* [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START
@ 2019-01-14 8:40 wentalou
[not found] ` <1547455236-30130-1-git-send-email-Wentao.Lou-5C7GfCeVMHo@public.gmane.org>
0 siblings, 1 reply; 25+ messages in thread
From: wentalou @ 2019-01-14 8:40 UTC (permalink / raw)
To: amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW; +Cc: wentalou
After removing unnecessary VM size calculations,
vm_manager.max_pfn would reach 0x10,0000,0000
max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding AMDGPU_GMC_HOLE_START
would caused GPU reset.
Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb
Signed-off-by: wentalou <Wentao.Lou@amd.com>
---
drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
index 7e22be7..dd3bd01 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c
@@ -26,7 +26,8 @@
uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev)
{
- uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT;
+ uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT,
+ AMDGPU_GMC_HOLE_START);
addr -= AMDGPU_VA_RESERVED_SIZE;
addr = amdgpu_gmc_sign_extend(addr);
--
2.7.4
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 25+ messages in thread[parent not found: <1547455236-30130-1-git-send-email-Wentao.Lou-5C7GfCeVMHo@public.gmane.org>]
* RE: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <1547455236-30130-1-git-send-email-Wentao.Lou-5C7GfCeVMHo@public.gmane.org> @ 2019-01-14 9:05 ` Liu, Monk 2019-01-14 13:05 ` Christian König 1 sibling, 0 replies; 25+ messages in thread From: Liu, Monk @ 2019-01-14 9:05 UTC (permalink / raw) To: amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW; +Cc: Lou, Wentao Good catch, patch is Reviewed-by: Monk Liu <monk.liu@amd.com> -----Original Message----- From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of wentalou Sent: Monday, January 14, 2019 4:41 PM To: amd-gfx@lists.freedesktop.org Cc: Lou, Wentao <Wentao.Lou@amd.com> Subject: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START After removing unnecessary VM size calculations, vm_manager.max_pfn would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding AMDGPU_GMC_HOLE_START would caused GPU reset. Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb Signed-off-by: wentalou <Wentao.Lou@amd.com> --- drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c index 7e22be7..dd3bd01 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c @@ -26,7 +26,8 @@ uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) { - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, + AMDGPU_GMC_HOLE_START); addr -= AMDGPU_VA_RESERVED_SIZE; addr = amdgpu_gmc_sign_extend(addr); -- 2.7.4 _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <1547455236-30130-1-git-send-email-Wentao.Lou-5C7GfCeVMHo@public.gmane.org> 2019-01-14 9:05 ` Liu, Monk @ 2019-01-14 13:05 ` Christian König [not found] ` <37b395b0-8503-238f-8775-66eb8795d23f-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 1 sibling, 1 reply; 25+ messages in thread From: Christian König @ 2019-01-14 13:05 UTC (permalink / raw) To: wentalou, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW Am 14.01.19 um 09:40 schrieb wentalou: > After removing unnecessary VM size calculations, > vm_manager.max_pfn would reach 0x10,0000,0000 > max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding AMDGPU_GMC_HOLE_START > would caused GPU reset. > > Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb > Signed-off-by: wentalou <Wentao.Lou@amd.com> NAK, that is incorrect. We intentionally place the csa above the GMC hole. Regards, Christian. > --- > drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c > index 7e22be7..dd3bd01 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c > @@ -26,7 +26,8 @@ > > uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) > { > - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; > + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, > + AMDGPU_GMC_HOLE_START); > > addr -= AMDGPU_VA_RESERVED_SIZE; > addr = amdgpu_gmc_sign_extend(addr); _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <37b395b0-8503-238f-8775-66eb8795d23f-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]
* RE: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <37b395b0-8503-238f-8775-66eb8795d23f-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> @ 2019-01-15 6:19 ` Liu, Monk [not found] ` <CY4PR1201MB0245AB79C8C8319D8DFB3A3884810-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org> 0 siblings, 1 reply; 25+ messages in thread From: Liu, Monk @ 2019-01-15 6:19 UTC (permalink / raw) To: Koenig, Christian, Lou, Wentao, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? Looks like we should modify this place /Monk -----Original Message----- From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Christian K?nig Sent: Monday, January 14, 2019 9:05 PM To: Lou, Wentao <Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 14.01.19 um 09:40 schrieb wentalou: > After removing unnecessary VM size calculations, vm_manager.max_pfn > would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding > AMDGPU_GMC_HOLE_START would caused GPU reset. > > Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb > Signed-off-by: wentalou <Wentao.Lou@amd.com> NAK, that is incorrect. We intentionally place the csa above the GMC hole. Regards, Christian. > --- > drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c > b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c > index 7e22be7..dd3bd01 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c > @@ -26,7 +26,8 @@ > > uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) > { > - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; > + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, > + AMDGPU_GMC_HOLE_START); > > addr -= AMDGPU_VA_RESERVED_SIZE; > addr = amdgpu_gmc_sign_extend(addr); _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <CY4PR1201MB0245AB79C8C8319D8DFB3A3884810-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org>]
* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <CY4PR1201MB0245AB79C8C8319D8DFB3A3884810-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org> @ 2019-01-15 8:02 ` Koenig, Christian [not found] ` <7f3f7b71-7acc-689a-e1d8-6fb184771f7b-5C7GfCeVMHo@public.gmane.org> 0 siblings, 1 reply; 25+ messages in thread From: Koenig, Christian @ 2019-01-15 8:02 UTC (permalink / raw) To: Liu, Monk, Lou, Wentao, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/amd-gfx _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <7f3f7b71-7acc-689a-e1d8-6fb184771f7b-5C7GfCeVMHo@public.gmane.org>]
* RE: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <7f3f7b71-7acc-689a-e1d8-6fb184771f7b-5C7GfCeVMHo@public.gmane.org> @ 2019-01-16 9:34 ` Lou, Wentao [not found] ` <BYAPR12MB2742181357E4D7DF7C171EE983820-ZGDeBxoHBPmbrehcvEBedAdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org> 0 siblings, 1 reply; 25+ messages in thread From: Lou, Wentao @ 2019-01-16 9:34 UTC (permalink / raw) To: Koenig, Christian, Liu, Monk, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 4410 bytes --] Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org; Zhu, Rex <Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 11364 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <BYAPR12MB2742181357E4D7DF7C171EE983820-ZGDeBxoHBPmbrehcvEBedAdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org>]
* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <BYAPR12MB2742181357E4D7DF7C171EE983820-ZGDeBxoHBPmbrehcvEBedAdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org> @ 2019-01-16 9:47 ` Koenig, Christian [not found] ` <574d6ca8-f1c8-8ac0-3dbe-23aed7c0125d-5C7GfCeVMHo@public.gmane.org> 0 siblings, 1 reply; 25+ messages in thread From: Koenig, Christian @ 2019-01-16 9:47 UTC (permalink / raw) To: Lou, Wentao, Liu, Monk, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 5159 bytes --] Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 12463 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <574d6ca8-f1c8-8ac0-3dbe-23aed7c0125d-5C7GfCeVMHo@public.gmane.org>]
* RE: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <574d6ca8-f1c8-8ac0-3dbe-23aed7c0125d-5C7GfCeVMHo@public.gmane.org> @ 2019-01-17 3:17 ` Lou, Wentao [not found] ` <BYAPR12MB27427158A5684A31BF96AB4783830-ZGDeBxoHBPmbrehcvEBedAdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org> 0 siblings, 1 reply; 25+ messages in thread From: Lou, Wentao @ 2019-01-17 3:17 UTC (permalink / raw) To: Koenig, Christian, Liu, Monk, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 6031 bytes --] Hi Christian, Your solution as: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = 0x10 0000, AMDGPU_PAGE_SHIFT = 12 Still got addr = 0xFFFF FFF0 0000, which would cause ring gfx timeout. Before commit 1bf621c42137926ac249af761c0190a9258aa0db, vm_size was 32GB, and csa_addr was under AMDGPU_GMC_HOLE_START. I didn’t understand why csa_addr need to be above AMDGPU_GMC_HOLE_START now. Thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com> Sent: Wednesday, January 16, 2019 5:48 PM To: Lou, Wentao <Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org; Zhu, Rex <Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 14621 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <BYAPR12MB27427158A5684A31BF96AB4783830-ZGDeBxoHBPmbrehcvEBedAdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org>]
* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <BYAPR12MB27427158A5684A31BF96AB4783830-ZGDeBxoHBPmbrehcvEBedAdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org> @ 2019-01-17 7:39 ` Koenig, Christian [not found] ` <361544e2-dd33-3b89-1211-4566b060b785-5C7GfCeVMHo@public.gmane.org> 0 siblings, 1 reply; 25+ messages in thread From: Koenig, Christian @ 2019-01-17 7:39 UTC (permalink / raw) To: Lou, Wentao, Liu, Monk, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 6653 bytes --] Am 17.01.19 um 04:17 schrieb Lou, Wentao: Hi Christian, Your solution as: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = 0x10 0000, AMDGPU_PAGE_SHIFT = 12 Still got addr = 0xFFFF FFF0 0000, which would cause ring gfx timeout. But 0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else. Please try to use AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as well. Does that work? Before commit 1bf621c42137926ac249af761c0190a9258aa0db, vm_size was 32GB, and csa_addr was under AMDGPU_GMC_HOLE_START. Wait a second why was the vm_size 32GB? This is on a Vega10 isn't it? I didn’t understand why csa_addr need to be above AMDGPU_GMC_HOLE_START now. On Vega10 the lower range, e.g. everything below AMDGPU_GMC_HOLE_START is reserved for SVA. Regards, Christian. Thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Wednesday, January 16, 2019 5:48 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 16344 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <361544e2-dd33-3b89-1211-4566b060b785-5C7GfCeVMHo@public.gmane.org>]
* RE: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <361544e2-dd33-3b89-1211-4566b060b785-5C7GfCeVMHo@public.gmane.org> @ 2019-01-17 8:04 ` Liu, Monk [not found] ` <CY4PR1201MB0245C5E3F6810C9157C9FDCF84830-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org> 0 siblings, 1 reply; 25+ messages in thread From: Liu, Monk @ 2019-01-17 8:04 UTC (permalink / raw) To: Koenig, Christian, Lou, Wentao, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 7873 bytes --] Hi Christian I believe Wentao can fix the issue we it by below step: 1. Return Virtual_address_max (UMD use it) to HOLE_START – RESERVED_SIZE 2. [optional] Still Keep virtual_address_offset to RESERVED_SIZE (current way, I think it’s because previously we put CSA in 0 --> RESERVED_SIZE space) 3. Put CSA in HOLE_START – RESERVED_SIZE ==> HOLE_START (it’s current design) I don’t get where above scheme is not correct … can you give more explain for the GMC_HOLE_START ? e.g. 1. why you set GMC_HOLE_START to 0x8’000’0000’0000 (half size of MAX of 48bit address space) ? is it for HSA purpose to make sure GPU address can also be used for CPU address ? 2. now MAX_PFN is 1’000’0000’0000, do you need to change GMC_HOLE_START ? thanks we need some catch up /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 3:39 PM To: Lou, Wentao <Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org; Zhu, Rex <Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 17.01.19 um 04:17 schrieb Lou, Wentao: Hi Christian, Your solution as: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = 0x10 0000, AMDGPU_PAGE_SHIFT = 12 Still got addr = 0xFFFF FFF0 0000, which would cause ring gfx timeout. But 0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else. Please try to use AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as well. Does that work? Before commit 1bf621c42137926ac249af761c0190a9258aa0db, vm_size was 32GB, and csa_addr was under AMDGPU_GMC_HOLE_START. Wait a second why was the vm_size 32GB? This is on a Vega10 isn't it? I didn’t understand why csa_addr need to be above AMDGPU_GMC_HOLE_START now. On Vega10 the lower range, e.g. everything below AMDGPU_GMC_HOLE_START is reserved for SVA. Regards, Christian. Thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Wednesday, January 16, 2019 5:48 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 22579 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <CY4PR1201MB0245C5E3F6810C9157C9FDCF84830-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org>]
* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <CY4PR1201MB0245C5E3F6810C9157C9FDCF84830-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org> @ 2019-01-17 8:30 ` Koenig, Christian [not found] ` <859dcf48-f4bf-d3ec-0893-369c7fad51ed-5C7GfCeVMHo@public.gmane.org> 0 siblings, 1 reply; 25+ messages in thread From: Koenig, Christian @ 2019-01-17 8:30 UTC (permalink / raw) To: Liu, Monk, Lou, Wentao, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 9343 bytes --] Hi Monk, ok let me explain a bit more how the hardware works. The GMC manages a virtual 64bit address space, but only 48bit of that virtual address space are handled by the page table walker. The 48bits of address space are sign extended, so bit 47 of that are extended into bits 48-63. This gives us the following memory layout: 0x0 .... virtual address space 0x8000 0000 0000 .... hole 0xFFFF 8000 0000 0000 .... virtual address space 0xFFFF FFFF FFFF FFFF Trying to access the hole results in a range fault interrupt IIRC. When doing the VM page table walk the topmost 16bits are ignored, so when programming the page table walker you cut those of and use a linear address again. This is what AMDGPU_GMC_HOLE_MASK is good for. Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. The most likely cause of this is that we still have a bug somewhere about this, e.g. not correctly using sign extended addresses *OR* using sign extended addresses where we should use linear instead. Regards, Christian. Am 17.01.19 um 09:04 schrieb Liu, Monk: Hi Christian I believe Wentao can fix the issue we it by below step: 1. Return Virtual_address_max (UMD use it) to HOLE_START – RESERVED_SIZE 2. [optional] Still Keep virtual_address_offset to RESERVED_SIZE (current way, I think it’s because previously we put CSA in 0 --> RESERVED_SIZE space) 3. Put CSA in HOLE_START – RESERVED_SIZE ==> HOLE_START (it’s current design) I don’t get where above scheme is not correct … can you give more explain for the GMC_HOLE_START ? e.g. 1. why you set GMC_HOLE_START to 0x8’000’0000’0000 (half size of MAX of 48bit address space) ? is it for HSA purpose to make sure GPU address can also be used for CPU address ? 2. now MAX_PFN is 1’000’0000’0000, do you need to change GMC_HOLE_START ? thanks we need some catch up /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 3:39 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 17.01.19 um 04:17 schrieb Lou, Wentao: Hi Christian, Your solution as: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = 0x10 0000, AMDGPU_PAGE_SHIFT = 12 Still got addr = 0xFFFF FFF0 0000, which would cause ring gfx timeout. But 0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else. Please try to use AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as well. Does that work? Before commit 1bf621c42137926ac249af761c0190a9258aa0db, vm_size was 32GB, and csa_addr was under AMDGPU_GMC_HOLE_START. Wait a second why was the vm_size 32GB? This is on a Vega10 isn't it? I didn’t understand why csa_addr need to be above AMDGPU_GMC_HOLE_START now. On Vega10 the lower range, e.g. everything below AMDGPU_GMC_HOLE_START is reserved for SVA. Regards, Christian. Thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Wednesday, January 16, 2019 5:48 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 24862 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <859dcf48-f4bf-d3ec-0893-369c7fad51ed-5C7GfCeVMHo@public.gmane.org>]
* RE: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <859dcf48-f4bf-d3ec-0893-369c7fad51ed-5C7GfCeVMHo@public.gmane.org> @ 2019-01-17 13:19 ` Liu, Monk [not found] ` <CY4PR1201MB0245C8076917864AA97A0FFA84830-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org> 2019-01-18 4:44 ` Lou, Wentao 1 sibling, 1 reply; 25+ messages in thread From: Liu, Monk @ 2019-01-17 13:19 UTC (permalink / raw) To: Koenig, Christian, Lou, Wentao, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 10510 bytes --] Hi Christian Thanks for explaining the HOLD for us, My understanding is we still could put CSA to 0~HOLE_START, because we can report UMD the max space is HOLD_START-CSA_SIZE , thus no colliding will hit. > Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 4:30 PM To: Liu, Monk <Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org; Zhu, Rex <Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, ok let me explain a bit more how the hardware works. The GMC manages a virtual 64bit address space, but only 48bit of that virtual address space are handled by the page table walker. The 48bits of address space are sign extended, so bit 47 of that are extended into bits 48-63. This gives us the following memory layout: 0x0 .... virtual address space 0x8000 0000 0000 .... hole 0xFFFF 8000 0000 0000 .... virtual address space 0xFFFF FFFF FFFF FFFF Trying to access the hole results in a range fault interrupt IIRC. When doing the VM page table walk the topmost 16bits are ignored, so when programming the page table walker you cut those of and use a linear address again. This is what AMDGPU_GMC_HOLE_MASK is good for. Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. The most likely cause of this is that we still have a bug somewhere about this, e.g. not correctly using sign extended addresses *OR* using sign extended addresses where we should use linear instead. Regards, Christian. Am 17.01.19 um 09:04 schrieb Liu, Monk: Hi Christian I believe Wentao can fix the issue we it by below step: 1. Return Virtual_address_max (UMD use it) to HOLE_START – RESERVED_SIZE 2. [optional] Still Keep virtual_address_offset to RESERVED_SIZE (current way, I think it’s because previously we put CSA in 0 --> RESERVED_SIZE space) 3. Put CSA in HOLE_START – RESERVED_SIZE ==> HOLE_START (it’s current design) I don’t get where above scheme is not correct … can you give more explain for the GMC_HOLE_START ? e.g. 1. why you set GMC_HOLE_START to 0x8’000’0000’0000 (half size of MAX of 48bit address space) ? is it for HSA purpose to make sure GPU address can also be used for CPU address ? 2. now MAX_PFN is 1’000’0000’0000, do you need to change GMC_HOLE_START ? thanks we need some catch up /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 3:39 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 17.01.19 um 04:17 schrieb Lou, Wentao: Hi Christian, Your solution as: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = 0x10 0000, AMDGPU_PAGE_SHIFT = 12 Still got addr = 0xFFFF FFF0 0000, which would cause ring gfx timeout. But 0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else. Please try to use AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as well. Does that work? Before commit 1bf621c42137926ac249af761c0190a9258aa0db, vm_size was 32GB, and csa_addr was under AMDGPU_GMC_HOLE_START. Wait a second why was the vm_size 32GB? This is on a Vega10 isn't it? I didn’t understand why csa_addr need to be above AMDGPU_GMC_HOLE_START now. On Vega10 the lower range, e.g. everything below AMDGPU_GMC_HOLE_START is reserved for SVA. Regards, Christian. Thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Wednesday, January 16, 2019 5:48 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 26939 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <CY4PR1201MB0245C8076917864AA97A0FFA84830-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org>]
* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <CY4PR1201MB0245C8076917864AA97A0FFA84830-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org> @ 2019-01-17 13:25 ` Koenig, Christian [not found] ` <cf3489f5-deb2-85ef-844a-b4c050f71d29-5C7GfCeVMHo@public.gmane.org> 0 siblings, 1 reply; 25+ messages in thread From: Koenig, Christian @ 2019-01-17 13:25 UTC (permalink / raw) To: Liu, Monk, Lou, Wentao, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 11524 bytes --] Hi Monk, Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Yes exactly. As I said the problem is that 0->HOLE_START is reserved for the ATC hardware, we should not touch it at all. Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? That turned out to be never a good idea in the first place. What we could do is reduce the max_pfn for SRIOV because the ATC doesn't work in that configuration anyway. But I would only do this as last resort. Any idea why an address above the hole doesn't work with SRIOV? It seems to work fine in the bare metal case. Regards, Christian. Am 17.01.19 um 14:19 schrieb Liu, Monk: Hi Christian Thanks for explaining the HOLD for us, My understanding is we still could put CSA to 0~HOLE_START, because we can report UMD the max space is HOLD_START-CSA_SIZE , thus no colliding will hit. > Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 4:30 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, ok let me explain a bit more how the hardware works. The GMC manages a virtual 64bit address space, but only 48bit of that virtual address space are handled by the page table walker. The 48bits of address space are sign extended, so bit 47 of that are extended into bits 48-63. This gives us the following memory layout: 0x0 .... virtual address space 0x8000 0000 0000 .... hole 0xFFFF 8000 0000 0000 .... virtual address space 0xFFFF FFFF FFFF FFFF Trying to access the hole results in a range fault interrupt IIRC. When doing the VM page table walk the topmost 16bits are ignored, so when programming the page table walker you cut those of and use a linear address again. This is what AMDGPU_GMC_HOLE_MASK is good for. Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. The most likely cause of this is that we still have a bug somewhere about this, e.g. not correctly using sign extended addresses *OR* using sign extended addresses where we should use linear instead. Regards, Christian. Am 17.01.19 um 09:04 schrieb Liu, Monk: Hi Christian I believe Wentao can fix the issue we it by below step: 1. Return Virtual_address_max (UMD use it) to HOLE_START – RESERVED_SIZE 2. [optional] Still Keep virtual_address_offset to RESERVED_SIZE (current way, I think it’s because previously we put CSA in 0 --> RESERVED_SIZE space) 3. Put CSA in HOLE_START – RESERVED_SIZE ==> HOLE_START (it’s current design) I don’t get where above scheme is not correct … can you give more explain for the GMC_HOLE_START ? e.g. 1. why you set GMC_HOLE_START to 0x8’000’0000’0000 (half size of MAX of 48bit address space) ? is it for HSA purpose to make sure GPU address can also be used for CPU address ? 2. now MAX_PFN is 1’000’0000’0000, do you need to change GMC_HOLE_START ? thanks we need some catch up /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 3:39 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 17.01.19 um 04:17 schrieb Lou, Wentao: Hi Christian, Your solution as: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = 0x10 0000, AMDGPU_PAGE_SHIFT = 12 Still got addr = 0xFFFF FFF0 0000, which would cause ring gfx timeout. But 0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else. Please try to use AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as well. Does that work? Before commit 1bf621c42137926ac249af761c0190a9258aa0db, vm_size was 32GB, and csa_addr was under AMDGPU_GMC_HOLE_START. Wait a second why was the vm_size 32GB? This is on a Vega10 isn't it? I didn’t understand why csa_addr need to be above AMDGPU_GMC_HOLE_START now. On Vega10 the lower range, e.g. everything below AMDGPU_GMC_HOLE_START is reserved for SVA. Regards, Christian. Thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Wednesday, January 16, 2019 5:48 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 29026 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <cf3489f5-deb2-85ef-844a-b4c050f71d29-5C7GfCeVMHo@public.gmane.org>]
* RE: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <cf3489f5-deb2-85ef-844a-b4c050f71d29-5C7GfCeVMHo@public.gmane.org> @ 2019-01-18 1:32 ` Liu, Monk [not found] ` <CY4PR1201MB02457ECA700E3AE8B321AC90849C0-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org> 0 siblings, 1 reply; 25+ messages in thread From: Liu, Monk @ 2019-01-18 1:32 UTC (permalink / raw) To: Koenig, Christian, Lou, Wentao, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 13011 bytes --] Thanks Christian, Questions I have now: 1. You see that for UMD, it can use 0 to HOLE_START, so why CSA cannot use that range although the range is as you said reserved to ATC h/w ? Be note that for windows KMD, the CSA is allocated by UMD driver so CSA shares the same aperture /space range with other UMD BO, which mean CSA in windows also located in ATC range, if that’s a problem why windows still works well. * Can you illustrate this limitation with more details ? we need to understand why CSA couldn’t be put in ATC range. 2. According to your previous description :” Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older user space as usable and also put the CSA in there.” * Do you mean even UMD should not use virtual address that dropped in range from 0 to HOLE_START ? that way where should UMD work in ?and I assume our UMD now still using this range, this part make me puzzle /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 9:26 PM To: Liu, Monk <Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org; Zhu, Rex <Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Yes exactly. As I said the problem is that 0->HOLE_START is reserved for the ATC hardware, we should not touch it at all. Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? That turned out to be never a good idea in the first place. What we could do is reduce the max_pfn for SRIOV because the ATC doesn't work in that configuration anyway. But I would only do this as last resort. Any idea why an address above the hole doesn't work with SRIOV? It seems to work fine in the bare metal case. Regards, Christian. Am 17.01.19 um 14:19 schrieb Liu, Monk: Hi Christian Thanks for explaining the HOLD for us, My understanding is we still could put CSA to 0~HOLE_START, because we can report UMD the max space is HOLD_START-CSA_SIZE , thus no colliding will hit. > Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 4:30 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, ok let me explain a bit more how the hardware works. The GMC manages a virtual 64bit address space, but only 48bit of that virtual address space are handled by the page table walker. The 48bits of address space are sign extended, so bit 47 of that are extended into bits 48-63. This gives us the following memory layout: 0x0 .... virtual address space 0x8000 0000 0000 .... hole 0xFFFF 8000 0000 0000 .... virtual address space 0xFFFF FFFF FFFF FFFF Trying to access the hole results in a range fault interrupt IIRC. When doing the VM page table walk the topmost 16bits are ignored, so when programming the page table walker you cut those of and use a linear address again. This is what AMDGPU_GMC_HOLE_MASK is good for. Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. The most likely cause of this is that we still have a bug somewhere about this, e.g. not correctly using sign extended addresses *OR* using sign extended addresses where we should use linear instead. Regards, Christian. Am 17.01.19 um 09:04 schrieb Liu, Monk: Hi Christian I believe Wentao can fix the issue we it by below step: 1. Return Virtual_address_max (UMD use it) to HOLE_START – RESERVED_SIZE 2. [optional] Still Keep virtual_address_offset to RESERVED_SIZE (current way, I think it’s because previously we put CSA in 0 --> RESERVED_SIZE space) 3. Put CSA in HOLE_START – RESERVED_SIZE ==> HOLE_START (it’s current design) I don’t get where above scheme is not correct … can you give more explain for the GMC_HOLE_START ? e.g. 1. why you set GMC_HOLE_START to 0x8’000’0000’0000 (half size of MAX of 48bit address space) ? is it for HSA purpose to make sure GPU address can also be used for CPU address ? 2. now MAX_PFN is 1’000’0000’0000, do you need to change GMC_HOLE_START ? thanks we need some catch up /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 3:39 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 17.01.19 um 04:17 schrieb Lou, Wentao: Hi Christian, Your solution as: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = 0x10 0000, AMDGPU_PAGE_SHIFT = 12 Still got addr = 0xFFFF FFF0 0000, which would cause ring gfx timeout. But 0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else. Please try to use AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as well. Does that work? Before commit 1bf621c42137926ac249af761c0190a9258aa0db, vm_size was 32GB, and csa_addr was under AMDGPU_GMC_HOLE_START. Wait a second why was the vm_size 32GB? This is on a Vega10 isn't it? I didn’t understand why csa_addr need to be above AMDGPU_GMC_HOLE_START now. On Vega10 the lower range, e.g. everything below AMDGPU_GMC_HOLE_START is reserved for SVA. Regards, Christian. Thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Wednesday, January 16, 2019 5:48 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 33164 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <CY4PR1201MB02457ECA700E3AE8B321AC90849C0-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org>]
* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <CY4PR1201MB02457ECA700E3AE8B321AC90849C0-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org> @ 2019-01-18 9:11 ` Christian König [not found] ` <c83cd7ec-03a5-45c0-b497-d5a5ec838880-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 0 siblings, 1 reply; 25+ messages in thread From: Christian König @ 2019-01-18 9:11 UTC (permalink / raw) To: Liu, Monk, Koenig, Christian, Lou, Wentao, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 19405 bytes --] Hi Monk, > You see that for UMD, it can use 0 to HOLE_START Let me say it once more: The UMD nor anybody else CAN'T use 0 to HOLE_START, that region is reserved for the ATC hardware! We unfortunately didn't knew that initially and also didn't used the ATC, so we didn't ran into a problem. But ROCm now uses the ATC on Raven/Picasso and I have a branch where I enable it on Vega as well. So when we don't fix that we will run into problems here. The ATC isn't usable in combination with SRIOV and I don't think Windows uses it either, so they probably never ran into any issues. > Do you mean even UMD should not use virtual address that dropped in > range from 0 to HOLE_START ? Yes, exactly! That in combination with ATC use can have quite a bunch of strange and hard to track down effects because two parts of the driver are using the same address space. > That way where should UMD work in ?and I assume our UMD now still > using this range, this part make me puzzle At least Mesa now uses the high address space from HOLE_END..0xFFFF FFFF FFFF FFFF. Regards, Christian. Am 18.01.19 um 02:32 schrieb Liu, Monk: > > Thanks Christian, > > Questions I have now: > > 1. You see that for UMD, it can use 0 to HOLE_START, so why CSA > cannot use that range although the range is as you said reserved > to ATC h/w ? Be note that for windows KMD, the CSA is allocated by > UMD driver so CSA shares the same aperture /space range with other > UMD BO, which mean CSA in windows also located in ATC range, if > that’s a problem why windows still works well. > 1. Can you illustrate this limitation with more details ? we need > to understand why CSA couldn’t be put in ATC range. > 2. According to your previous description :” Now on > Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range > (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we > *unfortunately didn't knew that initially we exposed those to > older user space as usable* and also put the CSA in there.” > 1. Do you mean even UMD should not use virtual address that > dropped in range from 0 to HOLE_START ? > > that way where should UMD work in ?and I assume our UMD now still > using this range, this part make me puzzle > > /Monk > > *From:*amd-gfx <amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> *On Behalf Of > *Koenig, Christian > *Sent:* Thursday, January 17, 2019 9:26 PM > *To:* Liu, Monk <Monk.Liu-5C7GfCeVMHo@public.gmane.org>; Lou, Wentao <Wentao.Lou-5C7GfCeVMHo@public.gmane.org>; > amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org; Zhu, Rex <Rex.Zhu-5C7GfCeVMHo@public.gmane.org> > *Cc:* Deng, Emily <Emily.Deng-5C7GfCeVMHo@public.gmane.org> > *Subject:* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Hi Monk, > > > Regarding with above sentence, do you mean this range > (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get > your point here … > > Yes exactly. As I said the problem is that 0->HOLE_START is reserved > for the ATC hardware, we should not touch it at all. > > > Putting CSA in 0~HOLD_START is the legacy approach we selected for > a long time since very early stage, how comes that you think it is > a problem now ? > > That turned out to be never a good idea in the first place. > > What we could do is reduce the max_pfn for SRIOV because the ATC > doesn't work in that configuration anyway. But I would only do this as > last resort. > > Any idea why an address above the hole doesn't work with SRIOV? It > seems to work fine in the bare metal case. > > Regards, > Christian. > > Am 17.01.19 um 14:19 schrieb Liu, Monk: > > Hi Christian > > Thanks for explaining the HOLD for us, > > My understanding is we still could put CSA to 0~HOLE_START, > because we can report UMD the max space is HOLD_START-CSA_SIZE , > thus no colliding will hit. > > > Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the > lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. > Since we unfortunately didn't knew that initially we exposed those > to older userspace as usable and also put the CSA in there. > > > Regarding with above sentence, do you mean this range > (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get > your point here … > > Putting CSA in 0~HOLD_START is the legacy approach we selected for > a long time since very early stage, how comes that you think it is > a problem now ? > > /Monk > > *From:*amd-gfx <amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> > <mailto:amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> *On Behalf Of > *Koenig, Christian > *Sent:* Thursday, January 17, 2019 4:30 PM > *To:* Liu, Monk <Monk.Liu-5C7GfCeVMHo@public.gmane.org> <mailto:Monk.Liu-5C7GfCeVMHo@public.gmane.org>; Lou, > Wentao <Wentao.Lou-5C7GfCeVMHo@public.gmane.org> <mailto:Wentao.Lou-5C7GfCeVMHo@public.gmane.org>; > amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org > <mailto:amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org>; Zhu, Rex <Rex.Zhu-5C7GfCeVMHo@public.gmane.org> > <mailto:Rex.Zhu-5C7GfCeVMHo@public.gmane.org> > *Cc:* Deng, Emily <Emily.Deng-5C7GfCeVMHo@public.gmane.org> <mailto:Emily.Deng-5C7GfCeVMHo@public.gmane.org> > *Subject:* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger > than AMDGPU_GMC_HOLE_START > > Hi Monk, > > ok let me explain a bit more how the hardware works. > > The GMC manages a virtual 64bit address space, but only 48bit of > that virtual address space are handled by the page table walker. > > The 48bits of address space are sign extended, so bit 47 of that > are extended into bits 48-63. > > This gives us the following memory layout: > 0x0 > .... virtual address space > 0x8000 0000 0000 > .... hole > 0xFFFF 8000 0000 0000 > .... virtual address space > 0xFFFF FFFF FFFF FFFF > > Trying to access the hole results in a range fault interrupt IIRC. > > When doing the VM page table walk the topmost 16bits are ignored, > so when programming the page table walker you cut those of and use > a linear address again. This is what AMDGPU_GMC_HOLE_MASK is good for. > > Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower > range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we > unfortunately didn't knew that initially we exposed those to older > userspace as usable and also put the CSA in there. > > The most likely cause of this is that we still have a bug > somewhere about this, e.g. not correctly using sign extended > addresses *OR* using sign extended addresses where we should use > linear instead. > > Regards, > Christian. > > Am 17.01.19 um 09:04 schrieb Liu, Monk: > > Hi Christian > > I believe Wentao can fix the issue we it by below step: > > 1. Return *Virtual_address_max* (UMD use it) to HOLE_START – > RESERVED_SIZE > 2. [optional] Still Keep virtual_address_offset to > RESERVED_SIZE (current way, I think it’s because > previously we put CSA in 0 à RESERVED_SIZE space) > 3. Put CSA in HOLE_START – RESERVED_SIZE è HOLE_START (it’s > current design) > > I don’t get where above scheme is not correct … can you give > more explain for the GMC_HOLE_START ? > > e.g. > > 1. why you set GMC_HOLE_START to 0x8’000’0000’0000 (half size > of MAX of 48bit address space) ? is it for HSA purpose to > make sure GPU address can also be used for CPU address ? > 2. now MAX_PFN is 1’000’0000’0000, do you need to change > GMC_HOLE_START ? > > thanks > > we need some catch up > > /Monk > > *From:*amd-gfx <amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> > <mailto:amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> *On Behalf Of > *Koenig, Christian > *Sent:* Thursday, January 17, 2019 3:39 PM > *To:* Lou, Wentao <Wentao.Lou-5C7GfCeVMHo@public.gmane.org> > <mailto:Wentao.Lou-5C7GfCeVMHo@public.gmane.org>; Liu, Monk <Monk.Liu-5C7GfCeVMHo@public.gmane.org> > <mailto:Monk.Liu-5C7GfCeVMHo@public.gmane.org>; amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org > <mailto:amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org>; Zhu, Rex > <Rex.Zhu-5C7GfCeVMHo@public.gmane.org> <mailto:Rex.Zhu-5C7GfCeVMHo@public.gmane.org> > *Cc:* Deng, Emily <Emily.Deng-5C7GfCeVMHo@public.gmane.org> <mailto:Emily.Deng-5C7GfCeVMHo@public.gmane.org> > *Subject:* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger > than AMDGPU_GMC_HOLE_START > > Am 17.01.19 um 04:17 schrieb Lou, Wentao: > > Hi Christian, > > Your solution as: > > addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> > AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; > > now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = > 0x10 0000, AMDGPU_PAGE_SHIFT = 12 > > Still got addr = 0xFFFF FFF0 0000, which would cause ring > gfx timeout. > > > But 0xFFFF FFF0 0000 is the correct address, if that is > causing a problem then there is a bug somewhere else. > > Please try to use > AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as well. Does > that work? > > > > > Before commit 1bf621c42137926ac249af761c0190a9258aa0db, > vm_size was 32GB, and csa_addr was under > AMDGPU_GMC_HOLE_START. > > > Wait a second why was the vm_size 32GB? This is on a Vega10 > isn't it? > > > > > I didn’t understand why csa_addr need to be above > AMDGPU_GMC_HOLE_START now. > > > On Vega10 the lower range, e.g. everything below > AMDGPU_GMC_HOLE_START is reserved for SVA. > > Regards, > Christian. > > > > > Thanks. > > BR, > > Wentao > > *From:* Koenig, Christian <Christian.Koenig-5C7GfCeVMHo@public.gmane.org> > <mailto:Christian.Koenig-5C7GfCeVMHo@public.gmane.org> > *Sent:* Wednesday, January 16, 2019 5:48 PM > *To:* Lou, Wentao <Wentao.Lou-5C7GfCeVMHo@public.gmane.org> > <mailto:Wentao.Lou-5C7GfCeVMHo@public.gmane.org>; Liu, Monk <Monk.Liu-5C7GfCeVMHo@public.gmane.org> > <mailto:Monk.Liu-5C7GfCeVMHo@public.gmane.org>; amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org > <mailto:amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org>; Zhu, Rex > <Rex.Zhu-5C7GfCeVMHo@public.gmane.org> <mailto:Rex.Zhu-5C7GfCeVMHo@public.gmane.org> > *Cc:* Deng, Emily <Emily.Deng-5C7GfCeVMHo@public.gmane.org> > <mailto:Emily.Deng-5C7GfCeVMHo@public.gmane.org> > *Subject:* Re: [PATCH] drm/amdgpu: csa_vaddr should not > larger than AMDGPU_GMC_HOLE_START > > Hi Wentao, > > well the problem is you don't seem to understand how the > hardware works. > > See the engines see an MC address space with a hole in the > middle, similar to the how x86 64bit CPU address space > works. But the page tables are programmed linearly. > > So the calculation in amdgpu_driver_open_kms() is correct > because it takes the MC address and mages a linear page > table index from it again. > > The only thing we might need to fix here is shifting > max_pfn before the subtraction and I doubt that even that > is necessary. > > Regards, > Christian. > > Am 16.01.19 um 10:34 schrieb Lou, Wentao: > > Hi Christian, > > Now vm_size was set to 0x4 0000 GB by below commit: > > 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: > Remove unnecessary VM size calculations > > So that max_pfn would be 0x10 0000 0000. > > amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 > 0000 0000 0000, and then minus > AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 > > unfortunately this number was between > AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that > amdgpu_gmc_sign_extend was called to make it 0xFFFF > FFFF FFF0 0000 > > in amdgpu_driver_open_kms, extended csa_addr cannot be > passed into amdgpu_map_static_csa directly, it would > be above the limit of max_pfn. > > So that csa_addr was restricted by > AMDGPU_GMC_HOLE_MASK to make it possible for > amdgpu_vm_alloc_pts. > > But this restriction by AMDGPU_GMC_HOLE_MASK would > make the address fall back into AMDGPU_GMC_HOLE again, > which causing GPU reset. > > We just put amdgpu_csa_vaddr back to > AMDGPU_GMC_HOLE_START, to avoid the address touching > AMDGPU_GMC_HOLE. > > By the way, if max_pfn was shift much to the left, it > would always get zero, with or without min(*,*). > > BR, > > Wentao > > -----Original Message----- > From: Koenig, Christian <Christian.Koenig-5C7GfCeVMHo@public.gmane.org> > <mailto:Christian.Koenig-5C7GfCeVMHo@public.gmane.org> > Sent: Tuesday, January 15, 2019 4:02 PM > To: Liu, Monk <Monk.Liu-5C7GfCeVMHo@public.gmane.org> > <mailto:Monk.Liu-5C7GfCeVMHo@public.gmane.org>; Lou, Wentao > <Wentao.Lou-5C7GfCeVMHo@public.gmane.org> <mailto:Wentao.Lou-5C7GfCeVMHo@public.gmane.org>; > amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org > <mailto:amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org>; Zhu, Rex > <Rex.Zhu-5C7GfCeVMHo@public.gmane.org> <mailto:Rex.Zhu-5C7GfCeVMHo@public.gmane.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not > larger than AMDGPU_GMC_HOLE_START > > Am 15.01.19 um 07:19 schrieb Liu, Monk: > > > The max_pfn is now 1'0000'0000'0000'0000 (bytes) > which is above 48 bit now, and it with > AMDGPU_GMC_HOLE_MASK make it to zero .... > > > > > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, > Rex write the code as : > > > > > > "csa_addr = amdgpu_csa_vadr(adev) & > AMDGPU_GMC_HOLE_MASK", I think this is wrong since you > intentionally place the csa above GMC hole, right ? > > The fix is just completely incorrect since > min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, > AMDGPU_GMC_HOLE_START) still gives you 0 when we shift > max_pfn to much to the left. > > The correct solution is to substract the reserved size > first and then shift. E.g.: > > addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> > AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; > > Regards, > > Christian. > > > > > > Looks like we should modify this place > > > > > > /Monk > > > > > > -----Original Message----- > > > From: amd-gfx <amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org > <mailto:amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org>> On > Behalf Of > > > Christian K?nig > > > Sent: Monday, January 14, 2019 9:05 PM > > > To: Lou, Wentao <Wentao.Lou-5C7GfCeVMHo@public.gmane.org > <mailto:Wentao.Lou-5C7GfCeVMHo@public.gmane.org>>; > amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org > <mailto:amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> > > > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should > not larger than > > > AMDGPU_GMC_HOLE_START > > > > > > Am 14.01.19 um 09:40 schrieb wentalou: > > >> After removing unnecessary VM size calculations, > vm_manager.max_pfn > > >> would reach 0x10,0000,0000 max_pfn << > AMDGPU_GPU_PAGE_SHIFT exceeding > > >> AMDGPU_GMC_HOLE_START would caused GPU reset. > > >> > > >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb > > >> Signed-off-by: wentalou <Wentao.Lou-5C7GfCeVMHo@public.gmane.org > <mailto:Wentao.Lou-5C7GfCeVMHo@public.gmane.org>> > > > NAK, that is incorrect. We intentionally place the > csa above the GMC hole. > > > > > > Regards, > > > Christian. > > > > > >> --- > > >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- > > >> 1 file changed, 2 insertions(+), 1 deletion(-) > > >> > > >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c > > >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c > > >> index 7e22be7..dd3bd01 100644 > > >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c > > >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c > > >> @@ -26,7 +26,8 @@ > > >> > > >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device > *adev) > > >> { > > >> - uint64_t addr = adev->vm_manager.max_pfn > << AMDGPU_GPU_PAGE_SHIFT; > > >> + uint64_t addr = > min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, > > >> + AMDGPU_GMC_HOLE_START); > > >> > > >> addr -= AMDGPU_VA_RESERVED_SIZE; > > >> addr = amdgpu_gmc_sign_extend(addr); > > > _______________________________________________ > > > amd-gfx mailing list > > > amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org > <mailto:amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> > > > https://lists.freedesktop.org/mailman/listinfo/amd-gfx > > > _______________________________________________ > amd-gfx mailing list > amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org > https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 46208 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <c83cd7ec-03a5-45c0-b497-d5a5ec838880-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]
* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <c83cd7ec-03a5-45c0-b497-d5a5ec838880-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> @ 2019-01-18 9:29 ` Chunming Zhou [not found] ` <effb374a-9803-d451-41c4-991a2cd0aa9a-5C7GfCeVMHo@public.gmane.org> 2019-01-18 10:57 ` Liu, Monk 1 sibling, 1 reply; 25+ messages in thread From: Chunming Zhou @ 2019-01-18 9:29 UTC (permalink / raw) To: Koenig, Christian, Liu, Monk, Lou, Wentao, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 15010 bytes --] 在 2019/1/18 17:11, Christian König 写道: Hi Monk, You see that for UMD, it can use 0 to HOLE_START Let me say it once more: The UMD nor anybody else CAN'T use 0 to HOLE_START, that region is reserved for the ATC hardware! We unfortunately didn't knew that initially and also didn't used the ATC, so we didn't ran into a problem. But ROCm now uses the ATC on Raven/Picasso and I have a branch where I enable it on Vega as well. So when we don't fix that we will run into problems here. The ATC isn't usable in combination with SRIOV and I don't think Windows uses it either, so they probably never ran into any issues. Do you mean even UMD should not use virtual address that dropped in range from 0 to HOLE_START ? Yes, exactly! That in combination with ATC use can have quite a bunch of strange and hard to track down effects because two parts of the driver are using the same address space. That way where should UMD work in ?and I assume our UMD now still using this range, this part make me puzzle At least Mesa now uses the high address space from HOLE_END..0xFFFF FFFF FFFF FFFF. That is why there is high_vamgr in libdrm. The extended va will be masked in kernel when doing vm mapping, right? If mesa already switches to use that, shouldn't internal opengl and vulkan switch to? -David Regards, Christian. Am 18.01.19 um 02:32 schrieb Liu, Monk: Thanks Christian, Questions I have now: 1. You see that for UMD, it can use 0 to HOLE_START, so why CSA cannot use that range although the range is as you said reserved to ATC h/w ? Be note that for windows KMD, the CSA is allocated by UMD driver so CSA shares the same aperture /space range with other UMD BO, which mean CSA in windows also located in ATC range, if that’s a problem why windows still works well. * Can you illustrate this limitation with more details ? we need to understand why CSA couldn’t be put in ATC range. 2. According to your previous description :” Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older user space as usable and also put the CSA in there.” * Do you mean even UMD should not use virtual address that dropped in range from 0 to HOLE_START ? that way where should UMD work in ?and I assume our UMD now still using this range, this part make me puzzle /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 9:26 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Yes exactly. As I said the problem is that 0->HOLE_START is reserved for the ATC hardware, we should not touch it at all. Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? That turned out to be never a good idea in the first place. What we could do is reduce the max_pfn for SRIOV because the ATC doesn't work in that configuration anyway. But I would only do this as last resort. Any idea why an address above the hole doesn't work with SRIOV? It seems to work fine in the bare metal case. Regards, Christian. Am 17.01.19 um 14:19 schrieb Liu, Monk: Hi Christian Thanks for explaining the HOLD for us, My understanding is we still could put CSA to 0~HOLE_START, because we can report UMD the max space is HOLD_START-CSA_SIZE , thus no colliding will hit. > Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 4:30 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, ok let me explain a bit more how the hardware works. The GMC manages a virtual 64bit address space, but only 48bit of that virtual address space are handled by the page table walker. The 48bits of address space are sign extended, so bit 47 of that are extended into bits 48-63. This gives us the following memory layout: 0x0 .... virtual address space 0x8000 0000 0000 .... hole 0xFFFF 8000 0000 0000 .... virtual address space 0xFFFF FFFF FFFF FFFF Trying to access the hole results in a range fault interrupt IIRC. When doing the VM page table walk the topmost 16bits are ignored, so when programming the page table walker you cut those of and use a linear address again. This is what AMDGPU_GMC_HOLE_MASK is good for. Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. The most likely cause of this is that we still have a bug somewhere about this, e.g. not correctly using sign extended addresses *OR* using sign extended addresses where we should use linear instead. Regards, Christian. Am 17.01.19 um 09:04 schrieb Liu, Monk: Hi Christian I believe Wentao can fix the issue we it by below step: 1. Return Virtual_address_max (UMD use it) to HOLE_START – RESERVED_SIZE 2. [optional] Still Keep virtual_address_offset to RESERVED_SIZE (current way, I think it’s because previously we put CSA in 0 --> RESERVED_SIZE space) 3. Put CSA in HOLE_START – RESERVED_SIZE ==> HOLE_START (it’s current design) I don’t get where above scheme is not correct … can you give more explain for the GMC_HOLE_START ? e.g. 1. why you set GMC_HOLE_START to 0x8’000’0000’0000 (half size of MAX of 48bit address space) ? is it for HSA purpose to make sure GPU address can also be used for CPU address ? 2. now MAX_PFN is 1’000’0000’0000, do you need to change GMC_HOLE_START ? thanks we need some catch up /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 3:39 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 17.01.19 um 04:17 schrieb Lou, Wentao: Hi Christian, Your solution as: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = 0x10 0000, AMDGPU_PAGE_SHIFT = 12 Still got addr = 0xFFFF FFF0 0000, which would cause ring gfx timeout. But 0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else. Please try to use AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as well. Does that work? Before commit 1bf621c42137926ac249af761c0190a9258aa0db, vm_size was 32GB, and csa_addr was under AMDGPU_GMC_HOLE_START. Wait a second why was the vm_size 32GB? This is on a Vega10 isn't it? I didn’t understand why csa_addr need to be above AMDGPU_GMC_HOLE_START now. On Vega10 the lower range, e.g. everything below AMDGPU_GMC_HOLE_START is reserved for SVA. Regards, Christian. Thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Wednesday, January 16, 2019 5:48 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> https://lists.freedesktop.org/mailman/listinfo/amd-gfx _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 37471 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <effb374a-9803-d451-41c4-991a2cd0aa9a-5C7GfCeVMHo@public.gmane.org>]
* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <effb374a-9803-d451-41c4-991a2cd0aa9a-5C7GfCeVMHo@public.gmane.org> @ 2019-01-18 10:39 ` Koenig, Christian 0 siblings, 0 replies; 25+ messages in thread From: Koenig, Christian @ 2019-01-18 10:39 UTC (permalink / raw) To: Zhou, David(ChunMing), Liu, Monk, Lou, Wentao, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 15247 bytes --] Am 18.01.19 um 10:29 schrieb Zhou, David(ChunMing): 在 2019/1/18 17:11, Christian König 写道: Hi Monk, You see that for UMD, it can use 0 to HOLE_START Let me say it once more: The UMD nor anybody else CAN'T use 0 to HOLE_START, that region is reserved for the ATC hardware! We unfortunately didn't knew that initially and also didn't used the ATC, so we didn't ran into a problem. But ROCm now uses the ATC on Raven/Picasso and I have a branch where I enable it on Vega as well. So when we don't fix that we will run into problems here. The ATC isn't usable in combination with SRIOV and I don't think Windows uses it either, so they probably never ran into any issues. Do you mean even UMD should not use virtual address that dropped in range from 0 to HOLE_START ? Yes, exactly! That in combination with ATC use can have quite a bunch of strange and hard to track down effects because two parts of the driver are using the same address space. That way where should UMD work in ?and I assume our UMD now still using this range, this part make me puzzle At least Mesa now uses the high address space from HOLE_END..0xFFFF FFFF FFFF FFFF. That is why there is high_vamgr in libdrm. The extended va will be masked in kernel when doing vm mapping, right? Yes, exactly. If mesa already switches to use that, shouldn't internal opengl and vulkan switch to? That is most likely a good idea. We should probably deprecate using the lower range completely for the next hardware generation. Regards, Christian. -David Regards, Christian. Am 18.01.19 um 02:32 schrieb Liu, Monk: Thanks Christian, Questions I have now: 1. You see that for UMD, it can use 0 to HOLE_START, so why CSA cannot use that range although the range is as you said reserved to ATC h/w ? Be note that for windows KMD, the CSA is allocated by UMD driver so CSA shares the same aperture /space range with other UMD BO, which mean CSA in windows also located in ATC range, if that’s a problem why windows still works well. * Can you illustrate this limitation with more details ? we need to understand why CSA couldn’t be put in ATC range. 2. According to your previous description :” Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older user space as usable and also put the CSA in there.” * Do you mean even UMD should not use virtual address that dropped in range from 0 to HOLE_START ? that way where should UMD work in ?and I assume our UMD now still using this range, this part make me puzzle /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 9:26 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Yes exactly. As I said the problem is that 0->HOLE_START is reserved for the ATC hardware, we should not touch it at all. Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? That turned out to be never a good idea in the first place. What we could do is reduce the max_pfn for SRIOV because the ATC doesn't work in that configuration anyway. But I would only do this as last resort. Any idea why an address above the hole doesn't work with SRIOV? It seems to work fine in the bare metal case. Regards, Christian. Am 17.01.19 um 14:19 schrieb Liu, Monk: Hi Christian Thanks for explaining the HOLD for us, My understanding is we still could put CSA to 0~HOLE_START, because we can report UMD the max space is HOLD_START-CSA_SIZE , thus no colliding will hit. > Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 4:30 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, ok let me explain a bit more how the hardware works. The GMC manages a virtual 64bit address space, but only 48bit of that virtual address space are handled by the page table walker. The 48bits of address space are sign extended, so bit 47 of that are extended into bits 48-63. This gives us the following memory layout: 0x0 .... virtual address space 0x8000 0000 0000 .... hole 0xFFFF 8000 0000 0000 .... virtual address space 0xFFFF FFFF FFFF FFFF Trying to access the hole results in a range fault interrupt IIRC. When doing the VM page table walk the topmost 16bits are ignored, so when programming the page table walker you cut those of and use a linear address again. This is what AMDGPU_GMC_HOLE_MASK is good for. Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. The most likely cause of this is that we still have a bug somewhere about this, e.g. not correctly using sign extended addresses *OR* using sign extended addresses where we should use linear instead. Regards, Christian. Am 17.01.19 um 09:04 schrieb Liu, Monk: Hi Christian I believe Wentao can fix the issue we it by below step: 1. Return Virtual_address_max (UMD use it) to HOLE_START – RESERVED_SIZE 2. [optional] Still Keep virtual_address_offset to RESERVED_SIZE (current way, I think it’s because previously we put CSA in 0 --> RESERVED_SIZE space) 3. Put CSA in HOLE_START – RESERVED_SIZE ==> HOLE_START (it’s current design) I don’t get where above scheme is not correct … can you give more explain for the GMC_HOLE_START ? e.g. 1. why you set GMC_HOLE_START to 0x8’000’0000’0000 (half size of MAX of 48bit address space) ? is it for HSA purpose to make sure GPU address can also be used for CPU address ? 2. now MAX_PFN is 1’000’0000’0000, do you need to change GMC_HOLE_START ? thanks we need some catch up /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 3:39 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 17.01.19 um 04:17 schrieb Lou, Wentao: Hi Christian, Your solution as: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = 0x10 0000, AMDGPU_PAGE_SHIFT = 12 Still got addr = 0xFFFF FFF0 0000, which would cause ring gfx timeout. But 0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else. Please try to use AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as well. Does that work? Before commit 1bf621c42137926ac249af761c0190a9258aa0db, vm_size was 32GB, and csa_addr was under AMDGPU_GMC_HOLE_START. Wait a second why was the vm_size 32GB? This is on a Vega10 isn't it? I didn’t understand why csa_addr need to be above AMDGPU_GMC_HOLE_START now. On Vega10 the lower range, e.g. everything below AMDGPU_GMC_HOLE_START is reserved for SVA. Regards, Christian. Thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Wednesday, January 16, 2019 5:48 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> https://lists.freedesktop.org/mailman/listinfo/amd-gfx _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 38165 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
* RE: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <c83cd7ec-03a5-45c0-b497-d5a5ec838880-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 2019-01-18 9:29 ` Chunming Zhou @ 2019-01-18 10:57 ` Liu, Monk [not found] ` <CY4PR1201MB0245D532D38DBA7F9BC445B9849C0-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org> 1 sibling, 1 reply; 25+ messages in thread From: Liu, Monk @ 2019-01-18 10:57 UTC (permalink / raw) To: Koenig, Christian, Lou, Wentao, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 15475 bytes --] You know what, … when you explained range 0 to HOLE-START is even not good to exposed to UMD I thought you made a typo and that’s why I repeat my question again … it’s the first time I heard that GMC9 cannot use 0 -> HOLE-START even for UMD general usage … With your assert in DEV_INFO the “virtual_address_offset/max” is now *totally* wrong … I saw current kmd still give that range from 0 to HOLE_START I need to check what you said with some HW guys, that sounds crazy … /Monk From: Christian König <ckoenig.leichtzumerken@gmail.com> Sent: Friday, January 18, 2019 5:12 PM To: Liu, Monk <Monk.Liu@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>; Lou, Wentao <Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org; Zhu, Rex <Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, You see that for UMD, it can use 0 to HOLE_START Let me say it once more: The UMD nor anybody else CAN'T use 0 to HOLE_START, that region is reserved for the ATC hardware! We unfortunately didn't knew that initially and also didn't used the ATC, so we didn't ran into a problem. But ROCm now uses the ATC on Raven/Picasso and I have a branch where I enable it on Vega as well. So when we don't fix that we will run into problems here. The ATC isn't usable in combination with SRIOV and I don't think Windows uses it either, so they probably never ran into any issues. Do you mean even UMD should not use virtual address that dropped in range from 0 to HOLE_START ? Yes, exactly! That in combination with ATC use can have quite a bunch of strange and hard to track down effects because two parts of the driver are using the same address space. That way where should UMD work in ?and I assume our UMD now still using this range, this part make me puzzle At least Mesa now uses the high address space from HOLE_END..0xFFFF FFFF FFFF FFFF. Regards, Christian. Am 18.01.19 um 02:32 schrieb Liu, Monk: Thanks Christian, Questions I have now: 1. You see that for UMD, it can use 0 to HOLE_START, so why CSA cannot use that range although the range is as you said reserved to ATC h/w ? Be note that for windows KMD, the CSA is allocated by UMD driver so CSA shares the same aperture /space range with other UMD BO, which mean CSA in windows also located in ATC range, if that’s a problem why windows still works well. * Can you illustrate this limitation with more details ? we need to understand why CSA couldn’t be put in ATC range. 1. According to your previous description :” Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older user space as usable and also put the CSA in there.” * Do you mean even UMD should not use virtual address that dropped in range from 0 to HOLE_START ? that way where should UMD work in ?and I assume our UMD now still using this range, this part make me puzzle /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 9:26 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Yes exactly. As I said the problem is that 0->HOLE_START is reserved for the ATC hardware, we should not touch it at all. Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? That turned out to be never a good idea in the first place. What we could do is reduce the max_pfn for SRIOV because the ATC doesn't work in that configuration anyway. But I would only do this as last resort. Any idea why an address above the hole doesn't work with SRIOV? It seems to work fine in the bare metal case. Regards, Christian. Am 17.01.19 um 14:19 schrieb Liu, Monk: Hi Christian Thanks for explaining the HOLD for us, My understanding is we still could put CSA to 0~HOLE_START, because we can report UMD the max space is HOLD_START-CSA_SIZE , thus no colliding will hit. > Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 4:30 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, ok let me explain a bit more how the hardware works. The GMC manages a virtual 64bit address space, but only 48bit of that virtual address space are handled by the page table walker. The 48bits of address space are sign extended, so bit 47 of that are extended into bits 48-63. This gives us the following memory layout: 0x0 .... virtual address space 0x8000 0000 0000 .... hole 0xFFFF 8000 0000 0000 .... virtual address space 0xFFFF FFFF FFFF FFFF Trying to access the hole results in a range fault interrupt IIRC. When doing the VM page table walk the topmost 16bits are ignored, so when programming the page table walker you cut those of and use a linear address again. This is what AMDGPU_GMC_HOLE_MASK is good for. Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. The most likely cause of this is that we still have a bug somewhere about this, e.g. not correctly using sign extended addresses *OR* using sign extended addresses where we should use linear instead. Regards, Christian. Am 17.01.19 um 09:04 schrieb Liu, Monk: Hi Christian I believe Wentao can fix the issue we it by below step: 1. Return Virtual_address_max (UMD use it) to HOLE_START – RESERVED_SIZE 2. [optional] Still Keep virtual_address_offset to RESERVED_SIZE (current way, I think it’s because previously we put CSA in 0 --> RESERVED_SIZE space) 3. Put CSA in HOLE_START – RESERVED_SIZE ==> HOLE_START (it’s current design) I don’t get where above scheme is not correct … can you give more explain for the GMC_HOLE_START ? e.g. 1. why you set GMC_HOLE_START to 0x8’000’0000’0000 (half size of MAX of 48bit address space) ? is it for HSA purpose to make sure GPU address can also be used for CPU address ? 2. now MAX_PFN is 1’000’0000’0000, do you need to change GMC_HOLE_START ? thanks we need some catch up /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 3:39 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 17.01.19 um 04:17 schrieb Lou, Wentao: Hi Christian, Your solution as: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = 0x10 0000, AMDGPU_PAGE_SHIFT = 12 Still got addr = 0xFFFF FFF0 0000, which would cause ring gfx timeout. But 0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else. Please try to use AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as well. Does that work? Before commit 1bf621c42137926ac249af761c0190a9258aa0db, vm_size was 32GB, and csa_addr was under AMDGPU_GMC_HOLE_START. Wait a second why was the vm_size 32GB? This is on a Vega10 isn't it? I didn’t understand why csa_addr need to be above AMDGPU_GMC_HOLE_START now. On Vega10 the lower range, e.g. everything below AMDGPU_GMC_HOLE_START is reserved for SVA. Regards, Christian. Thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Wednesday, January 16, 2019 5:48 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 38241 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <CY4PR1201MB0245D532D38DBA7F9BC445B9849C0-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org>]
* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <CY4PR1201MB0245D532D38DBA7F9BC445B9849C0-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org> @ 2019-01-18 12:20 ` Koenig, Christian [not found] ` <eff1c302-1c8d-e358-077e-704b67c3bd41-5C7GfCeVMHo@public.gmane.org> 0 siblings, 1 reply; 25+ messages in thread From: Koenig, Christian @ 2019-01-18 12:20 UTC (permalink / raw) To: Liu, Monk, Lou, Wentao, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 17333 bytes --] You know what, … when you explained range 0 to HOLE-START is even not good to exposed to UMD I thought you made a typo and that’s why I repeat my question again … Sorry my fault then. Didn't wanted to sound offending. it’s the first time I heard that GMC9 cannot use 0 -> HOLE-START even for UMD general usage … Well you actually can do it, but then you can't use the ATC or other SVA mechanism. With your assert in DEV_INFO the “virtual_address_offset/max” is now *totally* wrong … I saw current kmd still give that range from 0 to HOLE_START That is actually correct and for backward compatibility with old userspace. But since old userspace won't use the ATC that is also not a problem. As I said one possibility to solve this issue would be to use a low CSA address for SRIOV, because the ATC isn't usable with SRIOV anyway. I would just like to avoid that because it sounds like the CSA for some reason doesn't work at all in the higher address range and we will certainly then run into issues with that on bare metal as well. I need to check what you said with some HW guys, that sounds crazy … Actually that's not so crazy at all. See the ATC uses the CPU page tables to provide parts of the virtual GPU address space. E.g. when it is enabled you can then use the same pointer to memory on the CPU and the GPU. The problem is when the UMD now manually maps something into this range you can have a clash of the address space and the MC doesn't know any more if it should send a request to the CPU or the GPU page tables. Regards, Christian. Am 18.01.19 um 11:57 schrieb Liu, Monk: You know what, … when you explained range 0 to HOLE-START is even not good to exposed to UMD I thought you made a typo and that’s why I repeat my question again … it’s the first time I heard that GMC9 cannot use 0 -> HOLE-START even for UMD general usage … With your assert in DEV_INFO the “virtual_address_offset/max” is now *totally* wrong … I saw current kmd still give that range from 0 to HOLE_START I need to check what you said with some HW guys, that sounds crazy … /Monk From: Christian König <ckoenig.leichtzumerken@gmail.com><mailto:ckoenig.leichtzumerken@gmail.com> Sent: Friday, January 18, 2019 5:12 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, You see that for UMD, it can use 0 to HOLE_START Let me say it once more: The UMD nor anybody else CAN'T use 0 to HOLE_START, that region is reserved for the ATC hardware! We unfortunately didn't knew that initially and also didn't used the ATC, so we didn't ran into a problem. But ROCm now uses the ATC on Raven/Picasso and I have a branch where I enable it on Vega as well. So when we don't fix that we will run into problems here. The ATC isn't usable in combination with SRIOV and I don't think Windows uses it either, so they probably never ran into any issues. Do you mean even UMD should not use virtual address that dropped in range from 0 to HOLE_START ? Yes, exactly! That in combination with ATC use can have quite a bunch of strange and hard to track down effects because two parts of the driver are using the same address space. That way where should UMD work in ?and I assume our UMD now still using this range, this part make me puzzle At least Mesa now uses the high address space from HOLE_END..0xFFFF FFFF FFFF FFFF. Regards, Christian. Am 18.01.19 um 02:32 schrieb Liu, Monk: Thanks Christian, Questions I have now: 1. You see that for UMD, it can use 0 to HOLE_START, so why CSA cannot use that range although the range is as you said reserved to ATC h/w ? Be note that for windows KMD, the CSA is allocated by UMD driver so CSA shares the same aperture /space range with other UMD BO, which mean CSA in windows also located in ATC range, if that’s a problem why windows still works well. * Can you illustrate this limitation with more details ? we need to understand why CSA couldn’t be put in ATC range. 1. According to your previous description :” Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older user space as usable and also put the CSA in there.” * Do you mean even UMD should not use virtual address that dropped in range from 0 to HOLE_START ? that way where should UMD work in ?and I assume our UMD now still using this range, this part make me puzzle /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 9:26 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Yes exactly. As I said the problem is that 0->HOLE_START is reserved for the ATC hardware, we should not touch it at all. Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? That turned out to be never a good idea in the first place. What we could do is reduce the max_pfn for SRIOV because the ATC doesn't work in that configuration anyway. But I would only do this as last resort. Any idea why an address above the hole doesn't work with SRIOV? It seems to work fine in the bare metal case. Regards, Christian. Am 17.01.19 um 14:19 schrieb Liu, Monk: Hi Christian Thanks for explaining the HOLD for us, My understanding is we still could put CSA to 0~HOLE_START, because we can report UMD the max space is HOLD_START-CSA_SIZE , thus no colliding will hit. > Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 4:30 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, ok let me explain a bit more how the hardware works. The GMC manages a virtual 64bit address space, but only 48bit of that virtual address space are handled by the page table walker. The 48bits of address space are sign extended, so bit 47 of that are extended into bits 48-63. This gives us the following memory layout: 0x0 .... virtual address space 0x8000 0000 0000 .... hole 0xFFFF 8000 0000 0000 .... virtual address space 0xFFFF FFFF FFFF FFFF Trying to access the hole results in a range fault interrupt IIRC. When doing the VM page table walk the topmost 16bits are ignored, so when programming the page table walker you cut those of and use a linear address again. This is what AMDGPU_GMC_HOLE_MASK is good for. Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. The most likely cause of this is that we still have a bug somewhere about this, e.g. not correctly using sign extended addresses *OR* using sign extended addresses where we should use linear instead. Regards, Christian. Am 17.01.19 um 09:04 schrieb Liu, Monk: Hi Christian I believe Wentao can fix the issue we it by below step: 1. Return Virtual_address_max (UMD use it) to HOLE_START – RESERVED_SIZE 2. [optional] Still Keep virtual_address_offset to RESERVED_SIZE (current way, I think it’s because previously we put CSA in 0 --> RESERVED_SIZE space) 3. Put CSA in HOLE_START – RESERVED_SIZE ==> HOLE_START (it’s current design) I don’t get where above scheme is not correct … can you give more explain for the GMC_HOLE_START ? e.g. 1. why you set GMC_HOLE_START to 0x8’000’0000’0000 (half size of MAX of 48bit address space) ? is it for HSA purpose to make sure GPU address can also be used for CPU address ? 2. now MAX_PFN is 1’000’0000’0000, do you need to change GMC_HOLE_START ? thanks we need some catch up /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 3:39 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 17.01.19 um 04:17 schrieb Lou, Wentao: Hi Christian, Your solution as: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = 0x10 0000, AMDGPU_PAGE_SHIFT = 12 Still got addr = 0xFFFF FFF0 0000, which would cause ring gfx timeout. But 0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else. Please try to use AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as well. Does that work? Before commit 1bf621c42137926ac249af761c0190a9258aa0db, vm_size was 32GB, and csa_addr was under AMDGPU_GMC_HOLE_START. Wait a second why was the vm_size 32GB? This is on a Vega10 isn't it? I didn’t understand why csa_addr need to be above AMDGPU_GMC_HOLE_START now. On Vega10 the lower range, e.g. everything below AMDGPU_GMC_HOLE_START is reserved for SVA. Regards, Christian. Thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Wednesday, January 16, 2019 5:48 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 42014 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <eff1c302-1c8d-e358-077e-704b67c3bd41-5C7GfCeVMHo@public.gmane.org>]
* RE: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <eff1c302-1c8d-e358-077e-704b67c3bd41-5C7GfCeVMHo@public.gmane.org> @ 2019-01-21 4:35 ` Liu, Monk [not found] ` <DM5PR1201MB02516E33597C13BAA8C72299849F0-grEf7a3NxMClG5oLOQIDXGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org> 0 siblings, 1 reply; 25+ messages in thread From: Liu, Monk @ 2019-01-21 4:35 UTC (permalink / raw) To: Koenig, Christian, Lou, Wentao, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 18446 bytes --] > Actually that's not so crazy at all. See the ATC uses the CPU page tables to provide parts of the virtual GPU address space. So aperture 0->hole-start will be translated *not* by GMC9’s page table but instead by CPU’s (or IOMMU ?) MMU table after ATC enabled ? like GMC9 give up the role to do the translate but just deliver the address to CPU MMU (or IOMMU), is that correct ? Besides, where did you read that HOLE_START is set to be 0x800000000000 ? any registers actually controls this number ? I want to take a close look in the GMC registers regarding it No offending received, don’t be sorry at all, I don’t know that part before and thanks for sharing it. Thank you /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Friday, January 18, 2019 8:21 PM To: Liu, Monk <Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org; Zhu, Rex <Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START You know what, … when you explained range 0 to HOLE-START is even not good to exposed to UMD I thought you made a typo and that’s why I repeat my question again … Sorry my fault then. Didn't wanted to sound offending. it’s the first time I heard that GMC9 cannot use 0 -> HOLE-START even for UMD general usage … Well you actually can do it, but then you can't use the ATC or other SVA mechanism. With your assert in DEV_INFO the “virtual_address_offset/max” is now *totally* wrong … I saw current kmd still give that range from 0 to HOLE_START That is actually correct and for backward compatibility with old userspace. But since old userspace won't use the ATC that is also not a problem. As I said one possibility to solve this issue would be to use a low CSA address for SRIOV, because the ATC isn't usable with SRIOV anyway. I would just like to avoid that because it sounds like the CSA for some reason doesn't work at all in the higher address range and we will certainly then run into issues with that on bare metal as well. I need to check what you said with some HW guys, that sounds crazy … Actually that's not so crazy at all. See the ATC uses the CPU page tables to provide parts of the virtual GPU address space. E.g. when it is enabled you can then use the same pointer to memory on the CPU and the GPU. The problem is when the UMD now manually maps something into this range you can have a clash of the address space and the MC doesn't know any more if it should send a request to the CPU or the GPU page tables. Regards, Christian. Am 18.01.19 um 11:57 schrieb Liu, Monk: You know what, … when you explained range 0 to HOLE-START is even not good to exposed to UMD I thought you made a typo and that’s why I repeat my question again … it’s the first time I heard that GMC9 cannot use 0 -> HOLE-START even for UMD general usage … With your assert in DEV_INFO the “virtual_address_offset/max” is now *totally* wrong … I saw current kmd still give that range from 0 to HOLE_START I need to check what you said with some HW guys, that sounds crazy … /Monk From: Christian König <ckoenig.leichtzumerken@gmail.com><mailto:ckoenig.leichtzumerken@gmail.com> Sent: Friday, January 18, 2019 5:12 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, You see that for UMD, it can use 0 to HOLE_START Let me say it once more: The UMD nor anybody else CAN'T use 0 to HOLE_START, that region is reserved for the ATC hardware! We unfortunately didn't knew that initially and also didn't used the ATC, so we didn't ran into a problem. But ROCm now uses the ATC on Raven/Picasso and I have a branch where I enable it on Vega as well. So when we don't fix that we will run into problems here. The ATC isn't usable in combination with SRIOV and I don't think Windows uses it either, so they probably never ran into any issues. Do you mean even UMD should not use virtual address that dropped in range from 0 to HOLE_START ? Yes, exactly! That in combination with ATC use can have quite a bunch of strange and hard to track down effects because two parts of the driver are using the same address space. That way where should UMD work in ?and I assume our UMD now still using this range, this part make me puzzle At least Mesa now uses the high address space from HOLE_END..0xFFFF FFFF FFFF FFFF. Regards, Christian. Am 18.01.19 um 02:32 schrieb Liu, Monk: Thanks Christian, Questions I have now: 1. You see that for UMD, it can use 0 to HOLE_START, so why CSA cannot use that range although the range is as you said reserved to ATC h/w ? Be note that for windows KMD, the CSA is allocated by UMD driver so CSA shares the same aperture /space range with other UMD BO, which mean CSA in windows also located in ATC range, if that’s a problem why windows still works well. * Can you illustrate this limitation with more details ? we need to understand why CSA couldn’t be put in ATC range. 1. According to your previous description :” Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older user space as usable and also put the CSA in there.” * Do you mean even UMD should not use virtual address that dropped in range from 0 to HOLE_START ? that way where should UMD work in ?and I assume our UMD now still using this range, this part make me puzzle /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 9:26 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Yes exactly. As I said the problem is that 0->HOLE_START is reserved for the ATC hardware, we should not touch it at all. Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? That turned out to be never a good idea in the first place. What we could do is reduce the max_pfn for SRIOV because the ATC doesn't work in that configuration anyway. But I would only do this as last resort. Any idea why an address above the hole doesn't work with SRIOV? It seems to work fine in the bare metal case. Regards, Christian. Am 17.01.19 um 14:19 schrieb Liu, Monk: Hi Christian Thanks for explaining the HOLD for us, My understanding is we still could put CSA to 0~HOLE_START, because we can report UMD the max space is HOLD_START-CSA_SIZE , thus no colliding will hit. > Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 4:30 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, ok let me explain a bit more how the hardware works. The GMC manages a virtual 64bit address space, but only 48bit of that virtual address space are handled by the page table walker. The 48bits of address space are sign extended, so bit 47 of that are extended into bits 48-63. This gives us the following memory layout: 0x0 .... virtual address space 0x8000 0000 0000 .... hole 0xFFFF 8000 0000 0000 .... virtual address space 0xFFFF FFFF FFFF FFFF Trying to access the hole results in a range fault interrupt IIRC. When doing the VM page table walk the topmost 16bits are ignored, so when programming the page table walker you cut those of and use a linear address again. This is what AMDGPU_GMC_HOLE_MASK is good for. Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. The most likely cause of this is that we still have a bug somewhere about this, e.g. not correctly using sign extended addresses *OR* using sign extended addresses where we should use linear instead. Regards, Christian. Am 17.01.19 um 09:04 schrieb Liu, Monk: Hi Christian I believe Wentao can fix the issue we it by below step: 1. Return Virtual_address_max (UMD use it) to HOLE_START – RESERVED_SIZE 2. [optional] Still Keep virtual_address_offset to RESERVED_SIZE (current way, I think it’s because previously we put CSA in 0 --> RESERVED_SIZE space) 3. Put CSA in HOLE_START – RESERVED_SIZE ==> HOLE_START (it’s current design) I don’t get where above scheme is not correct … can you give more explain for the GMC_HOLE_START ? e.g. 1. why you set GMC_HOLE_START to 0x8’000’0000’0000 (half size of MAX of 48bit address space) ? is it for HSA purpose to make sure GPU address can also be used for CPU address ? 2. now MAX_PFN is 1’000’0000’0000, do you need to change GMC_HOLE_START ? thanks we need some catch up /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 3:39 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 17.01.19 um 04:17 schrieb Lou, Wentao: Hi Christian, Your solution as: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = 0x10 0000, AMDGPU_PAGE_SHIFT = 12 Still got addr = 0xFFFF FFF0 0000, which would cause ring gfx timeout. But 0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else. Please try to use AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as well. Does that work? Before commit 1bf621c42137926ac249af761c0190a9258aa0db, vm_size was 32GB, and csa_addr was under AMDGPU_GMC_HOLE_START. Wait a second why was the vm_size 32GB? This is on a Vega10 isn't it? I didn’t understand why csa_addr need to be above AMDGPU_GMC_HOLE_START now. On Vega10 the lower range, e.g. everything below AMDGPU_GMC_HOLE_START is reserved for SVA. Regards, Christian. Thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Wednesday, January 16, 2019 5:48 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 43456 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <DM5PR1201MB02516E33597C13BAA8C72299849F0-grEf7a3NxMClG5oLOQIDXGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org>]
* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <DM5PR1201MB02516E33597C13BAA8C72299849F0-grEf7a3NxMClG5oLOQIDXGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org> @ 2019-01-21 9:55 ` Koenig, Christian [not found] ` <c700df28-900c-f097-4ceb-0e6e8dcd35e8-5C7GfCeVMHo@public.gmane.org> 0 siblings, 1 reply; 25+ messages in thread From: Koenig, Christian @ 2019-01-21 9:55 UTC (permalink / raw) To: Liu, Monk, Lou, Wentao, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 19120 bytes --] Am 21.01.19 um 05:35 schrieb Liu, Monk: > Actually that's not so crazy at all. See the ATC uses the CPU page tables to provide parts of the virtual GPU address space. So aperture 0->hole-start will be translated *not* by GMC9’s page table but instead by CPU’s (or IOMMU ?) MMU table after ATC enabled ? like GMC9 give up the role to do the translate but just deliver the address to CPU MMU (or IOMMU), is that correct ? More or less yes. The GMC9 page tables still take precedence before the ATC IIRC, but the problem is simply that two hardware engines want to use the same address space. So when you have a collision you just mess things up no matter what you do. Besides, where did you read that HOLE_START is set to be 0x800000000000 ? any registers actually controls this number ? I want to take a close look in the GMC registers regarding it As far as I know that is hard wired on Vega. I was once in a meeting where they discussed if it should be configurable on Navi, but I'm not sure if they actually did that. Christian. No offending received, don’t be sorry at all, I don’t know that part before and thanks for sharing it. Thank you /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Friday, January 18, 2019 8:21 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START You know what, … when you explained range 0 to HOLE-START is even not good to exposed to UMD I thought you made a typo and that’s why I repeat my question again … Sorry my fault then. Didn't wanted to sound offending. it’s the first time I heard that GMC9 cannot use 0 -> HOLE-START even for UMD general usage … Well you actually can do it, but then you can't use the ATC or other SVA mechanism. With your assert in DEV_INFO the “virtual_address_offset/max” is now *totally* wrong … I saw current kmd still give that range from 0 to HOLE_START That is actually correct and for backward compatibility with old userspace. But since old userspace won't use the ATC that is also not a problem. As I said one possibility to solve this issue would be to use a low CSA address for SRIOV, because the ATC isn't usable with SRIOV anyway. I would just like to avoid that because it sounds like the CSA for some reason doesn't work at all in the higher address range and we will certainly then run into issues with that on bare metal as well. I need to check what you said with some HW guys, that sounds crazy … Actually that's not so crazy at all. See the ATC uses the CPU page tables to provide parts of the virtual GPU address space. E.g. when it is enabled you can then use the same pointer to memory on the CPU and the GPU. The problem is when the UMD now manually maps something into this range you can have a clash of the address space and the MC doesn't know any more if it should send a request to the CPU or the GPU page tables. Regards, Christian. Am 18.01.19 um 11:57 schrieb Liu, Monk: You know what, … when you explained range 0 to HOLE-START is even not good to exposed to UMD I thought you made a typo and that’s why I repeat my question again … it’s the first time I heard that GMC9 cannot use 0 -> HOLE-START even for UMD general usage … With your assert in DEV_INFO the “virtual_address_offset/max” is now *totally* wrong … I saw current kmd still give that range from 0 to HOLE_START I need to check what you said with some HW guys, that sounds crazy … /Monk From: Christian König <ckoenig.leichtzumerken@gmail.com><mailto:ckoenig.leichtzumerken@gmail.com> Sent: Friday, January 18, 2019 5:12 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, You see that for UMD, it can use 0 to HOLE_START Let me say it once more: The UMD nor anybody else CAN'T use 0 to HOLE_START, that region is reserved for the ATC hardware! We unfortunately didn't knew that initially and also didn't used the ATC, so we didn't ran into a problem. But ROCm now uses the ATC on Raven/Picasso and I have a branch where I enable it on Vega as well. So when we don't fix that we will run into problems here. The ATC isn't usable in combination with SRIOV and I don't think Windows uses it either, so they probably never ran into any issues. Do you mean even UMD should not use virtual address that dropped in range from 0 to HOLE_START ? Yes, exactly! That in combination with ATC use can have quite a bunch of strange and hard to track down effects because two parts of the driver are using the same address space. That way where should UMD work in ?and I assume our UMD now still using this range, this part make me puzzle At least Mesa now uses the high address space from HOLE_END..0xFFFF FFFF FFFF FFFF. Regards, Christian. Am 18.01.19 um 02:32 schrieb Liu, Monk: Thanks Christian, Questions I have now: 1. You see that for UMD, it can use 0 to HOLE_START, so why CSA cannot use that range although the range is as you said reserved to ATC h/w ? Be note that for windows KMD, the CSA is allocated by UMD driver so CSA shares the same aperture /space range with other UMD BO, which mean CSA in windows also located in ATC range, if that’s a problem why windows still works well. * Can you illustrate this limitation with more details ? we need to understand why CSA couldn’t be put in ATC range. 1. According to your previous description :” Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older user space as usable and also put the CSA in there.” * Do you mean even UMD should not use virtual address that dropped in range from 0 to HOLE_START ? that way where should UMD work in ?and I assume our UMD now still using this range, this part make me puzzle /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 9:26 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Yes exactly. As I said the problem is that 0->HOLE_START is reserved for the ATC hardware, we should not touch it at all. Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? That turned out to be never a good idea in the first place. What we could do is reduce the max_pfn for SRIOV because the ATC doesn't work in that configuration anyway. But I would only do this as last resort. Any idea why an address above the hole doesn't work with SRIOV? It seems to work fine in the bare metal case. Regards, Christian. Am 17.01.19 um 14:19 schrieb Liu, Monk: Hi Christian Thanks for explaining the HOLD for us, My understanding is we still could put CSA to 0~HOLE_START, because we can report UMD the max space is HOLD_START-CSA_SIZE , thus no colliding will hit. > Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. Regarding with above sentence, do you mean this range (0->HOLE_START) shouldn’t be exposed to user space ? I don’t get your point here … Putting CSA in 0~HOLD_START is the legacy approach we selected for a long time since very early stage, how comes that you think it is a problem now ? /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 4:30 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, ok let me explain a bit more how the hardware works. The GMC manages a virtual 64bit address space, but only 48bit of that virtual address space are handled by the page table walker. The 48bits of address space are sign extended, so bit 47 of that are extended into bits 48-63. This gives us the following memory layout: 0x0 .... virtual address space 0x8000 0000 0000 .... hole 0xFFFF 8000 0000 0000 .... virtual address space 0xFFFF FFFF FFFF FFFF Trying to access the hole results in a range fault interrupt IIRC. When doing the VM page table walk the topmost 16bits are ignored, so when programming the page table walker you cut those of and use a linear address again. This is what AMDGPU_GMC_HOLE_MASK is good for. Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. The most likely cause of this is that we still have a bug somewhere about this, e.g. not correctly using sign extended addresses *OR* using sign extended addresses where we should use linear instead. Regards, Christian. Am 17.01.19 um 09:04 schrieb Liu, Monk: Hi Christian I believe Wentao can fix the issue we it by below step: 1. Return Virtual_address_max (UMD use it) to HOLE_START – RESERVED_SIZE 2. [optional] Still Keep virtual_address_offset to RESERVED_SIZE (current way, I think it’s because previously we put CSA in 0 --> RESERVED_SIZE space) 3. Put CSA in HOLE_START – RESERVED_SIZE ==> HOLE_START (it’s current design) I don’t get where above scheme is not correct … can you give more explain for the GMC_HOLE_START ? e.g. 1. why you set GMC_HOLE_START to 0x8’000’0000’0000 (half size of MAX of 48bit address space) ? is it for HSA purpose to make sure GPU address can also be used for CPU address ? 2. now MAX_PFN is 1’000’0000’0000, do you need to change GMC_HOLE_START ? thanks we need some catch up /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 3:39 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 17.01.19 um 04:17 schrieb Lou, Wentao: Hi Christian, Your solution as: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = 0x10 0000, AMDGPU_PAGE_SHIFT = 12 Still got addr = 0xFFFF FFF0 0000, which would cause ring gfx timeout. But 0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else. Please try to use AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as well. Does that work? Before commit 1bf621c42137926ac249af761c0190a9258aa0db, vm_size was 32GB, and csa_addr was under AMDGPU_GMC_HOLE_START. Wait a second why was the vm_size 32GB? This is on a Vega10 isn't it? I didn’t understand why csa_addr need to be above AMDGPU_GMC_HOLE_START now. On Vega10 the lower range, e.g. everything below AMDGPU_GMC_HOLE_START is reserved for SVA. Regards, Christian. Thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Wednesday, January 16, 2019 5:48 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 46037 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <c700df28-900c-f097-4ceb-0e6e8dcd35e8-5C7GfCeVMHo@public.gmane.org>]
* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <c700df28-900c-f097-4ceb-0e6e8dcd35e8-5C7GfCeVMHo@public.gmane.org> @ 2019-01-29 22:33 ` Kuehling, Felix 0 siblings, 0 replies; 25+ messages in thread From: Kuehling, Felix @ 2019-01-29 22:33 UTC (permalink / raw) To: Koenig, Christian, Liu, Monk, Lou, Wentao, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily On 2019-01-21 4:55 a.m., Koenig, Christian wrote: > Am 21.01.19 um 05:35 schrieb Liu, Monk: >> >> > Actually that's not so crazy at all. See the ATC uses the CPU page >> tables to provide parts of the virtual GPU address space. >> >> >> >> So aperture 0->hole-start will be translated **not** by GMC9’s page >> table but instead by CPU’s (or IOMMU ?) MMU table after ATC enabled ? >> like GMC9 give up the role to do the translate but just deliver the >> address to CPU MMU (or IOMMU), is that correct ? >> > > More or less yes. The GMC9 page tables still take precedence before > the ATC IIRC, but the problem is simply that two hardware engines want > to use the same address space. The GPU looks at GPUVM page tables first. There is a special combination of PTE bits that tells it to forward the translation to ATC. IIRC the important bits are V=0, S=1. The full value of the ATC PTEs is defined in amdgpu_vm.h: #define AMDGPU_PTE_DEFAULT_ATC (AMDGPU_PTE_SYSTEM \ | AMDGPU_PTE_SNOOPED \ | AMDGPU_PTE_EXECUTABLE \ | AMDGPU_PTE_READABLE \ | AMDGPU_PTE_WRITEABLE \ | AMDGPU_PTE_MTYPE(AMDGPU_MTYPE_CC)) When we enable ATC for a VM, we program the PTEs and PDEs for unmapped addresses in the range from 0->hole-start to forward all address translation to ATC. That means, any addresses where nothing else is explicitly mapped, ATC handles the address translation so that the GPU can transparently access CPU virtual addresses. If you start mapping the CSA into that address range, you override PDEs and PTEs for the CSA address range, so they can no longer be handled by ATC. In the CSA address range, the GPU will no longer be able to access SVM memory in that range. Regards, Felix > > So when you have a collision you just mess things up no matter what > you do. > >> Besides, where did you read that HOLE_START is set to be >> 0x800000000000 ? any registers actually controls this number ? I want >> to take a close look in the GMC registers regarding it >> > > As far as I know that is hard wired on Vega. I was once in a meeting > where they discussed if it should be configurable on Navi, but I'm not > sure if they actually did that. > > Christian. > >> >> >> No offending received, don’t be sorry at all, I don’t know that part >> before and thanks for sharing it. >> >> >> >> Thank you >> >> /Monk >> >> >> >> *From:*amd-gfx <amd-gfx-bounces@lists.freedesktop.org> *On Behalf Of >> *Koenig, Christian >> *Sent:* Friday, January 18, 2019 8:21 PM >> *To:* Liu, Monk <Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com>; >> amd-gfx@lists.freedesktop.org; Zhu, Rex <Rex.Zhu@amd.com> >> *Cc:* Deng, Emily <Emily.Deng@amd.com> >> *Subject:* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than >> AMDGPU_GMC_HOLE_START >> >> >> >> You know what, … when you explained range 0 to HOLE-START is >> even not good to exposed to UMD I thought you made a typo and >> that’s why I repeat my question again … >> >> Sorry my fault then. Didn't wanted to sound offending. >> >> >> it’s the first time I heard that GMC9 cannot use 0 -> HOLE-START >> even for UMD general usage … >> >> Well you actually can do it, but then you can't use the ATC or other >> SVA mechanism. >> >> >> With your assert in DEV_INFO the “virtual_address_offset/max” is >> now **totally** wrong … I saw current kmd still give that range >> from 0 to HOLE_START >> >> That is actually correct and for backward compatibility with old >> userspace. But since old userspace won't use the ATC that is also not >> a problem. >> >> As I said one possibility to solve this issue would be to use a low >> CSA address for SRIOV, because the ATC isn't usable with SRIOV anyway. >> >> I would just like to avoid that because it sounds like the CSA for >> some reason doesn't work at all in the higher address range and we >> will certainly then run into issues with that on bare metal as well. >> >> >> I need to check what you said with some HW guys, that sounds crazy … >> >> Actually that's not so crazy at all. See the ATC uses the CPU page >> tables to provide parts of the virtual GPU address space. >> >> E.g. when it is enabled you can then use the same pointer to memory >> on the CPU and the GPU. >> >> The problem is when the UMD now manually maps something into this >> range you can have a clash of the address space and the MC doesn't >> know any more if it should send a request to the CPU or the GPU page >> tables. >> >> Regards, >> Christian. >> >> Am 18.01.19 um 11:57 schrieb Liu, Monk: >> >> You know what, … when you explained range 0 to HOLE-START is >> even not good to exposed to UMD I thought you made a typo and >> that’s why I repeat my question again … >> >> it’s the first time I heard that GMC9 cannot use 0 -> HOLE-START >> even for UMD general usage … >> >> With your assert in DEV_INFO the “virtual_address_offset/max” is >> now **totally** wrong … I saw current kmd still give that range >> from 0 to HOLE_START >> >> I need to check what you said with some HW guys, that sounds crazy … >> >> >> >> /Monk >> >> *From:*Christian König <ckoenig.leichtzumerken@gmail.com> >> <mailto:ckoenig.leichtzumerken@gmail.com> >> *Sent:* Friday, January 18, 2019 5:12 PM >> *To:* Liu, Monk <Monk.Liu@amd.com> <mailto:Monk.Liu@amd.com>; >> Koenig, Christian <Christian.Koenig@amd.com> >> <mailto:Christian.Koenig@amd.com>; Lou, Wentao >> <Wentao.Lou@amd.com> <mailto:Wentao.Lou@amd.com>; >> amd-gfx@lists.freedesktop.org >> <mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex >> <Rex.Zhu@amd.com> <mailto:Rex.Zhu@amd.com> >> *Cc:* Deng, Emily <Emily.Deng@amd.com> <mailto:Emily.Deng@amd.com> >> *Subject:* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger >> than AMDGPU_GMC_HOLE_START >> >> >> >> Hi Monk, >> >> >> >> You see that for UMD, it can use 0 to HOLE_START >> >> Let me say it once more: The UMD nor anybody else CAN'T use 0 to >> HOLE_START, that region is reserved for the ATC hardware! >> >> We unfortunately didn't knew that initially and also didn't used >> the ATC, so we didn't ran into a problem. >> >> But ROCm now uses the ATC on Raven/Picasso and I have a branch >> where I enable it on Vega as well. So when we don't fix that we >> will run into problems here. >> >> The ATC isn't usable in combination with SRIOV and I don't think >> Windows uses it either, so they probably never ran into any issues. >> >> >> >> Do you mean even UMD should not use virtual address that >> dropped in range from 0 to HOLE_START ? >> >> Yes, exactly! That in combination with ATC use can have quite a >> bunch of strange and hard to track down effects because two parts >> of the driver are using the same address space. >> >> >> >> That way where should UMD work in ?and I assume our UMD now >> still using this range, this part make me puzzle >> >> At least Mesa now uses the high address space from >> HOLE_END..0xFFFF FFFF FFFF FFFF. >> >> Regards, >> Christian. >> >> Am 18.01.19 um 02:32 schrieb Liu, Monk: >> >> Thanks Christian, >> >> >> >> Questions I have now: >> >> >> >> 1. You see that for UMD, it can use 0 to HOLE_START, so why >> CSA cannot use that range although the range is as you >> said reserved to ATC h/w ? Be note that for windows KMD, >> the CSA is allocated by UMD driver so CSA shares the same >> aperture /space range with other UMD BO, which mean CSA >> in windows also located in ATC range, if that’s a problem >> why windows still works well. >> >> 1. Can you illustrate this limitation with more details >> ? we need to understand why CSA couldn’t be put in >> ATC range. >> >> 2. According to your previous description :” Now on >> Vega/Raven/Picasso etc.. (everything with a GFX9) the >> lower range (0x0-0x8000 0000 0000) is reserved for >> SVA/ATC use. Since we *unfortunately didn't knew that >> initially we exposed those to older user space as usable* >> and also put the CSA in there.” >> >> 1. Do you mean even UMD should not use virtual address >> that dropped in range from 0 to HOLE_START ? >> >> that way where should UMD work in ?and I assume our UMD now >> still using this range, this part make me puzzle >> >> >> >> /Monk >> >> *From:*amd-gfx <amd-gfx-bounces@lists.freedesktop.org> >> <mailto:amd-gfx-bounces@lists.freedesktop.org> *On Behalf Of >> *Koenig, Christian >> *Sent:* Thursday, January 17, 2019 9:26 PM >> *To:* Liu, Monk <Monk.Liu@amd.com> <mailto:Monk.Liu@amd.com>; >> Lou, Wentao <Wentao.Lou@amd.com> <mailto:Wentao.Lou@amd.com>; >> amd-gfx@lists.freedesktop.org >> <mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex >> <Rex.Zhu@amd.com> <mailto:Rex.Zhu@amd.com> >> *Cc:* Deng, Emily <Emily.Deng@amd.com> >> <mailto:Emily.Deng@amd.com> >> *Subject:* Re: [PATCH] drm/amdgpu: csa_vaddr should not >> larger than AMDGPU_GMC_HOLE_START >> >> >> >> Hi Monk, >> >> >> >> >> Regarding with above sentence, do you mean this range >> (0->HOLE_START) shouldn’t be exposed to user space ? I >> don’t get your point here … >> >> Yes exactly. As I said the problem is that 0->HOLE_START is >> reserved for the ATC hardware, we should not touch it at all. >> >> >> >> >> Putting CSA in 0~HOLD_START is the legacy approach we >> selected for a long time since very early stage, how >> comes that you think it is a problem now ? >> >> That turned out to be never a good idea in the first place. >> >> What we could do is reduce the max_pfn for SRIOV because the >> ATC doesn't work in that configuration anyway. But I would >> only do this as last resort. >> >> Any idea why an address above the hole doesn't work with >> SRIOV? It seems to work fine in the bare metal case. >> >> Regards, >> Christian. >> >> Am 17.01.19 um 14:19 schrieb Liu, Monk: >> >> Hi Christian >> >> >> >> Thanks for explaining the HOLD for us, >> >> >> >> My understanding is we still could put CSA to >> 0~HOLE_START, because we can report UMD the max space is >> HOLD_START-CSA_SIZE , thus no colliding will hit. >> >> >> >> > Now on Vega/Raven/Picasso etc.. (everything with a GFX9) >> the lower range (0x0-0x8000 0000 0000) is reserved for >> SVA/ATC use. Since we unfortunately didn't knew that >> initially we exposed those to older userspace as usable >> and also put the CSA in there. >> >> >> >> >> Regarding with above sentence, do you mean this range >> (0->HOLE_START) shouldn’t be exposed to user space ? I >> don’t get your point here … >> >> >> >> Putting CSA in 0~HOLD_START is the legacy approach we >> selected for a long time since very early stage, how >> comes that you think it is a problem now ? >> >> >> >> /Monk >> >> *From:*amd-gfx <amd-gfx-bounces@lists.freedesktop.org> >> <mailto:amd-gfx-bounces@lists.freedesktop.org> *On Behalf >> Of *Koenig, Christian >> *Sent:* Thursday, January 17, 2019 4:30 PM >> *To:* Liu, Monk <Monk.Liu@amd.com> >> <mailto:Monk.Liu@amd.com>; Lou, Wentao >> <Wentao.Lou@amd.com> <mailto:Wentao.Lou@amd.com>; >> amd-gfx@lists.freedesktop.org >> <mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex >> <Rex.Zhu@amd.com> <mailto:Rex.Zhu@amd.com> >> *Cc:* Deng, Emily <Emily.Deng@amd.com> >> <mailto:Emily.Deng@amd.com> >> *Subject:* Re: [PATCH] drm/amdgpu: csa_vaddr should not >> larger than AMDGPU_GMC_HOLE_START >> >> >> >> Hi Monk, >> >> ok let me explain a bit more how the hardware works. >> >> The GMC manages a virtual 64bit address space, but only >> 48bit of that virtual address space are handled by the >> page table walker. >> >> The 48bits of address space are sign extended, so bit 47 >> of that are extended into bits 48-63. >> >> This gives us the following memory layout: >> 0x0 >> .... virtual address space >> 0x8000 0000 0000 >> .... hole >> 0xFFFF 8000 0000 0000 >> .... virtual address space >> 0xFFFF FFFF FFFF FFFF >> >> Trying to access the hole results in a range fault >> interrupt IIRC. >> >> When doing the VM page table walk the topmost 16bits are >> ignored, so when programming the page table walker you >> cut those of and use a linear address again. This is what >> AMDGPU_GMC_HOLE_MASK is good for. >> >> Now on Vega/Raven/Picasso etc.. (everything with a GFX9) >> the lower range (0x0-0x8000 0000 0000) is reserved for >> SVA/ATC use. Since we unfortunately didn't knew that >> initially we exposed those to older userspace as usable >> and also put the CSA in there. >> >> The most likely cause of this is that we still have a bug >> somewhere about this, e.g. not correctly using sign >> extended addresses *OR* using sign extended addresses >> where we should use linear instead. >> >> Regards, >> Christian. >> >> Am 17.01.19 um 09:04 schrieb Liu, Monk: >> >> Hi Christian >> >> >> >> I believe Wentao can fix the issue we it by below step: >> >> 1. Return *Virtual_address_max* (UMD use it) to >> HOLE_START – RESERVED_SIZE >> 2. [optional] Still Keep virtual_address_offset to >> RESERVED_SIZE (current way, I think it’s because >> previously we put CSA in 0 à RESERVED_SIZE space) >> 3. Put CSA in HOLE_START – RESERVED_SIZE è >> HOLE_START (it’s current design) >> >> >> >> I don’t get where above scheme is not correct … can >> you give more explain for the GMC_HOLE_START ? >> >> >> >> e.g. >> >> 1. why you set GMC_HOLE_START to 0x8’000’0000’0000 >> (half size of MAX of 48bit address space) ? is it >> for HSA purpose to make sure GPU address can also >> be used for CPU address ? >> 2. now MAX_PFN is 1’000’0000’0000, do you need to >> change GMC_HOLE_START ? >> >> >> >> thanks >> >> we need some catch up >> >> >> >> /Monk >> >> >> >> *From:*amd-gfx >> <amd-gfx-bounces@lists.freedesktop.org> >> <mailto:amd-gfx-bounces@lists.freedesktop.org> *On >> Behalf Of *Koenig, Christian >> *Sent:* Thursday, January 17, 2019 3:39 PM >> *To:* Lou, Wentao <Wentao.Lou@amd.com> >> <mailto:Wentao.Lou@amd.com>; Liu, Monk >> <Monk.Liu@amd.com> <mailto:Monk.Liu@amd.com>; >> amd-gfx@lists.freedesktop.org >> <mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex >> <Rex.Zhu@amd.com> <mailto:Rex.Zhu@amd.com> >> *Cc:* Deng, Emily <Emily.Deng@amd.com> >> <mailto:Emily.Deng@amd.com> >> *Subject:* Re: [PATCH] drm/amdgpu: csa_vaddr should >> not larger than AMDGPU_GMC_HOLE_START >> >> >> >> Am 17.01.19 um 04:17 schrieb Lou, Wentao: >> >> Hi Christian, >> >> >> >> Your solution as: >> >> addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; >> >> now max_pfn = 0x10 0000 0000, >> AMDGPU_VA_RESERVED_SIZE = 0x10 0000, >> AMDGPU_PAGE_SHIFT = 12 >> >> Still got addr = 0xFFFF FFF0 0000, which would >> cause ring gfx timeout. >> >> >> But 0xFFFF FFF0 0000 is the correct address, if that >> is causing a problem then there is a bug somewhere else. >> >> Please try to use >> AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as >> well. Does that work? >> >> >> >> >> >> >> >> >> Before commit >> 1bf621c42137926ac249af761c0190a9258aa0db, vm_size >> was 32GB, and csa_addr was under >> AMDGPU_GMC_HOLE_START. >> >> >> Wait a second why was the vm_size 32GB? This is on a >> Vega10 isn't it? >> >> >> >> >> >> >> I didn’t understand why csa_addr need to be above >> AMDGPU_GMC_HOLE_START now. >> >> >> On Vega10 the lower range, e.g. everything below >> AMDGPU_GMC_HOLE_START is reserved for SVA. >> >> Regards, >> Christian. >> >> >> >> >> >> >> Thanks. >> >> >> >> BR, >> >> Wentao >> >> >> >> >> >> >> >> *From:* Koenig, Christian >> <Christian.Koenig@amd.com> >> <mailto:Christian.Koenig@amd.com> >> *Sent:* Wednesday, January 16, 2019 5:48 PM >> *To:* Lou, Wentao <Wentao.Lou@amd.com> >> <mailto:Wentao.Lou@amd.com>; Liu, Monk >> <Monk.Liu@amd.com> <mailto:Monk.Liu@amd.com>; >> amd-gfx@lists.freedesktop.org >> <mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex >> <Rex.Zhu@amd.com> <mailto:Rex.Zhu@amd.com> >> *Cc:* Deng, Emily <Emily.Deng@amd.com> >> <mailto:Emily.Deng@amd.com> >> *Subject:* Re: [PATCH] drm/amdgpu: csa_vaddr >> should not larger than AMDGPU_GMC_HOLE_START >> >> >> >> Hi Wentao, >> >> well the problem is you don't seem to understand >> how the hardware works. >> >> See the engines see an MC address space with a >> hole in the middle, similar to the how x86 64bit >> CPU address space works. But the page tables are >> programmed linearly. >> >> So the calculation in amdgpu_driver_open_kms() is >> correct because it takes the MC address and mages >> a linear page table index from it again. >> >> The only thing we might need to fix here is >> shifting max_pfn before the subtraction and I >> doubt that even that is necessary. >> >> Regards, >> Christian. >> >> Am 16.01.19 um 10:34 schrieb Lou, Wentao: >> >> Hi Christian, >> >> >> >> Now vm_size was set to 0x4 0000 GB by below >> commit: >> >> 1bf621c42137926ac249af761c0190a9258aa0db >> drm/amdgpu: Remove unnecessary VM size >> calculations >> >> >> >> So that max_pfn would be 0x10 0000 0000. >> >> amdgpu_csa_vaddr would make max_pfn << 12 to >> get 0x1 0000 0000 0000, and then minus >> AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 >> >> unfortunately this number was between >> AMDGPU_GMC_HOLE_START and >> AMDGPU_GMC_HOLE_END, so that >> amdgpu_gmc_sign_extend was called to make it >> 0xFFFF FFFF FFF0 0000 >> >> >> >> in amdgpu_driver_open_kms, extended csa_addr >> cannot be passed into amdgpu_map_static_csa >> directly, it would be above the limit of max_pfn. >> >> So that csa_addr was restricted by >> AMDGPU_GMC_HOLE_MASK to make it possible for >> amdgpu_vm_alloc_pts. >> >> But this restriction by AMDGPU_GMC_HOLE_MASK >> would make the address fall back into >> AMDGPU_GMC_HOLE again, which causing GPU reset. >> >> We just put amdgpu_csa_vaddr back to >> AMDGPU_GMC_HOLE_START, to avoid the address >> touching AMDGPU_GMC_HOLE. >> >> By the way, if max_pfn was shift much to the >> left, it would always get zero, with or >> without min(*,*). >> >> >> >> >> >> BR, >> >> Wentao >> >> >> >> >> >> >> >> -----Original Message----- >> From: Koenig, Christian >> <Christian.Koenig@amd.com> >> <mailto:Christian.Koenig@amd.com> >> Sent: Tuesday, January 15, 2019 4:02 PM >> To: Liu, Monk <Monk.Liu@amd.com> >> <mailto:Monk.Liu@amd.com>; Lou, Wentao >> <Wentao.Lou@amd.com> >> <mailto:Wentao.Lou@amd.com>; >> amd-gfx@lists.freedesktop.org >> <mailto:amd-gfx@lists.freedesktop.org>; Zhu, >> Rex <Rex.Zhu@amd.com> <mailto:Rex.Zhu@amd.com> >> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr >> should not larger than AMDGPU_GMC_HOLE_START >> >> >> >> Am 15.01.19 um 07:19 schrieb Liu, Monk: >> >> > The max_pfn is now 1'0000'0000'0000'0000 >> (bytes) which is above 48 bit now, and it >> with AMDGPU_GMC_HOLE_MASK make it to zero .... >> >> > >> >> > And in code "amdgpu_driver_open_kms()" I >> saw @Zhu, Rex write the code as : >> >> > >> >> > "csa_addr = amdgpu_csa_vadr(adev) & >> AMDGPU_GMC_HOLE_MASK", I think this is wrong >> since you intentionally place the csa above >> GMC hole, right ? >> >> >> >> The fix is just completely incorrect since >> min(adev->vm_manager.max_pfn << >> AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) >> still gives you 0 when we shift max_pfn to >> much to the left. >> >> >> >> The correct solution is to substract the >> reserved size first and then shift. E.g.: >> >> >> >> addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; >> >> >> >> Regards, >> >> Christian. >> >> >> >> > >> >> > Looks like we should modify this place >> >> > >> >> > /Monk >> >> > >> >> > -----Original Message----- >> >> > From: amd-gfx >> <amd-gfx-bounces@lists.freedesktop.org >> <mailto:amd-gfx-bounces@lists.freedesktop.org>> >> On Behalf Of >> >> > Christian K?nig >> >> > Sent: Monday, January 14, 2019 9:05 PM >> >> > To: Lou, Wentao <Wentao.Lou@amd.com >> <mailto:Wentao.Lou@amd.com>>; >> amd-gfx@lists.freedesktop.org >> <mailto:amd-gfx@lists.freedesktop.org> >> >> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr >> should not larger than >> >> > AMDGPU_GMC_HOLE_START >> >> > >> >> > Am 14.01.19 um 09:40 schrieb wentalou: >> >> >> After removing unnecessary VM size >> calculations, vm_manager.max_pfn >> >> >> would reach 0x10,0000,0000 max_pfn << >> AMDGPU_GPU_PAGE_SHIFT exceeding >> >> >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> >> >> >> >> Change-Id: >> I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> >> >> Signed-off-by: wentalou >> <Wentao.Lou@amd.com <mailto:Wentao.Lou@amd.com>> >> >> > NAK, that is incorrect. We intentionally >> place the csa above the GMC hole. >> >> > >> >> > Regards, >> >> > Christian. >> >> > >> >> >> --- >> >> >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> | 3 ++- >> >> >> 1 file changed, 2 insertions(+), 1 >> deletion(-) >> >> >> >> >> >> diff --git >> a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> >> >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> >> >> index 7e22be7..dd3bd01 100644 >> >> >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> >> >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> >> >> @@ -26,7 +26,8 @@ >> >> >> >> >> >> uint64_t amdgpu_csa_vaddr(struct >> amdgpu_device *adev) >> >> >> { >> >> >> - uint64_t addr = >> adev->vm_manager.max_pfn << >> AMDGPU_GPU_PAGE_SHIFT; >> >> >> + uint64_t addr = >> min(adev->vm_manager.max_pfn << >> AMDGPU_GPU_PAGE_SHIFT, >> >> >> >> + >> AMDGPU_GMC_HOLE_START); >> >> >> >> >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> >> >> addr = amdgpu_gmc_sign_extend(addr); >> >> > _______________________________________________ >> >> > amd-gfx mailing list >> >> > amd-gfx@lists.freedesktop.org >> <mailto:amd-gfx@lists.freedesktop.org> >> >> > >> https://lists.freedesktop.org/mailman/listinfo/amd-gfx >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> amd-gfx mailing list >> >> amd-gfx@lists.freedesktop.org <mailto:amd-gfx@lists.freedesktop.org> >> >> https://lists.freedesktop.org/mailman/listinfo/amd-gfx >> >> >> >> >> > > > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/amd-gfx _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
* RE: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <859dcf48-f4bf-d3ec-0893-369c7fad51ed-5C7GfCeVMHo@public.gmane.org> 2019-01-17 13:19 ` Liu, Monk @ 2019-01-18 4:44 ` Lou, Wentao [not found] ` <BYAPR12MB274227E7B0D66F840747EB3C839C0-ZGDeBxoHBPmbrehcvEBedAdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org> 1 sibling, 1 reply; 25+ messages in thread From: Lou, Wentao @ 2019-01-18 4:44 UTC (permalink / raw) To: Koenig, Christian, Liu, Monk, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 10166 bytes --] Hi Christian, Thanks for the explanation. I have another question for the HOLE. As you mentioned “Trying to access the hole results in a range fault interrupt IIRC.” 0x8000 0000 0000 .... hole 0xFFFF 8000 0000 0000 But you said “0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else.” 0xFFFF FFF0 0000 should be inside the hole, so why it is a correct address? Much thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com> Sent: Thursday, January 17, 2019 4:30 PM To: Liu, Monk <Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org; Zhu, Rex <Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, ok let me explain a bit more how the hardware works. The GMC manages a virtual 64bit address space, but only 48bit of that virtual address space are handled by the page table walker. The 48bits of address space are sign extended, so bit 47 of that are extended into bits 48-63. This gives us the following memory layout: 0x0 .... virtual address space 0x8000 0000 0000 .... hole 0xFFFF 8000 0000 0000 .... virtual address space 0xFFFF FFFF FFFF FFFF Trying to access the hole results in a range fault interrupt IIRC. When doing the VM page table walk the topmost 16bits are ignored, so when programming the page table walker you cut those of and use a linear address again. This is what AMDGPU_GMC_HOLE_MASK is good for. Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. The most likely cause of this is that we still have a bug somewhere about this, e.g. not correctly using sign extended addresses *OR* using sign extended addresses where we should use linear instead. Regards, Christian. Am 17.01.19 um 09:04 schrieb Liu, Monk: Hi Christian I believe Wentao can fix the issue we it by below step: 1. Return Virtual_address_max (UMD use it) to HOLE_START – RESERVED_SIZE 2. [optional] Still Keep virtual_address_offset to RESERVED_SIZE (current way, I think it’s because previously we put CSA in 0 --> RESERVED_SIZE space) 3. Put CSA in HOLE_START – RESERVED_SIZE ==> HOLE_START (it’s current design) I don’t get where above scheme is not correct … can you give more explain for the GMC_HOLE_START ? e.g. 1. why you set GMC_HOLE_START to 0x8’000’0000’0000 (half size of MAX of 48bit address space) ? is it for HSA purpose to make sure GPU address can also be used for CPU address ? 2. now MAX_PFN is 1’000’0000’0000, do you need to change GMC_HOLE_START ? thanks we need some catch up /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 3:39 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 17.01.19 um 04:17 schrieb Lou, Wentao: Hi Christian, Your solution as: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = 0x10 0000, AMDGPU_PAGE_SHIFT = 12 Still got addr = 0xFFFF FFF0 0000, which would cause ring gfx timeout. But 0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else. Please try to use AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as well. Does that work? Before commit 1bf621c42137926ac249af761c0190a9258aa0db, vm_size was 32GB, and csa_addr was under AMDGPU_GMC_HOLE_START. Wait a second why was the vm_size 32GB? This is on a Vega10 isn't it? I didn’t understand why csa_addr need to be above AMDGPU_GMC_HOLE_START now. On Vega10 the lower range, e.g. everything below AMDGPU_GMC_HOLE_START is reserved for SVA. Regards, Christian. Thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Wednesday, January 16, 2019 5:48 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 25742 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <BYAPR12MB274227E7B0D66F840747EB3C839C0-ZGDeBxoHBPmbrehcvEBedAdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org>]
* Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <BYAPR12MB274227E7B0D66F840747EB3C839C0-ZGDeBxoHBPmbrehcvEBedAdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org> @ 2019-01-18 9:01 ` Koenig, Christian [not found] ` <b5d15c16-a023-3fb0-d5e3-b12276b6fc04-5C7GfCeVMHo@public.gmane.org> 0 siblings, 1 reply; 25+ messages in thread From: Koenig, Christian @ 2019-01-18 9:01 UTC (permalink / raw) To: Lou, Wentao, Liu, Monk, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 10720 bytes --] 0xFFFF FFF0 0000 should be inside the hole, so why it is a correct address? I was assuming that this is the linear address which is then "sign extended" to 0xFFFF FFFF FFF0 0000. If that address is used somewhere without extending bit 47 into bits 48-63 then we have found the root cause of the issue. Regards, Christian. Am 18.01.19 um 05:44 schrieb Lou, Wentao: Hi Christian, Thanks for the explanation. I have another question for the HOLE. As you mentioned “Trying to access the hole results in a range fault interrupt IIRC.” 0x8000 0000 0000 .... hole 0xFFFF 8000 0000 0000 But you said “0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else.” 0xFFFF FFF0 0000 should be inside the hole, so why it is a correct address? Much thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Thursday, January 17, 2019 4:30 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, ok let me explain a bit more how the hardware works. The GMC manages a virtual 64bit address space, but only 48bit of that virtual address space are handled by the page table walker. The 48bits of address space are sign extended, so bit 47 of that are extended into bits 48-63. This gives us the following memory layout: 0x0 .... virtual address space 0x8000 0000 0000 .... hole 0xFFFF 8000 0000 0000 .... virtual address space 0xFFFF FFFF FFFF FFFF Trying to access the hole results in a range fault interrupt IIRC. When doing the VM page table walk the topmost 16bits are ignored, so when programming the page table walker you cut those of and use a linear address again. This is what AMDGPU_GMC_HOLE_MASK is good for. Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. The most likely cause of this is that we still have a bug somewhere about this, e.g. not correctly using sign extended addresses *OR* using sign extended addresses where we should use linear instead. Regards, Christian. Am 17.01.19 um 09:04 schrieb Liu, Monk: Hi Christian I believe Wentao can fix the issue we it by below step: 1. Return Virtual_address_max (UMD use it) to HOLE_START – RESERVED_SIZE 2. [optional] Still Keep virtual_address_offset to RESERVED_SIZE (current way, I think it’s because previously we put CSA in 0 --> RESERVED_SIZE space) 3. Put CSA in HOLE_START – RESERVED_SIZE ==> HOLE_START (it’s current design) I don’t get where above scheme is not correct … can you give more explain for the GMC_HOLE_START ? e.g. 1. why you set GMC_HOLE_START to 0x8’000’0000’0000 (half size of MAX of 48bit address space) ? is it for HSA purpose to make sure GPU address can also be used for CPU address ? 2. now MAX_PFN is 1’000’0000’0000, do you need to change GMC_HOLE_START ? thanks we need some catch up /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 3:39 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 17.01.19 um 04:17 schrieb Lou, Wentao: Hi Christian, Your solution as: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = 0x10 0000, AMDGPU_PAGE_SHIFT = 12 Still got addr = 0xFFFF FFF0 0000, which would cause ring gfx timeout. But 0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else. Please try to use AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as well. Does that work? Before commit 1bf621c42137926ac249af761c0190a9258aa0db, vm_size was 32GB, and csa_addr was under AMDGPU_GMC_HOLE_START. Wait a second why was the vm_size 32GB? This is on a Vega10 isn't it? I didn’t understand why csa_addr need to be above AMDGPU_GMC_HOLE_START now. On Vega10 the lower range, e.g. everything below AMDGPU_GMC_HOLE_START is reserved for SVA. Regards, Christian. Thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Wednesday, January 16, 2019 5:48 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 27246 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
[parent not found: <b5d15c16-a023-3fb0-d5e3-b12276b6fc04-5C7GfCeVMHo@public.gmane.org>]
* RE: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START [not found] ` <b5d15c16-a023-3fb0-d5e3-b12276b6fc04-5C7GfCeVMHo@public.gmane.org> @ 2019-01-18 11:07 ` Liu, Monk 0 siblings, 0 replies; 25+ messages in thread From: Liu, Monk @ 2019-01-18 11:07 UTC (permalink / raw) To: Koenig, Christian, Lou, Wentao, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW, Zhu, Rex Cc: Deng, Emily [-- Attachment #1.1: Type: text/plain, Size: 11265 bytes --] HI Christian Regarding with range 0->hole_start being occupied by ATC, Can you share me where you see such limitation, any hardware document ? /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Friday, January 18, 2019 5:02 PM To: Lou, Wentao <Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org; Zhu, Rex <Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START 0xFFFF FFF0 0000 should be inside the hole, so why it is a correct address? I was assuming that this is the linear address which is then "sign extended" to 0xFFFF FFFF FFF0 0000. If that address is used somewhere without extending bit 47 into bits 48-63 then we have found the root cause of the issue. Regards, Christian. Am 18.01.19 um 05:44 schrieb Lou, Wentao: Hi Christian, Thanks for the explanation. I have another question for the HOLE. As you mentioned “Trying to access the hole results in a range fault interrupt IIRC.” 0x8000 0000 0000 .... hole 0xFFFF 8000 0000 0000 But you said “0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else.” 0xFFFF FFF0 0000 should be inside the hole, so why it is a correct address? Much thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Thursday, January 17, 2019 4:30 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Monk, ok let me explain a bit more how the hardware works. The GMC manages a virtual 64bit address space, but only 48bit of that virtual address space are handled by the page table walker. The 48bits of address space are sign extended, so bit 47 of that are extended into bits 48-63. This gives us the following memory layout: 0x0 .... virtual address space 0x8000 0000 0000 .... hole 0xFFFF 8000 0000 0000 .... virtual address space 0xFFFF FFFF FFFF FFFF Trying to access the hole results in a range fault interrupt IIRC. When doing the VM page table walk the topmost 16bits are ignored, so when programming the page table walker you cut those of and use a linear address again. This is what AMDGPU_GMC_HOLE_MASK is good for. Now on Vega/Raven/Picasso etc.. (everything with a GFX9) the lower range (0x0-0x8000 0000 0000) is reserved for SVA/ATC use. Since we unfortunately didn't knew that initially we exposed those to older userspace as usable and also put the CSA in there. The most likely cause of this is that we still have a bug somewhere about this, e.g. not correctly using sign extended addresses *OR* using sign extended addresses where we should use linear instead. Regards, Christian. Am 17.01.19 um 09:04 schrieb Liu, Monk: Hi Christian I believe Wentao can fix the issue we it by below step: 1. Return Virtual_address_max (UMD use it) to HOLE_START – RESERVED_SIZE 2. [optional] Still Keep virtual_address_offset to RESERVED_SIZE (current way, I think it’s because previously we put CSA in 0 --> RESERVED_SIZE space) 3. Put CSA in HOLE_START – RESERVED_SIZE ==> HOLE_START (it’s current design) I don’t get where above scheme is not correct … can you give more explain for the GMC_HOLE_START ? e.g. 1. why you set GMC_HOLE_START to 0x8’000’0000’0000 (half size of MAX of 48bit address space) ? is it for HSA purpose to make sure GPU address can also be used for CPU address ? 2. now MAX_PFN is 1’000’0000’0000, do you need to change GMC_HOLE_START ? thanks we need some catch up /Monk From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org><mailto:amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Koenig, Christian Sent: Thursday, January 17, 2019 3:39 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 17.01.19 um 04:17 schrieb Lou, Wentao: Hi Christian, Your solution as: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; now max_pfn = 0x10 0000 0000, AMDGPU_VA_RESERVED_SIZE = 0x10 0000, AMDGPU_PAGE_SHIFT = 12 Still got addr = 0xFFFF FFF0 0000, which would cause ring gfx timeout. But 0xFFFF FFF0 0000 is the correct address, if that is causing a problem then there is a bug somewhere else. Please try to use AMDGPU_GMC_HOLE_START-AMDGPU_VA_RESERVED_SIZE as well. Does that work? Before commit 1bf621c42137926ac249af761c0190a9258aa0db, vm_size was 32GB, and csa_addr was under AMDGPU_GMC_HOLE_START. Wait a second why was the vm_size 32GB? This is on a Vega10 isn't it? I didn’t understand why csa_addr need to be above AMDGPU_GMC_HOLE_START now. On Vega10 the lower range, e.g. everything below AMDGPU_GMC_HOLE_START is reserved for SVA. Regards, Christian. Thanks. BR, Wentao From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Wednesday, January 16, 2019 5:48 PM To: Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Cc: Deng, Emily <Emily.Deng@amd.com><mailto:Emily.Deng@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Hi Wentao, well the problem is you don't seem to understand how the hardware works. See the engines see an MC address space with a hole in the middle, similar to the how x86 64bit CPU address space works. But the page tables are programmed linearly. So the calculation in amdgpu_driver_open_kms() is correct because it takes the MC address and mages a linear page table index from it again. The only thing we might need to fix here is shifting max_pfn before the subtraction and I doubt that even that is necessary. Regards, Christian. Am 16.01.19 um 10:34 schrieb Lou, Wentao: Hi Christian, Now vm_size was set to 0x4 0000 GB by below commit: 1bf621c42137926ac249af761c0190a9258aa0db drm/amdgpu: Remove unnecessary VM size calculations So that max_pfn would be 0x10 0000 0000. amdgpu_csa_vaddr would make max_pfn << 12 to get 0x1 0000 0000 0000, and then minus AMDGPU_VA_RESERVED_SIZE, to get 0xFFFF FFF0 0000 unfortunately this number was between AMDGPU_GMC_HOLE_START and AMDGPU_GMC_HOLE_END, so that amdgpu_gmc_sign_extend was called to make it 0xFFFF FFFF FFF0 0000 in amdgpu_driver_open_kms, extended csa_addr cannot be passed into amdgpu_map_static_csa directly, it would be above the limit of max_pfn. So that csa_addr was restricted by AMDGPU_GMC_HOLE_MASK to make it possible for amdgpu_vm_alloc_pts. But this restriction by AMDGPU_GMC_HOLE_MASK would make the address fall back into AMDGPU_GMC_HOLE again, which causing GPU reset. We just put amdgpu_csa_vaddr back to AMDGPU_GMC_HOLE_START, to avoid the address touching AMDGPU_GMC_HOLE. By the way, if max_pfn was shift much to the left, it would always get zero, with or without min(*,*). BR, Wentao -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com><mailto:Christian.Koenig@amd.com> Sent: Tuesday, January 15, 2019 4:02 PM To: Liu, Monk <Monk.Liu@amd.com><mailto:Monk.Liu@amd.com>; Lou, Wentao <Wentao.Lou@amd.com><mailto:Wentao.Lou@amd.com>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Zhu, Rex <Rex.Zhu@amd.com><mailto:Rex.Zhu@amd.com> Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START Am 15.01.19 um 07:19 schrieb Liu, Monk: > The max_pfn is now 1'0000'0000'0000'0000 (bytes) which is above 48 bit now, and it with AMDGPU_GMC_HOLE_MASK make it to zero .... > > And in code "amdgpu_driver_open_kms()" I saw @Zhu, Rex write the code as : > > "csa_addr = amdgpu_csa_vadr(adev) & AMDGPU_GMC_HOLE_MASK", I think this is wrong since you intentionally place the csa above GMC hole, right ? The fix is just completely incorrect since min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, AMDGPU_GMC_HOLE_START) still gives you 0 when we shift max_pfn to much to the left. The correct solution is to substract the reserved size first and then shift. E.g.: addr = (max_pfn - (AMDGPU_VA_RESERVED_SIZE >> AMDGPU_PAGE_SHIFT)) << AMDGPU_PAGE_SHIFT; Regards, Christian. > > Looks like we should modify this place > > /Monk > > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of > Christian K?nig > Sent: Monday, January 14, 2019 9:05 PM > To: Lou, Wentao <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > Subject: Re: [PATCH] drm/amdgpu: csa_vaddr should not larger than > AMDGPU_GMC_HOLE_START > > Am 14.01.19 um 09:40 schrieb wentalou: >> After removing unnecessary VM size calculations, vm_manager.max_pfn >> would reach 0x10,0000,0000 max_pfn << AMDGPU_GPU_PAGE_SHIFT exceeding >> AMDGPU_GMC_HOLE_START would caused GPU reset. >> >> Change-Id: I47ad0be2b0bd9fb7490c4e1d7bb7bdacf71132cb >> Signed-off-by: wentalou <Wentao.Lou@amd.com<mailto:Wentao.Lou@amd.com>> > NAK, that is incorrect. We intentionally place the csa above the GMC hole. > > Regards, > Christian. > >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 3 ++- >> 1 file changed, 2 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> index 7e22be7..dd3bd01 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c >> @@ -26,7 +26,8 @@ >> >> uint64_t amdgpu_csa_vaddr(struct amdgpu_device *adev) >> { >> - uint64_t addr = adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT; >> + uint64_t addr = min(adev->vm_manager.max_pfn << AMDGPU_GPU_PAGE_SHIFT, >> + AMDGPU_GMC_HOLE_START); >> >> addr -= AMDGPU_VA_RESERVED_SIZE; >> addr = amdgpu_gmc_sign_extend(addr); > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx [-- Attachment #1.2: Type: text/html, Size: 28016 bytes --] [-- Attachment #2: Type: text/plain, Size: 154 bytes --] _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2019-01-29 22:33 UTC | newest]
Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-01-14 8:40 [PATCH] drm/amdgpu: csa_vaddr should not larger than AMDGPU_GMC_HOLE_START wentalou
[not found] ` <1547455236-30130-1-git-send-email-Wentao.Lou-5C7GfCeVMHo@public.gmane.org>
2019-01-14 9:05 ` Liu, Monk
2019-01-14 13:05 ` Christian König
[not found] ` <37b395b0-8503-238f-8775-66eb8795d23f-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2019-01-15 6:19 ` Liu, Monk
[not found] ` <CY4PR1201MB0245AB79C8C8319D8DFB3A3884810-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org>
2019-01-15 8:02 ` Koenig, Christian
[not found] ` <7f3f7b71-7acc-689a-e1d8-6fb184771f7b-5C7GfCeVMHo@public.gmane.org>
2019-01-16 9:34 ` Lou, Wentao
[not found] ` <BYAPR12MB2742181357E4D7DF7C171EE983820-ZGDeBxoHBPmbrehcvEBedAdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org>
2019-01-16 9:47 ` Koenig, Christian
[not found] ` <574d6ca8-f1c8-8ac0-3dbe-23aed7c0125d-5C7GfCeVMHo@public.gmane.org>
2019-01-17 3:17 ` Lou, Wentao
[not found] ` <BYAPR12MB27427158A5684A31BF96AB4783830-ZGDeBxoHBPmbrehcvEBedAdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org>
2019-01-17 7:39 ` Koenig, Christian
[not found] ` <361544e2-dd33-3b89-1211-4566b060b785-5C7GfCeVMHo@public.gmane.org>
2019-01-17 8:04 ` Liu, Monk
[not found] ` <CY4PR1201MB0245C5E3F6810C9157C9FDCF84830-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org>
2019-01-17 8:30 ` Koenig, Christian
[not found] ` <859dcf48-f4bf-d3ec-0893-369c7fad51ed-5C7GfCeVMHo@public.gmane.org>
2019-01-17 13:19 ` Liu, Monk
[not found] ` <CY4PR1201MB0245C8076917864AA97A0FFA84830-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org>
2019-01-17 13:25 ` Koenig, Christian
[not found] ` <cf3489f5-deb2-85ef-844a-b4c050f71d29-5C7GfCeVMHo@public.gmane.org>
2019-01-18 1:32 ` Liu, Monk
[not found] ` <CY4PR1201MB02457ECA700E3AE8B321AC90849C0-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org>
2019-01-18 9:11 ` Christian König
[not found] ` <c83cd7ec-03a5-45c0-b497-d5a5ec838880-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2019-01-18 9:29 ` Chunming Zhou
[not found] ` <effb374a-9803-d451-41c4-991a2cd0aa9a-5C7GfCeVMHo@public.gmane.org>
2019-01-18 10:39 ` Koenig, Christian
2019-01-18 10:57 ` Liu, Monk
[not found] ` <CY4PR1201MB0245D532D38DBA7F9BC445B9849C0-1iTaO6aE1DBfNQakwlCMTGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org>
2019-01-18 12:20 ` Koenig, Christian
[not found] ` <eff1c302-1c8d-e358-077e-704b67c3bd41-5C7GfCeVMHo@public.gmane.org>
2019-01-21 4:35 ` Liu, Monk
[not found] ` <DM5PR1201MB02516E33597C13BAA8C72299849F0-grEf7a3NxMClG5oLOQIDXGrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org>
2019-01-21 9:55 ` Koenig, Christian
[not found] ` <c700df28-900c-f097-4ceb-0e6e8dcd35e8-5C7GfCeVMHo@public.gmane.org>
2019-01-29 22:33 ` Kuehling, Felix
2019-01-18 4:44 ` Lou, Wentao
[not found] ` <BYAPR12MB274227E7B0D66F840747EB3C839C0-ZGDeBxoHBPmbrehcvEBedAdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org>
2019-01-18 9:01 ` Koenig, Christian
[not found] ` <b5d15c16-a023-3fb0-d5e3-b12276b6fc04-5C7GfCeVMHo@public.gmane.org>
2019-01-18 11:07 ` Liu, Monk
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.