[PATCH 1/2] drm/amdgpu: access stolen VRAM directly on KV/KB

[PATCH 1/2] drm/amdgpu: access stolen VRAM directly on KV/KB

[Christian König]

From: Christian König <christian.koenig@amd.com>

We don't need to use the PCI BAR on APUs. This allows us to access
the full VRAM directly without being limited by the BAR size.

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
index 3a25f72..70585b1 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
@@ -375,9 +375,16 @@ static int gmc_v7_0_mc_init(struct amdgpu_device *adev)
 	/* size in MB on si */
 	adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
 	adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
-	adev->mc.visible_vram_size = adev->mc.aper_size;
+
+#ifdef CONFIG_X86_64
+	if (adev->flags & AMD_IS_APU) {
+		adev->mc.aper_base = RREG32(mmMC_VM_FB_OFFSET) << 22ULL;
+		adev->mc.aper_size = adev->mc.real_vram_size;
+	}
+#endif
 
 	/* In case the PCI BAR is larger than the actual amount of vram */
+	adev->mc.visible_vram_size = adev->mc.aper_size;
 	if (adev->mc.visible_vram_size > adev->mc.real_vram_size)
 		adev->mc.visible_vram_size = adev->mc.real_vram_size;
 
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 2/2] drm/amdgpu: access stolen VRAM directly on CZ

[Christian König]

From: Christian König <christian.koenig@amd.com>

We don't need to use the PCI BAR on APUs. This allows us to access
the full VRAM directly without being limited by the BAR size.

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
index f7372d3..0323f5d 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
@@ -462,9 +462,16 @@ static int gmc_v8_0_mc_init(struct amdgpu_device *adev)
 	/* size in MB on si */
 	adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
 	adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
-	adev->mc.visible_vram_size = adev->mc.aper_size;
+
+#ifdef CONFIG_X86_64
+	if (adev->flags & AMD_IS_APU) {
+		adev->mc.aper_base = RREG32(mmMC_VM_FB_OFFSET) << 22ULL;
+		adev->mc.aper_size = adev->mc.real_vram_size;
+	}
+#endif
 
 	/* In case the PCI BAR is larger than the actual amount of vram */
+	adev->mc.visible_vram_size = adev->mc.aper_size;
 	if (adev->mc.visible_vram_size > adev->mc.real_vram_size)
 		adev->mc.visible_vram_size = adev->mc.real_vram_size;
 
-- 
2.7.4

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 1/2] drm/amdgpu: access stolen VRAM directly on KV/KB

[Michel Dänzer]

On 07/11/16 06:21 PM, Christian König wrote:
> From: Christian König <christian.koenig@amd.com>
> 
> We don't need to use the PCI BAR on APUs. This allows us to access
> the full VRAM directly without being limited by the BAR size.
> 
> Signed-off-by: Christian König <christian.koenig@amd.com>

The series is

Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>


We could also drop the HDP flushes / invalidates in this case, right?


-- 
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 1/2] drm/amdgpu: access stolen VRAM directly on KV/KB

[Christian König]

Am 07.11.2016 um 10:29 schrieb Michel Dänzer:
> On 07/11/16 06:21 PM, Christian König wrote:
>> From: Christian König <christian.koenig@amd.com>
>>
>> We don't need to use the PCI BAR on APUs. This allows us to access
>> the full VRAM directly without being limited by the BAR size.
>>
>> Signed-off-by: Christian König <christian.koenig@amd.com>
> The series is
>
> Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
>
>
> We could also drop the HDP flushes / invalidates in this case, right?
Could be, but I'm not 100% sure how those are actually wired up on APUs.

Might be that they are NOPs anyway, but could be that at least the 
flushes are still vital.

Regards,
Christian.
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 2/2] drm/amdgpu: access stolen VRAM directly on CZ

[Dave Airlie]

On 7 November 2016 at 19:21, Christian König <deathsimple@vodafone.de> wrote:
> From: Christian König <christian.koenig@amd.com>
>
> We don't need to use the PCI BAR on APUs. This allows us to access
> the full VRAM directly without being limited by the BAR size.

I'm feeling coherency issues, has this approach been validated with the hw team?

Dave.

>
> Signed-off-by: Christian König <christian.koenig@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
> index f7372d3..0323f5d 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
> @@ -462,9 +462,16 @@ static int gmc_v8_0_mc_init(struct amdgpu_device *adev)
>         /* size in MB on si */
>         adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>         adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
> -       adev->mc.visible_vram_size = adev->mc.aper_size;
> +
> +#ifdef CONFIG_X86_64
> +       if (adev->flags & AMD_IS_APU) {
> +               adev->mc.aper_base = RREG32(mmMC_VM_FB_OFFSET) << 22ULL;
> +               adev->mc.aper_size = adev->mc.real_vram_size;
> +       }
> +#endif
>
>         /* In case the PCI BAR is larger than the actual amount of vram */
> +       adev->mc.visible_vram_size = adev->mc.aper_size;
>         if (adev->mc.visible_vram_size > adev->mc.real_vram_size)
>                 adev->mc.visible_vram_size = adev->mc.real_vram_size;
>
> --
> 2.7.4
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 1/2] drm/amdgpu: access stolen VRAM directly on KV/KB

[Felix Kuehling]

As a next step, you could also remove HDP flushing on APUs.

Regards,
  Felix


On 16-11-07 04:21 AM, Christian König wrote:
> From: Christian König <christian.koenig@amd.com>
>
> We don't need to use the PCI BAR on APUs. This allows us to access
> the full VRAM directly without being limited by the BAR size.
>
> Signed-off-by: Christian König <christian.koenig@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> index 3a25f72..70585b1 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c
> @@ -375,9 +375,16 @@ static int gmc_v7_0_mc_init(struct amdgpu_device *adev)
>  	/* size in MB on si */
>  	adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
>  	adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
> -	adev->mc.visible_vram_size = adev->mc.aper_size;
> +
> +#ifdef CONFIG_X86_64
> +	if (adev->flags & AMD_IS_APU) {
> +		adev->mc.aper_base = RREG32(mmMC_VM_FB_OFFSET) << 22ULL;
> +		adev->mc.aper_size = adev->mc.real_vram_size;
> +	}
> +#endif
>  
>  	/* In case the PCI BAR is larger than the actual amount of vram */
> +	adev->mc.visible_vram_size = adev->mc.aper_size;
>  	if (adev->mc.visible_vram_size > adev->mc.real_vram_size)
>  		adev->mc.visible_vram_size = adev->mc.real_vram_size;
>  

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 2/2] drm/amdgpu: access stolen VRAM directly on CZ

[Michel Dänzer]

On 08/11/16 05:31 AM, Dave Airlie wrote:
> On 7 November 2016 at 19:21, Christian König <deathsimple@vodafone.de> wrote:
>> From: Christian König <christian.koenig@amd.com>
>>
>> We don't need to use the PCI BAR on APUs. This allows us to access
>> the full VRAM directly without being limited by the BAR size.
> 
> I'm feeling coherency issues, has this approach been validated with the hw team?

I raised this concern internally as well, but the response indicated
that it should be fine. Apparently x86 defines that write-combined
writes complete before any access to UC memory (such as writing to an
MMIO register or doorbell to submit a GPU operation).


-- 
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 1/2] drm/amdgpu: access stolen VRAM directly on KV/KB

[Michel Dänzer]

On 07/11/16 06:33 PM, Christian König wrote:
> Am 07.11.2016 um 10:29 schrieb Michel Dänzer:
>> On 07/11/16 06:21 PM, Christian König wrote:
>>> From: Christian König <christian.koenig@amd.com>
>>>
>>> We don't need to use the PCI BAR on APUs. This allows us to access
>>> the full VRAM directly without being limited by the BAR size.
>>>
>>> Signed-off-by: Christian König <christian.koenig@amd.com>
>> The series is
>>
>> Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
>>
>>
>> We could also drop the HDP flushes / invalidates in this case, right?
> Could be, but I'm not 100% sure how those are actually wired up on APUs.
> 
> Might be that they are NOPs anyway, but could be that at least the
> flushes are still vital.

My assumption is that since CPU access to VRAM doesn't go through the
HDP at all with this change, HDP flushes / invalidates can't have any
useful effect.


-- 
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 1/2] drm/amdgpu: access stolen VRAM directly on KV/KB

[Christian König]

Am 08.11.2016 um 08:31 schrieb Michel Dänzer:
> On 07/11/16 06:33 PM, Christian König wrote:
>> Am 07.11.2016 um 10:29 schrieb Michel Dänzer:
>>> On 07/11/16 06:21 PM, Christian König wrote:
>>>> From: Christian König <christian.koenig@amd.com>
>>>>
>>>> We don't need to use the PCI BAR on APUs. This allows us to access
>>>> the full VRAM directly without being limited by the BAR size.
>>>>
>>>> Signed-off-by: Christian König <christian.koenig@amd.com>
>>> The series is
>>>
>>> Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
>>>
>>>
>>> We could also drop the HDP flushes / invalidates in this case, right?
>> Could be, but I'm not 100% sure how those are actually wired up on APUs.
>>
>> Might be that they are NOPs anyway, but could be that at least the
>> flushes are still vital.
> My assumption is that since CPU access to VRAM doesn't go through the
> HDP at all with this change, HDP flushes / invalidates can't have any
> useful effect.

Well writes to GTT still use the HDP if I understand it correctly.

So we still need to handle either flushes or invalidations (I doesn't of 
hand remember which was that).

Regards,
Christian.
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 1/2] drm/amdgpu: access stolen VRAM directly on KV/KB

[Michel Dänzer]

On 08/11/16 06:23 PM, Christian König wrote:
> Am 08.11.2016 um 08:31 schrieb Michel Dänzer:
>> On 07/11/16 06:33 PM, Christian König wrote:
>>> Am 07.11.2016 um 10:29 schrieb Michel Dänzer:
>>>> On 07/11/16 06:21 PM, Christian König wrote:
>>>>> From: Christian König <christian.koenig@amd.com>
>>>>>
>>>>> We don't need to use the PCI BAR on APUs. This allows us to access
>>>>> the full VRAM directly without being limited by the BAR size.
>>>>>
>>>>> Signed-off-by: Christian König <christian.koenig@amd.com>
>>>> The series is
>>>>
>>>> Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
>>>>
>>>>
>>>> We could also drop the HDP flushes / invalidates in this case, right?
>>> Could be, but I'm not 100% sure how those are actually wired up on APUs.
>>>
>>> Might be that they are NOPs anyway, but could be that at least the
>>> flushes are still vital.
>> My assumption is that since CPU access to VRAM doesn't go through the
>> HDP at all with this change, HDP flushes / invalidates can't have any
>> useful effect.
> 
> Well writes to GTT still use the HDP if I understand it correctly.
> 
> So we still need to handle either flushes or invalidations (I doesn't of
> hand remember which was that).

Do you mean GPU access to GTT? If so, not sure why it would be writes only?


-- 
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 1/2] drm/amdgpu: access stolen VRAM directly on KV/KB

[Deucher, Alexander]

> -----Original Message-----
> From: amd-gfx [mailto:amd-gfx-bounces@lists.freedesktop.org] On Behalf
> Of Michel Dänzer
> Sent: Tuesday, November 08, 2016 4:37 AM
> To: Christian König
> Cc: amd-gfx@lists.freedesktop.org
> Subject: Re: [PATCH 1/2] drm/amdgpu: access stolen VRAM directly on
> KV/KB
> 
> On 08/11/16 06:23 PM, Christian König wrote:
> > Am 08.11.2016 um 08:31 schrieb Michel Dänzer:
> >> On 07/11/16 06:33 PM, Christian König wrote:
> >>> Am 07.11.2016 um 10:29 schrieb Michel Dänzer:
> >>>> On 07/11/16 06:21 PM, Christian König wrote:
> >>>>> From: Christian König <christian.koenig@amd.com>
> >>>>>
> >>>>> We don't need to use the PCI BAR on APUs. This allows us to access
> >>>>> the full VRAM directly without being limited by the BAR size.
> >>>>>
> >>>>> Signed-off-by: Christian König <christian.koenig@amd.com>
> >>>> The series is
> >>>>
> >>>> Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
> >>>>
> >>>>
> >>>> We could also drop the HDP flushes / invalidates in this case, right?
> >>> Could be, but I'm not 100% sure how those are actually wired up on
> APUs.
> >>>
> >>> Might be that they are NOPs anyway, but could be that at least the
> >>> flushes are still vital.
> >> My assumption is that since CPU access to VRAM doesn't go through the
> >> HDP at all with this change, HDP flushes / invalidates can't have any
> >> useful effect.
> >
> > Well writes to GTT still use the HDP if I understand it correctly.
> >
> > So we still need to handle either flushes or invalidations (I doesn't of
> > hand remember which was that).
> 
> Do you mean GPU access to GTT? If so, not sure why it would be writes only?

The HDP has a read cache and a write queue.  It only affects access via the BAR (either the CPU or another PCIe device).  AFAIK, the APU access would only use the HDP if you mapped the BAR via the GART.

Alex

> 
> 
> --
> Earthling Michel Dänzer               |               http://www.amd.com
> Libre software enthusiast             |             Mesa and X developer
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 1/2] drm/amdgpu: access stolen VRAM directly on KV/KB

[Christian König]

Am 08.11.2016 um 14:36 schrieb Deucher, Alexander:
>> -----Original Message-----
>> From: amd-gfx [mailto:amd-gfx-bounces@lists.freedesktop.org] On Behalf
>> Of Michel Dänzer
>> Sent: Tuesday, November 08, 2016 4:37 AM
>> To: Christian König
>> Cc: amd-gfx@lists.freedesktop.org
>> Subject: Re: [PATCH 1/2] drm/amdgpu: access stolen VRAM directly on
>> KV/KB
>>
>> On 08/11/16 06:23 PM, Christian König wrote:
>>> Am 08.11.2016 um 08:31 schrieb Michel Dänzer:
>>>> On 07/11/16 06:33 PM, Christian König wrote:
>>>>> Am 07.11.2016 um 10:29 schrieb Michel Dänzer:
>>>>>> On 07/11/16 06:21 PM, Christian König wrote:
>>>>>>> From: Christian König <christian.koenig@amd.com>
>>>>>>>
>>>>>>> We don't need to use the PCI BAR on APUs. This allows us to access
>>>>>>> the full VRAM directly without being limited by the BAR size.
>>>>>>>
>>>>>>> Signed-off-by: Christian König <christian.koenig@amd.com>
>>>>>> The series is
>>>>>>
>>>>>> Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
>>>>>>
>>>>>>
>>>>>> We could also drop the HDP flushes / invalidates in this case, right?
>>>>> Could be, but I'm not 100% sure how those are actually wired up on
>> APUs.
>>>>> Might be that they are NOPs anyway, but could be that at least the
>>>>> flushes are still vital.
>>>> My assumption is that since CPU access to VRAM doesn't go through the
>>>> HDP at all with this change, HDP flushes / invalidates can't have any
>>>> useful effect.
>>> Well writes to GTT still use the HDP if I understand it correctly.
>>>
>>> So we still need to handle either flushes or invalidations (I doesn't of
>>> hand remember which was that).
>> Do you mean GPU access to GTT? If so, not sure why it would be writes only?
> The HDP has a read cache and a write queue.  It only affects access via the BAR (either the CPU or another PCIe device).  AFAIK, the APU access would only use the HDP if you mapped the BAR via the GART.

Ok, my understanding was that the HDP was responsible for both 
directions. E.g. writes/reads from the CPU side to the BAR as well as 
writes/reads from the GPU to system memory.

Since this needs intense testing anyway I will just wait for the first 
two patches to land, wait for a release to make sure they don't cause 
problems and then disable HDP flushes on APUs as well.

Christian.

>
> Alex
>
>>
>> --
>> Earthling Michel Dänzer               |               http://www.amd.com
>> Libre software enthusiast             |             Mesa and X developer
>> _______________________________________________
>> amd-gfx mailing list
>> amd-gfx@lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx


_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx