Re: [PATCH] drm/amdgpu: force using sdma to update vm page table when mmio is blocked

[Felix Kuehling <felix.kuehling@amd.com>]

I made a related fix for HDP flushing under SRIOV a while ago, and that 
was confirmed to be working at the time. What changed since then? Is 
this on different HW or with a different BIOS? Or are you missing this 
patch? For reference:

commit 6295c4064d8d18c4480636076dbd5afb48885c02
Author: Felix Kuehling <Felix.Kuehling@amd.com>
Date:   Thu Nov 4 16:15:43 2021 -0400

     drm/amdgpu: Fix MMIO HDP flush on SRIOV
     
     Disable HDP register remapping on SRIOV and set rmmio_remap.reg_offset
     to the fixed address of the VF register for hdp_v*_flush_hdp.
     
     Signed-off-by: Felix Kuehling <Felix.Kuehling@amd.com>
     Tested-by: Bokun Zhang <bokun.zhang@amd.com>
     Reviewed-by: Lijo Lazar <lijo.lazar@amd.com>


Regards,
   Felix



Am 2022-01-20 um 04:39 schrieb Wang, Yang(Kevin):
>
> [AMD Official Use Only]
>
>
> [AMD Official Use Only]
>
>
> Thanks @Koenig, Christian <mailto:Christian.Koenig@amd.com>.
>
> Best Regards,
> Kevin
> ------------------------------------------------------------------------
> *From:* Koenig, Christian <Christian.Koenig@amd.com>
> *Sent:* Thursday, January 20, 2022 5:11 PM
> *To:* Wang, Yang(Kevin) <KevinYang.Wang@amd.com>; 
> amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>; 
> Deucher, Alexander <Alexander.Deucher@amd.com>; Liu, Monk 
> <Monk.Liu@amd.com>
> *Cc:* Min, Frank <Frank.Min@amd.com>; Chen, Horace <Horace.Chen@amd.com>
> *Subject:* Re: [PATCH] drm/amdgpu: force using sdma to update vm page 
> table when mmio is blocked
> Hi Kevin,
>
> well at least the HDP flush function has to work correctly or 
> otherwise the driver won't work correctly.
>
> If the registers are not accessible any more we need to find a proper 
> workaround for this.
>
> One possibility would be to use the KIQ, another is a dummy write/read 
> to make sure the HDP is flushed (check the hardware docs).
>
> The third option would be to question if blocking the HDP registers is 
> really a good idea.
>
> The solution is up to you, but a workaround like proposed below 
> doesn't really help in any way.
>
> Regards,
> Christian.
>
> Am 20.01.22 um 10:07 schrieb Wang, Yang(Kevin):
>>
>> [AMD Official Use Only]
>>
>>
>> Hi Chris,
>>
>> yes, I agree with your point.
>> and another way is that we can use KIQ to write HDP register to 
>> resolve HDP can't R/W issue.
>> but it will cause some performance drop if use KIQ to programing 
>> register.
>>
>> what is your ideas?
>>
>> Best Regards,
>> Kevin
>> ------------------------------------------------------------------------
>> *From:* Koenig, Christian <Christian.Koenig@amd.com> 
>> <mailto:Christian.Koenig@amd.com>
>> *Sent:* Thursday, January 20, 2022 4:58 PM
>> *To:* Wang, Yang(Kevin) <KevinYang.Wang@amd.com> 
>> <mailto:KevinYang.Wang@amd.com>; amd-gfx@lists.freedesktop.org 
>> <mailto:amd-gfx@lists.freedesktop.org> 
>> <amd-gfx@lists.freedesktop.org> 
>> <mailto:amd-gfx@lists.freedesktop.org>; Deucher, Alexander 
>> <Alexander.Deucher@amd.com> <mailto:Alexander.Deucher@amd.com>; Liu, 
>> Monk <Monk.Liu@amd.com> <mailto:Monk.Liu@amd.com>
>> *Cc:* Min, Frank <Frank.Min@amd.com> <mailto:Frank.Min@amd.com>; 
>> Chen, Horace <Horace.Chen@amd.com> <mailto:Horace.Chen@amd.com>
>> *Subject:* Re: [PATCH] drm/amdgpu: force using sdma to update vm page 
>> table when mmio is blocked
>> Well NAK.
>>
>> Even when we can't R/W HDP registers we need a way to invalidate the 
>> HDP or quite a bunch of functions won't work correctly.
>>
>> Blocking CPU base page table updates only works around the symptoms, 
>> but doesn't really solve anything.
>>
>> Regards,
>> Christian.
>>
>> Am 20.01.22 um 09:46 schrieb Wang, Yang(Kevin):
>>>
>>> [AMD Official Use Only]
>>>
>>>
>>> ping...
>>>
>>> add @Liu, Monk <mailto:Monk.Liu@amd.com> @Koenig, Christian 
>>> <mailto:Christian.Koenig@amd.com> @Deucher, Alexander 
>>> <mailto:Alexander.Deucher@amd.com>
>>>
>>> Best Regards,
>>> Kevin
>>> ------------------------------------------------------------------------
>>> *From:* Wang, Yang(Kevin) <KevinYang.Wang@amd.com> 
>>> <mailto:KevinYang.Wang@amd.com>
>>> *Sent:* Wednesday, January 19, 2022 11:16 AM
>>> *To:* amd-gfx@lists.freedesktop.org 
>>> <mailto:amd-gfx@lists.freedesktop.org> 
>>> <amd-gfx@lists.freedesktop.org> <mailto:amd-gfx@lists.freedesktop.org>
>>> *Cc:* Min, Frank <Frank.Min@amd.com> <mailto:Frank.Min@amd.com>; 
>>> Chen, Horace <Horace.Chen@amd.com> <mailto:Horace.Chen@amd.com>; 
>>> Wang, Yang(Kevin) <KevinYang.Wang@amd.com> 
>>> <mailto:KevinYang.Wang@amd.com>
>>> *Subject:* [PATCH] drm/amdgpu: force using sdma to update vm page 
>>> table when mmio is blocked
>>> when mmio protection feature is enabled in hypervisor,
>>> it will cause guest OS can't R/W HDP regiters,
>>> and using cpu to update page table is not working well.
>>>
>>> force using sdma to update page table when mmio is blocked.
>>>
>>> Signed-off-by: Yang Wang <KevinYang.Wang@amd.com> 
>>> <mailto:KevinYang.Wang@amd.com>
>>> ---
>>>  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 7 +++++++
>>>  1 file changed, 7 insertions(+)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c 
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> index b23cb463b106..0f86f0b2e183 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
>>> @@ -2959,6 +2959,9 @@ int amdgpu_vm_init(struct amdgpu_device *adev, 
>>> struct amdgpu_vm *vm)
>>>          vm->use_cpu_for_update = !!(adev->vm_manager.vm_update_mode &
>>> AMDGPU_VM_USE_CPU_FOR_GFX);
>>>
>>> +       if (vm->use_cpu_for_update && amdgpu_sriov_vf(adev) && 
>>> amdgpu_virt_mmio_blocked(adev))
>>> +               vm->use_cpu_for_update = false;
>>> +
>>>          DRM_DEBUG_DRIVER("VM update mode is %s\n",
>>> vm->use_cpu_for_update ? "CPU" : "SDMA");
>>> WARN_ONCE((vm->use_cpu_for_update &&
>>> @@ -3094,6 +3097,10 @@ int amdgpu_vm_make_compute(struct 
>>> amdgpu_device *adev, struct amdgpu_vm *vm)
>>>          /* Update VM state */
>>>          vm->use_cpu_for_update = !!(adev->vm_manager.vm_update_mode &
>>> AMDGPU_VM_USE_CPU_FOR_COMPUTE);
>>> +
>>> +       if (vm->use_cpu_for_update && amdgpu_sriov_vf(adev) && 
>>> amdgpu_virt_mmio_blocked(adev))
>>> +               vm->use_cpu_for_update = false;
>>> +
>>>          DRM_DEBUG_DRIVER("VM update mode is %s\n",
>>> vm->use_cpu_for_update ? "CPU" : "SDMA");
>>> WARN_ONCE((vm->use_cpu_for_update &&
>>> -- 
>>> 2.25.1
>>>
>>
>