Re: [RFC PATCH] drm/ttm, drm/vmwgfx: Have TTM support AMD SEV encryption

["Christian König" <ckoenig.leichtzumerken@gmail.com>]

Am 28.05.19 um 16:48 schrieb Lendacky, Thomas:
> On 5/28/19 2:31 AM, Thomas Hellstrom wrote:
>> Hi, Tom,
>>
>> Could you shed some light on this?
> I don't have a lot of GPU knowledge, so let me start with an overview of
> how everything should work and see if that answers the questions being
> asked.
>
> First, SME:
> The encryption bit is bit-47 of a physical address. So, if a device does
> not support at least 48-bit DMA, it will have to use the SWIOTLB and
> bounce buffer the data. This is handled automatically if the driver is
> using the Linux DMA-api as all of SWIOTLB has been marked un-encrypted.
> Data is bounced between the un-encrypted SWIOTLB and the (presumably)
> encrypted area of the driver.

Ok, that explains why we don't need to manually handle the encryption 
bit in TTM.

> For SEV:
> The encryption bit position is the same as SME. However, with SEV all
> DMA must use an un-encrypted area so all DMA goes through SWIOTLB. Just
> like SME, this is handled automatically if the driver is using the Linux
> DMA-api as all of SWIOTLB has been marked un-encrypted. And just like SME,
> data is bounced between the un-encrypted SWIOTLB and the (presumably)
> encrypted area of the driver.
>
> There is an optimization for dma_alloc_coherent() where the pages are
> allocated and marked un-encrypted, thus avoiding the bouncing (see file
> kernel/dma/direct.c, dma_direct_alloc_pages()).

And that explains why we have to use dma_alloc_coherent()....

> As for kernel vmaps and user-maps, those pages will be marked encrypted
> (unless explicitly made un-encrypted by calling set_memory_decrypted()).
> But, if you are copying to/from those areas into the un-encrypted DMA
> area then everything will be ok.
>
> Things get fuzzy for me when it comes to the GPU access of the memory
> and what and how it is accessed.

I can fill in those lose ends, but it's probably going to be a long mail 
and a bit off topic.

Thanks for filling in how that stuff actually works. And yeah, as far as 
I can see we actually don't need to do anything.

The only way to get un-encrypted pages which don't bounce through 
SWIOTLB is using dma_alloc_coherent().

It's probably a bit unfortunate that TTM can't control it, but I think 
Thomas has to live with that. The use case for sharing encrypted pages 
is probably not so common anyway.

Thanks,
Christian.

>
> Thanks,
> Tom
>
>> Thanks,
>> Thomas
>>
>>
>> On 5/24/19 5:08 PM, Alex Deucher wrote:
>>> + Tom
>>>
>>> He's been looking into SEV as well.
>>>
>>> On Fri, May 24, 2019 at 8:30 AM Thomas Hellstrom <thomas@shipmail.org>
>>> wrote:
>>>> On 5/24/19 2:03 PM, Koenig, Christian wrote:
>>>>> Am 24.05.19 um 12:37 schrieb Thomas Hellstrom:
>>>>>> [CAUTION: External Email]
>>>>>>
>>>>>> On 5/24/19 12:18 PM, Koenig, Christian wrote:
>>>>>>> Am 24.05.19 um 11:55 schrieb Thomas Hellstrom:
>>>>>>>> [CAUTION: External Email]
>>>>>>>>
>>>>>>>> On 5/24/19 11:11 AM, Thomas Hellstrom wrote:
>>>>>>>>> Hi, Christian,
>>>>>>>>>
>>>>>>>>> On 5/24/19 10:37 AM, Koenig, Christian wrote:
>>>>>>>>>> Am 24.05.19 um 10:11 schrieb Thomas Hellström (VMware):
>>>>>>>>>>> [CAUTION: External Email]
>>>>>>>>>>>
>>>>>>>>>>> From: Thomas Hellstrom <thellstrom@vmware.com>
>>>>>>>>>>>
>>>>>>>>>>> With SEV encryption, all DMA memory must be marked decrypted
>>>>>>>>>>> (AKA "shared") for devices to be able to read it. In the future we
>>>>>>>>>>> might
>>>>>>>>>>> want to be able to switch normal (encrypted) memory to decrypted in
>>>>>>>>>>> exactly
>>>>>>>>>>> the same way as we handle caching states, and that would require
>>>>>>>>>>> additional
>>>>>>>>>>> memory pools. But for now, rely on memory allocated with
>>>>>>>>>>> dma_alloc_coherent() which is already decrypted with SEV enabled.
>>>>>>>>>>> Set up
>>>>>>>>>>> the page protection accordingly. Drivers must detect SEV enabled
>>>>>>>>>>> and
>>>>>>>>>>> switch
>>>>>>>>>>> to the dma page pool.
>>>>>>>>>>>
>>>>>>>>>>> This patch has not yet been tested. As a follow-up, we might
>>>>>>>>>>> want to
>>>>>>>>>>> cache decrypted pages in the dma page pool regardless of their
>>>>>>>>>>> caching
>>>>>>>>>>> state.
>>>>>>>>>> This patch is unnecessary, SEV support already works fine with at
>>>>>>>>>> least
>>>>>>>>>> amdgpu and I would expect that it also works with other drivers as
>>>>>>>>>> well.
>>>>>>>>>>
>>>>>>>>>> Also see this patch:
>>>>>>>>>>
>>>>>>>>>> commit 64e1f830ea5b3516a4256ed1c504a265d7f2a65c
>>>>>>>>>> Author: Christian König <christian.koenig@amd.com>
>>>>>>>>>> Date:   Wed Mar 13 10:11:19 2019 +0100
>>>>>>>>>>
>>>>>>>>>>           drm: fallback to dma_alloc_coherent when memory
>>>>>>>>>> encryption is
>>>>>>>>>> active
>>>>>>>>>>
>>>>>>>>>>           We can't just map any randome page we get when memory
>>>>>>>>>> encryption is
>>>>>>>>>>           active.
>>>>>>>>>>
>>>>>>>>>>           Signed-off-by: Christian König <christian.koenig@amd.com>
>>>>>>>>>>           Acked-by: Alex Deucher <alexander.deucher@amd.com>
>>>>>>>>>>           Link: https://patchwork.kernel.org/patch/10850833/
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Christian.
>>>>>>>>> Yes, I noticed that. Although I fail to see where we automagically
>>>>>>>>> clear the PTE encrypted bit when mapping coherent memory? For the
>>>>>>>>> linear kernel map, that's done within dma_alloc_coherent() but for
>>>>>>>>> kernel vmaps and and user-space maps? Is that done automatically by
>>>>>>>>> the x86 platform layer?
>>>>>>> Yes, I think so. Haven't looked to closely at this either.
>>>>>> This sounds a bit odd. If that were the case, the natural place would be
>>>>>> the PAT tracking code, but it only handles caching flags AFAICT. Not
>>>>>> encryption flags.
>>>>>>
>>>>>> But when you tested AMD with SEV, was that running as hypervisor rather
>>>>>> than a guest, or did you run an SEV guest with PCI passthrough to the
>>>>>> AMD device?
>>>>> Yeah, well the problem is we never tested this ourself :)
>>>>>
>>>>>>>>> /Thomas
>>>>>>>>>
>>>>>>>> And, as a follow up question, why do we need dma_alloc_coherent() when
>>>>>>>> using SME? I thought the hardware performs the decryption when DMA-ing
>>>>>>>> to / from an encrypted page with SME, but not with SEV?
>>>>>>> I think the issue was that the DMA API would try to use a bounce buffer
>>>>>>> in this case.
>>>>>> SEV forces SWIOTLB bouncing on, but not SME. So it should probably be
>>>>>> possible to avoid dma_alloc_coherent() in the SME case.
>>>>> In this case I don't have an explanation for this.
>>>>>
>>>>> For the background what happened is that we got reports that SVE/SME
>>>>> doesn't work with amdgpu. So we told the people to try using the
>>>>> dma_alloc_coherent() path and that worked fine. Because of this we came
>>>>> up with the patch I noted earlier.
>>>>>
>>>>> I can confirm that it indeed works now for a couple of users, but we
>>>>> still don't have a test system for this in our team.
>>>>>
>>>>> Christian.
>>>> OK, undestood,
>>>>
>>>> But unless there is some strange magic going on, (which there might be
>>>> of course),I do think the patch I sent is correct, and the reason that
>>>> SEV works is that the AMD card is used by the hypervisor and not the
>>>> guest, and TTM is actually incorrectly creating conflicting maps and
>>>> treating the coherent memory as encrypted. But since the memory is only
>>>> accessed through encrypted PTEs, the hardware does the right thing,
>>>> using the hypervisor key for decryption....
>>>>
>>>> But that's only a guess, and this is not super-urgent. I will be able to
>>>> follow up if / when we bring vmwgfx up for SEV.
>>>>
>>>> /Thomas
>>>>
>>>>>> /Thomas
>>>>>>
>>>>>>
>>>>>>> Christian.
>>>>>>>
>>>>>>>> Thanks, Thomas
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>> _______________________________________________
>>>> dri-devel mailing list
>>>> dri-devel@lists.freedesktop.org
>>>> https://lists.freedesktop.org/mailman/listinfo/dri-devel
>>
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel