Re: [PATCH v11 10/22] vfio iommu type1: Add support for mediated devices

[Kirti Wankhede <kwankhede@nvidia.com>]

On 11/15/2016 12:57 PM, Alexey Kardashevskiy wrote:
> On 15/11/16 17:33, Kirti Wankhede wrote:
>>
>>
>> On 11/15/2016 10:47 AM, Alexey Kardashevskiy wrote:
>>> On 08/11/16 17:52, Alexey Kardashevskiy wrote:
>>>> On 05/11/16 08:10, Kirti Wankhede wrote:
>>>>> VFIO IOMMU drivers are designed for the devices which are IOMMU capable.
>>>>> Mediated device only uses IOMMU APIs, the underlying hardware can be
>>>>> managed by an IOMMU domain.
>>>>>
>>>>> Aim of this change is:
>>>>> - To use most of the code of TYPE1 IOMMU driver for mediated devices
>>>>> - To support direct assigned device and mediated device in single module
>>>>>
>>>>> This change adds pin and unpin support for mediated device to TYPE1 IOMMU
>>>>> backend module. More details:
>>>>> - vfio_pin_pages() callback here uses task and address space of vfio_dma,
>>>>>   that is, of the process who mapped that iova range.
>>>>> - Added pfn_list tracking logic to address space structure. All pages
>>>>>   pinned through this interface are trached in its address space.
>>>>> - Pinned pages list is used to verify unpinning request and to unpin
>>>>>   remaining pages while detaching the group for that device.
>>>>> - Page accounting is updated to account in its address space where the
>>>>>   pages are pinned/unpinned.
>>>>> -  Accouting for mdev device is only done if there is no iommu capable
>>>>>   domain in the container. When there is a direct device assigned to the
>>>>>   container and that domain is iommu capable, all pages are already pinned
>>>>>   during DMA_MAP.
>>>>> - Page accouting is updated on hot plug and unplug mdev device and pass
>>>>>   through device.
>>>>>
>>>>> Tested by assigning below combinations of devices to a single VM:
>>>>> - GPU pass through only
>>>>
>>>> This does not require this patchset, right?
>>>>
>>
>> Sorry I missed this earlier.
>> This testing is required for this patch, because this patch touches code
>> that is used for direct device assignment. Also for page accounting, all
>> cases are considered i.e. when there is only pass through device in a
>> container, when there is pass through device + vGPU device in a
>> container. Also have to test that pages are pinned properly when device
>> is hotplugged. In that case vfio_iommu_replay() is called to take
>> necessary action.
> 
> So in this particular test you are only testing that the patchset did not
> break the already existing functionality, is that correct?
> 
> 

Yes.

>>
>>>>> - vGPU device only
>>>>
>>>> Out of curiosity - how exactly did you test this? The exact GPU, how to
>>>> create vGPU, what was the QEMU command line and the guest does with this
>>>> passed device? Thanks.
>>>
>>> ping?
>>>
>>
>> I'm testing this code with M60, with custom changes in our driver.
> 
> 
> Is this shared anywhere? What does the mediated driver do? Can Tesla K80 do
> the same thing, or [10de:15fe] (whatever its name is)?
> 
> 

No, we are still in the process of upstreaming this patchset. After that
we would prepare official drivers. There is no timeline for the driver
release yet.

>> Steps how to create mediated device are listed in
>> Documentation/vfio-mediated-device.txt for sample mtty driver. Same
>> steps I'm following for GPU. Quoting those steps here for you:
> 
> 
> Nah, I saw this, I was wondering about actual hardware :) Like when you say
> "tested with vGPU" - I am wondering what is passed to the guest and how the
> guest is actually using it.
> 
> 

Its similar to mtty sample driver, that simulates serial port and so you
see serial ports in guest. For vGPU, you see a virtual GPU in guest.


Thanks,
Kirti

>>
>> 2. Create a mediated device by using the dummy device that you created
>> in the
>>    previous step.
>>
>>    # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" >      \
>>
>> /sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create
>>
>> 3. Add parameters to qemu-kvm.
>>
>>    -device vfio-pci,\
>>     sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001
> 
> 
> 
>