Re: [PATCH v11 10/22] vfio iommu type1: Add support for mediated devices

From: Kirti Wankhede <kwankhede@nvidia.com>
To: Alexey Kardashevskiy <aik@ozlabs.ru>,
	<alex.williamson@redhat.com>, <pbonzini@redhat.com>,
	<kraxel@redhat.com>, <cjia@nvidia.com>
Cc: <qemu-devel@nongnu.org>, <kvm@vger.kernel.org>,
	<kevin.tian@intel.com>, <jike.song@intel.com>,
	<bjsdjshi@linux.vnet.ibm.com>, <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v11 10/22] vfio iommu type1: Add support for mediated devices
Date: Tue, 15 Nov 2016 12:03:00 +0530	[thread overview]
Message-ID: <5219635a-011b-ed6a-9535-867085b072b0@nvidia.com> (raw)
In-Reply-To: <c775694e-782b-271c-1a2f-d276dfdf639b@ozlabs.ru>

On 11/15/2016 10:47 AM, Alexey Kardashevskiy wrote:
> On 08/11/16 17:52, Alexey Kardashevskiy wrote:
>> On 05/11/16 08:10, Kirti Wankhede wrote:
>>> VFIO IOMMU drivers are designed for the devices which are IOMMU capable.
>>> Mediated device only uses IOMMU APIs, the underlying hardware can be
>>> managed by an IOMMU domain.
>>>
>>> Aim of this change is:
>>> - To use most of the code of TYPE1 IOMMU driver for mediated devices
>>> - To support direct assigned device and mediated device in single module
>>>
>>> This change adds pin and unpin support for mediated device to TYPE1 IOMMU
>>> backend module. More details:
>>> - vfio_pin_pages() callback here uses task and address space of vfio_dma,
>>>   that is, of the process who mapped that iova range.
>>> - Added pfn_list tracking logic to address space structure. All pages
>>>   pinned through this interface are trached in its address space.
>>> - Pinned pages list is used to verify unpinning request and to unpin
>>>   remaining pages while detaching the group for that device.
>>> - Page accounting is updated to account in its address space where the
>>>   pages are pinned/unpinned.
>>> -  Accouting for mdev device is only done if there is no iommu capable
>>>   domain in the container. When there is a direct device assigned to the
>>>   container and that domain is iommu capable, all pages are already pinned
>>>   during DMA_MAP.
>>> - Page accouting is updated on hot plug and unplug mdev device and pass
>>>   through device.
>>>
>>> Tested by assigning below combinations of devices to a single VM:
>>> - GPU pass through only
>>
>> This does not require this patchset, right?
>>

Sorry I missed this earlier.
This testing is required for this patch, because this patch touches code
that is used for direct device assignment. Also for page accounting, all
cases are considered i.e. when there is only pass through device in a
container, when there is pass through device + vGPU device in a
container. Also have to test that pages are pinned properly when device
is hotplugged. In that case vfio_iommu_replay() is called to take
necessary action.

>>> - vGPU device only
>>
>> Out of curiosity - how exactly did you test this? The exact GPU, how to
>> create vGPU, what was the QEMU command line and the guest does with this
>> passed device? Thanks.
> 
> ping?
> 

I'm testing this code with M60, with custom changes in our driver.
Steps how to create mediated device are listed in
Documentation/vfio-mediated-device.txt for sample mtty driver. Same
steps I'm following for GPU. Quoting those steps here for you:

2. Create a mediated device by using the dummy device that you created
in the
   previous step.

   # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" >      \

/sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create

3. Add parameters to qemu-kvm.

   -device vfio-pci,\
    sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001

Thanks,
Kirti