Re: [Qemu-devel] [PATCH 1/5] VFIO KABI for migration interface

[Kirti Wankhede <kwankhede@nvidia.com>]

On 11/23/2018 12:24 AM, Dr. David Alan Gilbert wrote:
> * Kirti Wankhede (kwankhede@nvidia.com) wrote:
>> - Defined MIGRATION region type and sub-type.
>> - Defined VFIO device states during migration process.
>> - Defined vfio_device_migration_info structure which will be placed at 0th
>>   offset of migration region to get/set VFIO device related information.
>>   Defined actions and members of structure usage for each action:
>>     * To convey VFIO device state to be transitioned to.
>>     * To get pending bytes yet to be migrated for VFIO device
>>     * To ask driver to write data to migration region and return number of bytes
>>       written in the region
>>     * In migration resume path, user space app writes to migration region and
>>       communicates it to vendor driver.
>>     * Get bitmap of dirty pages from vendor driver from given start address
>>
>> Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
>> Reviewed-by: Neo Jia <cjia@nvidia.com>
> 
> <snip>
> 
>> + * Action Get buffer:
>> + *      On this action, vendor driver should write data to migration region and
>> + *      return number of bytes written in the region.
>> + *      data.offset [output] : offset in the region from where data is written.
>> + *      data.size [output] : number of bytes written in migration buffer by
>> + *          vendor driver.
> 
> <snip>
> 
>> + */
>> +
>> +struct vfio_device_migration_info {
>> +        __u32 device_state;         /* VFIO device state */
>> +        struct {
>> +            __u64 precopy_only;
>> +            __u64 compatible;
>> +            __u64 postcopy_only;
>> +            __u64 threshold_size;
>> +        } pending;
>> +        struct {
>> +            __u64 offset;           /* offset */
>> +            __u64 size;             /* size */
>> +        } data;
> 
> I'm curious how the offsets/size work; how does the 
> kernel driver know the maximum size of state it's allowed to write?


Migration region looks like:
 ----------------------------------------------------------------------
|vfio_device_migration_info|    data section			      |	
|                          |     ///////////////////////////////////  |
 ----------------------------------------------------------------------
 ^			         ^                                 ^
 offset 0-trapped part         data.offset                     data.size


Kernel driver defines the size of migration region and tells VFIO user
space application (QEMU here) through VFIO_DEVICE_GET_REGION_INFO ioctl.
So kernel driver can calculate the size of data section. Then kernel
driver can have (data.size >= data section size) or (data.size < data
section size), hence VFIO user space application need to know data.size
to copy only relevant data.

> Why would it pick a none-0 offset into the output region?

Data section is always followed by vfio_device_migration_info structure
in the region, so data.offset will always be none-0.
Offset from where data is copied is decided by kernel driver, data
section can be trapped or mapped depending on how kernel driver defines
data section. If mmapped, then data.offset should be page aligned, where
as initial section which contain vfio_device_migration_info structure
might not end at offset which is page aligned.

Thanks,
Kirti

> Without having dug further these feel like i/o rather than just output;
> i.e. the calling process says 'put it at that offset and you've got size
> bytes' and the kernel replies with 'I did put it at offset and I wrote
> only this size bytes'
> 
> Dave
> 
>> +        struct {
>> +            __u64 start_addr;
>> +            __u64 total;
>> +            __u64 copied;
>> +        } dirty_pfns;
>> +} __attribute__((packed));
>> +
>>  /* -------- API for Type1 VFIO IOMMU -------- */
>>  
>>  /**
>> -- 
>> 2.7.0
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>