Re: [Qemu-devel] [PATCH v4 00/13] Add migration support for VFIO device

[Kirti Wankhede <kwankhede@nvidia.com>]

On 6/21/2019 6:54 AM, Yan Zhao wrote:
> On Fri, Jun 21, 2019 at 08:25:18AM +0800, Yan Zhao wrote:
>> On Thu, Jun 20, 2019 at 10:37:28PM +0800, Kirti Wankhede wrote:
>>> Add migration support for VFIO device
>>>
>>> This Patch set include patches as below:
>>> - Define KABI for VFIO device for migration support.
>>> - Added save and restore functions for PCI configuration space
>>> - Generic migration functionality for VFIO device.
>>>   * This patch set adds functionality only for PCI devices, but can be
>>>     extended to other VFIO devices.
>>>   * Added all the basic functions required for pre-copy, stop-and-copy and
>>>     resume phases of migration.
>>>   * Added state change notifier and from that notifier function, VFIO
>>>     device's state changed is conveyed to VFIO device driver.
>>>   * During save setup phase and resume/load setup phase, migration region
>>>     is queried and is used to read/write VFIO device data.
>>>   * .save_live_pending and .save_live_iterate are implemented to use QEMU's
>>>     functionality of iteration during pre-copy phase.
>>>   * In .save_live_complete_precopy, that is in stop-and-copy phase,
>>>     iteration to read data from VFIO device driver is implemented till pending
>>>     bytes returned by driver are not zero.
>>>   * Added function to get dirty pages bitmap for the pages which are used by
>>>     driver.
>>> - Add vfio_listerner_log_sync to mark dirty pages.
>>> - Make VFIO PCI device migration capable. If migration region is not provided by
>>>   driver, migration is blocked.
>>>
>>> Below is the flow of state change for live migration where states in brackets
>>> represent VM state, migration state and VFIO device state as:
>>>     (VM state, MIGRATION_STATUS, VFIO_DEVICE_STATE)
>>>
>>> Live migration save path:
>>>         QEMU normal running state
>>>         (RUNNING, _NONE, _RUNNING)
>>>                         |
>>>     migrate_init spawns migration_thread.
>>>     (RUNNING, _SETUP, _RUNNING|_SAVING)
>>>     Migration thread then calls each device's .save_setup()
>>>                         |
>>>     (RUNNING, _ACTIVE, _RUNNING|_SAVING)
>>>     If device is active, get pending bytes by .save_live_pending()
>>>     if pending bytes >= threshold_size,  call save_live_iterate()
>>>     Data of VFIO device for pre-copy phase is copied.
>>>     Iterate till pending bytes converge and are less than threshold
>>>                         |
>>>     On migration completion, vCPUs stops and calls .save_live_complete_precopy
>>>     for each active device. VFIO device is then transitioned in
>>>      _SAVING state.
>>>     (FINISH_MIGRATE, _DEVICE, _SAVING)
>>>     For VFIO device, iterate in  .save_live_complete_precopy  until
>>>     pending data is 0.
>>>     (FINISH_MIGRATE, _DEVICE, _STOPPED)
>>
>> I suggest we also register to VMStateDescription, whose .pre_save
>> handler would get called after .save_live_complete_precopy in pre-copy
>> only case, and will called before .save_live_iterate in post-copy
>> enabled case.
>> In the .pre_save handler, we can save all device state which must be
>> copied after device stop in source vm and before device start in target vm.
>>
> hi
> to better describe this idea:
> 
> in pre-copy only case, the flow is
> 
> start migration --> .save_live_iterate (several round) -> stop source vm
> --> .save_live_complete_precopy --> .pre_save  -->start target vm
> -->migration complete
> 
> 
> in post-copy enabled case, the flow is
> 
> start migration --> .save_live_iterate (several round) --> start post copy --> 
> stop source vm --> .pre_save --> start target vm --> .save_live_iterate (several round) 
> -->migration complete
> 
> Therefore, we should put saving of device state in .pre_save interface
> rather than in .save_live_complete_precopy. 
> The device state includes pci config data, page tables, register state, etc.
> 
> The .save_live_iterate and .save_live_complete_precopy should only deal
> with saving dirty memory.
> 

Vendor driver can decide when to save device state depending on the VFIO
device state set by user. Vendor driver doesn't have to depend on which
callback function QEMU or user application calls. In pre-copy case,
save_live_complete_precopy sets VFIO device state to
VFIO_DEVICE_STATE_SAVING which means vCPUs are stopped and vendor driver
should save all device state.

> 
> I know current implementation does not support post-copy. but at least
> it should not require huge change when we decide to enable it in future.
> 

.has_postcopy and .save_live_complete_postcopy need to be implemented to
support post-copy. I think .save_live_complete_postcopy should be
similar to vfio_save_complete_precopy.

Thanks,
Kirti

> Thanks
> Yan
> 
>>>                         |
>>>     (FINISH_MIGRATE, _COMPLETED, STOPPED)
>>>     Migraton thread schedule cleanup bottom half and exit
>>>
>>> Live migration resume path:
>>>     Incomming migration calls .load_setup for each device
>>>     (RESTORE_VM, _ACTIVE, STOPPED)
>>>                         |
>>>     For each device, .load_state is called for that device section data
>>>                         |
>>>     At the end, called .load_cleanup for each device and vCPUs are started.
>>>                         |
>>>         (RUNNING, _NONE, _RUNNING)
>>>
>>> Note that:
>>> - Migration post copy is not supported.
>>>
>>> v3 -> v4:
>>> - Added one more bit for _RESUMING flag to be set explicitly.
>>> - data_offset field is read-only for user space application.
>>> - data_size is read for every iteration before reading data from migration, that
>>>   is removed assumption that data will be till end of migration region.
>>> - If vendor driver supports mappable sparsed region, map those region during
>>>   setup state of save/load, similarly unmap those from cleanup routines.
>>> - Handles race condition that causes data corruption in migration region during
>>>   save device state by adding mutex and serialiaing save_buffer and
>>>   get_dirty_pages routines.
>>> - Skip called get_dirty_pages routine for mapped MMIO region of device.
>>> - Added trace events.
>>> - Splitted into multiple functional patches.
>>>
>>> v2 -> v3:
>>> - Removed enum of VFIO device states. Defined VFIO device state with 2 bits.
>>> - Re-structured vfio_device_migration_info to keep it minimal and defined action
>>>   on read and write access on its members.
>>>
>>> v1 -> v2:
>>> - Defined MIGRATION region type and sub-type which should be used with region
>>>   type capability.
>>> - Re-structured vfio_device_migration_info. This structure will be placed at 0th
>>>   offset of migration region.
>>> - Replaced ioctl with read/write for trapped part of migration region.
>>> - Added both type of access support, trapped or mmapped, for data section of the
>>>   region.
>>> - Moved PCI device functions to pci file.
>>> - Added iteration to get dirty page bitmap until bitmap for all requested pages
>>>   are copied.
>>>
>>> Thanks,
>>> Kirti
>>>
>>>
>>> Kirti Wankhede (13):
>>>   vfio: KABI for migration interface
>>>   vfio: Add function to unmap VFIO region
>>>   vfio: Add save and load functions for VFIO PCI devices
>>>   vfio: Add migration region initialization and finalize function
>>>   vfio: Add VM state change handler to know state of VM
>>>   vfio: Add migration state change notifier
>>>   vfio: Register SaveVMHandlers for VFIO device
>>>   vfio: Add save state functions to SaveVMHandlers
>>>   vfio: Add load state functions to SaveVMHandlers
>>>   vfio: Add function to get dirty page list
>>>   vfio: Add vfio_listerner_log_sync to mark dirty pages
>>>   vfio: Make vfio-pci device migration capable.
>>>   vfio: Add trace events in migration code path
>>>
>>>  hw/vfio/Makefile.objs         |   2 +-
>>>  hw/vfio/common.c              |  55 +++
>>>  hw/vfio/migration.c           | 815 ++++++++++++++++++++++++++++++++++++++++++
>>>  hw/vfio/pci.c                 | 126 ++++++-
>>>  hw/vfio/pci.h                 |  29 ++
>>>  hw/vfio/trace-events          |  19 +
>>>  include/hw/vfio/vfio-common.h |  22 ++
>>>  linux-headers/linux/vfio.h    |  71 ++++
>>>  8 files changed, 1132 insertions(+), 7 deletions(-)
>>>  create mode 100644 hw/vfio/migration.c
>>>
>>> -- 
>>> 2.7.0
>>>