[RFC] vfio/migration: The way we start dirty tracking for the VFIO container?

[RFC] vfio/migration: The way we start dirty tracking for the VFIO container?

[Zenghui Yu]

Hi folks,

I'm looking at the way we start dirty tracking for the vfio container
(we start it at the save_setup stage of a device) and have a few
questions about it.  Please fix me up if I had missed something obvious.


- We may end up with the situation where we try to get dirty bitmap of
   the vfio container whilst the dirty tracking hasn't yet been enabled.

     ram_save_setup
         ram_init_bitmaps
             migration_bitmap_sync_precopy (i.e., vfio_listener_log_sync)
     vfio_save_setup
         vfio_migration_set_state += _SAVING
         vfio_set_dirty_page_tracking(true)

   Practically there is no problem since in vfio_listener_log_sync(), we
   will check that if all devices are having the _SAVING state *before*
   actually syncing dirty bitmap.  As we add _SAVING to the device state
   exactly before starting dirty tracking, we'll actually do nothing in
   the first run of vfio_listener_log_sync().

   But I'm wondering what prevents us from starting dirty tracking via
   the "standard" log_start() callback of the vfio container.


- Before start dirty tracking, we will check and ensure that the device
   is at _SAVING state and return error otherwise.  But the question is
   that what is the rationale?  Why does the VFIO_IOMMU_DIRTY_PAGES ioctl
   have something to do with the device state?

   VFIO_IOMMU_DIRTY_PAGES is clearly designed as a API for type1 VFIO
   IOMMU.  Is it possible that the *device* itself becomes the actual
   dirty bitmap provider of this ioctl (e.g., the device is smart enough
   to track its own dirty pages during migration)?  This is the only case
   I can think out where we may have to set the device to pre-copy state
   before starting dirty tracking, so that device can report dirty pages
   properly in response to the GET_BITMAP ioctl.  But afaict it isn't
   covered by the existing documentation [1][2]...

   Btw, it isn't clear that what possible dirty bitmap providers will be
   (IOMMU, devices, etc) behind the back of the VFIO_IOMMU_DIRTY_PAGES
   ioctl.  Although userspace can be unaware of any of this, it'd be
   appreciated if someone can clarify it.


- If there is indeed some dependency between the device state and the
   VFIO_IOMMU_DIRTY_PAGES_FLAG_START operation, shouldn't we check that
   *all* devices are now having the _SAVING state before START, rather
   than just checking the *given* device in its (per-device) save_setup
   stage?

[1] https://git.kernel.org/torvalds/c/a8a24f3f6e38
[2] https://git.kernel.org/torvalds/c/b704fd14a06f


Thanks,
Zenghui