From: Kirti Wankhede <kwankhede@nvidia.com>
To: alex.williamson@redhat.com, cjia@nvidia.com
Cc: kevin.tian@intel.com, ziye.yang@intel.com,
changpeng.liu@intel.com, yi.l.liu@intel.com, mlevitsk@redhat.com,
eskultet@redhat.com, cohuck@redhat.com, dgilbert@redhat.com,
jonathan.davies@nutanix.com, eauger@redhat.com, aik@ozlabs.ru,
pasic@linux.ibm.com, felipe@nutanix.com,
Zhengxiao.zx@Alibaba-inc.com, shuangtai.tst@alibaba-inc.com,
Ken.Xue@amd.com, zhi.a.wang@intel.com, qemu-devel@nongnu.org,
Kirti Wankhede <kwankhede@nvidia.com>
Subject: [Qemu-devel] [PATCH 0/5] Add migration support for VFIO device
Date: Wed, 21 Nov 2018 02:09:38 +0530 [thread overview]
Message-ID: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> (raw)
Add migration support for VFIO device
This Patch set include patches as below:
- Define KABI for VFIO device for migration support.
- Added save and restore functions for PCI configuration space
- Generic migration functionality for VFIO device.
* This patch set adds functionality only for PCI devices, but can be
extended to other VFIO devices.
* Added all the basic functions required for pre-copy, stop-and-copy and
resume phases of migration.
* Added state change notifier and from that notifier function, VFIO
device's state changed is conveyed to VFIO device driver.
* During save setup phase and resume/load setup phase, migration region
is queried and is used to read/write VFIO device data.
* .save_live_pending, .save_live_iterate and .is_active_iterate are
implemented to use QEMU's functionality of iteration during pre-copy
phase.
* In .save_live_complete_precopy, that is in stop-and-copy phase,
iteration to read data from VFIO device driver is implemented till pending
bytes returned by driver are not zero.
* .save_cleanup and .load_cleanup are implemented to unmap migration
region that was setup duing setup phase.
* Added function to get dirty pages bitmap for the pages which are used by
driver.
- Add vfio_listerner_log_sync to mark dirty pages.
- Make VFIO PCI device migration capable. If migration region is not provided by
driver, migration is blocked.
Below is the flow of state change for live migration where states in brackets
represent VM state, migration state and VFIO device state as:
(VM state, MIGRATION_STATUS, VFIO_DEVICE_STATE)
Live migration save path:
QEMU normal running state
(RUNNING, _NONE, _RUNNING)
|
migrate_init spawns migration_thread.
(RUNNING, _SETUP, _MIGRATION_SETUP)
Migration thread then calls each device's .save_setup()
|
(RUNNING, _ACTIVE, _MIGRATION_PRECOPY)
If device is active, get pending bytes by .save_live_pending()
if pending bytes >= threshold_size, call save_live_iterate()
Data of VFIO device for pre-copy phase is copied.
Iterate till pending bytes converge and are less than threshold
|
migration_completion() stops vCPUs and calls .save_live_complete_precopy
for each active device. VFIO device is then transitioned in
_MIGRATION_STOPNCOPY state.
(FINISH_MIGRATE, _DEVICE, _MIGRATION_STOPNCOPY)
For VFIO device, iterate in .save_live_complete_precopy until
pending data is 0. Change VFIO device state.
(FINISH_MIGRATE, _DEVICE, _MIGRATION_SAVE_COMPLETED)
|
(FINISH_MIGRATE, _COMPLETED, _MIGRATION_SAVE_COMPLETED)
Migraton thread schedule cleanup bottom half and exit
|
(POST_MIGRATE, _COMPLETED, _MIGRATION_SAVE_COMPLETED)
For each device, call .save_cleanup(). Unmap migration region.
Live migration resume path:
Incomming migration calls .load_setup for each device
(RESTORE_VM, _ACTIVE, _MIGRATION_RESUME)
|
For each device, .load_state is called for that device section data
|
At the end, called .load_cleanup for each device and vCPUs are started.
(RUNNING, _NONE, _RUNNING)
Note that:
- Migration post copy is not supported.
- VFIO device driver version compatibility is not taken care in this series.
v1 -> v2:
- Defined MIGRATION region type and sub-type which should be used with region
type capability.
- Re-structured vfio_device_migration_info. This structure will be placed at 0th
offset of migration region.
- Replaced ioctl with read/write for trapped part of migration region.
- Added both type of access support, trapped or mmapped, for data section of the
region.
- Moved PCI device functions to pci file.
- Added iteration to get dirty page bitmap until bitmap for all requested pages
are copied.
Thanks,
Kirti
Kirti Wankhede (5):
VFIO KABI for migration interface
Add save and load functions for VFIO PCI devices
Add migration functions for VFIO devices
Add vfio_listerner_log_sync to mark dirty pages
Make vfio-pci device migration capable.
hw/vfio/Makefile.objs | 2 +-
hw/vfio/common.c | 32 ++
hw/vfio/migration.c | 729 ++++++++++++++++++++++++++++++++++++++++++
hw/vfio/pci.c | 108 ++++++-
hw/vfio/pci.h | 29 ++
include/hw/vfio/vfio-common.h | 23 ++
linux-headers/linux/vfio.h | 130 ++++++++
7 files changed, 1045 insertions(+), 8 deletions(-)
create mode 100644 hw/vfio/migration.c
--
2.7.0
next reply other threads:[~2018-11-20 20:41 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-20 20:39 Kirti Wankhede [this message]
2018-11-20 20:39 ` [Qemu-devel] [PATCH 1/5] VFIO KABI for migration interface Kirti Wankhede
2018-11-21 0:26 ` Tian, Kevin
2018-11-21 4:24 ` Kirti Wankhede
2018-11-21 6:13 ` Tian, Kevin
2018-11-22 20:01 ` Kirti Wankhede
2018-11-26 7:14 ` Tian, Kevin
2018-11-21 17:26 ` Pierre Morel
2018-11-22 18:54 ` Dr. David Alan Gilbert
2018-11-22 20:43 ` Kirti Wankhede
2018-11-23 11:44 ` Dr. David Alan Gilbert
2018-11-23 5:47 ` Zhao Yan
2018-11-27 19:52 ` Alex Williamson
2018-12-04 10:53 ` Cornelia Huck
2018-12-04 17:14 ` Alex Williamson
2018-11-20 20:39 ` [Qemu-devel] [PATCH 2/5] Add save and load functions for VFIO PCI devices Kirti Wankhede
2018-11-21 5:32 ` Peter Xu
2018-11-20 20:39 ` [Qemu-devel] [PATCH 3/5] Add migration functions for VFIO devices Kirti Wankhede
2018-11-21 7:39 ` Zhao, Yan Y
2018-11-22 21:21 ` Kirti Wankhede
2018-11-23 5:29 ` Zhao Yan
2018-11-22 8:22 ` Zhao, Yan Y
2018-11-22 19:57 ` Dr. David Alan Gilbert
2018-11-29 8:04 ` Zhao Yan
2018-12-17 11:19 ` Gonglei (Arei)
2018-12-19 2:12 ` Zhao Yan
2018-12-21 7:36 ` Zhi Wang
2018-11-20 20:39 ` [Qemu-devel] [PATCH 4/5] Add vfio_listerner_log_sync to mark dirty pages Kirti Wankhede
2018-11-22 20:00 ` Dr. David Alan Gilbert
2018-11-20 20:39 ` [Qemu-devel] [PATCH 5/5] Make vfio-pci device migration capable Kirti Wankhede
2018-11-21 5:47 ` [Qemu-devel] [PATCH 0/5] Add migration support for VFIO device Peter Xu
2018-11-22 21:01 ` Kirti Wankhede
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1542746383-18288-1-git-send-email-kwankhede@nvidia.com \
--to=kwankhede@nvidia.com \
--cc=Ken.Xue@amd.com \
--cc=Zhengxiao.zx@Alibaba-inc.com \
--cc=aik@ozlabs.ru \
--cc=alex.williamson@redhat.com \
--cc=changpeng.liu@intel.com \
--cc=cjia@nvidia.com \
--cc=cohuck@redhat.com \
--cc=dgilbert@redhat.com \
--cc=eauger@redhat.com \
--cc=eskultet@redhat.com \
--cc=felipe@nutanix.com \
--cc=jonathan.davies@nutanix.com \
--cc=kevin.tian@intel.com \
--cc=mlevitsk@redhat.com \
--cc=pasic@linux.ibm.com \
--cc=qemu-devel@nongnu.org \
--cc=shuangtai.tst@alibaba-inc.com \
--cc=yi.l.liu@intel.com \
--cc=zhi.a.wang@intel.com \
--cc=ziye.yang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.