From: "Tarun Gupta (SW-GPU)" <email@example.com> To: Shenming Lu <firstname.lastname@example.org>, <email@example.com>, <firstname.lastname@example.org>, <email@example.com>, <firstname.lastname@example.org>, <email@example.com> Cc: firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org Subject: Re: [PATCH v3 1/1] docs/devel: Add VFIO device migration documentation Date: Thu, 1 Apr 2021 12:28:05 +0530 [thread overview] Message-ID: <email@example.com> (raw) In-Reply-To: <firstname.lastname@example.org> On 3/27/2021 11:34 AM, Shenming Lu wrote: > On 2021/3/26 21:18, Tarun Gupta wrote: >> Document interfaces used for VFIO device migration. Added flow of state changes >> during live migration with VFIO device. Tested by building docs with the new >> vfio-migration.rst file. >> >> v3: >> - Add introductory line about VM migration in general. >> - Remove occurcences of vfio_pin_pages() to describe pinning. >> - Incorporated comments from v2 >> >> v2: >> - Included the new vfio-migration.rst file in index.rst >> - Updated dirty page tracking section, also added details about >> 'pre-copy-dirty-page-tracking' opt-out option. >> - Incorporated comments around wording of doc. >> >> Signed-off-by: Tarun Gupta <email@example.com> >> Signed-off-by: Kirti Wankhede <firstname.lastname@example.org> >> --- >> MAINTAINERS | 1 + >> docs/devel/index.rst | 1 + >> docs/devel/vfio-migration.rst | 143 ++++++++++++++++++++++++++++++++++ >> 3 files changed, 145 insertions(+) >> create mode 100644 docs/devel/vfio-migration.rst >> >> diff --git a/MAINTAINERS b/MAINTAINERS >> index 738786146d..a2a80eee59 100644 >> --- a/MAINTAINERS >> +++ b/MAINTAINERS >> @@ -1801,6 +1801,7 @@ M: Alex Williamson <email@example.com> >> S: Supported >> F: hw/vfio/* >> F: include/hw/vfio/ >> +F: docs/devel/vfio-migration.rst >> >> vfio-ccw >> M: Cornelia Huck <firstname.lastname@example.org> >> diff --git a/docs/devel/index.rst b/docs/devel/index.rst >> index ae664da00c..5330f1ca1d 100644 >> --- a/docs/devel/index.rst >> +++ b/docs/devel/index.rst >> @@ -39,3 +39,4 @@ Contents: >> qom >> block-coroutine-wrapper >> multi-process >> + vfio-migration >> diff --git a/docs/devel/vfio-migration.rst b/docs/devel/vfio-migration.rst >> new file mode 100644 >> index 0000000000..24cb55991a >> --- /dev/null >> +++ b/docs/devel/vfio-migration.rst >> @@ -0,0 +1,143 @@ >> +===================== >> +VFIO device Migration >> +===================== >> + >> +Migration of virtual machine involves saving the state for each device that >> +the guest is running on source host and restoring this saved state on the >> +destination host. This document details how saving and restoring of VFIO >> +devices is done in QEMU. >> + >> +Migration of VFIO devices consists of two phases: the optional pre-copy phase, >> +and the stop-and-copy phase. The pre-copy phase is iterative and allows to >> +accommodate VFIO devices that have a large amount of data that needs to be >> +transferred. The iterative pre-copy phase of migration allows for the guest to >> +continue whilst the VFIO device state is transferred to the destination, this >> +helps to reduce the total downtime of the VM. VFIO devices can choose to skip >> +the pre-copy phase of migration by returning pending_bytes as zero during the >> +pre-copy phase. >> + >> +A detailed description of the UAPI for VFIO device migration can be found in >> +the comment for the ``vfio_device_migration_info`` structure in the header >> +file linux-headers/linux/vfio.h. >> + >> +VFIO device hooks for iterative approach: >> + >> +* A ``save_setup`` function that sets up the migration region, sets _SAVING >> + flag in the VFIO device state and informs the VFIO IOMMU module to start >> + dirty page tracking. >> + >> +* A ``load_setup`` function that sets up the migration region on the >> + destination and sets _RESUMING flag in the VFIO device state. >> + >> +* A ``save_live_pending`` function that reads pending_bytes from the vendor >> + driver, which indicates the amount of data that the vendor driver has yet to >> + save for the VFIO device. >> + >> +* A ``save_live_iterate`` function that reads the VFIO device's data from the >> + vendor driver through the migration region during iterative phase. >> + >> +* A ``save_live_complete_precopy`` function that resets _RUNNING flag from the >> + VFIO device state, saves the device config space, if any, and iteratively >> + copies the remaining data for the VFIO device until the vendor driver >> + indicates that no data remains (pending bytes is zero). > > Hi Tarun, > > We have moved the saving of the config space to the ``save_state`` function > added in commit d329f5032e1, do we need to add this change here? :-) > Thanks Shenming, I'll update it accordingly in the next version. Thanks, Tarun > Thanks, > Shenming > >> + >> +* A ``load_state`` function that loads the config section and the data >> + sections that are generated by the save functions above >> + >> +* ``cleanup`` functions for both save and load that perform any migration >> + related cleanup, including unmapping the migration region >> + >> +A VM state change handler is registered to change the VFIO device state when >> +the VM state changes. >> + >> +Similarly, a migration state change notifier is registered to get a >> +notification on migration state change. These states are translated to the >> +corresponding VFIO device state and conveyed to the vendor driver. >> + >> +System memory dirty pages tracking >> +---------------------------------- >> + >> +A ``log_sync`` memory listener callback marks those system memory pages >> +as dirty which are used for DMA by the VFIO device. The dirty pages bitmap is >> +queried per container. All pages pinned by the vendor driver through external >> +APIs have to be marked as dirty during migration. When there are CPU writes, >> +CPU dirty page tracking can identify dirtied pages, but any page pinned by the >> +vendor driver can also be written by device. There is currently no device or >> +IOMMU support for dirty page tracking in hardware. >> + >> +By default, dirty pages are tracked when the device is in pre-copy as well as >> +stop-and-copy phase. So, a page pinned by vendor driver will be copied to >> +destination in both the phases. Copying dirty pages in pre-copy phase helps >> +QEMU to predict if it can achieve its downtime tolerances. If QEMU during >> +pre-copy phase keeps finding dirty pages continuously, then it understands >> +that even in stop-and-copy phase, it is likely to find dirty pages and can >> +predict the downtime accordingly >> + >> +QEMU also provides per device opt-out option ``pre-copy-dirty-page-tracking`` >> +which disables querying dirty bitmap during pre-copy phase. If it is set to >> +off, all dirty pages will be copied to destination in stop-and-copy phase only >> + >> +System memory dirty pages tracking when vIOMMU is enabled >> +--------------------------------------------------------- >> + >> +With vIOMMU, an IO virtual address range can get unmapped while in pre-copy >> +phase of migration. In that case, the unmap ioctl returns any dirty pages in >> +that range and QEMU reports corresponding guest physical pages dirty. During >> +stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped >> +pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for those >> +mapped ranges. >> + >> +Flow of state changes during Live migration >> +=========================================== >> + >> +Below is the flow of state change during live migration. >> +The values in the brackets represent the VM state, the migration state, and >> +the VFIO device state, respectively. >> + >> +Live migration save path >> +------------------------ >> + >> +:: >> + >> + QEMU normal running state >> + (RUNNING, _NONE, _RUNNING) >> + | >> + migrate_init spawns migration_thread >> + Migration thread then calls each device's .save_setup() >> + (RUNNING, _SETUP, _RUNNING|_SAVING) >> + | >> + (RUNNING, _ACTIVE, _RUNNING|_SAVING) >> + If device is active, get pending_bytes by .save_live_pending() >> + If total pending_bytes >= threshold_size, call .save_live_iterate() >> + Data of VFIO device for pre-copy phase is copied >> + Iterate till total pending bytes converge and are less than threshold >> + | >> + On migration completion, vCPU stops and calls .save_live_complete_precopy for >> + each active device. The VFIO device is then transitioned into _SAVING state >> + (FINISH_MIGRATE, _DEVICE, _SAVING) >> + | >> + For the VFIO device, iterate in .save_live_complete_precopy until >> + pending data is 0 >> + (FINISH_MIGRATE, _DEVICE, _STOPPED) >> + | >> + (FINISH_MIGRATE, _COMPLETED, _STOPPED) >> + Migraton thread schedules cleanup bottom half and exits >> + >> +Live migration resume path >> +-------------------------- >> + >> +:: >> + >> + Incoming migration calls .load_setup for each device >> + (RESTORE_VM, _ACTIVE, _STOPPED) >> + | >> + For each device, .load_state is called for that device section data >> + (RESTORE_VM, _ACTIVE, _RESUMING) >> + | >> + At the end, .load_cleanup is called for each device and vCPUs are started >> + (RUNNING, _NONE, _RUNNING) >> + >> +Postcopy >> +======== >> + >> +Postcopy migration is currently not supported for VFIO devices. >>
next prev parent reply other threads:[~2021-04-01 7:14 UTC|newest] Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-03-26 13:18 Tarun Gupta 2021-03-27 6:04 ` Shenming Lu 2021-04-01 6:58 ` Tarun Gupta (SW-GPU) [this message] 2021-04-01 11:05 ` Cornelia Huck 2021-04-05 17:02 ` Tarun Gupta (SW-GPU) 2021-04-07 10:23 ` Cornelia Huck 2021-04-07 11:33 ` Tarun Gupta (SW-GPU)
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --subject='Re: [PATCH v3 1/1] docs/devel: Add VFIO device migration documentation' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).