From: "Tarun Gupta (SW-GPU)" <targupta@nvidia.com>
To: Cornelia Huck <cohuck@redhat.com>
Cc: kevin.tian@intel.com, cjia@nvidia.com, quintela@redhat.com,
qemu-devel@nongnu.org, Kirti Wankhede <kwankhede@nvidia.com>,
dgilbert@redhat.com, yan.y.zhao@intel.com, lushenming@huawei.com,
alex.williamson@redhat.com, dnigam@nvidia.com,
berrange@redhat.com, philmd@redhat.com
Subject: Re: [PATCH v2 1/1] docs/devel: Add VFIO device migration documentation
Date: Tue, 16 Mar 2021 21:48:38 +0530 [thread overview]
Message-ID: <c01589ee-d007-77da-83d1-fb6096f6f144@nvidia.com> (raw)
In-Reply-To: <20210315182258.586dbf23.cohuck@redhat.com>
On 3/15/2021 10:52 PM, Cornelia Huck wrote:
>
> On Thu, 11 Mar 2021 00:50:09 +0530
> Tarun Gupta <targupta@nvidia.com> wrote:
>
>> Document interfaces used for VFIO device migration. Added flow of state changes
>> during live migration with VFIO device. Tested by building docs with the new
>> vfio-migration.rst file.
>>
>> v2:
>> - Included the new vfio-migration.rst file in index.rst
>> - Updated dirty page tracking section, also added details about
>> 'pre-copy-dirty-page-tracking' opt-out option.
>> - Incorporated comments around wording of doc.
>>
>> Signed-off-by: Tarun Gupta <targupta@nvidia.com>
>> Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
>> ---
>> MAINTAINERS | 1 +
>> docs/devel/index.rst | 1 +
>> docs/devel/vfio-migration.rst | 135 ++++++++++++++++++++++++++++++++++
>> 3 files changed, 137 insertions(+)
>> create mode 100644 docs/devel/vfio-migration.rst
>>
>
> (...)
>
>> diff --git a/docs/devel/vfio-migration.rst b/docs/devel/vfio-migration.rst
>> new file mode 100644
>> index 0000000000..6196fb132c
>> --- /dev/null
>> +++ b/docs/devel/vfio-migration.rst
>> @@ -0,0 +1,135 @@
>> +=====================
>> +VFIO device Migration
>> +=====================
>
> Maybe add an introductory sentence or two describing the general
> approach? I.e. we have a general framework, and specific support for
> devices needs to be hooked up.
Ummm, the below paragraph does describe the approach we're using for the
migration framework involving pre-copy and stop-and-copy phase.
Can you help elaborate more on the general approach you'd like to have?
>
>> +
>> +VFIO devices use an iterative approach for migration because certain VFIO
>> +devices (e.g. GPU) have large amount of data to be transfered. The iterative
>> +pre-copy phase of migration allows for the guest to continue whilst the VFIO
>> +device state is transferred to the destination, this helps to reduce the total
>> +downtime of the VM. VFIO devices can choose to skip the pre-copy phase of
>> +migration by returning pending_bytes as zero during the pre-copy phase.
>
> What about something like:
>
> "Migration of VFIO devices consists of two phases: the optional
> pre-copy phase, and the stop-and-copy phase. The pre-copy phase is
> iterative and allows to accommodate VFIO devices that have a large
> amount of data that needs to be transferred. The iterative pre-copy
> phase..."
>
Thanks, this looks better. I'll update it in next version incorporating
the other comments too below.
Thanks,
Tarun
>> +
>> +A detailed description of the UAPI for VFIO device migration can be found in
>> +the comment for the ``vfio_device_migration_info`` structure in the header
>> +file linux-headers/linux/vfio.h.
>> +
>> +VFIO device hooks for iterative approach:
>> +
>> +* A ``save_setup`` function that sets up the migration region, sets _SAVING
>> + flag in the VFIO device state and informs the VFIO IOMMU module to start
>> + dirty page tracking.
>> +
>> +* A ``load_setup`` function that sets up the migration region on the
>> + destination and sets _RESUMING flag in the VFIO device state.
>> +
>> +* A ``save_live_pending`` function that reads pending_bytes from the vendor
>> + driver, which indicates the amount of data that the vendor driver has yet to
>> + save for the VFIO device.
>> +
>> +* A ``save_live_iterate`` function that reads the VFIO device's data from the
>> + vendor driver through the migration region during iterative phase.
>> +
>> +* A ``save_live_complete_precopy`` function that resets _RUNNING flag from the
>> + VFIO device state, saves the device config space, if any, and iteratively
>> + copies the remaining data for the VFIO device untill the vendor driver
>
> s/untill/until/
>
>> + indicates that no data remains (pending bytes is zero).
>> +
>> +* A ``load_state`` function that loads the config section and the data
>> + sections that are generated by the save functions above
>> +
>> +* ``cleanup`` functions for both save and load that perform any migration
>> + related cleanup, including unmapping the migration region
>> +
>> +A VM state change handler is registered to change the VFIO device state when
>> +the VM state changes.
>> +
>> +Similarly, a migration state change notifier is registered to get a
>> +notification on migration state change. These states are translated to VFIO
>
> s/to/to the corresponding/
>
>> +device state and conveyed to vendor driver.
>
> s/to/to the/
>
> (...)
>
>> +Postcopy
>> +========
>> +
>> +Postcopy migration is not supported for VFIO devices.
>
> s/is not/is currently not/ ?
>
next prev parent reply other threads:[~2021-03-16 16:24 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-10 19:20 [PATCH v2 1/1] docs/devel: Add VFIO device migration documentation Tarun Gupta
2021-03-11 9:40 ` Daniel P. Berrangé
2021-03-11 19:41 ` Dr. David Alan Gilbert
2021-03-12 2:30 ` Tian, Kevin
2021-03-16 15:46 ` Dr. David Alan Gilbert
2021-03-17 1:44 ` Tian, Kevin
2021-03-12 3:13 ` Tian, Kevin
2021-03-16 13:34 ` Tarun Gupta (SW-GPU)
2021-03-17 1:51 ` Tian, Kevin
2021-03-15 17:22 ` Cornelia Huck
2021-03-16 16:18 ` Tarun Gupta (SW-GPU) [this message]
2021-03-18 12:28 ` Cornelia Huck
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c01589ee-d007-77da-83d1-fb6096f6f144@nvidia.com \
--to=targupta@nvidia.com \
--cc=alex.williamson@redhat.com \
--cc=berrange@redhat.com \
--cc=cjia@nvidia.com \
--cc=cohuck@redhat.com \
--cc=dgilbert@redhat.com \
--cc=dnigam@nvidia.com \
--cc=kevin.tian@intel.com \
--cc=kwankhede@nvidia.com \
--cc=lushenming@huawei.com \
--cc=philmd@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=quintela@redhat.com \
--cc=yan.y.zhao@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).