From: Avihai Horon <avihaih@nvidia.com>
To: "Cédric Le Goater" <clg@redhat.com>,
"YangHang Liu" <yanghliu@redhat.com>
Cc: qemu-devel@nongnu.org,
"Alex Williamson" <alex.williamson@redhat.com>,
"Eduardo Habkost" <eduardo@habkost.net>,
"Marcel Apfelbaum" <marcel.apfelbaum@gmail.com>,
"Philippe Mathieu-Daudé" <philmd@linaro.org>,
"Yanan Wang" <wangyanan55@huawei.com>,
"Juan Quintela" <quintela@redhat.com>,
"Peter Xu" <peterx@redhat.com>,
"Leonardo Bras" <leobras@redhat.com>,
"Eric Blake" <eblake@redhat.com>,
"Markus Armbruster" <armbru@redhat.com>,
"Thomas Huth" <thuth@redhat.com>,
"Laurent Vivier" <lvivier@redhat.com>,
"Paolo Bonzini" <pbonzini@redhat.com>,
"Yishai Hadas" <yishaih@nvidia.com>,
"Jason Gunthorpe" <jgg@nvidia.com>,
"Maor Gottlieb" <maorg@nvidia.com>,
"Kirti Wankhede" <kwankhede@nvidia.com>,
"Tarun Gupta" <targupta@nvidia.com>,
"Joao Martins" <joao.m.martins@oracle.com>
Subject: Re: [PATCH v5 1/9] migration: Add switchover ack capability
Date: Mon, 19 Jun 2023 12:37:23 +0300 [thread overview]
Message-ID: <4786a4ce-bb1b-ab19-f038-744a32924192@nvidia.com> (raw)
In-Reply-To: <9fb7b40c-b6bf-07aa-5605-66b9fa7353cf@redhat.com>
On 15/06/2023 16:49, Cédric Le Goater wrote:
> External email: Use caution opening links or attachments
>
>
> On 6/15/23 14:38, YangHang Liu wrote:
>> Test in the following two scenarios:
>>
>> [1] Test scenario: Both source VM and target VM (in listening mode)
>> have enabled return-path and switchover-ack capability:
>>
>> Test result : The VFIO migration completed successfully
>>
>> [2] Test scenario : The source VM has enabled return-path and
>> switchover-ack capability while the target VM (in listening mode) not
>>
>> Test result : The VFIO migration fails
>>
>> The detailed error thrown by qemu-kvm when VFIO migration fails:
>> Target VM:
>> 0000:17:00.2: Received INIT_DATA_SENT but switchover ack
>> is not used
>> error while loading state section id
>> 81(0000:00:02.4:00.0/vfio)
>> load of migration failed: Invalid argument
>> Source VM:
>> failed to save SaveStateEntry with id(name): 2(ram): -5
>> Unable to write to socket: Connection reset by peer
>> Unable to write to socket: Connection reset by peer
>>
>> Tested-by: YangHang Liu <yanghliu@redhat.com>
>
> Some more info,
>
> Tests were performed with a mainline Linux and a mainline QEMU including
> this series - patch8.
>
> The amount of precopy data for a CX-7 VF is not very large. Any idea how
> to generate some more initial state with such devices ?
>
> I suppose pre-copy will be more important with vGPUs.
In CX-7 the precopy data is not expected to be very large, because it's
mainly used to pre-allocate resources in the destination.
However, precopy and switchover-ack are very important for CX-7, because
they allow doing the resource pre-allocation in the destination when the
source VM is running and reduce downtime significantly (see the example
I gave in the cover letter).
Thanks.
>
>
>
> YangHang,
>
> Could you please reply with a Tested-by on the cover letter, so that the
> whole series is tagged and not only patch 1.
>
> Thanks,
>
> C.
>
>>
>>
>> On Wed, May 31, 2023 at 1:46 AM Avihai Horon <avihaih@nvidia.com> wrote:
>>>
>>> Migration downtime estimation is calculated based on bandwidth and
>>> remaining migration data. This assumes that loading of migration
>>> data in
>>> the destination takes a negligible amount of time and that downtime
>>> depends only on network speed.
>>>
>>> While this may be true for RAM, it's not necessarily true for other
>>> migrated devices. For example, loading the data of a VFIO device in the
>>> destination might require from the device to allocate resources,
>>> prepare
>>> internal data structures and so on. These operations can take a
>>> significant amount of time which can increase migration downtime.
>>>
>>> This patch adds a new capability "switchover ack" that prevents the
>>> source from stopping the VM and completing the migration until an ACK
>>> is received from the destination that it's OK to do so.
>>>
>>> This can be used by migrated devices in various ways to reduce
>>> downtime.
>>> For example, a device can send initial precopy metadata to pre-allocate
>>> resources in the destination and use this capability to make sure that
>>> the pre-allocation is completed before the source VM is stopped, so it
>>> will have full effect.
>>>
>>> This new capability relies on the return path capability to communicate
>>> from the destination back to the source.
>>>
>>> The actual implementation of the capability will be added in the
>>> following patches.
>>>
>>> Signed-off-by: Avihai Horon <avihaih@nvidia.com>
>>> Reviewed-by: Peter Xu <peterx@redhat.com>
>>> Acked-by: Markus Armbruster <armbru@redhat.com>
>>> ---
>>> qapi/migration.json | 12 +++++++++++-
>>> migration/options.h | 1 +
>>> migration/options.c | 21 +++++++++++++++++++++
>>> 3 files changed, 33 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/qapi/migration.json b/qapi/migration.json
>>> index 179af0c4d8..061ea512e0 100644
>>> --- a/qapi/migration.json
>>> +++ b/qapi/migration.json
>>> @@ -487,6 +487,16 @@
>>> # and should not affect the correctness of postcopy migration.
>>> # (since 7.1)
>>> #
>>> +# @switchover-ack: If enabled, migration will not stop the source VM
>>> +# and complete the migration until an ACK is received from the
>>> +# destination that it's OK to do so. Exactly when this ACK is
>>> +# sent depends on the migrated devices that use this feature.
>>> +# For example, a device can use it to make sure some of its data
>>> +# is sent and loaded in the destination before doing switchover.
>>> +# This can reduce downtime if devices that support this capability
>>> +# are present. 'return-path' capability must be enabled to use
>>> +# it. (since 8.1)
>>> +#
>>> # Features:
>>> #
>>> # @unstable: Members @x-colo and @x-ignore-shared are experimental.
>>> @@ -502,7 +512,7 @@
>>> 'dirty-bitmaps', 'postcopy-blocktime',
>>> 'late-block-activate',
>>> { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] },
>>> 'validate-uuid', 'background-snapshot',
>>> - 'zero-copy-send', 'postcopy-preempt'] }
>>> + 'zero-copy-send', 'postcopy-preempt', 'switchover-ack'] }
>>>
>>> ##
>>> # @MigrationCapabilityStatus:
>>> diff --git a/migration/options.h b/migration/options.h
>>> index 45991af3c2..9aaf363322 100644
>>> --- a/migration/options.h
>>> +++ b/migration/options.h
>>> @@ -40,6 +40,7 @@ bool migrate_postcopy_ram(void);
>>> bool migrate_rdma_pin_all(void);
>>> bool migrate_release_ram(void);
>>> bool migrate_return_path(void);
>>> +bool migrate_switchover_ack(void);
>>> bool migrate_validate_uuid(void);
>>> bool migrate_xbzrle(void);
>>> bool migrate_zero_blocks(void);
>>> diff --git a/migration/options.c b/migration/options.c
>>> index b62ab30cd5..16007afca6 100644
>>> --- a/migration/options.c
>>> +++ b/migration/options.c
>>> @@ -185,6 +185,8 @@ Property migration_properties[] = {
>>> DEFINE_PROP_MIG_CAP("x-zero-copy-send",
>>> MIGRATION_CAPABILITY_ZERO_COPY_SEND),
>>> #endif
>>> + DEFINE_PROP_MIG_CAP("x-switchover-ack",
>>> + MIGRATION_CAPABILITY_SWITCHOVER_ACK),
>>>
>>> DEFINE_PROP_END_OF_LIST(),
>>> };
>>> @@ -308,6 +310,13 @@ bool migrate_return_path(void)
>>> return s->capabilities[MIGRATION_CAPABILITY_RETURN_PATH];
>>> }
>>>
>>> +bool migrate_switchover_ack(void)
>>> +{
>>> + MigrationState *s = migrate_get_current();
>>> +
>>> + return s->capabilities[MIGRATION_CAPABILITY_SWITCHOVER_ACK];
>>> +}
>>> +
>>> bool migrate_validate_uuid(void)
>>> {
>>> MigrationState *s = migrate_get_current();
>>> @@ -547,6 +556,18 @@ bool migrate_caps_check(bool *old_caps, bool
>>> *new_caps, Error **errp)
>>> }
>>> }
>>>
>>> + if (new_caps[MIGRATION_CAPABILITY_SWITCHOVER_ACK]) {
>>> + if (!new_caps[MIGRATION_CAPABILITY_RETURN_PATH]) {
>>> + error_setg(errp, "Capability 'switchover-ack' requires
>>> capability "
>>> + "'return-path'");
>>> + return false;
>>> + }
>>> +
>>> + /* Disable this capability until it's implemented */
>>> + error_setg(errp, "'switchover-ack' is not implemented yet");
>>> + return false;
>>> + }
>>> +
>>> return true;
>>> }
>>>
>>> --
>>> 2.26.3
>>>
>>>
>>
>
next prev parent reply other threads:[~2023-06-19 9:43 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-30 14:48 [PATCH v5 0/9] migration: Add switchover ack capability and VFIO precopy support Avihai Horon
2023-05-30 14:48 ` [PATCH v5 1/9] migration: Add switchover ack capability Avihai Horon
2023-06-15 12:38 ` YangHang Liu
2023-06-15 13:49 ` Cédric Le Goater
2023-06-19 9:37 ` Avihai Horon [this message]
2023-05-30 14:48 ` [PATCH v5 2/9] migration: Implement switchover ack logic Avihai Horon
2023-06-05 22:06 ` Alex Williamson
2023-06-06 12:12 ` Avihai Horon
2023-06-08 18:32 ` Alex Williamson
2023-06-11 7:45 ` Avihai Horon
2023-05-30 14:48 ` [PATCH v5 3/9] migration: Enable switchover ack capability Avihai Horon
2023-05-30 14:48 ` [PATCH v5 4/9] tests: Add migration switchover ack capability test Avihai Horon
2023-05-30 14:48 ` [PATCH v5 5/9] vfio/migration: Refactor vfio_save_block() to return saved data size Avihai Horon
2023-05-30 14:48 ` [PATCH v5 6/9] vfio/migration: Store VFIO migration flags in VFIOMigration Avihai Horon
2023-05-30 14:48 ` [PATCH v5 7/9] vfio/migration: Add VFIO migration pre-copy support Avihai Horon
2023-05-30 14:48 ` [PATCH v5 8/9] vfio/migration: Add x-allow-pre-copy VFIO device property Avihai Horon
2023-06-01 20:22 ` Alex Williamson
2023-06-04 9:33 ` Avihai Horon
2023-06-05 14:56 ` Alex Williamson
2023-06-06 11:59 ` Avihai Horon
2023-06-06 13:40 ` Cédric Le Goater
2023-06-07 7:41 ` Avihai Horon
2023-05-30 14:48 ` [PATCH v5 9/9] vfio/migration: Add support for switchover ack capability Avihai Horon
2023-05-30 15:15 ` Cédric Le Goater
2023-06-16 9:35 ` [PATCH v5 0/9] migration: Add switchover ack capability and VFIO precopy support YangHang Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4786a4ce-bb1b-ab19-f038-744a32924192@nvidia.com \
--to=avihaih@nvidia.com \
--cc=alex.williamson@redhat.com \
--cc=armbru@redhat.com \
--cc=clg@redhat.com \
--cc=eblake@redhat.com \
--cc=eduardo@habkost.net \
--cc=jgg@nvidia.com \
--cc=joao.m.martins@oracle.com \
--cc=kwankhede@nvidia.com \
--cc=leobras@redhat.com \
--cc=lvivier@redhat.com \
--cc=maorg@nvidia.com \
--cc=marcel.apfelbaum@gmail.com \
--cc=pbonzini@redhat.com \
--cc=peterx@redhat.com \
--cc=philmd@linaro.org \
--cc=qemu-devel@nongnu.org \
--cc=quintela@redhat.com \
--cc=targupta@nvidia.com \
--cc=thuth@redhat.com \
--cc=wangyanan55@huawei.com \
--cc=yanghliu@redhat.com \
--cc=yishaih@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).