* [PATCH 0/4] Multiple interface support on top of Multi-FD
@ 2022-06-09 7:33 Het Gala
2022-06-09 7:33 ` [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair Het Gala
` (4 more replies)
0 siblings, 5 replies; 34+ messages in thread
From: Het Gala @ 2022-06-09 7:33 UTC (permalink / raw)
To: qemu-devel
Cc: quintela, dgilbert, pbonzini, berrange, armbru, eblake, Het Gala
As of now, the multi-FD feature supports connection over the default network
only. This Patchset series is a Qemu side implementation of providing multiple
interfaces support for multi-FD. This enables us to fully utilize dedicated or
multiple NICs in case bonding of NICs is not possible.
Introduction
-------------
Multi-FD Qemu implementation currently supports connection only on the default
network. This forbids us from advantages like:
- Separating VM live migration traffic from the default network.
- Fully utilize all NICs’ capacity in cases where creating a LACP bond (Link
Aggregation Control Protocol) is not supported.
Multi-interface with Multi-FD
-----------------------------
Multiple-interface support over basic multi-FD has been implemented in the
patches. Advantages of this implementation are:
- Able to separate live migration traffic from default network interface by
creating multiFD channels on ip addresses of multiple non-default interfaces.
- Can optimize the number of multi-FD channels on a particular interface
depending upon the network bandwidth limit on a particular interface.
Implementation
--------------
Earlier the 'migrate' qmp command:
{ "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
Modified qmp command:
{ "execute": "migrate",
"arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
"source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
"multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
"destination-uri": "tcp:11.0.0.0:7789",
"multifd-channels": 5} ] } }
------------------------------------------------------------------------------
Earlier the 'migrate-incoming' qmp command:
{ "execute": "migrate-incoming", "arguments": { "uri": "tcp::4446" } }
Modified 'migrate-incoming' qmp command:
{ "execute": "migrate-incoming",
"arguments": {"uri": "tcp::6789",
"multi-fd-uri-list" : [ {"destination-uri" : "tcp::6900",
"multifd-channels": 4}, {"destination-uri" : "tcp:11.0.0.0:7789",
"multifd-channels": 5} ] } }
------------------------------------------------------------------------------
Introduced a new flag while spawning a qemu process for 'migrate-incoming' ip
addresses (-multi-fd-incoming flag):
-multi-fd-incoming "tcp::6900:4,tcp:11.0.0.0:7789:5"
---
Het Gala (4):
Modifying ‘migrate’ qmp command to add multi-FD socket on particular
source and destination pair
Adding multi-interface support for multi-FD on destination side
Establishing connection between any non-default source and destination
pair
Adding support for multi-FD connections dynamically
chardev/char-socket.c | 4 +-
include/io/channel-socket.h | 26 ++--
include/qapi/util.h | 10 ++
include/qemu/sockets.h | 6 +-
io/channel-socket.c | 50 +++++--
migration/migration.c | 211 ++++++++++++++++++++++------
migration/migration.h | 3 +-
migration/multifd.c | 42 +++---
migration/socket.c | 119 ++++++++++++----
migration/socket.h | 24 +++-
monitor/hmp-cmds.c | 68 ++++-----
nbd/client-connection.c | 2 +-
qapi/migration.json | 92 +++++++++---
qapi/qapi-util.c | 27 ++++
qemu-nbd.c | 4 +-
qemu-options.hx | 18 +++
scsi/pr-manager-helper.c | 1 +
softmmu/vl.c | 30 +++-
tests/unit/test-char.c | 8 +-
tests/unit/test-io-channel-socket.c | 4 +-
tests/unit/test-util-sockets.c | 16 +--
ui/input-barrier.c | 2 +-
ui/vnc.c | 3 +-
util/qemu-sockets.c | 71 +++++++---
24 files changed, 626 insertions(+), 215 deletions(-)
--
2.22.3
^ permalink raw reply [flat|nested] 34+ messages in thread
* [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair
2022-06-09 7:33 [PATCH 0/4] Multiple interface support on top of Multi-FD Het Gala
@ 2022-06-09 7:33 ` Het Gala
2022-06-16 17:26 ` Dr. David Alan Gilbert
2022-07-18 8:35 ` Markus Armbruster
2022-06-09 7:33 ` [PATCH 2/4] Adding multi-interface support for multi-FD on destination side Het Gala
` (3 subsequent siblings)
4 siblings, 2 replies; 34+ messages in thread
From: Het Gala @ 2022-06-09 7:33 UTC (permalink / raw)
To: qemu-devel
Cc: quintela, dgilbert, pbonzini, berrange, armbru, eblake, Het Gala,
Manish Mishra
i) Modified the format of the qemu monitor command : 'migrate' by adding a list,
each element in the list consists of multi-FD connection parameters: source
and destination uris and of the number of multi-fd channels between each pair.
ii) Information of all multi-FD connection parameters’ list, length of the list
and total number of multi-fd channels for all the connections together is
stored in ‘OutgoingArgs’ struct.
Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
Signed-off-by: Het Gala <het.gala@nutanix.com>
---
include/qapi/util.h | 9 ++++++++
migration/migration.c | 47 ++++++++++++++++++++++++++++++--------
migration/socket.c | 53 ++++++++++++++++++++++++++++++++++++++++---
migration/socket.h | 17 +++++++++++++-
monitor/hmp-cmds.c | 22 ++++++++++++++++--
qapi/migration.json | 43 +++++++++++++++++++++++++++++++----
6 files changed, 170 insertions(+), 21 deletions(-)
diff --git a/include/qapi/util.h b/include/qapi/util.h
index 81a2b13a33..3041feb3d9 100644
--- a/include/qapi/util.h
+++ b/include/qapi/util.h
@@ -56,4 +56,13 @@ int parse_qapi_name(const char *name, bool complete);
(tail) = &(*(tail))->next; \
} while (0)
+#define QAPI_LIST_LENGTH(list) ({ \
+ int _len = 0; \
+ typeof(list) _elem; \
+ for (_elem = list; _elem != NULL; _elem = _elem->next) { \
+ _len++; \
+ } \
+ _len; \
+})
+
#endif
diff --git a/migration/migration.c b/migration/migration.c
index 31739b2af9..c408175aeb 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -2328,13 +2328,14 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,
return true;
}
-void qmp_migrate(const char *uri, bool has_blk, bool blk,
+void qmp_migrate(const char *uri, bool has_multi_fd_uri_list,
+ MigrateUriParameterList *cap, bool has_blk, bool blk,
bool has_inc, bool inc, bool has_detach, bool detach,
bool has_resume, bool resume, Error **errp)
{
Error *local_err = NULL;
MigrationState *s = migrate_get_current();
- const char *p = NULL;
+ const char *dst_ptr = NULL;
if (!migrate_prepare(s, has_blk && blk, has_inc && inc,
has_resume && resume, errp)) {
@@ -2348,20 +2349,46 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
}
}
+ /*
+ * In case of Multi-FD migration parameters, if uri is provided,
+ * supports only tcp network protocol.
+ */
+ if (has_multi_fd_uri_list) {
+ int length = QAPI_LIST_LENGTH(cap);
+ init_multifd_array(length);
+ for (int i = 0; i < length; i++) {
+ const char *p1 = NULL, *p2 = NULL;
+ const char *multifd_dst_uri = cap->value->destination_uri;
+ const char *multifd_src_uri = cap->value->source_uri;
+ uint8_t multifd_channels = cap->value->multifd_channels;
+ if (!strstart(multifd_dst_uri, "tcp:", &p1) ||
+ !strstart(multifd_src_uri, "tcp:", &p2)) {
+ error_setg(errp, "multi-fd destination and multi-fd source "
+ "uri, both should be present and follows tcp protocol only");
+ break;
+ } else {
+ store_multifd_migration_params(p1 ? p1 : multifd_dst_uri,
+ p2 ? p2 : multifd_src_uri,
+ multifd_channels, i, &local_err);
+ }
+ cap = cap->next;
+ }
+ }
+
migrate_protocol_allow_multi_channels(false);
- if (strstart(uri, "tcp:", &p) ||
+ if (strstart(uri, "tcp:", &dst_ptr) ||
strstart(uri, "unix:", NULL) ||
strstart(uri, "vsock:", NULL)) {
migrate_protocol_allow_multi_channels(true);
- socket_start_outgoing_migration(s, p ? p : uri, &local_err);
+ socket_start_outgoing_migration(s, dst_ptr ? dst_ptr : uri, &local_err);
#ifdef CONFIG_RDMA
- } else if (strstart(uri, "rdma:", &p)) {
- rdma_start_outgoing_migration(s, p, &local_err);
+ } else if (strstart(uri, "rdma:", &dst_ptr)) {
+ rdma_start_outgoing_migration(s, dst_ptr, &local_err);
#endif
- } else if (strstart(uri, "exec:", &p)) {
- exec_start_outgoing_migration(s, p, &local_err);
- } else if (strstart(uri, "fd:", &p)) {
- fd_start_outgoing_migration(s, p, &local_err);
+ } else if (strstart(uri, "exec:", &dst_ptr)) {
+ exec_start_outgoing_migration(s, dst_ptr, &local_err);
+ } else if (strstart(uri, "fd:", &dst_ptr)) {
+ fd_start_outgoing_migration(s, dst_ptr, &local_err);
} else {
if (!(has_resume && resume)) {
yank_unregister_instance(MIGRATION_YANK_INSTANCE);
diff --git a/migration/socket.c b/migration/socket.c
index 4fd5e85f50..7ca6af8cca 100644
--- a/migration/socket.c
+++ b/migration/socket.c
@@ -32,6 +32,17 @@ struct SocketOutgoingArgs {
SocketAddress *saddr;
} outgoing_args;
+struct SocketArgs {
+ struct SrcDestAddr data;
+ uint8_t multifd_channels;
+};
+
+struct OutgoingMigrateParams {
+ struct SocketArgs *socket_args;
+ size_t length;
+ uint64_t total_multifd_channel;
+} outgoing_migrate_params;
+
void socket_send_channel_create(QIOTaskFunc f, void *data)
{
QIOChannelSocket *sioc = qio_channel_socket_new();
@@ -47,6 +58,14 @@ int socket_send_channel_destroy(QIOChannel *send)
qapi_free_SocketAddress(outgoing_args.saddr);
outgoing_args.saddr = NULL;
}
+
+ if (outgoing_migrate_params.socket_args != NULL) {
+ g_free(outgoing_migrate_params.socket_args);
+ outgoing_migrate_params.socket_args = NULL;
+ }
+ if (outgoing_migrate_params.length) {
+ outgoing_migrate_params.length = 0;
+ }
return 0;
}
@@ -117,13 +136,41 @@ socket_start_outgoing_migration_internal(MigrationState *s,
}
void socket_start_outgoing_migration(MigrationState *s,
- const char *str,
+ const char *dst_str,
Error **errp)
{
Error *err = NULL;
- SocketAddress *saddr = socket_parse(str, &err);
+ SocketAddress *dst_saddr = socket_parse(dst_str, &err);
+ if (!err) {
+ socket_start_outgoing_migration_internal(s, dst_saddr, &err);
+ }
+ error_propagate(errp, err);
+}
+
+void init_multifd_array(int length)
+{
+ outgoing_migrate_params.socket_args = g_new0(struct SocketArgs, length);
+ outgoing_migrate_params.length = length;
+ outgoing_migrate_params.total_multifd_channel = 0;
+}
+
+void store_multifd_migration_params(const char *dst_uri,
+ const char *src_uri,
+ uint8_t multifd_channels,
+ int idx, Error **errp)
+{
+ Error *err = NULL;
+ SocketAddress *src_addr = NULL;
+ SocketAddress *dst_addr = socket_parse(dst_uri, &err);
+ if (src_uri) {
+ src_addr = socket_parse(src_uri, &err);
+ }
if (!err) {
- socket_start_outgoing_migration_internal(s, saddr, &err);
+ outgoing_migrate_params.socket_args[idx].data.dst_addr = dst_addr;
+ outgoing_migrate_params.socket_args[idx].data.src_addr = src_addr;
+ outgoing_migrate_params.socket_args[idx].multifd_channels
+ = multifd_channels;
+ outgoing_migrate_params.total_multifd_channel += multifd_channels;
}
error_propagate(errp, err);
}
diff --git a/migration/socket.h b/migration/socket.h
index 891dbccceb..bba7f177fe 100644
--- a/migration/socket.h
+++ b/migration/socket.h
@@ -19,12 +19,27 @@
#include "io/channel.h"
#include "io/task.h"
+#include "migration.h"
+
+/* info regarding destination and source uri */
+struct SrcDestAddr {
+ SocketAddress *dst_addr;
+ SocketAddress *src_addr;
+};
void socket_send_channel_create(QIOTaskFunc f, void *data);
int socket_send_channel_destroy(QIOChannel *send);
void socket_start_incoming_migration(const char *str, Error **errp);
-void socket_start_outgoing_migration(MigrationState *s, const char *str,
+void socket_start_outgoing_migration(MigrationState *s, const char *dst_str,
Error **errp);
+
+int multifd_list_length(MigrateUriParameterList *list);
+
+void init_multifd_array(int length);
+
+void store_multifd_migration_params(const char *dst_uri, const char *src_uri,
+ uint8_t multifd_channels, int idx,
+ Error **erp);
#endif
diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
index 622c783c32..2db539016a 100644
--- a/monitor/hmp-cmds.c
+++ b/monitor/hmp-cmds.c
@@ -56,6 +56,9 @@
#include "migration/snapshot.h"
#include "migration/misc.h"
+/* Default number of multi-fd channels */
+#define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2
+
#ifdef CONFIG_SPICE
#include <spice/enums.h>
#endif
@@ -1574,10 +1577,25 @@ void hmp_migrate(Monitor *mon, const QDict *qdict)
bool inc = qdict_get_try_bool(qdict, "inc", false);
bool resume = qdict_get_try_bool(qdict, "resume", false);
const char *uri = qdict_get_str(qdict, "uri");
+
+ const char *src_uri = qdict_get_str(qdict, "source-uri");
+ const char *dst_uri = qdict_get_str(qdict, "destination-uri");
+ uint8_t multifd_channels = qdict_get_try_int(qdict, "multifd-channels",
+ DEFAULT_MIGRATE_MULTIFD_CHANNELS);
Error *err = NULL;
+ MigrateUriParameterList *caps = NULL;
+ MigrateUriParameter *value;
+
+ value = g_malloc0(sizeof(*value));
+ value->source_uri = (char *)src_uri;
+ value->destination_uri = (char *)dst_uri;
+ value->multifd_channels = multifd_channels;
+ QAPI_LIST_PREPEND(caps, value);
+
+ qmp_migrate(uri, !!caps, caps, !!blk, blk, !!inc,
+ inc, false, false, true, resume, &err);
+ qapi_free_MigrateUriParameterList(caps);
- qmp_migrate(uri, !!blk, blk, !!inc, inc,
- false, false, true, resume, &err);
if (hmp_handle_error(mon, err)) {
return;
}
diff --git a/qapi/migration.json b/qapi/migration.json
index 6130cd9fae..fb259d626b 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -1454,12 +1454,38 @@
##
{ 'command': 'migrate-continue', 'data': {'state': 'MigrationStatus'} }
+##
+# @MigrateUriParameter:
+#
+# Information regarding which source interface is connected to which
+# destination interface and number of multifd channels over each interface.
+#
+# @source-uri: the Uniform Resource Identifier of the source VM.
+# Default port number is 0.
+#
+# @destination-uri: the Uniform Resource Identifier of the destination VM
+#
+# @multifd-channels: number of parallel multifd channels used to migrate data
+# for specific source-uri and destination-uri. Default value
+# in this case is 2 (Since 4.0)
+#
+##
+{ 'struct' : 'MigrateUriParameter',
+ 'data' : { 'source-uri' : 'str',
+ 'destination-uri' : 'str',
+ '*multifd-channels' : 'uint8'} }
+
##
# @migrate:
#
# Migrates the current running guest to another Virtual Machine.
#
# @uri: the Uniform Resource Identifier of the destination VM
+# for migration thread
+#
+# @multi-fd-uri-list: list of pair of source and destination VM Uniform
+# Resource Identifiers with number of multifd-channels
+# for each pair
#
# @blk: do block migration (full disk copy)
#
@@ -1479,20 +1505,27 @@
# 1. The 'query-migrate' command should be used to check migration's progress
# and final result (this information is provided by the 'status' member)
#
-# 2. All boolean arguments default to false
+# 2. The uri argument should have the Uniform Resource Identifier of default
+# destination VM. This connection will be bound to default network
+#
+# 3. All boolean arguments default to false
#
-# 3. The user Monitor's "detach" argument is invalid in QMP and should not
+# 4. The user Monitor's "detach" argument is invalid in QMP and should not
# be used
#
# Example:
#
-# -> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
+# -> { "execute": "migrate",
+# "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
+# "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
+# "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
+# "destination-uri": "tcp:11.0.0.0:7789", "multifd-channels": 5} ] } }
# <- { "return": {} }
#
##
{ 'command': 'migrate',
- 'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool',
- '*detach': 'bool', '*resume': 'bool' } }
+ 'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateUriParameter'], '*blk': 'bool',
+ '*inc': 'bool', '*detach': 'bool', '*resume': 'bool' } }
##
# @migrate-incoming:
--
2.22.3
^ permalink raw reply related [flat|nested] 34+ messages in thread
* [PATCH 2/4] Adding multi-interface support for multi-FD on destination side
2022-06-09 7:33 [PATCH 0/4] Multiple interface support on top of Multi-FD Het Gala
2022-06-09 7:33 ` [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair Het Gala
@ 2022-06-09 7:33 ` Het Gala
2022-06-16 18:40 ` Dr. David Alan Gilbert
2022-06-09 7:33 ` [PATCH 3/4] Establishing connection between any non-default source and destination pair Het Gala
` (2 subsequent siblings)
4 siblings, 1 reply; 34+ messages in thread
From: Het Gala @ 2022-06-09 7:33 UTC (permalink / raw)
To: qemu-devel
Cc: quintela, dgilbert, pbonzini, berrange, armbru, eblake, Het Gala,
Manish Mishra
i) Modified the format of qemu monitor command: ‘migrate-incoming’ by adding
a list, each element in the list is to open listeners with a given number
of multiFD channels.
ii) Qemu starts with -incoming flag defer and -multi-fd-incoming defer to
allow the modified 'migrate-incoming' command to be used.
iii) Format for -multi-fd-incoming flag as a comma separated string has been
added with each substring containing listener socket address and number
of sockets to open.
Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
Signed-off-by: Het Gala <het.gala@nutanix.com>
---
include/qapi/util.h | 1 +
migration/migration.c | 149 ++++++++++++++++++++++++++++++++++++------
migration/migration.h | 2 +
migration/socket.c | 11 ++--
migration/socket.h | 3 +-
monitor/hmp-cmds.c | 42 ++++--------
qapi/migration.json | 43 ++++++++++--
qapi/qapi-util.c | 27 ++++++++
qemu-options.hx | 18 +++++
softmmu/vl.c | 30 ++++++++-
10 files changed, 265 insertions(+), 61 deletions(-)
diff --git a/include/qapi/util.h b/include/qapi/util.h
index 3041feb3d9..88fb2270db 100644
--- a/include/qapi/util.h
+++ b/include/qapi/util.h
@@ -29,6 +29,7 @@ bool qapi_bool_parse(const char *name, const char *value, bool *obj,
Error **errp);
int parse_qapi_name(const char *name, bool complete);
+struct strList *strList_from_string(const char *in, char c);
/*
* For any GenericList @list, insert @element at the front.
diff --git a/migration/migration.c b/migration/migration.c
index c408175aeb..9b0ad732e7 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -477,28 +477,39 @@ void migrate_add_address(SocketAddress *address)
QAPI_CLONE(SocketAddress, address));
}
-static void qemu_start_incoming_migration(const char *uri, Error **errp)
+static void qemu_start_incoming_migration(const char *uri, uint8_t number,
+ int idx, Error **errp)
{
const char *p = NULL;
- migrate_protocol_allow_multi_channels(false); /* reset it anyway */
- qapi_event_send_migration(MIGRATION_STATUS_SETUP);
- if (strstart(uri, "tcp:", &p) ||
- strstart(uri, "unix:", NULL) ||
- strstart(uri, "vsock:", NULL)) {
- migrate_protocol_allow_multi_channels(true);
- socket_start_incoming_migration(p ? p : uri, errp);
-#ifdef CONFIG_RDMA
- } else if (strstart(uri, "rdma:", &p)) {
- rdma_start_incoming_migration(p, errp);
-#endif
- } else if (strstart(uri, "exec:", &p)) {
- exec_start_incoming_migration(p, errp);
- } else if (strstart(uri, "fd:", &p)) {
- fd_start_incoming_migration(p, errp);
+ if (number == 0) {
+ migrate_protocol_allow_multi_channels(false); /* reset it anyway */
+ qapi_event_send_migration(MIGRATION_STATUS_SETUP);
+ if (strstart(uri, "tcp:", &p) ||
+ strstart(uri, "unix:", NULL) ||
+ strstart(uri, "vsock:", NULL)) {
+ migrate_protocol_allow_multi_channels(true);
+ #ifdef CONFIG_RDMA
+ } else if (strstart(uri, "rdma:", &p)) {
+ rdma_start_incoming_migration(p, errp);
+ #endif
+ } else if (strstart(uri, "exec:", &p)) {
+ exec_start_incoming_migration(p, errp);
+ } else if (strstart(uri, "fd:", &p)) {
+ fd_start_incoming_migration(p, errp);
+ } else {
+ error_setg(errp, "unknown migration protocol: %s", uri);
+ }
} else {
- error_setg(errp, "unknown migration protocol: %s", uri);
+ /* multi-FD parameters only support tcp network protocols */
+ if (!strstart(uri, "tcp:", &p)) {
+ error_setg(errp, "multifd-destination uri supports "
+ "tcp protocol only");
+ return;
+ }
+ store_multifd_migration_params(p ? p : uri, NULL, number, idx, errp);
}
+ socket_start_incoming_migration(p ? p : uri, number, errp);
}
static void process_incoming_migration_bh(void *opaque)
@@ -2140,7 +2151,17 @@ void migrate_del_blocker(Error *reason)
migration_blockers = g_slist_remove(migration_blockers, reason);
}
-void qmp_migrate_incoming(const char *uri, Error **errp)
+static inline int incoming_multi_fd_uri_parse(const char *str, char delim)
+{
+ int count = 0;
+ for (int i = 0; i < strlen(str); i++) {
+ count += (str[i] == delim);
+ }
+ return count;
+}
+
+/* migrate_incoming comes from -incoming flag in qemu process */
+void migrate_incoming(const char *uri, Error **errp)
{
Error *local_err = NULL;
static bool once = true;
@@ -2154,11 +2175,99 @@ void qmp_migrate_incoming(const char *uri, Error **errp)
return;
}
+ qemu_start_incoming_migration(uri, 0, 0, &local_err);
+
if (!yank_register_instance(MIGRATION_YANK_INSTANCE, errp)) {
return;
}
- qemu_start_incoming_migration(uri, &local_err);
+ if (local_err) {
+ yank_unregister_instance(MIGRATION_YANK_INSTANCE);
+ error_propagate(errp, local_err);
+ return;
+ }
+
+ once = false;
+}
+
+/* multi_fd_migrate_incoming comes from -multi-fd-migrate-incoming flag in qemu process */
+void multi_fd_migrate_incoming(const char *uri, Error **errp)
+{
+ Error *local_err = NULL;
+ static bool once = true;
+
+ if (!once) {
+ error_setg(errp, "The incoming migration has already been started");
+ return;
+ }
+ if (!runstate_check(RUN_STATE_INMIGRATE)) {
+ error_setg(errp, "'-multi-fd-incoming' was not specified on the command line");
+ return;
+ }
+
+ strList *st = strList_from_string(uri, ',');
+ strList *r = st;
+ int length = QAPI_LIST_LENGTH(st);
+ init_multifd_array(length);
+
+ for (int i = 0; i < length; i++) {
+ const char *uri = NULL, *ret = NULL;
+ const char *str = r->value;
+ uint8_t multifd_channels = DEFAULT_MIGRATE_MULTIFD_CHANNELS;
+ int parse_count = incoming_multi_fd_uri_parse(str, ':');
+ if (parse_count < 2 || parse_count > 3) {
+ error_setg(errp, "Invalid format of string-id %d in "
+ "'-multi-fd-incoming' flag", i);
+ return;
+ }
+ if (parse_count == 3) {
+ ret = strrchr(str, ':');
+ uri = g_strndup(str, strlen(str) - strlen(ret));
+ multifd_channels = atoi(ret + 1);
+ }
+ qemu_start_incoming_migration(parse_count == 2 ? str : uri,
+ multifd_channels, i, &local_err);
+ r = r->next;
+ }
+
+ if (local_err) {
+ yank_unregister_instance(MIGRATION_YANK_INSTANCE);
+ error_propagate(errp, local_err);
+ return;
+ }
+
+ once = false;
+}
+
+/* qmp_migrate_incoming comes from qemu qmp monitor command */
+void qmp_migrate_incoming(const char *uri, bool has_multi_fd_uri_list,
+ MigrateIncomingUriList *cap, Error **errp)
+{
+ Error *local_err = NULL;
+ static bool once = true;
+
+ if (!once) {
+ error_setg(errp, "The incoming migration has already been started");
+ return;
+ }
+
+ if (!yank_register_instance(MIGRATION_YANK_INSTANCE, errp)) {
+ return;
+ }
+
+ /* For migration thread */
+ qemu_start_incoming_migration(uri, 0, 0, &local_err);
+
+ /* For Multi-FD */
+ int length = QAPI_LIST_LENGTH(cap);
+ init_multifd_array(length);
+ for (int i = 0; i < length; i++) {
+ const char *multifd_dst_uri = cap->value->destination_uri;
+ uint8_t multifd_channels = cap->value->multifd_channels;
+ qemu_start_incoming_migration(multifd_dst_uri, multifd_channels,
+ i, &local_err);
+ cap = cap->next;
+ }
if (local_err) {
yank_unregister_instance(MIGRATION_YANK_INSTANCE);
@@ -2194,7 +2303,7 @@ void qmp_migrate_recover(const char *uri, Error **errp)
* only re-setup the migration stream and poke existing migration
* to continue using that newly established channel.
*/
- qemu_start_incoming_migration(uri, errp);
+ qemu_start_incoming_migration(uri, 0, 0, errp);
}
void qmp_migrate_pause(Error **errp)
diff --git a/migration/migration.h b/migration/migration.h
index 485d58b95f..fa8717ec9e 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -353,6 +353,8 @@ bool migration_is_setup_or_active(int state);
bool migration_is_running(int state);
void migrate_init(MigrationState *s);
+void migrate_incoming(const char *uri, Error **errp);
+void multi_fd_migrate_incoming(const char *uri_str, Error **errp);
bool migration_is_blocked(Error **errp);
/* True if outgoing migration has entered postcopy phase */
bool migration_in_postcopy(void);
diff --git a/migration/socket.c b/migration/socket.c
index 7ca6af8cca..21e0983df2 100644
--- a/migration/socket.c
+++ b/migration/socket.c
@@ -202,17 +202,17 @@ socket_incoming_migration_end(void *opaque)
static void
socket_start_incoming_migration_internal(SocketAddress *saddr,
- Error **errp)
+ uint8_t number, Error **errp)
{
QIONetListener *listener = qio_net_listener_new();
MigrationIncomingState *mis = migration_incoming_get_current();
size_t i;
- int num = 1;
+ uint8_t num = 1;
qio_net_listener_set_name(listener, "migration-socket-listener");
if (migrate_use_multifd()) {
- num = migrate_multifd_channels();
+ num = number;
}
if (qio_net_listener_open_sync(listener, saddr, num, errp) < 0) {
@@ -239,12 +239,13 @@ socket_start_incoming_migration_internal(SocketAddress *saddr,
}
}
-void socket_start_incoming_migration(const char *str, Error **errp)
+void socket_start_incoming_migration(const char *str,
+ uint8_t number, Error **errp)
{
Error *err = NULL;
SocketAddress *saddr = socket_parse(str, &err);
if (!err) {
- socket_start_incoming_migration_internal(saddr, &err);
+ socket_start_incoming_migration_internal(saddr, number, &err);
}
qapi_free_SocketAddress(saddr);
error_propagate(errp, err);
diff --git a/migration/socket.h b/migration/socket.h
index bba7f177fe..b9e3699167 100644
--- a/migration/socket.h
+++ b/migration/socket.h
@@ -30,7 +30,8 @@ struct SrcDestAddr {
void socket_send_channel_create(QIOTaskFunc f, void *data);
int socket_send_channel_destroy(QIOChannel *send);
-void socket_start_incoming_migration(const char *str, Error **errp);
+void socket_start_incoming_migration(const char *str, uint8_t number,
+ Error **errp);
void socket_start_outgoing_migration(MigrationState *s, const char *dst_str,
Error **errp);
diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
index 2db539016a..32a6b67d5f 100644
--- a/monitor/hmp-cmds.c
+++ b/monitor/hmp-cmds.c
@@ -42,6 +42,7 @@
#include "qapi/qapi-commands-run-state.h"
#include "qapi/qapi-commands-tpm.h"
#include "qapi/qapi-commands-ui.h"
+#include "qapi/util.h"
#include "qapi/qapi-visit-net.h"
#include "qapi/qapi-visit-migration.h"
#include "qapi/qmp/qdict.h"
@@ -72,32 +73,6 @@ bool hmp_handle_error(Monitor *mon, Error *err)
return false;
}
-/*
- * Produce a strList from a comma separated list.
- * A NULL or empty input string return NULL.
- */
-static strList *strList_from_comma_list(const char *in)
-{
- strList *res = NULL;
- strList **tail = &res;
-
- while (in && in[0]) {
- char *comma = strchr(in, ',');
- char *value;
-
- if (comma) {
- value = g_strndup(in, comma - in);
- in = comma + 1; /* skip the , */
- } else {
- value = g_strdup(in);
- in = NULL;
- }
- QAPI_LIST_APPEND(tail, value);
- }
-
- return res;
-}
-
void hmp_info_name(Monitor *mon, const QDict *qdict)
{
NameInfo *info;
@@ -1117,7 +1092,7 @@ void hmp_announce_self(Monitor *mon, const QDict *qdict)
migrate_announce_params());
qapi_free_strList(params->interfaces);
- params->interfaces = strList_from_comma_list(interfaces_str);
+ params->interfaces = strList_from_string(interfaces_str, ',');
params->has_interfaces = params->interfaces != NULL;
params->id = g_strdup(id);
params->has_id = !!params->id;
@@ -1147,8 +1122,19 @@ void hmp_migrate_incoming(Monitor *mon, const QDict *qdict)
{
Error *err = NULL;
const char *uri = qdict_get_str(qdict, "uri");
+ const char *dst_uri = qdict_get_str(qdict, "destination-uri");
+ uint8_t multifd_channels = qdict_get_try_int(qdict, "multifd-channels",
+ DEFAULT_MIGRATE_MULTIFD_CHANNELS);
+ MigrateIncomingUriList *caps = NULL;
+ MigrateIncomingUri *value;
+
+ value = g_malloc0(sizeof(*value));
+ value->destination_uri = (char *)dst_uri;
+ value->multifd_channels = multifd_channels;
+ QAPI_LIST_PREPEND(caps, value);
- qmp_migrate_incoming(uri, &err);
+ qmp_migrate_incoming(uri, !!caps, caps, &err);
+ qapi_free_MigrateIncomingUriList(caps);
hmp_handle_error(mon, err);
}
diff --git a/qapi/migration.json b/qapi/migration.json
index fb259d626b..62a7b22d19 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -1527,15 +1527,37 @@
'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateUriParameter'], '*blk': 'bool',
'*inc': 'bool', '*detach': 'bool', '*resume': 'bool' } }
+##
+# @MigrateIncomingUri:
+#
+# Information regarding which destination listening interface to be connected
+# and number of multifd channels over that interface.
+#
+# @destination-uri: the Uniform Resource Identifier of the destination VM
+#
+# @multifd-channels: number of channels used to migrate data in parallel for
+# for specific source-uri and destination-uri.
+# Default value in this case is 2 (Since 4.0)
+#
+##
+{ 'struct' : 'MigrateIncomingUri',
+ 'data' : { 'destination-uri' : 'str',
+ '*multifd-channels' : 'uint8'} }
+
##
# @migrate-incoming:
#
# Start an incoming migration, the qemu must have been started
-# with -incoming defer
+# with -incoming defer. qemu can also be started with optional
+# -multi-fd-incoming defer for opening multifd listening sockets
#
# @uri: The Uniform Resource Identifier identifying the source or
# address to listen on
#
+# @multi-fd-uri-list: list of pair of source and destination VM Uniform
+# Resource Identifiers with number of multifd-channels
+# for each pair
+#
# Returns: nothing on success
#
# Since: 2.3
@@ -1546,19 +1568,30 @@
# compatible with -incoming and the format of the uri is already exposed
# above libvirt.
#
-# 2. QEMU must be started with -incoming defer to allow migrate-incoming to
+# 2. multi-fd-uri-list will have list of destination uri as listening sockets
+# and multi-fd number of channels on each listening socket.
+#
+# 3. QEMU must be started with -incoming defer to allow migrate-incoming to
# be used.
#
-# 3. The uri format is the same as for -incoming
+# 4. multi-fd-uri-list format is not the same as for -multi-fd-incoming flag.
+# For -multi-fd-incoming flag, it is a comma separated list of listener
+# sockets with multifd channels.
+# Example: -multi-fd-incoming "tcp::6900:4,tcp:11.0.0.0:7789:5".
#
# Example:
#
# -> { "execute": "migrate-incoming",
-# "arguments": { "uri": "tcp::4446" } }
+# "arguments": {"uri": "tcp::6789",
+# "multi-fd-uri-list" : [ {"destination-uri" : "tcp::6900",
+# "multifd-channels": 4},
+# {"destination-uri" : "tcp:11.0.0.0:7789",
+# "multifd-channels": 5} ] } }
# <- { "return": {} }
#
##
-{ 'command': 'migrate-incoming', 'data': {'uri': 'str' } }
+{ 'command': 'migrate-incoming',
+ 'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateIncomingUri'] } }
##
# @xen-save-devices-state:
diff --git a/qapi/qapi-util.c b/qapi/qapi-util.c
index 63596e11c5..9cc43ebcd3 100644
--- a/qapi/qapi-util.c
+++ b/qapi/qapi-util.c
@@ -15,6 +15,7 @@
#include "qapi/error.h"
#include "qemu/ctype.h"
#include "qapi/qmp/qerror.h"
+#include "qapi/qapi-builtin-types.h"
CompatPolicy compat_policy;
@@ -152,3 +153,29 @@ int parse_qapi_name(const char *str, bool complete)
}
return p - str;
}
+
+/*
+ * Produce a strList from a delimiter separated list.
+ * A NULL or empty input string return NULL.
+ */
+strList *strList_from_string(const char *in, char c)
+{
+ strList *res = NULL;
+ strList **tail = &res;
+
+ while (in && in[0]) {
+ char *ch = strchr(in, c);
+ char *value;
+
+ if (ch) {
+ value = g_strndup(in, ch - in);
+ in = ch + 1; /* skip the , */
+ } else {
+ value = g_strdup(in);
+ in = NULL;
+ }
+ QAPI_LIST_APPEND(tail, value);
+ }
+
+ return res;
+}
diff --git a/qemu-options.hx b/qemu-options.hx
index 60cf188da4..2e82e41dd5 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -4480,6 +4480,24 @@ SRST
to issuing the migrate\_incoming to allow the migration to begin.
ERST
+DEF("multi-fd-incoming", HAS_ARG, QEMU_OPTION_multi_fd_incoming, \
+ "-multi-fd-incoming tcp:[host]:port[:channel][,to=maxport][,ipv4=on|off][,ipv6=on|off]\n" \
+ "-multi-fd-incoming defer\n" \
+ " wait for the URI to be specified via\n" \
+ " multi_fd_migrate_incoming\n",
+ QEMU_ARCH_ALL)
+SRST
+``-multi-fd-incoming tcp:[host]:port[:channel][,to=maxport][,ipv4=on|off][,ipv6=on|off]``
+ Prepare for multi-fd incoming migration, with multi-fd listening sockets
+ on that connection. Default number of multi-fd channels is 2.
+
+``-multi-fd-incoming defer``
+ Wait for the URI to be specified via multi_fd_migrate\_incoming. The
+ monitor can be used to change settings (such as migration parameters)
+ prior to issuing the multi_fd_migrate\_incoming to allow the migration
+ to begin.
+ERST
+
DEF("only-migratable", 0, QEMU_OPTION_only_migratable, \
"-only-migratable allow only migratable devices\n", QEMU_ARCH_ALL)
SRST
diff --git a/softmmu/vl.c b/softmmu/vl.c
index 4c1e94b00e..2f5cf18eff 100644
--- a/softmmu/vl.c
+++ b/softmmu/vl.c
@@ -45,7 +45,7 @@
#include "sysemu/seccomp.h"
#include "sysemu/tcg.h"
#include "sysemu/xen.h"
-
+#include "migration/migration.h"
#include "qemu/error-report.h"
#include "qemu/sockets.h"
#include "qemu/accel.h"
@@ -167,6 +167,7 @@ typedef struct DeviceOption {
static const char *cpu_option;
static const char *mem_path;
static const char *incoming;
+static const char *multi_fd_incoming;
static const char *loadvm;
static const char *accelerators;
static bool have_custom_ram_size;
@@ -2349,6 +2350,11 @@ static void qemu_validate_options(const QDict *machine_opts)
error_report("'preconfig' supports '-incoming defer' only");
exit(EXIT_FAILURE);
}
+ if (multi_fd_incoming && preconfig_requested &&
+ strcmp(multi_fd_incoming, "defer") != 0) {
+ error_report("'preconfig' supports '-multi-fd-incoming defer' only");
+ exit(EXIT_FAILURE);
+ }
#ifdef CONFIG_CURSES
if (is_daemonized() && dpy.type == DISPLAY_TYPE_CURSES) {
@@ -2621,7 +2627,7 @@ void qmp_x_exit_preconfig(Error **errp)
if (incoming) {
Error *local_err = NULL;
if (strcmp(incoming, "defer") != 0) {
- qmp_migrate_incoming(incoming, &local_err);
+ migrate_incoming(incoming, &local_err);
if (local_err) {
error_reportf_err(local_err, "-incoming %s: ", incoming);
exit(1);
@@ -2630,6 +2636,20 @@ void qmp_x_exit_preconfig(Error **errp)
} else if (autostart) {
qmp_cont(NULL);
}
+
+ if (multi_fd_incoming) {
+ Error *local_err = NULL;
+ if (strcmp(multi_fd_incoming, "defer") != 0) {
+ multi_fd_migrate_incoming(multi_fd_incoming, &local_err);
+ if (local_err) {
+ error_reportf_err(local_err, "-multi-fd-incoming %s: ",
+ multi_fd_incoming);
+ exit(1);
+ }
+ }
+ } else if (autostart) {
+ qmp_cont(NULL);
+ }
}
void qemu_init(int argc, char **argv, char **envp)
@@ -3355,6 +3375,12 @@ void qemu_init(int argc, char **argv, char **envp)
}
incoming = optarg;
break;
+ case QEMU_OPTION_multi_fd_incoming:
+ if (!multi_fd_incoming) {
+ runstate_set(RUN_STATE_INMIGRATE);
+ }
+ multi_fd_incoming = optarg;
+ break;
case QEMU_OPTION_only_migratable:
only_migratable = 1;
break;
--
2.22.3
^ permalink raw reply related [flat|nested] 34+ messages in thread
* [PATCH 3/4] Establishing connection between any non-default source and destination pair
2022-06-09 7:33 [PATCH 0/4] Multiple interface support on top of Multi-FD Het Gala
2022-06-09 7:33 ` [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair Het Gala
2022-06-09 7:33 ` [PATCH 2/4] Adding multi-interface support for multi-FD on destination side Het Gala
@ 2022-06-09 7:33 ` Het Gala
2022-06-16 17:39 ` Daniel P. Berrangé
2022-06-09 7:33 ` [PATCH 4/4] Adding support for multi-FD connections dynamically Het Gala
2022-06-09 15:47 ` [PATCH 0/4] Multiple interface support on top of Multi-FD Daniel P. Berrangé
4 siblings, 1 reply; 34+ messages in thread
From: Het Gala @ 2022-06-09 7:33 UTC (permalink / raw)
To: qemu-devel
Cc: quintela, dgilbert, pbonzini, berrange, armbru, eblake, Het Gala,
Manish Mishra
i) Binding of the socket to source ip address and port on the non-default
interface has been implemented for multi-FD connection, which was not
necessary earlier because the binding was on the default interface itself.
ii) Created an end to end connection between all multi-FD source and
destination pairs.
Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
Signed-off-by: Het Gala <het.gala@nutanix.com>
---
chardev/char-socket.c | 4 +-
include/io/channel-socket.h | 26 ++++++-----
include/qemu/sockets.h | 6 ++-
io/channel-socket.c | 50 ++++++++++++++------
migration/socket.c | 15 +++---
nbd/client-connection.c | 2 +-
qemu-nbd.c | 4 +-
scsi/pr-manager-helper.c | 1 +
tests/unit/test-char.c | 8 ++--
tests/unit/test-io-channel-socket.c | 4 +-
tests/unit/test-util-sockets.c | 16 +++----
ui/input-barrier.c | 2 +-
ui/vnc.c | 3 +-
util/qemu-sockets.c | 71 ++++++++++++++++++++---------
14 files changed, 135 insertions(+), 77 deletions(-)
diff --git a/chardev/char-socket.c b/chardev/char-socket.c
index dc4e218eeb..f3725238c5 100644
--- a/chardev/char-socket.c
+++ b/chardev/char-socket.c
@@ -932,7 +932,7 @@ static int tcp_chr_connect_client_sync(Chardev *chr, Error **errp)
QIOChannelSocket *sioc = qio_channel_socket_new();
tcp_chr_change_state(s, TCP_CHARDEV_STATE_CONNECTING);
tcp_chr_set_client_ioc_name(chr, sioc);
- if (qio_channel_socket_connect_sync(sioc, s->addr, errp) < 0) {
+ if (qio_channel_socket_connect_sync(sioc, s->addr, NULL, errp) < 0) {
tcp_chr_change_state(s, TCP_CHARDEV_STATE_DISCONNECTED);
object_unref(OBJECT(sioc));
return -1;
@@ -1120,7 +1120,7 @@ static void tcp_chr_connect_client_task(QIOTask *task,
SocketAddress *addr = opaque;
Error *err = NULL;
- qio_channel_socket_connect_sync(ioc, addr, &err);
+ qio_channel_socket_connect_sync(ioc, addr, NULL, &err);
qio_task_set_error(task, err);
}
diff --git a/include/io/channel-socket.h b/include/io/channel-socket.h
index 513c428fe4..59d5b1b349 100644
--- a/include/io/channel-socket.h
+++ b/include/io/channel-socket.h
@@ -83,41 +83,45 @@ qio_channel_socket_new_fd(int fd,
/**
* qio_channel_socket_connect_sync:
* @ioc: the socket channel object
- * @addr: the address to connect to
+ * @dst_addr: the destination address to connect to
+ * @src_addr: the source address to be connected
* @errp: pointer to a NULL-initialized error object
*
- * Attempt to connect to the address @addr. This method
- * will run in the foreground so the caller will not regain
- * execution control until the connection is established or
+ * Attempt to connect to the address @dst_addr with @src_addr.
+ * This method will run in the foreground so the caller will not
+ * regain execution control until the connection is established or
* an error occurs.
*/
int qio_channel_socket_connect_sync(QIOChannelSocket *ioc,
- SocketAddress *addr,
+ SocketAddress *dst_addr,
+ SocketAddress *src_addr,
Error **errp);
/**
* qio_channel_socket_connect_async:
* @ioc: the socket channel object
- * @addr: the address to connect to
+ * @dst_addr: the destination address to connect to
* @callback: the function to invoke on completion
* @opaque: user data to pass to @callback
* @destroy: the function to free @opaque
* @context: the context to run the async task. If %NULL, the default
* context will be used.
+ * @src_addr: the source address to be connected
*
- * Attempt to connect to the address @addr. This method
- * will run in the background so the caller will regain
+ * Attempt to connect to the address @dst_addr with the @src_addr.
+ * This method will run in the background so the caller will regain
* execution control immediately. The function @callback
- * will be invoked on completion or failure. The @addr
+ * will be invoked on completion or failure. The @dst_addr
* parameter will be copied, so may be freed as soon
* as this function returns without waiting for completion.
*/
void qio_channel_socket_connect_async(QIOChannelSocket *ioc,
- SocketAddress *addr,
+ SocketAddress *dst_addr,
QIOTaskFunc callback,
gpointer opaque,
GDestroyNotify destroy,
- GMainContext *context);
+ GMainContext *context,
+ SocketAddress *src_addr);
/**
diff --git a/include/qemu/sockets.h b/include/qemu/sockets.h
index 038faa157f..dc863c3df8 100644
--- a/include/qemu/sockets.h
+++ b/include/qemu/sockets.h
@@ -33,7 +33,8 @@ int inet_ai_family_from_address(InetSocketAddress *addr,
Error **errp);
int inet_parse(InetSocketAddress *addr, const char *str, Error **errp);
int inet_connect(const char *str, Error **errp);
-int inet_connect_saddr(InetSocketAddress *saddr, Error **errp);
+int inet_connect_saddr(InetSocketAddress *dst_addr,
+ InetSocketAddress *src_addr, Error **errp);
NetworkAddressFamily inet_netfamily(int family);
@@ -41,7 +42,8 @@ int unix_listen(const char *path, Error **errp);
int unix_connect(const char *path, Error **errp);
SocketAddress *socket_parse(const char *str, Error **errp);
-int socket_connect(SocketAddress *addr, Error **errp);
+int socket_connect(SocketAddress *dst_addr, SocketAddress *src_addr,
+ Error **errp);
int socket_listen(SocketAddress *addr, int num, Error **errp);
void socket_listen_cleanup(int fd, Error **errp);
int socket_dgram(SocketAddress *remote, SocketAddress *local, Error **errp);
diff --git a/io/channel-socket.c b/io/channel-socket.c
index dc9c165de1..f8746ad646 100644
--- a/io/channel-socket.c
+++ b/io/channel-socket.c
@@ -36,6 +36,12 @@
#define SOCKET_MAX_FDS 16
+struct SrcDestAddress {
+ SocketAddress *dst_addr;
+ SocketAddress *src_addr;
+};
+
+
SocketAddress *
qio_channel_socket_get_local_address(QIOChannelSocket *ioc,
Error **errp)
@@ -145,13 +151,14 @@ qio_channel_socket_new_fd(int fd,
int qio_channel_socket_connect_sync(QIOChannelSocket *ioc,
- SocketAddress *addr,
+ SocketAddress *dst_addr,
+ SocketAddress *src_addr,
Error **errp)
{
int fd;
- trace_qio_channel_socket_connect_sync(ioc, addr);
- fd = socket_connect(addr, errp);
+ trace_qio_channel_socket_connect_sync(ioc, dst_addr);
+ fd = socket_connect(dst_addr, src_addr, errp);
if (fd < 0) {
trace_qio_channel_socket_connect_fail(ioc);
return -1;
@@ -177,39 +184,56 @@ int qio_channel_socket_connect_sync(QIOChannelSocket *ioc,
}
+static void qio_channel_socket_worker_free(gpointer opaque)
+{
+ struct SrcDestAddress *data = opaque;
+ if (!data) {
+ return;
+ }
+ qapi_free_SocketAddress(data->dst_addr);
+ qapi_free_SocketAddress(data->src_addr);
+ g_free(data);
+}
+
+
static void qio_channel_socket_connect_worker(QIOTask *task,
gpointer opaque)
{
QIOChannelSocket *ioc = QIO_CHANNEL_SOCKET(qio_task_get_source(task));
- SocketAddress *addr = opaque;
+ struct SrcDestAddress *data = opaque;
Error *err = NULL;
- qio_channel_socket_connect_sync(ioc, addr, &err);
+ qio_channel_socket_connect_sync(ioc, data->dst_addr, data->src_addr, &err);
qio_task_set_error(task, err);
}
void qio_channel_socket_connect_async(QIOChannelSocket *ioc,
- SocketAddress *addr,
+ SocketAddress *dst_addr,
QIOTaskFunc callback,
gpointer opaque,
GDestroyNotify destroy,
- GMainContext *context)
+ GMainContext *context,
+ SocketAddress *src_addr)
{
QIOTask *task = qio_task_new(
OBJECT(ioc), callback, opaque, destroy);
- SocketAddress *addrCopy;
-
- addrCopy = QAPI_CLONE(SocketAddress, addr);
+ struct SrcDestAddress *data = g_new0(struct SrcDestAddress, 1);
+ data->dst_addr = QAPI_CLONE(SocketAddress, dst_addr);
+ if (src_addr) {
+ data->src_addr = QAPI_CLONE(SocketAddress, src_addr);
+ } else {
+ data->src_addr = NULL;
+ }
/* socket_connect() does a non-blocking connect(), but it
* still blocks in DNS lookups, so we must use a thread */
- trace_qio_channel_socket_connect_async(ioc, addr);
+ trace_qio_channel_socket_connect_async(ioc, dst_addr);
qio_task_run_in_thread(task,
qio_channel_socket_connect_worker,
- addrCopy,
- (GDestroyNotify)qapi_free_SocketAddress,
+ data,
+ qio_channel_socket_worker_free,
context);
}
diff --git a/migration/socket.c b/migration/socket.c
index 21e0983df2..d0cb7cc6a6 100644
--- a/migration/socket.c
+++ b/migration/socket.c
@@ -47,7 +47,7 @@ void socket_send_channel_create(QIOTaskFunc f, void *data)
{
QIOChannelSocket *sioc = qio_channel_socket_new();
qio_channel_socket_connect_async(sioc, outgoing_args.saddr,
- f, data, NULL, NULL);
+ f, data, NULL, NULL, NULL);
}
int socket_send_channel_destroy(QIOChannel *send)
@@ -110,7 +110,7 @@ out:
static void
socket_start_outgoing_migration_internal(MigrationState *s,
- SocketAddress *saddr,
+ SocketAddress *dst_addr,
Error **errp)
{
QIOChannelSocket *sioc = qio_channel_socket_new();
@@ -118,20 +118,17 @@ socket_start_outgoing_migration_internal(MigrationState *s,
data->s = s;
- /* in case previous migration leaked it */
- qapi_free_SocketAddress(outgoing_args.saddr);
- outgoing_args.saddr = saddr;
-
- if (saddr->type == SOCKET_ADDRESS_TYPE_INET) {
- data->hostname = g_strdup(saddr->u.inet.host);
+ if (dst_addr->type == SOCKET_ADDRESS_TYPE_INET) {
+ data->hostname = g_strdup(dst_addr->u.inet.host);
}
qio_channel_set_name(QIO_CHANNEL(sioc), "migration-socket-outgoing");
qio_channel_socket_connect_async(sioc,
- saddr,
+ dst_addr,
socket_outgoing_migration,
data,
socket_connect_data_free,
+ NULL,
NULL);
}
diff --git a/nbd/client-connection.c b/nbd/client-connection.c
index 2a632931c3..4bc505d26e 100644
--- a/nbd/client-connection.c
+++ b/nbd/client-connection.c
@@ -133,7 +133,7 @@ static int nbd_connect(QIOChannelSocket *sioc, SocketAddress *addr,
*outioc = NULL;
}
- ret = qio_channel_socket_connect_sync(sioc, addr, errp);
+ ret = qio_channel_socket_connect_sync(sioc, addr, NULL, errp);
if (ret < 0) {
return ret;
}
diff --git a/qemu-nbd.c b/qemu-nbd.c
index 0cd5aa6f02..fe178f271e 100644
--- a/qemu-nbd.c
+++ b/qemu-nbd.c
@@ -191,7 +191,7 @@ static int qemu_nbd_client_list(SocketAddress *saddr, QCryptoTLSCreds *tls,
int i, j;
sioc = qio_channel_socket_new();
- if (qio_channel_socket_connect_sync(sioc, saddr, &err) < 0) {
+ if (qio_channel_socket_connect_sync(sioc, saddr, NULL, &err) < 0) {
error_report_err(err);
goto out;
}
@@ -284,7 +284,7 @@ static void *nbd_client_thread(void *arg)
sioc = qio_channel_socket_new();
if (qio_channel_socket_connect_sync(sioc,
- saddr,
+ saddr, NULL,
&local_error) < 0) {
error_report_err(local_error);
goto out;
diff --git a/scsi/pr-manager-helper.c b/scsi/pr-manager-helper.c
index 3be52a98d5..6e3e00eadf 100644
--- a/scsi/pr-manager-helper.c
+++ b/scsi/pr-manager-helper.c
@@ -114,6 +114,7 @@ static int pr_manager_helper_initialize(PRManagerHelper *pr_mgr,
qio_channel_set_name(QIO_CHANNEL(sioc), "pr-manager-helper");
qio_channel_socket_connect_sync(sioc,
&saddr,
+ NULL,
&local_err);
g_free(path);
if (local_err) {
diff --git a/tests/unit/test-char.c b/tests/unit/test-char.c
index 5b3b48ebac..fd5f281777 100644
--- a/tests/unit/test-char.c
+++ b/tests/unit/test-char.c
@@ -691,7 +691,7 @@ char_socket_addr_to_opt_str(SocketAddress *addr, bool fd_pass,
if (is_listen) {
qio_channel_socket_listen_sync(ioc, addr, 1, &error_abort);
} else {
- qio_channel_socket_connect_sync(ioc, addr, &error_abort);
+ qio_channel_socket_connect_sync(ioc, addr, NULL, &error_abort);
}
fd = ioc->fd;
ioc->fd = -1;
@@ -748,7 +748,7 @@ char_socket_server_client_thread(gpointer data)
SocketAddress *addr = data;
QIOChannelSocket *ioc = qio_channel_socket_new();
- qio_channel_socket_connect_sync(ioc, addr, &error_abort);
+ qio_channel_socket_connect_sync(ioc, addr, NULL, &error_abort);
char_socket_ping_pong(QIO_CHANNEL(ioc), &error_abort);
@@ -1147,7 +1147,7 @@ static void char_socket_server_two_clients_test(gconstpointer opaque)
&closed, NULL, true);
ioc1 = qio_channel_socket_new();
- qio_channel_socket_connect_sync(ioc1, addr, &error_abort);
+ qio_channel_socket_connect_sync(ioc1, addr, NULL, &error_abort);
qemu_chr_wait_connected(chr, &error_abort);
/* switch the chardev to another context */
@@ -1161,7 +1161,7 @@ static void char_socket_server_two_clients_test(gconstpointer opaque)
* succeed immediately.
*/
ioc2 = qio_channel_socket_new();
- qio_channel_socket_connect_sync(ioc2, addr, &error_abort);
+ qio_channel_socket_connect_sync(ioc2, addr, NULL, &error_abort);
object_unref(OBJECT(ioc1));
/* The two connections should now be processed serially. */
diff --git a/tests/unit/test-io-channel-socket.c b/tests/unit/test-io-channel-socket.c
index 6713886d02..bc33ce6956 100644
--- a/tests/unit/test-io-channel-socket.c
+++ b/tests/unit/test-io-channel-socket.c
@@ -72,7 +72,7 @@ static void test_io_channel_setup_sync(SocketAddress *listen_addr,
*src = QIO_CHANNEL(qio_channel_socket_new());
qio_channel_socket_connect_sync(
- QIO_CHANNEL_SOCKET(*src), connect_addr, &error_abort);
+ QIO_CHANNEL_SOCKET(*src), connect_addr, NULL, &error_abort);
qio_channel_set_delay(*src, false);
qio_channel_wait(QIO_CHANNEL(lioc), G_IO_IN);
@@ -136,7 +136,7 @@ static void test_io_channel_setup_async(SocketAddress *listen_addr,
qio_channel_socket_connect_async(
QIO_CHANNEL_SOCKET(*src), connect_addr,
- test_io_channel_complete, &data, NULL, NULL);
+ test_io_channel_complete, &data, NULL, NULL, NULL);
g_main_loop_run(data.loop);
g_main_context_iteration(g_main_context_default(), FALSE);
diff --git a/tests/unit/test-util-sockets.c b/tests/unit/test-util-sockets.c
index 63909ccb2b..aa26630045 100644
--- a/tests/unit/test-util-sockets.c
+++ b/tests/unit/test-util-sockets.c
@@ -89,7 +89,7 @@ static void test_socket_fd_pass_name_good(void)
addr.type = SOCKET_ADDRESS_TYPE_FD;
addr.u.fd.str = g_strdup(mon_fdname);
- fd = socket_connect(&addr, &error_abort);
+ fd = socket_connect(&addr, NULL, &error_abort);
g_assert_cmpint(fd, !=, -1);
g_assert_cmpint(fd, !=, mon_fd);
close(fd);
@@ -121,7 +121,7 @@ static void test_socket_fd_pass_name_bad(void)
addr.type = SOCKET_ADDRESS_TYPE_FD;
addr.u.fd.str = g_strdup(mon_fdname);
- fd = socket_connect(&addr, &err);
+ fd = socket_connect(&addr, NULL, &err);
g_assert_cmpint(fd, ==, -1);
error_free_or_abort(&err);
@@ -148,7 +148,7 @@ static void test_socket_fd_pass_name_nomon(void)
addr.type = SOCKET_ADDRESS_TYPE_FD;
addr.u.fd.str = g_strdup("myfd");
- fd = socket_connect(&addr, &err);
+ fd = socket_connect(&addr, NULL, &err);
g_assert_cmpint(fd, ==, -1);
error_free_or_abort(&err);
@@ -172,7 +172,7 @@ static void test_socket_fd_pass_num_good(void)
addr.type = SOCKET_ADDRESS_TYPE_FD;
addr.u.fd.str = g_strdup_printf("%d", sfd);
- fd = socket_connect(&addr, &error_abort);
+ fd = socket_connect(&addr, NULL, &error_abort);
g_assert_cmpint(fd, ==, sfd);
fd = socket_listen(&addr, 1, &error_abort);
@@ -194,7 +194,7 @@ static void test_socket_fd_pass_num_bad(void)
addr.type = SOCKET_ADDRESS_TYPE_FD;
addr.u.fd.str = g_strdup_printf("%d", sfd);
- fd = socket_connect(&addr, &err);
+ fd = socket_connect(&addr, NULL, &err);
g_assert_cmpint(fd, ==, -1);
error_free_or_abort(&err);
@@ -217,7 +217,7 @@ static void test_socket_fd_pass_num_nocli(void)
addr.type = SOCKET_ADDRESS_TYPE_FD;
addr.u.fd.str = g_strdup_printf("%d", STDOUT_FILENO);
- fd = socket_connect(&addr, &err);
+ fd = socket_connect(&addr, NULL, &err);
g_assert_cmpint(fd, ==, -1);
error_free_or_abort(&err);
@@ -246,10 +246,10 @@ static gpointer unix_client_thread_func(gpointer user_data)
for (i = 0; i < ABSTRACT_SOCKET_VARIANTS; i++) {
if (row->expect_connect[i]) {
- fd = socket_connect(row->client[i], &error_abort);
+ fd = socket_connect(row->client[i], NULL, &error_abort);
g_assert_cmpint(fd, >=, 0);
} else {
- fd = socket_connect(row->client[i], &err);
+ fd = socket_connect(row->client[i], NULL, &err);
g_assert_cmpint(fd, ==, -1);
error_free_or_abort(&err);
}
diff --git a/ui/input-barrier.c b/ui/input-barrier.c
index 2d57ca7079..ee78654e3b 100644
--- a/ui/input-barrier.c
+++ b/ui/input-barrier.c
@@ -506,7 +506,7 @@ static void input_barrier_complete(UserCreatable *uc, Error **errp)
ib->sioc = qio_channel_socket_new();
qio_channel_set_name(QIO_CHANNEL(ib->sioc), "barrier-client");
- qio_channel_socket_connect_sync(ib->sioc, &ib->saddr, &local_err);
+ qio_channel_socket_connect_sync(ib->sioc, &ib->saddr, NULL, &local_err);
if (local_err) {
error_propagate(errp, local_err);
return;
diff --git a/ui/vnc.c b/ui/vnc.c
index 6a05d06147..c3dd84026a 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -3931,7 +3931,8 @@ static int vnc_display_connect(VncDisplay *vd,
vd->is_unix = saddr_list->value->type == SOCKET_ADDRESS_TYPE_UNIX;
sioc = qio_channel_socket_new();
qio_channel_set_name(QIO_CHANNEL(sioc), "vnc-reverse");
- if (qio_channel_socket_connect_sync(sioc, saddr_list->value, errp) < 0) {
+ if (qio_channel_socket_connect_sync(sioc, saddr_list->value,
+ NULL, errp) < 0) {
object_unref(OBJECT(sioc));
return -1;
}
diff --git a/util/qemu-sockets.c b/util/qemu-sockets.c
index 13b5b197f9..bbe0dc0ee0 100644
--- a/util/qemu-sockets.c
+++ b/util/qemu-sockets.c
@@ -226,7 +226,7 @@ static int inet_listen_saddr(InetSocketAddress *saddr,
return -1;
}
- memset(&ai,0, sizeof(ai));
+ memset(&ai,0,sizeof(ai));
ai.ai_flags = AI_PASSIVE;
if (saddr->has_numeric && saddr->numeric) {
ai.ai_flags |= AI_NUMERICHOST | AI_NUMERICSERV;
@@ -282,8 +282,8 @@ static int inet_listen_saddr(InetSocketAddress *saddr,
e->ai_protocol = IPPROTO_MPTCP;
}
#endif
- getnameinfo((struct sockaddr*)e->ai_addr,e->ai_addrlen,
- uaddr,INET6_ADDRSTRLEN,uport,32,
+ getnameinfo((struct sockaddr *)e->ai_addr, e->ai_addrlen,
+ uaddr, INET6_ADDRSTRLEN, uport, 32,
NI_NUMERICHOST | NI_NUMERICSERV);
port_min = inet_getport(e);
@@ -358,7 +358,8 @@ listen_ok:
((rc) == -EINPROGRESS)
#endif
-static int inet_connect_addr(const InetSocketAddress *saddr,
+static int inet_connect_addr(const InetSocketAddress *dst_addr,
+ const InetSocketAddress *src_addr,
struct addrinfo *addr, Error **errp)
{
int sock, rc;
@@ -371,8 +372,28 @@ static int inet_connect_addr(const InetSocketAddress *saddr,
}
socket_set_fast_reuse(sock);
+ /* to bind the socket if src_addr is available */
+
+ if (src_addr) {
+ struct sockaddr_in servaddr;
+
+ /* bind to a specific interface in the internet domain */
+ /* to make sure the sin_zero filed is cleared */
+ memset(&servaddr, 0, sizeof(servaddr));
+
+ servaddr.sin_family = AF_INET;
+ servaddr.sin_addr.s_addr = inet_addr(src_addr->host);
+ servaddr.sin_port = 0;
+
+ if (bind(sock, (struct sockaddr *)&servaddr, sizeof(servaddr)) < 0) {
+ error_setg_errno(errp, errno, "Failed to bind socket");
+ return -1;
+ }
+ }
+
/* connect to peer */
do {
+
rc = 0;
if (connect(sock, addr->ai_addr, addr->ai_addrlen) < 0) {
rc = -errno;
@@ -380,8 +401,14 @@ static int inet_connect_addr(const InetSocketAddress *saddr,
} while (rc == -EINTR);
if (rc < 0) {
- error_setg_errno(errp, errno, "Failed to connect to '%s:%s'",
- saddr->host, saddr->port);
+ if (src_addr) {
+ error_setg_errno(errp, errno, "Failed to connect '%s:%s' to "
+ "'%s:%s'", dst_addr->host, dst_addr->port,
+ src_addr->host, src_addr->port);
+ } else {
+ error_setg_errno(errp, errno, "Failed to connect '%s:%s'",
+ dst_addr->host, dst_addr->port);
+ }
closesocket(sock);
return -1;
}
@@ -446,13 +473,14 @@ static struct addrinfo *inet_parse_connect_saddr(InetSocketAddress *saddr,
*
* Returns: -1 on error, file descriptor on success.
*/
-int inet_connect_saddr(InetSocketAddress *saddr, Error **errp)
+int inet_connect_saddr(InetSocketAddress *dst_addr,
+ InetSocketAddress *src_addr, Error **errp)
{
Error *local_err = NULL;
struct addrinfo *res, *e;
int sock = -1;
- res = inet_parse_connect_saddr(saddr, errp);
+ res = inet_parse_connect_saddr(dst_addr, errp);
if (!res) {
return -1;
}
@@ -462,12 +490,12 @@ int inet_connect_saddr(InetSocketAddress *saddr, Error **errp)
local_err = NULL;
#ifdef HAVE_IPPROTO_MPTCP
- if (saddr->has_mptcp && saddr->mptcp) {
+ if (dst_addr->has_mptcp && dst_addr->mptcp) {
e->ai_protocol = IPPROTO_MPTCP;
}
#endif
- sock = inet_connect_addr(saddr, e, &local_err);
+ sock = inet_connect_addr(dst_addr, src_addr, e, &local_err);
if (sock >= 0) {
break;
}
@@ -480,7 +508,7 @@ int inet_connect_saddr(InetSocketAddress *saddr, Error **errp)
return sock;
}
- if (saddr->keep_alive) {
+ if (dst_addr->keep_alive) {
int val = 1;
int ret = setsockopt(sock, SOL_SOCKET, SO_KEEPALIVE,
&val, sizeof(val));
@@ -506,7 +534,7 @@ static int inet_dgram_saddr(InetSocketAddress *sraddr,
Error *err = NULL;
/* lookup peer addr */
- memset(&ai,0, sizeof(ai));
+ memset(&ai, 0, sizeof(ai));
ai.ai_flags = AI_CANONNAME | AI_V4MAPPED | AI_ADDRCONFIG;
ai.ai_family = inet_ai_family_from_address(sraddr, &err);
ai.ai_socktype = SOCK_DGRAM;
@@ -533,7 +561,7 @@ static int inet_dgram_saddr(InetSocketAddress *sraddr,
}
/* lookup local addr */
- memset(&ai,0, sizeof(ai));
+ memset(&ai, 0, sizeof(ai));
ai.ai_flags = AI_PASSIVE;
ai.ai_family = peer->ai_family;
ai.ai_socktype = SOCK_DGRAM;
@@ -574,7 +602,7 @@ static int inet_dgram_saddr(InetSocketAddress *sraddr,
}
/* connect to peer */
- if (connect(sock,peer->ai_addr,peer->ai_addrlen) < 0) {
+ if (connect(sock, peer->ai_addr, peer->ai_addrlen) < 0) {
error_setg_errno(errp, errno, "Failed to connect to '%s:%s'",
addr, port);
goto err;
@@ -727,7 +755,7 @@ int inet_connect(const char *str, Error **errp)
InetSocketAddress *addr = g_new(InetSocketAddress, 1);
if (!inet_parse(addr, str, errp)) {
- sock = inet_connect_saddr(addr, errp);
+ sock = inet_connect_saddr(addr, NULL, errp);
}
qapi_free_InetSocketAddress(addr);
return sock;
@@ -1182,25 +1210,26 @@ int socket_address_parse_named_fd(SocketAddress *addr, Error **errp)
return 0;
}
-int socket_connect(SocketAddress *addr, Error **errp)
+int socket_connect(SocketAddress *dst_addr, SocketAddress *src_addr, Error **errp)
{
int fd;
- switch (addr->type) {
+ switch (dst_addr->type) {
case SOCKET_ADDRESS_TYPE_INET:
- fd = inet_connect_saddr(&addr->u.inet, errp);
+ fd = inet_connect_saddr(&dst_addr->u.inet, src_addr ?
+ &src_addr->u.inet : NULL, errp);
break;
case SOCKET_ADDRESS_TYPE_UNIX:
- fd = unix_connect_saddr(&addr->u.q_unix, errp);
+ fd = unix_connect_saddr(&dst_addr->u.q_unix, errp);
break;
case SOCKET_ADDRESS_TYPE_FD:
- fd = socket_get_fd(addr->u.fd.str, errp);
+ fd = socket_get_fd(dst_addr->u.fd.str, errp);
break;
case SOCKET_ADDRESS_TYPE_VSOCK:
- fd = vsock_connect_saddr(&addr->u.vsock, errp);
+ fd = vsock_connect_saddr(&dst_addr->u.vsock, errp);
break;
default:
--
2.22.3
^ permalink raw reply related [flat|nested] 34+ messages in thread
* [PATCH 4/4] Adding support for multi-FD connections dynamically
2022-06-09 7:33 [PATCH 0/4] Multiple interface support on top of Multi-FD Het Gala
` (2 preceding siblings ...)
2022-06-09 7:33 ` [PATCH 3/4] Establishing connection between any non-default source and destination pair Het Gala
@ 2022-06-09 7:33 ` Het Gala
2022-06-16 18:47 ` Dr. David Alan Gilbert
2022-06-09 15:47 ` [PATCH 0/4] Multiple interface support on top of Multi-FD Daniel P. Berrangé
4 siblings, 1 reply; 34+ messages in thread
From: Het Gala @ 2022-06-09 7:33 UTC (permalink / raw)
To: qemu-devel
Cc: quintela, dgilbert, pbonzini, berrange, armbru, eblake, Het Gala,
Manish Mishra
i) Dynamically decide appropriate source and destination ip pairs for the
corresponding multi-FD channel to be connected.
ii) Removed the support for setting the number of multi-fd channels from qmp
commands. As now all multiFD parameters will be passed via qmp: migrate
command or incoming flag itself.
Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
Signed-off-by: Het Gala <het.gala@nutanix.com>
---
migration/migration.c | 15 ---------------
migration/migration.h | 1 -
migration/multifd.c | 42 +++++++++++++++++++++---------------------
migration/socket.c | 42 +++++++++++++++++++++++++++++++++---------
migration/socket.h | 4 +++-
monitor/hmp-cmds.c | 4 ----
qapi/migration.json | 6 ------
7 files changed, 57 insertions(+), 57 deletions(-)
diff --git a/migration/migration.c b/migration/migration.c
index 9b0ad732e7..57dd4494b4 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1585,9 +1585,6 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
if (params->has_block_incremental) {
dest->block_incremental = params->block_incremental;
}
- if (params->has_multifd_channels) {
- dest->multifd_channels = params->multifd_channels;
- }
if (params->has_multifd_compression) {
dest->multifd_compression = params->multifd_compression;
}
@@ -1702,9 +1699,6 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
if (params->has_block_incremental) {
s->parameters.block_incremental = params->block_incremental;
}
- if (params->has_multifd_channels) {
- s->parameters.multifd_channels = params->multifd_channels;
- }
if (params->has_multifd_compression) {
s->parameters.multifd_compression = params->multifd_compression;
}
@@ -2686,15 +2680,6 @@ bool migrate_pause_before_switchover(void)
MIGRATION_CAPABILITY_PAUSE_BEFORE_SWITCHOVER];
}
-int migrate_multifd_channels(void)
-{
- MigrationState *s;
-
- s = migrate_get_current();
-
- return s->parameters.multifd_channels;
-}
-
MultiFDCompression migrate_multifd_compression(void)
{
MigrationState *s;
diff --git a/migration/migration.h b/migration/migration.h
index fa8717ec9e..9464de8ef7 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -372,7 +372,6 @@ bool migrate_validate_uuid(void);
bool migrate_auto_converge(void);
bool migrate_use_multifd(void);
bool migrate_pause_before_switchover(void);
-int migrate_multifd_channels(void);
MultiFDCompression migrate_multifd_compression(void);
int migrate_multifd_zlib_level(void);
int migrate_multifd_zstd_level(void);
diff --git a/migration/multifd.c b/migration/multifd.c
index 9282ab6aa4..ce017436fb 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -225,7 +225,7 @@ static int multifd_recv_initial_packet(QIOChannel *c, Error **errp)
return -1;
}
- if (msg.id > migrate_multifd_channels()) {
+ if (msg.id > total_multifd_channels()) {
error_setg(errp, "multifd: received channel version %u "
"expected %u", msg.version, MULTIFD_VERSION);
return -1;
@@ -410,8 +410,8 @@ static int multifd_send_pages(QEMUFile *f)
* using more channels, so ensure it doesn't overflow if the
* limit is lower now.
*/
- next_channel %= migrate_multifd_channels();
- for (i = next_channel;; i = (i + 1) % migrate_multifd_channels()) {
+ next_channel %= total_multifd_channels();
+ for (i = next_channel;; i = (i + 1) % total_multifd_channels()) {
p = &multifd_send_state->params[i];
qemu_mutex_lock(&p->mutex);
@@ -422,7 +422,7 @@ static int multifd_send_pages(QEMUFile *f)
}
if (!p->pending_job) {
p->pending_job++;
- next_channel = (i + 1) % migrate_multifd_channels();
+ next_channel = (i + 1) % total_multifd_channels();
break;
}
qemu_mutex_unlock(&p->mutex);
@@ -500,7 +500,7 @@ static void multifd_send_terminate_threads(Error *err)
return;
}
- for (i = 0; i < migrate_multifd_channels(); i++) {
+ for (i = 0; i < total_multifd_channels(); i++) {
MultiFDSendParams *p = &multifd_send_state->params[i];
qemu_mutex_lock(&p->mutex);
@@ -521,14 +521,14 @@ void multifd_save_cleanup(void)
return;
}
multifd_send_terminate_threads(NULL);
- for (i = 0; i < migrate_multifd_channels(); i++) {
+ for (i = 0; i < total_multifd_channels(); i++) {
MultiFDSendParams *p = &multifd_send_state->params[i];
if (p->running) {
qemu_thread_join(&p->thread);
}
}
- for (i = 0; i < migrate_multifd_channels(); i++) {
+ for (i = 0; i < total_multifd_channels(); i++) {
MultiFDSendParams *p = &multifd_send_state->params[i];
Error *local_err = NULL;
@@ -594,7 +594,7 @@ int multifd_send_sync_main(QEMUFile *f)
flush_zero_copy = migrate_use_zero_copy_send();
- for (i = 0; i < migrate_multifd_channels(); i++) {
+ for (i = 0; i < total_multifd_channels(); i++) {
MultiFDSendParams *p = &multifd_send_state->params[i];
trace_multifd_send_sync_main_signal(p->id);
@@ -627,7 +627,7 @@ int multifd_send_sync_main(QEMUFile *f)
}
}
}
- for (i = 0; i < migrate_multifd_channels(); i++) {
+ for (i = 0; i < total_multifd_channels(); i++) {
MultiFDSendParams *p = &multifd_send_state->params[i];
trace_multifd_send_sync_main_wait(p->id);
@@ -903,7 +903,7 @@ int multifd_save_setup(Error **errp)
int thread_count;
uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
uint8_t i;
-
+ int idx;
if (!migrate_use_multifd()) {
return 0;
}
@@ -912,7 +912,7 @@ int multifd_save_setup(Error **errp)
return -1;
}
- thread_count = migrate_multifd_channels();
+ thread_count = total_multifd_channels();
multifd_send_state = g_malloc0(sizeof(*multifd_send_state));
multifd_send_state->params = g_new0(MultiFDSendParams, thread_count);
multifd_send_state->pages = multifd_pages_init(page_count);
@@ -945,8 +945,8 @@ int multifd_save_setup(Error **errp)
} else {
p->write_flags = 0;
}
-
- socket_send_channel_create(multifd_new_send_channel_async, p);
+ idx = multifd_index(i);
+ socket_send_channel_create(multifd_new_send_channel_async, p, idx);
}
for (i = 0; i < thread_count; i++) {
@@ -991,7 +991,7 @@ static void multifd_recv_terminate_threads(Error *err)
}
}
- for (i = 0; i < migrate_multifd_channels(); i++) {
+ for (i = 0; i < total_multifd_channels(); i++) {
MultiFDRecvParams *p = &multifd_recv_state->params[i];
qemu_mutex_lock(&p->mutex);
@@ -1017,7 +1017,7 @@ int multifd_load_cleanup(Error **errp)
return 0;
}
multifd_recv_terminate_threads(NULL);
- for (i = 0; i < migrate_multifd_channels(); i++) {
+ for (i = 0; i < total_multifd_channels(); i++) {
MultiFDRecvParams *p = &multifd_recv_state->params[i];
if (p->running) {
@@ -1030,7 +1030,7 @@ int multifd_load_cleanup(Error **errp)
qemu_thread_join(&p->thread);
}
}
- for (i = 0; i < migrate_multifd_channels(); i++) {
+ for (i = 0; i < total_multifd_channels(); i++) {
MultiFDRecvParams *p = &multifd_recv_state->params[i];
migration_ioc_unregister_yank(p->c);
@@ -1065,13 +1065,13 @@ void multifd_recv_sync_main(void)
if (!migrate_use_multifd()) {
return;
}
- for (i = 0; i < migrate_multifd_channels(); i++) {
+ for (i = 0; i < total_multifd_channels(); i++) {
MultiFDRecvParams *p = &multifd_recv_state->params[i];
trace_multifd_recv_sync_main_wait(p->id);
qemu_sem_wait(&multifd_recv_state->sem_sync);
}
- for (i = 0; i < migrate_multifd_channels(); i++) {
+ for (i = 0; i < total_multifd_channels(); i++) {
MultiFDRecvParams *p = &multifd_recv_state->params[i];
WITH_QEMU_LOCK_GUARD(&p->mutex) {
@@ -1166,7 +1166,7 @@ int multifd_load_setup(Error **errp)
error_setg(errp, "multifd is not supported by current protocol");
return -1;
}
- thread_count = migrate_multifd_channels();
+ thread_count = total_multifd_channels();
multifd_recv_state = g_malloc0(sizeof(*multifd_recv_state));
multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count);
qatomic_set(&multifd_recv_state->count, 0);
@@ -1204,7 +1204,7 @@ int multifd_load_setup(Error **errp)
bool multifd_recv_all_channels_created(void)
{
- int thread_count = migrate_multifd_channels();
+ int thread_count = total_multifd_channels();
if (!migrate_use_multifd()) {
return true;
@@ -1259,5 +1259,5 @@ bool multifd_recv_new_channel(QIOChannel *ioc, Error **errp)
QEMU_THREAD_JOINABLE);
qatomic_inc(&multifd_recv_state->count);
return qatomic_read(&multifd_recv_state->count) ==
- migrate_multifd_channels();
+ total_multifd_channels();
}
diff --git a/migration/socket.c b/migration/socket.c
index d0cb7cc6a6..c0ac6dbbe2 100644
--- a/migration/socket.c
+++ b/migration/socket.c
@@ -28,9 +28,6 @@
#include "trace.h"
-struct SocketOutgoingArgs {
- SocketAddress *saddr;
-} outgoing_args;
struct SocketArgs {
struct SrcDestAddr data;
@@ -43,20 +40,47 @@ struct OutgoingMigrateParams {
uint64_t total_multifd_channel;
} outgoing_migrate_params;
-void socket_send_channel_create(QIOTaskFunc f, void *data)
+
+int total_multifd_channels(void)
+{
+ return outgoing_migrate_params.total_multifd_channel;
+}
+
+int multifd_index(int i)
+{
+ int length = outgoing_migrate_params.length;
+ int j = 0;
+ int runn_sum = 0;
+ while (j < length) {
+ runn_sum += outgoing_migrate_params.socket_args[j].multifd_channels;
+ if (i >= runn_sum) {
+ j++;
+ } else {
+ break;
+ }
+ }
+ return j;
+}
+
+void socket_send_channel_create(QIOTaskFunc f, void *data, int idx)
{
QIOChannelSocket *sioc = qio_channel_socket_new();
- qio_channel_socket_connect_async(sioc, outgoing_args.saddr,
- f, data, NULL, NULL, NULL);
+ qio_channel_socket_connect_async(sioc,
+ outgoing_migrate_params.socket_args[idx].data.dst_addr,
+ f, data, NULL, NULL,
+ outgoing_migrate_params.socket_args[idx].data.src_addr);
}
int socket_send_channel_destroy(QIOChannel *send)
{
/* Remove channel */
object_unref(OBJECT(send));
- if (outgoing_args.saddr) {
- qapi_free_SocketAddress(outgoing_args.saddr);
- outgoing_args.saddr = NULL;
+ if (outgoing_migrate_params.socket_args != NULL) {
+ g_free(outgoing_migrate_params.socket_args);
+ outgoing_migrate_params.socket_args = NULL;
+ }
+ if (outgoing_migrate_params.length) {
+ outgoing_migrate_params.length = 0;
}
if (outgoing_migrate_params.socket_args != NULL) {
diff --git a/migration/socket.h b/migration/socket.h
index b9e3699167..c8b9252384 100644
--- a/migration/socket.h
+++ b/migration/socket.h
@@ -27,7 +27,9 @@ struct SrcDestAddr {
SocketAddress *src_addr;
};
-void socket_send_channel_create(QIOTaskFunc f, void *data);
+int total_multifd_channels(void);
+int multifd_index(int i);
+void socket_send_channel_create(QIOTaskFunc f, void *data, int idx);
int socket_send_channel_destroy(QIOChannel *send);
void socket_start_incoming_migration(const char *str, uint8_t number,
diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
index 32a6b67d5f..9a3d76d6ba 100644
--- a/monitor/hmp-cmds.c
+++ b/monitor/hmp-cmds.c
@@ -1281,10 +1281,6 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
p->has_block_incremental = true;
visit_type_bool(v, param, &p->block_incremental, &err);
break;
- case MIGRATION_PARAMETER_MULTIFD_CHANNELS:
- p->has_multifd_channels = true;
- visit_type_uint8(v, param, &p->multifd_channels, &err);
- break;
case MIGRATION_PARAMETER_MULTIFD_COMPRESSION:
p->has_multifd_compression = true;
visit_type_MultiFDCompression(v, param, &p->multifd_compression,
diff --git a/qapi/migration.json b/qapi/migration.json
index 62a7b22d19..1b1c6d01d3 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -877,11 +877,6 @@
# migrated and the destination must already have access to the
# same backing chain as was used on the source. (since 2.10)
#
-# @multifd-channels: Number of channels used to migrate data in
-# parallel. This is the same number that the
-# number of sockets used for migration. The
-# default value is 2 (since 4.0)
-#
# @xbzrle-cache-size: cache size to be used by XBZRLE migration. It
# needs to be a multiple of the target page size
# and a power of 2
@@ -965,7 +960,6 @@
'*x-checkpoint-delay': { 'type': 'uint32',
'features': [ 'unstable' ] },
'*block-incremental': 'bool',
- '*multifd-channels': 'uint8',
'*xbzrle-cache-size': 'size',
'*max-postcopy-bandwidth': 'size',
'*max-cpu-throttle': 'uint8',
--
2.22.3
^ permalink raw reply related [flat|nested] 34+ messages in thread
* Re: [PATCH 0/4] Multiple interface support on top of Multi-FD
2022-06-09 7:33 [PATCH 0/4] Multiple interface support on top of Multi-FD Het Gala
` (3 preceding siblings ...)
2022-06-09 7:33 ` [PATCH 4/4] Adding support for multi-FD connections dynamically Het Gala
@ 2022-06-09 15:47 ` Daniel P. Berrangé
2022-06-10 12:28 ` manish.mishra
4 siblings, 1 reply; 34+ messages in thread
From: Daniel P. Berrangé @ 2022-06-09 15:47 UTC (permalink / raw)
To: Het Gala; +Cc: qemu-devel, quintela, dgilbert, pbonzini, armbru, eblake
On Thu, Jun 09, 2022 at 07:33:01AM +0000, Het Gala wrote:
>
> As of now, the multi-FD feature supports connection over the default network
> only. This Patchset series is a Qemu side implementation of providing multiple
> interfaces support for multi-FD. This enables us to fully utilize dedicated or
> multiple NICs in case bonding of NICs is not possible.
>
>
> Introduction
> -------------
> Multi-FD Qemu implementation currently supports connection only on the default
> network. This forbids us from advantages like:
> - Separating VM live migration traffic from the default network.
Perhaps I'm mis-understanding your intent here, but AFAIK it
has been possible to separate VM migration traffic from general
host network traffic essentially forever.
If you have two NICs with IP addresses on different subnets,
then the kernel will pick which NIC to use automatically
based on the IP address of the target matching the kernel
routing table entries.
Management apps have long used this ability in order to
control which NIC migration traffic flows over.
> - Fully utilize all NICs’ capacity in cases where creating a LACP bond (Link
> Aggregation Control Protocol) is not supported.
Can you elaborate on scenarios in which it is impossible to use LACP
bonding at the kernel level ?
> Multi-interface with Multi-FD
> -----------------------------
> Multiple-interface support over basic multi-FD has been implemented in the
> patches. Advantages of this implementation are:
> - Able to separate live migration traffic from default network interface by
> creating multiFD channels on ip addresses of multiple non-default interfaces.
> - Can optimize the number of multi-FD channels on a particular interface
> depending upon the network bandwidth limit on a particular interface.
Manually assigning individual channels to different NICs is a pretty
inefficient way to optimizing traffic. Feels like you could easily get
into a situation where one NIC ends up idle while the other is busy,
especially if the traffic patterns are different. For example with
post-copy there's an extra channel for OOB async page requests, and
its far from clear that manually picking NICs per chanel upfront is
going work for that. The kernel can continually dynamically balance
load on the fly and so do much better than any static mapping QEMU
tries to apply, especially if there are multiple distinct QEMU's
competing for bandwidth.
> Implementation
> --------------
>
> Earlier the 'migrate' qmp command:
> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
>
> Modified qmp command:
> { "execute": "migrate",
> "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
> "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
> "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
> "destination-uri": "tcp:11.0.0.0:7789",
> "multifd-channels": 5} ] } }
> ------------------------------------------------------------------------------
>
> Earlier the 'migrate-incoming' qmp command:
> { "execute": "migrate-incoming", "arguments": { "uri": "tcp::4446" } }
>
> Modified 'migrate-incoming' qmp command:
> { "execute": "migrate-incoming",
> "arguments": {"uri": "tcp::6789",
> "multi-fd-uri-list" : [ {"destination-uri" : "tcp::6900",
> "multifd-channels": 4}, {"destination-uri" : "tcp:11.0.0.0:7789",
> "multifd-channels": 5} ] } }
> ------------------------------------------------------------------------------
These examples pretty nicely illustrate my concern with this
proposal. It is making QEMU configuration of migration
massively more complicated, while duplicating functionality
the kernel can provide via NIC teaming, but without having
ability to balance it on the fly as the kernel would.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 0/4] Multiple interface support on top of Multi-FD
2022-06-09 15:47 ` [PATCH 0/4] Multiple interface support on top of Multi-FD Daniel P. Berrangé
@ 2022-06-10 12:28 ` manish.mishra
2022-06-15 16:43 ` Daniel P. Berrangé
0 siblings, 1 reply; 34+ messages in thread
From: manish.mishra @ 2022-06-10 12:28 UTC (permalink / raw)
To: Daniel P. Berrangé, Het Gala
Cc: qemu-devel, quintela, dgilbert, pbonzini, armbru, eblake
On 09/06/22 9:17 pm, Daniel P. Berrangé wrote:
> On Thu, Jun 09, 2022 at 07:33:01AM +0000, Het Gala wrote:
>> As of now, the multi-FD feature supports connection over the default network
>> only. This Patchset series is a Qemu side implementation of providing multiple
>> interfaces support for multi-FD. This enables us to fully utilize dedicated or
>> multiple NICs in case bonding of NICs is not possible.
>>
>>
>> Introduction
>> -------------
>> Multi-FD Qemu implementation currently supports connection only on the default
>> network. This forbids us from advantages like:
>> - Separating VM live migration traffic from the default network.
Hi Daniel,
I totally understand your concern around this approach increasing compexity inside qemu,
when similar things can be done with NIC teaming. But we thought this approach provides
much more flexibility to user in few cases like.
1. We checked our customer data, almost all of the host had multiple NIC, but LACP support
in their setups was very rare. So for those cases this approach can help in utilise multiple
NICs as teaming is not possible there.
2. We have seen requests recently to separate out traffic of storage, VM netwrok, migration
over different vswitch which can be backed by 1 or more NICs as this give better
predictability and assurance. So host with multiple ips/vswitches can be very common
environment. In this kind of enviroment this approach gives per vm or migration level
flexibilty, like for critical VM we can still use bandwidth from all available vswitch/interface
but for normal VM they can keep live migration only on dedicated NICs without changing
complete host network topology.
At final we want it to be something like this [<ip-pair>, <multiFD-channels>, <bandwidth_control>]
to provide bandwidth_control per interface.
3. Dedicated NIC we mentioned as a use case, agree with you it can be done without this
approach too.
> Perhaps I'm mis-understanding your intent here, but AFAIK it
> has been possible to separate VM migration traffic from general
> host network traffic essentially forever.
>
> If you have two NICs with IP addresses on different subnets,
> then the kernel will pick which NIC to use automatically
> based on the IP address of the target matching the kernel
> routing table entries.
>
> Management apps have long used this ability in order to
> control which NIC migration traffic flows over.
>
>> - Fully utilize all NICs’ capacity in cases where creating a LACP bond (Link
>> Aggregation Control Protocol) is not supported.
> Can you elaborate on scenarios in which it is impossible to use LACP
> bonding at the kernel level ?
Yes, as mentioned above LACP support was rare in customer setups.
>> Multi-interface with Multi-FD
>> -----------------------------
>> Multiple-interface support over basic multi-FD has been implemented in the
>> patches. Advantages of this implementation are:
>> - Able to separate live migration traffic from default network interface by
>> creating multiFD channels on ip addresses of multiple non-default interfaces.
>> - Can optimize the number of multi-FD channels on a particular interface
>> depending upon the network bandwidth limit on a particular interface.
> Manually assigning individual channels to different NICs is a pretty
> inefficient way to optimizing traffic. Feels like you could easily get
> into a situation where one NIC ends up idle while the other is busy,
> especially if the traffic patterns are different. For example with
> post-copy there's an extra channel for OOB async page requests, and
> its far from clear that manually picking NICs per chanel upfront is
> going work for that. The kernel can continually dynamically balance
> load on the fly and so do much better than any static mapping QEMU
> tries to apply, especially if there are multiple distinct QEMU's
> competing for bandwidth.
>
Yes, Daniel current solution is only for pre-copy. As with postcopy
multiFD is not yet supported but in future we can extend it for postcopy
channels too.
>> Implementation
>> --------------
>>
>> Earlier the 'migrate' qmp command:
>> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
>>
>> Modified qmp command:
>> { "execute": "migrate",
>> "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
>> "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
>> "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
>> "destination-uri": "tcp:11.0.0.0:7789",
>> "multifd-channels": 5} ] } }
>> ------------------------------------------------------------------------------
>>
>> Earlier the 'migrate-incoming' qmp command:
>> { "execute": "migrate-incoming", "arguments": { "uri": "tcp::4446" } }
>>
>> Modified 'migrate-incoming' qmp command:
>> { "execute": "migrate-incoming",
>> "arguments": {"uri": "tcp::6789",
>> "multi-fd-uri-list" : [ {"destination-uri" : "tcp::6900",
>> "multifd-channels": 4}, {"destination-uri" : "tcp:11.0.0.0:7789",
>> "multifd-channels": 5} ] } }
>> ------------------------------------------------------------------------------
> These examples pretty nicely illustrate my concern with this
> proposal. It is making QEMU configuration of migration
> massively more complicated, while duplicating functionality
> the kernel can provide via NIC teaming, but without having
> ability to balance it on the fly as the kernel would.
Yes, agree Daniel this raises complexity but we will make sure that it does not
change/imapct anything existing and we provide new options as optional.
Few of the things about this may not be taken care currently as this was to
get some early feedback but we will definately modify that.
>
> With regards,
> Daniel
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 0/4] Multiple interface support on top of Multi-FD
2022-06-10 12:28 ` manish.mishra
@ 2022-06-15 16:43 ` Daniel P. Berrangé
2022-06-15 19:14 ` Dr. David Alan Gilbert
2022-06-16 8:27 ` Daniel P. Berrangé
0 siblings, 2 replies; 34+ messages in thread
From: Daniel P. Berrangé @ 2022-06-15 16:43 UTC (permalink / raw)
To: manish.mishra
Cc: Het Gala, qemu-devel, quintela, dgilbert, pbonzini, armbru, eblake
On Fri, Jun 10, 2022 at 05:58:31PM +0530, manish.mishra wrote:
>
> On 09/06/22 9:17 pm, Daniel P. Berrangé wrote:
> > On Thu, Jun 09, 2022 at 07:33:01AM +0000, Het Gala wrote:
> > > As of now, the multi-FD feature supports connection over the default network
> > > only. This Patchset series is a Qemu side implementation of providing multiple
> > > interfaces support for multi-FD. This enables us to fully utilize dedicated or
> > > multiple NICs in case bonding of NICs is not possible.
> > >
> > >
> > > Introduction
> > > -------------
> > > Multi-FD Qemu implementation currently supports connection only on the default
> > > network. This forbids us from advantages like:
> > > - Separating VM live migration traffic from the default network.
>
> Hi Daniel,
>
> I totally understand your concern around this approach increasing compexity inside qemu,
>
> when similar things can be done with NIC teaming. But we thought this approach provides
>
> much more flexibility to user in few cases like.
>
> 1. We checked our customer data, almost all of the host had multiple NIC, but LACP support
>
> in their setups was very rare. So for those cases this approach can help in utilise multiple
>
> NICs as teaming is not possible there.
AFAIK, LACP is not required in order to do link aggregation with Linux.
Traditional Linux bonding has no special NIC hardware or switch requirements,
so LACP is merely a "nice to have" in order to simplify some aspects.
IOW, migration with traffic spread across multiple NICs is already
possible AFAICT.
I can understand that some people may not have actually configured
bonding on their hosts, but it is not unreasonable to request that
they do so, if they want to take advantage fo aggrated bandwidth.
It has the further benefit that it will be fault tolerant. With
this proposal if any single NIC has a problem, the whole migration
will get stuck. With kernel level bonding, if any single NIC haus
a problem, it'll get offlined by the kernel and migration will
continue to work across remaining active NICs.
> 2. We have seen requests recently to separate out traffic of storage, VM netwrok, migration
>
> over different vswitch which can be backed by 1 or more NICs as this give better
>
> predictability and assurance. So host with multiple ips/vswitches can be very common
>
> environment. In this kind of enviroment this approach gives per vm or migration level
>
> flexibilty, like for critical VM we can still use bandwidth from all available vswitch/interface
>
> but for normal VM they can keep live migration only on dedicated NICs without changing
>
> complete host network topology.
>
> At final we want it to be something like this [<ip-pair>, <multiFD-channels>, <bandwidth_control>]
>
> to provide bandwidth_control per interface.
Again, it is already possible to separate migration traffic from storage
traffic, from other network traffic. The target IP given will influence
which NIC is used based on routing table and I know this is already
done widely with OpenStack deployments.
> 3. Dedicated NIC we mentioned as a use case, agree with you it can be done without this
>
> approach too.
> > > Multi-interface with Multi-FD
> > > -----------------------------
> > > Multiple-interface support over basic multi-FD has been implemented in the
> > > patches. Advantages of this implementation are:
> > > - Able to separate live migration traffic from default network interface by
> > > creating multiFD channels on ip addresses of multiple non-default interfaces.
> > > - Can optimize the number of multi-FD channels on a particular interface
> > > depending upon the network bandwidth limit on a particular interface.
> > Manually assigning individual channels to different NICs is a pretty
> > inefficient way to optimizing traffic. Feels like you could easily get
> > into a situation where one NIC ends up idle while the other is busy,
> > especially if the traffic patterns are different. For example with
> > post-copy there's an extra channel for OOB async page requests, and
> > its far from clear that manually picking NICs per chanel upfront is
> > going work for that. The kernel can continually dynamically balance
> > load on the fly and so do much better than any static mapping QEMU
> > tries to apply, especially if there are multiple distinct QEMU's
> > competing for bandwidth.
> >
> Yes, Daniel current solution is only for pre-copy. As with postcopy
> multiFD is not yet supported but in future we can extend it for postcopy
>
> channels too.
>
> > > Implementation
> > > --------------
> > >
> > > Earlier the 'migrate' qmp command:
> > > { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
> > >
> > > Modified qmp command:
> > > { "execute": "migrate",
> > > "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
> > > "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
> > > "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
> > > "destination-uri": "tcp:11.0.0.0:7789",
> > > "multifd-channels": 5} ] } }
> > > ------------------------------------------------------------------------------
> > >
> > > Earlier the 'migrate-incoming' qmp command:
> > > { "execute": "migrate-incoming", "arguments": { "uri": "tcp::4446" } }
> > >
> > > Modified 'migrate-incoming' qmp command:
> > > { "execute": "migrate-incoming",
> > > "arguments": {"uri": "tcp::6789",
> > > "multi-fd-uri-list" : [ {"destination-uri" : "tcp::6900",
> > > "multifd-channels": 4}, {"destination-uri" : "tcp:11.0.0.0:7789",
> > > "multifd-channels": 5} ] } }
> > > ------------------------------------------------------------------------------
> > These examples pretty nicely illustrate my concern with this
> > proposal. It is making QEMU configuration of migration
> > massively more complicated, while duplicating functionality
> > the kernel can provide via NIC teaming, but without having
> > ability to balance it on the fly as the kernel would.
>
> Yes, agree Daniel this raises complexity but we will make sure that it does not
>
> change/imapct anything existing and we provide new options as optional.
The added code is certainly going to impact ongoing maint of QEMU I/O
layer and migration in particular. I'm not convinced this complexity
is compelling enough compared to leveraging kernel native bonding
to justify the maint burden it will impose.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 0/4] Multiple interface support on top of Multi-FD
2022-06-15 16:43 ` Daniel P. Berrangé
@ 2022-06-15 19:14 ` Dr. David Alan Gilbert
2022-06-16 8:16 ` Daniel P. Berrangé
2022-06-16 8:27 ` Daniel P. Berrangé
1 sibling, 1 reply; 34+ messages in thread
From: Dr. David Alan Gilbert @ 2022-06-15 19:14 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: manish.mishra, Het Gala, qemu-devel, quintela, pbonzini, armbru, eblake
* Daniel P. Berrangé (berrange@redhat.com) wrote:
> On Fri, Jun 10, 2022 at 05:58:31PM +0530, manish.mishra wrote:
> >
> > On 09/06/22 9:17 pm, Daniel P. Berrangé wrote:
> > > On Thu, Jun 09, 2022 at 07:33:01AM +0000, Het Gala wrote:
> > > > As of now, the multi-FD feature supports connection over the default network
> > > > only. This Patchset series is a Qemu side implementation of providing multiple
> > > > interfaces support for multi-FD. This enables us to fully utilize dedicated or
> > > > multiple NICs in case bonding of NICs is not possible.
> > > >
> > > >
> > > > Introduction
> > > > -------------
> > > > Multi-FD Qemu implementation currently supports connection only on the default
> > > > network. This forbids us from advantages like:
> > > > - Separating VM live migration traffic from the default network.
> >
> > Hi Daniel,
> >
> > I totally understand your concern around this approach increasing compexity inside qemu,
> >
> > when similar things can be done with NIC teaming. But we thought this approach provides
> >
> > much more flexibility to user in few cases like.
> >
> > 1. We checked our customer data, almost all of the host had multiple NIC, but LACP support
> >
> > in their setups was very rare. So for those cases this approach can help in utilise multiple
> >
> > NICs as teaming is not possible there.
>
> AFAIK, LACP is not required in order to do link aggregation with Linux.
> Traditional Linux bonding has no special NIC hardware or switch requirements,
> so LACP is merely a "nice to have" in order to simplify some aspects.
>
> IOW, migration with traffic spread across multiple NICs is already
> possible AFAICT.
Are we sure that works with multifd? I've seen a lot of bonding NIC
setups which spread based on a hash of source/destination IP and port
numbers; given that we use the same dest port and IP at the moment what
happens in reality? That hashing can be quite delicate for high
bandwidth single streams.
> I can understand that some people may not have actually configured
> bonding on their hosts, but it is not unreasonable to request that
> they do so, if they want to take advantage fo aggrated bandwidth.
>
> It has the further benefit that it will be fault tolerant. With
> this proposal if any single NIC has a problem, the whole migration
> will get stuck. With kernel level bonding, if any single NIC haus
> a problem, it'll get offlined by the kernel and migration will
> continue to work across remaining active NICs.
>
> > 2. We have seen requests recently to separate out traffic of storage, VM netwrok, migration
> >
> > over different vswitch which can be backed by 1 or more NICs as this give better
> >
> > predictability and assurance. So host with multiple ips/vswitches can be very common
> >
> > environment. In this kind of enviroment this approach gives per vm or migration level
> >
> > flexibilty, like for critical VM we can still use bandwidth from all available vswitch/interface
> >
> > but for normal VM they can keep live migration only on dedicated NICs without changing
> >
> > complete host network topology.
> >
> > At final we want it to be something like this [<ip-pair>, <multiFD-channels>, <bandwidth_control>]
> >
> > to provide bandwidth_control per interface.
>
> Again, it is already possible to separate migration traffic from storage
> traffic, from other network traffic. The target IP given will influence
> which NIC is used based on routing table and I know this is already
> done widely with OpenStack deployments.
>
> > 3. Dedicated NIC we mentioned as a use case, agree with you it can be done without this
> >
> > approach too.
>
>
> > > > Multi-interface with Multi-FD
> > > > -----------------------------
> > > > Multiple-interface support over basic multi-FD has been implemented in the
> > > > patches. Advantages of this implementation are:
> > > > - Able to separate live migration traffic from default network interface by
> > > > creating multiFD channels on ip addresses of multiple non-default interfaces.
> > > > - Can optimize the number of multi-FD channels on a particular interface
> > > > depending upon the network bandwidth limit on a particular interface.
> > > Manually assigning individual channels to different NICs is a pretty
> > > inefficient way to optimizing traffic. Feels like you could easily get
> > > into a situation where one NIC ends up idle while the other is busy,
> > > especially if the traffic patterns are different. For example with
> > > post-copy there's an extra channel for OOB async page requests, and
> > > its far from clear that manually picking NICs per chanel upfront is
> > > going work for that. The kernel can continually dynamically balance
> > > load on the fly and so do much better than any static mapping QEMU
> > > tries to apply, especially if there are multiple distinct QEMU's
> > > competing for bandwidth.
> > >
> > Yes, Daniel current solution is only for pre-copy. As with postcopy
> > multiFD is not yet supported but in future we can extend it for postcopy
I had been thinking about explicit selection of network device for NUMA
use though; ideally I'd like to be able to associate a set of multifd
threads to each NUMA node, and then associate a NIC with that set of
threads; so that the migration happens down the NIC that's on the node
the RAM is on. On a really good day you'd have one NIC per top level
NUMA node.
> > channels too.
> >
> > > > Implementation
> > > > --------------
> > > >
> > > > Earlier the 'migrate' qmp command:
> > > > { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
> > > >
> > > > Modified qmp command:
> > > > { "execute": "migrate",
> > > > "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
> > > > "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
> > > > "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
> > > > "destination-uri": "tcp:11.0.0.0:7789",
> > > > "multifd-channels": 5} ] } }
> > > > ------------------------------------------------------------------------------
> > > >
> > > > Earlier the 'migrate-incoming' qmp command:
> > > > { "execute": "migrate-incoming", "arguments": { "uri": "tcp::4446" } }
> > > >
> > > > Modified 'migrate-incoming' qmp command:
> > > > { "execute": "migrate-incoming",
> > > > "arguments": {"uri": "tcp::6789",
> > > > "multi-fd-uri-list" : [ {"destination-uri" : "tcp::6900",
> > > > "multifd-channels": 4}, {"destination-uri" : "tcp:11.0.0.0:7789",
> > > > "multifd-channels": 5} ] } }
> > > > ------------------------------------------------------------------------------
> > > These examples pretty nicely illustrate my concern with this
> > > proposal. It is making QEMU configuration of migration
> > > massively more complicated, while duplicating functionality
> > > the kernel can provide via NIC teaming, but without having
> > > ability to balance it on the fly as the kernel would.
> >
> > Yes, agree Daniel this raises complexity but we will make sure that it does not
> >
> > change/imapct anything existing and we provide new options as optional.
>
> The added code is certainly going to impact ongoing maint of QEMU I/O
> layer and migration in particular. I'm not convinced this complexity
> is compelling enough compared to leveraging kernel native bonding
> to justify the maint burden it will impose.
Dave
> With regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 0/4] Multiple interface support on top of Multi-FD
2022-06-15 19:14 ` Dr. David Alan Gilbert
@ 2022-06-16 8:16 ` Daniel P. Berrangé
2022-06-16 10:14 ` manish.mishra
0 siblings, 1 reply; 34+ messages in thread
From: Daniel P. Berrangé @ 2022-06-16 8:16 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: manish.mishra, Het Gala, qemu-devel, quintela, pbonzini, armbru, eblake
On Wed, Jun 15, 2022 at 08:14:26PM +0100, Dr. David Alan Gilbert wrote:
> * Daniel P. Berrangé (berrange@redhat.com) wrote:
> > On Fri, Jun 10, 2022 at 05:58:31PM +0530, manish.mishra wrote:
> > >
> > > On 09/06/22 9:17 pm, Daniel P. Berrangé wrote:
> > > > On Thu, Jun 09, 2022 at 07:33:01AM +0000, Het Gala wrote:
> > > > > As of now, the multi-FD feature supports connection over the default network
> > > > > only. This Patchset series is a Qemu side implementation of providing multiple
> > > > > interfaces support for multi-FD. This enables us to fully utilize dedicated or
> > > > > multiple NICs in case bonding of NICs is not possible.
> > > > >
> > > > >
> > > > > Introduction
> > > > > -------------
> > > > > Multi-FD Qemu implementation currently supports connection only on the default
> > > > > network. This forbids us from advantages like:
> > > > > - Separating VM live migration traffic from the default network.
> > >
> > > Hi Daniel,
> > >
> > > I totally understand your concern around this approach increasing compexity inside qemu,
> > >
> > > when similar things can be done with NIC teaming. But we thought this approach provides
> > >
> > > much more flexibility to user in few cases like.
> > >
> > > 1. We checked our customer data, almost all of the host had multiple NIC, but LACP support
> > >
> > > in their setups was very rare. So for those cases this approach can help in utilise multiple
> > >
> > > NICs as teaming is not possible there.
> >
> > AFAIK, LACP is not required in order to do link aggregation with Linux.
> > Traditional Linux bonding has no special NIC hardware or switch requirements,
> > so LACP is merely a "nice to have" in order to simplify some aspects.
> >
> > IOW, migration with traffic spread across multiple NICs is already
> > possible AFAICT.
>
> Are we sure that works with multifd? I've seen a lot of bonding NIC
> setups which spread based on a hash of source/destination IP and port
> numbers; given that we use the same dest port and IP at the moment what
> happens in reality? That hashing can be quite delicate for high
> bandwidth single streams.
The simplest Linux bonding mode does per-packet round-robin across
NICs, so traffic from the collection of multifd connections should
fill up all the NICs in the bond. There are of course other modes
which may be sub-optimal for the reasons you describe. Which mode
to pick depends on the type of service traffic patterns you're
aiming to balance.
> > > > > Multi-interface with Multi-FD
> > > > > -----------------------------
> > > > > Multiple-interface support over basic multi-FD has been implemented in the
> > > > > patches. Advantages of this implementation are:
> > > > > - Able to separate live migration traffic from default network interface by
> > > > > creating multiFD channels on ip addresses of multiple non-default interfaces.
> > > > > - Can optimize the number of multi-FD channels on a particular interface
> > > > > depending upon the network bandwidth limit on a particular interface.
> > > > Manually assigning individual channels to different NICs is a pretty
> > > > inefficient way to optimizing traffic. Feels like you could easily get
> > > > into a situation where one NIC ends up idle while the other is busy,
> > > > especially if the traffic patterns are different. For example with
> > > > post-copy there's an extra channel for OOB async page requests, and
> > > > its far from clear that manually picking NICs per chanel upfront is
> > > > going work for that. The kernel can continually dynamically balance
> > > > load on the fly and so do much better than any static mapping QEMU
> > > > tries to apply, especially if there are multiple distinct QEMU's
> > > > competing for bandwidth.
> > > >
> > > Yes, Daniel current solution is only for pre-copy. As with postcopy
> > > multiFD is not yet supported but in future we can extend it for postcopy
>
> I had been thinking about explicit selection of network device for NUMA
> use though; ideally I'd like to be able to associate a set of multifd
> threads to each NUMA node, and then associate a NIC with that set of
> threads; so that the migration happens down the NIC that's on the node
> the RAM is on. On a really good day you'd have one NIC per top level
> NUMA node.
Now that's an interesting idea, and not one that can be dealt with
by bonding, since the network layer won't be aware of the NUMA
affinity constraints.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 0/4] Multiple interface support on top of Multi-FD
2022-06-15 16:43 ` Daniel P. Berrangé
2022-06-15 19:14 ` Dr. David Alan Gilbert
@ 2022-06-16 8:27 ` Daniel P. Berrangé
2022-06-16 15:50 ` Dr. David Alan Gilbert
1 sibling, 1 reply; 34+ messages in thread
From: Daniel P. Berrangé @ 2022-06-16 8:27 UTC (permalink / raw)
To: manish.mishra, Het Gala, qemu-devel, quintela, dgilbert,
pbonzini, armbru, eblake
On Wed, Jun 15, 2022 at 05:43:28PM +0100, Daniel P. Berrangé wrote:
> On Fri, Jun 10, 2022 at 05:58:31PM +0530, manish.mishra wrote:
> >
> > On 09/06/22 9:17 pm, Daniel P. Berrangé wrote:
> > > On Thu, Jun 09, 2022 at 07:33:01AM +0000, Het Gala wrote:
> > > > As of now, the multi-FD feature supports connection over the default network
> > > > only. This Patchset series is a Qemu side implementation of providing multiple
> > > > interfaces support for multi-FD. This enables us to fully utilize dedicated or
> > > > multiple NICs in case bonding of NICs is not possible.
> > > >
> > > >
> > > > Introduction
> > > > -------------
> > > > Multi-FD Qemu implementation currently supports connection only on the default
> > > > network. This forbids us from advantages like:
> > > > - Separating VM live migration traffic from the default network.
> >
> > Hi Daniel,
> >
> > I totally understand your concern around this approach increasing compexity inside qemu,
> >
> > when similar things can be done with NIC teaming. But we thought this approach provides
> >
> > much more flexibility to user in few cases like.
> >
> > 1. We checked our customer data, almost all of the host had multiple NIC, but LACP support
> >
> > in their setups was very rare. So for those cases this approach can help in utilise multiple
> >
> > NICs as teaming is not possible there.
>
> AFAIK, LACP is not required in order to do link aggregation with Linux.
> Traditional Linux bonding has no special NIC hardware or switch requirements,
> so LACP is merely a "nice to have" in order to simplify some aspects.
>
> IOW, migration with traffic spread across multiple NICs is already
> possible AFAICT.
>
> I can understand that some people may not have actually configured
> bonding on their hosts, but it is not unreasonable to request that
> they do so, if they want to take advantage fo aggrated bandwidth.
>
> It has the further benefit that it will be fault tolerant. With
> this proposal if any single NIC has a problem, the whole migration
> will get stuck. With kernel level bonding, if any single NIC haus
> a problem, it'll get offlined by the kernel and migration will
> continue to work across remaining active NICs.
>
> > 2. We have seen requests recently to separate out traffic of storage, VM netwrok, migration
> >
> > over different vswitch which can be backed by 1 or more NICs as this give better
> >
> > predictability and assurance. So host with multiple ips/vswitches can be very common
> >
> > environment. In this kind of enviroment this approach gives per vm or migration level
> >
> > flexibilty, like for critical VM we can still use bandwidth from all available vswitch/interface
> >
> > but for normal VM they can keep live migration only on dedicated NICs without changing
> >
> > complete host network topology.
> >
> > At final we want it to be something like this [<ip-pair>, <multiFD-channels>, <bandwidth_control>]
> >
> > to provide bandwidth_control per interface.
>
> Again, it is already possible to separate migration traffic from storage
> traffic, from other network traffic. The target IP given will influence
> which NIC is used based on routing table and I know this is already
> done widely with OpenStack deployments.
Actually I should clarify this is only practical if the two NICs are
using different IP subnets, otherwise routing rules are not viable.
So needing to set source IP would be needed to select between a pair
of NICs on the same IP subnet.
Previous usage I've seen has always setup fully distinct IP subnets
for generic vs storage vs migration network traffic.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 0/4] Multiple interface support on top of Multi-FD
2022-06-16 8:16 ` Daniel P. Berrangé
@ 2022-06-16 10:14 ` manish.mishra
2022-06-16 17:32 ` Daniel P. Berrangé
0 siblings, 1 reply; 34+ messages in thread
From: manish.mishra @ 2022-06-16 10:14 UTC (permalink / raw)
To: Daniel P. Berrangé, Dr. David Alan Gilbert
Cc: Het Gala, qemu-devel, quintela, pbonzini, armbru, eblake
On 16/06/22 1:46 pm, Daniel P. Berrangé wrote:
> On Wed, Jun 15, 2022 at 08:14:26PM +0100, Dr. David Alan Gilbert wrote:
>> * Daniel P. Berrangé (berrange@redhat.com) wrote:
>>> On Fri, Jun 10, 2022 at 05:58:31PM +0530, manish.mishra wrote:
>>>> On 09/06/22 9:17 pm, Daniel P. Berrangé wrote:
>>>>> On Thu, Jun 09, 2022 at 07:33:01AM +0000, Het Gala wrote:
>>>>>> As of now, the multi-FD feature supports connection over the default network
>>>>>> only. This Patchset series is a Qemu side implementation of providing multiple
>>>>>> interfaces support for multi-FD. This enables us to fully utilize dedicated or
>>>>>> multiple NICs in case bonding of NICs is not possible.
>>>>>>
>>>>>>
>>>>>> Introduction
>>>>>> -------------
>>>>>> Multi-FD Qemu implementation currently supports connection only on the default
>>>>>> network. This forbids us from advantages like:
>>>>>> - Separating VM live migration traffic from the default network.
>>>> Hi Daniel,
>>>>
>>>> I totally understand your concern around this approach increasing compexity inside qemu,
>>>>
>>>> when similar things can be done with NIC teaming. But we thought this approach provides
>>>>
>>>> much more flexibility to user in few cases like.
>>>>
>>>> 1. We checked our customer data, almost all of the host had multiple NIC, but LACP support
>>>>
>>>> in their setups was very rare. So for those cases this approach can help in utilise multiple
>>>>
>>>> NICs as teaming is not possible there.
>>> AFAIK, LACP is not required in order to do link aggregation with Linux.
>>> Traditional Linux bonding has no special NIC hardware or switch requirements,
>>> so LACP is merely a "nice to have" in order to simplify some aspects.
>>>
>>> IOW, migration with traffic spread across multiple NICs is already
>>> possible AFAICT.
>> Are we sure that works with multifd? I've seen a lot of bonding NIC
>> setups which spread based on a hash of source/destination IP and port
>> numbers; given that we use the same dest port and IP at the moment what
>> happens in reality? That hashing can be quite delicate for high
>> bandwidth single streams.
> The simplest Linux bonding mode does per-packet round-robin across
> NICs, so traffic from the collection of multifd connections should
> fill up all the NICs in the bond. There are of course other modes
> which may be sub-optimal for the reasons you describe. Which mode
> to pick depends on the type of service traffic patterns you're
> aiming to balance.
My understanding on networking is not good enough so apologies in advance if something
does not make sense. As per my understanding it is easy to do load balancing on sender
side because we have full control where to send packet but complicated on receive side
if we do not have LACP like support. I see there are some teaming technique which does
load balancing of incoming traffic by possibly sending different slaves mac address on arp
requests but that does not work for our use case and may require a complicated setup
for proper usage. Our use case can be something like this e.g. both source and destination
has 2-2 NICs of 10Gbps each and we want to get a throughput of 20Gbps for live migration.
thanks
Manish Mishra
>
>>>>>> Multi-interface with Multi-FD
>>>>>> -----------------------------
>>>>>> Multiple-interface support over basic multi-FD has been implemented in the
>>>>>> patches. Advantages of this implementation are:
>>>>>> - Able to separate live migration traffic from default network interface by
>>>>>> creating multiFD channels on ip addresses of multiple non-default interfaces.
>>>>>> - Can optimize the number of multi-FD channels on a particular interface
>>>>>> depending upon the network bandwidth limit on a particular interface.
>>>>> Manually assigning individual channels to different NICs is a pretty
>>>>> inefficient way to optimizing traffic. Feels like you could easily get
>>>>> into a situation where one NIC ends up idle while the other is busy,
>>>>> especially if the traffic patterns are different. For example with
>>>>> post-copy there's an extra channel for OOB async page requests, and
>>>>> its far from clear that manually picking NICs per chanel upfront is
>>>>> going work for that. The kernel can continually dynamically balance
>>>>> load on the fly and so do much better than any static mapping QEMU
>>>>> tries to apply, especially if there are multiple distinct QEMU's
>>>>> competing for bandwidth.
>>>>>
>>>> Yes, Daniel current solution is only for pre-copy. As with postcopy
>>>> multiFD is not yet supported but in future we can extend it for postcopy
>> I had been thinking about explicit selection of network device for NUMA
>> use though; ideally I'd like to be able to associate a set of multifd
>> threads to each NUMA node, and then associate a NIC with that set of
>> threads; so that the migration happens down the NIC that's on the node
>> the RAM is on. On a really good day you'd have one NIC per top level
>> NUMA node.
> Now that's an interesting idea, and not one that can be dealt with
> by bonding, since the network layer won't be aware of the NUMA
> affinity constraints.
>
>
> With regards,
> Daniel
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 0/4] Multiple interface support on top of Multi-FD
2022-06-16 8:27 ` Daniel P. Berrangé
@ 2022-06-16 15:50 ` Dr. David Alan Gilbert
2022-06-21 16:16 ` manish.mishra
0 siblings, 1 reply; 34+ messages in thread
From: Dr. David Alan Gilbert @ 2022-06-16 15:50 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: manish.mishra, Het Gala, qemu-devel, quintela, pbonzini, armbru, eblake
* Daniel P. Berrangé (berrange@redhat.com) wrote:
> On Wed, Jun 15, 2022 at 05:43:28PM +0100, Daniel P. Berrangé wrote:
> > On Fri, Jun 10, 2022 at 05:58:31PM +0530, manish.mishra wrote:
> > >
> > > On 09/06/22 9:17 pm, Daniel P. Berrangé wrote:
> > > > On Thu, Jun 09, 2022 at 07:33:01AM +0000, Het Gala wrote:
> > > > > As of now, the multi-FD feature supports connection over the default network
> > > > > only. This Patchset series is a Qemu side implementation of providing multiple
> > > > > interfaces support for multi-FD. This enables us to fully utilize dedicated or
> > > > > multiple NICs in case bonding of NICs is not possible.
> > > > >
> > > > >
> > > > > Introduction
> > > > > -------------
> > > > > Multi-FD Qemu implementation currently supports connection only on the default
> > > > > network. This forbids us from advantages like:
> > > > > - Separating VM live migration traffic from the default network.
> > >
> > > Hi Daniel,
> > >
> > > I totally understand your concern around this approach increasing compexity inside qemu,
> > >
> > > when similar things can be done with NIC teaming. But we thought this approach provides
> > >
> > > much more flexibility to user in few cases like.
> > >
> > > 1. We checked our customer data, almost all of the host had multiple NIC, but LACP support
> > >
> > > in their setups was very rare. So for those cases this approach can help in utilise multiple
> > >
> > > NICs as teaming is not possible there.
> >
> > AFAIK, LACP is not required in order to do link aggregation with Linux.
> > Traditional Linux bonding has no special NIC hardware or switch requirements,
> > so LACP is merely a "nice to have" in order to simplify some aspects.
> >
> > IOW, migration with traffic spread across multiple NICs is already
> > possible AFAICT.
> >
> > I can understand that some people may not have actually configured
> > bonding on their hosts, but it is not unreasonable to request that
> > they do so, if they want to take advantage fo aggrated bandwidth.
> >
> > It has the further benefit that it will be fault tolerant. With
> > this proposal if any single NIC has a problem, the whole migration
> > will get stuck. With kernel level bonding, if any single NIC haus
> > a problem, it'll get offlined by the kernel and migration will
> > continue to work across remaining active NICs.
> >
> > > 2. We have seen requests recently to separate out traffic of storage, VM netwrok, migration
> > >
> > > over different vswitch which can be backed by 1 or more NICs as this give better
> > >
> > > predictability and assurance. So host with multiple ips/vswitches can be very common
> > >
> > > environment. In this kind of enviroment this approach gives per vm or migration level
> > >
> > > flexibilty, like for critical VM we can still use bandwidth from all available vswitch/interface
> > >
> > > but for normal VM they can keep live migration only on dedicated NICs without changing
> > >
> > > complete host network topology.
> > >
> > > At final we want it to be something like this [<ip-pair>, <multiFD-channels>, <bandwidth_control>]
> > >
> > > to provide bandwidth_control per interface.
> >
> > Again, it is already possible to separate migration traffic from storage
> > traffic, from other network traffic. The target IP given will influence
> > which NIC is used based on routing table and I know this is already
> > done widely with OpenStack deployments.
>
> Actually I should clarify this is only practical if the two NICs are
> using different IP subnets, otherwise routing rules are not viable.
> So needing to set source IP would be needed to select between a pair
> of NICs on the same IP subnet.
Yeh so I think that's one reason that the idea in this series is OK
(together with the idea for the NUMA stuff) and I suspect there are
other cases as well.
Dave
> Previous usage I've seen has always setup fully distinct IP subnets
> for generic vs storage vs migration network traffic.
>
> With regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair
2022-06-09 7:33 ` [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair Het Gala
@ 2022-06-16 17:26 ` Dr. David Alan Gilbert
2022-07-13 8:08 ` Het Gala
2022-07-13 12:54 ` Claudio Fontana
2022-07-18 8:35 ` Markus Armbruster
1 sibling, 2 replies; 34+ messages in thread
From: Dr. David Alan Gilbert @ 2022-06-16 17:26 UTC (permalink / raw)
To: Het Gala, cfontana
Cc: qemu-devel, quintela, pbonzini, berrange, armbru, eblake, Manish Mishra
* Het Gala (het.gala@nutanix.com) wrote:
> i) Modified the format of the qemu monitor command : 'migrate' by adding a list,
> each element in the list consists of multi-FD connection parameters: source
> and destination uris and of the number of multi-fd channels between each pair.
>
> ii) Information of all multi-FD connection parameters’ list, length of the list
> and total number of multi-fd channels for all the connections together is
> stored in ‘OutgoingArgs’ struct.
>
> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
> Signed-off-by: Het Gala <het.gala@nutanix.com>
> ---
> include/qapi/util.h | 9 ++++++++
> migration/migration.c | 47 ++++++++++++++++++++++++++++++--------
> migration/socket.c | 53 ++++++++++++++++++++++++++++++++++++++++---
> migration/socket.h | 17 +++++++++++++-
> monitor/hmp-cmds.c | 22 ++++++++++++++++--
> qapi/migration.json | 43 +++++++++++++++++++++++++++++++----
> 6 files changed, 170 insertions(+), 21 deletions(-)
>
> diff --git a/include/qapi/util.h b/include/qapi/util.h
> index 81a2b13a33..3041feb3d9 100644
> --- a/include/qapi/util.h
> +++ b/include/qapi/util.h
> @@ -56,4 +56,13 @@ int parse_qapi_name(const char *name, bool complete);
> (tail) = &(*(tail))->next; \
> } while (0)
>
> +#define QAPI_LIST_LENGTH(list) ({ \
> + int _len = 0; \
> + typeof(list) _elem; \
> + for (_elem = list; _elem != NULL; _elem = _elem->next) { \
> + _len++; \
> + } \
> + _len; \
> +})
> +
> #endif
This looks like it should be a separate patch to me (and perhaps size_t
for len?)
> diff --git a/migration/migration.c b/migration/migration.c
> index 31739b2af9..c408175aeb 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -2328,13 +2328,14 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,
> return true;
> }
>
> -void qmp_migrate(const char *uri, bool has_blk, bool blk,
> +void qmp_migrate(const char *uri, bool has_multi_fd_uri_list,
> + MigrateUriParameterList *cap, bool has_blk, bool blk,
> bool has_inc, bool inc, bool has_detach, bool detach,
> bool has_resume, bool resume, Error **errp)
> {
> Error *local_err = NULL;
> MigrationState *s = migrate_get_current();
> - const char *p = NULL;
> + const char *dst_ptr = NULL;
>
> if (!migrate_prepare(s, has_blk && blk, has_inc && inc,
> has_resume && resume, errp)) {
> @@ -2348,20 +2349,46 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
> }
> }
>
> + /*
> + * In case of Multi-FD migration parameters, if uri is provided,
I think you mean 'if uri list is provided'
> + * supports only tcp network protocol.
> + */
> + if (has_multi_fd_uri_list) {
> + int length = QAPI_LIST_LENGTH(cap);
> + init_multifd_array(length);
> + for (int i = 0; i < length; i++) {
> + const char *p1 = NULL, *p2 = NULL;
Keep these as ps/pd to make it clear which is source and dest.
> + const char *multifd_dst_uri = cap->value->destination_uri;
> + const char *multifd_src_uri = cap->value->source_uri;
> + uint8_t multifd_channels = cap->value->multifd_channels;
> + if (!strstart(multifd_dst_uri, "tcp:", &p1) ||
> + !strstart(multifd_src_uri, "tcp:", &p2)) {
I've copied in Claudio Fontana; Claudio is fighting to make snapshots
faster and has been playing with various multithread schemes for multifd
with files and fd's; perhaps the syntax you're proposing doesn't need
to be limited to tcp.
> + error_setg(errp, "multi-fd destination and multi-fd source "
> + "uri, both should be present and follows tcp protocol only");
> + break;
> + } else {
> + store_multifd_migration_params(p1 ? p1 : multifd_dst_uri,
> + p2 ? p2 : multifd_src_uri,
> + multifd_channels, i, &local_err);
> + }
> + cap = cap->next;
> + }
> + }
> +
> migrate_protocol_allow_multi_channels(false);
> - if (strstart(uri, "tcp:", &p) ||
> + if (strstart(uri, "tcp:", &dst_ptr) ||
> strstart(uri, "unix:", NULL) ||
> strstart(uri, "vsock:", NULL)) {
> migrate_protocol_allow_multi_channels(true);
> - socket_start_outgoing_migration(s, p ? p : uri, &local_err);
> + socket_start_outgoing_migration(s, dst_ptr ? dst_ptr : uri, &local_err);
> #ifdef CONFIG_RDMA
> - } else if (strstart(uri, "rdma:", &p)) {
> - rdma_start_outgoing_migration(s, p, &local_err);
> + } else if (strstart(uri, "rdma:", &dst_ptr)) {
> + rdma_start_outgoing_migration(s, dst_ptr, &local_err);
> #endif
> - } else if (strstart(uri, "exec:", &p)) {
> - exec_start_outgoing_migration(s, p, &local_err);
> - } else if (strstart(uri, "fd:", &p)) {
> - fd_start_outgoing_migration(s, p, &local_err);
> + } else if (strstart(uri, "exec:", &dst_ptr)) {
> + exec_start_outgoing_migration(s, dst_ptr, &local_err);
> + } else if (strstart(uri, "fd:", &dst_ptr)) {
> + fd_start_outgoing_migration(s, dst_ptr, &local_err);
> } else {
> if (!(has_resume && resume)) {
> yank_unregister_instance(MIGRATION_YANK_INSTANCE);
> diff --git a/migration/socket.c b/migration/socket.c
> index 4fd5e85f50..7ca6af8cca 100644
> --- a/migration/socket.c
> +++ b/migration/socket.c
> @@ -32,6 +32,17 @@ struct SocketOutgoingArgs {
> SocketAddress *saddr;
> } outgoing_args;
>
> +struct SocketArgs {
> + struct SrcDestAddr data;
'data' is an odd name; 'addresses' perhaps?
> + uint8_t multifd_channels;
> +};
> +
> +struct OutgoingMigrateParams {
> + struct SocketArgs *socket_args;
> + size_t length;
> + uint64_t total_multifd_channel;
> +} outgoing_migrate_params;
> +
> void socket_send_channel_create(QIOTaskFunc f, void *data)
> {
> QIOChannelSocket *sioc = qio_channel_socket_new();
> @@ -47,6 +58,14 @@ int socket_send_channel_destroy(QIOChannel *send)
> qapi_free_SocketAddress(outgoing_args.saddr);
> outgoing_args.saddr = NULL;
> }
> +
> + if (outgoing_migrate_params.socket_args != NULL) {
> + g_free(outgoing_migrate_params.socket_args);
> + outgoing_migrate_params.socket_args = NULL;
I think g_free is safe on NULL; so I think you can just do this without
the if.
> + }
> + if (outgoing_migrate_params.length) {
Does that ever differ from the != NULL test ?
I think you can always just set this to 0 without the test.
> + outgoing_migrate_params.length = 0;
> + }
> return 0;
> }
>
> @@ -117,13 +136,41 @@ socket_start_outgoing_migration_internal(MigrationState *s,
> }
>
> void socket_start_outgoing_migration(MigrationState *s,
> - const char *str,
> + const char *dst_str,
> Error **errp)
> {
> Error *err = NULL;
> - SocketAddress *saddr = socket_parse(str, &err);
> + SocketAddress *dst_saddr = socket_parse(dst_str, &err);
> + if (!err) {
> + socket_start_outgoing_migration_internal(s, dst_saddr, &err);
> + }
> + error_propagate(errp, err);
> +}
> +
> +void init_multifd_array(int length)
> +{
> + outgoing_migrate_params.socket_args = g_new0(struct SocketArgs, length);
> + outgoing_migrate_params.length = length;
> + outgoing_migrate_params.total_multifd_channel = 0;
> +}
> +
> +void store_multifd_migration_params(const char *dst_uri,
> + const char *src_uri,
> + uint8_t multifd_channels,
> + int idx, Error **errp)
> +{
> + Error *err = NULL;
> + SocketAddress *src_addr = NULL;
> + SocketAddress *dst_addr = socket_parse(dst_uri, &err);
> + if (src_uri) {
> + src_addr = socket_parse(src_uri, &err);
> + }
> if (!err) {
> - socket_start_outgoing_migration_internal(s, saddr, &err);
> + outgoing_migrate_params.socket_args[idx].data.dst_addr = dst_addr;
> + outgoing_migrate_params.socket_args[idx].data.src_addr = src_addr;
> + outgoing_migrate_params.socket_args[idx].multifd_channels
> + = multifd_channels;
> + outgoing_migrate_params.total_multifd_channel += multifd_channels;
> }
> error_propagate(errp, err);
> }
> diff --git a/migration/socket.h b/migration/socket.h
> index 891dbccceb..bba7f177fe 100644
> --- a/migration/socket.h
> +++ b/migration/socket.h
> @@ -19,12 +19,27 @@
>
> #include "io/channel.h"
> #include "io/task.h"
> +#include "migration.h"
> +
> +/* info regarding destination and source uri */
> +struct SrcDestAddr {
> + SocketAddress *dst_addr;
> + SocketAddress *src_addr;
> +};
>
> void socket_send_channel_create(QIOTaskFunc f, void *data);
> int socket_send_channel_destroy(QIOChannel *send);
>
> void socket_start_incoming_migration(const char *str, Error **errp);
>
> -void socket_start_outgoing_migration(MigrationState *s, const char *str,
> +void socket_start_outgoing_migration(MigrationState *s, const char *dst_str,
> Error **errp);
> +
> +int multifd_list_length(MigrateUriParameterList *list);
> +
> +void init_multifd_array(int length);
> +
> +void store_multifd_migration_params(const char *dst_uri, const char *src_uri,
> + uint8_t multifd_channels, int idx,
> + Error **erp);
> #endif
> diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
> index 622c783c32..2db539016a 100644
> --- a/monitor/hmp-cmds.c
> +++ b/monitor/hmp-cmds.c
> @@ -56,6 +56,9 @@
> #include "migration/snapshot.h"
> #include "migration/misc.h"
>
> +/* Default number of multi-fd channels */
> +#define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2
> +
> #ifdef CONFIG_SPICE
> #include <spice/enums.h>
> #endif
> @@ -1574,10 +1577,25 @@ void hmp_migrate(Monitor *mon, const QDict *qdict)
> bool inc = qdict_get_try_bool(qdict, "inc", false);
> bool resume = qdict_get_try_bool(qdict, "resume", false);
> const char *uri = qdict_get_str(qdict, "uri");
> +
> + const char *src_uri = qdict_get_str(qdict, "source-uri");
> + const char *dst_uri = qdict_get_str(qdict, "destination-uri");
> + uint8_t multifd_channels = qdict_get_try_int(qdict, "multifd-channels",
> + DEFAULT_MIGRATE_MULTIFD_CHANNELS);
> Error *err = NULL;
> + MigrateUriParameterList *caps = NULL;
> + MigrateUriParameter *value;
> +
> + value = g_malloc0(sizeof(*value));
> + value->source_uri = (char *)src_uri;
> + value->destination_uri = (char *)dst_uri;
> + value->multifd_channels = multifd_channels;
> + QAPI_LIST_PREPEND(caps, value);
> +
> + qmp_migrate(uri, !!caps, caps, !!blk, blk, !!inc,
> + inc, false, false, true, resume, &err);
> + qapi_free_MigrateUriParameterList(caps);
>
> - qmp_migrate(uri, !!blk, blk, !!inc, inc,
> - false, false, true, resume, &err);
> if (hmp_handle_error(mon, err)) {
> return;
> }
Please split the HMP changes into a separate patch.
> diff --git a/qapi/migration.json b/qapi/migration.json
> index 6130cd9fae..fb259d626b 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -1454,12 +1454,38 @@
> ##
> { 'command': 'migrate-continue', 'data': {'state': 'MigrationStatus'} }
>
> +##
> +# @MigrateUriParameter:
> +#
> +# Information regarding which source interface is connected to which
> +# destination interface and number of multifd channels over each interface.
> +#
> +# @source-uri: the Uniform Resource Identifier of the source VM.
> +# Default port number is 0.
> +#
> +# @destination-uri: the Uniform Resource Identifier of the destination VM
I would just say 'uri' rather than spelling it out.
> +# @multifd-channels: number of parallel multifd channels used to migrate data
> +# for specific source-uri and destination-uri. Default value
> +# in this case is 2 (Since 4.0)
7.1 at the moment.
> +#
> +##
> +{ 'struct' : 'MigrateUriParameter',
> + 'data' : { 'source-uri' : 'str',
> + 'destination-uri' : 'str',
> + '*multifd-channels' : 'uint8'} }
OK, so much higher level question - why do we specify both URIs on
each end? Is it just the source that is used on the source side to say
which NIC to route down? On the destination side I guess there's no
need?
Do we have some rule about needing to specify enough channels for all
the multifd channels we specify (i.e. if we specify 4 multifd channels
in the migration parameter do we have to supply 4 channels here?)
What happens with say Peter's preemption channel?
Is there some logical ordering rule; i.e. if we were to start ordering
particular multifd threads, then can we say that we allocate these
channels in the same order as this list?
> ##
> # @migrate:
> #
> # Migrates the current running guest to another Virtual Machine.
> #
> # @uri: the Uniform Resource Identifier of the destination VM
> +# for migration thread
> +#
> +# @multi-fd-uri-list: list of pair of source and destination VM Uniform
> +# Resource Identifiers with number of multifd-channels
> +# for each pair
> #
> # @blk: do block migration (full disk copy)
> #
> @@ -1479,20 +1505,27 @@
> # 1. The 'query-migrate' command should be used to check migration's progress
> # and final result (this information is provided by the 'status' member)
> #
> -# 2. All boolean arguments default to false
> +# 2. The uri argument should have the Uniform Resource Identifier of default
> +# destination VM. This connection will be bound to default network
> +#
> +# 3. All boolean arguments default to false
> #
> -# 3. The user Monitor's "detach" argument is invalid in QMP and should not
> +# 4. The user Monitor's "detach" argument is invalid in QMP and should not
> # be used
> #
> # Example:
> #
> -# -> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
> +# -> { "execute": "migrate",
> +# "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
> +# "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
> +# "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
> +# "destination-uri": "tcp:11.0.0.0:7789", "multifd-channels": 5} ] } }
> # <- { "return": {} }
> #
> ##
> { 'command': 'migrate',
> - 'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool',
> - '*detach': 'bool', '*resume': 'bool' } }
> + 'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateUriParameter'], '*blk': 'bool',
> + '*inc': 'bool', '*detach': 'bool', '*resume': 'bool' } }
>
> ##
> # @migrate-incoming:
> --
> 2.22.3
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 0/4] Multiple interface support on top of Multi-FD
2022-06-16 10:14 ` manish.mishra
@ 2022-06-16 17:32 ` Daniel P. Berrangé
0 siblings, 0 replies; 34+ messages in thread
From: Daniel P. Berrangé @ 2022-06-16 17:32 UTC (permalink / raw)
To: manish.mishra
Cc: Dr. David Alan Gilbert, Het Gala, qemu-devel, quintela, pbonzini,
armbru, eblake
On Thu, Jun 16, 2022 at 03:44:09PM +0530, manish.mishra wrote:
>
> On 16/06/22 1:46 pm, Daniel P. Berrangé wrote:
> > On Wed, Jun 15, 2022 at 08:14:26PM +0100, Dr. David Alan Gilbert wrote:
> > > * Daniel P. Berrangé (berrange@redhat.com) wrote:
> > > > On Fri, Jun 10, 2022 at 05:58:31PM +0530, manish.mishra wrote:
> > > > > On 09/06/22 9:17 pm, Daniel P. Berrangé wrote:
> > > > > > On Thu, Jun 09, 2022 at 07:33:01AM +0000, Het Gala wrote:
> > > > > > > As of now, the multi-FD feature supports connection over the default network
> > > > > > > only. This Patchset series is a Qemu side implementation of providing multiple
> > > > > > > interfaces support for multi-FD. This enables us to fully utilize dedicated or
> > > > > > > multiple NICs in case bonding of NICs is not possible.
> > > > > > >
> > > > > > >
> > > > > > > Introduction
> > > > > > > -------------
> > > > > > > Multi-FD Qemu implementation currently supports connection only on the default
> > > > > > > network. This forbids us from advantages like:
> > > > > > > - Separating VM live migration traffic from the default network.
> > > > > Hi Daniel,
> > > > >
> > > > > I totally understand your concern around this approach increasing compexity inside qemu,
> > > > >
> > > > > when similar things can be done with NIC teaming. But we thought this approach provides
> > > > >
> > > > > much more flexibility to user in few cases like.
> > > > >
> > > > > 1. We checked our customer data, almost all of the host had multiple NIC, but LACP support
> > > > >
> > > > > in their setups was very rare. So for those cases this approach can help in utilise multiple
> > > > >
> > > > > NICs as teaming is not possible there.
> > > > AFAIK, LACP is not required in order to do link aggregation with Linux.
> > > > Traditional Linux bonding has no special NIC hardware or switch requirements,
> > > > so LACP is merely a "nice to have" in order to simplify some aspects.
> > > >
> > > > IOW, migration with traffic spread across multiple NICs is already
> > > > possible AFAICT.
> > > Are we sure that works with multifd? I've seen a lot of bonding NIC
> > > setups which spread based on a hash of source/destination IP and port
> > > numbers; given that we use the same dest port and IP at the moment what
> > > happens in reality? That hashing can be quite delicate for high
> > > bandwidth single streams.
> > The simplest Linux bonding mode does per-packet round-robin across
> > NICs, so traffic from the collection of multifd connections should
> > fill up all the NICs in the bond. There are of course other modes
> > which may be sub-optimal for the reasons you describe. Which mode
> > to pick depends on the type of service traffic patterns you're
> > aiming to balance.
>
> My understanding on networking is not good enough so apologies in advance if something
> does not make sense. As per my understanding it is easy to do load balancing on sender
> side because we have full control where to send packet but complicated on receive side
> if we do not have LACP like support. I see there are some teaming technique which does
> load balancing of incoming traffic by possibly sending different slaves mac address on arp
> requests but that does not work for our use case and may require a complicated setup
> for proper usage. Our use case can be something like this e.g. both source and destination
> has 2-2 NICs of 10Gbps each and we want to get a throughput of 20Gbps for live migration.
I believe you are right. The Linux bonding will give us full 20 Gpbs
throughput on the transmit side, without any hardware dependancies.
On the receive side, however, there is a dependancy on the network
switch to be able to balance the traffic it forwards to the target.
This is fairly common in switches, but the typical policies based on
hashing the MAC/IP addr will not be sufficient in this case.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 3/4] Establishing connection between any non-default source and destination pair
2022-06-09 7:33 ` [PATCH 3/4] Establishing connection between any non-default source and destination pair Het Gala
@ 2022-06-16 17:39 ` Daniel P. Berrangé
2022-06-21 16:09 ` manish.mishra
[not found] ` <54ca00c7-a108-11e3-1c8d-8771798aed6a@nutanix.com>
0 siblings, 2 replies; 34+ messages in thread
From: Daniel P. Berrangé @ 2022-06-16 17:39 UTC (permalink / raw)
To: Het Gala
Cc: qemu-devel, quintela, dgilbert, pbonzini, armbru, eblake, Manish Mishra
On Thu, Jun 09, 2022 at 07:33:04AM +0000, Het Gala wrote:
> i) Binding of the socket to source ip address and port on the non-default
> interface has been implemented for multi-FD connection, which was not
> necessary earlier because the binding was on the default interface itself.
>
> ii) Created an end to end connection between all multi-FD source and
> destination pairs.
>
> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
> Signed-off-by: Het Gala <het.gala@nutanix.com>
> ---
> chardev/char-socket.c | 4 +-
> include/io/channel-socket.h | 26 ++++++-----
> include/qemu/sockets.h | 6 ++-
> io/channel-socket.c | 50 ++++++++++++++------
> migration/socket.c | 15 +++---
> nbd/client-connection.c | 2 +-
> qemu-nbd.c | 4 +-
> scsi/pr-manager-helper.c | 1 +
> tests/unit/test-char.c | 8 ++--
> tests/unit/test-io-channel-socket.c | 4 +-
> tests/unit/test-util-sockets.c | 16 +++----
> ui/input-barrier.c | 2 +-
> ui/vnc.c | 3 +-
> util/qemu-sockets.c | 71 ++++++++++++++++++++---------
> 14 files changed, 135 insertions(+), 77 deletions(-)
>
> diff --git a/chardev/char-socket.c b/chardev/char-socket.c
> index dc4e218eeb..f3725238c5 100644
> --- a/chardev/char-socket.c
> +++ b/chardev/char-socket.c
> @@ -932,7 +932,7 @@ static int tcp_chr_connect_client_sync(Chardev *chr, Error **errp)
> QIOChannelSocket *sioc = qio_channel_socket_new();
> tcp_chr_change_state(s, TCP_CHARDEV_STATE_CONNECTING);
> tcp_chr_set_client_ioc_name(chr, sioc);
> - if (qio_channel_socket_connect_sync(sioc, s->addr, errp) < 0) {
> + if (qio_channel_socket_connect_sync(sioc, s->addr, NULL, errp) < 0) {
> tcp_chr_change_state(s, TCP_CHARDEV_STATE_DISCONNECTED);
> object_unref(OBJECT(sioc));
> return -1;
> @@ -1120,7 +1120,7 @@ static void tcp_chr_connect_client_task(QIOTask *task,
> SocketAddress *addr = opaque;
> Error *err = NULL;
>
> - qio_channel_socket_connect_sync(ioc, addr, &err);
> + qio_channel_socket_connect_sync(ioc, addr, NULL, &err);
>
> qio_task_set_error(task, err);
> }
> diff --git a/include/io/channel-socket.h b/include/io/channel-socket.h
> index 513c428fe4..59d5b1b349 100644
> --- a/include/io/channel-socket.h
> +++ b/include/io/channel-socket.h
> @@ -83,41 +83,45 @@ qio_channel_socket_new_fd(int fd,
> /**
> * qio_channel_socket_connect_sync:
> * @ioc: the socket channel object
> - * @addr: the address to connect to
> + * @dst_addr: the destination address to connect to
> + * @src_addr: the source address to be connected
> * @errp: pointer to a NULL-initialized error object
> *
> - * Attempt to connect to the address @addr. This method
> - * will run in the foreground so the caller will not regain
> - * execution control until the connection is established or
> + * Attempt to connect to the address @dst_addr with @src_addr.
> + * This method will run in the foreground so the caller will not
> + * regain execution control until the connection is established or
> * an error occurs.
> */
> int qio_channel_socket_connect_sync(QIOChannelSocket *ioc,
> - SocketAddress *addr,
> + SocketAddress *dst_addr,
> + SocketAddress *src_addr,
> Error **errp);
>
> /**
> * qio_channel_socket_connect_async:
> * @ioc: the socket channel object
> - * @addr: the address to connect to
> + * @dst_addr: the destination address to connect to
> * @callback: the function to invoke on completion
> * @opaque: user data to pass to @callback
> * @destroy: the function to free @opaque
> * @context: the context to run the async task. If %NULL, the default
> * context will be used.
> + * @src_addr: the source address to be connected
> *
> - * Attempt to connect to the address @addr. This method
> - * will run in the background so the caller will regain
> + * Attempt to connect to the address @dst_addr with the @src_addr.
> + * This method will run in the background so the caller will regain
> * execution control immediately. The function @callback
> - * will be invoked on completion or failure. The @addr
> + * will be invoked on completion or failure. The @dst_addr
> * parameter will be copied, so may be freed as soon
> * as this function returns without waiting for completion.
> */
> void qio_channel_socket_connect_async(QIOChannelSocket *ioc,
> - SocketAddress *addr,
> + SocketAddress *dst_addr,
> QIOTaskFunc callback,
> gpointer opaque,
> GDestroyNotify destroy,
> - GMainContext *context);
> + GMainContext *context,
> + SocketAddress *src_addr);
Lets avoid modifying these two methods, and thus avoid
updating countless callers.
In common with typical pattern in QIO code, lets add
a second variant
qio_channel_socket_connect_full
qio_channel_socket_connect_full_async
which has the extra parameters, then make the existing
simpler methods call the new ones.
ie qio_channel_socket_connect should call
qio_channel_socket_connect_full, pasing NULL for the
src_addr.
> diff --git a/io/channel-socket.c b/io/channel-socket.c
> index dc9c165de1..f8746ad646 100644
> --- a/io/channel-socket.c
> +++ b/io/channel-socket.c
> @@ -36,6 +36,12 @@
>
> #define SOCKET_MAX_FDS 16
>
> +struct SrcDestAddress {
> + SocketAddress *dst_addr;
> + SocketAddress *src_addr;
> +};
Nitpick, just call this 'struct ConnectData'.
> SocketAddress *
> qio_channel_socket_get_local_address(QIOChannelSocket *ioc,
> Error **errp)
> @@ -145,13 +151,14 @@ qio_channel_socket_new_fd(int fd,
>
>
> int qio_channel_socket_connect_sync(QIOChannelSocket *ioc,
> - SocketAddress *addr,
> + SocketAddress *dst_addr,
> + SocketAddress *src_addr,
> Error **errp)
> {
> int fd;
>
> - trace_qio_channel_socket_connect_sync(ioc, addr);
> - fd = socket_connect(addr, errp);
> + trace_qio_channel_socket_connect_sync(ioc, dst_addr);
> + fd = socket_connect(dst_addr, src_addr, errp);
> if (fd < 0) {
> trace_qio_channel_socket_connect_fail(ioc);
> return -1;
> @@ -177,39 +184,56 @@ int qio_channel_socket_connect_sync(QIOChannelSocket *ioc,
> }
>
>
> +static void qio_channel_socket_worker_free(gpointer opaque)
> +{
> + struct SrcDestAddress *data = opaque;
> + if (!data) {
> + return;
> + }
> + qapi_free_SocketAddress(data->dst_addr);
> + qapi_free_SocketAddress(data->src_addr);
> + g_free(data);
> +}
> +
> +
> static void qio_channel_socket_connect_worker(QIOTask *task,
> gpointer opaque)
> {
> QIOChannelSocket *ioc = QIO_CHANNEL_SOCKET(qio_task_get_source(task));
> - SocketAddress *addr = opaque;
> + struct SrcDestAddress *data = opaque;
> Error *err = NULL;
>
> - qio_channel_socket_connect_sync(ioc, addr, &err);
> + qio_channel_socket_connect_sync(ioc, data->dst_addr, data->src_addr, &err);
>
> qio_task_set_error(task, err);
> }
>
>
> void qio_channel_socket_connect_async(QIOChannelSocket *ioc,
> - SocketAddress *addr,
> + SocketAddress *dst_addr,
> QIOTaskFunc callback,
> gpointer opaque,
> GDestroyNotify destroy,
> - GMainContext *context)
> + GMainContext *context,
> + SocketAddress *src_addr)
> {
> QIOTask *task = qio_task_new(
> OBJECT(ioc), callback, opaque, destroy);
> - SocketAddress *addrCopy;
> -
> - addrCopy = QAPI_CLONE(SocketAddress, addr);
> + struct SrcDestAddress *data = g_new0(struct SrcDestAddress, 1);
>
> + data->dst_addr = QAPI_CLONE(SocketAddress, dst_addr);
> + if (src_addr) {
> + data->src_addr = QAPI_CLONE(SocketAddress, src_addr);
> + } else {
> + data->src_addr = NULL;
> + }
> /* socket_connect() does a non-blocking connect(), but it
> * still blocks in DNS lookups, so we must use a thread */
> - trace_qio_channel_socket_connect_async(ioc, addr);
> + trace_qio_channel_socket_connect_async(ioc, dst_addr);
> qio_task_run_in_thread(task,
> qio_channel_socket_connect_worker,
> - addrCopy,
> - (GDestroyNotify)qapi_free_SocketAddress,
> + data,
> + qio_channel_socket_worker_free,
> context);
> }
>
> diff --git a/migration/socket.c b/migration/socket.c
> index 21e0983df2..d0cb7cc6a6 100644
> --- a/migration/socket.c
> +++ b/migration/socket.c
> @@ -47,7 +47,7 @@ void socket_send_channel_create(QIOTaskFunc f, void *data)
> {
> QIOChannelSocket *sioc = qio_channel_socket_new();
> qio_channel_socket_connect_async(sioc, outgoing_args.saddr,
> - f, data, NULL, NULL);
> + f, data, NULL, NULL, NULL);
> }
>
> int socket_send_channel_destroy(QIOChannel *send)
> @@ -110,7 +110,7 @@ out:
>
> static void
> socket_start_outgoing_migration_internal(MigrationState *s,
> - SocketAddress *saddr,
> + SocketAddress *dst_addr,
> Error **errp)
> {
> QIOChannelSocket *sioc = qio_channel_socket_new();
> @@ -118,20 +118,17 @@ socket_start_outgoing_migration_internal(MigrationState *s,
>
> data->s = s;
>
> - /* in case previous migration leaked it */
> - qapi_free_SocketAddress(outgoing_args.saddr);
> - outgoing_args.saddr = saddr;
> -
> - if (saddr->type == SOCKET_ADDRESS_TYPE_INET) {
> - data->hostname = g_strdup(saddr->u.inet.host);
> + if (dst_addr->type == SOCKET_ADDRESS_TYPE_INET) {
> + data->hostname = g_strdup(dst_addr->u.inet.host);
> }
>
> qio_channel_set_name(QIO_CHANNEL(sioc), "migration-socket-outgoing");
> qio_channel_socket_connect_async(sioc,
> - saddr,
> + dst_addr,
> socket_outgoing_migration,
> data,
> socket_connect_data_free,
> + NULL,
> NULL);
> }
>
> diff --git a/util/qemu-sockets.c b/util/qemu-sockets.c
> index 13b5b197f9..bbe0dc0ee0 100644
> --- a/util/qemu-sockets.c
> +++ b/util/qemu-sockets.c
> @@ -226,7 +226,7 @@ static int inet_listen_saddr(InetSocketAddress *saddr,
> return -1;
> }
>
> - memset(&ai,0, sizeof(ai));
> + memset(&ai,0,sizeof(ai));
Add whitespace before the '0', rather than removing it after.
> ai.ai_flags = AI_PASSIVE;
> if (saddr->has_numeric && saddr->numeric) {
> ai.ai_flags |= AI_NUMERICHOST | AI_NUMERICSERV;
> @@ -282,8 +282,8 @@ static int inet_listen_saddr(InetSocketAddress *saddr,
> e->ai_protocol = IPPROTO_MPTCP;
> }
> #endif
> - getnameinfo((struct sockaddr*)e->ai_addr,e->ai_addrlen,
> - uaddr,INET6_ADDRSTRLEN,uport,32,
> + getnameinfo((struct sockaddr *)e->ai_addr, e->ai_addrlen,
> + uaddr, INET6_ADDRSTRLEN, uport, 32,
> NI_NUMERICHOST | NI_NUMERICSERV);
Both thsi & the above whitespace changes should be a separate
patch from any functional changes
> @@ -371,8 +372,28 @@ static int inet_connect_addr(const InetSocketAddress *saddr,
> }
> socket_set_fast_reuse(sock);
>
> + /* to bind the socket if src_addr is available */
> +
> + if (src_addr) {
> + struct sockaddr_in servaddr;
> +
> + /* bind to a specific interface in the internet domain */
> + /* to make sure the sin_zero filed is cleared */
> + memset(&servaddr, 0, sizeof(servaddr));
> +
> + servaddr.sin_family = AF_INET;
> + servaddr.sin_addr.s_addr = inet_addr(src_addr->host);
We can't assume src_addr is a raw IPv4 address.
THis needs to go through getaddrinfo in the caller like the
dst address has done.
For sanity, we should also validate that there isn't a mismatch
of IPv4 vs IPv6 for thte src & dst addrs.
> + servaddr.sin_port = 0;
> +
> + if (bind(sock, (struct sockaddr *)&servaddr, sizeof(servaddr)) < 0) {
> + error_setg_errno(errp, errno, "Failed to bind socket");
> + return -1;
> + }
> + }
> +
> /* connect to peer */
> do {
> +
> rc = 0;
> if (connect(sock, addr->ai_addr, addr->ai_addrlen) < 0) {
> rc = -errno;
> @@ -380,8 +401,14 @@ static int inet_connect_addr(const InetSocketAddress *saddr,
> } while (rc == -EINTR);
>
> if (rc < 0) {
> - error_setg_errno(errp, errno, "Failed to connect to '%s:%s'",
> - saddr->host, saddr->port);
> + if (src_addr) {
> + error_setg_errno(errp, errno, "Failed to connect '%s:%s' to "
> + "'%s:%s'", dst_addr->host, dst_addr->port,
> + src_addr->host, src_addr->port);
> + } else {
> + error_setg_errno(errp, errno, "Failed to connect '%s:%s'",
> + dst_addr->host, dst_addr->port);
> + }
> closesocket(sock);
> return -1;
> }
> @@ -506,7 +534,7 @@ static int inet_dgram_saddr(InetSocketAddress *sraddr,
> Error *err = NULL;
>
> /* lookup peer addr */
> - memset(&ai,0, sizeof(ai));
> + memset(&ai, 0, sizeof(ai));
> ai.ai_flags = AI_CANONNAME | AI_V4MAPPED | AI_ADDRCONFIG;
> ai.ai_family = inet_ai_family_from_address(sraddr, &err);
> ai.ai_socktype = SOCK_DGRAM;
> @@ -533,7 +561,7 @@ static int inet_dgram_saddr(InetSocketAddress *sraddr,
> }
>
> /* lookup local addr */
> - memset(&ai,0, sizeof(ai));
> + memset(&ai, 0, sizeof(ai));
> ai.ai_flags = AI_PASSIVE;
> ai.ai_family = peer->ai_family;
> ai.ai_socktype = SOCK_DGRAM;
> @@ -574,7 +602,7 @@ static int inet_dgram_saddr(InetSocketAddress *sraddr,
> }
>
> /* connect to peer */
> - if (connect(sock,peer->ai_addr,peer->ai_addrlen) < 0) {
> + if (connect(sock, peer->ai_addr, peer->ai_addrlen) < 0) {
> error_setg_errno(errp, errno, "Failed to connect to '%s:%s'",
> addr, port);
> goto err;
More whitespace changes for a separate patch
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 2/4] Adding multi-interface support for multi-FD on destination side
2022-06-09 7:33 ` [PATCH 2/4] Adding multi-interface support for multi-FD on destination side Het Gala
@ 2022-06-16 18:40 ` Dr. David Alan Gilbert
2022-07-13 14:36 ` Het Gala
0 siblings, 1 reply; 34+ messages in thread
From: Dr. David Alan Gilbert @ 2022-06-16 18:40 UTC (permalink / raw)
To: Het Gala
Cc: qemu-devel, quintela, pbonzini, berrange, armbru, eblake, Manish Mishra
* Het Gala (het.gala@nutanix.com) wrote:
> i) Modified the format of qemu monitor command: ‘migrate-incoming’ by adding
> a list, each element in the list is to open listeners with a given number
> of multiFD channels.
>
> ii) Qemu starts with -incoming flag defer and -multi-fd-incoming defer to
> allow the modified 'migrate-incoming' command to be used.
>
> iii) Format for -multi-fd-incoming flag as a comma separated string has been
> added with each substring containing listener socket address and number
> of sockets to open.
>
> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
> Signed-off-by: Het Gala <het.gala@nutanix.com>
> ---
> include/qapi/util.h | 1 +
> migration/migration.c | 149 ++++++++++++++++++++++++++++++++++++------
> migration/migration.h | 2 +
> migration/socket.c | 11 ++--
> migration/socket.h | 3 +-
> monitor/hmp-cmds.c | 42 ++++--------
> qapi/migration.json | 43 ++++++++++--
> qapi/qapi-util.c | 27 ++++++++
> qemu-options.hx | 18 +++++
> softmmu/vl.c | 30 ++++++++-
> 10 files changed, 265 insertions(+), 61 deletions(-)
>
> diff --git a/include/qapi/util.h b/include/qapi/util.h
> index 3041feb3d9..88fb2270db 100644
> --- a/include/qapi/util.h
> +++ b/include/qapi/util.h
> @@ -29,6 +29,7 @@ bool qapi_bool_parse(const char *name, const char *value, bool *obj,
> Error **errp);
>
> int parse_qapi_name(const char *name, bool complete);
> +struct strList *strList_from_string(const char *in, char c);
>
> /*
> * For any GenericList @list, insert @element at the front.
> diff --git a/migration/migration.c b/migration/migration.c
> index c408175aeb..9b0ad732e7 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -477,28 +477,39 @@ void migrate_add_address(SocketAddress *address)
> QAPI_CLONE(SocketAddress, address));
> }
>
> -static void qemu_start_incoming_migration(const char *uri, Error **errp)
> +static void qemu_start_incoming_migration(const char *uri, uint8_t number,
> + int idx, Error **errp)
> {
> const char *p = NULL;
>
> - migrate_protocol_allow_multi_channels(false); /* reset it anyway */
> - qapi_event_send_migration(MIGRATION_STATUS_SETUP);
> - if (strstart(uri, "tcp:", &p) ||
> - strstart(uri, "unix:", NULL) ||
> - strstart(uri, "vsock:", NULL)) {
> - migrate_protocol_allow_multi_channels(true);
> - socket_start_incoming_migration(p ? p : uri, errp);
> -#ifdef CONFIG_RDMA
> - } else if (strstart(uri, "rdma:", &p)) {
> - rdma_start_incoming_migration(p, errp);
> -#endif
> - } else if (strstart(uri, "exec:", &p)) {
> - exec_start_incoming_migration(p, errp);
> - } else if (strstart(uri, "fd:", &p)) {
> - fd_start_incoming_migration(p, errp);
> + if (number == 0) {
> + migrate_protocol_allow_multi_channels(false); /* reset it anyway */
> + qapi_event_send_migration(MIGRATION_STATUS_SETUP);
> + if (strstart(uri, "tcp:", &p) ||
> + strstart(uri, "unix:", NULL) ||
> + strstart(uri, "vsock:", NULL)) {
> + migrate_protocol_allow_multi_channels(true);
> + #ifdef CONFIG_RDMA
> + } else if (strstart(uri, "rdma:", &p)) {
> + rdma_start_incoming_migration(p, errp);
> + #endif
> + } else if (strstart(uri, "exec:", &p)) {
> + exec_start_incoming_migration(p, errp);
> + } else if (strstart(uri, "fd:", &p)) {
> + fd_start_incoming_migration(p, errp);
> + } else {
> + error_setg(errp, "unknown migration protocol: %s", uri);
> + }
> } else {
> - error_setg(errp, "unknown migration protocol: %s", uri);
> + /* multi-FD parameters only support tcp network protocols */
> + if (!strstart(uri, "tcp:", &p)) {
> + error_setg(errp, "multifd-destination uri supports "
> + "tcp protocol only");
> + return;
> + }
> + store_multifd_migration_params(p ? p : uri, NULL, number, idx, errp);
> }
> + socket_start_incoming_migration(p ? p : uri, number, errp);
> }
>
> static void process_incoming_migration_bh(void *opaque)
> @@ -2140,7 +2151,17 @@ void migrate_del_blocker(Error *reason)
> migration_blockers = g_slist_remove(migration_blockers, reason);
> }
>
> -void qmp_migrate_incoming(const char *uri, Error **errp)
> +static inline int incoming_multi_fd_uri_parse(const char *str, char delim)
> +{
> + int count = 0;
> + for (int i = 0; i < strlen(str); i++) {
> + count += (str[i] == delim);
> + }
> + return count;
> +}
That's a bit more general little helper function; I guess it could go in
util/ somewhere (something like qemu_string_count_delim ???)
> +/* migrate_incoming comes from -incoming flag in qemu process */
> +void migrate_incoming(const char *uri, Error **errp)
> {
> Error *local_err = NULL;
> static bool once = true;
> @@ -2154,11 +2175,99 @@ void qmp_migrate_incoming(const char *uri, Error **errp)
> return;
> }
>
> + qemu_start_incoming_migration(uri, 0, 0, &local_err);
> +
> if (!yank_register_instance(MIGRATION_YANK_INSTANCE, errp)) {
> return;
> }
>
> - qemu_start_incoming_migration(uri, &local_err);
> + if (local_err) {
> + yank_unregister_instance(MIGRATION_YANK_INSTANCE);
> + error_propagate(errp, local_err);
> + return;
> + }
> +
> + once = false;
Can you explain what this change is for?
> +}
> +
> +/* multi_fd_migrate_incoming comes from -multi-fd-migrate-incoming flag in qemu process */
> +void multi_fd_migrate_incoming(const char *uri, Error **errp)
> +{
> + Error *local_err = NULL;
> + static bool once = true;
> +
> + if (!once) {
> + error_setg(errp, "The incoming migration has already been started");
> + return;
> + }
> + if (!runstate_check(RUN_STATE_INMIGRATE)) {
> + error_setg(errp, "'-multi-fd-incoming' was not specified on the command line");
> + return;
> + }
> +
> + strList *st = strList_from_string(uri, ',');
> + strList *r = st;
> + int length = QAPI_LIST_LENGTH(st);
> + init_multifd_array(length);
> +
> + for (int i = 0; i < length; i++) {
> + const char *uri = NULL, *ret = NULL;
> + const char *str = r->value;
> + uint8_t multifd_channels = DEFAULT_MIGRATE_MULTIFD_CHANNELS;
> + int parse_count = incoming_multi_fd_uri_parse(str, ':');
> + if (parse_count < 2 || parse_count > 3) {
> + error_setg(errp, "Invalid format of string-id %d in "
> + "'-multi-fd-incoming' flag", i);
> + return;
> + }
> + if (parse_count == 3) {
> + ret = strrchr(str, ':');
> + uri = g_strndup(str, strlen(str) - strlen(ret));
> + multifd_channels = atoi(ret + 1);
> + }
> + qemu_start_incoming_migration(parse_count == 2 ? str : uri,
> + multifd_channels, i, &local_err);
> + r = r->next;
> + }
> +
> + if (local_err) {
> + yank_unregister_instance(MIGRATION_YANK_INSTANCE);
> + error_propagate(errp, local_err);
> + return;
> + }
> +
> + once = false;
> +}
> +
> +/* qmp_migrate_incoming comes from qemu qmp monitor command */
> +void qmp_migrate_incoming(const char *uri, bool has_multi_fd_uri_list,
> + MigrateIncomingUriList *cap, Error **errp)
> +{
> + Error *local_err = NULL;
> + static bool once = true;
> +
> + if (!once) {
> + error_setg(errp, "The incoming migration has already been started");
> + return;
> + }
> +
> + if (!yank_register_instance(MIGRATION_YANK_INSTANCE, errp)) {
> + return;
> + }
> +
> + /* For migration thread */
> + qemu_start_incoming_migration(uri, 0, 0, &local_err);
> +
> + /* For Multi-FD */
> + int length = QAPI_LIST_LENGTH(cap);
> + init_multifd_array(length);
> + for (int i = 0; i < length; i++) {
> + const char *multifd_dst_uri = cap->value->destination_uri;
> + uint8_t multifd_channels = cap->value->multifd_channels;
> + qemu_start_incoming_migration(multifd_dst_uri, multifd_channels,
> + i, &local_err);
> + cap = cap->next;
> + }
>
> if (local_err) {
> yank_unregister_instance(MIGRATION_YANK_INSTANCE);
> @@ -2194,7 +2303,7 @@ void qmp_migrate_recover(const char *uri, Error **errp)
> * only re-setup the migration stream and poke existing migration
> * to continue using that newly established channel.
> */
> - qemu_start_incoming_migration(uri, errp);
> + qemu_start_incoming_migration(uri, 0, 0, errp);
> }
>
> void qmp_migrate_pause(Error **errp)
> diff --git a/migration/migration.h b/migration/migration.h
> index 485d58b95f..fa8717ec9e 100644
> --- a/migration/migration.h
> +++ b/migration/migration.h
> @@ -353,6 +353,8 @@ bool migration_is_setup_or_active(int state);
> bool migration_is_running(int state);
>
> void migrate_init(MigrationState *s);
> +void migrate_incoming(const char *uri, Error **errp);
> +void multi_fd_migrate_incoming(const char *uri_str, Error **errp);
> bool migration_is_blocked(Error **errp);
> /* True if outgoing migration has entered postcopy phase */
> bool migration_in_postcopy(void);
> diff --git a/migration/socket.c b/migration/socket.c
> index 7ca6af8cca..21e0983df2 100644
> --- a/migration/socket.c
> +++ b/migration/socket.c
> @@ -202,17 +202,17 @@ socket_incoming_migration_end(void *opaque)
>
> static void
> socket_start_incoming_migration_internal(SocketAddress *saddr,
> - Error **errp)
> + uint8_t number, Error **errp)
> {
> QIONetListener *listener = qio_net_listener_new();
> MigrationIncomingState *mis = migration_incoming_get_current();
> size_t i;
> - int num = 1;
> + uint8_t num = 1;
>
> qio_net_listener_set_name(listener, "migration-socket-listener");
>
> if (migrate_use_multifd()) {
> - num = migrate_multifd_channels();
> + num = number;
That's a strange line - it suggests you need a better name for both
parameters!
> }
>
> if (qio_net_listener_open_sync(listener, saddr, num, errp) < 0) {
> @@ -239,12 +239,13 @@ socket_start_incoming_migration_internal(SocketAddress *saddr,
> }
> }
>
> -void socket_start_incoming_migration(const char *str, Error **errp)
> +void socket_start_incoming_migration(const char *str,
> + uint8_t number, Error **errp)
> {
> Error *err = NULL;
> SocketAddress *saddr = socket_parse(str, &err);
> if (!err) {
> - socket_start_incoming_migration_internal(saddr, &err);
> + socket_start_incoming_migration_internal(saddr, number, &err);
> }
> qapi_free_SocketAddress(saddr);
> error_propagate(errp, err);
> diff --git a/migration/socket.h b/migration/socket.h
> index bba7f177fe..b9e3699167 100644
> --- a/migration/socket.h
> +++ b/migration/socket.h
> @@ -30,7 +30,8 @@ struct SrcDestAddr {
> void socket_send_channel_create(QIOTaskFunc f, void *data);
> int socket_send_channel_destroy(QIOChannel *send);
>
> -void socket_start_incoming_migration(const char *str, Error **errp);
> +void socket_start_incoming_migration(const char *str, uint8_t number,
> + Error **errp);
>
> void socket_start_outgoing_migration(MigrationState *s, const char *dst_str,
> Error **errp);
> diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
> index 2db539016a..32a6b67d5f 100644
> --- a/monitor/hmp-cmds.c
> +++ b/monitor/hmp-cmds.c
> @@ -42,6 +42,7 @@
> #include "qapi/qapi-commands-run-state.h"
> #include "qapi/qapi-commands-tpm.h"
> #include "qapi/qapi-commands-ui.h"
> +#include "qapi/util.h"
> #include "qapi/qapi-visit-net.h"
> #include "qapi/qapi-visit-migration.h"
> #include "qapi/qmp/qdict.h"
> @@ -72,32 +73,6 @@ bool hmp_handle_error(Monitor *mon, Error *err)
> return false;
> }
>
> -/*
> - * Produce a strList from a comma separated list.
> - * A NULL or empty input string return NULL.
> - */
> -static strList *strList_from_comma_list(const char *in)
> -{
> - strList *res = NULL;
> - strList **tail = &res;
> -
> - while (in && in[0]) {
> - char *comma = strchr(in, ',');
> - char *value;
> -
> - if (comma) {
> - value = g_strndup(in, comma - in);
> - in = comma + 1; /* skip the , */
> - } else {
> - value = g_strdup(in);
> - in = NULL;
> - }
> - QAPI_LIST_APPEND(tail, value);
> - }
> -
> - return res;
> -}
> -
> void hmp_info_name(Monitor *mon, const QDict *qdict)
> {
> NameInfo *info;
> @@ -1117,7 +1092,7 @@ void hmp_announce_self(Monitor *mon, const QDict *qdict)
> migrate_announce_params());
>
> qapi_free_strList(params->interfaces);
> - params->interfaces = strList_from_comma_list(interfaces_str);
> + params->interfaces = strList_from_string(interfaces_str, ',');
> params->has_interfaces = params->interfaces != NULL;
> params->id = g_strdup(id);
> params->has_id = !!params->id;
> @@ -1147,8 +1122,19 @@ void hmp_migrate_incoming(Monitor *mon, const QDict *qdict)
> {
> Error *err = NULL;
> const char *uri = qdict_get_str(qdict, "uri");
> + const char *dst_uri = qdict_get_str(qdict, "destination-uri");
> + uint8_t multifd_channels = qdict_get_try_int(qdict, "multifd-channels",
> + DEFAULT_MIGRATE_MULTIFD_CHANNELS);
> + MigrateIncomingUriList *caps = NULL;
> + MigrateIncomingUri *value;
> +
> + value = g_malloc0(sizeof(*value));
> + value->destination_uri = (char *)dst_uri;
> + value->multifd_channels = multifd_channels;
> + QAPI_LIST_PREPEND(caps, value);
>
> - qmp_migrate_incoming(uri, &err);
> + qmp_migrate_incoming(uri, !!caps, caps, &err);
> + qapi_free_MigrateIncomingUriList(caps);
>
> hmp_handle_error(mon, err);
> }
> diff --git a/qapi/migration.json b/qapi/migration.json
> index fb259d626b..62a7b22d19 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -1527,15 +1527,37 @@
> 'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateUriParameter'], '*blk': 'bool',
> '*inc': 'bool', '*detach': 'bool', '*resume': 'bool' } }
>
> +##
> +# @MigrateIncomingUri:
> +#
> +# Information regarding which destination listening interface to be connected
> +# and number of multifd channels over that interface.
> +#
> +# @destination-uri: the Uniform Resource Identifier of the destination VM
> +#
> +# @multifd-channels: number of channels used to migrate data in parallel for
> +# for specific source-uri and destination-uri.
> +# Default value in this case is 2 (Since 4.0)
> +#
> +##
> +{ 'struct' : 'MigrateIncomingUri',
> + 'data' : { 'destination-uri' : 'str',
> + '*multifd-channels' : 'uint8'} }
> +
> ##
> # @migrate-incoming:
> #
> # Start an incoming migration, the qemu must have been started
> -# with -incoming defer
> +# with -incoming defer. qemu can also be started with optional
> +# -multi-fd-incoming defer for opening multifd listening sockets
> #
> # @uri: The Uniform Resource Identifier identifying the source or
> # address to listen on
> #
> +# @multi-fd-uri-list: list of pair of source and destination VM Uniform
> +# Resource Identifiers with number of multifd-channels
> +# for each pair
> +#
> # Returns: nothing on success
> #
> # Since: 2.3
> @@ -1546,19 +1568,30 @@
> # compatible with -incoming and the format of the uri is already exposed
> # above libvirt.
> #
> -# 2. QEMU must be started with -incoming defer to allow migrate-incoming to
> +# 2. multi-fd-uri-list will have list of destination uri as listening sockets
> +# and multi-fd number of channels on each listening socket.
> +#
> +# 3. QEMU must be started with -incoming defer to allow migrate-incoming to
> # be used.
> #
> -# 3. The uri format is the same as for -incoming
> +# 4. multi-fd-uri-list format is not the same as for -multi-fd-incoming flag.
> +# For -multi-fd-incoming flag, it is a comma separated list of listener
> +# sockets with multifd channels.
> +# Example: -multi-fd-incoming "tcp::6900:4,tcp:11.0.0.0:7789:5".
Why bother with -multi-fd-incoming ? Just insist on people using
-incoming defer - I think we'd like to do away with the other forms
of -incoming, it simplifies stuff a lot that way rather than having two
schemes.
> # Example:
> #
> # -> { "execute": "migrate-incoming",
> -# "arguments": { "uri": "tcp::4446" } }
> +# "arguments": {"uri": "tcp::6789",
> +# "multi-fd-uri-list" : [ {"destination-uri" : "tcp::6900",
> +# "multifd-channels": 4},
> +# {"destination-uri" : "tcp:11.0.0.0:7789",
> +# "multifd-channels": 5} ] } }
> # <- { "return": {} }
> #
> ##
> -{ 'command': 'migrate-incoming', 'data': {'uri': 'str' } }
> +{ 'command': 'migrate-incoming',
> + 'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateIncomingUri'] } }
>
> ##
> # @xen-save-devices-state:
> diff --git a/qapi/qapi-util.c b/qapi/qapi-util.c
> index 63596e11c5..9cc43ebcd3 100644
> --- a/qapi/qapi-util.c
> +++ b/qapi/qapi-util.c
> @@ -15,6 +15,7 @@
> #include "qapi/error.h"
> #include "qemu/ctype.h"
> #include "qapi/qmp/qerror.h"
> +#include "qapi/qapi-builtin-types.h"
>
> CompatPolicy compat_policy;
>
> @@ -152,3 +153,29 @@ int parse_qapi_name(const char *str, bool complete)
> }
> return p - str;
> }
> +
> +/*
> + * Produce a strList from a delimiter separated list.
> + * A NULL or empty input string return NULL.
> + */
> +strList *strList_from_string(const char *in, char c)
> +{
> + strList *res = NULL;
> + strList **tail = &res;
> +
> + while (in && in[0]) {
> + char *ch = strchr(in, c);
> + char *value;
> +
> + if (ch) {
> + value = g_strndup(in, ch - in);
> + in = ch + 1; /* skip the , */
> + } else {
> + value = g_strdup(in);
> + in = NULL;
> + }
> + QAPI_LIST_APPEND(tail, value);
> + }
> +
> + return res;
> +}
> diff --git a/qemu-options.hx b/qemu-options.hx
> index 60cf188da4..2e82e41dd5 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -4480,6 +4480,24 @@ SRST
> to issuing the migrate\_incoming to allow the migration to begin.
> ERST
>
> +DEF("multi-fd-incoming", HAS_ARG, QEMU_OPTION_multi_fd_incoming, \
> + "-multi-fd-incoming tcp:[host]:port[:channel][,to=maxport][,ipv4=on|off][,ipv6=on|off]\n" \
> + "-multi-fd-incoming defer\n" \
> + " wait for the URI to be specified via\n" \
> + " multi_fd_migrate_incoming\n",
> + QEMU_ARCH_ALL)
> +SRST
> +``-multi-fd-incoming tcp:[host]:port[:channel][,to=maxport][,ipv4=on|off][,ipv6=on|off]``
> + Prepare for multi-fd incoming migration, with multi-fd listening sockets
> + on that connection. Default number of multi-fd channels is 2.
> +
> +``-multi-fd-incoming defer``
> + Wait for the URI to be specified via multi_fd_migrate\_incoming. The
> + monitor can be used to change settings (such as migration parameters)
> + prior to issuing the multi_fd_migrate\_incoming to allow the migration
> + to begin.
> +ERST
> +
> DEF("only-migratable", 0, QEMU_OPTION_only_migratable, \
> "-only-migratable allow only migratable devices\n", QEMU_ARCH_ALL)
> SRST
> diff --git a/softmmu/vl.c b/softmmu/vl.c
> index 4c1e94b00e..2f5cf18eff 100644
> --- a/softmmu/vl.c
> +++ b/softmmu/vl.c
> @@ -45,7 +45,7 @@
> #include "sysemu/seccomp.h"
> #include "sysemu/tcg.h"
> #include "sysemu/xen.h"
> -
> +#include "migration/migration.h"
> #include "qemu/error-report.h"
> #include "qemu/sockets.h"
> #include "qemu/accel.h"
> @@ -167,6 +167,7 @@ typedef struct DeviceOption {
> static const char *cpu_option;
> static const char *mem_path;
> static const char *incoming;
> +static const char *multi_fd_incoming;
> static const char *loadvm;
> static const char *accelerators;
> static bool have_custom_ram_size;
> @@ -2349,6 +2350,11 @@ static void qemu_validate_options(const QDict *machine_opts)
> error_report("'preconfig' supports '-incoming defer' only");
> exit(EXIT_FAILURE);
> }
> + if (multi_fd_incoming && preconfig_requested &&
> + strcmp(multi_fd_incoming, "defer") != 0) {
> + error_report("'preconfig' supports '-multi-fd-incoming defer' only");
> + exit(EXIT_FAILURE);
> + }
>
> #ifdef CONFIG_CURSES
> if (is_daemonized() && dpy.type == DISPLAY_TYPE_CURSES) {
> @@ -2621,7 +2627,7 @@ void qmp_x_exit_preconfig(Error **errp)
> if (incoming) {
> Error *local_err = NULL;
> if (strcmp(incoming, "defer") != 0) {
> - qmp_migrate_incoming(incoming, &local_err);
> + migrate_incoming(incoming, &local_err);
> if (local_err) {
> error_reportf_err(local_err, "-incoming %s: ", incoming);
> exit(1);
> @@ -2630,6 +2636,20 @@ void qmp_x_exit_preconfig(Error **errp)
> } else if (autostart) {
> qmp_cont(NULL);
> }
> +
> + if (multi_fd_incoming) {
> + Error *local_err = NULL;
> + if (strcmp(multi_fd_incoming, "defer") != 0) {
> + multi_fd_migrate_incoming(multi_fd_incoming, &local_err);
> + if (local_err) {
> + error_reportf_err(local_err, "-multi-fd-incoming %s: ",
> + multi_fd_incoming);
> + exit(1);
> + }
> + }
> + } else if (autostart) {
> + qmp_cont(NULL);
> + }
> }
>
> void qemu_init(int argc, char **argv, char **envp)
> @@ -3355,6 +3375,12 @@ void qemu_init(int argc, char **argv, char **envp)
> }
> incoming = optarg;
> break;
> + case QEMU_OPTION_multi_fd_incoming:
> + if (!multi_fd_incoming) {
> + runstate_set(RUN_STATE_INMIGRATE);
> + }
> + multi_fd_incoming = optarg;
> + break;
> case QEMU_OPTION_only_migratable:
> only_migratable = 1;
> break;
> --
> 2.22.3
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 4/4] Adding support for multi-FD connections dynamically
2022-06-09 7:33 ` [PATCH 4/4] Adding support for multi-FD connections dynamically Het Gala
@ 2022-06-16 18:47 ` Dr. David Alan Gilbert
2022-06-21 16:12 ` manish.mishra
0 siblings, 1 reply; 34+ messages in thread
From: Dr. David Alan Gilbert @ 2022-06-16 18:47 UTC (permalink / raw)
To: Het Gala
Cc: qemu-devel, quintela, pbonzini, berrange, armbru, eblake, Manish Mishra
* Het Gala (het.gala@nutanix.com) wrote:
> i) Dynamically decide appropriate source and destination ip pairs for the
> corresponding multi-FD channel to be connected.
>
> ii) Removed the support for setting the number of multi-fd channels from qmp
> commands. As now all multiFD parameters will be passed via qmp: migrate
> command or incoming flag itself.
We can't do that, because it's part of the API already; what you'll need
to do is check that the number of entries in your list corresponds to
the value set there and error if it's different.
Dave
> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
> Signed-off-by: Het Gala <het.gala@nutanix.com>
> ---
> migration/migration.c | 15 ---------------
> migration/migration.h | 1 -
> migration/multifd.c | 42 +++++++++++++++++++++---------------------
> migration/socket.c | 42 +++++++++++++++++++++++++++++++++---------
> migration/socket.h | 4 +++-
> monitor/hmp-cmds.c | 4 ----
> qapi/migration.json | 6 ------
> 7 files changed, 57 insertions(+), 57 deletions(-)
>
> diff --git a/migration/migration.c b/migration/migration.c
> index 9b0ad732e7..57dd4494b4 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -1585,9 +1585,6 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
> if (params->has_block_incremental) {
> dest->block_incremental = params->block_incremental;
> }
> - if (params->has_multifd_channels) {
> - dest->multifd_channels = params->multifd_channels;
> - }
> if (params->has_multifd_compression) {
> dest->multifd_compression = params->multifd_compression;
> }
> @@ -1702,9 +1699,6 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
> if (params->has_block_incremental) {
> s->parameters.block_incremental = params->block_incremental;
> }
> - if (params->has_multifd_channels) {
> - s->parameters.multifd_channels = params->multifd_channels;
> - }
> if (params->has_multifd_compression) {
> s->parameters.multifd_compression = params->multifd_compression;
> }
> @@ -2686,15 +2680,6 @@ bool migrate_pause_before_switchover(void)
> MIGRATION_CAPABILITY_PAUSE_BEFORE_SWITCHOVER];
> }
>
> -int migrate_multifd_channels(void)
> -{
> - MigrationState *s;
> -
> - s = migrate_get_current();
> -
> - return s->parameters.multifd_channels;
> -}
> -
> MultiFDCompression migrate_multifd_compression(void)
> {
> MigrationState *s;
> diff --git a/migration/migration.h b/migration/migration.h
> index fa8717ec9e..9464de8ef7 100644
> --- a/migration/migration.h
> +++ b/migration/migration.h
> @@ -372,7 +372,6 @@ bool migrate_validate_uuid(void);
> bool migrate_auto_converge(void);
> bool migrate_use_multifd(void);
> bool migrate_pause_before_switchover(void);
> -int migrate_multifd_channels(void);
> MultiFDCompression migrate_multifd_compression(void);
> int migrate_multifd_zlib_level(void);
> int migrate_multifd_zstd_level(void);
> diff --git a/migration/multifd.c b/migration/multifd.c
> index 9282ab6aa4..ce017436fb 100644
> --- a/migration/multifd.c
> +++ b/migration/multifd.c
> @@ -225,7 +225,7 @@ static int multifd_recv_initial_packet(QIOChannel *c, Error **errp)
> return -1;
> }
>
> - if (msg.id > migrate_multifd_channels()) {
> + if (msg.id > total_multifd_channels()) {
> error_setg(errp, "multifd: received channel version %u "
> "expected %u", msg.version, MULTIFD_VERSION);
> return -1;
> @@ -410,8 +410,8 @@ static int multifd_send_pages(QEMUFile *f)
> * using more channels, so ensure it doesn't overflow if the
> * limit is lower now.
> */
> - next_channel %= migrate_multifd_channels();
> - for (i = next_channel;; i = (i + 1) % migrate_multifd_channels()) {
> + next_channel %= total_multifd_channels();
> + for (i = next_channel;; i = (i + 1) % total_multifd_channels()) {
> p = &multifd_send_state->params[i];
>
> qemu_mutex_lock(&p->mutex);
> @@ -422,7 +422,7 @@ static int multifd_send_pages(QEMUFile *f)
> }
> if (!p->pending_job) {
> p->pending_job++;
> - next_channel = (i + 1) % migrate_multifd_channels();
> + next_channel = (i + 1) % total_multifd_channels();
> break;
> }
> qemu_mutex_unlock(&p->mutex);
> @@ -500,7 +500,7 @@ static void multifd_send_terminate_threads(Error *err)
> return;
> }
>
> - for (i = 0; i < migrate_multifd_channels(); i++) {
> + for (i = 0; i < total_multifd_channels(); i++) {
> MultiFDSendParams *p = &multifd_send_state->params[i];
>
> qemu_mutex_lock(&p->mutex);
> @@ -521,14 +521,14 @@ void multifd_save_cleanup(void)
> return;
> }
> multifd_send_terminate_threads(NULL);
> - for (i = 0; i < migrate_multifd_channels(); i++) {
> + for (i = 0; i < total_multifd_channels(); i++) {
> MultiFDSendParams *p = &multifd_send_state->params[i];
>
> if (p->running) {
> qemu_thread_join(&p->thread);
> }
> }
> - for (i = 0; i < migrate_multifd_channels(); i++) {
> + for (i = 0; i < total_multifd_channels(); i++) {
> MultiFDSendParams *p = &multifd_send_state->params[i];
> Error *local_err = NULL;
>
> @@ -594,7 +594,7 @@ int multifd_send_sync_main(QEMUFile *f)
>
> flush_zero_copy = migrate_use_zero_copy_send();
>
> - for (i = 0; i < migrate_multifd_channels(); i++) {
> + for (i = 0; i < total_multifd_channels(); i++) {
> MultiFDSendParams *p = &multifd_send_state->params[i];
>
> trace_multifd_send_sync_main_signal(p->id);
> @@ -627,7 +627,7 @@ int multifd_send_sync_main(QEMUFile *f)
> }
> }
> }
> - for (i = 0; i < migrate_multifd_channels(); i++) {
> + for (i = 0; i < total_multifd_channels(); i++) {
> MultiFDSendParams *p = &multifd_send_state->params[i];
>
> trace_multifd_send_sync_main_wait(p->id);
> @@ -903,7 +903,7 @@ int multifd_save_setup(Error **errp)
> int thread_count;
> uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
> uint8_t i;
> -
> + int idx;
> if (!migrate_use_multifd()) {
> return 0;
> }
> @@ -912,7 +912,7 @@ int multifd_save_setup(Error **errp)
> return -1;
> }
>
> - thread_count = migrate_multifd_channels();
> + thread_count = total_multifd_channels();
> multifd_send_state = g_malloc0(sizeof(*multifd_send_state));
> multifd_send_state->params = g_new0(MultiFDSendParams, thread_count);
> multifd_send_state->pages = multifd_pages_init(page_count);
> @@ -945,8 +945,8 @@ int multifd_save_setup(Error **errp)
> } else {
> p->write_flags = 0;
> }
> -
> - socket_send_channel_create(multifd_new_send_channel_async, p);
> + idx = multifd_index(i);
> + socket_send_channel_create(multifd_new_send_channel_async, p, idx);
> }
>
> for (i = 0; i < thread_count; i++) {
> @@ -991,7 +991,7 @@ static void multifd_recv_terminate_threads(Error *err)
> }
> }
>
> - for (i = 0; i < migrate_multifd_channels(); i++) {
> + for (i = 0; i < total_multifd_channels(); i++) {
> MultiFDRecvParams *p = &multifd_recv_state->params[i];
>
> qemu_mutex_lock(&p->mutex);
> @@ -1017,7 +1017,7 @@ int multifd_load_cleanup(Error **errp)
> return 0;
> }
> multifd_recv_terminate_threads(NULL);
> - for (i = 0; i < migrate_multifd_channels(); i++) {
> + for (i = 0; i < total_multifd_channels(); i++) {
> MultiFDRecvParams *p = &multifd_recv_state->params[i];
>
> if (p->running) {
> @@ -1030,7 +1030,7 @@ int multifd_load_cleanup(Error **errp)
> qemu_thread_join(&p->thread);
> }
> }
> - for (i = 0; i < migrate_multifd_channels(); i++) {
> + for (i = 0; i < total_multifd_channels(); i++) {
> MultiFDRecvParams *p = &multifd_recv_state->params[i];
>
> migration_ioc_unregister_yank(p->c);
> @@ -1065,13 +1065,13 @@ void multifd_recv_sync_main(void)
> if (!migrate_use_multifd()) {
> return;
> }
> - for (i = 0; i < migrate_multifd_channels(); i++) {
> + for (i = 0; i < total_multifd_channels(); i++) {
> MultiFDRecvParams *p = &multifd_recv_state->params[i];
>
> trace_multifd_recv_sync_main_wait(p->id);
> qemu_sem_wait(&multifd_recv_state->sem_sync);
> }
> - for (i = 0; i < migrate_multifd_channels(); i++) {
> + for (i = 0; i < total_multifd_channels(); i++) {
> MultiFDRecvParams *p = &multifd_recv_state->params[i];
>
> WITH_QEMU_LOCK_GUARD(&p->mutex) {
> @@ -1166,7 +1166,7 @@ int multifd_load_setup(Error **errp)
> error_setg(errp, "multifd is not supported by current protocol");
> return -1;
> }
> - thread_count = migrate_multifd_channels();
> + thread_count = total_multifd_channels();
> multifd_recv_state = g_malloc0(sizeof(*multifd_recv_state));
> multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count);
> qatomic_set(&multifd_recv_state->count, 0);
> @@ -1204,7 +1204,7 @@ int multifd_load_setup(Error **errp)
>
> bool multifd_recv_all_channels_created(void)
> {
> - int thread_count = migrate_multifd_channels();
> + int thread_count = total_multifd_channels();
>
> if (!migrate_use_multifd()) {
> return true;
> @@ -1259,5 +1259,5 @@ bool multifd_recv_new_channel(QIOChannel *ioc, Error **errp)
> QEMU_THREAD_JOINABLE);
> qatomic_inc(&multifd_recv_state->count);
> return qatomic_read(&multifd_recv_state->count) ==
> - migrate_multifd_channels();
> + total_multifd_channels();
> }
> diff --git a/migration/socket.c b/migration/socket.c
> index d0cb7cc6a6..c0ac6dbbe2 100644
> --- a/migration/socket.c
> +++ b/migration/socket.c
> @@ -28,9 +28,6 @@
> #include "trace.h"
>
>
> -struct SocketOutgoingArgs {
> - SocketAddress *saddr;
> -} outgoing_args;
>
> struct SocketArgs {
> struct SrcDestAddr data;
> @@ -43,20 +40,47 @@ struct OutgoingMigrateParams {
> uint64_t total_multifd_channel;
> } outgoing_migrate_params;
>
> -void socket_send_channel_create(QIOTaskFunc f, void *data)
> +
> +int total_multifd_channels(void)
> +{
> + return outgoing_migrate_params.total_multifd_channel;
> +}
> +
> +int multifd_index(int i)
> +{
> + int length = outgoing_migrate_params.length;
> + int j = 0;
> + int runn_sum = 0;
> + while (j < length) {
> + runn_sum += outgoing_migrate_params.socket_args[j].multifd_channels;
> + if (i >= runn_sum) {
> + j++;
> + } else {
> + break;
> + }
> + }
> + return j;
> +}
> +
> +void socket_send_channel_create(QIOTaskFunc f, void *data, int idx)
> {
> QIOChannelSocket *sioc = qio_channel_socket_new();
> - qio_channel_socket_connect_async(sioc, outgoing_args.saddr,
> - f, data, NULL, NULL, NULL);
> + qio_channel_socket_connect_async(sioc,
> + outgoing_migrate_params.socket_args[idx].data.dst_addr,
> + f, data, NULL, NULL,
> + outgoing_migrate_params.socket_args[idx].data.src_addr);
> }
>
> int socket_send_channel_destroy(QIOChannel *send)
> {
> /* Remove channel */
> object_unref(OBJECT(send));
> - if (outgoing_args.saddr) {
> - qapi_free_SocketAddress(outgoing_args.saddr);
> - outgoing_args.saddr = NULL;
> + if (outgoing_migrate_params.socket_args != NULL) {
> + g_free(outgoing_migrate_params.socket_args);
> + outgoing_migrate_params.socket_args = NULL;
> + }
> + if (outgoing_migrate_params.length) {
> + outgoing_migrate_params.length = 0;
> }
>
> if (outgoing_migrate_params.socket_args != NULL) {
> diff --git a/migration/socket.h b/migration/socket.h
> index b9e3699167..c8b9252384 100644
> --- a/migration/socket.h
> +++ b/migration/socket.h
> @@ -27,7 +27,9 @@ struct SrcDestAddr {
> SocketAddress *src_addr;
> };
>
> -void socket_send_channel_create(QIOTaskFunc f, void *data);
> +int total_multifd_channels(void);
> +int multifd_index(int i);
> +void socket_send_channel_create(QIOTaskFunc f, void *data, int idx);
> int socket_send_channel_destroy(QIOChannel *send);
>
> void socket_start_incoming_migration(const char *str, uint8_t number,
> diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
> index 32a6b67d5f..9a3d76d6ba 100644
> --- a/monitor/hmp-cmds.c
> +++ b/monitor/hmp-cmds.c
> @@ -1281,10 +1281,6 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
> p->has_block_incremental = true;
> visit_type_bool(v, param, &p->block_incremental, &err);
> break;
> - case MIGRATION_PARAMETER_MULTIFD_CHANNELS:
> - p->has_multifd_channels = true;
> - visit_type_uint8(v, param, &p->multifd_channels, &err);
> - break;
> case MIGRATION_PARAMETER_MULTIFD_COMPRESSION:
> p->has_multifd_compression = true;
> visit_type_MultiFDCompression(v, param, &p->multifd_compression,
> diff --git a/qapi/migration.json b/qapi/migration.json
> index 62a7b22d19..1b1c6d01d3 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -877,11 +877,6 @@
> # migrated and the destination must already have access to the
> # same backing chain as was used on the source. (since 2.10)
> #
> -# @multifd-channels: Number of channels used to migrate data in
> -# parallel. This is the same number that the
> -# number of sockets used for migration. The
> -# default value is 2 (since 4.0)
> -#
> # @xbzrle-cache-size: cache size to be used by XBZRLE migration. It
> # needs to be a multiple of the target page size
> # and a power of 2
> @@ -965,7 +960,6 @@
> '*x-checkpoint-delay': { 'type': 'uint32',
> 'features': [ 'unstable' ] },
> '*block-incremental': 'bool',
> - '*multifd-channels': 'uint8',
> '*xbzrle-cache-size': 'size',
> '*max-postcopy-bandwidth': 'size',
> '*max-cpu-throttle': 'uint8',
> --
> 2.22.3
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 3/4] Establishing connection between any non-default source and destination pair
2022-06-16 17:39 ` Daniel P. Berrangé
@ 2022-06-21 16:09 ` manish.mishra
[not found] ` <54ca00c7-a108-11e3-1c8d-8771798aed6a@nutanix.com>
1 sibling, 0 replies; 34+ messages in thread
From: manish.mishra @ 2022-06-21 16:09 UTC (permalink / raw)
To: Daniel P. Berrangé, Het Gala
Cc: qemu-devel, quintela, dgilbert, pbonzini, armbru, eblake
[-- Attachment #1: Type: text/plain, Size: 16449 bytes --]
Hi Daniel, David,
Thank you so much for review on patches. I am posting this message on
behalf of Het. We wanted to get a early feedback so sorry if code was not
in best of shape. Het is currently on break intership break so does not have
access to nutanix mail, he will join in first week of july and will definately post
v2 on this by 2nd week of july.
thanks
Manish MIshra
On 16/06/22 11:09 pm, Daniel P. Berrangé wrote:
> On Thu, Jun 09, 2022 at 07:33:04AM +0000, Het Gala wrote:
>> i) Binding of the socket to source ip address and port on the non-default
>> interface has been implemented for multi-FD connection, which was not
>> necessary earlier because the binding was on the default interface itself.
>>
>> ii) Created an end to end connection between all multi-FD source and
>> destination pairs.
>>
>> Suggested-by: Manish Mishra<manish.mishra@nutanix.com>
>> Signed-off-by: Het Gala<het.gala@nutanix.com>
>> ---
>> chardev/char-socket.c | 4 +-
>> include/io/channel-socket.h | 26 ++++++-----
>> include/qemu/sockets.h | 6 ++-
>> io/channel-socket.c | 50 ++++++++++++++------
>> migration/socket.c | 15 +++---
>> nbd/client-connection.c | 2 +-
>> qemu-nbd.c | 4 +-
>> scsi/pr-manager-helper.c | 1 +
>> tests/unit/test-char.c | 8 ++--
>> tests/unit/test-io-channel-socket.c | 4 +-
>> tests/unit/test-util-sockets.c | 16 +++----
>> ui/input-barrier.c | 2 +-
>> ui/vnc.c | 3 +-
>> util/qemu-sockets.c | 71 ++++++++++++++++++++---------
>> 14 files changed, 135 insertions(+), 77 deletions(-)
>>
>> diff --git a/chardev/char-socket.c b/chardev/char-socket.c
>> index dc4e218eeb..f3725238c5 100644
>> --- a/chardev/char-socket.c
>> +++ b/chardev/char-socket.c
>> @@ -932,7 +932,7 @@ static int tcp_chr_connect_client_sync(Chardev *chr, Error **errp)
>> QIOChannelSocket *sioc = qio_channel_socket_new();
>> tcp_chr_change_state(s, TCP_CHARDEV_STATE_CONNECTING);
>> tcp_chr_set_client_ioc_name(chr, sioc);
>> - if (qio_channel_socket_connect_sync(sioc, s->addr, errp) < 0) {
>> + if (qio_channel_socket_connect_sync(sioc, s->addr, NULL, errp) < 0) {
>> tcp_chr_change_state(s, TCP_CHARDEV_STATE_DISCONNECTED);
>> object_unref(OBJECT(sioc));
>> return -1;
>> @@ -1120,7 +1120,7 @@ static void tcp_chr_connect_client_task(QIOTask *task,
>> SocketAddress *addr = opaque;
>> Error *err = NULL;
>>
>> - qio_channel_socket_connect_sync(ioc, addr, &err);
>> + qio_channel_socket_connect_sync(ioc, addr, NULL, &err);
>>
>> qio_task_set_error(task, err);
>> }
>> diff --git a/include/io/channel-socket.h b/include/io/channel-socket.h
>> index 513c428fe4..59d5b1b349 100644
>> --- a/include/io/channel-socket.h
>> +++ b/include/io/channel-socket.h
>> @@ -83,41 +83,45 @@ qio_channel_socket_new_fd(int fd,
>> /**
>> * qio_channel_socket_connect_sync:
>> * @ioc: the socket channel object
>> - * @addr: the address to connect to
>> + * @dst_addr: the destination address to connect to
>> + * @src_addr: the source address to be connected
>> * @errp: pointer to a NULL-initialized error object
>> *
>> - * Attempt to connect to the address @addr. This method
>> - * will run in the foreground so the caller will not regain
>> - * execution control until the connection is established or
>> + * Attempt to connect to the address @dst_addr with @src_addr.
>> + * This method will run in the foreground so the caller will not
>> + * regain execution control until the connection is established or
>> * an error occurs.
>> */
>> int qio_channel_socket_connect_sync(QIOChannelSocket *ioc,
>> - SocketAddress *addr,
>> + SocketAddress *dst_addr,
>> + SocketAddress *src_addr,
>> Error **errp);
>>
>> /**
>> * qio_channel_socket_connect_async:
>> * @ioc: the socket channel object
>> - * @addr: the address to connect to
>> + * @dst_addr: the destination address to connect to
>> * @callback: the function to invoke on completion
>> * @opaque: user data to pass to @callback
>> * @destroy: the function to free @opaque
>> * @context: the context to run the async task. If %NULL, the default
>> * context will be used.
>> + * @src_addr: the source address to be connected
>> *
>> - * Attempt to connect to the address @addr. This method
>> - * will run in the background so the caller will regain
>> + * Attempt to connect to the address @dst_addr with the @src_addr.
>> + * This method will run in the background so the caller will regain
>> * execution control immediately. The function @callback
>> - * will be invoked on completion or failure. The @addr
>> + * will be invoked on completion or failure. The @dst_addr
>> * parameter will be copied, so may be freed as soon
>> * as this function returns without waiting for completion.
>> */
>> void qio_channel_socket_connect_async(QIOChannelSocket *ioc,
>> - SocketAddress *addr,
>> + SocketAddress *dst_addr,
>> QIOTaskFunc callback,
>> gpointer opaque,
>> GDestroyNotify destroy,
>> - GMainContext *context);
>> + GMainContext *context,
>> + SocketAddress *src_addr);
> Lets avoid modifying these two methods, and thus avoid
> updating countless callers.
>
> In common with typical pattern in QIO code, lets add
> a second variant
>
> qio_channel_socket_connect_full
> qio_channel_socket_connect_full_async
>
> which has the extra parameters, then make the existing
> simpler methods call the new ones.
>
> ie qio_channel_socket_connect should call
> qio_channel_socket_connect_full, pasing NULL for the
> src_addr.
>
>
>
>> diff --git a/io/channel-socket.c b/io/channel-socket.c
>> index dc9c165de1..f8746ad646 100644
>> --- a/io/channel-socket.c
>> +++ b/io/channel-socket.c
>> @@ -36,6 +36,12 @@
>>
>> #define SOCKET_MAX_FDS 16
>>
>> +struct SrcDestAddress {
>> + SocketAddress *dst_addr;
>> + SocketAddress *src_addr;
>> +};
> Nitpick, just call this 'struct ConnectData'.
>
>> SocketAddress *
>> qio_channel_socket_get_local_address(QIOChannelSocket *ioc,
>> Error **errp)
>> @@ -145,13 +151,14 @@ qio_channel_socket_new_fd(int fd,
>>
>>
>> int qio_channel_socket_connect_sync(QIOChannelSocket *ioc,
>> - SocketAddress *addr,
>> + SocketAddress *dst_addr,
>> + SocketAddress *src_addr,
>> Error **errp)
>> {
>> int fd;
>>
>> - trace_qio_channel_socket_connect_sync(ioc, addr);
>> - fd = socket_connect(addr, errp);
>> + trace_qio_channel_socket_connect_sync(ioc, dst_addr);
>> + fd = socket_connect(dst_addr, src_addr, errp);
>> if (fd < 0) {
>> trace_qio_channel_socket_connect_fail(ioc);
>> return -1;
>> @@ -177,39 +184,56 @@ int qio_channel_socket_connect_sync(QIOChannelSocket *ioc,
>> }
>>
>>
>> +static void qio_channel_socket_worker_free(gpointer opaque)
>> +{
>> + struct SrcDestAddress *data = opaque;
>> + if (!data) {
>> + return;
>> + }
>> + qapi_free_SocketAddress(data->dst_addr);
>> + qapi_free_SocketAddress(data->src_addr);
>> + g_free(data);
>> +}
>> +
>> +
>> static void qio_channel_socket_connect_worker(QIOTask *task,
>> gpointer opaque)
>> {
>> QIOChannelSocket *ioc = QIO_CHANNEL_SOCKET(qio_task_get_source(task));
>> - SocketAddress *addr = opaque;
>> + struct SrcDestAddress *data = opaque;
>> Error *err = NULL;
>>
>> - qio_channel_socket_connect_sync(ioc, addr, &err);
>> + qio_channel_socket_connect_sync(ioc, data->dst_addr, data->src_addr, &err);
>>
>> qio_task_set_error(task, err);
>> }
>>
>>
>> void qio_channel_socket_connect_async(QIOChannelSocket *ioc,
>> - SocketAddress *addr,
>> + SocketAddress *dst_addr,
>> QIOTaskFunc callback,
>> gpointer opaque,
>> GDestroyNotify destroy,
>> - GMainContext *context)
>> + GMainContext *context,
>> + SocketAddress *src_addr)
>> {
>> QIOTask *task = qio_task_new(
>> OBJECT(ioc), callback, opaque, destroy);
>> - SocketAddress *addrCopy;
>> -
>> - addrCopy = QAPI_CLONE(SocketAddress, addr);
>> + struct SrcDestAddress *data = g_new0(struct SrcDestAddress, 1);
>>
>> + data->dst_addr = QAPI_CLONE(SocketAddress, dst_addr);
>> + if (src_addr) {
>> + data->src_addr = QAPI_CLONE(SocketAddress, src_addr);
>> + } else {
>> + data->src_addr = NULL;
>> + }
>> /* socket_connect() does a non-blocking connect(), but it
>> * still blocks in DNS lookups, so we must use a thread */
>> - trace_qio_channel_socket_connect_async(ioc, addr);
>> + trace_qio_channel_socket_connect_async(ioc, dst_addr);
>> qio_task_run_in_thread(task,
>> qio_channel_socket_connect_worker,
>> - addrCopy,
>> - (GDestroyNotify)qapi_free_SocketAddress,
>> + data,
>> + qio_channel_socket_worker_free,
>> context);
>> }
>>
>> diff --git a/migration/socket.c b/migration/socket.c
>> index 21e0983df2..d0cb7cc6a6 100644
>> --- a/migration/socket.c
>> +++ b/migration/socket.c
>> @@ -47,7 +47,7 @@ void socket_send_channel_create(QIOTaskFunc f, void *data)
>> {
>> QIOChannelSocket *sioc = qio_channel_socket_new();
>> qio_channel_socket_connect_async(sioc, outgoing_args.saddr,
>> - f, data, NULL, NULL);
>> + f, data, NULL, NULL, NULL);
>> }
>>
>> int socket_send_channel_destroy(QIOChannel *send)
>> @@ -110,7 +110,7 @@ out:
>>
>> static void
>> socket_start_outgoing_migration_internal(MigrationState *s,
>> - SocketAddress *saddr,
>> + SocketAddress *dst_addr,
>> Error **errp)
>> {
>> QIOChannelSocket *sioc = qio_channel_socket_new();
>> @@ -118,20 +118,17 @@ socket_start_outgoing_migration_internal(MigrationState *s,
>>
>> data->s = s;
>>
>> - /* in case previous migration leaked it */
>> - qapi_free_SocketAddress(outgoing_args.saddr);
>> - outgoing_args.saddr = saddr;
>> -
>> - if (saddr->type == SOCKET_ADDRESS_TYPE_INET) {
>> - data->hostname = g_strdup(saddr->u.inet.host);
>> + if (dst_addr->type == SOCKET_ADDRESS_TYPE_INET) {
>> + data->hostname = g_strdup(dst_addr->u.inet.host);
>> }
>>
>> qio_channel_set_name(QIO_CHANNEL(sioc), "migration-socket-outgoing");
>> qio_channel_socket_connect_async(sioc,
>> - saddr,
>> + dst_addr,
>> socket_outgoing_migration,
>> data,
>> socket_connect_data_free,
>> + NULL,
>> NULL);
>> }
>>
>> diff --git a/util/qemu-sockets.c b/util/qemu-sockets.c
>> index 13b5b197f9..bbe0dc0ee0 100644
>> --- a/util/qemu-sockets.c
>> +++ b/util/qemu-sockets.c
>> @@ -226,7 +226,7 @@ static int inet_listen_saddr(InetSocketAddress *saddr,
>> return -1;
>> }
>>
>> - memset(&ai,0, sizeof(ai));
>> + memset(&ai,0,sizeof(ai));
> Add whitespace before the '0', rather than removing it after.
>
>> ai.ai_flags = AI_PASSIVE;
>> if (saddr->has_numeric && saddr->numeric) {
>> ai.ai_flags |= AI_NUMERICHOST | AI_NUMERICSERV;
>> @@ -282,8 +282,8 @@ static int inet_listen_saddr(InetSocketAddress *saddr,
>> e->ai_protocol = IPPROTO_MPTCP;
>> }
>> #endif
>> - getnameinfo((struct sockaddr*)e->ai_addr,e->ai_addrlen,
>> - uaddr,INET6_ADDRSTRLEN,uport,32,
>> + getnameinfo((struct sockaddr *)e->ai_addr, e->ai_addrlen,
>> + uaddr, INET6_ADDRSTRLEN, uport, 32,
>> NI_NUMERICHOST | NI_NUMERICSERV);
> Both thsi & the above whitespace changes should be a separate
> patch from any functional changes
>
>
>> @@ -371,8 +372,28 @@ static int inet_connect_addr(const InetSocketAddress *saddr,
>> }
>> socket_set_fast_reuse(sock);
>>
>> + /* to bind the socket if src_addr is available */
>> +
>> + if (src_addr) {
>> + struct sockaddr_in servaddr;
>> +
>> + /* bind to a specific interface in the internet domain */
>> + /* to make sure the sin_zero filed is cleared */
>> + memset(&servaddr, 0, sizeof(servaddr));
>> +
>> + servaddr.sin_family = AF_INET;
>> + servaddr.sin_addr.s_addr = inet_addr(src_addr->host);
> We can't assume src_addr is a raw IPv4 address.
>
> THis needs to go through getaddrinfo in the caller like the
> dst address has done.
>
> For sanity, we should also validate that there isn't a mismatch
> of IPv4 vs IPv6 for thte src & dst addrs.
>
>> + servaddr.sin_port = 0;
>
>
>> +
>> + if (bind(sock, (struct sockaddr *)&servaddr, sizeof(servaddr)) < 0) {
>> + error_setg_errno(errp, errno, "Failed to bind socket");
>> + return -1;
>> + }
>> + }
>> +
>> /* connect to peer */
>> do {
>> +
>> rc = 0;
>> if (connect(sock, addr->ai_addr, addr->ai_addrlen) < 0) {
>> rc = -errno;
>> @@ -380,8 +401,14 @@ static int inet_connect_addr(const InetSocketAddress *saddr,
>> } while (rc == -EINTR);
>>
>> if (rc < 0) {
>> - error_setg_errno(errp, errno, "Failed to connect to '%s:%s'",
>> - saddr->host, saddr->port);
>> + if (src_addr) {
>> + error_setg_errno(errp, errno, "Failed to connect '%s:%s' to "
>> + "'%s:%s'", dst_addr->host, dst_addr->port,
>> + src_addr->host, src_addr->port);
>> + } else {
>> + error_setg_errno(errp, errno, "Failed to connect '%s:%s'",
>> + dst_addr->host, dst_addr->port);
>> + }
>> closesocket(sock);
>> return -1;
>> }
>> @@ -506,7 +534,7 @@ static int inet_dgram_saddr(InetSocketAddress *sraddr,
>> Error *err = NULL;
>>
>> /* lookup peer addr */
>> - memset(&ai,0, sizeof(ai));
>> + memset(&ai, 0, sizeof(ai));
>> ai.ai_flags = AI_CANONNAME | AI_V4MAPPED | AI_ADDRCONFIG;
>> ai.ai_family = inet_ai_family_from_address(sraddr, &err);
>> ai.ai_socktype = SOCK_DGRAM;
>> @@ -533,7 +561,7 @@ static int inet_dgram_saddr(InetSocketAddress *sraddr,
>> }
>>
>> /* lookup local addr */
>> - memset(&ai,0, sizeof(ai));
>> + memset(&ai, 0, sizeof(ai));
>> ai.ai_flags = AI_PASSIVE;
>> ai.ai_family = peer->ai_family;
>> ai.ai_socktype = SOCK_DGRAM;
>
>> @@ -574,7 +602,7 @@ static int inet_dgram_saddr(InetSocketAddress *sraddr,
>> }
>>
>> /* connect to peer */
>> - if (connect(sock,peer->ai_addr,peer->ai_addrlen) < 0) {
>> + if (connect(sock, peer->ai_addr, peer->ai_addrlen) < 0) {
>> error_setg_errno(errp, errno, "Failed to connect to '%s:%s'",
>> addr, port);
>> goto err;
> More whitespace changes for a separate patch
>
>
>
>
> With regards,
> Daniel
[-- Attachment #2: Type: text/html, Size: 17714 bytes --]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 4/4] Adding support for multi-FD connections dynamically
2022-06-16 18:47 ` Dr. David Alan Gilbert
@ 2022-06-21 16:12 ` manish.mishra
0 siblings, 0 replies; 34+ messages in thread
From: manish.mishra @ 2022-06-21 16:12 UTC (permalink / raw)
To: Dr. David Alan Gilbert, Het Gala
Cc: qemu-devel, quintela, pbonzini, berrange, armbru, eblake
On 17/06/22 12:17 am, Dr. David Alan Gilbert wrote:
> * Het Gala (het.gala@nutanix.com) wrote:
>> i) Dynamically decide appropriate source and destination ip pairs for the
>> corresponding multi-FD channel to be connected.
>>
>> ii) Removed the support for setting the number of multi-fd channels from qmp
>> commands. As now all multiFD parameters will be passed via qmp: migrate
>> command or incoming flag itself.
> We can't do that, because it's part of the API already; what you'll need
> to do is check that the number of entries in your list corresponds to
> the value set there and error if it's different.
>
> Dave
>
thanks for review David. Yes, we will make sure in V2 that nothing existing breaks.
- Manish Mishra
>> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
>> Signed-off-by: Het Gala <het.gala@nutanix.com>
>> ---
>> migration/migration.c | 15 ---------------
>> migration/migration.h | 1 -
>> migration/multifd.c | 42 +++++++++++++++++++++---------------------
>> migration/socket.c | 42 +++++++++++++++++++++++++++++++++---------
>> migration/socket.h | 4 +++-
>> monitor/hmp-cmds.c | 4 ----
>> qapi/migration.json | 6 ------
>> 7 files changed, 57 insertions(+), 57 deletions(-)
>>
>> diff --git a/migration/migration.c b/migration/migration.c
>> index 9b0ad732e7..57dd4494b4 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -1585,9 +1585,6 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
>> if (params->has_block_incremental) {
>> dest->block_incremental = params->block_incremental;
>> }
>> - if (params->has_multifd_channels) {
>> - dest->multifd_channels = params->multifd_channels;
>> - }
>> if (params->has_multifd_compression) {
>> dest->multifd_compression = params->multifd_compression;
>> }
>> @@ -1702,9 +1699,6 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
>> if (params->has_block_incremental) {
>> s->parameters.block_incremental = params->block_incremental;
>> }
>> - if (params->has_multifd_channels) {
>> - s->parameters.multifd_channels = params->multifd_channels;
>> - }
>> if (params->has_multifd_compression) {
>> s->parameters.multifd_compression = params->multifd_compression;
>> }
>> @@ -2686,15 +2680,6 @@ bool migrate_pause_before_switchover(void)
>> MIGRATION_CAPABILITY_PAUSE_BEFORE_SWITCHOVER];
>> }
>>
>> -int migrate_multifd_channels(void)
>> -{
>> - MigrationState *s;
>> -
>> - s = migrate_get_current();
>> -
>> - return s->parameters.multifd_channels;
>> -}
>> -
>> MultiFDCompression migrate_multifd_compression(void)
>> {
>> MigrationState *s;
>> diff --git a/migration/migration.h b/migration/migration.h
>> index fa8717ec9e..9464de8ef7 100644
>> --- a/migration/migration.h
>> +++ b/migration/migration.h
>> @@ -372,7 +372,6 @@ bool migrate_validate_uuid(void);
>> bool migrate_auto_converge(void);
>> bool migrate_use_multifd(void);
>> bool migrate_pause_before_switchover(void);
>> -int migrate_multifd_channels(void);
>> MultiFDCompression migrate_multifd_compression(void);
>> int migrate_multifd_zlib_level(void);
>> int migrate_multifd_zstd_level(void);
>> diff --git a/migration/multifd.c b/migration/multifd.c
>> index 9282ab6aa4..ce017436fb 100644
>> --- a/migration/multifd.c
>> +++ b/migration/multifd.c
>> @@ -225,7 +225,7 @@ static int multifd_recv_initial_packet(QIOChannel *c, Error **errp)
>> return -1;
>> }
>>
>> - if (msg.id > migrate_multifd_channels()) {
>> + if (msg.id > total_multifd_channels()) {
>> error_setg(errp, "multifd: received channel version %u "
>> "expected %u", msg.version, MULTIFD_VERSION);
>> return -1;
>> @@ -410,8 +410,8 @@ static int multifd_send_pages(QEMUFile *f)
>> * using more channels, so ensure it doesn't overflow if the
>> * limit is lower now.
>> */
>> - next_channel %= migrate_multifd_channels();
>> - for (i = next_channel;; i = (i + 1) % migrate_multifd_channels()) {
>> + next_channel %= total_multifd_channels();
>> + for (i = next_channel;; i = (i + 1) % total_multifd_channels()) {
>> p = &multifd_send_state->params[i];
>>
>> qemu_mutex_lock(&p->mutex);
>> @@ -422,7 +422,7 @@ static int multifd_send_pages(QEMUFile *f)
>> }
>> if (!p->pending_job) {
>> p->pending_job++;
>> - next_channel = (i + 1) % migrate_multifd_channels();
>> + next_channel = (i + 1) % total_multifd_channels();
>> break;
>> }
>> qemu_mutex_unlock(&p->mutex);
>> @@ -500,7 +500,7 @@ static void multifd_send_terminate_threads(Error *err)
>> return;
>> }
>>
>> - for (i = 0; i < migrate_multifd_channels(); i++) {
>> + for (i = 0; i < total_multifd_channels(); i++) {
>> MultiFDSendParams *p = &multifd_send_state->params[i];
>>
>> qemu_mutex_lock(&p->mutex);
>> @@ -521,14 +521,14 @@ void multifd_save_cleanup(void)
>> return;
>> }
>> multifd_send_terminate_threads(NULL);
>> - for (i = 0; i < migrate_multifd_channels(); i++) {
>> + for (i = 0; i < total_multifd_channels(); i++) {
>> MultiFDSendParams *p = &multifd_send_state->params[i];
>>
>> if (p->running) {
>> qemu_thread_join(&p->thread);
>> }
>> }
>> - for (i = 0; i < migrate_multifd_channels(); i++) {
>> + for (i = 0; i < total_multifd_channels(); i++) {
>> MultiFDSendParams *p = &multifd_send_state->params[i];
>> Error *local_err = NULL;
>>
>> @@ -594,7 +594,7 @@ int multifd_send_sync_main(QEMUFile *f)
>>
>> flush_zero_copy = migrate_use_zero_copy_send();
>>
>> - for (i = 0; i < migrate_multifd_channels(); i++) {
>> + for (i = 0; i < total_multifd_channels(); i++) {
>> MultiFDSendParams *p = &multifd_send_state->params[i];
>>
>> trace_multifd_send_sync_main_signal(p->id);
>> @@ -627,7 +627,7 @@ int multifd_send_sync_main(QEMUFile *f)
>> }
>> }
>> }
>> - for (i = 0; i < migrate_multifd_channels(); i++) {
>> + for (i = 0; i < total_multifd_channels(); i++) {
>> MultiFDSendParams *p = &multifd_send_state->params[i];
>>
>> trace_multifd_send_sync_main_wait(p->id);
>> @@ -903,7 +903,7 @@ int multifd_save_setup(Error **errp)
>> int thread_count;
>> uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
>> uint8_t i;
>> -
>> + int idx;
>> if (!migrate_use_multifd()) {
>> return 0;
>> }
>> @@ -912,7 +912,7 @@ int multifd_save_setup(Error **errp)
>> return -1;
>> }
>>
>> - thread_count = migrate_multifd_channels();
>> + thread_count = total_multifd_channels();
>> multifd_send_state = g_malloc0(sizeof(*multifd_send_state));
>> multifd_send_state->params = g_new0(MultiFDSendParams, thread_count);
>> multifd_send_state->pages = multifd_pages_init(page_count);
>> @@ -945,8 +945,8 @@ int multifd_save_setup(Error **errp)
>> } else {
>> p->write_flags = 0;
>> }
>> -
>> - socket_send_channel_create(multifd_new_send_channel_async, p);
>> + idx = multifd_index(i);
>> + socket_send_channel_create(multifd_new_send_channel_async, p, idx);
>> }
>>
>> for (i = 0; i < thread_count; i++) {
>> @@ -991,7 +991,7 @@ static void multifd_recv_terminate_threads(Error *err)
>> }
>> }
>>
>> - for (i = 0; i < migrate_multifd_channels(); i++) {
>> + for (i = 0; i < total_multifd_channels(); i++) {
>> MultiFDRecvParams *p = &multifd_recv_state->params[i];
>>
>> qemu_mutex_lock(&p->mutex);
>> @@ -1017,7 +1017,7 @@ int multifd_load_cleanup(Error **errp)
>> return 0;
>> }
>> multifd_recv_terminate_threads(NULL);
>> - for (i = 0; i < migrate_multifd_channels(); i++) {
>> + for (i = 0; i < total_multifd_channels(); i++) {
>> MultiFDRecvParams *p = &multifd_recv_state->params[i];
>>
>> if (p->running) {
>> @@ -1030,7 +1030,7 @@ int multifd_load_cleanup(Error **errp)
>> qemu_thread_join(&p->thread);
>> }
>> }
>> - for (i = 0; i < migrate_multifd_channels(); i++) {
>> + for (i = 0; i < total_multifd_channels(); i++) {
>> MultiFDRecvParams *p = &multifd_recv_state->params[i];
>>
>> migration_ioc_unregister_yank(p->c);
>> @@ -1065,13 +1065,13 @@ void multifd_recv_sync_main(void)
>> if (!migrate_use_multifd()) {
>> return;
>> }
>> - for (i = 0; i < migrate_multifd_channels(); i++) {
>> + for (i = 0; i < total_multifd_channels(); i++) {
>> MultiFDRecvParams *p = &multifd_recv_state->params[i];
>>
>> trace_multifd_recv_sync_main_wait(p->id);
>> qemu_sem_wait(&multifd_recv_state->sem_sync);
>> }
>> - for (i = 0; i < migrate_multifd_channels(); i++) {
>> + for (i = 0; i < total_multifd_channels(); i++) {
>> MultiFDRecvParams *p = &multifd_recv_state->params[i];
>>
>> WITH_QEMU_LOCK_GUARD(&p->mutex) {
>> @@ -1166,7 +1166,7 @@ int multifd_load_setup(Error **errp)
>> error_setg(errp, "multifd is not supported by current protocol");
>> return -1;
>> }
>> - thread_count = migrate_multifd_channels();
>> + thread_count = total_multifd_channels();
>> multifd_recv_state = g_malloc0(sizeof(*multifd_recv_state));
>> multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count);
>> qatomic_set(&multifd_recv_state->count, 0);
>> @@ -1204,7 +1204,7 @@ int multifd_load_setup(Error **errp)
>>
>> bool multifd_recv_all_channels_created(void)
>> {
>> - int thread_count = migrate_multifd_channels();
>> + int thread_count = total_multifd_channels();
>>
>> if (!migrate_use_multifd()) {
>> return true;
>> @@ -1259,5 +1259,5 @@ bool multifd_recv_new_channel(QIOChannel *ioc, Error **errp)
>> QEMU_THREAD_JOINABLE);
>> qatomic_inc(&multifd_recv_state->count);
>> return qatomic_read(&multifd_recv_state->count) ==
>> - migrate_multifd_channels();
>> + total_multifd_channels();
>> }
>> diff --git a/migration/socket.c b/migration/socket.c
>> index d0cb7cc6a6..c0ac6dbbe2 100644
>> --- a/migration/socket.c
>> +++ b/migration/socket.c
>> @@ -28,9 +28,6 @@
>> #include "trace.h"
>>
>>
>> -struct SocketOutgoingArgs {
>> - SocketAddress *saddr;
>> -} outgoing_args;
>>
>> struct SocketArgs {
>> struct SrcDestAddr data;
>> @@ -43,20 +40,47 @@ struct OutgoingMigrateParams {
>> uint64_t total_multifd_channel;
>> } outgoing_migrate_params;
>>
>> -void socket_send_channel_create(QIOTaskFunc f, void *data)
>> +
>> +int total_multifd_channels(void)
>> +{
>> + return outgoing_migrate_params.total_multifd_channel;
>> +}
>> +
>> +int multifd_index(int i)
>> +{
>> + int length = outgoing_migrate_params.length;
>> + int j = 0;
>> + int runn_sum = 0;
>> + while (j < length) {
>> + runn_sum += outgoing_migrate_params.socket_args[j].multifd_channels;
>> + if (i >= runn_sum) {
>> + j++;
>> + } else {
>> + break;
>> + }
>> + }
>> + return j;
>> +}
>> +
>> +void socket_send_channel_create(QIOTaskFunc f, void *data, int idx)
>> {
>> QIOChannelSocket *sioc = qio_channel_socket_new();
>> - qio_channel_socket_connect_async(sioc, outgoing_args.saddr,
>> - f, data, NULL, NULL, NULL);
>> + qio_channel_socket_connect_async(sioc,
>> + outgoing_migrate_params.socket_args[idx].data.dst_addr,
>> + f, data, NULL, NULL,
>> + outgoing_migrate_params.socket_args[idx].data.src_addr);
>> }
>>
>> int socket_send_channel_destroy(QIOChannel *send)
>> {
>> /* Remove channel */
>> object_unref(OBJECT(send));
>> - if (outgoing_args.saddr) {
>> - qapi_free_SocketAddress(outgoing_args.saddr);
>> - outgoing_args.saddr = NULL;
>> + if (outgoing_migrate_params.socket_args != NULL) {
>> + g_free(outgoing_migrate_params.socket_args);
>> + outgoing_migrate_params.socket_args = NULL;
>> + }
>> + if (outgoing_migrate_params.length) {
>> + outgoing_migrate_params.length = 0;
>> }
>>
>> if (outgoing_migrate_params.socket_args != NULL) {
>> diff --git a/migration/socket.h b/migration/socket.h
>> index b9e3699167..c8b9252384 100644
>> --- a/migration/socket.h
>> +++ b/migration/socket.h
>> @@ -27,7 +27,9 @@ struct SrcDestAddr {
>> SocketAddress *src_addr;
>> };
>>
>> -void socket_send_channel_create(QIOTaskFunc f, void *data);
>> +int total_multifd_channels(void);
>> +int multifd_index(int i);
>> +void socket_send_channel_create(QIOTaskFunc f, void *data, int idx);
>> int socket_send_channel_destroy(QIOChannel *send);
>>
>> void socket_start_incoming_migration(const char *str, uint8_t number,
>> diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
>> index 32a6b67d5f..9a3d76d6ba 100644
>> --- a/monitor/hmp-cmds.c
>> +++ b/monitor/hmp-cmds.c
>> @@ -1281,10 +1281,6 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
>> p->has_block_incremental = true;
>> visit_type_bool(v, param, &p->block_incremental, &err);
>> break;
>> - case MIGRATION_PARAMETER_MULTIFD_CHANNELS:
>> - p->has_multifd_channels = true;
>> - visit_type_uint8(v, param, &p->multifd_channels, &err);
>> - break;
>> case MIGRATION_PARAMETER_MULTIFD_COMPRESSION:
>> p->has_multifd_compression = true;
>> visit_type_MultiFDCompression(v, param, &p->multifd_compression,
>> diff --git a/qapi/migration.json b/qapi/migration.json
>> index 62a7b22d19..1b1c6d01d3 100644
>> --- a/qapi/migration.json
>> +++ b/qapi/migration.json
>> @@ -877,11 +877,6 @@
>> # migrated and the destination must already have access to the
>> # same backing chain as was used on the source. (since 2.10)
>> #
>> -# @multifd-channels: Number of channels used to migrate data in
>> -# parallel. This is the same number that the
>> -# number of sockets used for migration. The
>> -# default value is 2 (since 4.0)
>> -#
>> # @xbzrle-cache-size: cache size to be used by XBZRLE migration. It
>> # needs to be a multiple of the target page size
>> # and a power of 2
>> @@ -965,7 +960,6 @@
>> '*x-checkpoint-delay': { 'type': 'uint32',
>> 'features': [ 'unstable' ] },
>> '*block-incremental': 'bool',
>> - '*multifd-channels': 'uint8',
>> '*xbzrle-cache-size': 'size',
>> '*max-postcopy-bandwidth': 'size',
>> '*max-cpu-throttle': 'uint8',
>> --
>> 2.22.3
>>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 0/4] Multiple interface support on top of Multi-FD
2022-06-16 15:50 ` Dr. David Alan Gilbert
@ 2022-06-21 16:16 ` manish.mishra
0 siblings, 0 replies; 34+ messages in thread
From: manish.mishra @ 2022-06-21 16:16 UTC (permalink / raw)
To: Dr. David Alan Gilbert, Daniel P. Berrangé
Cc: Het Gala, qemu-devel, quintela, pbonzini, armbru, eblake
On 16/06/22 9:20 pm, Dr. David Alan Gilbert wrote:
> * Daniel P. Berrangé (berrange@redhat.com) wrote:
>> On Wed, Jun 15, 2022 at 05:43:28PM +0100, Daniel P. Berrangé wrote:
>>> On Fri, Jun 10, 2022 at 05:58:31PM +0530, manish.mishra wrote:
>>>> On 09/06/22 9:17 pm, Daniel P. Berrangé wrote:
>>>>> On Thu, Jun 09, 2022 at 07:33:01AM +0000, Het Gala wrote:
>>>>>> As of now, the multi-FD feature supports connection over the default network
>>>>>> only. This Patchset series is a Qemu side implementation of providing multiple
>>>>>> interfaces support for multi-FD. This enables us to fully utilize dedicated or
>>>>>> multiple NICs in case bonding of NICs is not possible.
>>>>>>
>>>>>>
>>>>>> Introduction
>>>>>> -------------
>>>>>> Multi-FD Qemu implementation currently supports connection only on the default
>>>>>> network. This forbids us from advantages like:
>>>>>> - Separating VM live migration traffic from the default network.
>>>> Hi Daniel,
>>>>
>>>> I totally understand your concern around this approach increasing compexity inside qemu,
>>>>
>>>> when similar things can be done with NIC teaming. But we thought this approach provides
>>>>
>>>> much more flexibility to user in few cases like.
>>>>
>>>> 1. We checked our customer data, almost all of the host had multiple NIC, but LACP support
>>>>
>>>> in their setups was very rare. So for those cases this approach can help in utilise multiple
>>>>
>>>> NICs as teaming is not possible there.
>>> AFAIK, LACP is not required in order to do link aggregation with Linux.
>>> Traditional Linux bonding has no special NIC hardware or switch requirements,
>>> so LACP is merely a "nice to have" in order to simplify some aspects.
>>>
>>> IOW, migration with traffic spread across multiple NICs is already
>>> possible AFAICT.
>>>
>>> I can understand that some people may not have actually configured
>>> bonding on their hosts, but it is not unreasonable to request that
>>> they do so, if they want to take advantage fo aggrated bandwidth.
>>>
>>> It has the further benefit that it will be fault tolerant. With
>>> this proposal if any single NIC has a problem, the whole migration
>>> will get stuck. With kernel level bonding, if any single NIC haus
>>> a problem, it'll get offlined by the kernel and migration will
>>> continue to work across remaining active NICs.
>>>
>>>> 2. We have seen requests recently to separate out traffic of storage, VM netwrok, migration
>>>>
>>>> over different vswitch which can be backed by 1 or more NICs as this give better
>>>>
>>>> predictability and assurance. So host with multiple ips/vswitches can be very common
>>>>
>>>> environment. In this kind of enviroment this approach gives per vm or migration level
>>>>
>>>> flexibilty, like for critical VM we can still use bandwidth from all available vswitch/interface
>>>>
>>>> but for normal VM they can keep live migration only on dedicated NICs without changing
>>>>
>>>> complete host network topology.
>>>>
>>>> At final we want it to be something like this [<ip-pair>, <multiFD-channels>, <bandwidth_control>]
>>>>
>>>> to provide bandwidth_control per interface.
>>> Again, it is already possible to separate migration traffic from storage
>>> traffic, from other network traffic. The target IP given will influence
>>> which NIC is used based on routing table and I know this is already
>>> done widely with OpenStack deployments.
>> Actually I should clarify this is only practical if the two NICs are
>> using different IP subnets, otherwise routing rules are not viable.
>> So needing to set source IP would be needed to select between a pair
>> of NICs on the same IP subnet.
> Yeh so I think that's one reason that the idea in this series is OK
> (together with the idea for the NUMA stuff) and I suspect there are
> other cases as well.
>
> Dave
>
yes, David multiFD per NUMA seems interesting idea, I was just curious
how much throughput diff we can experience per multiFD channel
with local vs remote NIC?
thanks
Manish Mishra
>> Previous usage I've seen has always setup fully distinct IP subnets
>> for generic vs storage vs migration network traffic.
>>
>> With regards,
>> Daniel
>> --
>> |: https://urldefense.proofpoint.com/v2/url?u=https-3A__berrange.com&d=DwIDAw&c=s883GpUCOChKOHiocYtGcg&r=c4KON2DiMd-szjwjggQcuUvTsPWblztAL0gVzaHnNmc&m=qfclRDP-GXttuWQ3knJS2RHXmg2XjmG7Pju002cBrHugZE8hpO3DRbKdHphItFr-&s=1RKIz6cO82_JwgkJ-QLP3SRWaG2Lo6J8w4O0Z2YVJ4Q&e= -o- https://urldefense.proofpoint.com/v2/url?u=https-3A__www.flickr.com_photos_dberrange&d=DwIDAw&c=s883GpUCOChKOHiocYtGcg&r=c4KON2DiMd-szjwjggQcuUvTsPWblztAL0gVzaHnNmc&m=qfclRDP-GXttuWQ3knJS2RHXmg2XjmG7Pju002cBrHugZE8hpO3DRbKdHphItFr-&s=BkGiCLXloxlYYBJeJ_0XGRUgkUraRPJdIu26ukR6erI&e= :|
>> |: https://urldefense.proofpoint.com/v2/url?u=https-3A__libvirt.org&d=DwIDAw&c=s883GpUCOChKOHiocYtGcg&r=c4KON2DiMd-szjwjggQcuUvTsPWblztAL0gVzaHnNmc&m=qfclRDP-GXttuWQ3knJS2RHXmg2XjmG7Pju002cBrHugZE8hpO3DRbKdHphItFr-&s=KOz_zQXuQzFxwhNLINm-FrADPcBgnVjjmULmZ6iZTi4&e= -o- https://urldefense.proofpoint.com/v2/url?u=https-3A__fstop138.berrange.com&d=DwIDAw&c=s883GpUCOChKOHiocYtGcg&r=c4KON2DiMd-szjwjggQcuUvTsPWblztAL0gVzaHnNmc&m=qfclRDP-GXttuWQ3knJS2RHXmg2XjmG7Pju002cBrHugZE8hpO3DRbKdHphItFr-&s=Ez_j93m7dz0aJe9mjyynk8mJ122ZeXre2F-ylFXj2og&e= :|
>> |: https://urldefense.proofpoint.com/v2/url?u=https-3A__entangle-2Dphoto.org&d=DwIDAw&c=s883GpUCOChKOHiocYtGcg&r=c4KON2DiMd-szjwjggQcuUvTsPWblztAL0gVzaHnNmc&m=qfclRDP-GXttuWQ3knJS2RHXmg2XjmG7Pju002cBrHugZE8hpO3DRbKdHphItFr-&s=ID9hDsAkt6zO_o85XqDIjhoxLiwrOhyfAhEqJSukAbw&e= -o- https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instagram.com_dberrange&d=DwIDAw&c=s883GpUCOChKOHiocYtGcg&r=c4KON2DiMd-szjwjggQcuUvTsPWblztAL0gVzaHnNmc&m=qfclRDP-GXttuWQ3knJS2RHXmg2XjmG7Pju002cBrHugZE8hpO3DRbKdHphItFr-&s=EK5bGerh1gLCXnMyUV1FlC8EyMN2lWa-r1MVxp6_A_s&e= :|
>>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair
2022-06-16 17:26 ` Dr. David Alan Gilbert
@ 2022-07-13 8:08 ` Het Gala
2022-07-15 8:07 ` Het Gala
2022-07-13 12:54 ` Claudio Fontana
1 sibling, 1 reply; 34+ messages in thread
From: Het Gala @ 2022-07-13 8:08 UTC (permalink / raw)
To: Dr. David Alan Gilbert, cfontana
Cc: qemu-devel, quintela, pbonzini, berrange, armbru, eblake, Manish Mishra
On 16/06/22 10:56 pm, Dr. David Alan Gilbert wrote:
> * Het Gala (het.gala@nutanix.com) wrote:
> First of all, I apologise for the late reply. I was on a leave after
internship ended
at Nutanix. Hope to learn a lot from you all in the process of
upstreaming multifd
patches.
>> i) Modified the format of the qemu monitor command : 'migrate' by adding a list,
>> each element in the list consists of multi-FD connection parameters: source
>> and destination uris and of the number of multi-fd channels between each pair.
>>
>> ii) Information of all multi-FD connection parameters’ list, length of the list
>> and total number of multi-fd channels for all the connections together is
>> stored in ‘OutgoingArgs’ struct.
>>
>> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
>> Signed-off-by: Het Gala <het.gala@nutanix.com>
>> ---
>> include/qapi/util.h | 9 ++++++++
>> migration/migration.c | 47 ++++++++++++++++++++++++++++++--------
>> migration/socket.c | 53 ++++++++++++++++++++++++++++++++++++++++---
>> migration/socket.h | 17 +++++++++++++-
>> monitor/hmp-cmds.c | 22 ++++++++++++++++--
>> qapi/migration.json | 43 +++++++++++++++++++++++++++++++----
>> 6 files changed, 170 insertions(+), 21 deletions(-)
>>
>> diff --git a/include/qapi/util.h b/include/qapi/util.h
>> index 81a2b13a33..3041feb3d9 100644
>> --- a/include/qapi/util.h
>> +++ b/include/qapi/util.h
>> @@ -56,4 +56,13 @@ int parse_qapi_name(const char *name, bool complete);
>> (tail) = &(*(tail))->next; \
>> } while (0)
>>
>> +#define QAPI_LIST_LENGTH(list) ({ \
>> + int _len = 0; \
>> + typeof(list) _elem; \
>> + for (_elem = list; _elem != NULL; _elem = _elem->next) { \
>> + _len++; \
>> + } \
>> + _len; \
>> +})
>> +
>> #endif
> This looks like it should be a separate patch to me (and perhaps size_t
> for len?)
> Sure, will try to make a seperate patch for QAPI_LIST_LENGTH, and other
such utility functions from the other patches.
>
>> diff --git a/migration/migration.c b/migration/migration.c
>> index 31739b2af9..c408175aeb 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -2328,13 +2328,14 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,
>> return true;
>> }
>>
>> -void qmp_migrate(const char *uri, bool has_blk, bool blk,
>> +void qmp_migrate(const char *uri, bool has_multi_fd_uri_list,
>> + MigrateUriParameterList *cap, bool has_blk, bool blk,
>> bool has_inc, bool inc, bool has_detach, bool detach,
>> bool has_resume, bool resume, Error **errp)
>> {
>> Error *local_err = NULL;
>> MigrationState *s = migrate_get_current();
>> - const char *p = NULL;
>> + const char *dst_ptr = NULL;
>>
>> if (!migrate_prepare(s, has_blk && blk, has_inc && inc,
>> has_resume && resume, errp)) {
>> @@ -2348,20 +2349,46 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
>> }
>> }
>>
>> + /*
>> + * In case of Multi-FD migration parameters, if uri is provided,
> I think you mean 'if uri list is provided'
> Acknowledged.
>
>> + * supports only tcp network protocol.
>> + */
>> + if (has_multi_fd_uri_list) {
>> + int length = QAPI_LIST_LENGTH(cap);
>> + init_multifd_array(length);
>> + for (int i = 0; i < length; i++) {
>> + const char *p1 = NULL, *p2 = NULL;
> Keep these as ps/pd to make it clear which is source and dest.
> Acknowledged. Will change in the upcoming patchset.
>
>> + const char *multifd_dst_uri = cap->value->destination_uri;
>> + const char *multifd_src_uri = cap->value->source_uri;
>> + uint8_t multifd_channels = cap->value->multifd_channels;
>> + if (!strstart(multifd_dst_uri, "tcp:", &p1) ||
>> + !strstart(multifd_src_uri, "tcp:", &p2)) {
> I've copied in Claudio Fontana; Claudio is fighting to make snapshots
> faster and has been playing with various multithread schemes for multifd
> with files and fd's; perhaps the syntax you're proposing doesn't need
> to be limited to tcp.
> For now, we are just aiming to include multifd for existing tcp
protocol.
We would be happy to take any suggestions from Claudio Fontana and try to
include them in the upcoming patchset series.
>
>> + error_setg(errp, "multi-fd destination and multi-fd source "
>> + "uri, both should be present and follows tcp protocol only");
>> + break;
>> + } else {
>> + store_multifd_migration_params(p1 ? p1 : multifd_dst_uri,
>> + p2 ? p2 : multifd_src_uri,
>> + multifd_channels, i, &local_err);
>> + }
>> + cap = cap->next;
>> + }
>> + }
>> +
>> migrate_protocol_allow_multi_channels(false);
>> - if (strstart(uri, "tcp:", &p) ||
>> + if (strstart(uri, "tcp:", &dst_ptr) ||
>> strstart(uri, "unix:", NULL) ||
>> strstart(uri, "vsock:", NULL)) {
>> migrate_protocol_allow_multi_channels(true);
>> - socket_start_outgoing_migration(s, p ? p : uri, &local_err);
>> + socket_start_outgoing_migration(s, dst_ptr ? dst_ptr : uri, &local_err);
>> #ifdef CONFIG_RDMA
>> - } else if (strstart(uri, "rdma:", &p)) {
>> - rdma_start_outgoing_migration(s, p, &local_err);
>> + } else if (strstart(uri, "rdma:", &dst_ptr)) {
>> + rdma_start_outgoing_migration(s, dst_ptr, &local_err);
>> #endif
>> - } else if (strstart(uri, "exec:", &p)) {
>> - exec_start_outgoing_migration(s, p, &local_err);
>> - } else if (strstart(uri, "fd:", &p)) {
>> - fd_start_outgoing_migration(s, p, &local_err);
>> + } else if (strstart(uri, "exec:", &dst_ptr)) {
>> + exec_start_outgoing_migration(s, dst_ptr, &local_err);
>> + } else if (strstart(uri, "fd:", &dst_ptr)) {
>> + fd_start_outgoing_migration(s, dst_ptr, &local_err);
>> } else {
>> if (!(has_resume && resume)) {
>> yank_unregister_instance(MIGRATION_YANK_INSTANCE);
>> diff --git a/migration/socket.c b/migration/socket.c
>> index 4fd5e85f50..7ca6af8cca 100644
>> --- a/migration/socket.c
>> +++ b/migration/socket.c
>> @@ -32,6 +32,17 @@ struct SocketOutgoingArgs {
>> SocketAddress *saddr;
>> } outgoing_args;
>>
>> +struct SocketArgs {
>> + struct SrcDestAddr data;
> 'data' is an odd name; 'addresses' perhaps?
> Sure, Acknowledged.
>
>> + uint8_t multifd_channels;
>> +};
>> +
>> +struct OutgoingMigrateParams {
>> + struct SocketArgs *socket_args;
>> + size_t length;
>> + uint64_t total_multifd_channel;
>> +} outgoing_migrate_params;
>> +
>> void socket_send_channel_create(QIOTaskFunc f, void *data)
>> {
>> QIOChannelSocket *sioc = qio_channel_socket_new();
>> @@ -47,6 +58,14 @@ int socket_send_channel_destroy(QIOChannel *send)
>> qapi_free_SocketAddress(outgoing_args.saddr);
>> outgoing_args.saddr = NULL;
>> }
>> +
>> + if (outgoing_migrate_params.socket_args != NULL) {
>> + g_free(outgoing_migrate_params.socket_args);
>> + outgoing_migrate_params.socket_args = NULL;
> I think g_free is safe on NULL; so I think you can just do this without
> the if.
> Okay, thanks for the suggestion there David.
>
>> + }
>> + if (outgoing_migrate_params.length) {
> Does that ever differ from the != NULL test ?
> I think you can always just set this to 0 without the test.
> Sure.
>
>> + outgoing_migrate_params.length = 0;
>> + }
>> return 0;
>> }
>>
>> @@ -117,13 +136,41 @@ socket_start_outgoing_migration_internal(MigrationState *s,
>> }
>>
>> void socket_start_outgoing_migration(MigrationState *s,
>> - const char *str,
>> + const char *dst_str,
>> Error **errp)
>> {
>> Error *err = NULL;
>> - SocketAddress *saddr = socket_parse(str, &err);
>> + SocketAddress *dst_saddr = socket_parse(dst_str, &err);
>> + if (!err) {
>> + socket_start_outgoing_migration_internal(s, dst_saddr, &err);
>> + }
>> + error_propagate(errp, err);
>> +}
>> +
>> +void init_multifd_array(int length)
>> +{
>> + outgoing_migrate_params.socket_args = g_new0(struct SocketArgs, length);
>> + outgoing_migrate_params.length = length;
>> + outgoing_migrate_params.total_multifd_channel = 0;
>> +}
>> +
>> +void store_multifd_migration_params(const char *dst_uri,
>> + const char *src_uri,
>> + uint8_t multifd_channels,
>> + int idx, Error **errp)
>> +{
>> + Error *err = NULL;
>> + SocketAddress *src_addr = NULL;
>> + SocketAddress *dst_addr = socket_parse(dst_uri, &err);
>> + if (src_uri) {
>> + src_addr = socket_parse(src_uri, &err);
>> + }
>> if (!err) {
>> - socket_start_outgoing_migration_internal(s, saddr, &err);
>> + outgoing_migrate_params.socket_args[idx].data.dst_addr = dst_addr;
>> + outgoing_migrate_params.socket_args[idx].data.src_addr = src_addr;
>> + outgoing_migrate_params.socket_args[idx].multifd_channels
>> + = multifd_channels;
>> + outgoing_migrate_params.total_multifd_channel += multifd_channels;
>> }
>> error_propagate(errp, err);
>> }
>> diff --git a/migration/socket.h b/migration/socket.h
>> index 891dbccceb..bba7f177fe 100644
>> --- a/migration/socket.h
>> +++ b/migration/socket.h
>> @@ -19,12 +19,27 @@
>>
>> #include "io/channel.h"
>> #include "io/task.h"
>> +#include "migration.h"
>> +
>> +/* info regarding destination and source uri */
>> +struct SrcDestAddr {
>> + SocketAddress *dst_addr;
>> + SocketAddress *src_addr;
>> +};
>>
>> void socket_send_channel_create(QIOTaskFunc f, void *data);
>> int socket_send_channel_destroy(QIOChannel *send);
>>
>> void socket_start_incoming_migration(const char *str, Error **errp);
>>
>> -void socket_start_outgoing_migration(MigrationState *s, const char *str,
>> +void socket_start_outgoing_migration(MigrationState *s, const char *dst_str,
>> Error **errp);
>> +
>> +int multifd_list_length(MigrateUriParameterList *list);
>> +
>> +void init_multifd_array(int length);
>> +
>> +void store_multifd_migration_params(const char *dst_uri, const char *src_uri,
>> + uint8_t multifd_channels, int idx,
>> + Error **erp);
>> #endif
>> diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
>> index 622c783c32..2db539016a 100644
>> --- a/monitor/hmp-cmds.c
>> +++ b/monitor/hmp-cmds.c
>> @@ -56,6 +56,9 @@
>> #include "migration/snapshot.h"
>> #include "migration/misc.h"
>>
>> +/* Default number of multi-fd channels */
>> +#define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2
>> +
>> #ifdef CONFIG_SPICE
>> #include <spice/enums.h>
>> #endif
>> @@ -1574,10 +1577,25 @@ void hmp_migrate(Monitor *mon, const QDict *qdict)
>> bool inc = qdict_get_try_bool(qdict, "inc", false);
>> bool resume = qdict_get_try_bool(qdict, "resume", false);
>> const char *uri = qdict_get_str(qdict, "uri");
>> +
>> + const char *src_uri = qdict_get_str(qdict, "source-uri");
>> + const char *dst_uri = qdict_get_str(qdict, "destination-uri");
>> + uint8_t multifd_channels = qdict_get_try_int(qdict, "multifd-channels",
>> + DEFAULT_MIGRATE_MULTIFD_CHANNELS);
>> Error *err = NULL;
>> + MigrateUriParameterList *caps = NULL;
>> + MigrateUriParameter *value;
>> +
>> + value = g_malloc0(sizeof(*value));
>> + value->source_uri = (char *)src_uri;
>> + value->destination_uri = (char *)dst_uri;
>> + value->multifd_channels = multifd_channels;
>> + QAPI_LIST_PREPEND(caps, value);
>> +
>> + qmp_migrate(uri, !!caps, caps, !!blk, blk, !!inc,
>> + inc, false, false, true, resume, &err);
>> + qapi_free_MigrateUriParameterList(caps);
>>
>> - qmp_migrate(uri, !!blk, blk, !!inc, inc,
>> - false, false, true, resume, &err);
>> if (hmp_handle_error(mon, err)) {
>> return;
>> }
> Please split the HMP changes into a separate patch.
> Okay sure. Will include both on destination and source side HMP changes
into a seperate patch.
>
>> diff --git a/qapi/migration.json b/qapi/migration.json
>> index 6130cd9fae..fb259d626b 100644
>> --- a/qapi/migration.json
>> +++ b/qapi/migration.json
>> @@ -1454,12 +1454,38 @@
>> ##
>> { 'command': 'migrate-continue', 'data': {'state': 'MigrationStatus'} }
>>
>> +##
>> +# @MigrateUriParameter:
>> +#
>> +# Information regarding which source interface is connected to which
>> +# destination interface and number of multifd channels over each interface.
>> +#
>> +# @source-uri: the Uniform Resource Identifier of the source VM.
>> +# Default port number is 0.
>> +#
>> +# @destination-uri: the Uniform Resource Identifier of the destination VM
> I would just say 'uri' rather than spelling it out.
> Okay, acknowledged.
>
>> +# @multifd-channels: number of parallel multifd channels used to migrate data
>> +# for specific source-uri and destination-uri. Default value
>> +# in this case is 2 (Since 4.0)
> 7.1 at the moment.
> Thanks for pointing it out.
>
>> +#
>> +##
>> +{ 'struct' : 'MigrateUriParameter',
>> + 'data' : { 'source-uri' : 'str',
>> + 'destination-uri' : 'str',
>> + '*multifd-channels' : 'uint8'} }
> OK, so much higher level question - why do we specify both URIs on
> each end? Is it just the source that is used on the source side to say
> which NIC to route down? On the destination side I guess there's no
> need?
>
> Do we have some rule about needing to specify enough channels for all
> the multifd channels we specify (i.e. if we specify 4 multifd channels
> in the migration parameter do we have to supply 4 channels here?)
> What happens with say Peter's preemption channel?
>
> Is there some logical ordering rule; i.e. if we were to start ordering
> particular multifd threads, then can we say that we allocate these
> channels in the same order as this list?
> I certainly did not get your first point here David. On the
destination side,
I think we certainly need both, destination and source uri's for making
a connection
but on the source side, we do not require source uri, which I have not
included
if you look at the 'Adding multi-interface support for multi-FD on
destination
side' patch.
> Yes, I agree with you. I will inlcude this feature in the next
version of patchset,
where it will check the number of multifd channels coming from API and
total
multifd channel number from qmp monitor command, and should be equal.
> Yes David, multifd threads will be allocated in the same order, the
user will
specify in the qmp monitor command.
>> ##
>> # @migrate:
>> #
>> # Migrates the current running guest to another Virtual Machine.
>> #
>> # @uri: the Uniform Resource Identifier of the destination VM
>> +# for migration thread
>> +#
>> +# @multi-fd-uri-list: list of pair of source and destination VM Uniform
>> +# Resource Identifiers with number of multifd-channels
>> +# for each pair
>> #
>> # @blk: do block migration (full disk copy)
>> #
>> @@ -1479,20 +1505,27 @@
>> # 1. The 'query-migrate' command should be used to check migration's progress
>> # and final result (this information is provided by the 'status' member)
>> #
>> -# 2. All boolean arguments default to false
>> +# 2. The uri argument should have the Uniform Resource Identifier of default
>> +# destination VM. This connection will be bound to default network
>> +#
>> +# 3. All boolean arguments default to false
>> #
>> -# 3. The user Monitor's "detach" argument is invalid in QMP and should not
>> +# 4. The user Monitor's "detach" argument is invalid in QMP and should not
>> # be used
>> #
>> # Example:
>> #
>> -# -> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
>> +# -> { "execute": "migrate",
>> +# "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
>> +# "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
>> +# "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
>> +# "destination-uri": "tcp:11.0.0.0:7789", "multifd-channels": 5} ] } }
>> # <- { "return": {} }
>> #
>> ##
>> { 'command': 'migrate',
>> - 'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool',
>> - '*detach': 'bool', '*resume': 'bool' } }
>> + 'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateUriParameter'], '*blk': 'bool',
>> + '*inc': 'bool', '*detach': 'bool', '*resume': 'bool' } }
>>
>> ##
>> # @migrate-incoming:
>> --
>> 2.22.3
>>
Regards
Het Gala
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair
2022-06-16 17:26 ` Dr. David Alan Gilbert
2022-07-13 8:08 ` Het Gala
@ 2022-07-13 12:54 ` Claudio Fontana
1 sibling, 0 replies; 34+ messages in thread
From: Claudio Fontana @ 2022-07-13 12:54 UTC (permalink / raw)
To: Dr. David Alan Gilbert, Het Gala
Cc: qemu-devel, quintela, pbonzini, berrange, armbru, eblake,
Manish Mishra, Daniel P . Berrangé,
Jim Fehlig
On 6/16/22 19:26, Dr. David Alan Gilbert wrote:
> * Het Gala (het.gala@nutanix.com) wrote:
>> i) Modified the format of the qemu monitor command : 'migrate' by adding a list,
>> each element in the list consists of multi-FD connection parameters: source
>> and destination uris and of the number of multi-fd channels between each pair.
>>
>> ii) Information of all multi-FD connection parameters’ list, length of the list
>> and total number of multi-fd channels for all the connections together is
>> stored in ‘OutgoingArgs’ struct.
>>
>> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
>> Signed-off-by: Het Gala <het.gala@nutanix.com>
>> ---
>> include/qapi/util.h | 9 ++++++++
>> migration/migration.c | 47 ++++++++++++++++++++++++++++++--------
>> migration/socket.c | 53 ++++++++++++++++++++++++++++++++++++++++---
>> migration/socket.h | 17 +++++++++++++-
>> monitor/hmp-cmds.c | 22 ++++++++++++++++--
>> qapi/migration.json | 43 +++++++++++++++++++++++++++++++----
>> 6 files changed, 170 insertions(+), 21 deletions(-)
>>
>> diff --git a/include/qapi/util.h b/include/qapi/util.h
>> index 81a2b13a33..3041feb3d9 100644
>> --- a/include/qapi/util.h
>> +++ b/include/qapi/util.h
>> @@ -56,4 +56,13 @@ int parse_qapi_name(const char *name, bool complete);
>> (tail) = &(*(tail))->next; \
>> } while (0)
>>
>> +#define QAPI_LIST_LENGTH(list) ({ \
>> + int _len = 0; \
>> + typeof(list) _elem; \
>> + for (_elem = list; _elem != NULL; _elem = _elem->next) { \
>> + _len++; \
>> + } \
>> + _len; \
>> +})
>> +
>> #endif
>
> This looks like it should be a separate patch to me (and perhaps size_t
> for len?)
>
>> diff --git a/migration/migration.c b/migration/migration.c
>> index 31739b2af9..c408175aeb 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -2328,13 +2328,14 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,
>> return true;
>> }
>>
>> -void qmp_migrate(const char *uri, bool has_blk, bool blk,
>> +void qmp_migrate(const char *uri, bool has_multi_fd_uri_list,
>> + MigrateUriParameterList *cap, bool has_blk, bool blk,
>> bool has_inc, bool inc, bool has_detach, bool detach,
>> bool has_resume, bool resume, Error **errp)
>> {
>> Error *local_err = NULL;
>> MigrationState *s = migrate_get_current();
>> - const char *p = NULL;
>> + const char *dst_ptr = NULL;
>>
>> if (!migrate_prepare(s, has_blk && blk, has_inc && inc,
>> has_resume && resume, errp)) {
>> @@ -2348,20 +2349,46 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
>> }
>> }
>>
>> + /*
>> + * In case of Multi-FD migration parameters, if uri is provided,
>
> I think you mean 'if uri list is provided'
>
>> + * supports only tcp network protocol.
>> + */
>> + if (has_multi_fd_uri_list) {
>> + int length = QAPI_LIST_LENGTH(cap);
>> + init_multifd_array(length);
>> + for (int i = 0; i < length; i++) {
>> + const char *p1 = NULL, *p2 = NULL;
>
> Keep these as ps/pd to make it clear which is source and dest.
>
>> + const char *multifd_dst_uri = cap->value->destination_uri;
>> + const char *multifd_src_uri = cap->value->source_uri;
>> + uint8_t multifd_channels = cap->value->multifd_channels;
>> + if (!strstart(multifd_dst_uri, "tcp:", &p1) ||
>> + !strstart(multifd_src_uri, "tcp:", &p2)) {
>
> I've copied in Claudio Fontana; Claudio is fighting to make snapshots
> faster and has been playing with various multithread schemes for multifd
> with files and fd's; perhaps the syntax you're proposing doesn't need
> to be limited to tcp.
Hi,
I will try to express our current problem, and see where there might be some overlap, maybe you can see more.
The current problem we are facing is, saving or restoring of VM state to disk, which in libvirt terms is "virsh save" or "virsh managedsave" and "virsh restore" or "virsh start",
is currently needlessly slow with large VMs, using upstream libvirt and qemu.
We need to get the transfer speeds of VM state (mainly RAM) to disk in the multiple GiB/s range with modern processors and NVMe disks; we have shown it is feasible.
Mainline libvirt uses QEMU migration to "fd://" to implement saving of VMs to disk, but adds copying though pipes via a libvirt "iohelper" process to work around a set of problems that are still not 100% clear to me;
(I cannot find the right email where this was discussed).
One clearly factual issue is that the QEMU migration stream is not currently O_DIRECT friendly, as it assumes it goes to network, and unfortunately for large transfers of this kind, O_DIRECT is needed due to the kernel file cache trashing problem.
So already, if your series were to address "fd://", it would potentially automatically provide an additional feature for libvirt's current save VM implementation; but I am not sure if what you are trying to achieve applies here.
Our temporary solution for libvirt to the throughput problem takes advantage of multifd migration to a "unix://" socket target to save in parallel,
with a new helper process (multifd-helper) taking the place of iohelper and performing the parallel multithreaded copy from the UNIX socket to a single file (in the latest iteration of the series),
or to multiple files in previous iterations, one for each multifd channel.
It works very well in practice, achieving dramatic throughput improvements by parallelizing the transfer reaching the GiB/s range.
This temporary solution is available here:
https://listman.redhat.com/archives/libvir-list/2022-June/232252.html
Libvirt is not accepting this approach, because the maintainer (Daniel, in Cc:) argues that the problem needs to be solved in QEMU instead,
while solving it in libvirt is an unwanted hack. My understanding is that this new feature is no more of a hack than the existing libvirt iohelper solution for basic VM save currently in mainline.
I don't really know how really this QEMU solution could look like yet.
If we code up a new QEMU "disk://" migration transport to save to a local file, and parameters to specify whether the transfer should happen in parallel, and how many parallel channels to use,
then we could solve the problem entirely in QEMU (possibly reusing some multifd code, or even not reusing that at all), but we end up with libvirt unable to efficiently put its own header as part of the savefile format libvirt expects.
An alternative could be instead to adjust the QEMU "fd://" migration protocol to add "parallel" parameters, and so keep the existing mechanism for libvirt/qemu communication for save vm,
change libvirt header read/write to be O_DIRECT friendly, and have qemu migrate in parallel directly to the open fd.
In both cases I presume that the QEMU migration stream code, including the code for all device state save, would need to be adjusted to be O_DIRECT friendly.
This modulo some additional details is my current understanding of the situation, I hope it helps.
Ciao,
Claudio
>
>> + error_setg(errp, "multi-fd destination and multi-fd source "
>> + "uri, both should be present and follows tcp protocol only");
>> + break;
>> + } else {
>> + store_multifd_migration_params(p1 ? p1 : multifd_dst_uri,
>> + p2 ? p2 : multifd_src_uri,
>> + multifd_channels, i, &local_err);
>> + }
>> + cap = cap->next;
>> + }
>> + }
>> +
>> migrate_protocol_allow_multi_channels(false);
>> - if (strstart(uri, "tcp:", &p) ||
>> + if (strstart(uri, "tcp:", &dst_ptr) ||
>> strstart(uri, "unix:", NULL) ||
>> strstart(uri, "vsock:", NULL)) {
>> migrate_protocol_allow_multi_channels(true);
>> - socket_start_outgoing_migration(s, p ? p : uri, &local_err);
>> + socket_start_outgoing_migration(s, dst_ptr ? dst_ptr : uri, &local_err);
>> #ifdef CONFIG_RDMA
>> - } else if (strstart(uri, "rdma:", &p)) {
>> - rdma_start_outgoing_migration(s, p, &local_err);
>> + } else if (strstart(uri, "rdma:", &dst_ptr)) {
>> + rdma_start_outgoing_migration(s, dst_ptr, &local_err);
>> #endif
>> - } else if (strstart(uri, "exec:", &p)) {
>> - exec_start_outgoing_migration(s, p, &local_err);
>> - } else if (strstart(uri, "fd:", &p)) {
>> - fd_start_outgoing_migration(s, p, &local_err);
>> + } else if (strstart(uri, "exec:", &dst_ptr)) {
>> + exec_start_outgoing_migration(s, dst_ptr, &local_err);
>> + } else if (strstart(uri, "fd:", &dst_ptr)) {
>> + fd_start_outgoing_migration(s, dst_ptr, &local_err);
>> } else {
>> if (!(has_resume && resume)) {
>> yank_unregister_instance(MIGRATION_YANK_INSTANCE);
>> diff --git a/migration/socket.c b/migration/socket.c
>> index 4fd5e85f50..7ca6af8cca 100644
>> --- a/migration/socket.c
>> +++ b/migration/socket.c
>> @@ -32,6 +32,17 @@ struct SocketOutgoingArgs {
>> SocketAddress *saddr;
>> } outgoing_args;
>>
>> +struct SocketArgs {
>> + struct SrcDestAddr data;
>
> 'data' is an odd name; 'addresses' perhaps?
>
>> + uint8_t multifd_channels;
>> +};
>> +
>> +struct OutgoingMigrateParams {
>> + struct SocketArgs *socket_args;
>> + size_t length;
>> + uint64_t total_multifd_channel;
>> +} outgoing_migrate_params;
>> +
>> void socket_send_channel_create(QIOTaskFunc f, void *data)
>> {
>> QIOChannelSocket *sioc = qio_channel_socket_new();
>> @@ -47,6 +58,14 @@ int socket_send_channel_destroy(QIOChannel *send)
>> qapi_free_SocketAddress(outgoing_args.saddr);
>> outgoing_args.saddr = NULL;
>> }
>> +
>> + if (outgoing_migrate_params.socket_args != NULL) {
>> + g_free(outgoing_migrate_params.socket_args);
>> + outgoing_migrate_params.socket_args = NULL;
>
> I think g_free is safe on NULL; so I think you can just do this without
> the if.
>
>> + }
>> + if (outgoing_migrate_params.length) {
>
> Does that ever differ from the != NULL test ?
> I think you can always just set this to 0 without the test.
>
>> + outgoing_migrate_params.length = 0;
>> + }
>> return 0;
>> }
>>
>> @@ -117,13 +136,41 @@ socket_start_outgoing_migration_internal(MigrationState *s,
>> }
>>
>> void socket_start_outgoing_migration(MigrationState *s,
>> - const char *str,
>> + const char *dst_str,
>> Error **errp)
>> {
>> Error *err = NULL;
>> - SocketAddress *saddr = socket_parse(str, &err);
>> + SocketAddress *dst_saddr = socket_parse(dst_str, &err);
>> + if (!err) {
>> + socket_start_outgoing_migration_internal(s, dst_saddr, &err);
>> + }
>> + error_propagate(errp, err);
>> +}
>> +
>> +void init_multifd_array(int length)
>> +{
>> + outgoing_migrate_params.socket_args = g_new0(struct SocketArgs, length);
>> + outgoing_migrate_params.length = length;
>> + outgoing_migrate_params.total_multifd_channel = 0;
>> +}
>> +
>> +void store_multifd_migration_params(const char *dst_uri,
>> + const char *src_uri,
>> + uint8_t multifd_channels,
>> + int idx, Error **errp)
>> +{
>> + Error *err = NULL;
>> + SocketAddress *src_addr = NULL;
>> + SocketAddress *dst_addr = socket_parse(dst_uri, &err);
>> + if (src_uri) {
>> + src_addr = socket_parse(src_uri, &err);
>> + }
>> if (!err) {
>> - socket_start_outgoing_migration_internal(s, saddr, &err);
>> + outgoing_migrate_params.socket_args[idx].data.dst_addr = dst_addr;
>> + outgoing_migrate_params.socket_args[idx].data.src_addr = src_addr;
>> + outgoing_migrate_params.socket_args[idx].multifd_channels
>> + = multifd_channels;
>> + outgoing_migrate_params.total_multifd_channel += multifd_channels;
>> }
>> error_propagate(errp, err);
>> }
>> diff --git a/migration/socket.h b/migration/socket.h
>> index 891dbccceb..bba7f177fe 100644
>> --- a/migration/socket.h
>> +++ b/migration/socket.h
>> @@ -19,12 +19,27 @@
>>
>> #include "io/channel.h"
>> #include "io/task.h"
>> +#include "migration.h"
>> +
>> +/* info regarding destination and source uri */
>> +struct SrcDestAddr {
>> + SocketAddress *dst_addr;
>> + SocketAddress *src_addr;
>> +};
>>
>> void socket_send_channel_create(QIOTaskFunc f, void *data);
>> int socket_send_channel_destroy(QIOChannel *send);
>>
>> void socket_start_incoming_migration(const char *str, Error **errp);
>>
>> -void socket_start_outgoing_migration(MigrationState *s, const char *str,
>> +void socket_start_outgoing_migration(MigrationState *s, const char *dst_str,
>> Error **errp);
>> +
>> +int multifd_list_length(MigrateUriParameterList *list);
>> +
>> +void init_multifd_array(int length);
>> +
>> +void store_multifd_migration_params(const char *dst_uri, const char *src_uri,
>> + uint8_t multifd_channels, int idx,
>> + Error **erp);
>> #endif
>> diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
>> index 622c783c32..2db539016a 100644
>> --- a/monitor/hmp-cmds.c
>> +++ b/monitor/hmp-cmds.c
>> @@ -56,6 +56,9 @@
>> #include "migration/snapshot.h"
>> #include "migration/misc.h"
>>
>> +/* Default number of multi-fd channels */
>> +#define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2
>> +
>> #ifdef CONFIG_SPICE
>> #include <spice/enums.h>
>> #endif
>> @@ -1574,10 +1577,25 @@ void hmp_migrate(Monitor *mon, const QDict *qdict)
>> bool inc = qdict_get_try_bool(qdict, "inc", false);
>> bool resume = qdict_get_try_bool(qdict, "resume", false);
>> const char *uri = qdict_get_str(qdict, "uri");
>> +
>> + const char *src_uri = qdict_get_str(qdict, "source-uri");
>> + const char *dst_uri = qdict_get_str(qdict, "destination-uri");
>> + uint8_t multifd_channels = qdict_get_try_int(qdict, "multifd-channels",
>> + DEFAULT_MIGRATE_MULTIFD_CHANNELS);
>> Error *err = NULL;
>> + MigrateUriParameterList *caps = NULL;
>> + MigrateUriParameter *value;
>> +
>> + value = g_malloc0(sizeof(*value));
>> + value->source_uri = (char *)src_uri;
>> + value->destination_uri = (char *)dst_uri;
>> + value->multifd_channels = multifd_channels;
>> + QAPI_LIST_PREPEND(caps, value);
>> +
>> + qmp_migrate(uri, !!caps, caps, !!blk, blk, !!inc,
>> + inc, false, false, true, resume, &err);
>> + qapi_free_MigrateUriParameterList(caps);
>>
>> - qmp_migrate(uri, !!blk, blk, !!inc, inc,
>> - false, false, true, resume, &err);
>> if (hmp_handle_error(mon, err)) {
>> return;
>> }
>
> Please split the HMP changes into a separate patch.
>
>> diff --git a/qapi/migration.json b/qapi/migration.json
>> index 6130cd9fae..fb259d626b 100644
>> --- a/qapi/migration.json
>> +++ b/qapi/migration.json
>> @@ -1454,12 +1454,38 @@
>> ##
>> { 'command': 'migrate-continue', 'data': {'state': 'MigrationStatus'} }
>>
>> +##
>> +# @MigrateUriParameter:
>> +#
>> +# Information regarding which source interface is connected to which
>> +# destination interface and number of multifd channels over each interface.
>> +#
>> +# @source-uri: the Uniform Resource Identifier of the source VM.
>> +# Default port number is 0.
>> +#
>> +# @destination-uri: the Uniform Resource Identifier of the destination VM
>
> I would just say 'uri' rather than spelling it out.
>
>> +# @multifd-channels: number of parallel multifd channels used to migrate data
>> +# for specific source-uri and destination-uri. Default value
>> +# in this case is 2 (Since 4.0)
>
> 7.1 at the moment.
>
>> +#
>> +##
>> +{ 'struct' : 'MigrateUriParameter',
>> + 'data' : { 'source-uri' : 'str',
>> + 'destination-uri' : 'str',
>> + '*multifd-channels' : 'uint8'} }
>
> OK, so much higher level question - why do we specify both URIs on
> each end? Is it just the source that is used on the source side to say
> which NIC to route down? On the destination side I guess there's no
> need?
>
> Do we have some rule about needing to specify enough channels for all
> the multifd channels we specify (i.e. if we specify 4 multifd channels
> in the migration parameter do we have to supply 4 channels here?)
> What happens with say Peter's preemption channel?
>
> Is there some logical ordering rule; i.e. if we were to start ordering
> particular multifd threads, then can we say that we allocate these
> channels in the same order as this list?
>
>> ##
>> # @migrate:
>> #
>> # Migrates the current running guest to another Virtual Machine.
>> #
>> # @uri: the Uniform Resource Identifier of the destination VM
>> +# for migration thread
>> +#
>> +# @multi-fd-uri-list: list of pair of source and destination VM Uniform
>> +# Resource Identifiers with number of multifd-channels
>> +# for each pair
>> #
>> # @blk: do block migration (full disk copy)
>> #
>> @@ -1479,20 +1505,27 @@
>> # 1. The 'query-migrate' command should be used to check migration's progress
>> # and final result (this information is provided by the 'status' member)
>> #
>> -# 2. All boolean arguments default to false
>> +# 2. The uri argument should have the Uniform Resource Identifier of default
>> +# destination VM. This connection will be bound to default network
>> +#
>> +# 3. All boolean arguments default to false
>> #
>> -# 3. The user Monitor's "detach" argument is invalid in QMP and should not
>> +# 4. The user Monitor's "detach" argument is invalid in QMP and should not
>> # be used
>> #
>> # Example:
>> #
>> -# -> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
>> +# -> { "execute": "migrate",
>> +# "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
>> +# "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
>> +# "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
>> +# "destination-uri": "tcp:11.0.0.0:7789", "multifd-channels": 5} ] } }
>> # <- { "return": {} }
>> #
>> ##
>> { 'command': 'migrate',
>> - 'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool',
>> - '*detach': 'bool', '*resume': 'bool' } }
>> + 'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateUriParameter'], '*blk': 'bool',
>> + '*inc': 'bool', '*detach': 'bool', '*resume': 'bool' } }
>>
>> ##
>> # @migrate-incoming:
>> --
>> 2.22.3
>>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 2/4] Adding multi-interface support for multi-FD on destination side
2022-06-16 18:40 ` Dr. David Alan Gilbert
@ 2022-07-13 14:36 ` Het Gala
0 siblings, 0 replies; 34+ messages in thread
From: Het Gala @ 2022-07-13 14:36 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: qemu-devel, quintela, pbonzini, berrange, armbru, eblake, Manish Mishra
On 17/06/22 12:10 am, Dr. David Alan Gilbert wrote:
> * Het Gala (het.gala@nutanix.com) wrote:
>> i) Modified the format of qemu monitor command: ‘migrate-incoming’ by adding
>> a list, each element in the list is to open listeners with a given number
>> of multiFD channels.
>>
>> ii) Qemu starts with -incoming flag defer and -multi-fd-incoming defer to
>> allow the modified 'migrate-incoming' command to be used.
>>
>> iii) Format for -multi-fd-incoming flag as a comma separated string has been
>> added with each substring containing listener socket address and number
>> of sockets to open.
>>
>> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
>> Signed-off-by: Het Gala <het.gala@nutanix.com>
>> ---
>> include/qapi/util.h | 1 +
>> migration/migration.c | 149 ++++++++++++++++++++++++++++++++++++------
>> migration/migration.h | 2 +
>> migration/socket.c | 11 ++--
>> migration/socket.h | 3 +-
>> monitor/hmp-cmds.c | 42 ++++--------
>> qapi/migration.json | 43 ++++++++++--
>> qapi/qapi-util.c | 27 ++++++++
>> qemu-options.hx | 18 +++++
>> softmmu/vl.c | 30 ++++++++-
>> 10 files changed, 265 insertions(+), 61 deletions(-)
>>
>> diff --git a/include/qapi/util.h b/include/qapi/util.h
>> index 3041feb3d9..88fb2270db 100644
>> --- a/include/qapi/util.h
>> +++ b/include/qapi/util.h
>> @@ -29,6 +29,7 @@ bool qapi_bool_parse(const char *name, const char *value, bool *obj,
>> Error **errp);
>>
>> int parse_qapi_name(const char *name, bool complete);
>> +struct strList *strList_from_string(const char *in, char c);
>>
>> /*
>> * For any GenericList @list, insert @element at the front.
>> diff --git a/migration/migration.c b/migration/migration.c
>> index c408175aeb..9b0ad732e7 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -477,28 +477,39 @@ void migrate_add_address(SocketAddress *address)
>> QAPI_CLONE(SocketAddress, address));
>> }
>>
>> -static void qemu_start_incoming_migration(const char *uri, Error **errp)
>> +static void qemu_start_incoming_migration(const char *uri, uint8_t number,
>> + int idx, Error **errp)
>> {
>> const char *p = NULL;
>>
>> - migrate_protocol_allow_multi_channels(false); /* reset it anyway */
>> - qapi_event_send_migration(MIGRATION_STATUS_SETUP);
>> - if (strstart(uri, "tcp:", &p) ||
>> - strstart(uri, "unix:", NULL) ||
>> - strstart(uri, "vsock:", NULL)) {
>> - migrate_protocol_allow_multi_channels(true);
>> - socket_start_incoming_migration(p ? p : uri, errp);
>> -#ifdef CONFIG_RDMA
>> - } else if (strstart(uri, "rdma:", &p)) {
>> - rdma_start_incoming_migration(p, errp);
>> -#endif
>> - } else if (strstart(uri, "exec:", &p)) {
>> - exec_start_incoming_migration(p, errp);
>> - } else if (strstart(uri, "fd:", &p)) {
>> - fd_start_incoming_migration(p, errp);
>> + if (number == 0) {
>> + migrate_protocol_allow_multi_channels(false); /* reset it anyway */
>> + qapi_event_send_migration(MIGRATION_STATUS_SETUP);
>> + if (strstart(uri, "tcp:", &p) ||
>> + strstart(uri, "unix:", NULL) ||
>> + strstart(uri, "vsock:", NULL)) {
>> + migrate_protocol_allow_multi_channels(true);
>> + #ifdef CONFIG_RDMA
>> + } else if (strstart(uri, "rdma:", &p)) {
>> + rdma_start_incoming_migration(p, errp);
>> + #endif
>> + } else if (strstart(uri, "exec:", &p)) {
>> + exec_start_incoming_migration(p, errp);
>> + } else if (strstart(uri, "fd:", &p)) {
>> + fd_start_incoming_migration(p, errp);
>> + } else {
>> + error_setg(errp, "unknown migration protocol: %s", uri);
>> + }
>> } else {
>> - error_setg(errp, "unknown migration protocol: %s", uri);
>> + /* multi-FD parameters only support tcp network protocols */
>> + if (!strstart(uri, "tcp:", &p)) {
>> + error_setg(errp, "multifd-destination uri supports "
>> + "tcp protocol only");
>> + return;
>> + }
>> + store_multifd_migration_params(p ? p : uri, NULL, number, idx, errp);
>> }
>> + socket_start_incoming_migration(p ? p : uri, number, errp);
>> }
>>
>> static void process_incoming_migration_bh(void *opaque)
>> @@ -2140,7 +2151,17 @@ void migrate_del_blocker(Error *reason)
>> migration_blockers = g_slist_remove(migration_blockers, reason);
>> }
>>
>> -void qmp_migrate_incoming(const char *uri, Error **errp)
>> +static inline int incoming_multi_fd_uri_parse(const char *str, char delim)
>> +{
>> + int count = 0;
>> + for (int i = 0; i < strlen(str); i++) {
>> + count += (str[i] == delim);
>> + }
>> + return count;
>> +}
> That's a bit more general little helper function; I guess it could go in
> util/ somewhere (something like qemu_string_count_delim ???)
> Yes David, I will include this helper function in util folder, in the
same patch
with all other helper functions I am trying to include.
>
>> +/* migrate_incoming comes from -incoming flag in qemu process */
>> +void migrate_incoming(const char *uri, Error **errp)
>> {
>> Error *local_err = NULL;
>> static bool once = true;
>> @@ -2154,11 +2175,99 @@ void qmp_migrate_incoming(const char *uri, Error **errp)
>> return;
>> }
>>
>> + qemu_start_incoming_migration(uri, 0, 0, &local_err);
>> +
>> if (!yank_register_instance(MIGRATION_YANK_INSTANCE, errp)) {
>> return;
>> }
>>
>> - qemu_start_incoming_migration(uri, &local_err);
>> + if (local_err) {
>> + yank_unregister_instance(MIGRATION_YANK_INSTANCE);
>> + error_propagate(errp, local_err);
>> + return;
>> + }
>> +
>> + once = false;
> Can you explain what this change is for?
> Sorry David, its just small confusion here. the upstream function
migrate_incoming
was divided into 3 different functions, because of which it has created
this confusion.
I will definitely clear this out in the v2 version of patches.
>> +}
>> +
>> +/* multi_fd_migrate_incoming comes from -multi-fd-migrate-incoming flag in qemu process */
>> +void multi_fd_migrate_incoming(const char *uri, Error **errp)
>> +{
>> + Error *local_err = NULL;
>> + static bool once = true;
>> +
>> + if (!once) {
>> + error_setg(errp, "The incoming migration has already been started");
>> + return;
>> + }
>> + if (!runstate_check(RUN_STATE_INMIGRATE)) {
>> + error_setg(errp, "'-multi-fd-incoming' was not specified on the command line");
>> + return;
>> + }
>> +
>> + strList *st = strList_from_string(uri, ',');
>> + strList *r = st;
>> + int length = QAPI_LIST_LENGTH(st);
>> + init_multifd_array(length);
>> +
>> + for (int i = 0; i < length; i++) {
>> + const char *uri = NULL, *ret = NULL;
>> + const char *str = r->value;
>> + uint8_t multifd_channels = DEFAULT_MIGRATE_MULTIFD_CHANNELS;
>> + int parse_count = incoming_multi_fd_uri_parse(str, ':');
>> + if (parse_count < 2 || parse_count > 3) {
>> + error_setg(errp, "Invalid format of string-id %d in "
>> + "'-multi-fd-incoming' flag", i);
>> + return;
>> + }
>> + if (parse_count == 3) {
>> + ret = strrchr(str, ':');
>> + uri = g_strndup(str, strlen(str) - strlen(ret));
>> + multifd_channels = atoi(ret + 1);
>> + }
>> + qemu_start_incoming_migration(parse_count == 2 ? str : uri,
>> + multifd_channels, i, &local_err);
>> + r = r->next;
>> + }
>> +
>> + if (local_err) {
>> + yank_unregister_instance(MIGRATION_YANK_INSTANCE);
>> + error_propagate(errp, local_err);
>> + return;
>> + }
>> +
>> + once = false;
>> +}
>> +
>> +/* qmp_migrate_incoming comes from qemu qmp monitor command */
>> +void qmp_migrate_incoming(const char *uri, bool has_multi_fd_uri_list,
>> + MigrateIncomingUriList *cap, Error **errp)
>> +{
>> + Error *local_err = NULL;
>> + static bool once = true;
>> +
>> + if (!once) {
>> + error_setg(errp, "The incoming migration has already been started");
>> + return;
>> + }
>> +
>> + if (!yank_register_instance(MIGRATION_YANK_INSTANCE, errp)) {
>> + return;
>> + }
>> +
>> + /* For migration thread */
>> + qemu_start_incoming_migration(uri, 0, 0, &local_err);
>> +
>> + /* For Multi-FD */
>> + int length = QAPI_LIST_LENGTH(cap);
>> + init_multifd_array(length);
>> + for (int i = 0; i < length; i++) {
>> + const char *multifd_dst_uri = cap->value->destination_uri;
>> + uint8_t multifd_channels = cap->value->multifd_channels;
>> + qemu_start_incoming_migration(multifd_dst_uri, multifd_channels,
>> + i, &local_err);
>> + cap = cap->next;
>> + }
>>
>> if (local_err) {
>> yank_unregister_instance(MIGRATION_YANK_INSTANCE);
>> @@ -2194,7 +2303,7 @@ void qmp_migrate_recover(const char *uri, Error **errp)
>> * only re-setup the migration stream and poke existing migration
>> * to continue using that newly established channel.
>> */
>> - qemu_start_incoming_migration(uri, errp);
>> + qemu_start_incoming_migration(uri, 0, 0, errp);
>> }
>>
>> void qmp_migrate_pause(Error **errp)
>> diff --git a/migration/migration.h b/migration/migration.h
>> index 485d58b95f..fa8717ec9e 100644
>> --- a/migration/migration.h
>> +++ b/migration/migration.h
>> @@ -353,6 +353,8 @@ bool migration_is_setup_or_active(int state);
>> bool migration_is_running(int state);
>>
>> void migrate_init(MigrationState *s);
>> +void migrate_incoming(const char *uri, Error **errp);
>> +void multi_fd_migrate_incoming(const char *uri_str, Error **errp);
>> bool migration_is_blocked(Error **errp);
>> /* True if outgoing migration has entered postcopy phase */
>> bool migration_in_postcopy(void);
>> diff --git a/migration/socket.c b/migration/socket.c
>> index 7ca6af8cca..21e0983df2 100644
>> --- a/migration/socket.c
>> +++ b/migration/socket.c
>> @@ -202,17 +202,17 @@ socket_incoming_migration_end(void *opaque)
>>
>> static void
>> socket_start_incoming_migration_internal(SocketAddress *saddr,
>> - Error **errp)
>> + uint8_t number, Error **errp)
>> {
>> QIONetListener *listener = qio_net_listener_new();
>> MigrationIncomingState *mis = migration_incoming_get_current();
>> size_t i;
>> - int num = 1;
>> + uint8_t num = 1;
>>
>> qio_net_listener_set_name(listener, "migration-socket-listener");
>>
>> if (migrate_use_multifd()) {
>> - num = migrate_multifd_channels();
>> + num = number;
> That's a strange line - it suggests you need a better name for both
> parameters!
> Sure David. will change the parameter names for both.
>
>> }
>>
>> if (qio_net_listener_open_sync(listener, saddr, num, errp) < 0) {
>> @@ -239,12 +239,13 @@ socket_start_incoming_migration_internal(SocketAddress *saddr,
>> }
>> }
>>
>> -void socket_start_incoming_migration(const char *str, Error **errp)
>> +void socket_start_incoming_migration(const char *str,
>> + uint8_t number, Error **errp)
>> {
>> Error *err = NULL;
>> SocketAddress *saddr = socket_parse(str, &err);
>> if (!err) {
>> - socket_start_incoming_migration_internal(saddr, &err);
>> + socket_start_incoming_migration_internal(saddr, number, &err);
>> }
>> qapi_free_SocketAddress(saddr);
>> error_propagate(errp, err);
>> diff --git a/migration/socket.h b/migration/socket.h
>> index bba7f177fe..b9e3699167 100644
>> --- a/migration/socket.h
>> +++ b/migration/socket.h
>> @@ -30,7 +30,8 @@ struct SrcDestAddr {
>> void socket_send_channel_create(QIOTaskFunc f, void *data);
>> int socket_send_channel_destroy(QIOChannel *send);
>>
>> -void socket_start_incoming_migration(const char *str, Error **errp);
>> +void socket_start_incoming_migration(const char *str, uint8_t number,
>> + Error **errp);
>>
>> void socket_start_outgoing_migration(MigrationState *s, const char *dst_str,
>> Error **errp);
>> diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
>> index 2db539016a..32a6b67d5f 100644
>> --- a/monitor/hmp-cmds.c
>> +++ b/monitor/hmp-cmds.c
>> @@ -42,6 +42,7 @@
>> #include "qapi/qapi-commands-run-state.h"
>> #include "qapi/qapi-commands-tpm.h"
>> #include "qapi/qapi-commands-ui.h"
>> +#include "qapi/util.h"
>> #include "qapi/qapi-visit-net.h"
>> #include "qapi/qapi-visit-migration.h"
>> #include "qapi/qmp/qdict.h"
>> @@ -72,32 +73,6 @@ bool hmp_handle_error(Monitor *mon, Error *err)
>> return false;
>> }
>>
>> -/*
>> - * Produce a strList from a comma separated list.
>> - * A NULL or empty input string return NULL.
>> - */
>> -static strList *strList_from_comma_list(const char *in)
>> -{
>> - strList *res = NULL;
>> - strList **tail = &res;
>> -
>> - while (in && in[0]) {
>> - char *comma = strchr(in, ',');
>> - char *value;
>> -
>> - if (comma) {
>> - value = g_strndup(in, comma - in);
>> - in = comma + 1; /* skip the , */
>> - } else {
>> - value = g_strdup(in);
>> - in = NULL;
>> - }
>> - QAPI_LIST_APPEND(tail, value);
>> - }
>> -
>> - return res;
>> -}
>> -
>> void hmp_info_name(Monitor *mon, const QDict *qdict)
>> {
>> NameInfo *info;
>> @@ -1117,7 +1092,7 @@ void hmp_announce_self(Monitor *mon, const QDict *qdict)
>> migrate_announce_params());
>>
>> qapi_free_strList(params->interfaces);
>> - params->interfaces = strList_from_comma_list(interfaces_str);
>> + params->interfaces = strList_from_string(interfaces_str, ',');
>> params->has_interfaces = params->interfaces != NULL;
>> params->id = g_strdup(id);
>> params->has_id = !!params->id;
>> @@ -1147,8 +1122,19 @@ void hmp_migrate_incoming(Monitor *mon, const QDict *qdict)
>> {
>> Error *err = NULL;
>> const char *uri = qdict_get_str(qdict, "uri");
>> + const char *dst_uri = qdict_get_str(qdict, "destination-uri");
>> + uint8_t multifd_channels = qdict_get_try_int(qdict, "multifd-channels",
>> + DEFAULT_MIGRATE_MULTIFD_CHANNELS);
>> + MigrateIncomingUriList *caps = NULL;
>> + MigrateIncomingUri *value;
>> +
>> + value = g_malloc0(sizeof(*value));
>> + value->destination_uri = (char *)dst_uri;
>> + value->multifd_channels = multifd_channels;
>> + QAPI_LIST_PREPEND(caps, value);
>>
>> - qmp_migrate_incoming(uri, &err);
>> + qmp_migrate_incoming(uri, !!caps, caps, &err);
>> + qapi_free_MigrateIncomingUriList(caps);
>>
>> hmp_handle_error(mon, err);
>> }
>> diff --git a/qapi/migration.json b/qapi/migration.json
>> index fb259d626b..62a7b22d19 100644
>> --- a/qapi/migration.json
>> +++ b/qapi/migration.json
>> @@ -1527,15 +1527,37 @@
>> 'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateUriParameter'], '*blk': 'bool',
>> '*inc': 'bool', '*detach': 'bool', '*resume': 'bool' } }
>>
>> +##
>> +# @MigrateIncomingUri:
>> +#
>> +# Information regarding which destination listening interface to be connected
>> +# and number of multifd channels over that interface.
>> +#
>> +# @destination-uri: the Uniform Resource Identifier of the destination VM
>> +#
>> +# @multifd-channels: number of channels used to migrate data in parallel for
>> +# for specific source-uri and destination-uri.
>> +# Default value in this case is 2 (Since 4.0)
>> +#
>> +##
>> +{ 'struct' : 'MigrateIncomingUri',
>> + 'data' : { 'destination-uri' : 'str',
>> + '*multifd-channels' : 'uint8'} }
>> +
>> ##
>> # @migrate-incoming:
>> #
>> # Start an incoming migration, the qemu must have been started
>> -# with -incoming defer
>> +# with -incoming defer. qemu can also be started with optional
>> +# -multi-fd-incoming defer for opening multifd listening sockets
>> #
>> # @uri: The Uniform Resource Identifier identifying the source or
>> # address to listen on
>> #
>> +# @multi-fd-uri-list: list of pair of source and destination VM Uniform
>> +# Resource Identifiers with number of multifd-channels
>> +# for each pair
>> +#
>> # Returns: nothing on success
>> #
>> # Since: 2.3
>> @@ -1546,19 +1568,30 @@
>> # compatible with -incoming and the format of the uri is already exposed
>> # above libvirt.
>> #
>> -# 2. QEMU must be started with -incoming defer to allow migrate-incoming to
>> +# 2. multi-fd-uri-list will have list of destination uri as listening sockets
>> +# and multi-fd number of channels on each listening socket.
>> +#
>> +# 3. QEMU must be started with -incoming defer to allow migrate-incoming to
>> # be used.
>> #
>> -# 3. The uri format is the same as for -incoming
>> +# 4. multi-fd-uri-list format is not the same as for -multi-fd-incoming flag.
>> +# For -multi-fd-incoming flag, it is a comma separated list of listener
>> +# sockets with multifd channels.
>> +# Example: -multi-fd-incoming "tcp::6900:4,tcp:11.0.0.0:7789:5".
> Why bother with -multi-fd-incoming ? Just insist on people using
> -incoming defer - I think we'd like to do away with the other forms
> of -incoming, it simplifies stuff a lot that way rather than having two
> schemes.
> Our idea here behind introducing -multi-fd-incoming was just that if
some user wanted to open the multifd listening sockets before migration
process starts, he could able todo it, just like -incoming flag. But if you
think, it should only be via qmp monitor command, we could discard this
change.
>
>
>> # Example:
>> #
>> # -> { "execute": "migrate-incoming",
>> -# "arguments": { "uri": "tcp::4446" } }
>> +# "arguments": {"uri": "tcp::6789",
>> +# "multi-fd-uri-list" : [ {"destination-uri" : "tcp::6900",
>> +# "multifd-channels": 4},
>> +# {"destination-uri" : "tcp:11.0.0.0:7789",
>> +# "multifd-channels": 5} ] } }
>> # <- { "return": {} }
>> #
>> ##
>> -{ 'command': 'migrate-incoming', 'data': {'uri': 'str' } }
>> +{ 'command': 'migrate-incoming',
>> + 'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateIncomingUri'] } }
>>
>> ##
>> # @xen-save-devices-state:
>> diff --git a/qapi/qapi-util.c b/qapi/qapi-util.c
>> index 63596e11c5..9cc43ebcd3 100644
>> --- a/qapi/qapi-util.c
>> +++ b/qapi/qapi-util.c
>> @@ -15,6 +15,7 @@
>> #include "qapi/error.h"
>> #include "qemu/ctype.h"
>> #include "qapi/qmp/qerror.h"
>> +#include "qapi/qapi-builtin-types.h"
>>
>> CompatPolicy compat_policy;
>>
>> @@ -152,3 +153,29 @@ int parse_qapi_name(const char *str, bool complete)
>> }
>> return p - str;
>> }
>> +
>> +/*
>> + * Produce a strList from a delimiter separated list.
>> + * A NULL or empty input string return NULL.
>> + */
>> +strList *strList_from_string(const char *in, char c)
>> +{
>> + strList *res = NULL;
>> + strList **tail = &res;
>> +
>> + while (in && in[0]) {
>> + char *ch = strchr(in, c);
>> + char *value;
>> +
>> + if (ch) {
>> + value = g_strndup(in, ch - in);
>> + in = ch + 1; /* skip the , */
>> + } else {
>> + value = g_strdup(in);
>> + in = NULL;
>> + }
>> + QAPI_LIST_APPEND(tail, value);
>> + }
>> +
>> + return res;
>> +}
>> diff --git a/qemu-options.hx b/qemu-options.hx
>> index 60cf188da4..2e82e41dd5 100644
>> --- a/qemu-options.hx
>> +++ b/qemu-options.hx
>> @@ -4480,6 +4480,24 @@ SRST
>> to issuing the migrate\_incoming to allow the migration to begin.
>> ERST
>>
>> +DEF("multi-fd-incoming", HAS_ARG, QEMU_OPTION_multi_fd_incoming, \
>> + "-multi-fd-incoming tcp:[host]:port[:channel][,to=maxport][,ipv4=on|off][,ipv6=on|off]\n" \
>> + "-multi-fd-incoming defer\n" \
>> + " wait for the URI to be specified via\n" \
>> + " multi_fd_migrate_incoming\n",
>> + QEMU_ARCH_ALL)
>> +SRST
>> +``-multi-fd-incoming tcp:[host]:port[:channel][,to=maxport][,ipv4=on|off][,ipv6=on|off]``
>> + Prepare for multi-fd incoming migration, with multi-fd listening sockets
>> + on that connection. Default number of multi-fd channels is 2.
>> +
>> +``-multi-fd-incoming defer``
>> + Wait for the URI to be specified via multi_fd_migrate\_incoming. The
>> + monitor can be used to change settings (such as migration parameters)
>> + prior to issuing the multi_fd_migrate\_incoming to allow the migration
>> + to begin.
>> +ERST
>> +
>> DEF("only-migratable", 0, QEMU_OPTION_only_migratable, \
>> "-only-migratable allow only migratable devices\n", QEMU_ARCH_ALL)
>> SRST
>> diff --git a/softmmu/vl.c b/softmmu/vl.c
>> index 4c1e94b00e..2f5cf18eff 100644
>> --- a/softmmu/vl.c
>> +++ b/softmmu/vl.c
>> @@ -45,7 +45,7 @@
>> #include "sysemu/seccomp.h"
>> #include "sysemu/tcg.h"
>> #include "sysemu/xen.h"
>> -
>> +#include "migration/migration.h"
>> #include "qemu/error-report.h"
>> #include "qemu/sockets.h"
>> #include "qemu/accel.h"
>> @@ -167,6 +167,7 @@ typedef struct DeviceOption {
>> static const char *cpu_option;
>> static const char *mem_path;
>> static const char *incoming;
>> +static const char *multi_fd_incoming;
>> static const char *loadvm;
>> static const char *accelerators;
>> static bool have_custom_ram_size;
>> @@ -2349,6 +2350,11 @@ static void qemu_validate_options(const QDict *machine_opts)
>> error_report("'preconfig' supports '-incoming defer' only");
>> exit(EXIT_FAILURE);
>> }
>> + if (multi_fd_incoming && preconfig_requested &&
>> + strcmp(multi_fd_incoming, "defer") != 0) {
>> + error_report("'preconfig' supports '-multi-fd-incoming defer' only");
>> + exit(EXIT_FAILURE);
>> + }
>>
>> #ifdef CONFIG_CURSES
>> if (is_daemonized() && dpy.type == DISPLAY_TYPE_CURSES) {
>> @@ -2621,7 +2627,7 @@ void qmp_x_exit_preconfig(Error **errp)
>> if (incoming) {
>> Error *local_err = NULL;
>> if (strcmp(incoming, "defer") != 0) {
>> - qmp_migrate_incoming(incoming, &local_err);
>> + migrate_incoming(incoming, &local_err);
>> if (local_err) {
>> error_reportf_err(local_err, "-incoming %s: ", incoming);
>> exit(1);
>> @@ -2630,6 +2636,20 @@ void qmp_x_exit_preconfig(Error **errp)
>> } else if (autostart) {
>> qmp_cont(NULL);
>> }
>> +
>> + if (multi_fd_incoming) {
>> + Error *local_err = NULL;
>> + if (strcmp(multi_fd_incoming, "defer") != 0) {
>> + multi_fd_migrate_incoming(multi_fd_incoming, &local_err);
>> + if (local_err) {
>> + error_reportf_err(local_err, "-multi-fd-incoming %s: ",
>> + multi_fd_incoming);
>> + exit(1);
>> + }
>> + }
>> + } else if (autostart) {
>> + qmp_cont(NULL);
>> + }
>> }
>>
>> void qemu_init(int argc, char **argv, char **envp)
>> @@ -3355,6 +3375,12 @@ void qemu_init(int argc, char **argv, char **envp)
>> }
>> incoming = optarg;
>> break;
>> + case QEMU_OPTION_multi_fd_incoming:
>> + if (!multi_fd_incoming) {
>> + runstate_set(RUN_STATE_INMIGRATE);
>> + }
>> + multi_fd_incoming = optarg;
>> + break;
>> case QEMU_OPTION_only_migratable:
>> only_migratable = 1;
>> break;
>> --
>> 2.22.3
>>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair
2022-07-13 8:08 ` Het Gala
@ 2022-07-15 8:07 ` Het Gala
0 siblings, 0 replies; 34+ messages in thread
From: Het Gala @ 2022-07-15 8:07 UTC (permalink / raw)
To: Dr. David Alan Gilbert, cfontana
Cc: qemu-devel, quintela, pbonzini, berrange, armbru, eblake, Manish Mishra
On 13/07/22 1:38 pm, Het Gala wrote:
>
> On 16/06/22 10:56 pm, Dr. David Alan Gilbert wrote:
>> * Het Gala (het.gala@nutanix.com) wrote:
>
> > First of all, I apologise for the late reply. I was on a leave after
> internship ended
>
> at Nutanix. Hope to learn a lot from you all in the process of
> upstreaming multifd
>
> patches.
>
>>> i) Modified the format of the qemu monitor command : 'migrate' by
>>> adding a list,
>>> each element in the list consists of multi-FD connection
>>> parameters: source
>>> and destination uris and of the number of multi-fd channels
>>> between each pair.
>>>
>>> ii) Information of all multi-FD connection parameters’ list, length
>>> of the list
>>> and total number of multi-fd channels for all the connections
>>> together is
>>> stored in ‘OutgoingArgs’ struct.
>>>
>>> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
>>> Signed-off-by: Het Gala <het.gala@nutanix.com>
>>> ---
>>> include/qapi/util.h | 9 ++++++++
>>> migration/migration.c | 47 ++++++++++++++++++++++++++++++--------
>>> migration/socket.c | 53
>>> ++++++++++++++++++++++++++++++++++++++++---
>>> migration/socket.h | 17 +++++++++++++-
>>> monitor/hmp-cmds.c | 22 ++++++++++++++++--
>>> qapi/migration.json | 43 +++++++++++++++++++++++++++++++----
>>> 6 files changed, 170 insertions(+), 21 deletions(-)
>>>
>>> diff --git a/include/qapi/util.h b/include/qapi/util.h
>>> index 81a2b13a33..3041feb3d9 100644
>>> --- a/include/qapi/util.h
>>> +++ b/include/qapi/util.h
>>> @@ -56,4 +56,13 @@ int parse_qapi_name(const char *name, bool
>>> complete);
>>> (tail) = &(*(tail))->next; \
>>> } while (0)
>>> +#define QAPI_LIST_LENGTH(list) ({ \
>>> + int _len = 0; \
>>> + typeof(list) _elem; \
>>> + for (_elem = list; _elem != NULL; _elem = _elem->next) { \
>>> + _len++; \
>>> + } \
>>> + _len; \
>>> +})
>>> +
>>> #endif
>> This looks like it should be a separate patch to me (and perhaps size_t
>> for len?)
>
> > Sure, will try to make a seperate patch for QAPI_LIST_LENGTH, and other
>
> such utility functions from the other patches.
>
>>
>>> diff --git a/migration/migration.c b/migration/migration.c
>>> index 31739b2af9..c408175aeb 100644
>>> --- a/migration/migration.c
>>> +++ b/migration/migration.c
>>> @@ -2328,13 +2328,14 @@ static bool migrate_prepare(MigrationState
>>> *s, bool blk, bool blk_inc,
>>> return true;
>>> }
>>> -void qmp_migrate(const char *uri, bool has_blk, bool blk,
>>> +void qmp_migrate(const char *uri, bool has_multi_fd_uri_list,
>>> + MigrateUriParameterList *cap, bool has_blk, bool blk,
>>> bool has_inc, bool inc, bool has_detach, bool
>>> detach,
>>> bool has_resume, bool resume, Error **errp)
>>> {
>>> Error *local_err = NULL;
>>> MigrationState *s = migrate_get_current();
>>> - const char *p = NULL;
>>> + const char *dst_ptr = NULL;
>>> if (!migrate_prepare(s, has_blk && blk, has_inc && inc,
>>> has_resume && resume, errp)) {
>>> @@ -2348,20 +2349,46 @@ void qmp_migrate(const char *uri, bool
>>> has_blk, bool blk,
>>> }
>>> }
>>> + /*
>>> + * In case of Multi-FD migration parameters, if uri is provided,
>> I think you mean 'if uri list is provided'
> > Acknowledged.
>>
>>> + * supports only tcp network protocol.
>>> + */
>>> + if (has_multi_fd_uri_list) {
>>> + int length = QAPI_LIST_LENGTH(cap);
>>> + init_multifd_array(length);
>>> + for (int i = 0; i < length; i++) {
>>> + const char *p1 = NULL, *p2 = NULL;
>> Keep these as ps/pd to make it clear which is source and dest.
> > Acknowledged. Will change in the upcoming patchset.
>>
>>> + const char *multifd_dst_uri = cap->value->destination_uri;
>>> + const char *multifd_src_uri = cap->value->source_uri;
>>> + uint8_t multifd_channels = cap->value->multifd_channels;
>>> + if (!strstart(multifd_dst_uri, "tcp:", &p1) ||
>>> + !strstart(multifd_src_uri, "tcp:", &p2)) {
>> I've copied in Claudio Fontana; Claudio is fighting to make snapshots
>> faster and has been playing with various multithread schemes for multifd
>> with files and fd's; perhaps the syntax you're proposing doesn't need
>> to be limited to tcp.
>
> > For now, we are just aiming to include multifd for existing tcp
> protocol.
>
> We would be happy to take any suggestions from Claudio Fontana and try to
>
> include them in the upcoming patchset series.
>
>>
>>> + error_setg(errp, "multi-fd destination and multi-fd
>>> source "
>>> + "uri, both should be present and follows tcp
>>> protocol only");
>>> + break;
>>> + } else {
>>> + store_multifd_migration_params(p1 ? p1 :
>>> multifd_dst_uri,
>>> + p2 ? p2 : multifd_src_uri,
>>> + multifd_channels, i,
>>> &local_err);
>>> + }
>>> + cap = cap->next;
>>> + }
>>> + }
>>> +
>>> migrate_protocol_allow_multi_channels(false);
>>> - if (strstart(uri, "tcp:", &p) ||
>>> + if (strstart(uri, "tcp:", &dst_ptr) ||
>>> strstart(uri, "unix:", NULL) ||
>>> strstart(uri, "vsock:", NULL)) {
>>> migrate_protocol_allow_multi_channels(true);
>>> - socket_start_outgoing_migration(s, p ? p : uri, &local_err);
>>> + socket_start_outgoing_migration(s, dst_ptr ? dst_ptr : uri,
>>> &local_err);
>>> #ifdef CONFIG_RDMA
>>> - } else if (strstart(uri, "rdma:", &p)) {
>>> - rdma_start_outgoing_migration(s, p, &local_err);
>>> + } else if (strstart(uri, "rdma:", &dst_ptr)) {
>>> + rdma_start_outgoing_migration(s, dst_ptr, &local_err);
>>> #endif
>>> - } else if (strstart(uri, "exec:", &p)) {
>>> - exec_start_outgoing_migration(s, p, &local_err);
>>> - } else if (strstart(uri, "fd:", &p)) {
>>> - fd_start_outgoing_migration(s, p, &local_err);
>>> + } else if (strstart(uri, "exec:", &dst_ptr)) {
>>> + exec_start_outgoing_migration(s, dst_ptr, &local_err);
>>> + } else if (strstart(uri, "fd:", &dst_ptr)) {
>>> + fd_start_outgoing_migration(s, dst_ptr, &local_err);
>>> } else {
>>> if (!(has_resume && resume)) {
>>> yank_unregister_instance(MIGRATION_YANK_INSTANCE);
>>> diff --git a/migration/socket.c b/migration/socket.c
>>> index 4fd5e85f50..7ca6af8cca 100644
>>> --- a/migration/socket.c
>>> +++ b/migration/socket.c
>>> @@ -32,6 +32,17 @@ struct SocketOutgoingArgs {
>>> SocketAddress *saddr;
>>> } outgoing_args;
>>> +struct SocketArgs {
>>> + struct SrcDestAddr data;
>> 'data' is an odd name; 'addresses' perhaps?
> > Sure, Acknowledged.
>>
>>> + uint8_t multifd_channels;
>>> +};
>>> +
>>> +struct OutgoingMigrateParams {
>>> + struct SocketArgs *socket_args;
>>> + size_t length;
>>> + uint64_t total_multifd_channel;
>>> +} outgoing_migrate_params;
>>> +
>>> void socket_send_channel_create(QIOTaskFunc f, void *data)
>>> {
>>> QIOChannelSocket *sioc = qio_channel_socket_new();
>>> @@ -47,6 +58,14 @@ int socket_send_channel_destroy(QIOChannel *send)
>>> qapi_free_SocketAddress(outgoing_args.saddr);
>>> outgoing_args.saddr = NULL;
>>> }
>>> +
>>> + if (outgoing_migrate_params.socket_args != NULL) {
>>> + g_free(outgoing_migrate_params.socket_args);
>>> + outgoing_migrate_params.socket_args = NULL;
>> I think g_free is safe on NULL; so I think you can just do this without
>> the if.
> > Okay, thanks for the suggestion there David.
>>
>>> + }
>>> + if (outgoing_migrate_params.length) {
>> Does that ever differ from the != NULL test ?
>> I think you can always just set this to 0 without the test.
> > Sure.
>>
>>> + outgoing_migrate_params.length = 0;
>>> + }
>>> return 0;
>>> }
>>> @@ -117,13 +136,41 @@
>>> socket_start_outgoing_migration_internal(MigrationState *s,
>>> }
>>> void socket_start_outgoing_migration(MigrationState *s,
>>> - const char *str,
>>> + const char *dst_str,
>>> Error **errp)
>>> {
>>> Error *err = NULL;
>>> - SocketAddress *saddr = socket_parse(str, &err);
>>> + SocketAddress *dst_saddr = socket_parse(dst_str, &err);
>>> + if (!err) {
>>> + socket_start_outgoing_migration_internal(s, dst_saddr, &err);
>>> + }
>>> + error_propagate(errp, err);
>>> +}
>>> +
>>> +void init_multifd_array(int length)
>>> +{
>>> + outgoing_migrate_params.socket_args = g_new0(struct SocketArgs,
>>> length);
>>> + outgoing_migrate_params.length = length;
>>> + outgoing_migrate_params.total_multifd_channel = 0;
>>> +}
>>> +
>>> +void store_multifd_migration_params(const char *dst_uri,
>>> + const char *src_uri,
>>> + uint8_t multifd_channels,
>>> + int idx, Error **errp)
>>> +{
>>> + Error *err = NULL;
>>> + SocketAddress *src_addr = NULL;
>>> + SocketAddress *dst_addr = socket_parse(dst_uri, &err);
>>> + if (src_uri) {
>>> + src_addr = socket_parse(src_uri, &err);
>>> + }
>>> if (!err) {
>>> - socket_start_outgoing_migration_internal(s, saddr, &err);
>>> + outgoing_migrate_params.socket_args[idx].data.dst_addr = dst_addr;
>>> + outgoing_migrate_params.socket_args[idx].data.src_addr = src_addr;
>>> + outgoing_migrate_params.socket_args[idx].multifd_channels
>>> + =
>>> multifd_channels;
>>> + outgoing_migrate_params.total_multifd_channel +=
>>> multifd_channels;
>>> }
>>> error_propagate(errp, err);
>>> }
>>> diff --git a/migration/socket.h b/migration/socket.h
>>> index 891dbccceb..bba7f177fe 100644
>>> --- a/migration/socket.h
>>> +++ b/migration/socket.h
>>> @@ -19,12 +19,27 @@
>>> #include "io/channel.h"
>>> #include "io/task.h"
>>> +#include "migration.h"
>>> +
>>> +/* info regarding destination and source uri */
>>> +struct SrcDestAddr {
>>> + SocketAddress *dst_addr;
>>> + SocketAddress *src_addr;
>>> +};
>>> void socket_send_channel_create(QIOTaskFunc f, void *data);
>>> int socket_send_channel_destroy(QIOChannel *send);
>>> void socket_start_incoming_migration(const char *str, Error
>>> **errp);
>>> -void socket_start_outgoing_migration(MigrationState *s, const
>>> char *str,
>>> +void socket_start_outgoing_migration(MigrationState *s, const char
>>> *dst_str,
>>> Error **errp);
>>> +
>>> +int multifd_list_length(MigrateUriParameterList *list);
>>> +
>>> +void init_multifd_array(int length);
>>> +
>>> +void store_multifd_migration_params(const char *dst_uri, const char
>>> *src_uri,
>>> + uint8_t multifd_channels, int idx,
>>> + Error **erp);
>>> #endif
>>> diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
>>> index 622c783c32..2db539016a 100644
>>> --- a/monitor/hmp-cmds.c
>>> +++ b/monitor/hmp-cmds.c
>>> @@ -56,6 +56,9 @@
>>> #include "migration/snapshot.h"
>>> #include "migration/misc.h"
>>> +/* Default number of multi-fd channels */
>>> +#define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2
>>> +
>>> #ifdef CONFIG_SPICE
>>> #include <spice/enums.h>
>>> #endif
>>> @@ -1574,10 +1577,25 @@ void hmp_migrate(Monitor *mon, const QDict
>>> *qdict)
>>> bool inc = qdict_get_try_bool(qdict, "inc", false);
>>> bool resume = qdict_get_try_bool(qdict, "resume", false);
>>> const char *uri = qdict_get_str(qdict, "uri");
>>> +
>>> + const char *src_uri = qdict_get_str(qdict, "source-uri");
>>> + const char *dst_uri = qdict_get_str(qdict, "destination-uri");
>>> + uint8_t multifd_channels = qdict_get_try_int(qdict,
>>> "multifd-channels",
>>> + DEFAULT_MIGRATE_MULTIFD_CHANNELS);
>>> Error *err = NULL;
>>> + MigrateUriParameterList *caps = NULL;
>>> + MigrateUriParameter *value;
>>> +
>>> + value = g_malloc0(sizeof(*value));
>>> + value->source_uri = (char *)src_uri;
>>> + value->destination_uri = (char *)dst_uri;
>>> + value->multifd_channels = multifd_channels;
>>> + QAPI_LIST_PREPEND(caps, value);
>>> +
>>> + qmp_migrate(uri, !!caps, caps, !!blk, blk, !!inc,
>>> + inc, false, false, true, resume, &err);
>>> + qapi_free_MigrateUriParameterList(caps);
>>> - qmp_migrate(uri, !!blk, blk, !!inc, inc,
>>> - false, false, true, resume, &err);
>>> if (hmp_handle_error(mon, err)) {
>>> return;
>>> }
>> Please split the HMP changes into a separate patch.
>
> > Okay sure. Will include both on destination and source side HMP changes
>
> into a seperate patch.
> Hi David. I am very new to upstream changes so I apologise if something
very obvious is not understandable to me. I tried to seperate HMP
changes from
source and destination side, but the build is failing because it has
dependencies
from qapi/migration.json and qapi/qapi-commands-migration.h files. I
also reffered
to this commit
https://github.com/qemu/qemu/commit/abb6295b3ace5d17c3a65936913fc346616dbf14
and they have also put the QMP/HMP changes into a single commit. Let me
know if there
is a better way we can put the HMP changes into a seperate patch.
>
>>
>>> diff --git a/qapi/migration.json b/qapi/migration.json
>>> index 6130cd9fae..fb259d626b 100644
>>> --- a/qapi/migration.json
>>> +++ b/qapi/migration.json
>>> @@ -1454,12 +1454,38 @@
>>> ##
>>> { 'command': 'migrate-continue', 'data': {'state':
>>> 'MigrationStatus'} }
>>> +##
>>> +# @MigrateUriParameter:
>>> +#
>>> +# Information regarding which source interface is connected to which
>>> +# destination interface and number of multifd channels over each
>>> interface.
>>> +#
>>> +# @source-uri: the Uniform Resource Identifier of the source VM.
>>> +# Default port number is 0.
>>> +#
>>> +# @destination-uri: the Uniform Resource Identifier of the
>>> destination VM
>> I would just say 'uri' rather than spelling it out.
> > Okay, acknowledged.
>>
>>> +# @multifd-channels: number of parallel multifd channels used to
>>> migrate data
>>> +# for specific source-uri and destination-uri.
>>> Default value
>>> +# in this case is 2 (Since 4.0)
>> 7.1 at the moment.
> > Thanks for pointing it out.
>>
>>> +#
>>> +##
>>> +{ 'struct' : 'MigrateUriParameter',
>>> + 'data' : { 'source-uri' : 'str',
>>> + 'destination-uri' : 'str',
>>> + '*multifd-channels' : 'uint8'} }
>> OK, so much higher level question - why do we specify both URIs on
>> each end? Is it just the source that is used on the source side to say
>> which NIC to route down? On the destination side I guess there's no
>> need?
>>
>> Do we have some rule about needing to specify enough channels for all
>> the multifd channels we specify (i.e. if we specify 4 multifd channels
>> in the migration parameter do we have to supply 4 channels here?)
>> What happens with say Peter's preemption channel?
>>
>> Is there some logical ordering rule; i.e. if we were to start ordering
>> particular multifd threads, then can we say that we allocate these
>> channels in the same order as this list?
>
> > I certainly did not get your first point here David. On the
> destination side,
>
> I think we certainly need both, destination and source uri's for
> making a connection
>
> but on the source side, we do not require source uri, which I have not
> included
>
> if you look at the 'Adding multi-interface support for multi-FD on
> destination
>
> side' patch.
>
> > Yes, I agree with you. I will inlcude this feature in the next
> version of patchset,
>
> where it will check the number of multifd channels coming from API and
> total
>
> multifd channel number from qmp monitor command, and should be equal.
>
> > Yes David, multifd threads will be allocated in the same order, the
> user will
>
> specify in the qmp monitor command.
>
>>> ##
>>> # @migrate:
>>> #
>>> # Migrates the current running guest to another Virtual Machine.
>>> #
>>> # @uri: the Uniform Resource Identifier of the destination VM
>>> +# for migration thread
>>> +#
>>> +# @multi-fd-uri-list: list of pair of source and destination VM
>>> Uniform
>>> +# Resource Identifiers with number of
>>> multifd-channels
>>> +# for each pair
>>> #
>>> # @blk: do block migration (full disk copy)
>>> #
>>> @@ -1479,20 +1505,27 @@
>>> # 1. The 'query-migrate' command should be used to check
>>> migration's progress
>>> # and final result (this information is provided by the
>>> 'status' member)
>>> #
>>> -# 2. All boolean arguments default to false
>>> +# 2. The uri argument should have the Uniform Resource Identifier
>>> of default
>>> +# destination VM. This connection will be bound to default network
>>> +#
>>> +# 3. All boolean arguments default to false
>>> #
>>> -# 3. The user Monitor's "detach" argument is invalid in QMP and
>>> should not
>>> +# 4. The user Monitor's "detach" argument is invalid in QMP and
>>> should not
>>> # be used
>>> #
>>> # Example:
>>> #
>>> -# -> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
>>> +# -> { "execute": "migrate",
>>> +# "arguments": { "uri": "tcp:0:4446",
>>> "multi-fd-uri-list": [ {
>>> +# "source-uri": "tcp::6900",
>>> "destination-uri": "tcp:0:4480",
>>> +# "multifd-channels": 4}, {
>>> "source-uri": "tcp:10.0.0.0: ",
>>> +# "destination-uri":
>>> "tcp:11.0.0.0:7789", "multifd-channels": 5} ] } }
>>> # <- { "return": {} }
>>> #
>>> ##
>>> { 'command': 'migrate',
>>> - 'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool',
>>> - '*detach': 'bool', '*resume': 'bool' } }
>>> + 'data': {'uri': 'str', '*multi-fd-uri-list':
>>> ['MigrateUriParameter'], '*blk': 'bool',
>>> + '*inc': 'bool', '*detach': 'bool', '*resume': 'bool' } }
>>> ##
>>> # @migrate-incoming:
>>> --
>>> 2.22.3
>>>
> Regards
>
> Het Gala
>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair
2022-06-09 7:33 ` [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair Het Gala
2022-06-16 17:26 ` Dr. David Alan Gilbert
@ 2022-07-18 8:35 ` Markus Armbruster
2022-07-18 13:33 ` Het Gala
1 sibling, 1 reply; 34+ messages in thread
From: Markus Armbruster @ 2022-07-18 8:35 UTC (permalink / raw)
To: Het Gala
Cc: qemu-devel, quintela, dgilbert, pbonzini, berrange, eblake,
Manish Mishra
Het Gala <het.gala@nutanix.com> writes:
> i) Modified the format of the qemu monitor command : 'migrate' by adding a list,
> each element in the list consists of multi-FD connection parameters: source
> and destination uris and of the number of multi-fd channels between each pair.
>
> ii) Information of all multi-FD connection parameters’ list, length of the list
> and total number of multi-fd channels for all the connections together is
> stored in ‘OutgoingArgs’ struct.
>
> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
> Signed-off-by: Het Gala <het.gala@nutanix.com>
> ---
> include/qapi/util.h | 9 ++++++++
> migration/migration.c | 47 ++++++++++++++++++++++++++++++--------
> migration/socket.c | 53 ++++++++++++++++++++++++++++++++++++++++---
> migration/socket.h | 17 +++++++++++++-
> monitor/hmp-cmds.c | 22 ++++++++++++++++--
> qapi/migration.json | 43 +++++++++++++++++++++++++++++++----
> 6 files changed, 170 insertions(+), 21 deletions(-)
>
> diff --git a/include/qapi/util.h b/include/qapi/util.h
> index 81a2b13a33..3041feb3d9 100644
> --- a/include/qapi/util.h
> +++ b/include/qapi/util.h
> @@ -56,4 +56,13 @@ int parse_qapi_name(const char *name, bool complete);
> (tail) = &(*(tail))->next; \
> } while (0)
>
> +#define QAPI_LIST_LENGTH(list) ({ \
> + int _len = 0; \
> + typeof(list) _elem; \
> + for (_elem = list; _elem != NULL; _elem = _elem->next) { \
> + _len++; \
> + } \
> + _len; \
> +})
> +
Unless there is a compelling reason for open-coding this, make it a
(non-inline) function.
> #endif
> diff --git a/migration/migration.c b/migration/migration.c
> index 31739b2af9..c408175aeb 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -2328,13 +2328,14 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,
> return true;
> }
>
> -void qmp_migrate(const char *uri, bool has_blk, bool blk,
> +void qmp_migrate(const char *uri, bool has_multi_fd_uri_list,
> + MigrateUriParameterList *cap, bool has_blk, bool blk,
> bool has_inc, bool inc, bool has_detach, bool detach,
> bool has_resume, bool resume, Error **errp)
> {
> Error *local_err = NULL;
> MigrationState *s = migrate_get_current();
> - const char *p = NULL;
> + const char *dst_ptr = NULL;
>
> if (!migrate_prepare(s, has_blk && blk, has_inc && inc,
> has_resume && resume, errp)) {
> @@ -2348,20 +2349,46 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
> }
> }
>
> + /*
> + * In case of Multi-FD migration parameters, if uri is provided,
> + * supports only tcp network protocol.
> + */
> + if (has_multi_fd_uri_list) {
> + int length = QAPI_LIST_LENGTH(cap);
> + init_multifd_array(length);
> + for (int i = 0; i < length; i++) {
> + const char *p1 = NULL, *p2 = NULL;
> + const char *multifd_dst_uri = cap->value->destination_uri;
> + const char *multifd_src_uri = cap->value->source_uri;
> + uint8_t multifd_channels = cap->value->multifd_channels;
> + if (!strstart(multifd_dst_uri, "tcp:", &p1) ||
> + !strstart(multifd_src_uri, "tcp:", &p2)) {
> + error_setg(errp, "multi-fd destination and multi-fd source "
> + "uri, both should be present and follows tcp protocol only");
> + break;
> + } else {
> + store_multifd_migration_params(p1 ? p1 : multifd_dst_uri,
> + p2 ? p2 : multifd_src_uri,
> + multifd_channels, i, &local_err);
> + }
> + cap = cap->next;
> + }
> + }
> +
> migrate_protocol_allow_multi_channels(false);
> - if (strstart(uri, "tcp:", &p) ||
> + if (strstart(uri, "tcp:", &dst_ptr) ||
> strstart(uri, "unix:", NULL) ||
> strstart(uri, "vsock:", NULL)) {
> migrate_protocol_allow_multi_channels(true);
> - socket_start_outgoing_migration(s, p ? p : uri, &local_err);
> + socket_start_outgoing_migration(s, dst_ptr ? dst_ptr : uri, &local_err);
> #ifdef CONFIG_RDMA
> - } else if (strstart(uri, "rdma:", &p)) {
> - rdma_start_outgoing_migration(s, p, &local_err);
> + } else if (strstart(uri, "rdma:", &dst_ptr)) {
> + rdma_start_outgoing_migration(s, dst_ptr, &local_err);
> #endif
> - } else if (strstart(uri, "exec:", &p)) {
> - exec_start_outgoing_migration(s, p, &local_err);
> - } else if (strstart(uri, "fd:", &p)) {
> - fd_start_outgoing_migration(s, p, &local_err);
> + } else if (strstart(uri, "exec:", &dst_ptr)) {
> + exec_start_outgoing_migration(s, dst_ptr, &local_err);
> + } else if (strstart(uri, "fd:", &dst_ptr)) {
> + fd_start_outgoing_migration(s, dst_ptr, &local_err);
> } else {
> if (!(has_resume && resume)) {
> yank_unregister_instance(MIGRATION_YANK_INSTANCE);
> diff --git a/migration/socket.c b/migration/socket.c
> index 4fd5e85f50..7ca6af8cca 100644
> --- a/migration/socket.c
> +++ b/migration/socket.c
> @@ -32,6 +32,17 @@ struct SocketOutgoingArgs {
> SocketAddress *saddr;
> } outgoing_args;
>
> +struct SocketArgs {
> + struct SrcDestAddr data;
> + uint8_t multifd_channels;
> +};
> +
> +struct OutgoingMigrateParams {
> + struct SocketArgs *socket_args;
> + size_t length;
Length of what?
> + uint64_t total_multifd_channel;
@total_multifd_channels appears to be the sum of the
socket_args[*].multifd_channels. Correct?
> +} outgoing_migrate_params;
> +
> void socket_send_channel_create(QIOTaskFunc f, void *data)
> {
> QIOChannelSocket *sioc = qio_channel_socket_new();
> @@ -47,6 +58,14 @@ int socket_send_channel_destroy(QIOChannel *send)
> qapi_free_SocketAddress(outgoing_args.saddr);
> outgoing_args.saddr = NULL;
> }
> +
> + if (outgoing_migrate_params.socket_args != NULL) {
> + g_free(outgoing_migrate_params.socket_args);
> + outgoing_migrate_params.socket_args = NULL;
> + }
> + if (outgoing_migrate_params.length) {
> + outgoing_migrate_params.length = 0;
> + }
> return 0;
> }
>
> @@ -117,13 +136,41 @@ socket_start_outgoing_migration_internal(MigrationState *s,
> }
>
> void socket_start_outgoing_migration(MigrationState *s,
> - const char *str,
> + const char *dst_str,
> Error **errp)
> {
> Error *err = NULL;
> - SocketAddress *saddr = socket_parse(str, &err);
> + SocketAddress *dst_saddr = socket_parse(dst_str, &err);
> + if (!err) {
> + socket_start_outgoing_migration_internal(s, dst_saddr, &err);
> + }
> + error_propagate(errp, err);
> +}
> +
> +void init_multifd_array(int length)
> +{
> + outgoing_migrate_params.socket_args = g_new0(struct SocketArgs, length);
> + outgoing_migrate_params.length = length;
> + outgoing_migrate_params.total_multifd_channel = 0;
> +}
> +
> +void store_multifd_migration_params(const char *dst_uri,
> + const char *src_uri,
> + uint8_t multifd_channels,
> + int idx, Error **errp)
> +{
> + Error *err = NULL;
> + SocketAddress *src_addr = NULL;
> + SocketAddress *dst_addr = socket_parse(dst_uri, &err);
> + if (src_uri) {
> + src_addr = socket_parse(src_uri, &err);
> + }
Incorrect use of &err. error.h's big comment:
* Receive and accumulate multiple errors (first one wins):
* Error *err = NULL, *local_err = NULL;
* foo(arg, &err);
* bar(arg, &local_err);
* error_propagate(&err, local_err);
* if (err) {
* handle the error...
* }
*
* Do *not* "optimize" this to
* Error *err = NULL;
* foo(arg, &err);
* bar(arg, &err); // WRONG!
* if (err) {
* handle the error...
* }
* because this may pass a non-null err to bar().
> if (!err) {
> - socket_start_outgoing_migration_internal(s, saddr, &err);
> + outgoing_migrate_params.socket_args[idx].data.dst_addr = dst_addr;
> + outgoing_migrate_params.socket_args[idx].data.src_addr = src_addr;
> + outgoing_migrate_params.socket_args[idx].multifd_channels
> + = multifd_channels;
> + outgoing_migrate_params.total_multifd_channel += multifd_channels;
> }
> error_propagate(errp, err);
Consider
struct SocketArgs *sa = &outgoing_migrate_params.socket_args[idx];
SocketAddress *src_addr, *dst_addr;
src_addr = socketaddress(src_uri, errp);
if (!src_addr) {
return;
}
dst_addr = socketaddress(dst_uri, errp);
if (!dst_addr) {
return;
}
sa->data.dst_addr = dst_addr;
sa->data.src_addr = src_addr;
sa->multifd_channels = multifd_channels;
outgoing_migrate_params.total_multifd_channel += multifd_channels;
> }
> diff --git a/migration/socket.h b/migration/socket.h
> index 891dbccceb..bba7f177fe 100644
> --- a/migration/socket.h
> +++ b/migration/socket.h
> @@ -19,12 +19,27 @@
>
> #include "io/channel.h"
> #include "io/task.h"
> +#include "migration.h"
> +
> +/* info regarding destination and source uri */
> +struct SrcDestAddr {
> + SocketAddress *dst_addr;
> + SocketAddress *src_addr;
> +};
QEMU coding style wants a typedef.
>
> void socket_send_channel_create(QIOTaskFunc f, void *data);
> int socket_send_channel_destroy(QIOChannel *send);
>
> void socket_start_incoming_migration(const char *str, Error **errp);
>
> -void socket_start_outgoing_migration(MigrationState *s, const char *str,
> +void socket_start_outgoing_migration(MigrationState *s, const char *dst_str,
> Error **errp);
> +
> +int multifd_list_length(MigrateUriParameterList *list);
> +
> +void init_multifd_array(int length);
> +
> +void store_multifd_migration_params(const char *dst_uri, const char *src_uri,
> + uint8_t multifd_channels, int idx,
> + Error **erp);
> #endif
> diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
> index 622c783c32..2db539016a 100644
> --- a/monitor/hmp-cmds.c
> +++ b/monitor/hmp-cmds.c
> @@ -56,6 +56,9 @@
> #include "migration/snapshot.h"
> #include "migration/misc.h"
>
> +/* Default number of multi-fd channels */
> +#define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2
> +
> #ifdef CONFIG_SPICE
> #include <spice/enums.h>
> #endif
> @@ -1574,10 +1577,25 @@ void hmp_migrate(Monitor *mon, const QDict *qdict)
> bool inc = qdict_get_try_bool(qdict, "inc", false);
> bool resume = qdict_get_try_bool(qdict, "resume", false);
> const char *uri = qdict_get_str(qdict, "uri");
> +
> + const char *src_uri = qdict_get_str(qdict, "source-uri");
> + const char *dst_uri = qdict_get_str(qdict, "destination-uri");
> + uint8_t multifd_channels = qdict_get_try_int(qdict, "multifd-channels",
> + DEFAULT_MIGRATE_MULTIFD_CHANNELS);
> Error *err = NULL;
> + MigrateUriParameterList *caps = NULL;
> + MigrateUriParameter *value;
> +
> + value = g_malloc0(sizeof(*value));
> + value->source_uri = (char *)src_uri;
> + value->destination_uri = (char *)dst_uri;
> + value->multifd_channels = multifd_channels;
> + QAPI_LIST_PREPEND(caps, value);
> +
> + qmp_migrate(uri, !!caps, caps, !!blk, blk, !!inc,
> + inc, false, false, true, resume, &err);
> + qapi_free_MigrateUriParameterList(caps);
>
> - qmp_migrate(uri, !!blk, blk, !!inc, inc,
> - false, false, true, resume, &err);
> if (hmp_handle_error(mon, err)) {
> return;
> }
> diff --git a/qapi/migration.json b/qapi/migration.json
> index 6130cd9fae..fb259d626b 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -1454,12 +1454,38 @@
> ##
> { 'command': 'migrate-continue', 'data': {'state': 'MigrationStatus'} }
>
> +##
> +# @MigrateUriParameter:
> +#
> +# Information regarding which source interface is connected to which
> +# destination interface and number of multifd channels over each interface.
> +#
> +# @source-uri: the Uniform Resource Identifier of the source VM.
> +# Default port number is 0.
> +#
> +# @destination-uri: the Uniform Resource Identifier of the destination VM
> +#
> +# @multifd-channels: number of parallel multifd channels used to migrate data
> +# for specific source-uri and destination-uri. Default value
> +# in this case is 2 (Since 4.0)
You mean "(Since 7.1)", I guess.
> +#
> +##
> +{ 'struct' : 'MigrateUriParameter',
> + 'data' : { 'source-uri' : 'str',
> + 'destination-uri' : 'str',
> + '*multifd-channels' : 'uint8'} }
> +
> ##
> # @migrate:
> #
> # Migrates the current running guest to another Virtual Machine.
> #
> # @uri: the Uniform Resource Identifier of the destination VM
> +# for migration thread
> +#
> +# @multi-fd-uri-list: list of pair of source and destination VM Uniform
> +# Resource Identifiers with number of multifd-channels
> +# for each pair
> #
> # @blk: do block migration (full disk copy)
> #
> @@ -1479,20 +1505,27 @@
> # 1. The 'query-migrate' command should be used to check migration's progress
> # and final result (this information is provided by the 'status' member)
> #
> -# 2. All boolean arguments default to false
> +# 2. The uri argument should have the Uniform Resource Identifier of default
> +# destination VM. This connection will be bound to default network
> +#
> +# 3. All boolean arguments default to false
> #
> -# 3. The user Monitor's "detach" argument is invalid in QMP and should not
> +# 4. The user Monitor's "detach" argument is invalid in QMP and should not
> # be used
> #
> # Example:
> #
> -# -> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
> +# -> { "execute": "migrate",
> +# "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
> +# "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
> +# "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
> +# "destination-uri": "tcp:11.0.0.0:7789", "multifd-channels": 5} ] } }
> # <- { "return": {} }
> #
> ##
> { 'command': 'migrate',
> - 'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool',
> - '*detach': 'bool', '*resume': 'bool' } }
> + 'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateUriParameter'], '*blk': 'bool',
Long line.
> + '*inc': 'bool', '*detach': 'bool', '*resume': 'bool' } }
>
> ##
> # @migrate-incoming:
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair
2022-07-18 8:35 ` Markus Armbruster
@ 2022-07-18 13:33 ` Het Gala
2022-07-18 14:33 ` Markus Armbruster
0 siblings, 1 reply; 34+ messages in thread
From: Het Gala @ 2022-07-18 13:33 UTC (permalink / raw)
To: Markus Armbruster
Cc: qemu-devel, quintela, dgilbert, pbonzini, berrange, eblake,
Manish Mishra
On 18/07/22 2:05 pm, Markus Armbruster wrote:
> Het Gala <het.gala@nutanix.com> writes:
>
>> i) Modified the format of the qemu monitor command : 'migrate' by adding a list,
>> each element in the list consists of multi-FD connection parameters: source
>> and destination uris and of the number of multi-fd channels between each pair.
>>
>> ii) Information of all multi-FD connection parameters’ list, length of the list
>> and total number of multi-fd channels for all the connections together is
>> stored in ‘OutgoingArgs’ struct.
>>
>> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
>> Signed-off-by: Het Gala <het.gala@nutanix.com>
>> ---
>> include/qapi/util.h | 9 ++++++++
>> migration/migration.c | 47 ++++++++++++++++++++++++++++++--------
>> migration/socket.c | 53 ++++++++++++++++++++++++++++++++++++++++---
>> migration/socket.h | 17 +++++++++++++-
>> monitor/hmp-cmds.c | 22 ++++++++++++++++--
>> qapi/migration.json | 43 +++++++++++++++++++++++++++++++----
>> 6 files changed, 170 insertions(+), 21 deletions(-)
>>
>> diff --git a/include/qapi/util.h b/include/qapi/util.h
>> index 81a2b13a33..3041feb3d9 100644
>> --- a/include/qapi/util.h
>> +++ b/include/qapi/util.h
>> @@ -56,4 +56,13 @@ int parse_qapi_name(const char *name, bool complete);
>> (tail) = &(*(tail))->next; \
>> } while (0)
>>
>> +#define QAPI_LIST_LENGTH(list) ({ \
>> + int _len = 0; \
>> + typeof(list) _elem; \
>> + for (_elem = list; _elem != NULL; _elem = _elem->next) { \
>> + _len++; \
>> + } \
>> + _len; \
>> +})
>> +
> Unless there is a compelling reason for open-coding this, make it a
> (non-inline) function.
> if kept as a function, the datatype of list is different every time
in the qemu code, and that led to multiple copies of this function. So
we decided, it would be best to keep it as a macro defined function. We
would be happy to hear any suggstions to solve this problem while the
function non-inline.
>> #endif
>> diff --git a/migration/migration.c b/migration/migration.c
>> index 31739b2af9..c408175aeb 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -2328,13 +2328,14 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,
>> return true;
>> }
>>
>> -void qmp_migrate(const char *uri, bool has_blk, bool blk,
>> +void qmp_migrate(const char *uri, bool has_multi_fd_uri_list,
>> + MigrateUriParameterList *cap, bool has_blk, bool blk,
>> bool has_inc, bool inc, bool has_detach, bool detach,
>> bool has_resume, bool resume, Error **errp)
>> {
>> Error *local_err = NULL;
>> MigrationState *s = migrate_get_current();
>> - const char *p = NULL;
>> + const char *dst_ptr = NULL;
>>
>> if (!migrate_prepare(s, has_blk && blk, has_inc && inc,
>> has_resume && resume, errp)) {
>> @@ -2348,20 +2349,46 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
>> }
>> }
>>
>> + /*
>> + * In case of Multi-FD migration parameters, if uri is provided,
>> + * supports only tcp network protocol.
>> + */
>> + if (has_multi_fd_uri_list) {
>> + int length = QAPI_LIST_LENGTH(cap);
>> + init_multifd_array(length);
>> + for (int i = 0; i < length; i++) {
>> + const char *p1 = NULL, *p2 = NULL;
>> + const char *multifd_dst_uri = cap->value->destination_uri;
>> + const char *multifd_src_uri = cap->value->source_uri;
>> + uint8_t multifd_channels = cap->value->multifd_channels;
>> + if (!strstart(multifd_dst_uri, "tcp:", &p1) ||
>> + !strstart(multifd_src_uri, "tcp:", &p2)) {
>> + error_setg(errp, "multi-fd destination and multi-fd source "
>> + "uri, both should be present and follows tcp protocol only");
>> + break;
>> + } else {
>> + store_multifd_migration_params(p1 ? p1 : multifd_dst_uri,
>> + p2 ? p2 : multifd_src_uri,
>> + multifd_channels, i, &local_err);
>> + }
>> + cap = cap->next;
>> + }
>> + }
>> +
>> migrate_protocol_allow_multi_channels(false);
>> - if (strstart(uri, "tcp:", &p) ||
>> + if (strstart(uri, "tcp:", &dst_ptr) ||
>> strstart(uri, "unix:", NULL) ||
>> strstart(uri, "vsock:", NULL)) {
>> migrate_protocol_allow_multi_channels(true);
>> - socket_start_outgoing_migration(s, p ? p : uri, &local_err);
>> + socket_start_outgoing_migration(s, dst_ptr ? dst_ptr : uri, &local_err);
>> #ifdef CONFIG_RDMA
>> - } else if (strstart(uri, "rdma:", &p)) {
>> - rdma_start_outgoing_migration(s, p, &local_err);
>> + } else if (strstart(uri, "rdma:", &dst_ptr)) {
>> + rdma_start_outgoing_migration(s, dst_ptr, &local_err);
>> #endif
>> - } else if (strstart(uri, "exec:", &p)) {
>> - exec_start_outgoing_migration(s, p, &local_err);
>> - } else if (strstart(uri, "fd:", &p)) {
>> - fd_start_outgoing_migration(s, p, &local_err);
>> + } else if (strstart(uri, "exec:", &dst_ptr)) {
>> + exec_start_outgoing_migration(s, dst_ptr, &local_err);
>> + } else if (strstart(uri, "fd:", &dst_ptr)) {
>> + fd_start_outgoing_migration(s, dst_ptr, &local_err);
>> } else {
>> if (!(has_resume && resume)) {
>> yank_unregister_instance(MIGRATION_YANK_INSTANCE);
>> diff --git a/migration/socket.c b/migration/socket.c
>> index 4fd5e85f50..7ca6af8cca 100644
>> --- a/migration/socket.c
>> +++ b/migration/socket.c
>> @@ -32,6 +32,17 @@ struct SocketOutgoingArgs {
>> SocketAddress *saddr;
>> } outgoing_args;
>>
>> +struct SocketArgs {
>> + struct SrcDestAddr data;
>> + uint8_t multifd_channels;
>> +};
>> +
>> +struct OutgoingMigrateParams {
>> + struct SocketArgs *socket_args;
>> + size_t length;
> Length of what?
> length of the socket_args[] array. Thanks for pointing it out. I will
be more specific for this variable in the v2 patchset series.
>
>> + uint64_t total_multifd_channel;
> @total_multifd_channels appears to be the sum of the
> socket_args[*].multifd_channels. Correct?
> Yes Markus, you are correct.
>
>> +} outgoing_migrate_params;
>> +
>> void socket_send_channel_create(QIOTaskFunc f, void *data)
>> {
>> QIOChannelSocket *sioc = qio_channel_socket_new();
>> @@ -47,6 +58,14 @@ int socket_send_channel_destroy(QIOChannel *send)
>> qapi_free_SocketAddress(outgoing_args.saddr);
>> outgoing_args.saddr = NULL;
>> }
>> +
>> + if (outgoing_migrate_params.socket_args != NULL) {
>> + g_free(outgoing_migrate_params.socket_args);
>> + outgoing_migrate_params.socket_args = NULL;
>> + }
>> + if (outgoing_migrate_params.length) {
>> + outgoing_migrate_params.length = 0;
>> + }
>> return 0;
>> }
>>
>> @@ -117,13 +136,41 @@ socket_start_outgoing_migration_internal(MigrationState *s,
>> }
>>
>> void socket_start_outgoing_migration(MigrationState *s,
>> - const char *str,
>> + const char *dst_str,
>> Error **errp)
>> {
>> Error *err = NULL;
>> - SocketAddress *saddr = socket_parse(str, &err);
>> + SocketAddress *dst_saddr = socket_parse(dst_str, &err);
>> + if (!err) {
>> + socket_start_outgoing_migration_internal(s, dst_saddr, &err);
>> + }
>> + error_propagate(errp, err);
>> +}
>> +
>> +void init_multifd_array(int length)
>> +{
>> + outgoing_migrate_params.socket_args = g_new0(struct SocketArgs, length);
>> + outgoing_migrate_params.length = length;
>> + outgoing_migrate_params.total_multifd_channel = 0;
>> +}
>> +
>> +void store_multifd_migration_params(const char *dst_uri,
>> + const char *src_uri,
>> + uint8_t multifd_channels,
>> + int idx, Error **errp)
>> +{
>> + Error *err = NULL;
>> + SocketAddress *src_addr = NULL;
>> + SocketAddress *dst_addr = socket_parse(dst_uri, &err);
>> + if (src_uri) {
>> + src_addr = socket_parse(src_uri, &err);
>> + }
> Incorrect use of &err. error.h's big comment:
>
> * Receive and accumulate multiple errors (first one wins):
> * Error *err = NULL, *local_err = NULL;
> * foo(arg, &err);
> * bar(arg, &local_err);
> * error_propagate(&err, local_err);
> * if (err) {
> * handle the error...
> * }
> *
> * Do *not* "optimize" this to
> * Error *err = NULL;
> * foo(arg, &err);
> * bar(arg, &err); // WRONG!
> * if (err) {
> * handle the error...
> * }
> * because this may pass a non-null err to bar().
> Thankyou Markus for sharing this knowledge. I was unaware of the
dont's with &err.
>> if (!err) {
>> - socket_start_outgoing_migration_internal(s, saddr, &err);
>> + outgoing_migrate_params.socket_args[idx].data.dst_addr = dst_addr;
>> + outgoing_migrate_params.socket_args[idx].data.src_addr = src_addr;
>> + outgoing_migrate_params.socket_args[idx].multifd_channels
>> + = multifd_channels;
>> + outgoing_migrate_params.total_multifd_channel += multifd_channels;
>> }
>> error_propagate(errp, err);
> Consider
>
> struct SocketArgs *sa = &outgoing_migrate_params.socket_args[idx];
> SocketAddress *src_addr, *dst_addr;
>
> src_addr = socketaddress(src_uri, errp);
> if (!src_addr) {
> return;
> }
>
> dst_addr = socketaddress(dst_uri, errp);
> if (!dst_addr) {
> return;
> }
>
> sa->data.dst_addr = dst_addr;
> sa->data.src_addr = src_addr;
> sa->multifd_channels = multifd_channels;
> outgoing_migrate_params.total_multifd_channel += multifd_channels;
> Thanks Markus for this amazing suggestion. Your approach looks
simpler to understand and also resolves the error it had with &err. I
will surely implement this in the upcoming v2 patchset.
>> }
>> diff --git a/migration/socket.h b/migration/socket.h
>> index 891dbccceb..bba7f177fe 100644
>> --- a/migration/socket.h
>> +++ b/migration/socket.h
>> @@ -19,12 +19,27 @@
>>
>> #include "io/channel.h"
>> #include "io/task.h"
>> +#include "migration.h"
>> +
>> +/* info regarding destination and source uri */
>> +struct SrcDestAddr {
>> + SocketAddress *dst_addr;
>> + SocketAddress *src_addr;
>> +};
> QEMU coding style wants a typedef.
> Okay, acknowledged.
>>
>> void socket_send_channel_create(QIOTaskFunc f, void *data);
>> int socket_send_channel_destroy(QIOChannel *send);
>>
>> void socket_start_incoming_migration(const char *str, Error **errp);
>>
>> -void socket_start_outgoing_migration(MigrationState *s, const char *str,
>> +void socket_start_outgoing_migration(MigrationState *s, const char *dst_str,
>> Error **errp);
>> +
>> +int multifd_list_length(MigrateUriParameterList *list);
>> +
>> +void init_multifd_array(int length);
>> +
>> +void store_multifd_migration_params(const char *dst_uri, const char *src_uri,
>> + uint8_t multifd_channels, int idx,
>> + Error **erp);
>> #endif
>> diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
>> index 622c783c32..2db539016a 100644
>> --- a/monitor/hmp-cmds.c
>> +++ b/monitor/hmp-cmds.c
>> @@ -56,6 +56,9 @@
>> #include "migration/snapshot.h"
>> #include "migration/misc.h"
>>
>> +/* Default number of multi-fd channels */
>> +#define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2
>> +
>> #ifdef CONFIG_SPICE
>> #include <spice/enums.h>
>> #endif
>> @@ -1574,10 +1577,25 @@ void hmp_migrate(Monitor *mon, const QDict *qdict)
>> bool inc = qdict_get_try_bool(qdict, "inc", false);
>> bool resume = qdict_get_try_bool(qdict, "resume", false);
>> const char *uri = qdict_get_str(qdict, "uri");
>> +
>> + const char *src_uri = qdict_get_str(qdict, "source-uri");
>> + const char *dst_uri = qdict_get_str(qdict, "destination-uri");
>> + uint8_t multifd_channels = qdict_get_try_int(qdict, "multifd-channels",
>> + DEFAULT_MIGRATE_MULTIFD_CHANNELS);
>> Error *err = NULL;
>> + MigrateUriParameterList *caps = NULL;
>> + MigrateUriParameter *value;
>> +
>> + value = g_malloc0(sizeof(*value));
>> + value->source_uri = (char *)src_uri;
>> + value->destination_uri = (char *)dst_uri;
>> + value->multifd_channels = multifd_channels;
>> + QAPI_LIST_PREPEND(caps, value);
>> +
>> + qmp_migrate(uri, !!caps, caps, !!blk, blk, !!inc,
>> + inc, false, false, true, resume, &err);
>> + qapi_free_MigrateUriParameterList(caps);
>>
>> - qmp_migrate(uri, !!blk, blk, !!inc, inc,
>> - false, false, true, resume, &err);
>> if (hmp_handle_error(mon, err)) {
>> return;
>> }
>> diff --git a/qapi/migration.json b/qapi/migration.json
>> index 6130cd9fae..fb259d626b 100644
>> --- a/qapi/migration.json
>> +++ b/qapi/migration.json
>> @@ -1454,12 +1454,38 @@
>> ##
>> { 'command': 'migrate-continue', 'data': {'state': 'MigrationStatus'} }
>>
>> +##
>> +# @MigrateUriParameter:
>> +#
>> +# Information regarding which source interface is connected to which
>> +# destination interface and number of multifd channels over each interface.
>> +#
>> +# @source-uri: the Uniform Resource Identifier of the source VM.
>> +# Default port number is 0.
>> +#
>> +# @destination-uri: the Uniform Resource Identifier of the destination VM
>> +#
>> +# @multifd-channels: number of parallel multifd channels used to migrate data
>> +# for specific source-uri and destination-uri. Default value
>> +# in this case is 2 (Since 4.0)
> You mean "(Since 7.1)", I guess.
> Yes yes. Also David pointed this thing out. I will update the version
in the v2 patchset.
>> +#
>> +##
>> +{ 'struct' : 'MigrateUriParameter',
>> + 'data' : { 'source-uri' : 'str',
>> + 'destination-uri' : 'str',
>> + '*multifd-channels' : 'uint8'} }
>> +
>> ##
>> # @migrate:
>> #
>> # Migrates the current running guest to another Virtual Machine.
>> #
>> # @uri: the Uniform Resource Identifier of the destination VM
>> +# for migration thread
>> +#
>> +# @multi-fd-uri-list: list of pair of source and destination VM Uniform
>> +# Resource Identifiers with number of multifd-channels
>> +# for each pair
>> #
>> # @blk: do block migration (full disk copy)
>> #
>> @@ -1479,20 +1505,27 @@
>> # 1. The 'query-migrate' command should be used to check migration's progress
>> # and final result (this information is provided by the 'status' member)
>> #
>> -# 2. All boolean arguments default to false
>> +# 2. The uri argument should have the Uniform Resource Identifier of default
>> +# destination VM. This connection will be bound to default network
>> +#
>> +# 3. All boolean arguments default to false
>> #
>> -# 3. The user Monitor's "detach" argument is invalid in QMP and should not
>> +# 4. The user Monitor's "detach" argument is invalid in QMP and should not
>> # be used
>> #
>> # Example:
>> #
>> -# -> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
>> +# -> { "execute": "migrate",
>> +# "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
>> +# "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
>> +# "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
>> +# "destination-uri": "tcp:11.0.0.0:7789", "multifd-channels": 5} ] } }
>> # <- { "return": {} }
>> #
>> ##
>> { 'command': 'migrate',
>> - 'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool',
>> - '*detach': 'bool', '*resume': 'bool' } }
>> + 'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateUriParameter'], '*blk': 'bool',
> Long line.
> Okay, acknowledged. Even for example, should it be under 80
characters per line, or that is fine?
>
>> + '*inc': 'bool', '*detach': 'bool', '*resume': 'bool' } }
>>
>> ##
>> # @migrate-incoming:
Regards,
Het Gala
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair
2022-07-18 13:33 ` Het Gala
@ 2022-07-18 14:33 ` Markus Armbruster
2022-07-18 15:17 ` Het Gala
0 siblings, 1 reply; 34+ messages in thread
From: Markus Armbruster @ 2022-07-18 14:33 UTC (permalink / raw)
To: Het Gala
Cc: qemu-devel, quintela, dgilbert, pbonzini, berrange, eblake,
Manish Mishra
Het Gala <het.gala@nutanix.com> writes:
> On 18/07/22 2:05 pm, Markus Armbruster wrote:
>> Het Gala <het.gala@nutanix.com> writes:
>>
>>> i) Modified the format of the qemu monitor command : 'migrate' by adding a list,
>>> each element in the list consists of multi-FD connection parameters: source
>>> and destination uris and of the number of multi-fd channels between each pair.
>>>
>>> ii) Information of all multi-FD connection parameters’ list, length of the list
>>> and total number of multi-fd channels for all the connections together is
>>> stored in ‘OutgoingArgs’ struct.
>>>
>>> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
>>> Signed-off-by: Het Gala <het.gala@nutanix.com>
>>> ---
>>> include/qapi/util.h | 9 ++++++++
>>> migration/migration.c | 47 ++++++++++++++++++++++++++++++--------
>>> migration/socket.c | 53 ++++++++++++++++++++++++++++++++++++++++---
>>> migration/socket.h | 17 +++++++++++++-
>>> monitor/hmp-cmds.c | 22 ++++++++++++++++--
>>> qapi/migration.json | 43 +++++++++++++++++++++++++++++++----
>>> 6 files changed, 170 insertions(+), 21 deletions(-)
>>>
>>> diff --git a/include/qapi/util.h b/include/qapi/util.h
>>> index 81a2b13a33..3041feb3d9 100644
>>> --- a/include/qapi/util.h
>>> +++ b/include/qapi/util.h
>>> @@ -56,4 +56,13 @@ int parse_qapi_name(const char *name, bool complete);
>>> (tail) = &(*(tail))->next; \
>>> } while (0)
>>>
>>> +#define QAPI_LIST_LENGTH(list) ({ \
>>> + int _len = 0; \
>>> + typeof(list) _elem; \
>>> + for (_elem = list; _elem != NULL; _elem = _elem->next) { \
>>> + _len++; \
>>> + } \
>>> + _len; \
>>> +})
>>> +
>>
>> Unless there is a compelling reason for open-coding this, make it a
>> (non-inline) function.
>>
> if kept as a function, the datatype of list is different every time
> in the qemu code, and that led to multiple copies of this function. So
> we decided, it would be best to keep it as a macro defined function. We
> would be happy to hear any suggstions to solve this problem while the
> function non-inline.
Point taken.
You could make it a function taking a GenericList *, but then you'd have
to cast actual list pointers to GenericList, which isn't nice.
>>> #endif
>>> diff --git a/migration/migration.c b/migration/migration.c
>>> index 31739b2af9..c408175aeb 100644
>>> --- a/migration/migration.c
>>> +++ b/migration/migration.c
>>> @@ -2328,13 +2328,14 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,
>>> return true;
>>> }
>>>
>>> -void qmp_migrate(const char *uri, bool has_blk, bool blk,
>>> +void qmp_migrate(const char *uri, bool has_multi_fd_uri_list,
>>> + MigrateUriParameterList *cap, bool has_blk, bool blk,
>>> bool has_inc, bool inc, bool has_detach, bool detach,
>>> bool has_resume, bool resume, Error **errp)
>>> {
>>> Error *local_err = NULL;
>>> MigrationState *s = migrate_get_current();
>>> - const char *p = NULL;
>>> + const char *dst_ptr = NULL;
>>>
>>> if (!migrate_prepare(s, has_blk && blk, has_inc && inc,
>>> has_resume && resume, errp)) {
>>> @@ -2348,20 +2349,46 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
>>> }
>>> }
>>>
>>> + /*
>>> + * In case of Multi-FD migration parameters, if uri is provided,
>>> + * supports only tcp network protocol.
>>> + */
>>> + if (has_multi_fd_uri_list) {
>>> + int length = QAPI_LIST_LENGTH(cap);
>>> + init_multifd_array(length);
>>> + for (int i = 0; i < length; i++) {
>>> + const char *p1 = NULL, *p2 = NULL;
>>> + const char *multifd_dst_uri = cap->value->destination_uri;
>>> + const char *multifd_src_uri = cap->value->source_uri;
>>> + uint8_t multifd_channels = cap->value->multifd_channels;
>>> + if (!strstart(multifd_dst_uri, "tcp:", &p1) ||
>>> + !strstart(multifd_src_uri, "tcp:", &p2)) {
>>> + error_setg(errp, "multi-fd destination and multi-fd source "
>>> + "uri, both should be present and follows tcp protocol only");
>>> + break;
>>> + } else {
>>> + store_multifd_migration_params(p1 ? p1 : multifd_dst_uri,
>>> + p2 ? p2 : multifd_src_uri,
>>> + multifd_channels, i, &local_err);
>>> + }
>>> + cap = cap->next;
>>> + }
>>> + }
>>> +
>>> migrate_protocol_allow_multi_channels(false);
>>> - if (strstart(uri, "tcp:", &p) ||
>>> + if (strstart(uri, "tcp:", &dst_ptr) ||
>>> strstart(uri, "unix:", NULL) ||
>>> strstart(uri, "vsock:", NULL)) {
>>> migrate_protocol_allow_multi_channels(true);
>>> - socket_start_outgoing_migration(s, p ? p : uri, &local_err);
>>> + socket_start_outgoing_migration(s, dst_ptr ? dst_ptr : uri, &local_err);
>>> #ifdef CONFIG_RDMA
>>> - } else if (strstart(uri, "rdma:", &p)) {
>>> - rdma_start_outgoing_migration(s, p, &local_err);
>>> + } else if (strstart(uri, "rdma:", &dst_ptr)) {
>>> + rdma_start_outgoing_migration(s, dst_ptr, &local_err);
>>> #endif
>>> - } else if (strstart(uri, "exec:", &p)) {
>>> - exec_start_outgoing_migration(s, p, &local_err);
>>> - } else if (strstart(uri, "fd:", &p)) {
>>> - fd_start_outgoing_migration(s, p, &local_err);
>>> + } else if (strstart(uri, "exec:", &dst_ptr)) {
>>> + exec_start_outgoing_migration(s, dst_ptr, &local_err);
>>> + } else if (strstart(uri, "fd:", &dst_ptr)) {
>>> + fd_start_outgoing_migration(s, dst_ptr, &local_err);
>>> } else {
>>> if (!(has_resume && resume)) {
>>> yank_unregister_instance(MIGRATION_YANK_INSTANCE);
>>> diff --git a/migration/socket.c b/migration/socket.c
>>> index 4fd5e85f50..7ca6af8cca 100644
>>> --- a/migration/socket.c
>>> +++ b/migration/socket.c
>>> @@ -32,6 +32,17 @@ struct SocketOutgoingArgs {
>>> SocketAddress *saddr;
>>> } outgoing_args;
>>>
>>> +struct SocketArgs {
>>> + struct SrcDestAddr data;
>>> + uint8_t multifd_channels;
>>> +};
>>> +
>>> +struct OutgoingMigrateParams {
>>> + struct SocketArgs *socket_args;
>>> + size_t length;
>>
>> Length of what?
>
> length of the socket_args[] array. Thanks for pointing it out. I will
> be more specific for this variable in the v2 patchset series.
>
>>> + uint64_t total_multifd_channel;
>>
>> @total_multifd_channels appears to be the sum of the
>> socket_args[*].multifd_channels. Correct?
>
> Yes Markus, you are correct.
Sure you need to keep the sum separately?
>>> +} outgoing_migrate_params;
>>> +
>>> void socket_send_channel_create(QIOTaskFunc f, void *data)
>>> {
>>> QIOChannelSocket *sioc = qio_channel_socket_new();
>>> @@ -47,6 +58,14 @@ int socket_send_channel_destroy(QIOChannel *send)
>>> qapi_free_SocketAddress(outgoing_args.saddr);
>>> outgoing_args.saddr = NULL;
>>> }
>>> +
>>> + if (outgoing_migrate_params.socket_args != NULL) {
>>> + g_free(outgoing_migrate_params.socket_args);
>>> + outgoing_migrate_params.socket_args = NULL;
>>> + }
>>> + if (outgoing_migrate_params.length) {
>>> + outgoing_migrate_params.length = 0;
>>> + }
>>> return 0;
>>> }
>>>
>>> @@ -117,13 +136,41 @@ socket_start_outgoing_migration_internal(MigrationState *s,
>>> }
>>>
>>> void socket_start_outgoing_migration(MigrationState *s,
>>> - const char *str,
>>> + const char *dst_str,
>>> Error **errp)
>>> {
>>> Error *err = NULL;
>>> - SocketAddress *saddr = socket_parse(str, &err);
>>> + SocketAddress *dst_saddr = socket_parse(dst_str, &err);
>>> + if (!err) {
>>> + socket_start_outgoing_migration_internal(s, dst_saddr, &err);
>>> + }
>>> + error_propagate(errp, err);
>>> +}
>>> +
>>> +void init_multifd_array(int length)
>>> +{
>>> + outgoing_migrate_params.socket_args = g_new0(struct SocketArgs, length);
>>> + outgoing_migrate_params.length = length;
>>> + outgoing_migrate_params.total_multifd_channel = 0;
>>> +}
>>> +
>>> +void store_multifd_migration_params(const char *dst_uri,
>>> + const char *src_uri,
>>> + uint8_t multifd_channels,
>>> + int idx, Error **errp)
>>> +{
>>> + Error *err = NULL;
>>> + SocketAddress *src_addr = NULL;
>>> + SocketAddress *dst_addr = socket_parse(dst_uri, &err);
>>> + if (src_uri) {
>>> + src_addr = socket_parse(src_uri, &err);
>>> + }
>>
>> Incorrect use of &err. error.h's big comment:
>>
>> * Receive and accumulate multiple errors (first one wins):
>> * Error *err = NULL, *local_err = NULL;
>> * foo(arg, &err);
>> * bar(arg, &local_err);
>> * error_propagate(&err, local_err);
>> * if (err) {
>> * handle the error...
>> * }
>> *
>> * Do *not* "optimize" this to
>> * Error *err = NULL;
>> * foo(arg, &err);
>> * bar(arg, &err); // WRONG!
>> * if (err) {
>> * handle the error...
>> * }
>> * because this may pass a non-null err to bar().
>>
> Thankyou Markus for sharing this knowledge. I was unaware of the
> dont's with &err.
The big comment should help you along. If it doesn't, just ask.
>>> if (!err) {
>>> - socket_start_outgoing_migration_internal(s, saddr, &err);
>>> + outgoing_migrate_params.socket_args[idx].data.dst_addr = dst_addr;
>>> + outgoing_migrate_params.socket_args[idx].data.src_addr = src_addr;
>>> + outgoing_migrate_params.socket_args[idx].multifd_channels
>>> + = multifd_channels;
>>> + outgoing_migrate_params.total_multifd_channel += multifd_channels;
>>> }
>>> error_propagate(errp, err);
>>
>> Consider
>>
>> struct SocketArgs *sa = &outgoing_migrate_params.socket_args[idx];
>> SocketAddress *src_addr, *dst_addr;
>>
>> src_addr = socketaddress(src_uri, errp);
>> if (!src_addr) {
>> return;
>> }
>>
>> dst_addr = socketaddress(dst_uri, errp);
>> if (!dst_addr) {
>> return;
>> }
>>
>> sa->data.dst_addr = dst_addr;
>> sa->data.src_addr = src_addr;
>> sa->multifd_channels = multifd_channels;
>> outgoing_migrate_params.total_multifd_channel += multifd_channels;
>
> Thanks Markus for this amazing suggestion. Your approach looks
> simpler to understand and also resolves the error it had with &err. I
> will surely implement this in the upcoming v2 patchset.
You're welcome :)
>>> }
>>> diff --git a/migration/socket.h b/migration/socket.h
>>> index 891dbccceb..bba7f177fe 100644
>>> --- a/migration/socket.h
>>> +++ b/migration/socket.h
>>> @@ -19,12 +19,27 @@
>>>
>>> #include "io/channel.h"
>>> #include "io/task.h"
>>> +#include "migration.h"
>>> +
>>> +/* info regarding destination and source uri */
>>> +struct SrcDestAddr {
>>> + SocketAddress *dst_addr;
>>> + SocketAddress *src_addr;
>>> +};
>>
>> QEMU coding style wants a typedef.
>
> Okay, acknowledged.
>
>>>
>>> void socket_send_channel_create(QIOTaskFunc f, void *data);
>>> int socket_send_channel_destroy(QIOChannel *send);
>>>
>>> void socket_start_incoming_migration(const char *str, Error **errp);
>>>
>>> -void socket_start_outgoing_migration(MigrationState *s, const char *str,
>>> +void socket_start_outgoing_migration(MigrationState *s, const char *dst_str,
>>> Error **errp);
>>> +
>>> +int multifd_list_length(MigrateUriParameterList *list);
>>> +
>>> +void init_multifd_array(int length);
>>> +
>>> +void store_multifd_migration_params(const char *dst_uri, const char *src_uri,
>>> + uint8_t multifd_channels, int idx,
>>> + Error **erp);
>>> #endif
>>> diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
>>> index 622c783c32..2db539016a 100644
>>> --- a/monitor/hmp-cmds.c
>>> +++ b/monitor/hmp-cmds.c
>>> @@ -56,6 +56,9 @@
>>> #include "migration/snapshot.h"
>>> #include "migration/misc.h"
>>>
>>> +/* Default number of multi-fd channels */
>>> +#define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2
>>> +
>>> #ifdef CONFIG_SPICE
>>> #include <spice/enums.h>
>>> #endif
>>> @@ -1574,10 +1577,25 @@ void hmp_migrate(Monitor *mon, const QDict *qdict)
>>> bool inc = qdict_get_try_bool(qdict, "inc", false);
>>> bool resume = qdict_get_try_bool(qdict, "resume", false);
>>> const char *uri = qdict_get_str(qdict, "uri");
>>> +
>>> + const char *src_uri = qdict_get_str(qdict, "source-uri");
>>> + const char *dst_uri = qdict_get_str(qdict, "destination-uri");
>>> + uint8_t multifd_channels = qdict_get_try_int(qdict, "multifd-channels",
>>> + DEFAULT_MIGRATE_MULTIFD_CHANNELS);
>>> Error *err = NULL;
>>> + MigrateUriParameterList *caps = NULL;
>>> + MigrateUriParameter *value;
>>> +
>>> + value = g_malloc0(sizeof(*value));
>>> + value->source_uri = (char *)src_uri;
>>> + value->destination_uri = (char *)dst_uri;
>>> + value->multifd_channels = multifd_channels;
>>> + QAPI_LIST_PREPEND(caps, value);
>>> +
>>> + qmp_migrate(uri, !!caps, caps, !!blk, blk, !!inc,
>>> + inc, false, false, true, resume, &err);
>>> + qapi_free_MigrateUriParameterList(caps);
>>>
>>> - qmp_migrate(uri, !!blk, blk, !!inc, inc,
>>> - false, false, true, resume, &err);
>>> if (hmp_handle_error(mon, err)) {
>>> return;
>>> }
>>> diff --git a/qapi/migration.json b/qapi/migration.json
>>> index 6130cd9fae..fb259d626b 100644
>>> --- a/qapi/migration.json
>>> +++ b/qapi/migration.json
>>> @@ -1454,12 +1454,38 @@
>>> ##
>>> { 'command': 'migrate-continue', 'data': {'state': 'MigrationStatus'} }
>>>
>>> +##
>>> +# @MigrateUriParameter:
>>> +#
>>> +# Information regarding which source interface is connected to which
>>> +# destination interface and number of multifd channels over each interface.
>>> +#
>>> +# @source-uri: the Uniform Resource Identifier of the source VM.
>>> +# Default port number is 0.
>>> +#
>>> +# @destination-uri: the Uniform Resource Identifier of the destination VM
>>> +#
>>> +# @multifd-channels: number of parallel multifd channels used to migrate data
>>> +# for specific source-uri and destination-uri. Default value
>>> +# in this case is 2 (Since 4.0)
>>
>> You mean "(Since 7.1)", I guess.
>
> Yes yes. Also David pointed this thing out. I will update the version
> in the v2 patchset.
>
>>> +#
>>> +##
>>> +{ 'struct' : 'MigrateUriParameter',
>>> + 'data' : { 'source-uri' : 'str',
>>> + 'destination-uri' : 'str',
>>> + '*multifd-channels' : 'uint8'} }
>>> +
>>> ##
>>> # @migrate:
>>> #
>>> # Migrates the current running guest to another Virtual Machine.
>>> #
>>> # @uri: the Uniform Resource Identifier of the destination VM
>>> +# for migration thread
>>> +#
>>> +# @multi-fd-uri-list: list of pair of source and destination VM Uniform
>>> +# Resource Identifiers with number of multifd-channels
>>> +# for each pair
>>> #
>>> # @blk: do block migration (full disk copy)
>>> #
>>> @@ -1479,20 +1505,27 @@
>>> # 1. The 'query-migrate' command should be used to check migration's progress
>>> # and final result (this information is provided by the 'status' member)
>>> #
>>> -# 2. All boolean arguments default to false
>>> +# 2. The uri argument should have the Uniform Resource Identifier of default
>>> +# destination VM. This connection will be bound to default network
>>> +#
>>> +# 3. All boolean arguments default to false
>>> #
>>> -# 3. The user Monitor's "detach" argument is invalid in QMP and should not
>>> +# 4. The user Monitor's "detach" argument is invalid in QMP and should not
>>> # be used
>>> #
>>> # Example:
>>> #
>>> -# -> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
>>> +# -> { "execute": "migrate",
>>> +# "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
>>> +# "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
>>> +# "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
>>> +# "destination-uri": "tcp:11.0.0.0:7789", "multifd-channels": 5} ] } }
>>> # <- { "return": {} }
>>> #
>>> ##
>>> { 'command': 'migrate',
>>> - 'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool',
>>> - '*detach': 'bool', '*resume': 'bool' } }
>>> + 'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateUriParameter'], '*blk': 'bool',
??
>> Long line.
>
> Okay, acknowledged. Even for example, should it be under 80
> characters per line, or that is fine?
docs/devel/style.rst:
Line width
==========
Lines should be 80 characters; try not to make them longer.
Sometimes it is hard to do, especially when dealing with QEMU subsystems
that use long function or symbol names. If wrapping the line at 80 columns
is obviously less readable and more awkward, prefer not to wrap it; better
to have an 85 character line than one which is awkwardly wrapped.
Even in that case, try not to make lines much longer than 80 characters.
(The checkpatch script will warn at 100 characters, but this is intended
as a guard against obviously-overlength lines, not a target.)
Personally, I very much prefer to wrap between 70 and 75 except where it
impairs legibility.
>>
>>> + '*inc': 'bool', '*detach': 'bool', '*resume': 'bool' } }
>>>
>>> ##
>>> # @migrate-incoming:
>
> Regards,
>
> Het Gala
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair
2022-07-18 14:33 ` Markus Armbruster
@ 2022-07-18 15:17 ` Het Gala
2022-07-19 7:06 ` Markus Armbruster
0 siblings, 1 reply; 34+ messages in thread
From: Het Gala @ 2022-07-18 15:17 UTC (permalink / raw)
To: Markus Armbruster
Cc: qemu-devel, quintela, dgilbert, pbonzini, berrange, eblake,
Manish Mishra
On 18/07/22 8:03 pm, Markus Armbruster wrote:
> Het Gala <het.gala@nutanix.com> writes:
>
>> On 18/07/22 2:05 pm, Markus Armbruster wrote:
>>> Het Gala <het.gala@nutanix.com> writes:
>>>
>>>> i) Modified the format of the qemu monitor command : 'migrate' by adding a list,
>>>> each element in the list consists of multi-FD connection parameters: source
>>>> and destination uris and of the number of multi-fd channels between each pair.
>>>>
>>>> ii) Information of all multi-FD connection parameters’ list, length of the list
>>>> and total number of multi-fd channels for all the connections together is
>>>> stored in ‘OutgoingArgs’ struct.
>>>>
>>>> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
>>>> Signed-off-by: Het Gala <het.gala@nutanix.com>
>>>> ---
>>>> include/qapi/util.h | 9 ++++++++
>>>> migration/migration.c | 47 ++++++++++++++++++++++++++++++--------
>>>> migration/socket.c | 53 ++++++++++++++++++++++++++++++++++++++++---
>>>> migration/socket.h | 17 +++++++++++++-
>>>> monitor/hmp-cmds.c | 22 ++++++++++++++++--
>>>> qapi/migration.json | 43 +++++++++++++++++++++++++++++++----
>>>> 6 files changed, 170 insertions(+), 21 deletions(-)
>>>>
>>>> diff --git a/include/qapi/util.h b/include/qapi/util.h
>>>> index 81a2b13a33..3041feb3d9 100644
>>>> --- a/include/qapi/util.h
>>>> +++ b/include/qapi/util.h
>>>> @@ -56,4 +56,13 @@ int parse_qapi_name(const char *name, bool complete);
>>>> (tail) = &(*(tail))->next; \
>>>> } while (0)
>>>>
>>>> +#define QAPI_LIST_LENGTH(list) ({ \
>>>> + int _len = 0; \
>>>> + typeof(list) _elem; \
>>>> + for (_elem = list; _elem != NULL; _elem = _elem->next) { \
>>>> + _len++; \
>>>> + } \
>>>> + _len; \
>>>> +})
>>>> +
>>> Unless there is a compelling reason for open-coding this, make it a
>>> (non-inline) function.
>>>
>> if kept as a function, the datatype of list is different every time
>> in the qemu code, and that led to multiple copies of this function. So
>> we decided, it would be best to keep it as a macro defined function. We
>> would be happy to hear any suggstions to solve this problem while the
>> function non-inline.
> Point taken.
>
> You could make it a function taking a GenericList *, but then you'd have
> to cast actual list pointers to GenericList, which isn't nice.
>
>>>> #endif
>>>> diff --git a/migration/migration.c b/migration/migration.c
>>>> index 31739b2af9..c408175aeb 100644
>>>> --- a/migration/migration.c
>>>> +++ b/migration/migration.c
>>>> @@ -2328,13 +2328,14 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,
>>>> return true;
>>>> }
>>>>
>>>> -void qmp_migrate(const char *uri, bool has_blk, bool blk,
>>>> +void qmp_migrate(const char *uri, bool has_multi_fd_uri_list,
>>>> + MigrateUriParameterList *cap, bool has_blk, bool blk,
>>>> bool has_inc, bool inc, bool has_detach, bool detach,
>>>> bool has_resume, bool resume, Error **errp)
>>>> {
>>>> Error *local_err = NULL;
>>>> MigrationState *s = migrate_get_current();
>>>> - const char *p = NULL;
>>>> + const char *dst_ptr = NULL;
>>>>
>>>> if (!migrate_prepare(s, has_blk && blk, has_inc && inc,
>>>> has_resume && resume, errp)) {
>>>> @@ -2348,20 +2349,46 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
>>>> }
>>>> }
>>>>
>>>> + /*
>>>> + * In case of Multi-FD migration parameters, if uri is provided,
>>>> + * supports only tcp network protocol.
>>>> + */
>>>> + if (has_multi_fd_uri_list) {
>>>> + int length = QAPI_LIST_LENGTH(cap);
>>>> + init_multifd_array(length);
>>>> + for (int i = 0; i < length; i++) {
>>>> + const char *p1 = NULL, *p2 = NULL;
>>>> + const char *multifd_dst_uri = cap->value->destination_uri;
>>>> + const char *multifd_src_uri = cap->value->source_uri;
>>>> + uint8_t multifd_channels = cap->value->multifd_channels;
>>>> + if (!strstart(multifd_dst_uri, "tcp:", &p1) ||
>>>> + !strstart(multifd_src_uri, "tcp:", &p2)) {
>>>> + error_setg(errp, "multi-fd destination and multi-fd source "
>>>> + "uri, both should be present and follows tcp protocol only");
>>>> + break;
>>>> + } else {
>>>> + store_multifd_migration_params(p1 ? p1 : multifd_dst_uri,
>>>> + p2 ? p2 : multifd_src_uri,
>>>> + multifd_channels, i, &local_err);
>>>> + }
>>>> + cap = cap->next;
>>>> + }
>>>> + }
>>>> +
>>>> migrate_protocol_allow_multi_channels(false);
>>>> - if (strstart(uri, "tcp:", &p) ||
>>>> + if (strstart(uri, "tcp:", &dst_ptr) ||
>>>> strstart(uri, "unix:", NULL) ||
>>>> strstart(uri, "vsock:", NULL)) {
>>>> migrate_protocol_allow_multi_channels(true);
>>>> - socket_start_outgoing_migration(s, p ? p : uri, &local_err);
>>>> + socket_start_outgoing_migration(s, dst_ptr ? dst_ptr : uri, &local_err);
>>>> #ifdef CONFIG_RDMA
>>>> - } else if (strstart(uri, "rdma:", &p)) {
>>>> - rdma_start_outgoing_migration(s, p, &local_err);
>>>> + } else if (strstart(uri, "rdma:", &dst_ptr)) {
>>>> + rdma_start_outgoing_migration(s, dst_ptr, &local_err);
>>>> #endif
>>>> - } else if (strstart(uri, "exec:", &p)) {
>>>> - exec_start_outgoing_migration(s, p, &local_err);
>>>> - } else if (strstart(uri, "fd:", &p)) {
>>>> - fd_start_outgoing_migration(s, p, &local_err);
>>>> + } else if (strstart(uri, "exec:", &dst_ptr)) {
>>>> + exec_start_outgoing_migration(s, dst_ptr, &local_err);
>>>> + } else if (strstart(uri, "fd:", &dst_ptr)) {
>>>> + fd_start_outgoing_migration(s, dst_ptr, &local_err);
>>>> } else {
>>>> if (!(has_resume && resume)) {
>>>> yank_unregister_instance(MIGRATION_YANK_INSTANCE);
>>>> diff --git a/migration/socket.c b/migration/socket.c
>>>> index 4fd5e85f50..7ca6af8cca 100644
>>>> --- a/migration/socket.c
>>>> +++ b/migration/socket.c
>>>> @@ -32,6 +32,17 @@ struct SocketOutgoingArgs {
>>>> SocketAddress *saddr;
>>>> } outgoing_args;
>>>>
>>>> +struct SocketArgs {
>>>> + struct SrcDestAddr data;
>>>> + uint8_t multifd_channels;
>>>> +};
>>>> +
>>>> +struct OutgoingMigrateParams {
>>>> + struct SocketArgs *socket_args;
>>>> + size_t length;
>>> Length of what?
>> length of the socket_args[] array. Thanks for pointing it out. I will
>> be more specific for this variable in the v2 patchset series.
>>
>>>> + uint64_t total_multifd_channel;
>>> @total_multifd_channels appears to be the sum of the
>>> socket_args[*].multifd_channels. Correct?
>> Yes Markus, you are correct.
> Sure you need to keep the sum separately?
> So earlier, the idea behind this was that, we had this intention to
depreciate the migrate_multifd_channels() API from the live migration
process. We made total_multifd_channels() function, where it used to
calculate total number of multifd channels every time, for whichever
function called was computation internsive so we replaced it by
returning sum of socket_args[*].multifd_channels i.e.
total_multifd_channel in the later patches.
But now in the v2 patchset onwards, Thanks to inputs from Dr. David
and Daniel, we are not depricating migrate_multifd_channels() API but
the value from the API will be cross-referenced with sum of
socket_args[*].multifd_channels i.e. total_multifd_channel, and error if
they are not equal.
>
>>>> +} outgoing_migrate_params;
>>>> +
>>>> void socket_send_channel_create(QIOTaskFunc f, void *data)
>>>> {
>>>> QIOChannelSocket *sioc = qio_channel_socket_new();
>>>> @@ -47,6 +58,14 @@ int socket_send_channel_destroy(QIOChannel *send)
>>>> qapi_free_SocketAddress(outgoing_args.saddr);
>>>> outgoing_args.saddr = NULL;
>>>> }
>>>> +
>>>> + if (outgoing_migrate_params.socket_args != NULL) {
>>>> + g_free(outgoing_migrate_params.socket_args);
>>>> + outgoing_migrate_params.socket_args = NULL;
>>>> + }
>>>> + if (outgoing_migrate_params.length) {
>>>> + outgoing_migrate_params.length = 0;
>>>> + }
>>>> return 0;
>>>> }
>>>>
>>>> @@ -117,13 +136,41 @@ socket_start_outgoing_migration_internal(MigrationState *s,
>>>> }
>>>>
>>>> void socket_start_outgoing_migration(MigrationState *s,
>>>> - const char *str,
>>>> + const char *dst_str,
>>>> Error **errp)
>>>> {
>>>> Error *err = NULL;
>>>> - SocketAddress *saddr = socket_parse(str, &err);
>>>> + SocketAddress *dst_saddr = socket_parse(dst_str, &err);
>>>> + if (!err) {
>>>> + socket_start_outgoing_migration_internal(s, dst_saddr, &err);
>>>> + }
>>>> + error_propagate(errp, err);
>>>> +}
>>>> +
>>>> +void init_multifd_array(int length)
>>>> +{
>>>> + outgoing_migrate_params.socket_args = g_new0(struct SocketArgs, length);
>>>> + outgoing_migrate_params.length = length;
>>>> + outgoing_migrate_params.total_multifd_channel = 0;
>>>> +}
>>>> +
>>>> +void store_multifd_migration_params(const char *dst_uri,
>>>> + const char *src_uri,
>>>> + uint8_t multifd_channels,
>>>> + int idx, Error **errp)
>>>> +{
>>>> + Error *err = NULL;
>>>> + SocketAddress *src_addr = NULL;
>>>> + SocketAddress *dst_addr = socket_parse(dst_uri, &err);
>>>> + if (src_uri) {
>>>> + src_addr = socket_parse(src_uri, &err);
>>>> + }
>>> Incorrect use of &err. error.h's big comment:
>>>
>>> * Receive and accumulate multiple errors (first one wins):
>>> * Error *err = NULL, *local_err = NULL;
>>> * foo(arg, &err);
>>> * bar(arg, &local_err);
>>> * error_propagate(&err, local_err);
>>> * if (err) {
>>> * handle the error...
>>> * }
>>> *
>>> * Do *not* "optimize" this to
>>> * Error *err = NULL;
>>> * foo(arg, &err);
>>> * bar(arg, &err); // WRONG!
>>> * if (err) {
>>> * handle the error...
>>> * }
>>> * because this may pass a non-null err to bar().
>>>
>> Thankyou Markus for sharing this knowledge. I was unaware of the
>> dont's with &err.
> The big comment should help you along. If it doesn't, just ask.
> I read the comment, and it is pretty well explained out there.
>
>>>> if (!err) {
>>>> - socket_start_outgoing_migration_internal(s, saddr, &err);
>>>> + outgoing_migrate_params.socket_args[idx].data.dst_addr = dst_addr;
>>>> + outgoing_migrate_params.socket_args[idx].data.src_addr = src_addr;
>>>> + outgoing_migrate_params.socket_args[idx].multifd_channels
>>>> + = multifd_channels;
>>>> + outgoing_migrate_params.total_multifd_channel += multifd_channels;
>>>> }
>>>> error_propagate(errp, err);
>>> Consider
>>>
>>> struct SocketArgs *sa = &outgoing_migrate_params.socket_args[idx];
>>> SocketAddress *src_addr, *dst_addr;
>>>
>>> src_addr = socketaddress(src_uri, errp);
>>> if (!src_addr) {
>>> return;
>>> }
>>>
>>> dst_addr = socketaddress(dst_uri, errp);
>>> if (!dst_addr) {
>>> return;
>>> }
>>>
>>> sa->data.dst_addr = dst_addr;
>>> sa->data.src_addr = src_addr;
>>> sa->multifd_channels = multifd_channels;
>>> outgoing_migrate_params.total_multifd_channel += multifd_channels;
>> Thanks Markus for this amazing suggestion. Your approach looks
>> simpler to understand and also resolves the error it had with &err. I
>> will surely implement this in the upcoming v2 patchset.
> You're welcome :)
> I just wanted to have a double check on the solution you gave above
Markus. The suggestion you have given there has been deliberately
written in that way right, because
src_addr = socketaddress(src_uri, errp);
dst_addr = socketaddress(dst_uri, errp);
if (!src_addr) {
return;
}
if (!dst_addr) {
return;
}
would still be an error right according to the &err guidelines from
error.h file.
>
>>>> }
>>>> diff --git a/migration/socket.h b/migration/socket.h
>>>> index 891dbccceb..bba7f177fe 100644
>>>> --- a/migration/socket.h
>>>> +++ b/migration/socket.h
>>>> @@ -19,12 +19,27 @@
>>>>
>>>> #include "io/channel.h"
>>>> #include "io/task.h"
>>>> +#include "migration.h"
>>>> +
>>>> +/* info regarding destination and source uri */
>>>> +struct SrcDestAddr {
>>>> + SocketAddress *dst_addr;
>>>> + SocketAddress *src_addr;
>>>> +};
>>> QEMU coding style wants a typedef.
>> Okay, acknowledged.
>>
>>>> void socket_send_channel_create(QIOTaskFunc f, void *data);
>>>> int socket_send_channel_destroy(QIOChannel *send);
>>>>
>>>> void socket_start_incoming_migration(const char *str, Error **errp);
>>>>
>>>> -void socket_start_outgoing_migration(MigrationState *s, const char *str,
>>>> +void socket_start_outgoing_migration(MigrationState *s, const char *dst_str,
>>>> Error **errp);
>>>> +
>>>> +int multifd_list_length(MigrateUriParameterList *list);
>>>> +
>>>> +void init_multifd_array(int length);
>>>> +
>>>> +void store_multifd_migration_params(const char *dst_uri, const char *src_uri,
>>>> + uint8_t multifd_channels, int idx,
>>>> + Error **erp);
>>>> #endif
>>>> diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
>>>> index 622c783c32..2db539016a 100644
>>>> --- a/monitor/hmp-cmds.c
>>>> +++ b/monitor/hmp-cmds.c
>>>> @@ -56,6 +56,9 @@
>>>> #include "migration/snapshot.h"
>>>> #include "migration/misc.h"
>>>>
>>>> +/* Default number of multi-fd channels */
>>>> +#define DEFAULT_MIGRATE_MULTIFD_CHANNELS 2
>>>> +
>>>> #ifdef CONFIG_SPICE
>>>> #include <spice/enums.h>
>>>> #endif
>>>> @@ -1574,10 +1577,25 @@ void hmp_migrate(Monitor *mon, const QDict *qdict)
>>>> bool inc = qdict_get_try_bool(qdict, "inc", false);
>>>> bool resume = qdict_get_try_bool(qdict, "resume", false);
>>>> const char *uri = qdict_get_str(qdict, "uri");
>>>> +
>>>> + const char *src_uri = qdict_get_str(qdict, "source-uri");
>>>> + const char *dst_uri = qdict_get_str(qdict, "destination-uri");
>>>> + uint8_t multifd_channels = qdict_get_try_int(qdict, "multifd-channels",
>>>> + DEFAULT_MIGRATE_MULTIFD_CHANNELS);
>>>> Error *err = NULL;
>>>> + MigrateUriParameterList *caps = NULL;
>>>> + MigrateUriParameter *value;
>>>> +
>>>> + value = g_malloc0(sizeof(*value));
>>>> + value->source_uri = (char *)src_uri;
>>>> + value->destination_uri = (char *)dst_uri;
>>>> + value->multifd_channels = multifd_channels;
>>>> + QAPI_LIST_PREPEND(caps, value);
>>>> +
>>>> + qmp_migrate(uri, !!caps, caps, !!blk, blk, !!inc,
>>>> + inc, false, false, true, resume, &err);
>>>> + qapi_free_MigrateUriParameterList(caps);
>>>>
>>>> - qmp_migrate(uri, !!blk, blk, !!inc, inc,
>>>> - false, false, true, resume, &err);
>>>> if (hmp_handle_error(mon, err)) {
>>>> return;
>>>> }
>>>> diff --git a/qapi/migration.json b/qapi/migration.json
>>>> index 6130cd9fae..fb259d626b 100644
>>>> --- a/qapi/migration.json
>>>> +++ b/qapi/migration.json
>>>> @@ -1454,12 +1454,38 @@
>>>> ##
>>>> { 'command': 'migrate-continue', 'data': {'state': 'MigrationStatus'} }
>>>>
>>>> +##
>>>> +# @MigrateUriParameter:
>>>> +#
>>>> +# Information regarding which source interface is connected to which
>>>> +# destination interface and number of multifd channels over each interface.
>>>> +#
>>>> +# @source-uri: the Uniform Resource Identifier of the source VM.
>>>> +# Default port number is 0.
>>>> +#
>>>> +# @destination-uri: the Uniform Resource Identifier of the destination VM
>>>> +#
>>>> +# @multifd-channels: number of parallel multifd channels used to migrate data
>>>> +# for specific source-uri and destination-uri. Default value
>>>> +# in this case is 2 (Since 4.0)
>>> You mean "(Since 7.1)", I guess.
>> Yes yes. Also David pointed this thing out. I will update the version
>> in the v2 patchset.
>>
>>>> +#
>>>> +##
>>>> +{ 'struct' : 'MigrateUriParameter',
>>>> + 'data' : { 'source-uri' : 'str',
>>>> + 'destination-uri' : 'str',
>>>> + '*multifd-channels' : 'uint8'} }
>>>> +
>>>> ##
>>>> # @migrate:
>>>> #
>>>> # Migrates the current running guest to another Virtual Machine.
>>>> #
>>>> # @uri: the Uniform Resource Identifier of the destination VM
>>>> +# for migration thread
>>>> +#
>>>> +# @multi-fd-uri-list: list of pair of source and destination VM Uniform
>>>> +# Resource Identifiers with number of multifd-channels
>>>> +# for each pair
>>>> #
>>>> # @blk: do block migration (full disk copy)
>>>> #
>>>> @@ -1479,20 +1505,27 @@
>>>> # 1. The 'query-migrate' command should be used to check migration's progress
>>>> # and final result (this information is provided by the 'status' member)
>>>> #
>>>> -# 2. All boolean arguments default to false
>>>> +# 2. The uri argument should have the Uniform Resource Identifier of default
>>>> +# destination VM. This connection will be bound to default network
>>>> +#
>>>> +# 3. All boolean arguments default to false
>>>> #
>>>> -# 3. The user Monitor's "detach" argument is invalid in QMP and should not
>>>> +# 4. The user Monitor's "detach" argument is invalid in QMP and should not
>>>> # be used
>>>> #
>>>> # Example:
>>>> #
>>>> -# -> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
>>>> +# -> { "execute": "migrate",
>>>> +# "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
>>>> +# "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
>>>> +# "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
>>>> +# "destination-uri": "tcp:11.0.0.0:7789", "multifd-channels": 5} ] } }
>>>> # <- { "return": {} }
>>>> #
>>>> ##
>>>> { 'command': 'migrate',
>>>> - 'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool',
>>>> - '*detach': 'bool', '*resume': 'bool' } }
>>>> + 'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateUriParameter'], '*blk': 'bool',
> ??
> Sorry Markus, I think the statement I wrote did not make sense, I
apologise for that. I meant to say example in the sense:
# Example:
#
# -> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
# -> { "execute": "migrate",
# "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
# "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
# "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
# "destination-uri": "tcp:11.0.0.0:7789", "multifd-channels": 5} ] } }
even this we should try to wrap within 80 character column right? or is
that okay to go beyond 80.
>>> Long line.
>> Okay, acknowledged. Even for example, should it be under 80
>> characters per line, or that is fine?
> docs/devel/style.rst:
>
> Line width
> ==========
>
> Lines should be 80 characters; try not to make them longer.
>
> Sometimes it is hard to do, especially when dealing with QEMU subsystems
> that use long function or symbol names. If wrapping the line at 80 columns
> is obviously less readable and more awkward, prefer not to wrap it; better
> to have an 85 character line than one which is awkwardly wrapped.
>
> Even in that case, try not to make lines much longer than 80 characters.
> (The checkpatch script will warn at 100 characters, but this is intended
> as a guard against obviously-overlength lines, not a target.)
>
> Personally, I very much prefer to wrap between 70 and 75 except where it
> impairs legibility.
> Okay thanks again Markus for your valuable suggestion. I will try to
wrap within 75 in almost all the cases.
>>>> + '*inc': 'bool', '*detach': 'bool', '*resume': 'bool' } }
>>>>
>>>> ##
>>>> # @migrate-incoming:
>> Regards,
>>
>> Het Gala
Regards,
Het Gala
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair
2022-07-18 15:17 ` Het Gala
@ 2022-07-19 7:06 ` Markus Armbruster
2022-07-19 7:51 ` Het Gala
0 siblings, 1 reply; 34+ messages in thread
From: Markus Armbruster @ 2022-07-19 7:06 UTC (permalink / raw)
To: Het Gala
Cc: qemu-devel, quintela, dgilbert, pbonzini, berrange, eblake,
Manish Mishra
Het Gala <het.gala@nutanix.com> writes:
> On 18/07/22 8:03 pm, Markus Armbruster wrote:
>> Het Gala <het.gala@nutanix.com> writes:
>>
>>> On 18/07/22 2:05 pm, Markus Armbruster wrote:
>>>> Het Gala <het.gala@nutanix.com> writes:
>>>>
>>>>> i) Modified the format of the qemu monitor command : 'migrate' by adding a list,
>>>>> each element in the list consists of multi-FD connection parameters: source
>>>>> and destination uris and of the number of multi-fd channels between each pair.
>>>>>
>>>>> ii) Information of all multi-FD connection parameters’ list, length of the list
>>>>> and total number of multi-fd channels for all the connections together is
>>>>> stored in ‘OutgoingArgs’ struct.
>>>>>
>>>>> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
>>>>> Signed-off-by: Het Gala <het.gala@nutanix.com>
>>>>> ---
[...]
>>>>> diff --git a/migration/socket.c b/migration/socket.c
>>>>> index 4fd5e85f50..7ca6af8cca 100644
>>>>> --- a/migration/socket.c
>>>>> +++ b/migration/socket.c
>>>>> @@ -32,6 +32,17 @@ struct SocketOutgoingArgs {
>>>>> SocketAddress *saddr;
>>>>> } outgoing_args;
>>>>>
>>>>> +struct SocketArgs {
>>>>> + struct SrcDestAddr data;
>>>>> + uint8_t multifd_channels;
>>>>> +};
>>>>> +
>>>>> +struct OutgoingMigrateParams {
>>>>> + struct SocketArgs *socket_args;
>>>>> + size_t length;
>>>> Length of what?
>>> length of the socket_args[] array. Thanks for pointing it out. I will
>>> be more specific for this variable in the v2 patchset series.
>>>
>>>>> + uint64_t total_multifd_channel;
>>>> @total_multifd_channels appears to be the sum of the
>>>> socket_args[*].multifd_channels. Correct?
>>> Yes Markus, you are correct.
>> Sure you need to keep the sum separately?
>
> So earlier, the idea behind this was that, we had this intention to depreciate the migrate_multifd_channels() API from the live migration
> process. We made total_multifd_channels() function, where it used to calculate total number of multifd channels every time, for whichever
> function called was computation internsive so we replaced it by returning sum of socket_args[*].multifd_channels i.e.
> total_multifd_channel in the later patches.
>
> But now in the v2 patchset onwards, Thanks to inputs from Dr. David and Daniel, we are not depricating migrate_multifd_channels() API but
> the value from the API will be cross-referenced with sum of socket_args[*].multifd_channels i.e. total_multifd_channel, and error if
> they are not equal.
I'm afraid I don't understand. I'm not sure I have to. Let me loop
back to my question.
If @total_multifd_channel is always the sum of the
socket_args[*].multifd_channels, then you can always compute it on the
fly.
I.e. you can replace @total_multifd_channel by a function that returns
the sum.
Precomputing it instead is more complex, because then you need to
document that the two are the same. Also, bug oppertunity: letting them
deviate somehow. I figure that's worthwhile only if computing on the
fly is too expensive.
>>>>> +} outgoing_migrate_params;
>>>>> +
>>>>> void socket_send_channel_create(QIOTaskFunc f, void *data)
>>>>> {
>>>>> QIOChannelSocket *sioc = qio_channel_socket_new();
>>>>> @@ -47,6 +58,14 @@ int socket_send_channel_destroy(QIOChannel *send)
>>>>> qapi_free_SocketAddress(outgoing_args.saddr);
>>>>> outgoing_args.saddr = NULL;
>>>>> }
>>>>> +
>>>>> + if (outgoing_migrate_params.socket_args != NULL) {
>>>>> + g_free(outgoing_migrate_params.socket_args);
>>>>> + outgoing_migrate_params.socket_args = NULL;
>>>>> + }
>>>>> + if (outgoing_migrate_params.length) {
>>>>> + outgoing_migrate_params.length = 0;
>>>>> + }
>>>>> return 0;
>>>>> }
>>>>>
>>>>> @@ -117,13 +136,41 @@ socket_start_outgoing_migration_internal(MigrationState *s,
>>>>> }
>>>>>
>>>>> void socket_start_outgoing_migration(MigrationState *s,
>>>>> - const char *str,
>>>>> + const char *dst_str,
>>>>> Error **errp)
>>>>> {
>>>>> Error *err = NULL;
>>>>> - SocketAddress *saddr = socket_parse(str, &err);
>>>>> + SocketAddress *dst_saddr = socket_parse(dst_str, &err);
>>>>> + if (!err) {
>>>>> + socket_start_outgoing_migration_internal(s, dst_saddr, &err);
>>>>> + }
>>>>> + error_propagate(errp, err);
>>>>> +}
>>>>> +
>>>>> +void init_multifd_array(int length)
>>>>> +{
>>>>> + outgoing_migrate_params.socket_args = g_new0(struct SocketArgs, length);
>>>>> + outgoing_migrate_params.length = length;
>>>>> + outgoing_migrate_params.total_multifd_channel = 0;
>>>>> +}
>>>>> +
>>>>> +void store_multifd_migration_params(const char *dst_uri,
>>>>> + const char *src_uri,
>>>>> + uint8_t multifd_channels,
>>>>> + int idx, Error **errp)
>>>>> +{
>>>>> + Error *err = NULL;
>>>>> + SocketAddress *src_addr = NULL;
>>>>> + SocketAddress *dst_addr = socket_parse(dst_uri, &err);
>>>>> + if (src_uri) {
>>>>> + src_addr = socket_parse(src_uri, &err);
>>>>> + }
>>>> Incorrect use of &err. error.h's big comment:
>>>>
>>>> * Receive and accumulate multiple errors (first one wins):
>>>> * Error *err = NULL, *local_err = NULL;
>>>> * foo(arg, &err);
>>>> * bar(arg, &local_err);
>>>> * error_propagate(&err, local_err);
>>>> * if (err) {
>>>> * handle the error...
>>>> * }
>>>> *
>>>> * Do *not* "optimize" this to
>>>> * Error *err = NULL;
>>>> * foo(arg, &err);
>>>> * bar(arg, &err); // WRONG!
>>>> * if (err) {
>>>> * handle the error...
>>>> * }
>>>> * because this may pass a non-null err to bar().
>>>>
>>> Thankyou Markus for sharing this knowledge. I was unaware of the
>>> dont's with &err.
>> The big comment should help you along. If it doesn't, just ask.
>> I read the comment, and it is pretty well explained out there.
>>
>>>>> if (!err) {
>>>>> - socket_start_outgoing_migration_internal(s, saddr, &err);
>>>>> + outgoing_migrate_params.socket_args[idx].data.dst_addr = dst_addr;
>>>>> + outgoing_migrate_params.socket_args[idx].data.src_addr = src_addr;
>>>>> + outgoing_migrate_params.socket_args[idx].multifd_channels
>>>>> + = multifd_channels;
>>>>> + outgoing_migrate_params.total_multifd_channel += multifd_channels;
>>>>> }
>>>>> error_propagate(errp, err);
>>>> Consider
>>>>
>>>> struct SocketArgs *sa = &outgoing_migrate_params.socket_args[idx];
>>>> SocketAddress *src_addr, *dst_addr;
>>>>
>>>> src_addr = socketaddress(src_uri, errp);
>>>> if (!src_addr) {
>>>> return;
>>>> }
>>>>
>>>> dst_addr = socketaddress(dst_uri, errp);
>>>> if (!dst_addr) {
>>>> return;
>>>> }
>>>>
>>>> sa->data.dst_addr = dst_addr;
>>>> sa->data.src_addr = src_addr;
>>>> sa->multifd_channels = multifd_channels;
>>>> outgoing_migrate_params.total_multifd_channel += multifd_channels;
>>> Thanks Markus for this amazing suggestion. Your approach looks
>>> simpler to understand and also resolves the error it had with &err. I
>>> will surely implement this in the upcoming v2 patchset.
>> You're welcome :)
>
> I just wanted to have a double check on the solution you gave above Markus. The suggestion you have given there has been deliberately
> written in that way right, because
>
> src_addr = socketaddress(src_uri, errp);
> dst_addr = socketaddress(dst_uri, errp);
> if (!src_addr) {
> return;
> }
> if (!dst_addr) {
> return;
> }
>
> would still be an error right according to the &err guidelines from error.h file.
Correct.
>>>>> }
[...]
>>>>> diff --git a/qapi/migration.json b/qapi/migration.json
>>>>> index 6130cd9fae..fb259d626b 100644
>>>>> --- a/qapi/migration.json
>>>>> +++ b/qapi/migration.json
>>>>> @@ -1454,12 +1454,38 @@
>>>>> ##
>>>>> { 'command': 'migrate-continue', 'data': {'state': 'MigrationStatus'} }
>>>>>
>>>>> +##
>>>>> +# @MigrateUriParameter:
>>>>> +#
>>>>> +# Information regarding which source interface is connected to which
>>>>> +# destination interface and number of multifd channels over each interface.
>>>>> +#
>>>>> +# @source-uri: the Uniform Resource Identifier of the source VM.
>>>>> +# Default port number is 0.
>>>>> +#
>>>>> +# @destination-uri: the Uniform Resource Identifier of the destination VM
>>>>> +#
>>>>> +# @multifd-channels: number of parallel multifd channels used to migrate data
>>>>> +# for specific source-uri and destination-uri. Default value
>>>>> +# in this case is 2 (Since 4.0)
>>>> You mean "(Since 7.1)", I guess.
>>> Yes yes. Also David pointed this thing out. I will update the version
>>> in the v2 patchset.
>>>
>>>>> +#
>>>>> +##
>>>>> +{ 'struct' : 'MigrateUriParameter',
>>>>> + 'data' : { 'source-uri' : 'str',
>>>>> + 'destination-uri' : 'str',
>>>>> + '*multifd-channels' : 'uint8'} }
>>>>> +
>>>>> ##
>>>>> # @migrate:
>>>>> #
>>>>> # Migrates the current running guest to another Virtual Machine.
>>>>> #
>>>>> # @uri: the Uniform Resource Identifier of the destination VM
>>>>> +# for migration thread
>>>>> +#
>>>>> +# @multi-fd-uri-list: list of pair of source and destination VM Uniform
>>>>> +# Resource Identifiers with number of multifd-channels
>>>>> +# for each pair
>>>>> #
>>>>> # @blk: do block migration (full disk copy)
>>>>> #
>>>>> @@ -1479,20 +1505,27 @@
>>>>> # 1. The 'query-migrate' command should be used to check migration's progress
>>>>> # and final result (this information is provided by the 'status' member)
>>>>> #
>>>>> -# 2. All boolean arguments default to false
>>>>> +# 2. The uri argument should have the Uniform Resource Identifier of default
>>>>> +# destination VM. This connection will be bound to default network
>>>>> +#
>>>>> +# 3. All boolean arguments default to false
>>>>> #
>>>>> -# 3. The user Monitor's "detach" argument is invalid in QMP and should not
>>>>> +# 4. The user Monitor's "detach" argument is invalid in QMP and should not
>>>>> # be used
>>>>> #
>>>>> # Example:
>>>>> #
>>>>> -# -> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
>>>>> +# -> { "execute": "migrate",
>>>>> +# "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
>>>>> +# "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
>>>>> +# "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
>>>>> +# "destination-uri": "tcp:11.0.0.0:7789", "multifd-channels": 5} ] } }
>>>>> # <- { "return": {} }
>>>>> #
>>>>> ##
>>>>> { 'command': 'migrate',
>>>>> - 'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool',
>>>>> - '*detach': 'bool', '*resume': 'bool' } }
>>>>> + 'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateUriParameter'], '*blk': 'bool',
> ??
>
> Sorry Markus, I think the statement I wrote did not make sense, I apologise for that. I meant to say example in the sense:
>
> # Example:
> #
> # -> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
> # -> { "execute": "migrate",
> # "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
> # "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
> # "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
> # "destination-uri": "tcp:11.0.0.0:7789", "multifd-channels": 5} ] } }
>
> even this we should try to wrap within 80 character column right? or is that okay to go beyond 80.
I'd format it like
# -> { "execute": "migrate",
# "arguments": {
# "uri": "tcp:0:4446",
# "multi-fd-uri-list": [
# { "source-uri": "tcp::6900",
# "destination-uri": "tcp:0:4480",
# "multifd-channels": 4 },
# { "source-uri": "tcp:10.0.0.0: ",
# "destination-uri": "tcp:11.0.0.0:7789",
# "multifd-channels": 5} ] } }
>>>> Long line.
>>> Okay, acknowledged. Even for example, should it be under 80
>>> characters per line, or that is fine?
>> docs/devel/style.rst:
>>
>> Line width
>> ==========
>>
>> Lines should be 80 characters; try not to make them longer.
>>
>> Sometimes it is hard to do, especially when dealing with QEMU subsystems
>> that use long function or symbol names. If wrapping the line at 80 columns
>> is obviously less readable and more awkward, prefer not to wrap it; better
>> to have an 85 character line than one which is awkwardly wrapped.
>>
>> Even in that case, try not to make lines much longer than 80 characters.
>> (The checkpatch script will warn at 100 characters, but this is intended
>> as a guard against obviously-overlength lines, not a target.)
>>
>> Personally, I very much prefer to wrap between 70 and 75 except where it
>> impairs legibility.
> Okay thanks again Markus for your valuable suggestion. I will try to wrap within 75 in almost all the cases.
>>>>> + '*inc': 'bool', '*detach': 'bool', '*resume': 'bool' } }
>>>>> ##
>>>>> # @migrate-incoming:
>>> Regards,
>>>
>>> Het Gala
>
> Regards,
>
> Het Gala
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair
2022-07-19 7:06 ` Markus Armbruster
@ 2022-07-19 7:51 ` Het Gala
2022-07-19 9:48 ` Markus Armbruster
0 siblings, 1 reply; 34+ messages in thread
From: Het Gala @ 2022-07-19 7:51 UTC (permalink / raw)
To: Markus Armbruster
Cc: qemu-devel, quintela, dgilbert, pbonzini, berrange, eblake,
Manish Mishra
On 19/07/22 12:36 pm, Markus Armbruster wrote:
> Het Gala <het.gala@nutanix.com> writes:
>
>> On 18/07/22 8:03 pm, Markus Armbruster wrote:
>>> Het Gala <het.gala@nutanix.com> writes:
>>>
>>>> On 18/07/22 2:05 pm, Markus Armbruster wrote:
>>>>> Het Gala <het.gala@nutanix.com> writes:
>>>>>
>>>>>> i) Modified the format of the qemu monitor command : 'migrate' by adding a list,
>>>>>> each element in the list consists of multi-FD connection parameters: source
>>>>>> and destination uris and of the number of multi-fd channels between each pair.
>>>>>>
>>>>>> ii) Information of all multi-FD connection parameters’ list, length of the list
>>>>>> and total number of multi-fd channels for all the connections together is
>>>>>> stored in ‘OutgoingArgs’ struct.
>>>>>>
>>>>>> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
>>>>>> Signed-off-by: Het Gala <het.gala@nutanix.com>
>>>>>> ---
> [...]
>
>>>>>> diff --git a/migration/socket.c b/migration/socket.c
>>>>>> index 4fd5e85f50..7ca6af8cca 100644
>>>>>> --- a/migration/socket.c
>>>>>> +++ b/migration/socket.c
>>>>>> @@ -32,6 +32,17 @@ struct SocketOutgoingArgs {
>>>>>> SocketAddress *saddr;
>>>>>> } outgoing_args;
>>>>>>
>>>>>> +struct SocketArgs {
>>>>>> + struct SrcDestAddr data;
>>>>>> + uint8_t multifd_channels;
>>>>>> +};
>>>>>> +
>>>>>> +struct OutgoingMigrateParams {
>>>>>> + struct SocketArgs *socket_args;
>>>>>> + size_t length;
>>>>> Length of what?
>>>> length of the socket_args[] array. Thanks for pointing it out. I will
>>>> be more specific for this variable in the v2 patchset series.
>>>>
>>>>>> + uint64_t total_multifd_channel;
>>>>> @total_multifd_channels appears to be the sum of the
>>>>> socket_args[*].multifd_channels. Correct?
>>>> Yes Markus, you are correct.
>>> Sure you need to keep the sum separately?
>> So earlier, the idea behind this was that, we had this intention to depreciate the migrate_multifd_channels() API from the live migration
>> process. We made total_multifd_channels() function, where it used to calculate total number of multifd channels every time, for whichever
>> function called was computation internsive so we replaced it by returning sum of socket_args[*].multifd_channels i.e.
>> total_multifd_channel in the later patches.
>>
>> But now in the v2 patchset onwards, Thanks to inputs from Dr. David and Daniel, we are not depricating migrate_multifd_channels() API but
>> the value from the API will be cross-referenced with sum of socket_args[*].multifd_channels i.e. total_multifd_channel, and error if
>> they are not equal.
> I'm afraid I don't understand. I'm not sure I have to. Let me loop
> back to my question.
>
> If @total_multifd_channel is always the sum of the
> socket_args[*].multifd_channels, then you can always compute it on the
> fly.
>
> I.e. you can replace @total_multifd_channel by a function that returns
> the sum.
>
> Precomputing it instead is more complex, because then you need to
> document that the two are the same. Also, bug oppertunity: letting them
> deviate somehow. I figure that's worthwhile only if computing on the
> fly is too expensive.
> Okay, I understand your concern. I am okay with your approach too,
but these things are not expected to change out of qmp command context.
So is keeping @total_multifd_channel variable should be fine? or making
a function is better?
>>>>>> +} outgoing_migrate_params;
>>>>>> +
>>>>>> void socket_send_channel_create(QIOTaskFunc f, void *data)
>>>>>> {
>>>>>> QIOChannelSocket *sioc = qio_channel_socket_new();
>>>>>> @@ -47,6 +58,14 @@ int socket_send_channel_destroy(QIOChannel *send)
>>>>>> qapi_free_SocketAddress(outgoing_args.saddr);
>>>>>> outgoing_args.saddr = NULL;
>>>>>> }
>>>>>> +
>>>>>> + if (outgoing_migrate_params.socket_args != NULL) {
>>>>>> + g_free(outgoing_migrate_params.socket_args);
>>>>>> + outgoing_migrate_params.socket_args = NULL;
>>>>>> + }
>>>>>> + if (outgoing_migrate_params.length) {
>>>>>> + outgoing_migrate_params.length = 0;
>>>>>> + }
>>>>>> return 0;
>>>>>> }
>>>>>>
>>>>>> @@ -117,13 +136,41 @@ socket_start_outgoing_migration_internal(MigrationState *s,
>>>>>> }
>>>>>>
>>>>>> void socket_start_outgoing_migration(MigrationState *s,
>>>>>> - const char *str,
>>>>>> + const char *dst_str,
>>>>>> Error **errp)
>>>>>> {
>>>>>> Error *err = NULL;
>>>>>> - SocketAddress *saddr = socket_parse(str, &err);
>>>>>> + SocketAddress *dst_saddr = socket_parse(dst_str, &err);
>>>>>> + if (!err) {
>>>>>> + socket_start_outgoing_migration_internal(s, dst_saddr, &err);
>>>>>> + }
>>>>>> + error_propagate(errp, err);
>>>>>> +}
>>>>>> +
>>>>>> +void init_multifd_array(int length)
>>>>>> +{
>>>>>> + outgoing_migrate_params.socket_args = g_new0(struct SocketArgs, length);
>>>>>> + outgoing_migrate_params.length = length;
>>>>>> + outgoing_migrate_params.total_multifd_channel = 0;
>>>>>> +}
>>>>>> +
>>>>>> +void store_multifd_migration_params(const char *dst_uri,
>>>>>> + const char *src_uri,
>>>>>> + uint8_t multifd_channels,
>>>>>> + int idx, Error **errp)
>>>>>> +{
>>>>>> + Error *err = NULL;
>>>>>> + SocketAddress *src_addr = NULL;
>>>>>> + SocketAddress *dst_addr = socket_parse(dst_uri, &err);
>>>>>> + if (src_uri) {
>>>>>> + src_addr = socket_parse(src_uri, &err);
>>>>>> + }
>>>>> Incorrect use of &err. error.h's big comment:
>>>>>
>>>>> * Receive and accumulate multiple errors (first one wins):
>>>>> * Error *err = NULL, *local_err = NULL;
>>>>> * foo(arg, &err);
>>>>> * bar(arg, &local_err);
>>>>> * error_propagate(&err, local_err);
>>>>> * if (err) {
>>>>> * handle the error...
>>>>> * }
>>>>> *
>>>>> * Do *not* "optimize" this to
>>>>> * Error *err = NULL;
>>>>> * foo(arg, &err);
>>>>> * bar(arg, &err); // WRONG!
>>>>> * if (err) {
>>>>> * handle the error...
>>>>> * }
>>>>> * because this may pass a non-null err to bar().
>>>>>
>>>> Thankyou Markus for sharing this knowledge. I was unaware of the
>>>> dont's with &err.
>>> The big comment should help you along. If it doesn't, just ask.
>>> I read the comment, and it is pretty well explained out there.
>>>
>>>>>> if (!err) {
>>>>>> - socket_start_outgoing_migration_internal(s, saddr, &err);
>>>>>> + outgoing_migrate_params.socket_args[idx].data.dst_addr = dst_addr;
>>>>>> + outgoing_migrate_params.socket_args[idx].data.src_addr = src_addr;
>>>>>> + outgoing_migrate_params.socket_args[idx].multifd_channels
>>>>>> + = multifd_channels;
>>>>>> + outgoing_migrate_params.total_multifd_channel += multifd_channels;
>>>>>> }
>>>>>> error_propagate(errp, err);
>>>>> Consider
>>>>>
>>>>> struct SocketArgs *sa = &outgoing_migrate_params.socket_args[idx];
>>>>> SocketAddress *src_addr, *dst_addr;
>>>>>
>>>>> src_addr = socketaddress(src_uri, errp);
>>>>> if (!src_addr) {
>>>>> return;
>>>>> }
>>>>>
>>>>> dst_addr = socketaddress(dst_uri, errp);
>>>>> if (!dst_addr) {
>>>>> return;
>>>>> }
>>>>>
>>>>> sa->data.dst_addr = dst_addr;
>>>>> sa->data.src_addr = src_addr;
>>>>> sa->multifd_channels = multifd_channels;
>>>>> outgoing_migrate_params.total_multifd_channel += multifd_channels;
>>>> Thanks Markus for this amazing suggestion. Your approach looks
>>>> simpler to understand and also resolves the error it had with &err. I
>>>> will surely implement this in the upcoming v2 patchset.
>>> You're welcome :)
>> I just wanted to have a double check on the solution you gave above Markus. The suggestion you have given there has been deliberately
>> written in that way right, because
>>
>> src_addr = socketaddress(src_uri, errp);
>> dst_addr = socketaddress(dst_uri, errp);
>> if (!src_addr) {
>> return;
>> }
>> if (!dst_addr) {
>> return;
>> }
>>
>> would still be an error right according to the &err guidelines from error.h file.
> Correct.
> Thankyou Markus.
>>>>>> }
> [...]
>
>>>>>> diff --git a/qapi/migration.json b/qapi/migration.json
>>>>>> index 6130cd9fae..fb259d626b 100644
>>>>>> --- a/qapi/migration.json
>>>>>> +++ b/qapi/migration.json
>>>>>> @@ -1454,12 +1454,38 @@
>>>>>> ##
>>>>>> { 'command': 'migrate-continue', 'data': {'state': 'MigrationStatus'} }
>>>>>>
>>>>>> +##
>>>>>> +# @MigrateUriParameter:
>>>>>> +#
>>>>>> +# Information regarding which source interface is connected to which
>>>>>> +# destination interface and number of multifd channels over each interface.
>>>>>> +#
>>>>>> +# @source-uri: the Uniform Resource Identifier of the source VM.
>>>>>> +# Default port number is 0.
>>>>>> +#
>>>>>> +# @destination-uri: the Uniform Resource Identifier of the destination VM
>>>>>> +#
>>>>>> +# @multifd-channels: number of parallel multifd channels used to migrate data
>>>>>> +# for specific source-uri and destination-uri. Default value
>>>>>> +# in this case is 2 (Since 4.0)
>>>>> You mean "(Since 7.1)", I guess.
>>>> Yes yes. Also David pointed this thing out. I will update the version
>>>> in the v2 patchset.
>>>>
>>>>>> +#
>>>>>> +##
>>>>>> +{ 'struct' : 'MigrateUriParameter',
>>>>>> + 'data' : { 'source-uri' : 'str',
>>>>>> + 'destination-uri' : 'str',
>>>>>> + '*multifd-channels' : 'uint8'} }
>>>>>> +
>>>>>> ##
>>>>>> # @migrate:
>>>>>> #
>>>>>> # Migrates the current running guest to another Virtual Machine.
>>>>>> #
>>>>>> # @uri: the Uniform Resource Identifier of the destination VM
>>>>>> +# for migration thread
>>>>>> +#
>>>>>> +# @multi-fd-uri-list: list of pair of source and destination VM Uniform
>>>>>> +# Resource Identifiers with number of multifd-channels
>>>>>> +# for each pair
>>>>>> #
>>>>>> # @blk: do block migration (full disk copy)
>>>>>> #
>>>>>> @@ -1479,20 +1505,27 @@
>>>>>> # 1. The 'query-migrate' command should be used to check migration's progress
>>>>>> # and final result (this information is provided by the 'status' member)
>>>>>> #
>>>>>> -# 2. All boolean arguments default to false
>>>>>> +# 2. The uri argument should have the Uniform Resource Identifier of default
>>>>>> +# destination VM. This connection will be bound to default network
>>>>>> +#
>>>>>> +# 3. All boolean arguments default to false
>>>>>> #
>>>>>> -# 3. The user Monitor's "detach" argument is invalid in QMP and should not
>>>>>> +# 4. The user Monitor's "detach" argument is invalid in QMP and should not
>>>>>> # be used
>>>>>> #
>>>>>> # Example:
>>>>>> #
>>>>>> -# -> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
>>>>>> +# -> { "execute": "migrate",
>>>>>> +# "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
>>>>>> +# "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
>>>>>> +# "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
>>>>>> +# "destination-uri": "tcp:11.0.0.0:7789", "multifd-channels": 5} ] } }
>>>>>> # <- { "return": {} }
>>>>>> #
>>>>>> ##
>>>>>> { 'command': 'migrate',
>>>>>> - 'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool',
>>>>>> - '*detach': 'bool', '*resume': 'bool' } }
>>>>>> + 'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateUriParameter'], '*blk': 'bool',
>> ??
>>
>> Sorry Markus, I think the statement I wrote did not make sense, I apologise for that. I meant to say example in the sense:
>>
>> # Example:
>> #
>> # -> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
>> # -> { "execute": "migrate",
>> # "arguments": { "uri": "tcp:0:4446", "multi-fd-uri-list": [ {
>> # "source-uri": "tcp::6900", "destination-uri": "tcp:0:4480",
>> # "multifd-channels": 4}, { "source-uri": "tcp:10.0.0.0: ",
>> # "destination-uri": "tcp:11.0.0.0:7789", "multifd-channels": 5} ] } }
>>
>> even this we should try to wrap within 80 character column right? or is that okay to go beyond 80.
> I'd format it like
>
> # -> { "execute": "migrate",
> # "arguments": {
> # "uri": "tcp:0:4446",
> # "multi-fd-uri-list": [
> # { "source-uri": "tcp::6900",
> # "destination-uri": "tcp:0:4480",
> # "multifd-channels": 4 },
> # { "source-uri": "tcp:10.0.0.0: ",
> # "destination-uri": "tcp:11.0.0.0:7789",
> # "multifd-channels": 5} ] } }
> Yeah sure Markus.
>>>>> Long line.
>>>> Okay, acknowledged. Even for example, should it be under 80
>>>> characters per line, or that is fine?
>>> docs/devel/style.rst:
>>>
>>> Line width
>>> ==========
>>>
>>> Lines should be 80 characters; try not to make them longer.
>>>
>>> Sometimes it is hard to do, especially when dealing with QEMU subsystems
>>> that use long function or symbol names. If wrapping the line at 80 columns
>>> is obviously less readable and more awkward, prefer not to wrap it; better
>>> to have an 85 character line than one which is awkwardly wrapped.
>>>
>>> Even in that case, try not to make lines much longer than 80 characters.
>>> (The checkpatch script will warn at 100 characters, but this is intended
>>> as a guard against obviously-overlength lines, not a target.)
>>>
>>> Personally, I very much prefer to wrap between 70 and 75 except where it
>>> impairs legibility.
>> Okay thanks again Markus for your valuable suggestion. I will try to wrap within 75 in almost all the cases.
>>>>>> + '*inc': 'bool', '*detach': 'bool', '*resume': 'bool' } }
>>>>>> ##
>>>>>> # @migrate-incoming:
>>>> Regards,
>>>>
>>>> Het Gala
>> Regards,
>>
>> Het Gala
Regards,
Het Gala
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair
2022-07-19 7:51 ` Het Gala
@ 2022-07-19 9:48 ` Markus Armbruster
2022-07-19 10:40 ` Het Gala
0 siblings, 1 reply; 34+ messages in thread
From: Markus Armbruster @ 2022-07-19 9:48 UTC (permalink / raw)
To: Het Gala
Cc: qemu-devel, quintela, dgilbert, pbonzini, berrange, eblake,
Manish Mishra
Het Gala <het.gala@nutanix.com> writes:
> On 19/07/22 12:36 pm, Markus Armbruster wrote:
>> Het Gala <het.gala@nutanix.com> writes:
>>
>>> On 18/07/22 8:03 pm, Markus Armbruster wrote:
>>>> Het Gala <het.gala@nutanix.com> writes:
>>>>
>>>>> On 18/07/22 2:05 pm, Markus Armbruster wrote:
>>>>>> Het Gala <het.gala@nutanix.com> writes:
>>>>>>
>>>>>>> i) Modified the format of the qemu monitor command : 'migrate' by adding a list,
>>>>>>> each element in the list consists of multi-FD connection parameters: source
>>>>>>> and destination uris and of the number of multi-fd channels between each pair.
>>>>>>>
>>>>>>> ii) Information of all multi-FD connection parameters’ list, length of the list
>>>>>>> and total number of multi-fd channels for all the connections together is
>>>>>>> stored in ‘OutgoingArgs’ struct.
>>>>>>>
>>>>>>> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
>>>>>>> Signed-off-by: Het Gala <het.gala@nutanix.com>
>>>>>>> ---
>> [...]
>>
>>>>>>> diff --git a/migration/socket.c b/migration/socket.c
>>>>>>> index 4fd5e85f50..7ca6af8cca 100644
>>>>>>> --- a/migration/socket.c
>>>>>>> +++ b/migration/socket.c
>>>>>>> @@ -32,6 +32,17 @@ struct SocketOutgoingArgs {
>>>>>>> SocketAddress *saddr;
>>>>>>> } outgoing_args;
>>>>>>>
>>>>>>> +struct SocketArgs {
>>>>>>> + struct SrcDestAddr data;
>>>>>>> + uint8_t multifd_channels;
>>>>>>> +};
>>>>>>> +
>>>>>>> +struct OutgoingMigrateParams {
>>>>>>> + struct SocketArgs *socket_args;
>>>>>>> + size_t length;
>>>>>> Length of what?
>>>>> length of the socket_args[] array. Thanks for pointing it out. I will
>>>>> be more specific for this variable in the v2 patchset series.
>>>>>
>>>>>>> + uint64_t total_multifd_channel;
>>>>>> @total_multifd_channels appears to be the sum of the
>>>>>> socket_args[*].multifd_channels. Correct?
>>>>> Yes Markus, you are correct.
>>>> Sure you need to keep the sum separately?
>>> So earlier, the idea behind this was that, we had this intention to depreciate the migrate_multifd_channels() API from the live migration
>>> process. We made total_multifd_channels() function, where it used to calculate total number of multifd channels every time, for whichever
>>> function called was computation internsive so we replaced it by returning sum of socket_args[*].multifd_channels i.e.
>>> total_multifd_channel in the later patches.
>>>
>>> But now in the v2 patchset onwards, Thanks to inputs from Dr. David and Daniel, we are not depricating migrate_multifd_channels() API but
>>> the value from the API will be cross-referenced with sum of socket_args[*].multifd_channels i.e. total_multifd_channel, and error if
>>> they are not equal.
>> I'm afraid I don't understand. I'm not sure I have to. Let me loop
>> back to my question.
>>
>> If @total_multifd_channel is always the sum of the
>> socket_args[*].multifd_channels, then you can always compute it on the
>> fly.
>>
>> I.e. you can replace @total_multifd_channel by a function that returns
>> the sum.
>>
>> Precomputing it instead is more complex, because then you need to
>> document that the two are the same. Also, bug oppertunity: letting them
>> deviate somehow. I figure that's worthwhile only if computing on the
>> fly is too expensive.
>> Okay, I understand your concern. I am okay with your approach too, but these things are not expected to change out of qmp command context.
>
> So is keeping @total_multifd_channel variable should be fine? or making a function is better?
I recommend making it a function unless we need a variable for
performance.
[...]
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair
2022-07-19 9:48 ` Markus Armbruster
@ 2022-07-19 10:40 ` Het Gala
0 siblings, 0 replies; 34+ messages in thread
From: Het Gala @ 2022-07-19 10:40 UTC (permalink / raw)
To: Markus Armbruster
Cc: qemu-devel, quintela, dgilbert, pbonzini, berrange, eblake,
Manish Mishra
On 19/07/22 3:18 pm, Markus Armbruster wrote:
> Het Gala <het.gala@nutanix.com> writes:
>
>> On 19/07/22 12:36 pm, Markus Armbruster wrote:
>>> Het Gala <het.gala@nutanix.com> writes:
>>>
>>>> On 18/07/22 8:03 pm, Markus Armbruster wrote:
>>>>> Het Gala <het.gala@nutanix.com> writes:
>>>>>
>>>>>> On 18/07/22 2:05 pm, Markus Armbruster wrote:
>>>>>>> Het Gala <het.gala@nutanix.com> writes:
>>>>>>>
>>>>>>>> i) Modified the format of the qemu monitor command : 'migrate' by adding a list,
>>>>>>>> each element in the list consists of multi-FD connection parameters: source
>>>>>>>> and destination uris and of the number of multi-fd channels between each pair.
>>>>>>>>
>>>>>>>> ii) Information of all multi-FD connection parameters’ list, length of the list
>>>>>>>> and total number of multi-fd channels for all the connections together is
>>>>>>>> stored in ‘OutgoingArgs’ struct.
>>>>>>>>
>>>>>>>> Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
>>>>>>>> Signed-off-by: Het Gala <het.gala@nutanix.com>
>>>>>>>> ---
>>> [...]
>>>
>>>>>>>> diff --git a/migration/socket.c b/migration/socket.c
>>>>>>>> index 4fd5e85f50..7ca6af8cca 100644
>>>>>>>> --- a/migration/socket.c
>>>>>>>> +++ b/migration/socket.c
>>>>>>>> @@ -32,6 +32,17 @@ struct SocketOutgoingArgs {
>>>>>>>> SocketAddress *saddr;
>>>>>>>> } outgoing_args;
>>>>>>>>
>>>>>>>> +struct SocketArgs {
>>>>>>>> + struct SrcDestAddr data;
>>>>>>>> + uint8_t multifd_channels;
>>>>>>>> +};
>>>>>>>> +
>>>>>>>> +struct OutgoingMigrateParams {
>>>>>>>> + struct SocketArgs *socket_args;
>>>>>>>> + size_t length;
>>>>>>> Length of what?
>>>>>> length of the socket_args[] array. Thanks for pointing it out. I will
>>>>>> be more specific for this variable in the v2 patchset series.
>>>>>>
>>>>>>>> + uint64_t total_multifd_channel;
>>>>>>> @total_multifd_channels appears to be the sum of the
>>>>>>> socket_args[*].multifd_channels. Correct?
>>>>>> Yes Markus, you are correct.
>>>>> Sure you need to keep the sum separately?
>>>> So earlier, the idea behind this was that, we had this intention to depreciate the migrate_multifd_channels() API from the live migration
>>>> process. We made total_multifd_channels() function, where it used to calculate total number of multifd channels every time, for whichever
>>>> function called was computation internsive so we replaced it by returning sum of socket_args[*].multifd_channels i.e.
>>>> total_multifd_channel in the later patches.
>>>>
>>>> But now in the v2 patchset onwards, Thanks to inputs from Dr. David and Daniel, we are not depricating migrate_multifd_channels() API but
>>>> the value from the API will be cross-referenced with sum of socket_args[*].multifd_channels i.e. total_multifd_channel, and error if
>>>> they are not equal.
>>> I'm afraid I don't understand. I'm not sure I have to. Let me loop
>>> back to my question.
>>>
>>> If @total_multifd_channel is always the sum of the
>>> socket_args[*].multifd_channels, then you can always compute it on the
>>> fly.
>>>
>>> I.e. you can replace @total_multifd_channel by a function that returns
>>> the sum.
>>>
>>> Precomputing it instead is more complex, because then you need to
>>> document that the two are the same. Also, bug oppertunity: letting them
>>> deviate somehow. I figure that's worthwhile only if computing on the
>>> fly is too expensive.
>>> Okay, I understand your concern. I am okay with your approach too, but these things are not expected to change out of qmp command context.
>> So is keeping @total_multifd_channel variable should be fine? or making a function is better?
> I recommend making it a function unless we need a variable for
> performance.
> Okay Markus. I will make it a function rather than a variable
>
> [...]
>
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [PATCH 3/4] Establishing connection between any non-default source and destination pair
[not found] ` <de0646c1-75d7-5f9d-32db-07c498c45715@nutanix.com>
@ 2022-07-20 6:52 ` Daniel P. Berrangé
0 siblings, 0 replies; 34+ messages in thread
From: Daniel P. Berrangé @ 2022-07-20 6:52 UTC (permalink / raw)
To: Het Gala, qemu-devel
Re-adding the mailing list, please don't drop the list in
replies to discussions.
On Wed, Jul 20, 2022 at 02:08:23AM +0530, Het Gala wrote:
>
> On 13/07/22 3:10 pm, Het Gala wrote:
> >
> > On 16/06/22 11:09 pm, Daniel P. Berrangé wrote:
> > > On Thu, Jun 09, 2022 at 07:33:04AM +0000, Het Gala wrote:
> > > > i) Binding of the socket to source ip address and port on the
> > > > non-default
> > > > interface has been implemented for multi-FD connection,
> > > > which was not
> > > > necessary earlier because the binding was on the default
> > > > interface itself.
> > > >
> > > > ii) Created an end to end connection between all multi-FD source and
> > > > destination pairs.
> > > >
> > > > Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
> > > > Signed-off-by: Het Gala <het.gala@nutanix.com>
> > > > ---
> > > > chardev/char-socket.c | 4 +-
> > > > include/io/channel-socket.h | 26 ++++++-----
> > > > include/qemu/sockets.h | 6 ++-
> > > > io/channel-socket.c | 50 ++++++++++++++------
> > > > migration/socket.c | 15 +++---
> > > > nbd/client-connection.c | 2 +-
> > > > qemu-nbd.c | 4 +-
> > > > scsi/pr-manager-helper.c | 1 +
> > > > tests/unit/test-char.c | 8 ++--
> > > > tests/unit/test-io-channel-socket.c | 4 +-
> > > > tests/unit/test-util-sockets.c | 16 +++----
> > > > ui/input-barrier.c | 2 +-
> > > > ui/vnc.c | 3 +-
> > > > util/qemu-sockets.c | 71
> > > > ++++++++++++++++++++---------
> > > > 14 files changed, 135 insertions(+), 77 deletions(-)
> > > >
> > > > diff --git a/chardev/char-socket.c b/chardev/char-socket.c
> > > > index dc4e218eeb..f3725238c5 100644
> > > > --- a/chardev/char-socket.c
> > > > +++ b/chardev/char-socket.c
> > > > @@ -932,7 +932,7 @@ static int
> > > > tcp_chr_connect_client_sync(Chardev *chr, Error **errp)
> > > > QIOChannelSocket *sioc = qio_channel_socket_new();
> > > > tcp_chr_change_state(s, TCP_CHARDEV_STATE_CONNECTING);
> > > > tcp_chr_set_client_ioc_name(chr, sioc);
> > > > - if (qio_channel_socket_connect_sync(sioc, s->addr, errp) < 0) {
> > > > + if (qio_channel_socket_connect_sync(sioc, s->addr, NULL,
> > > > errp) < 0) {
> > > > tcp_chr_change_state(s, TCP_CHARDEV_STATE_DISCONNECTED);
> > > > object_unref(OBJECT(sioc));
> > > > return -1;
> > > > @@ -1120,7 +1120,7 @@ static void
> > > > tcp_chr_connect_client_task(QIOTask *task,
> > > > SocketAddress *addr = opaque;
> > > > Error *err = NULL;
> > > > - qio_channel_socket_connect_sync(ioc, addr, &err);
> > > > + qio_channel_socket_connect_sync(ioc, addr, NULL, &err);
> > > > qio_task_set_error(task, err);
> > > > }
> > > > diff --git a/include/io/channel-socket.h b/include/io/channel-socket.h
> > > > index 513c428fe4..59d5b1b349 100644
> > > > --- a/include/io/channel-socket.h
> > > > +++ b/include/io/channel-socket.h
> > > > @@ -83,41 +83,45 @@ qio_channel_socket_new_fd(int fd,
> > > > /**
> > > > * qio_channel_socket_connect_sync:
> > > > * @ioc: the socket channel object
> > > > - * @addr: the address to connect to
> > > > + * @dst_addr: the destination address to connect to
> > > > + * @src_addr: the source address to be connected
> > > > * @errp: pointer to a NULL-initialized error object
> > > > *
> > > > - * Attempt to connect to the address @addr. This method
> > > > - * will run in the foreground so the caller will not regain
> > > > - * execution control until the connection is established or
> > > > + * Attempt to connect to the address @dst_addr with @src_addr.
> > > > + * This method will run in the foreground so the caller will not
> > > > + * regain execution control until the connection is established or
> > > > * an error occurs.
> > > > */
> > > > int qio_channel_socket_connect_sync(QIOChannelSocket *ioc,
> > > > - SocketAddress *addr,
> > > > + SocketAddress *dst_addr,
> > > > + SocketAddress *src_addr,
> > > > Error **errp);
> > > > /**
> > > > * qio_channel_socket_connect_async:
> > > > * @ioc: the socket channel object
> > > > - * @addr: the address to connect to
> > > > + * @dst_addr: the destination address to connect to
> > > > * @callback: the function to invoke on completion
> > > > * @opaque: user data to pass to @callback
> > > > * @destroy: the function to free @opaque
> > > > * @context: the context to run the async task. If %NULL, the default
> > > > * context will be used.
> > > > + * @src_addr: the source address to be connected
> > > > *
> > > > - * Attempt to connect to the address @addr. This method
> > > > - * will run in the background so the caller will regain
> > > > + * Attempt to connect to the address @dst_addr with the @src_addr.
> > > > + * This method will run in the background so the caller will regain
> > > > * execution control immediately. The function @callback
> > > > - * will be invoked on completion or failure. The @addr
> > > > + * will be invoked on completion or failure. The @dst_addr
> > > > * parameter will be copied, so may be freed as soon
> > > > * as this function returns without waiting for completion.
> > > > */
> > > > void qio_channel_socket_connect_async(QIOChannelSocket *ioc,
> > > > - SocketAddress *addr,
> > > > + SocketAddress *dst_addr,
> > > > QIOTaskFunc callback,
> > > > gpointer opaque,
> > > > GDestroyNotify destroy,
> > > > - GMainContext *context);
> > > > + GMainContext *context,
> > > > + SocketAddress *src_addr);
> > > Lets avoid modifying these two methods, and thus avoid
> > > updating countless callers.
> > >
> > > In common with typical pattern in QIO code, lets add
> > > a second variant
> > >
> > > qio_channel_socket_connect_full
> > > qio_channel_socket_connect_full_async
> > >
> > > which has the extra parameters, then make the existing
> > > simpler methods call the new ones.
> > >
> > > ie qio_channel_socket_connect should call
> > > qio_channel_socket_connect_full, pasing NULL for the
> > > src_addr.
> > >
> > > Thanks for the suggestion Daniel. Will modify the same structure as
> >
> > suggested above in the v2 patchset.
>
> > Hi Daniel. I agree with your suggestion here, but I have couple of doubts
> in implementing this type.
>
> 1. You meant to say qio_channel_socket_connect_async calls ->
> qio_channel_socket_connect_all_async and the later function would have a
> extra parameter for src_addr as NULL right. But if you see this approach
> works well for connecting non-multifd channels where source uri is passed as
> NULL, but for multifd channels, as you see the function
> socket_send_channel_create also calls qio_channel_socket_connect_async, but
> this time instead of NULL, it should actually pass a src_addr parameter. So
> in my opion, whatever function multifd function is calling it should have
> extra parameter to pass src_addr.
>
> 2. Same goes for qio_channel_socket_connect_sync func, for multifd path, it
> should be passed with src_addr instead of NULL.
>
> 3. I agree, modifying these methods would lead to updating endless callers
> from test cases. But I don't see a better way that this at the moment. And
> out of the two methods, one method is called only for single unit test case
> in qemu.
>
> We would love to have suggestions from your side Daniel.
Do not modify this existing method signature at all:
int qio_channel_socket_connect_sync(QIOChannelSocket *ioc,
SocketAddress *addr,
Error **errp);
Only add a new method:
int qio_channel_socket_connect_full_sync(QIOChannelSocket *ioc,
SocketAddress *dst_addr,
SocketAddress *src_addr,
Error **errp);
Internally the former method calls the latter, assing NULL for
src_addr.
Externally, only the migration code needs to use the new method,
all the rest of QEMU code must remain unchanged calling the simpler
method.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 34+ messages in thread
end of thread, other threads:[~2022-07-20 6:54 UTC | newest]
Thread overview: 34+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-09 7:33 [PATCH 0/4] Multiple interface support on top of Multi-FD Het Gala
2022-06-09 7:33 ` [PATCH 1/4] Modifying ‘migrate’ qmp command to add multi-FD socket on particular source and destination pair Het Gala
2022-06-16 17:26 ` Dr. David Alan Gilbert
2022-07-13 8:08 ` Het Gala
2022-07-15 8:07 ` Het Gala
2022-07-13 12:54 ` Claudio Fontana
2022-07-18 8:35 ` Markus Armbruster
2022-07-18 13:33 ` Het Gala
2022-07-18 14:33 ` Markus Armbruster
2022-07-18 15:17 ` Het Gala
2022-07-19 7:06 ` Markus Armbruster
2022-07-19 7:51 ` Het Gala
2022-07-19 9:48 ` Markus Armbruster
2022-07-19 10:40 ` Het Gala
2022-06-09 7:33 ` [PATCH 2/4] Adding multi-interface support for multi-FD on destination side Het Gala
2022-06-16 18:40 ` Dr. David Alan Gilbert
2022-07-13 14:36 ` Het Gala
2022-06-09 7:33 ` [PATCH 3/4] Establishing connection between any non-default source and destination pair Het Gala
2022-06-16 17:39 ` Daniel P. Berrangé
2022-06-21 16:09 ` manish.mishra
[not found] ` <54ca00c7-a108-11e3-1c8d-8771798aed6a@nutanix.com>
[not found] ` <de0646c1-75d7-5f9d-32db-07c498c45715@nutanix.com>
2022-07-20 6:52 ` Daniel P. Berrangé
2022-06-09 7:33 ` [PATCH 4/4] Adding support for multi-FD connections dynamically Het Gala
2022-06-16 18:47 ` Dr. David Alan Gilbert
2022-06-21 16:12 ` manish.mishra
2022-06-09 15:47 ` [PATCH 0/4] Multiple interface support on top of Multi-FD Daniel P. Berrangé
2022-06-10 12:28 ` manish.mishra
2022-06-15 16:43 ` Daniel P. Berrangé
2022-06-15 19:14 ` Dr. David Alan Gilbert
2022-06-16 8:16 ` Daniel P. Berrangé
2022-06-16 10:14 ` manish.mishra
2022-06-16 17:32 ` Daniel P. Berrangé
2022-06-16 8:27 ` Daniel P. Berrangé
2022-06-16 15:50 ` Dr. David Alan Gilbert
2022-06-21 16:16 ` manish.mishra
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.