* [PATCH RFC 01/14] migration: add the 'migrate_use_rdma_pin_all' function
2020-02-13 9:37 [PATCH RFC 00/14] *** multifd for RDMA v2 *** Zhimin Feng
@ 2020-02-13 9:37 ` Zhimin Feng
2020-02-13 10:02 ` Juan Quintela
2020-02-13 9:37 ` [PATCH RFC 02/14] migration: judge whether or not the RDMA is used for migration Zhimin Feng
` (13 subsequent siblings)
14 siblings, 1 reply; 22+ messages in thread
From: Zhimin Feng @ 2020-02-13 9:37 UTC (permalink / raw)
To: quintela, dgilbert, armbru, eblake
Cc: jemmy858585, Zhimin Feng, qemu-devel, zhang.zhanghailiang
Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
---
migration/migration.c | 9 +++++++++
migration/migration.h | 1 +
2 files changed, 10 insertions(+)
diff --git a/migration/migration.c b/migration/migration.c
index 3a21a4686c..10a13e0c79 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -2208,6 +2208,15 @@ bool migrate_use_events(void)
return s->enabled_capabilities[MIGRATION_CAPABILITY_EVENTS];
}
+bool migrate_use_rdma_pin_all(void)
+{
+ MigrationState *s;
+
+ s = migrate_get_current();
+
+ return s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL];
+}
+
bool migrate_use_multifd(void)
{
MigrationState *s;
diff --git a/migration/migration.h b/migration/migration.h
index 8473ddfc88..50fc2693c7 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -297,6 +297,7 @@ bool migrate_ignore_shared(void);
bool migrate_validate_uuid(void);
bool migrate_auto_converge(void);
+bool migrate_use_rdma_pin_all(void);
bool migrate_use_multifd(void);
bool migrate_pause_before_switchover(void);
int migrate_multifd_channels(void);
--
2.19.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH RFC 02/14] migration: judge whether or not the RDMA is used for migration
2020-02-13 9:37 [PATCH RFC 00/14] *** multifd for RDMA v2 *** Zhimin Feng
2020-02-13 9:37 ` [PATCH RFC 01/14] migration: add the 'migrate_use_rdma_pin_all' function Zhimin Feng
@ 2020-02-13 9:37 ` Zhimin Feng
2020-02-13 10:04 ` Juan Quintela
2020-02-13 9:37 ` [PATCH RFC 03/14] migration/rdma: Create multiFd migration threads Zhimin Feng
` (12 subsequent siblings)
14 siblings, 1 reply; 22+ messages in thread
From: Zhimin Feng @ 2020-02-13 9:37 UTC (permalink / raw)
To: quintela, dgilbert, armbru, eblake
Cc: jemmy858585, Zhimin Feng, qemu-devel, zhang.zhanghailiang
Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
---
migration/migration.c | 10 ++++++++++
migration/migration.h | 1 +
2 files changed, 11 insertions(+)
diff --git a/migration/migration.c b/migration/migration.c
index 10a13e0c79..819089a7ea 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -107,6 +107,7 @@ static NotifierList migration_state_notifiers =
NOTIFIER_LIST_INITIALIZER(migration_state_notifiers);
static bool deferred_incoming;
+static bool enabled_rdma_migration;
/* Messages sent on the return path from destination to source */
enum mig_rp_message_type {
@@ -354,6 +355,7 @@ void migrate_add_address(SocketAddress *address)
void qemu_start_incoming_migration(const char *uri, Error **errp)
{
const char *p;
+ enabled_rdma_migration = false;
qapi_event_send_migration(MIGRATION_STATUS_SETUP);
if (!strcmp(uri, "defer")) {
@@ -362,6 +364,7 @@ void qemu_start_incoming_migration(const char *uri, Error **errp)
tcp_start_incoming_migration(p, errp);
#ifdef CONFIG_RDMA
} else if (strstart(uri, "rdma:", &p)) {
+ enabled_rdma_migration = true;
rdma_start_incoming_migration(p, errp);
#endif
} else if (strstart(uri, "exec:", &p)) {
@@ -1982,6 +1985,7 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
Error *local_err = NULL;
MigrationState *s = migrate_get_current();
const char *p;
+ enabled_rdma_migration = false;
if (!migrate_prepare(s, has_blk && blk, has_inc && inc,
has_resume && resume, errp)) {
@@ -1993,6 +1997,7 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
tcp_start_outgoing_migration(s, p, &local_err);
#ifdef CONFIG_RDMA
} else if (strstart(uri, "rdma:", &p)) {
+ enabled_rdma_migration = true;
rdma_start_outgoing_migration(s, p, &local_err);
#endif
} else if (strstart(uri, "exec:", &p)) {
@@ -2208,6 +2213,11 @@ bool migrate_use_events(void)
return s->enabled_capabilities[MIGRATION_CAPABILITY_EVENTS];
}
+bool migrate_use_rdma(void)
+{
+ return enabled_rdma_migration;
+}
+
bool migrate_use_rdma_pin_all(void)
{
MigrationState *s;
diff --git a/migration/migration.h b/migration/migration.h
index 50fc2693c7..9b37320d50 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -297,6 +297,7 @@ bool migrate_ignore_shared(void);
bool migrate_validate_uuid(void);
bool migrate_auto_converge(void);
+bool migrate_use_rdma(void);
bool migrate_use_rdma_pin_all(void);
bool migrate_use_multifd(void);
bool migrate_pause_before_switchover(void);
--
2.19.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* Re: [PATCH RFC 02/14] migration: judge whether or not the RDMA is used for migration
2020-02-13 9:37 ` [PATCH RFC 02/14] migration: judge whether or not the RDMA is used for migration Zhimin Feng
@ 2020-02-13 10:04 ` Juan Quintela
0 siblings, 0 replies; 22+ messages in thread
From: Juan Quintela @ 2020-02-13 10:04 UTC (permalink / raw)
To: Zhimin Feng
Cc: zhang.zhanghailiang, armbru, qemu-devel, dgilbert, jemmy858585
Zhimin Feng <fengzhimin1@huawei.com> wrote:
> Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
> ---
> migration/migration.c | 10 ++++++++++
> migration/migration.h | 1 +
> 2 files changed, 11 insertions(+)
>
> diff --git a/migration/migration.c b/migration/migration.c
> index 10a13e0c79..819089a7ea 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -107,6 +107,7 @@ static NotifierList migration_state_notifiers =
> NOTIFIER_LIST_INITIALIZER(migration_state_notifiers);
>
> static bool deferred_incoming;
> +static bool enabled_rdma_migration;
Please no.
Use a field in migration state.
No problem with the rest ofthe patch.
Later, Juan.
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH RFC 03/14] migration/rdma: Create multiFd migration threads
2020-02-13 9:37 [PATCH RFC 00/14] *** multifd for RDMA v2 *** Zhimin Feng
2020-02-13 9:37 ` [PATCH RFC 01/14] migration: add the 'migrate_use_rdma_pin_all' function Zhimin Feng
2020-02-13 9:37 ` [PATCH RFC 02/14] migration: judge whether or not the RDMA is used for migration Zhimin Feng
@ 2020-02-13 9:37 ` Zhimin Feng
2020-02-13 10:12 ` Juan Quintela
2020-02-13 9:37 ` [PATCH RFC 04/14] migration/rdma: Export the RDMAContext struct Zhimin Feng
` (11 subsequent siblings)
14 siblings, 1 reply; 22+ messages in thread
From: Zhimin Feng @ 2020-02-13 9:37 UTC (permalink / raw)
To: quintela, dgilbert, armbru, eblake
Cc: jemmy858585, Zhimin Feng, qemu-devel, zhang.zhanghailiang
Creation of the multifd send threads for RDMA migration,
nothing inside yet.
Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
---
migration/multifd.c | 33 +++++++++++++---
migration/multifd.h | 2 +
migration/qemu-file.c | 5 +++
migration/qemu-file.h | 1 +
migration/rdma.c | 88 ++++++++++++++++++++++++++++++++++++++++++-
migration/rdma.h | 3 ++
6 files changed, 125 insertions(+), 7 deletions(-)
diff --git a/migration/multifd.c b/migration/multifd.c
index b3e8ae9bcc..63678d7fdd 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -424,7 +424,7 @@ void multifd_send_sync_main(QEMUFile *f)
{
int i;
- if (!migrate_use_multifd()) {
+ if (!migrate_use_multifd() || migrate_use_rdma()) {
return;
}
if (multifd_send_state->pages->used) {
@@ -562,6 +562,20 @@ out:
return NULL;
}
+static void rdma_send_channel_create(MultiFDSendParams *p)
+{
+ Error *local_err = NULL;
+
+ if (p->quit) {
+ error_setg(&local_err, "multifd: send id %d already quit", p->id);
+ return ;
+ }
+ p->running = true;
+
+ qemu_thread_create(&p->thread, p->name, multifd_rdma_send_thread, p,
+ QEMU_THREAD_JOINABLE);
+}
+
static void multifd_new_send_channel_async(QIOTask *task, gpointer opaque)
{
MultiFDSendParams *p = opaque;
@@ -621,7 +635,11 @@ int multifd_save_setup(Error **errp)
p->packet->magic = cpu_to_be32(MULTIFD_MAGIC);
p->packet->version = cpu_to_be32(MULTIFD_VERSION);
p->name = g_strdup_printf("multifdsend_%d", i);
- socket_send_channel_create(multifd_new_send_channel_async, p);
+ if (!migrate_use_rdma()) {
+ socket_send_channel_create(multifd_new_send_channel_async, p);
+ } else {
+ rdma_send_channel_create(p);
+ }
}
return 0;
}
@@ -720,7 +738,7 @@ void multifd_recv_sync_main(void)
{
int i;
- if (!migrate_use_multifd()) {
+ if (!migrate_use_multifd() || migrate_use_rdma()) {
return;
}
for (i = 0; i < migrate_multifd_channels(); i++) {
@@ -890,8 +908,13 @@ bool multifd_recv_new_channel(QIOChannel *ioc, Error **errp)
p->num_packets = 1;
p->running = true;
- qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p,
- QEMU_THREAD_JOINABLE);
+ if (!migrate_use_rdma()) {
+ qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p,
+ QEMU_THREAD_JOINABLE);
+ } else {
+ qemu_thread_create(&p->thread, p->name, multifd_rdma_recv_thread, p,
+ QEMU_THREAD_JOINABLE);
+ }
atomic_inc(&multifd_recv_state->count);
return atomic_read(&multifd_recv_state->count) ==
migrate_multifd_channels();
diff --git a/migration/multifd.h b/migration/multifd.h
index d8b0205977..c9c11ad140 100644
--- a/migration/multifd.h
+++ b/migration/multifd.h
@@ -13,6 +13,8 @@
#ifndef QEMU_MIGRATION_MULTIFD_H
#define QEMU_MIGRATION_MULTIFD_H
+#include "migration/rdma.h"
+
int multifd_save_setup(Error **errp);
void multifd_save_cleanup(void);
int multifd_load_setup(Error **errp);
diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 1c3a358a14..f0ed8f1381 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -248,6 +248,11 @@ void qemu_fflush(QEMUFile *f)
f->iovcnt = 0;
}
+void *getQIOChannel(QEMUFile *f)
+{
+ return f->opaque;
+}
+
void ram_control_before_iterate(QEMUFile *f, uint64_t flags)
{
int ret = 0;
diff --git a/migration/qemu-file.h b/migration/qemu-file.h
index a9b6d6ccb7..fc656a3b72 100644
--- a/migration/qemu-file.h
+++ b/migration/qemu-file.h
@@ -161,6 +161,7 @@ int qemu_file_shutdown(QEMUFile *f);
QEMUFile *qemu_file_get_return_path(QEMUFile *f);
void qemu_fflush(QEMUFile *f);
void qemu_file_set_blocking(QEMUFile *f, bool block);
+void *getQIOChannel(QEMUFile *f);
void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
diff --git a/migration/rdma.c b/migration/rdma.c
index 2379b8345b..f086ab5a82 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -34,6 +34,7 @@
#include <arpa/inet.h>
#include <rdma/rdma_cma.h>
#include "trace.h"
+#include "multifd.h"
/*
* Print and error on both the Monitor and the Log file.
@@ -3975,6 +3976,34 @@ static QEMUFile *qemu_fopen_rdma(RDMAContext *rdma, const char *mode)
return rioc->file;
}
+static void migration_rdma_process_incoming(QEMUFile *f, Error **errp)
+{
+ MigrationIncomingState *mis = migration_incoming_get_current();
+ Error *local_err = NULL;
+ QIOChannel *ioc = NULL;
+ bool start_migration;
+
+ if (!mis->from_src_file) {
+ mis->from_src_file = f;
+ qemu_file_set_blocking(f, false);
+
+ start_migration = migrate_use_multifd();
+ } else {
+ ioc = QIO_CHANNEL(getQIOChannel(f));
+ /* Multiple connections */
+ assert(migrate_use_multifd());
+ start_migration = multifd_recv_new_channel(ioc, &local_err);
+ if (local_err) {
+ error_propagate(errp, local_err);
+ return;
+ }
+ }
+
+ if (start_migration) {
+ migration_incoming_process();
+ }
+}
+
static void rdma_accept_incoming_migration(void *opaque)
{
RDMAContext *rdma = opaque;
@@ -4003,8 +4032,12 @@ static void rdma_accept_incoming_migration(void *opaque)
return;
}
- rdma->migration_started_on_destination = 1;
- migration_fd_process_incoming(f, errp);
+ if (migrate_use_multifd()) {
+ migration_rdma_process_incoming(f, errp);
+ } else {
+ rdma->migration_started_on_destination = 1;
+ migration_fd_process_incoming(f, errp);
+ }
}
void rdma_start_incoming_migration(const char *host_port, Error **errp)
@@ -4048,6 +4081,15 @@ void rdma_start_incoming_migration(const char *host_port, Error **errp)
qemu_rdma_return_path_dest_init(rdma_return_path, rdma);
}
+ if (multifd_load_setup(&local_err) != 0) {
+ /*
+ * We haven't been able to create multifd threads
+ * nothing better to do
+ */
+ error_report_err(local_err);
+ goto err;
+ }
+
qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migration,
NULL, (void *)(intptr_t)rdma);
return;
@@ -4118,3 +4160,45 @@ err:
g_free(rdma);
g_free(rdma_return_path);
}
+
+void *multifd_rdma_recv_thread(void *opaque)
+{
+ MultiFDRecvParams *p = opaque;
+
+ while (true) {
+ qemu_mutex_lock(&p->mutex);
+ if (p->quit) {
+ qemu_mutex_unlock(&p->mutex);
+ break;
+ }
+ qemu_mutex_unlock(&p->mutex);
+ qemu_sem_wait(&p->sem_sync);
+ }
+
+ qemu_mutex_lock(&p->mutex);
+ p->running = false;
+ qemu_mutex_unlock(&p->mutex);
+
+ return NULL;
+}
+
+void *multifd_rdma_send_thread(void *opaque)
+{
+ MultiFDSendParams *p = opaque;
+
+ while (true) {
+ qemu_mutex_lock(&p->mutex);
+ if (p->quit) {
+ qemu_mutex_unlock(&p->mutex);
+ break;
+ }
+ qemu_mutex_unlock(&p->mutex);
+ qemu_sem_wait(&p->sem);
+ }
+
+ qemu_mutex_lock(&p->mutex);
+ p->running = false;
+ qemu_mutex_unlock(&p->mutex);
+
+ return NULL;
+}
diff --git a/migration/rdma.h b/migration/rdma.h
index de2ba09dc5..3a00573083 100644
--- a/migration/rdma.h
+++ b/migration/rdma.h
@@ -17,6 +17,9 @@
#ifndef QEMU_MIGRATION_RDMA_H
#define QEMU_MIGRATION_RDMA_H
+void *multifd_rdma_recv_thread(void *opaque);
+void *multifd_rdma_send_thread(void *opaque);
+
void rdma_start_outgoing_migration(void *opaque, const char *host_port,
Error **errp);
--
2.19.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* Re: [PATCH RFC 03/14] migration/rdma: Create multiFd migration threads
2020-02-13 9:37 ` [PATCH RFC 03/14] migration/rdma: Create multiFd migration threads Zhimin Feng
@ 2020-02-13 10:12 ` Juan Quintela
2020-02-14 9:51 ` fengzhimin
0 siblings, 1 reply; 22+ messages in thread
From: Juan Quintela @ 2020-02-13 10:12 UTC (permalink / raw)
To: Zhimin Feng
Cc: zhang.zhanghailiang, armbru, qemu-devel, dgilbert, jemmy858585
Zhimin Feng <fengzhimin1@huawei.com> wrote:
> Creation of the multifd send threads for RDMA migration,
> nothing inside yet.
>
> Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
> ---
> migration/multifd.c | 33 +++++++++++++---
> migration/multifd.h | 2 +
> migration/qemu-file.c | 5 +++
> migration/qemu-file.h | 1 +
> migration/rdma.c | 88 ++++++++++++++++++++++++++++++++++++++++++-
> migration/rdma.h | 3 ++
> 6 files changed, 125 insertions(+), 7 deletions(-)
>
> diff --git a/migration/multifd.c b/migration/multifd.c
> index b3e8ae9bcc..63678d7fdd 100644
> --- a/migration/multifd.c
> +++ b/migration/multifd.c
> @@ -424,7 +424,7 @@ void multifd_send_sync_main(QEMUFile *f)
> {
> int i;
>
> - if (!migrate_use_multifd()) {
> + if (!migrate_use_multifd() || migrate_use_rdma()) {
You don't need sync with main channel on rdma?
> +static void rdma_send_channel_create(MultiFDSendParams *p)
> +{
> + Error *local_err = NULL;
> +
> + if (p->quit) {
> + error_setg(&local_err, "multifd: send id %d already quit", p->id);
> + return ;
> + }
> + p->running = true;
> +
> + qemu_thread_create(&p->thread, p->name, multifd_rdma_send_thread, p,
> + QEMU_THREAD_JOINABLE);
> +}
> +
> static void multifd_new_send_channel_async(QIOTask *task, gpointer opaque)
> {
> MultiFDSendParams *p = opaque;
> @@ -621,7 +635,11 @@ int multifd_save_setup(Error **errp)
> p->packet->magic = cpu_to_be32(MULTIFD_MAGIC);
> p->packet->version = cpu_to_be32(MULTIFD_VERSION);
> p->name = g_strdup_printf("multifdsend_%d", i);
> - socket_send_channel_create(multifd_new_send_channel_async, p);
> + if (!migrate_use_rdma()) {
> + socket_send_channel_create(multifd_new_send_channel_async, p);
> + } else {
> + rdma_send_channel_create(p);
> + }
This is what we are trying to avoid. Just create a struct ops, where we
have a
ops->create_channel(new_channel_async, p)
or whatever, and fill it differently for rdma and for tcp.
> }
> return 0;
> }
> @@ -720,7 +738,7 @@ void multifd_recv_sync_main(void)
> {
> int i;
>
> - if (!migrate_use_multifd()) {
> + if (!migrate_use_multifd() || migrate_use_rdma()) {
> return;
> }
Ok. you can just put an empty function for you here.
> for (i = 0; i < migrate_multifd_channels(); i++) {
> @@ -890,8 +908,13 @@ bool multifd_recv_new_channel(QIOChannel *ioc, Error **errp)
> p->num_packets = 1;
>
> p->running = true;
> - qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p,
> - QEMU_THREAD_JOINABLE);
> + if (!migrate_use_rdma()) {
> + qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p,
> + QEMU_THREAD_JOINABLE);
> + } else {
> + qemu_thread_create(&p->thread, p->name, multifd_rdma_recv_thread, p,
> + QEMU_THREAD_JOINABLE);
> + }
new_recv_chanel() member function.
> atomic_inc(&multifd_recv_state->count);
> return atomic_read(&multifd_recv_state->count) ==
> migrate_multifd_channels();
> diff --git a/migration/multifd.h b/migration/multifd.h
> index d8b0205977..c9c11ad140 100644
> --- a/migration/multifd.h
> +++ b/migration/multifd.h
> @@ -13,6 +13,8 @@
> #ifndef QEMU_MIGRATION_MULTIFD_H
> #define QEMU_MIGRATION_MULTIFD_H
>
> +#include "migration/rdma.h"
> +
> int multifd_save_setup(Error **errp);
> void multifd_save_cleanup(void);
> int multifd_load_setup(Error **errp);
You are not exporting anything rdma related from here, are you?
> diff --git a/migration/qemu-file.c b/migration/qemu-file.c
> index 1c3a358a14..f0ed8f1381 100644
> --- a/migration/qemu-file.c
> +++ b/migration/qemu-file.c
> @@ -248,6 +248,11 @@ void qemu_fflush(QEMUFile *f)
> f->iovcnt = 0;
> }
>
> +void *getQIOChannel(QEMUFile *f)
> +{
> + return f->opaque;
> +}
> +
We really want this to return a void? and not a better type?
> +static void migration_rdma_process_incoming(QEMUFile *f, Error **errp)
> +{
> + MigrationIncomingState *mis = migration_incoming_get_current();
> + Error *local_err = NULL;
> + QIOChannel *ioc = NULL;
> + bool start_migration;
> +
> + if (!mis->from_src_file) {
> + mis->from_src_file = f;
> + qemu_file_set_blocking(f, false);
> +
> + start_migration = migrate_use_multifd();
> + } else {
> + ioc = QIO_CHANNEL(getQIOChannel(f));
> + /* Multiple connections */
> + assert(migrate_use_multifd());
I am not sure that you can make this incompatible change.
You need to have *both*, old method and new multifd one.
I would have been happy to remove old precopy tcp method, but we
*assure* backwards compatibility.
> @@ -4003,8 +4032,12 @@ static void rdma_accept_incoming_migration(void *opaque)
> return;
> }
>
> - rdma->migration_started_on_destination = 1;
> - migration_fd_process_incoming(f, errp);
> + if (migrate_use_multifd()) {
> + migration_rdma_process_incoming(f, errp);
> + } else {
> + rdma->migration_started_on_destination = 1;
> + migration_fd_process_incoming(f, errp);
> + }
But here you allow that multifd is not defined?
> +
> +void *multifd_rdma_recv_thread(void *opaque)
> +{
Why can't you use the multifd_recv_thread() directly, just creating
different ops when you need them?
Later, Juan.
^ permalink raw reply [flat|nested] 22+ messages in thread* RE: [PATCH RFC 03/14] migration/rdma: Create multiFd migration threads
2020-02-13 10:12 ` Juan Quintela
@ 2020-02-14 9:51 ` fengzhimin
0 siblings, 0 replies; 22+ messages in thread
From: fengzhimin @ 2020-02-14 9:51 UTC (permalink / raw)
To: quintela; +Cc: Zhanghailiang, armbru, qemu-devel, dgilbert, jemmy858585
Thanks for your review. I will fix these errors in the next version(V3).
Due to migration data transfer using RDMA WRITE operation, we don't need to receive data in the destination.
We only need to poll the CQE in the destination, so multifd_recv_thread() can't be used directly.
-----Original Message-----
From: Juan Quintela [mailto:quintela@redhat.com]
Sent: Thursday, February 13, 2020 6:13 PM
To: fengzhimin <fengzhimin1@huawei.com>
Cc: dgilbert@redhat.com; armbru@redhat.com; eblake@redhat.com; qemu-devel@nongnu.org; Zhanghailiang <zhang.zhanghailiang@huawei.com>; jemmy858585@gmail.com
Subject: Re: [PATCH RFC 03/14] migration/rdma: Create multiFd migration threads
Zhimin Feng <fengzhimin1@huawei.com> wrote:
> Creation of the multifd send threads for RDMA migration, nothing
> inside yet.
>
> Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
> ---
> migration/multifd.c | 33 +++++++++++++---
> migration/multifd.h | 2 +
> migration/qemu-file.c | 5 +++
> migration/qemu-file.h | 1 +
> migration/rdma.c | 88 ++++++++++++++++++++++++++++++++++++++++++-
> migration/rdma.h | 3 ++
> 6 files changed, 125 insertions(+), 7 deletions(-)
>
> diff --git a/migration/multifd.c b/migration/multifd.c index
> b3e8ae9bcc..63678d7fdd 100644
> --- a/migration/multifd.c
> +++ b/migration/multifd.c
> @@ -424,7 +424,7 @@ void multifd_send_sync_main(QEMUFile *f) {
> int i;
>
> - if (!migrate_use_multifd()) {
> + if (!migrate_use_multifd() || migrate_use_rdma()) {
You don't need sync with main channel on rdma?
> +static void rdma_send_channel_create(MultiFDSendParams *p) {
> + Error *local_err = NULL;
> +
> + if (p->quit) {
> + error_setg(&local_err, "multifd: send id %d already quit", p->id);
> + return ;
> + }
> + p->running = true;
> +
> + qemu_thread_create(&p->thread, p->name, multifd_rdma_send_thread, p,
> + QEMU_THREAD_JOINABLE); }
> +
> static void multifd_new_send_channel_async(QIOTask *task, gpointer
> opaque) {
> MultiFDSendParams *p = opaque;
> @@ -621,7 +635,11 @@ int multifd_save_setup(Error **errp)
> p->packet->magic = cpu_to_be32(MULTIFD_MAGIC);
> p->packet->version = cpu_to_be32(MULTIFD_VERSION);
> p->name = g_strdup_printf("multifdsend_%d", i);
> - socket_send_channel_create(multifd_new_send_channel_async, p);
> + if (!migrate_use_rdma()) {
> + socket_send_channel_create(multifd_new_send_channel_async, p);
> + } else {
> + rdma_send_channel_create(p);
> + }
This is what we are trying to avoid. Just create a struct ops, where we have a
ops->create_channel(new_channel_async, p)
or whatever, and fill it differently for rdma and for tcp.
> }
> return 0;
> }
> @@ -720,7 +738,7 @@ void multifd_recv_sync_main(void) {
> int i;
>
> - if (!migrate_use_multifd()) {
> + if (!migrate_use_multifd() || migrate_use_rdma()) {
> return;
> }
Ok. you can just put an empty function for you here.
> for (i = 0; i < migrate_multifd_channels(); i++) { @@ -890,8
> +908,13 @@ bool multifd_recv_new_channel(QIOChannel *ioc, Error **errp)
> p->num_packets = 1;
>
> p->running = true;
> - qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p,
> - QEMU_THREAD_JOINABLE);
> + if (!migrate_use_rdma()) {
> + qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p,
> + QEMU_THREAD_JOINABLE);
> + } else {
> + qemu_thread_create(&p->thread, p->name, multifd_rdma_recv_thread, p,
> + QEMU_THREAD_JOINABLE);
> + }
new_recv_chanel() member function.
> atomic_inc(&multifd_recv_state->count);
> return atomic_read(&multifd_recv_state->count) ==
> migrate_multifd_channels(); diff --git
> a/migration/multifd.h b/migration/multifd.h index
> d8b0205977..c9c11ad140 100644
> --- a/migration/multifd.h
> +++ b/migration/multifd.h
> @@ -13,6 +13,8 @@
> #ifndef QEMU_MIGRATION_MULTIFD_H
> #define QEMU_MIGRATION_MULTIFD_H
>
> +#include "migration/rdma.h"
> +
> int multifd_save_setup(Error **errp); void
> multifd_save_cleanup(void); int multifd_load_setup(Error **errp);
You are not exporting anything rdma related from here, are you?
> diff --git a/migration/qemu-file.c b/migration/qemu-file.c index
> 1c3a358a14..f0ed8f1381 100644
> --- a/migration/qemu-file.c
> +++ b/migration/qemu-file.c
> @@ -248,6 +248,11 @@ void qemu_fflush(QEMUFile *f)
> f->iovcnt = 0;
> }
>
> +void *getQIOChannel(QEMUFile *f)
> +{
> + return f->opaque;
> +}
> +
We really want this to return a void? and not a better type?
> +static void migration_rdma_process_incoming(QEMUFile *f, Error
> +**errp) {
> + MigrationIncomingState *mis = migration_incoming_get_current();
> + Error *local_err = NULL;
> + QIOChannel *ioc = NULL;
> + bool start_migration;
> +
> + if (!mis->from_src_file) {
> + mis->from_src_file = f;
> + qemu_file_set_blocking(f, false);
> +
> + start_migration = migrate_use_multifd();
> + } else {
> + ioc = QIO_CHANNEL(getQIOChannel(f));
> + /* Multiple connections */
> + assert(migrate_use_multifd());
I am not sure that you can make this incompatible change.
You need to have *both*, old method and new multifd one.
I would have been happy to remove old precopy tcp method, but we
*assure* backwards compatibility.
> @@ -4003,8 +4032,12 @@ static void rdma_accept_incoming_migration(void *opaque)
> return;
> }
>
> - rdma->migration_started_on_destination = 1;
> - migration_fd_process_incoming(f, errp);
> + if (migrate_use_multifd()) {
> + migration_rdma_process_incoming(f, errp);
> + } else {
> + rdma->migration_started_on_destination = 1;
> + migration_fd_process_incoming(f, errp);
> + }
But here you allow that multifd is not defined?
> +
> +void *multifd_rdma_recv_thread(void *opaque) {
Why can't you use the multifd_recv_thread() directly, just creating different ops when you need them?
Later, Juan.
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH RFC 04/14] migration/rdma: Export the RDMAContext struct
2020-02-13 9:37 [PATCH RFC 00/14] *** multifd for RDMA v2 *** Zhimin Feng
` (2 preceding siblings ...)
2020-02-13 9:37 ` [PATCH RFC 03/14] migration/rdma: Create multiFd migration threads Zhimin Feng
@ 2020-02-13 9:37 ` Zhimin Feng
2020-02-13 9:37 ` [PATCH RFC 05/14] migration/rdma: Create the multifd channels for RDMA Zhimin Feng
` (10 subsequent siblings)
14 siblings, 0 replies; 22+ messages in thread
From: Zhimin Feng @ 2020-02-13 9:37 UTC (permalink / raw)
To: quintela, dgilbert, armbru, eblake
Cc: jemmy858585, Zhimin Feng, qemu-devel, zhang.zhanghailiang
We need to use the RDMAContext in migration/multifd.c
so it has to be exported.
Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
---
migration/rdma.c | 243 ----------------------------------------------
migration/rdma.h | 247 +++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 247 insertions(+), 243 deletions(-)
diff --git a/migration/rdma.c b/migration/rdma.c
index f086ab5a82..a76823986e 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -19,9 +19,7 @@
#include "qemu/cutils.h"
#include "rdma.h"
#include "migration.h"
-#include "qemu-file.h"
#include "ram.h"
-#include "qemu-file-channel.h"
#include "qemu/error-report.h"
#include "qemu/main-loop.h"
#include "qemu/module.h"
@@ -47,34 +45,6 @@
} \
} while (0)
-#define RDMA_RESOLVE_TIMEOUT_MS 10000
-
-/* Do not merge data if larger than this. */
-#define RDMA_MERGE_MAX (2 * 1024 * 1024)
-#define RDMA_SIGNALED_SEND_MAX (RDMA_MERGE_MAX / 4096)
-
-#define RDMA_REG_CHUNK_SHIFT 20 /* 1 MB */
-
-/*
- * This is only for non-live state being migrated.
- * Instead of RDMA_WRITE messages, we use RDMA_SEND
- * messages for that state, which requires a different
- * delivery design than main memory.
- */
-#define RDMA_SEND_INCREMENT 32768
-
-/*
- * Maximum size infiniband SEND message
- */
-#define RDMA_CONTROL_MAX_BUFFER (512 * 1024)
-#define RDMA_CONTROL_MAX_COMMANDS_PER_MESSAGE 4096
-
-#define RDMA_CONTROL_VERSION_CURRENT 1
-/*
- * Capabilities for negotiation.
- */
-#define RDMA_CAPABILITY_PIN_ALL 0x01
-
/*
* Add the other flags above to this list of known capabilities
* as they are introduced.
@@ -117,18 +87,6 @@ static uint32_t known_capabilities = RDMA_CAPABILITY_PIN_ALL;
#define RDMA_WRID_CHUNK_MASK (~RDMA_WRID_BLOCK_MASK & ~RDMA_WRID_TYPE_MASK)
-/*
- * RDMA migration protocol:
- * 1. RDMA Writes (data messages, i.e. RAM)
- * 2. IB Send/Recv (control channel messages)
- */
-enum {
- RDMA_WRID_NONE = 0,
- RDMA_WRID_RDMA_WRITE = 1,
- RDMA_WRID_SEND_CONTROL = 2000,
- RDMA_WRID_RECV_CONTROL = 4000,
-};
-
static const char *wrid_desc[] = {
[RDMA_WRID_NONE] = "NONE",
[RDMA_WRID_RDMA_WRITE] = "WRITE RDMA",
@@ -136,50 +94,6 @@ static const char *wrid_desc[] = {
[RDMA_WRID_RECV_CONTROL] = "CONTROL RECV",
};
-/*
- * Work request IDs for IB SEND messages only (not RDMA writes).
- * This is used by the migration protocol to transmit
- * control messages (such as device state and registration commands)
- *
- * We could use more WRs, but we have enough for now.
- */
-enum {
- RDMA_WRID_READY = 0,
- RDMA_WRID_DATA,
- RDMA_WRID_CONTROL,
- RDMA_WRID_MAX,
-};
-
-/*
- * SEND/RECV IB Control Messages.
- */
-enum {
- RDMA_CONTROL_NONE = 0,
- RDMA_CONTROL_ERROR,
- RDMA_CONTROL_READY, /* ready to receive */
- RDMA_CONTROL_QEMU_FILE, /* QEMUFile-transmitted bytes */
- RDMA_CONTROL_RAM_BLOCKS_REQUEST, /* RAMBlock synchronization */
- RDMA_CONTROL_RAM_BLOCKS_RESULT, /* RAMBlock synchronization */
- RDMA_CONTROL_COMPRESS, /* page contains repeat values */
- RDMA_CONTROL_REGISTER_REQUEST, /* dynamic page registration */
- RDMA_CONTROL_REGISTER_RESULT, /* key to use after registration */
- RDMA_CONTROL_REGISTER_FINISHED, /* current iteration finished */
- RDMA_CONTROL_UNREGISTER_REQUEST, /* dynamic UN-registration */
- RDMA_CONTROL_UNREGISTER_FINISHED, /* unpinning finished */
-};
-
-
-/*
- * Memory and MR structures used to represent an IB Send/Recv work request.
- * This is *not* used for RDMA writes, only IB Send/Recv.
- */
-typedef struct {
- uint8_t control[RDMA_CONTROL_MAX_BUFFER]; /* actual buffer to register */
- struct ibv_mr *control_mr; /* registration metadata */
- size_t control_len; /* length of the message */
- uint8_t *control_curr; /* start of unconsumed bytes */
-} RDMAWorkRequestData;
-
/*
* Negotiate RDMA capabilities during connection-setup time.
*/
@@ -200,46 +114,6 @@ static void network_to_caps(RDMACapabilities *cap)
cap->flags = ntohl(cap->flags);
}
-/*
- * Representation of a RAMBlock from an RDMA perspective.
- * This is not transmitted, only local.
- * This and subsequent structures cannot be linked lists
- * because we're using a single IB message to transmit
- * the information. It's small anyway, so a list is overkill.
- */
-typedef struct RDMALocalBlock {
- char *block_name;
- uint8_t *local_host_addr; /* local virtual address */
- uint64_t remote_host_addr; /* remote virtual address */
- uint64_t offset;
- uint64_t length;
- struct ibv_mr **pmr; /* MRs for chunk-level registration */
- struct ibv_mr *mr; /* MR for non-chunk-level registration */
- uint32_t *remote_keys; /* rkeys for chunk-level registration */
- uint32_t remote_rkey; /* rkeys for non-chunk-level registration */
- int index; /* which block are we */
- unsigned int src_index; /* (Only used on dest) */
- bool is_ram_block;
- int nb_chunks;
- unsigned long *transit_bitmap;
- unsigned long *unregister_bitmap;
-} RDMALocalBlock;
-
-/*
- * Also represents a RAMblock, but only on the dest.
- * This gets transmitted by the dest during connection-time
- * to the source VM and then is used to populate the
- * corresponding RDMALocalBlock with
- * the information needed to perform the actual RDMA.
- */
-typedef struct QEMU_PACKED RDMADestBlock {
- uint64_t remote_host_addr;
- uint64_t offset;
- uint64_t length;
- uint32_t remote_rkey;
- uint32_t padding;
-} RDMADestBlock;
-
static const char *control_desc(unsigned int rdma_control)
{
static const char *strs[] = {
@@ -294,123 +168,6 @@ static void network_to_dest_block(RDMADestBlock *db)
db->remote_rkey = ntohl(db->remote_rkey);
}
-/*
- * Virtual address of the above structures used for transmitting
- * the RAMBlock descriptions at connection-time.
- * This structure is *not* transmitted.
- */
-typedef struct RDMALocalBlocks {
- int nb_blocks;
- bool init; /* main memory init complete */
- RDMALocalBlock *block;
-} RDMALocalBlocks;
-
-/*
- * Main data structure for RDMA state.
- * While there is only one copy of this structure being allocated right now,
- * this is the place where one would start if you wanted to consider
- * having more than one RDMA connection open at the same time.
- */
-typedef struct RDMAContext {
- char *host;
- int port;
-
- RDMAWorkRequestData wr_data[RDMA_WRID_MAX];
-
- /*
- * This is used by *_exchange_send() to figure out whether or not
- * the initial "READY" message has already been received or not.
- * This is because other functions may potentially poll() and detect
- * the READY message before send() does, in which case we need to
- * know if it completed.
- */
- int control_ready_expected;
-
- /* number of outstanding writes */
- int nb_sent;
-
- /* store info about current buffer so that we can
- merge it with future sends */
- uint64_t current_addr;
- uint64_t current_length;
- /* index of ram block the current buffer belongs to */
- int current_index;
- /* index of the chunk in the current ram block */
- int current_chunk;
-
- bool pin_all;
-
- /*
- * infiniband-specific variables for opening the device
- * and maintaining connection state and so forth.
- *
- * cm_id also has ibv_context, rdma_event_channel, and ibv_qp in
- * cm_id->verbs, cm_id->channel, and cm_id->qp.
- */
- struct rdma_cm_id *cm_id; /* connection manager ID */
- struct rdma_cm_id *listen_id;
- bool connected;
-
- struct ibv_context *verbs;
- struct rdma_event_channel *channel;
- struct ibv_qp *qp; /* queue pair */
- struct ibv_comp_channel *comp_channel; /* completion channel */
- struct ibv_pd *pd; /* protection domain */
- struct ibv_cq *cq; /* completion queue */
-
- /*
- * If a previous write failed (perhaps because of a failed
- * memory registration, then do not attempt any future work
- * and remember the error state.
- */
- int error_state;
- int error_reported;
- int received_error;
-
- /*
- * Description of ram blocks used throughout the code.
- */
- RDMALocalBlocks local_ram_blocks;
- RDMADestBlock *dest_blocks;
-
- /* Index of the next RAMBlock received during block registration */
- unsigned int next_src_index;
-
- /*
- * Migration on *destination* started.
- * Then use coroutine yield function.
- * Source runs in a thread, so we don't care.
- */
- int migration_started_on_destination;
-
- int total_registrations;
- int total_writes;
-
- int unregister_current, unregister_next;
- uint64_t unregistrations[RDMA_SIGNALED_SEND_MAX];
-
- GHashTable *blockmap;
-
- /* the RDMAContext for return path */
- struct RDMAContext *return_path;
- bool is_return_path;
-} RDMAContext;
-
-#define TYPE_QIO_CHANNEL_RDMA "qio-channel-rdma"
-#define QIO_CHANNEL_RDMA(obj) \
- OBJECT_CHECK(QIOChannelRDMA, (obj), TYPE_QIO_CHANNEL_RDMA)
-
-typedef struct QIOChannelRDMA QIOChannelRDMA;
-
-
-struct QIOChannelRDMA {
- QIOChannel parent;
- RDMAContext *rdmain;
- RDMAContext *rdmaout;
- QEMUFile *file;
- bool blocking; /* XXX we don't actually honour this yet */
-};
-
/*
* Main structure for IB Send/Recv control messages.
* This gets prepended at the beginning of every Send/Recv.
diff --git a/migration/rdma.h b/migration/rdma.h
index 3a00573083..cb206c7004 100644
--- a/migration/rdma.h
+++ b/migration/rdma.h
@@ -17,6 +17,253 @@
#ifndef QEMU_MIGRATION_RDMA_H
#define QEMU_MIGRATION_RDMA_H
+#include "qemu-file.h"
+#include "qemu-file-channel.h"
+
+#define RDMA_RESOLVE_TIMEOUT_MS 10000
+
+/* Do not merge data if larger than this. */
+#define RDMA_MERGE_MAX (2 * 1024 * 1024)
+#define RDMA_SIGNALED_SEND_MAX (RDMA_MERGE_MAX / 4096)
+
+#define RDMA_REG_CHUNK_SHIFT 20 /* 1 MB */
+
+/*
+ * This is only for non-live state being migrated.
+ * Instead of RDMA_WRITE messages, we use RDMA_SEND
+ * messages for that state, which requires a different
+ * delivery design than main memory.
+ */
+#define RDMA_SEND_INCREMENT 32768
+
+/*
+ * Maximum size infiniband SEND message
+ */
+#define RDMA_CONTROL_MAX_BUFFER (512 * 1024)
+#define RDMA_CONTROL_MAX_COMMANDS_PER_MESSAGE 4096
+
+#define RDMA_CONTROL_VERSION_CURRENT 1
+/*
+ * Capabilities for negotiation.
+ */
+#define RDMA_CAPABILITY_PIN_ALL 0x01
+
+/*
+ * RDMA migration protocol:
+ * 1. RDMA Writes (data messages, i.e. RAM)
+ * 2. IB Send/Recv (control channel messages)
+ */
+enum {
+ RDMA_WRID_NONE = 0,
+ RDMA_WRID_RDMA_WRITE = 1,
+ RDMA_WRID_SEND_CONTROL = 2000,
+ RDMA_WRID_RECV_CONTROL = 4000,
+};
+
+/*
+ * Work request IDs for IB SEND messages only (not RDMA writes).
+ * This is used by the migration protocol to transmit
+ * control messages (such as device state and registration commands)
+ *
+ * We could use more WRs, but we have enough for now.
+ */
+enum {
+ RDMA_WRID_READY = 0,
+ RDMA_WRID_DATA,
+ RDMA_WRID_CONTROL,
+ RDMA_WRID_MAX,
+};
+
+/*
+ * SEND/RECV IB Control Messages.
+ */
+enum {
+ RDMA_CONTROL_NONE = 0,
+ RDMA_CONTROL_ERROR,
+ RDMA_CONTROL_READY, /* ready to receive */
+ RDMA_CONTROL_QEMU_FILE, /* QEMUFile-transmitted bytes */
+ RDMA_CONTROL_RAM_BLOCKS_REQUEST, /* RAMBlock synchronization */
+ RDMA_CONTROL_RAM_BLOCKS_RESULT, /* RAMBlock synchronization */
+ RDMA_CONTROL_COMPRESS, /* page contains repeat values */
+ RDMA_CONTROL_REGISTER_REQUEST, /* dynamic page registration */
+ RDMA_CONTROL_REGISTER_RESULT, /* key to use after registration */
+ RDMA_CONTROL_REGISTER_FINISHED, /* current iteration finished */
+ RDMA_CONTROL_UNREGISTER_REQUEST, /* dynamic UN-registration */
+ RDMA_CONTROL_UNREGISTER_FINISHED, /* unpinning finished */
+};
+
+
+/*
+ * Memory and MR structures used to represent an IB Send/Recv work request.
+ * This is *not* used for RDMA writes, only IB Send/Recv.
+ */
+typedef struct {
+ uint8_t control[RDMA_CONTROL_MAX_BUFFER]; /* actual buffer to register */
+ struct ibv_mr *control_mr; /* registration metadata */
+ size_t control_len; /* length of the message */
+ uint8_t *control_curr; /* start of unconsumed bytes */
+} RDMAWorkRequestData;
+
+/*
+ * Representation of a RAMBlock from an RDMA perspective.
+ * This is not transmitted, only local.
+ * This and subsequent structures cannot be linked lists
+ * because we're using a single IB message to transmit
+ * the information. It's small anyway, so a list is overkill.
+ */
+typedef struct RDMALocalBlock {
+ char *block_name;
+ uint8_t *local_host_addr; /* local virtual address */
+ uint64_t remote_host_addr; /* remote virtual address */
+ uint64_t offset;
+ uint64_t length;
+ struct ibv_mr **pmr; /* MRs for chunk-level registration */
+ struct ibv_mr *mr; /* MR for non-chunk-level registration */
+ uint32_t *remote_keys; /* rkeys for chunk-level registration */
+ uint32_t remote_rkey; /* rkeys for non-chunk-level registration */
+ int index; /* which block are we */
+ unsigned int src_index; /* (Only used on dest) */
+ bool is_ram_block;
+ int nb_chunks;
+ unsigned long *transit_bitmap;
+ unsigned long *unregister_bitmap;
+} RDMALocalBlock;
+
+/*
+ * Also represents a RAMblock, but only on the dest.
+ * This gets transmitted by the dest during connection-time
+ * to the source VM and then is used to populate the
+ * corresponding RDMALocalBlock with
+ * the information needed to perform the actual RDMA.
+ */
+typedef struct QEMU_PACKED RDMADestBlock {
+ uint64_t remote_host_addr;
+ uint64_t offset;
+ uint64_t length;
+ uint32_t remote_rkey;
+ uint32_t padding;
+} RDMADestBlock;
+
+/*
+ * Virtual address of the above structures used for transmitting
+ * the RAMBlock descriptions at connection-time.
+ * This structure is *not* transmitted.
+ */
+typedef struct RDMALocalBlocks {
+ int nb_blocks;
+ bool init; /* main memory init complete */
+ RDMALocalBlock *block;
+} RDMALocalBlocks;
+
+/*
+ * Main data structure for RDMA state.
+ * While there is only one copy of this structure being allocated right now,
+ * this is the place where one would start if you wanted to consider
+ * having more than one RDMA connection open at the same time.
+ */
+typedef struct RDMAContext {
+ char *host;
+ int port;
+
+ RDMAWorkRequestData wr_data[RDMA_WRID_MAX];
+
+ /*
+ * This is used by *_exchange_send() to figure out whether or not
+ * the initial "READY" message has already been received or not.
+ * This is because other functions may potentially poll() and detect
+ * the READY message before send() does, in which case we need to
+ * know if it completed.
+ */
+ int control_ready_expected;
+
+ /* number of outstanding writes */
+ int nb_sent;
+
+ /*
+ * store info about current buffer so that we can
+ * merge it with future sends
+ */
+ uint64_t current_addr;
+ uint64_t current_length;
+ /* index of ram block the current buffer belongs to */
+ int current_index;
+ /* index of the chunk in the current ram block */
+ int current_chunk;
+
+ bool pin_all;
+
+ /*
+ * infiniband-specific variables for opening the device
+ * and maintaining connection state and so forth.
+ *
+ * cm_id also has ibv_context, rdma_event_channel, and ibv_qp in
+ * cm_id->verbs, cm_id->channel, and cm_id->qp.
+ */
+ struct rdma_cm_id *cm_id; /* connection manager ID */
+ struct rdma_cm_id *listen_id;
+ bool connected;
+
+ struct ibv_context *verbs;
+ struct rdma_event_channel *channel;
+ struct ibv_qp *qp; /* queue pair */
+ struct ibv_comp_channel *comp_channel; /* completion channel */
+ struct ibv_pd *pd; /* protection domain */
+ struct ibv_cq *cq; /* completion queue */
+
+ /*
+ * If a previous write failed (perhaps because of a failed
+ * memory registration, then do not attempt any future work
+ * and remember the error state.
+ */
+ int error_state;
+ int error_reported;
+ int received_error;
+
+ /*
+ * Description of ram blocks used throughout the code.
+ */
+ RDMALocalBlocks local_ram_blocks;
+ RDMADestBlock *dest_blocks;
+
+ /* Index of the next RAMBlock received during block registration */
+ unsigned int next_src_index;
+
+ /*
+ * Migration on *destination* started.
+ * Then use coroutine yield function.
+ * Source runs in a thread, so we don't care.
+ */
+ int migration_started_on_destination;
+
+ int total_registrations;
+ int total_writes;
+
+ int unregister_current, unregister_next;
+ uint64_t unregistrations[RDMA_SIGNALED_SEND_MAX];
+
+ GHashTable *blockmap;
+
+ /* the RDMAContext for return path */
+ struct RDMAContext *return_path;
+ bool is_return_path;
+} RDMAContext;
+
+#define TYPE_QIO_CHANNEL_RDMA "qio-channel-rdma"
+#define QIO_CHANNEL_RDMA(obj) \
+ OBJECT_CHECK(QIOChannelRDMA, (obj), TYPE_QIO_CHANNEL_RDMA)
+
+typedef struct QIOChannelRDMA QIOChannelRDMA;
+
+
+struct QIOChannelRDMA {
+ QIOChannel parent;
+ RDMAContext *rdmain;
+ RDMAContext *rdmaout;
+ QEMUFile *file;
+ bool blocking; /* XXX we don't actually honour this yet */
+};
+
+
void *multifd_rdma_recv_thread(void *opaque);
void *multifd_rdma_send_thread(void *opaque);
--
2.19.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH RFC 05/14] migration/rdma: Create the multifd channels for RDMA
2020-02-13 9:37 [PATCH RFC 00/14] *** multifd for RDMA v2 *** Zhimin Feng
` (3 preceding siblings ...)
2020-02-13 9:37 ` [PATCH RFC 04/14] migration/rdma: Export the RDMAContext struct Zhimin Feng
@ 2020-02-13 9:37 ` Zhimin Feng
2020-02-13 9:37 ` [PATCH RFC 06/14] migration/rdma: Transmit initial packet Zhimin Feng
` (9 subsequent siblings)
14 siblings, 0 replies; 22+ messages in thread
From: Zhimin Feng @ 2020-02-13 9:37 UTC (permalink / raw)
To: quintela, dgilbert, armbru, eblake
Cc: jemmy858585, Zhimin Feng, qemu-devel, zhang.zhanghailiang
In both sides. We still don't transmit anything through them,
and we only build the RDMA connections.
Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
---
migration/multifd.c | 103 ++++++++++++++++++++++++++++++++++++---
migration/multifd.h | 10 ++++
migration/rdma.c | 115 ++++++++++++++++++++++++++++++++------------
migration/rdma.h | 4 +-
4 files changed, 189 insertions(+), 43 deletions(-)
diff --git a/migration/multifd.c b/migration/multifd.c
index 63678d7fdd..acdfd3d5b3 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -248,6 +248,19 @@ struct {
int exiting;
} *multifd_send_state;
+int get_multifd_send_param(int id, MultiFDSendParams **param)
+{
+ int ret = 0;
+
+ if (id < 0 || id >= migrate_multifd_channels()) {
+ ret = -1;
+ } else {
+ *param = &(multifd_send_state->params[id]);
+ }
+
+ return ret;
+}
+
/*
* How we use multifd_send_state->pages and channel->pages?
*
@@ -410,6 +423,9 @@ void multifd_save_cleanup(void)
p->packet_len = 0;
g_free(p->packet);
p->packet = NULL;
+ if (migrate_use_rdma()) {
+ g_free(p->rdma);
+ }
}
qemu_sem_destroy(&multifd_send_state->channels_ready);
g_free(multifd_send_state->params);
@@ -464,6 +480,27 @@ void multifd_send_sync_main(QEMUFile *f)
trace_multifd_send_sync_main(multifd_send_state->packet_num);
}
+static void *multifd_rdma_send_thread(void *opaque)
+{
+ MultiFDSendParams *p = opaque;
+
+ while (true) {
+ qemu_mutex_lock(&p->mutex);
+ if (p->quit) {
+ qemu_mutex_unlock(&p->mutex);
+ break;
+ }
+ qemu_mutex_unlock(&p->mutex);
+ qemu_sem_wait(&p->sem);
+ }
+
+ qemu_mutex_lock(&p->mutex);
+ p->running = false;
+ qemu_mutex_unlock(&p->mutex);
+
+ return NULL;
+}
+
static void *multifd_send_thread(void *opaque)
{
MultiFDSendParams *p = opaque;
@@ -566,6 +603,12 @@ static void rdma_send_channel_create(MultiFDSendParams *p)
{
Error *local_err = NULL;
+ if (multifd_channel_rdma_connect(p)) {
+ error_setg(&local_err, "multifd: rdma channel %d not established",
+ p->id);
+ return ;
+ }
+
if (p->quit) {
error_setg(&local_err, "multifd: send id %d already quit", p->id);
return ;
@@ -654,6 +697,19 @@ struct {
uint64_t packet_num;
} *multifd_recv_state;
+int get_multifd_recv_param(int id, MultiFDRecvParams **param)
+{
+ int ret = 0;
+
+ if (id < 0 || id >= migrate_multifd_channels()) {
+ ret = -1;
+ } else {
+ *param = &(multifd_recv_state->params[id]);
+ }
+
+ return ret;
+}
+
static void multifd_recv_terminate_threads(Error *err)
{
int i;
@@ -724,6 +780,9 @@ int multifd_load_cleanup(Error **errp)
p->packet_len = 0;
g_free(p->packet);
p->packet = NULL;
+ if (migrate_use_rdma()) {
+ g_free(p->rdma);
+ }
}
qemu_sem_destroy(&multifd_recv_state->sem_sync);
g_free(multifd_recv_state->params);
@@ -761,6 +820,27 @@ void multifd_recv_sync_main(void)
trace_multifd_recv_sync_main(multifd_recv_state->packet_num);
}
+static void *multifd_rdma_recv_thread(void *opaque)
+{
+ MultiFDRecvParams *p = opaque;
+
+ while (true) {
+ qemu_mutex_lock(&p->mutex);
+ if (p->quit) {
+ qemu_mutex_unlock(&p->mutex);
+ break;
+ }
+ qemu_mutex_unlock(&p->mutex);
+ qemu_sem_wait(&p->sem_sync);
+ }
+
+ qemu_mutex_lock(&p->mutex);
+ p->running = false;
+ qemu_mutex_unlock(&p->mutex);
+
+ return NULL;
+}
+
static void *multifd_recv_thread(void *opaque)
{
MultiFDRecvParams *p = opaque;
@@ -880,18 +960,24 @@ bool multifd_recv_all_channels_created(void)
bool multifd_recv_new_channel(QIOChannel *ioc, Error **errp)
{
MultiFDRecvParams *p;
+ QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc);
Error *local_err = NULL;
int id;
- id = multifd_recv_initial_packet(ioc, &local_err);
- if (id < 0) {
- multifd_recv_terminate_threads(local_err);
- error_propagate_prepend(errp, local_err,
- "failed to receive packet"
- " via multifd channel %d: ",
- atomic_read(&multifd_recv_state->count));
- return false;
+ if (migrate_use_rdma()) {
+ id = multifd_recv_state->count;
+ } else {
+ id = multifd_recv_initial_packet(ioc, &local_err);
+ if (id < 0) {
+ multifd_recv_terminate_threads(local_err);
+ error_propagate_prepend(errp, local_err,
+ "failed to receive packet"
+ " via multifd channel %d: ",
+ atomic_read(&multifd_recv_state->count));
+ return false;
+ }
}
+
trace_multifd_recv_new_channel(id);
p = &multifd_recv_state->params[id];
@@ -903,6 +989,7 @@ bool multifd_recv_new_channel(QIOChannel *ioc, Error **errp)
return false;
}
p->c = ioc;
+ p->file = rioc->file;
object_ref(OBJECT(ioc));
/* initial packet */
p->num_packets = 1;
diff --git a/migration/multifd.h b/migration/multifd.h
index c9c11ad140..1eae427f8c 100644
--- a/migration/multifd.h
+++ b/migration/multifd.h
@@ -67,6 +67,10 @@ typedef struct {
char *name;
/* channel thread id */
QemuThread thread;
+ /* RDMAContext channel */
+ RDMAContext *rdma;
+ /* communication channel */
+ QEMUFile *file;
/* communication channel */
QIOChannel *c;
/* sem where to wait for more work */
@@ -108,6 +112,10 @@ typedef struct {
char *name;
/* channel thread id */
QemuThread thread;
+ /* RDMAContext channel */
+ RDMAContext *rdma;
+ /* communication channel */
+ QEMUFile *file;
/* communication channel */
QIOChannel *c;
/* this mutex protects the following parameters */
@@ -137,5 +145,7 @@ typedef struct {
QemuSemaphore sem_sync;
} MultiFDRecvParams;
+int get_multifd_send_param(int id, MultiFDSendParams **param);
+int get_multifd_recv_param(int id, MultiFDRecvParams **param);
#endif
diff --git a/migration/rdma.c b/migration/rdma.c
index a76823986e..48615fcaad 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -94,6 +94,8 @@ static const char *wrid_desc[] = {
[RDMA_WRID_RECV_CONTROL] = "CONTROL RECV",
};
+static const char *rdma_host_port;
+
/*
* Negotiate RDMA capabilities during connection-setup time.
*/
@@ -3122,6 +3124,33 @@ static int qemu_rdma_accept(RDMAContext *rdma)
qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migration,
NULL,
(void *)(intptr_t)rdma->return_path);
+ } else if (migrate_use_multifd()) {
+ int thread_count;
+ int i;
+ MultiFDRecvParams *multifd_recv_param;
+ RDMAContext *multifd_rdma = NULL;
+ thread_count = migrate_multifd_channels();
+ /* create the multifd channels for RDMA */
+ for (i = 0; i < thread_count; i++) {
+ if (get_multifd_recv_param(i, &multifd_recv_param) < 0) {
+ error_report("rdma: error getting multifd_recv_param(%d)", i);
+ goto err_rdma_dest_wait;
+ }
+
+ if (multifd_recv_param->rdma->cm_id == NULL) {
+ multifd_rdma = multifd_recv_param->rdma;
+ break;
+ }
+ }
+
+ if (multifd_rdma) {
+ qemu_set_fd_handler(rdma->channel->fd,
+ rdma_accept_incoming_migration,
+ NULL, (void *)(intptr_t)multifd_rdma);
+ } else {
+ qemu_set_fd_handler(rdma->channel->fd, rdma_cm_poll_handler,
+ NULL, rdma);
+ }
} else {
qemu_set_fd_handler(rdma->channel->fd, rdma_cm_poll_handler,
NULL, rdma);
@@ -3744,7 +3773,7 @@ static void migration_rdma_process_incoming(QEMUFile *f, Error **errp)
mis->from_src_file = f;
qemu_file_set_blocking(f, false);
- start_migration = migrate_use_multifd();
+ start_migration = !migrate_use_multifd();
} else {
ioc = QIO_CHANNEL(getQIOChannel(f));
/* Multiple connections */
@@ -3847,6 +3876,30 @@ void rdma_start_incoming_migration(const char *host_port, Error **errp)
goto err;
}
+ if (migrate_use_multifd()) {
+ int thread_count;
+ int i;
+ int idx;
+ MultiFDRecvParams *multifd_recv_param;
+ thread_count = migrate_multifd_channels();
+ for (i = 0; i < thread_count; i++) {
+ if (get_multifd_recv_param(i, &multifd_recv_param) < 0) {
+ error_report("rdma: error getting multifd_recv_param(%d)", i);
+ goto err;
+ }
+
+ multifd_recv_param->rdma = qemu_rdma_data_init(host_port,
+ &local_err);
+ for (idx = 0; idx < RDMA_WRID_MAX; idx++) {
+ multifd_recv_param->rdma->wr_data[idx].control_len = 0;
+ multifd_recv_param->rdma->wr_data[idx].control_curr = NULL;
+ }
+ /* the CM channel and CM id is shared */
+ multifd_recv_param->rdma->channel = rdma->channel;
+ multifd_recv_param->rdma->listen_id = rdma->listen_id;
+ }
+ }
+
qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migration,
NULL, (void *)(intptr_t)rdma);
return;
@@ -3868,6 +3921,10 @@ void rdma_start_outgoing_migration(void *opaque,
goto err;
}
+ if (migrate_use_multifd()) {
+ rdma_host_port = g_strdup(host_port);
+ }
+
ret = qemu_rdma_source_init(rdma,
s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL], errp);
@@ -3918,44 +3975,38 @@ err:
g_free(rdma_return_path);
}
-void *multifd_rdma_recv_thread(void *opaque)
+int multifd_channel_rdma_connect(void *opaque)
{
- MultiFDRecvParams *p = opaque;
+ MultiFDSendParams *p = opaque;
+ Error *local_err = NULL;
+ int ret = 0;
- while (true) {
- qemu_mutex_lock(&p->mutex);
- if (p->quit) {
- qemu_mutex_unlock(&p->mutex);
- break;
- }
- qemu_mutex_unlock(&p->mutex);
- qemu_sem_wait(&p->sem_sync);
+ p->rdma = qemu_rdma_data_init(rdma_host_port, &local_err);
+ if (p->rdma == NULL) {
+ goto out;
}
- qemu_mutex_lock(&p->mutex);
- p->running = false;
- qemu_mutex_unlock(&p->mutex);
-
- return NULL;
-}
+ ret = qemu_rdma_source_init(p->rdma,
+ migrate_use_rdma_pin_all(),
+ &local_err);
+ if (ret) {
+ goto out;
+ }
-void *multifd_rdma_send_thread(void *opaque)
-{
- MultiFDSendParams *p = opaque;
+ ret = qemu_rdma_connect(p->rdma, &local_err);
+ if (ret) {
+ goto out;
+ }
- while (true) {
- qemu_mutex_lock(&p->mutex);
- if (p->quit) {
- qemu_mutex_unlock(&p->mutex);
- break;
- }
- qemu_mutex_unlock(&p->mutex);
- qemu_sem_wait(&p->sem);
+ p->file = qemu_fopen_rdma(p->rdma, "wb");
+ if (p->file == NULL) {
+ goto out;
}
- qemu_mutex_lock(&p->mutex);
- p->running = false;
- qemu_mutex_unlock(&p->mutex);
+out:
+ if (local_err) {
+ trace_multifd_send_error(p->id);
+ }
- return NULL;
+ return ret;
}
diff --git a/migration/rdma.h b/migration/rdma.h
index cb206c7004..ace6e5be90 100644
--- a/migration/rdma.h
+++ b/migration/rdma.h
@@ -263,9 +263,7 @@ struct QIOChannelRDMA {
bool blocking; /* XXX we don't actually honour this yet */
};
-
-void *multifd_rdma_recv_thread(void *opaque);
-void *multifd_rdma_send_thread(void *opaque);
+int multifd_channel_rdma_connect(void *opaque);
void rdma_start_outgoing_migration(void *opaque, const char *host_port,
Error **errp);
--
2.19.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH RFC 06/14] migration/rdma: Transmit initial packet
2020-02-13 9:37 [PATCH RFC 00/14] *** multifd for RDMA v2 *** Zhimin Feng
` (4 preceding siblings ...)
2020-02-13 9:37 ` [PATCH RFC 05/14] migration/rdma: Create the multifd channels for RDMA Zhimin Feng
@ 2020-02-13 9:37 ` Zhimin Feng
2020-02-14 16:31 ` Dr. David Alan Gilbert
2020-02-13 9:37 ` [PATCH RFC 07/14] migration/rdma: Export the 'qemu_rdma_registration_handle' and 'qemu_rdma_exchange_send' functions Zhimin Feng
` (8 subsequent siblings)
14 siblings, 1 reply; 22+ messages in thread
From: Zhimin Feng @ 2020-02-13 9:37 UTC (permalink / raw)
To: quintela, dgilbert, armbru, eblake
Cc: jemmy858585, Zhimin Feng, qemu-devel, zhang.zhanghailiang
Transmit initial packet through the multifd RDMA channels,
so that we can identify the multifd channels.
Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
---
migration/multifd.c | 33 +++++++++++++++++++++------------
migration/rdma.c | 2 ++
2 files changed, 23 insertions(+), 12 deletions(-)
diff --git a/migration/multifd.c b/migration/multifd.c
index acdfd3d5b3..a57d7a2eab 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -483,6 +483,13 @@ void multifd_send_sync_main(QEMUFile *f)
static void *multifd_rdma_send_thread(void *opaque)
{
MultiFDSendParams *p = opaque;
+ Error *local_err = NULL;
+
+ trace_multifd_send_thread_start(p->id);
+
+ if (multifd_send_initial_packet(p, &local_err) < 0) {
+ goto out;
+ }
while (true) {
qemu_mutex_lock(&p->mutex);
@@ -494,6 +501,12 @@ static void *multifd_rdma_send_thread(void *opaque)
qemu_sem_wait(&p->sem);
}
+out:
+ if (local_err) {
+ trace_multifd_send_error(p->id);
+ multifd_send_terminate_threads(local_err);
+ }
+
qemu_mutex_lock(&p->mutex);
p->running = false;
qemu_mutex_unlock(&p->mutex);
@@ -964,18 +977,14 @@ bool multifd_recv_new_channel(QIOChannel *ioc, Error **errp)
Error *local_err = NULL;
int id;
- if (migrate_use_rdma()) {
- id = multifd_recv_state->count;
- } else {
- id = multifd_recv_initial_packet(ioc, &local_err);
- if (id < 0) {
- multifd_recv_terminate_threads(local_err);
- error_propagate_prepend(errp, local_err,
- "failed to receive packet"
- " via multifd channel %d: ",
- atomic_read(&multifd_recv_state->count));
- return false;
- }
+ id = multifd_recv_initial_packet(ioc, &local_err);
+ if (id < 0) {
+ multifd_recv_terminate_threads(local_err);
+ error_propagate_prepend(errp, local_err,
+ "failed to receive packet"
+ " via multifd channel %d: ",
+ atomic_read(&multifd_recv_state->count));
+ return false;
}
trace_multifd_recv_new_channel(id);
diff --git a/migration/rdma.c b/migration/rdma.c
index 48615fcaad..2f1e69197f 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -4003,6 +4003,8 @@ int multifd_channel_rdma_connect(void *opaque)
goto out;
}
+ p->c = QIO_CHANNEL(getQIOChannel(p->file));
+
out:
if (local_err) {
trace_multifd_send_error(p->id);
--
2.19.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* Re: [PATCH RFC 06/14] migration/rdma: Transmit initial packet
2020-02-13 9:37 ` [PATCH RFC 06/14] migration/rdma: Transmit initial packet Zhimin Feng
@ 2020-02-14 16:31 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 22+ messages in thread
From: Dr. David Alan Gilbert @ 2020-02-14 16:31 UTC (permalink / raw)
To: Zhimin Feng
Cc: zhang.zhanghailiang, quintela, qemu-devel, armbru, jemmy858585
* Zhimin Feng (fengzhimin1@huawei.com) wrote:
> Transmit initial packet through the multifd RDMA channels,
> so that we can identify the multifd channels.
>
> Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
> ---
> migration/multifd.c | 33 +++++++++++++++++++++------------
> migration/rdma.c | 2 ++
> 2 files changed, 23 insertions(+), 12 deletions(-)
>
> diff --git a/migration/multifd.c b/migration/multifd.c
> index acdfd3d5b3..a57d7a2eab 100644
> --- a/migration/multifd.c
> +++ b/migration/multifd.c
> @@ -483,6 +483,13 @@ void multifd_send_sync_main(QEMUFile *f)
> static void *multifd_rdma_send_thread(void *opaque)
> {
> MultiFDSendParams *p = opaque;
> + Error *local_err = NULL;
> +
> + trace_multifd_send_thread_start(p->id);
> +
> + if (multifd_send_initial_packet(p, &local_err) < 0) {
> + goto out;
> + }
>
> while (true) {
> qemu_mutex_lock(&p->mutex);
> @@ -494,6 +501,12 @@ static void *multifd_rdma_send_thread(void *opaque)
> qemu_sem_wait(&p->sem);
> }
>
> +out:
> + if (local_err) {
> + trace_multifd_send_error(p->id);
> + multifd_send_terminate_threads(local_err);
> + }
> +
> qemu_mutex_lock(&p->mutex);
> p->running = false;
> qemu_mutex_unlock(&p->mutex);
> @@ -964,18 +977,14 @@ bool multifd_recv_new_channel(QIOChannel *ioc, Error **errp)
> Error *local_err = NULL;
> int id;
>
> - if (migrate_use_rdma()) {
> - id = multifd_recv_state->count;
> - } else {
> - id = multifd_recv_initial_packet(ioc, &local_err);
> - if (id < 0) {
> - multifd_recv_terminate_threads(local_err);
> - error_propagate_prepend(errp, local_err,
> - "failed to receive packet"
> - " via multifd channel %d: ",
> - atomic_read(&multifd_recv_state->count));
> - return false;
> - }
> + id = multifd_recv_initial_packet(ioc, &local_err);
> + if (id < 0) {
> + multifd_recv_terminate_threads(local_err);
> + error_propagate_prepend(errp, local_err,
> + "failed to receive packet"
> + " via multifd channel %d: ",
> + atomic_read(&multifd_recv_state->count));
> + return false;
I'm confused; hasn't that just undone part of patch 5 ?
Dave
> }
>
> trace_multifd_recv_new_channel(id);
> diff --git a/migration/rdma.c b/migration/rdma.c
> index 48615fcaad..2f1e69197f 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -4003,6 +4003,8 @@ int multifd_channel_rdma_connect(void *opaque)
> goto out;
> }
>
> + p->c = QIO_CHANNEL(getQIOChannel(p->file));
> +
> out:
> if (local_err) {
> trace_multifd_send_error(p->id);
> --
> 2.19.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH RFC 07/14] migration/rdma: Export the 'qemu_rdma_registration_handle' and 'qemu_rdma_exchange_send' functions
2020-02-13 9:37 [PATCH RFC 00/14] *** multifd for RDMA v2 *** Zhimin Feng
` (5 preceding siblings ...)
2020-02-13 9:37 ` [PATCH RFC 06/14] migration/rdma: Transmit initial packet Zhimin Feng
@ 2020-02-13 9:37 ` Zhimin Feng
2020-02-13 9:37 ` [PATCH RFC 08/14] migration/rdma: Add the function for dynamic page registration Zhimin Feng
` (7 subsequent siblings)
14 siblings, 0 replies; 22+ messages in thread
From: Zhimin Feng @ 2020-02-13 9:37 UTC (permalink / raw)
To: quintela, dgilbert, armbru, eblake
Cc: jemmy858585, Zhimin Feng, qemu-devel, zhang.zhanghailiang
Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
---
migration/rdma.c | 25 +++++--------------------
migration/rdma.h | 16 ++++++++++++++++
2 files changed, 21 insertions(+), 20 deletions(-)
diff --git a/migration/rdma.c b/migration/rdma.c
index 2f1e69197f..23f7f525f4 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -170,17 +170,6 @@ static void network_to_dest_block(RDMADestBlock *db)
db->remote_rkey = ntohl(db->remote_rkey);
}
-/*
- * Main structure for IB Send/Recv control messages.
- * This gets prepended at the beginning of every Send/Recv.
- */
-typedef struct QEMU_PACKED {
- uint32_t len; /* Total length of data portion */
- uint32_t type; /* which control command to perform */
- uint32_t repeat; /* number of commands in data portion of same type */
- uint32_t padding;
-} RDMAControlHeader;
-
static void control_to_network(RDMAControlHeader *control)
{
control->type = htonl(control->type);
@@ -289,10 +278,6 @@ static void network_to_result(RDMARegisterResult *result)
};
const char *print_wrid(int wrid);
-static int qemu_rdma_exchange_send(RDMAContext *rdma, RDMAControlHeader *head,
- uint8_t *data, RDMAControlHeader *resp,
- int *resp_idx,
- int (*callback)(RDMAContext *rdma));
static inline uint64_t ram_chunk_index(const uint8_t *start,
const uint8_t *host)
@@ -1590,10 +1575,10 @@ static void qemu_rdma_move_header(RDMAContext *rdma, int idx,
* to perform an *additional* exchange of message just to provide a response by
* instead piggy-backing on the acknowledgement.
*/
-static int qemu_rdma_exchange_send(RDMAContext *rdma, RDMAControlHeader *head,
- uint8_t *data, RDMAControlHeader *resp,
- int *resp_idx,
- int (*callback)(RDMAContext *rdma))
+int qemu_rdma_exchange_send(RDMAContext *rdma, RDMAControlHeader *head,
+ uint8_t *data, RDMAControlHeader *resp,
+ int *resp_idx,
+ int (*callback)(RDMAContext *rdma))
{
int ret = 0;
@@ -3210,7 +3195,7 @@ static int dest_ram_sort_func(const void *a, const void *b)
*
* Keep doing this until the source tells us to stop.
*/
-static int qemu_rdma_registration_handle(QEMUFile *f, void *opaque)
+int qemu_rdma_registration_handle(QEMUFile *f, void *opaque)
{
RDMAControlHeader reg_resp = { .len = sizeof(RDMARegisterResult),
.type = RDMA_CONTROL_REGISTER_RESULT,
diff --git a/migration/rdma.h b/migration/rdma.h
index ace6e5be90..8e1a6edf57 100644
--- a/migration/rdma.h
+++ b/migration/rdma.h
@@ -144,6 +144,17 @@ typedef struct QEMU_PACKED RDMADestBlock {
uint32_t padding;
} RDMADestBlock;
+/*
+ * Main structure for IB Send/Recv control messages.
+ * This gets prepended at the beginning of every Send/Recv.
+ */
+typedef struct QEMU_PACKED {
+ uint32_t len; /* Total length of data portion */
+ uint32_t type; /* which control command to perform */
+ uint32_t repeat; /* number of commands in data portion of same type */
+ uint32_t padding;
+} RDMAControlHeader;
+
/*
* Virtual address of the above structures used for transmitting
* the RAMBlock descriptions at connection-time.
@@ -264,6 +275,11 @@ struct QIOChannelRDMA {
};
int multifd_channel_rdma_connect(void *opaque);
+int qemu_rdma_registration_handle(QEMUFile *f, void *opaque);
+int qemu_rdma_exchange_send(RDMAContext *rdma, RDMAControlHeader *head,
+ uint8_t *data, RDMAControlHeader *resp,
+ int *resp_idx,
+ int (*callback)(RDMAContext *rdma));
void rdma_start_outgoing_migration(void *opaque, const char *host_port,
Error **errp);
--
2.19.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH RFC 08/14] migration/rdma: Add the function for dynamic page registration
2020-02-13 9:37 [PATCH RFC 00/14] *** multifd for RDMA v2 *** Zhimin Feng
` (6 preceding siblings ...)
2020-02-13 9:37 ` [PATCH RFC 07/14] migration/rdma: Export the 'qemu_rdma_registration_handle' and 'qemu_rdma_exchange_send' functions Zhimin Feng
@ 2020-02-13 9:37 ` Zhimin Feng
2020-02-13 9:37 ` [PATCH RFC 09/14] migration/rdma: register memory for multifd RDMA channels Zhimin Feng
` (6 subsequent siblings)
14 siblings, 0 replies; 22+ messages in thread
From: Zhimin Feng @ 2020-02-13 9:37 UTC (permalink / raw)
To: quintela, dgilbert, armbru, eblake
Cc: jemmy858585, Zhimin Feng, qemu-devel, zhang.zhanghailiang
Add the 'qemu_rdma_registration' function,
multifd send threads call it to register memory.
Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
---
migration/rdma.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++++
migration/rdma.h | 1 +
2 files changed, 52 insertions(+)
diff --git a/migration/rdma.c b/migration/rdma.c
index 23f7f525f4..19a238be30 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -3471,6 +3471,57 @@ out:
return ret;
}
+/*
+ * Dynamic page registrations for multifd RDMA threads.
+ */
+int qemu_rdma_registration(void *opaque)
+{
+ RDMAContext *rdma = opaque;
+ RDMAControlHeader resp = {.type = RDMA_CONTROL_RAM_BLOCKS_RESULT };
+ RDMALocalBlocks *local = &rdma->local_ram_blocks;
+ int reg_result_idx, i, nb_dest_blocks;
+ RDMAControlHeader head = { .len = 0, .repeat = 1 };
+ int ret = 0;
+
+ head.type = RDMA_CONTROL_RAM_BLOCKS_REQUEST;
+
+ ret = qemu_rdma_exchange_send(rdma, &head, NULL, &resp,
+ ®_result_idx, rdma->pin_all ?
+ qemu_rdma_reg_whole_ram_blocks : NULL);
+ if (ret < 0) {
+ goto out;
+ }
+
+ nb_dest_blocks = resp.len / sizeof(RDMADestBlock);
+
+ if (local->nb_blocks != nb_dest_blocks) {
+ rdma->error_state = -EINVAL;
+ ret = -1;
+ goto out;
+ }
+
+ qemu_rdma_move_header(rdma, reg_result_idx, &resp);
+ memcpy(rdma->dest_blocks,
+ rdma->wr_data[reg_result_idx].control_curr, resp.len);
+
+ for (i = 0; i < nb_dest_blocks; i++) {
+ network_to_dest_block(&rdma->dest_blocks[i]);
+
+ /* We require that the blocks are in the same order */
+ if (rdma->dest_blocks[i].length != local->block[i].length) {
+ rdma->error_state = -EINVAL;
+ ret = -1;
+ goto out;
+ }
+ local->block[i].remote_host_addr =
+ rdma->dest_blocks[i].remote_host_addr;
+ local->block[i].remote_rkey = rdma->dest_blocks[i].remote_rkey;
+ }
+
+out:
+ return ret;
+}
+
/* Destination:
* Called via a ram_control_load_hook during the initial RAM load section which
* lists the RAMBlocks by name. This lets us know the order of the RAMBlocks
diff --git a/migration/rdma.h b/migration/rdma.h
index 8e1a6edf57..86c89bdd1f 100644
--- a/migration/rdma.h
+++ b/migration/rdma.h
@@ -280,6 +280,7 @@ int qemu_rdma_exchange_send(RDMAContext *rdma, RDMAControlHeader *head,
uint8_t *data, RDMAControlHeader *resp,
int *resp_idx,
int (*callback)(RDMAContext *rdma));
+int qemu_rdma_registration(void *opaque);
void rdma_start_outgoing_migration(void *opaque, const char *host_port,
Error **errp);
--
2.19.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH RFC 09/14] migration/rdma: register memory for multifd RDMA channels
2020-02-13 9:37 [PATCH RFC 00/14] *** multifd for RDMA v2 *** Zhimin Feng
` (7 preceding siblings ...)
2020-02-13 9:37 ` [PATCH RFC 08/14] migration/rdma: Add the function for dynamic page registration Zhimin Feng
@ 2020-02-13 9:37 ` Zhimin Feng
2020-02-13 9:37 ` [PATCH RFC 10/14] migration/rdma: Wait for all multifd to complete registration Zhimin Feng
` (5 subsequent siblings)
14 siblings, 0 replies; 22+ messages in thread
From: Zhimin Feng @ 2020-02-13 9:37 UTC (permalink / raw)
To: quintela, dgilbert, armbru, eblake
Cc: jemmy858585, Zhimin Feng, qemu-devel, zhang.zhanghailiang
register memory for multifd RDMA channels and transmit the destination
the keys to source to use including the virtual addresses.
Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
---
migration/multifd.c | 34 +++++++++++++++++++++++++++++---
migration/rdma.c | 48 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 79 insertions(+), 3 deletions(-)
diff --git a/migration/multifd.c b/migration/multifd.c
index a57d7a2eab..4ae25fc88f 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -388,7 +388,11 @@ static void multifd_send_terminate_threads(Error *err)
qemu_mutex_lock(&p->mutex);
p->quit = true;
- qemu_sem_post(&p->sem);
+ if (migrate_use_rdma()) {
+ qemu_sem_post(&p->sem_sync);
+ } else {
+ qemu_sem_post(&p->sem);
+ }
qemu_mutex_unlock(&p->mutex);
}
}
@@ -484,6 +488,8 @@ static void *multifd_rdma_send_thread(void *opaque)
{
MultiFDSendParams *p = opaque;
Error *local_err = NULL;
+ int ret = 0;
+ RDMAControlHeader head = { .len = 0, .repeat = 1 };
trace_multifd_send_thread_start(p->id);
@@ -491,14 +497,28 @@ static void *multifd_rdma_send_thread(void *opaque)
goto out;
}
+ /* wait for semaphore notification to register memory */
+ qemu_sem_wait(&p->sem_sync);
+ if (qemu_rdma_registration(p->rdma) < 0) {
+ goto out;
+ }
+
while (true) {
+ qemu_sem_wait(&p->sem_sync);
+
qemu_mutex_lock(&p->mutex);
if (p->quit) {
qemu_mutex_unlock(&p->mutex);
break;
}
qemu_mutex_unlock(&p->mutex);
- qemu_sem_wait(&p->sem);
+
+ /* Send FINISHED to the destination */
+ head.type = RDMA_CONTROL_REGISTER_FINISHED;
+ ret = qemu_rdma_exchange_send(p->rdma, &head, NULL, NULL, NULL, NULL);
+ if (ret < 0) {
+ return NULL;
+ }
}
out:
@@ -836,15 +856,23 @@ void multifd_recv_sync_main(void)
static void *multifd_rdma_recv_thread(void *opaque)
{
MultiFDRecvParams *p = opaque;
+ int ret = 0;
while (true) {
+ qemu_sem_wait(&p->sem_sync);
+
qemu_mutex_lock(&p->mutex);
if (p->quit) {
qemu_mutex_unlock(&p->mutex);
break;
}
qemu_mutex_unlock(&p->mutex);
- qemu_sem_wait(&p->sem_sync);
+
+ ret = qemu_rdma_registration_handle(p->file, p->c);
+ if (ret < 0) {
+ qemu_file_set_error(p->file, ret);
+ break;
+ }
}
qemu_mutex_lock(&p->mutex);
diff --git a/migration/rdma.c b/migration/rdma.c
index 19a238be30..5de3a29712 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -3570,6 +3570,19 @@ static int rdma_load_hook(QEMUFile *f, void *opaque, uint64_t flags, void *data)
return rdma_block_notification_handle(opaque, data);
case RAM_CONTROL_HOOK:
+ if (migrate_use_multifd()) {
+ int i;
+ MultiFDRecvParams *multifd_recv_param = NULL;
+ int thread_count = migrate_multifd_channels();
+ /* Inform dest recv_thread to poll */
+ for (i = 0; i < thread_count; i++) {
+ if (get_multifd_recv_param(i, &multifd_recv_param)) {
+ return -1;
+ }
+ qemu_sem_post(&multifd_recv_param->sem_sync);
+ }
+ }
+
return qemu_rdma_registration_handle(f, opaque);
default:
@@ -3643,6 +3656,25 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque,
head.type = RDMA_CONTROL_RAM_BLOCKS_REQUEST;
trace_qemu_rdma_registration_stop_ram();
+ if (migrate_use_multifd()) {
+ /*
+ * Inform the multifd channels to register memory
+ */
+ int i;
+ int thread_count = migrate_multifd_channels();
+ MultiFDSendParams *multifd_send_param = NULL;
+ for (i = 0; i < thread_count; i++) {
+ ret = get_multifd_send_param(i, &multifd_send_param);
+ if (ret) {
+ ERROR(errp, "rdma: error getting"
+ "multifd_send_param(%d)", i);
+ return ret;
+ }
+
+ qemu_sem_post(&multifd_send_param->sem_sync);
+ }
+ }
+
/*
* Make sure that we parallelize the pinning on both sides.
* For very large guests, doing this serially takes a really
@@ -3708,6 +3740,22 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque,
head.type = RDMA_CONTROL_REGISTER_FINISHED;
ret = qemu_rdma_exchange_send(rdma, &head, NULL, NULL, NULL, NULL);
+ if (migrate_use_multifd()) {
+ /* Inform src send_thread to send FINISHED signal */
+ int i;
+ int thread_count = migrate_multifd_channels();
+ MultiFDSendParams *multifd_send_param = NULL;
+ for (i = 0; i < thread_count; i++) {
+ ret = get_multifd_send_param(i, &multifd_send_param);
+ if (ret) {
+ ERROR(errp, "rdma: error getting multifd_send_param(%d)", i);
+ return ret;
+ }
+
+ qemu_sem_post(&multifd_send_param->sem_sync);
+ }
+ }
+
if (ret < 0) {
goto err;
}
--
2.19.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH RFC 10/14] migration/rdma: Wait for all multifd to complete registration
2020-02-13 9:37 [PATCH RFC 00/14] *** multifd for RDMA v2 *** Zhimin Feng
` (8 preceding siblings ...)
2020-02-13 9:37 ` [PATCH RFC 09/14] migration/rdma: register memory for multifd RDMA channels Zhimin Feng
@ 2020-02-13 9:37 ` Zhimin Feng
2020-02-13 9:37 ` [PATCH RFC 11/14] migration/rdma: use multifd to migrate VM for rdma-pin-all mode Zhimin Feng
` (4 subsequent siblings)
14 siblings, 0 replies; 22+ messages in thread
From: Zhimin Feng @ 2020-02-13 9:37 UTC (permalink / raw)
To: quintela, dgilbert, armbru, eblake
Cc: jemmy858585, Zhimin Feng, qemu-devel, zhang.zhanghailiang
Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
---
migration/multifd.c | 6 ++++++
migration/rdma.c | 17 +++++++++++++++++
2 files changed, 23 insertions(+)
diff --git a/migration/multifd.c b/migration/multifd.c
index 4ae25fc88f..c986d4c247 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -389,6 +389,7 @@ static void multifd_send_terminate_threads(Error *err)
qemu_mutex_lock(&p->mutex);
p->quit = true;
if (migrate_use_rdma()) {
+ qemu_sem_post(&p->sem);
qemu_sem_post(&p->sem_sync);
} else {
qemu_sem_post(&p->sem);
@@ -502,6 +503,11 @@ static void *multifd_rdma_send_thread(void *opaque)
if (qemu_rdma_registration(p->rdma) < 0) {
goto out;
}
+ /*
+ * Inform the main RDMA thread to run when multifd
+ * RDMA thread have completed registration.
+ */
+ qemu_sem_post(&p->sem);
while (true) {
qemu_sem_wait(&p->sem_sync);
diff --git a/migration/rdma.c b/migration/rdma.c
index 5de3a29712..4c48e9832c 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -3733,6 +3733,23 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque,
rdma->dest_blocks[i].remote_host_addr;
local->block[i].remote_rkey = rdma->dest_blocks[i].remote_rkey;
}
+
+ /* Wait for all multifd channels to complete registration */
+ if (migrate_use_multifd()) {
+ int i;
+ int thread_count = migrate_multifd_channels();
+ MultiFDSendParams *multifd_send_param = NULL;
+ for (i = 0; i < thread_count; i++) {
+ ret = get_multifd_send_param(i, &multifd_send_param);
+ if (ret) {
+ ERROR(errp, "rdma: error"
+ "getting multifd_send_param(%d)", i);
+ return ret;
+ }
+
+ qemu_sem_wait(&multifd_send_param->sem);
+ }
+ }
}
trace_qemu_rdma_registration_stop(flags);
--
2.19.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH RFC 11/14] migration/rdma: use multifd to migrate VM for rdma-pin-all mode
2020-02-13 9:37 [PATCH RFC 00/14] *** multifd for RDMA v2 *** Zhimin Feng
` (9 preceding siblings ...)
2020-02-13 9:37 ` [PATCH RFC 10/14] migration/rdma: Wait for all multifd to complete registration Zhimin Feng
@ 2020-02-13 9:37 ` Zhimin Feng
2020-02-13 9:37 ` [PATCH RFC 12/14] migration/rdma: use multifd to migrate VM for NOT " Zhimin Feng
` (3 subsequent siblings)
14 siblings, 0 replies; 22+ messages in thread
From: Zhimin Feng @ 2020-02-13 9:37 UTC (permalink / raw)
To: quintela, dgilbert, armbru, eblake
Cc: jemmy858585, Zhimin Feng, qemu-devel, zhang.zhanghailiang
Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
---
migration/multifd.c | 15 ++++++++++++
migration/rdma.c | 58 +++++++++++++++++++++++++++++++++++++++++----
migration/rdma.h | 2 ++
3 files changed, 70 insertions(+), 5 deletions(-)
diff --git a/migration/multifd.c b/migration/multifd.c
index c986d4c247..ba5e0b11d0 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -519,12 +519,27 @@ static void *multifd_rdma_send_thread(void *opaque)
}
qemu_mutex_unlock(&p->mutex);
+ /* To complete polling(CQE) */
+ while (p->rdma->nb_sent) {
+ ret = qemu_rdma_block_for_wrid(p->rdma, RDMA_WRID_RDMA_WRITE, NULL);
+ if (ret < 0) {
+ error_report("multifd RDMA migration: "
+ "complete polling error!");
+ return NULL;
+ }
+ }
+
/* Send FINISHED to the destination */
head.type = RDMA_CONTROL_REGISTER_FINISHED;
ret = qemu_rdma_exchange_send(p->rdma, &head, NULL, NULL, NULL, NULL);
if (ret < 0) {
+ error_report("multifd RDMA migration: "
+ "receiving remote info!");
return NULL;
}
+
+ /* sync main thread */
+ qemu_sem_post(&p->sem);
}
out:
diff --git a/migration/rdma.c b/migration/rdma.c
index 4c48e9832c..873c17dc03 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -96,6 +96,23 @@ static const char *wrid_desc[] = {
static const char *rdma_host_port;
+/*
+ * index of current RDMA channel for multifd
+ */
+static int current_RDMA_index;
+
+/*
+ * Get the multifd RDMA channel used to send data.
+ */
+static int get_multifd_RDMA_channel(void)
+{
+ int thread_count = migrate_multifd_channels();
+ current_RDMA_index++;
+ current_RDMA_index %= thread_count;
+
+ return current_RDMA_index;
+}
+
/*
* Negotiate RDMA capabilities during connection-setup time.
*/
@@ -1328,8 +1345,8 @@ static int qemu_rdma_wait_comp_channel(RDMAContext *rdma)
* completions only need to be recorded, but do not actually
* need further processing.
*/
-static int qemu_rdma_block_for_wrid(RDMAContext *rdma, int wrid_requested,
- uint32_t *byte_len)
+int qemu_rdma_block_for_wrid(RDMAContext *rdma, int wrid_requested,
+ uint32_t *byte_len)
{
int num_cq_events = 0, ret = 0;
struct ibv_cq *cq;
@@ -1731,6 +1748,20 @@ static int qemu_rdma_write_one(QEMUFile *f, RDMAContext *rdma,
.repeat = 1,
};
+ /* use multifd to send data */
+ if (migrate_use_multifd() && migrate_use_rdma_pin_all()) {
+ int channel = get_multifd_RDMA_channel();
+ int ret = 0;
+ MultiFDSendParams *multifd_send_param = NULL;
+ ret = get_multifd_send_param(channel, &multifd_send_param);
+ if (ret) {
+ error_report("rdma: error getting multifd_send_param(%d)", channel);
+ return -EINVAL;
+ }
+ rdma = multifd_send_param->rdma;
+ block = &(rdma->local_ram_blocks.block[current_index]);
+ }
+
retry:
sge.addr = (uintptr_t)(block->local_host_addr +
(current_addr - block->offset));
@@ -1948,8 +1979,21 @@ static int qemu_rdma_write_flush(QEMUFile *f, RDMAContext *rdma)
}
if (ret == 0) {
- rdma->nb_sent++;
- trace_qemu_rdma_write_flush(rdma->nb_sent);
+ if (migrate_use_multifd() && migrate_use_rdma_pin_all()) {
+ /* The multifd RDMA threads send data */
+ MultiFDSendParams *multifd_send_param = NULL;
+ ret = get_multifd_send_param(current_RDMA_index,
+ &multifd_send_param);
+ if (ret) {
+ error_report("rdma: error getting multifd_send_param(%d)",
+ current_RDMA_index);
+ return ret;
+ }
+ multifd_send_param->rdma->nb_sent++;
+ } else {
+ rdma->nb_sent++;
+ trace_qemu_rdma_write_flush(rdma->nb_sent);
+ }
}
rdma->current_length = 0;
@@ -3758,7 +3802,10 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque,
ret = qemu_rdma_exchange_send(rdma, &head, NULL, NULL, NULL, NULL);
if (migrate_use_multifd()) {
- /* Inform src send_thread to send FINISHED signal */
+ /*
+ * Inform src send_thread to send FINISHED signal.
+ * Wait for multifd RDMA send threads to poll the CQE.
+ */
int i;
int thread_count = migrate_multifd_channels();
MultiFDSendParams *multifd_send_param = NULL;
@@ -3770,6 +3817,7 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque,
}
qemu_sem_post(&multifd_send_param->sem_sync);
+ qemu_sem_wait(&multifd_send_param->sem);
}
}
diff --git a/migration/rdma.h b/migration/rdma.h
index 86c89bdd1f..7dc3895698 100644
--- a/migration/rdma.h
+++ b/migration/rdma.h
@@ -281,6 +281,8 @@ int qemu_rdma_exchange_send(RDMAContext *rdma, RDMAControlHeader *head,
int *resp_idx,
int (*callback)(RDMAContext *rdma));
int qemu_rdma_registration(void *opaque);
+int qemu_rdma_block_for_wrid(RDMAContext *rdma, int wrid_requested,
+ uint32_t *byte_len);
void rdma_start_outgoing_migration(void *opaque, const char *host_port,
Error **errp);
--
2.19.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH RFC 12/14] migration/rdma: use multifd to migrate VM for NOT rdma-pin-all mode
2020-02-13 9:37 [PATCH RFC 00/14] *** multifd for RDMA v2 *** Zhimin Feng
` (10 preceding siblings ...)
2020-02-13 9:37 ` [PATCH RFC 11/14] migration/rdma: use multifd to migrate VM for rdma-pin-all mode Zhimin Feng
@ 2020-02-13 9:37 ` Zhimin Feng
2020-02-13 9:37 ` [PATCH RFC 13/14] migration/rdma: only register the memory for multifd channels Zhimin Feng
` (2 subsequent siblings)
14 siblings, 0 replies; 22+ messages in thread
From: Zhimin Feng @ 2020-02-13 9:37 UTC (permalink / raw)
To: quintela, dgilbert, armbru, eblake
Cc: jemmy858585, Zhimin Feng, qemu-devel, zhang.zhanghailiang
Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
---
migration/rdma.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/migration/rdma.c b/migration/rdma.c
index 873c17dc03..eb7c2edbe7 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -1749,7 +1749,7 @@ static int qemu_rdma_write_one(QEMUFile *f, RDMAContext *rdma,
};
/* use multifd to send data */
- if (migrate_use_multifd() && migrate_use_rdma_pin_all()) {
+ if (migrate_use_multifd()) {
int channel = get_multifd_RDMA_channel();
int ret = 0;
MultiFDSendParams *multifd_send_param = NULL;
@@ -1979,7 +1979,7 @@ static int qemu_rdma_write_flush(QEMUFile *f, RDMAContext *rdma)
}
if (ret == 0) {
- if (migrate_use_multifd() && migrate_use_rdma_pin_all()) {
+ if (migrate_use_multifd()) {
/* The multifd RDMA threads send data */
MultiFDSendParams *multifd_send_param = NULL;
ret = get_multifd_send_param(current_RDMA_index,
--
2.19.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH RFC 13/14] migration/rdma: only register the memory for multifd channels
2020-02-13 9:37 [PATCH RFC 00/14] *** multifd for RDMA v2 *** Zhimin Feng
` (11 preceding siblings ...)
2020-02-13 9:37 ` [PATCH RFC 12/14] migration/rdma: use multifd to migrate VM for NOT " Zhimin Feng
@ 2020-02-13 9:37 ` Zhimin Feng
2020-02-13 9:37 ` [PATCH RFC 14/14] migration/rdma: RDMA cleanup for multifd migration Zhimin Feng
2020-02-13 10:14 ` [PATCH RFC 00/14] *** multifd for RDMA v2 *** no-reply
14 siblings, 0 replies; 22+ messages in thread
From: Zhimin Feng @ 2020-02-13 9:37 UTC (permalink / raw)
To: quintela, dgilbert, armbru, eblake
Cc: jemmy858585, Zhimin Feng, qemu-devel, zhang.zhanghailiang
All data is sent by multifd Channels, so we only register its for
multifd channels and main channel don't register its.
Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
---
migration/rdma.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/migration/rdma.c b/migration/rdma.c
index eb7c2edbe7..b7b56c0493 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -3717,6 +3717,12 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque,
qemu_sem_post(&multifd_send_param->sem_sync);
}
+
+ /*
+ * Use multifd to migrate, we only register memory for
+ * multifd RDMA channel and main channel don't register it.
+ */
+ goto wait_reg_complete;
}
/*
@@ -3778,6 +3784,7 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque,
local->block[i].remote_rkey = rdma->dest_blocks[i].remote_rkey;
}
+wait_reg_complete:
/* Wait for all multifd channels to complete registration */
if (migrate_use_multifd()) {
int i;
--
2.19.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* [PATCH RFC 14/14] migration/rdma: RDMA cleanup for multifd migration
2020-02-13 9:37 [PATCH RFC 00/14] *** multifd for RDMA v2 *** Zhimin Feng
` (12 preceding siblings ...)
2020-02-13 9:37 ` [PATCH RFC 13/14] migration/rdma: only register the memory for multifd channels Zhimin Feng
@ 2020-02-13 9:37 ` Zhimin Feng
2020-02-13 10:14 ` [PATCH RFC 00/14] *** multifd for RDMA v2 *** no-reply
14 siblings, 0 replies; 22+ messages in thread
From: Zhimin Feng @ 2020-02-13 9:37 UTC (permalink / raw)
To: quintela, dgilbert, armbru, eblake
Cc: jemmy858585, Zhimin Feng, qemu-devel, zhang.zhanghailiang
Signed-off-by: Zhimin Feng <fengzhimin1@huawei.com>
---
migration/multifd.c | 6 ++++++
migration/rdma.c | 5 ++---
migration/rdma.h | 1 +
3 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/migration/multifd.c b/migration/multifd.c
index ba5e0b11d0..886c8e1271 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -429,6 +429,9 @@ void multifd_save_cleanup(void)
g_free(p->packet);
p->packet = NULL;
if (migrate_use_rdma()) {
+ p->rdma->listen_id = NULL;
+ p->rdma->channel = NULL;
+ qemu_rdma_cleanup(p->rdma);
g_free(p->rdma);
}
}
@@ -835,6 +838,9 @@ int multifd_load_cleanup(Error **errp)
g_free(p->packet);
p->packet = NULL;
if (migrate_use_rdma()) {
+ p->rdma->listen_id = NULL;
+ p->rdma->channel = NULL;
+ qemu_rdma_cleanup(p->rdma);
g_free(p->rdma);
}
}
diff --git a/migration/rdma.c b/migration/rdma.c
index b7b56c0493..0a48713d03 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -2096,11 +2096,11 @@ static int qemu_rdma_write(QEMUFile *f, RDMAContext *rdma,
return 0;
}
-static void qemu_rdma_cleanup(RDMAContext *rdma)
+void qemu_rdma_cleanup(RDMAContext *rdma)
{
int idx;
- if (rdma->cm_id && rdma->connected) {
+ if (rdma->channel && rdma->cm_id && rdma->connected) {
if ((rdma->error_state ||
migrate_get_current()->state == MIGRATION_STATUS_CANCELLING) &&
!rdma->received_error) {
@@ -2181,7 +2181,6 @@ static void qemu_rdma_cleanup(RDMAContext *rdma)
rdma->host = NULL;
}
-
static int qemu_rdma_source_init(RDMAContext *rdma, bool pin_all, Error **errp)
{
int ret, idx;
diff --git a/migration/rdma.h b/migration/rdma.h
index 7dc3895698..b78f79ddc2 100644
--- a/migration/rdma.h
+++ b/migration/rdma.h
@@ -283,6 +283,7 @@ int qemu_rdma_exchange_send(RDMAContext *rdma, RDMAControlHeader *head,
int qemu_rdma_registration(void *opaque);
int qemu_rdma_block_for_wrid(RDMAContext *rdma, int wrid_requested,
uint32_t *byte_len);
+void qemu_rdma_cleanup(RDMAContext *rdma);
void rdma_start_outgoing_migration(void *opaque, const char *host_port,
Error **errp);
--
2.19.1
^ permalink raw reply related [flat|nested] 22+ messages in thread* Re: [PATCH RFC 00/14] *** multifd for RDMA v2 ***
2020-02-13 9:37 [PATCH RFC 00/14] *** multifd for RDMA v2 *** Zhimin Feng
` (13 preceding siblings ...)
2020-02-13 9:37 ` [PATCH RFC 14/14] migration/rdma: RDMA cleanup for multifd migration Zhimin Feng
@ 2020-02-13 10:14 ` no-reply
2020-02-14 13:23 ` Dr. David Alan Gilbert
14 siblings, 1 reply; 22+ messages in thread
From: no-reply @ 2020-02-13 10:14 UTC (permalink / raw)
To: fengzhimin1
Cc: jemmy858585, quintela, qemu-devel, fengzhimin1, armbru, dgilbert,
zhang.zhanghailiang
Patchew URL: https://patchew.org/QEMU/20200213093755.370-1-fengzhimin1@huawei.com/
Hi,
This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.
=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===
/tmp/qemu-test/src/migration/multifd.c:663: undefined reference to `multifd_channel_rdma_connect'
../migration/multifd.o: In function `multifd_load_cleanup':
/tmp/qemu-test/src/migration/multifd.c:843: undefined reference to `qemu_rdma_cleanup'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:206: qemu-system-x86_64w.exe] Error 1
make: *** [Makefile:497: x86_64-softmmu/all] Error 2
make: *** Waiting for unfinished jobs....
../migration/multifd.o: In function `multifd_rdma_recv_thread':
/tmp/qemu-test/src/migration/multifd.c:898: undefined reference to `qemu_rdma_registration_handle'
---
/tmp/qemu-test/src/migration/multifd.c:663: undefined reference to `multifd_channel_rdma_connect'
../migration/multifd.o: In function `multifd_load_cleanup':
/tmp/qemu-test/src/migration/multifd.c:843: undefined reference to `qemu_rdma_cleanup'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:206: qemu-system-aarch64w.exe] Error 1
make: *** [Makefile:497: aarch64-softmmu/all] Error 2
Traceback (most recent call last):
File "./tests/docker/docker.py", line 664, in <module>
sys.exit(main())
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=88473d634d6543ea992045cbe9a806e1', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-yiq7aevf/src/docker-src.2020-02-13-05.11.35.1374:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=88473d634d6543ea992045cbe9a806e1
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-yiq7aevf/src'
make: *** [docker-run-test-mingw@fedora] Error 2
real 2m35.791s
user 0m7.717s
The full log is available at
http://patchew.org/logs/20200213093755.370-1-fengzhimin1@huawei.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [PATCH RFC 00/14] *** multifd for RDMA v2 ***
2020-02-13 10:14 ` [PATCH RFC 00/14] *** multifd for RDMA v2 *** no-reply
@ 2020-02-14 13:23 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 22+ messages in thread
From: Dr. David Alan Gilbert @ 2020-02-14 13:23 UTC (permalink / raw)
To: fengzhimin1, qemu-devel
Cc: zhang.zhanghailiang, jemmy858585, armbru, quintela
Make sure that it compiles OK with RDMA compiled out; I think this is a
windows cross build that's failing, but more generally even a Linux box
with no-RDMA libraries.
Dave
* no-reply@patchew.org (no-reply@patchew.org) wrote:
> Patchew URL: https://patchew.org/QEMU/20200213093755.370-1-fengzhimin1@huawei.com/
>
>
>
> Hi,
>
> This series failed the docker-mingw@fedora build test. Please find the testing commands and
> their output below. If you have Docker installed, you can probably reproduce it
> locally.
>
> === TEST SCRIPT BEGIN ===
> #! /bin/bash
> export ARCH=x86_64
> make docker-image-fedora V=1 NETWORK=1
> time make docker-test-mingw@fedora J=14 NETWORK=1
> === TEST SCRIPT END ===
>
> /tmp/qemu-test/src/migration/multifd.c:663: undefined reference to `multifd_channel_rdma_connect'
> ../migration/multifd.o: In function `multifd_load_cleanup':
> /tmp/qemu-test/src/migration/multifd.c:843: undefined reference to `qemu_rdma_cleanup'
> collect2: error: ld returned 1 exit status
> make[1]: *** [Makefile:206: qemu-system-x86_64w.exe] Error 1
> make: *** [Makefile:497: x86_64-softmmu/all] Error 2
> make: *** Waiting for unfinished jobs....
> ../migration/multifd.o: In function `multifd_rdma_recv_thread':
> /tmp/qemu-test/src/migration/multifd.c:898: undefined reference to `qemu_rdma_registration_handle'
> ---
> /tmp/qemu-test/src/migration/multifd.c:663: undefined reference to `multifd_channel_rdma_connect'
> ../migration/multifd.o: In function `multifd_load_cleanup':
> /tmp/qemu-test/src/migration/multifd.c:843: undefined reference to `qemu_rdma_cleanup'
> collect2: error: ld returned 1 exit status
> make[1]: *** [Makefile:206: qemu-system-aarch64w.exe] Error 1
> make: *** [Makefile:497: aarch64-softmmu/all] Error 2
> Traceback (most recent call last):
> File "./tests/docker/docker.py", line 664, in <module>
> sys.exit(main())
> ---
> raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=88473d634d6543ea992045cbe9a806e1', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-yiq7aevf/src/docker-src.2020-02-13-05.11.35.1374:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
> filter=--filter=label=com.qemu.instance.uuid=88473d634d6543ea992045cbe9a806e1
> make[1]: *** [docker-run] Error 1
> make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-yiq7aevf/src'
> make: *** [docker-run-test-mingw@fedora] Error 2
>
> real 2m35.791s
> user 0m7.717s
>
>
> The full log is available at
> http://patchew.org/logs/20200213093755.370-1-fengzhimin1@huawei.com/testing.docker-mingw@fedora/?type=message.
> ---
> Email generated automatically by Patchew [https://patchew.org/].
> Please send your feedback to patchew-devel@redhat.com
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 22+ messages in thread