[PATCH v2 0/3] Fix multifd + cancel + multifd

[PATCH v2 0/3] Fix multifd + cancel + multifd

[Juan Quintela]

Hi

In the v2 series:
- get the multifd test patch
- drop the ->shutdown fix
  it break postcopy recovery test.  Still trying to determine if the
  problem is inside the recover test or the recover code.
- upgrade the migrate_cancel test

Please review.

[v1]
This series:
- create a test that does:
  launch multifd on target
  migrate to target
  cancel on source
  create another target
  migrate again

- And fixes the cases that made it fail:
* Make sure that we don't try ever IO after shutdown/error

Please, review.

Juan Quintela (3):
  migration-test: Add migration multifd test
  migration: Make sure that we don't call write() in case of error
  migration-test: Make sure that multifd and cancel works

 migration/ram.c        |  25 +++++++
 tests/migration-test.c | 166 ++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 190 insertions(+), 1 deletion(-)

-- 
2.24.1

[PATCH v2 1/3] migration-test: Add migration multifd test

[Juan Quintela]

We set multifd-channels.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Tested-by: Wei Yang <richardw.yang@linux.intel.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 tests/migration-test.c | 56 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)

diff --git a/tests/migration-test.c b/tests/migration-test.c
index 53afec4395..1a8d06718e 100644
--- a/tests/migration-test.c
+++ b/tests/migration-test.c
@@ -1202,6 +1202,61 @@ static void test_migrate_auto_converge(void)
     test_migrate_end(from, to, true);
 }
 
+static void test_multifd_tcp(void)
+{
+    MigrateStart *args = migrate_start_new();
+    QTestState *from, *to;
+    QDict *rsp;
+    char *uri;
+
+    if (test_migrate_start(&from, &to, "defer", args)) {
+        return;
+    }
+
+    /*
+     * We want to pick a speed slow enough that the test completes
+     * quickly, but that it doesn't complete precopy even on a slow
+     * machine, so also set the downtime.
+     */
+    /* 1 ms should make it not converge*/
+    migrate_set_parameter_int(from, "downtime-limit", 1);
+    /* 1GB/s */
+    migrate_set_parameter_int(from, "max-bandwidth", 1000000000);
+
+    migrate_set_parameter_int(from, "multifd-channels", 16);
+    migrate_set_parameter_int(to, "multifd-channels", 16);
+
+    migrate_set_capability(from, "multifd", "true");
+    migrate_set_capability(to, "multifd", "true");
+
+    /* Start incoming migration from the 1st socket */
+    rsp = wait_command(to, "{ 'execute': 'migrate-incoming',"
+                           "  'arguments': { 'uri': 'tcp:127.0.0.1:0' }}");
+    qobject_unref(rsp);
+
+    /* Wait for the first serial output from the source */
+    wait_for_serial("src_serial");
+
+    uri = migrate_get_socket_address(to, "socket-address");
+
+    migrate_qmp(from, uri, "{}");
+
+    wait_for_migration_pass(from);
+
+    /* 300ms it should converge */
+    migrate_set_parameter_int(from, "downtime-limit", 600);
+
+    if (!got_stop) {
+        qtest_qmp_eventwait(from, "STOP");
+    }
+    qtest_qmp_eventwait(to, "RESUME");
+
+    wait_for_serial("dest_serial");
+    wait_for_migration_complete(from);
+    test_migrate_end(from, to, true);
+    free(uri);
+}
+
 int main(int argc, char **argv)
 {
     char template[] = "/tmp/migration-test-XXXXXX";
@@ -1266,6 +1321,7 @@ int main(int argc, char **argv)
                    test_validate_uuid_dst_not_set);
 
     qtest_add_func("/migration/auto_converge", test_migrate_auto_converge);
+    qtest_add_func("/migration/multifd/tcp", test_multifd_tcp);
 
     ret = g_test_run();
 
-- 
2.24.1

[PATCH v2 2/3] migration: Make sure that we don't call write() in case of error

[Juan Quintela]

If we are exiting due to an error/finish/.... Just don't try to even
touch the channel with one IO operation.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/ram.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/migration/ram.c b/migration/ram.c
index 96feb4062c..6e678dbd2e 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -900,6 +900,12 @@ struct {
     uint64_t packet_num;
     /* send channels ready */
     QemuSemaphore channels_ready;
+    /*
+     * Have we already run terminate threads.  There is a race when it
+     * happens that we got one error while we are exiting.
+     * We will use atomic operations.  Only valid values are 0 and 1.
+     */
+    int exiting;
 } *multifd_send_state;
 
 /*
@@ -928,6 +934,10 @@ static int multifd_send_pages(RAMState *rs)
     MultiFDPages_t *pages = multifd_send_state->pages;
     uint64_t transferred;
 
+    if (atomic_read(&multifd_send_state->exiting)) {
+        return -1;
+    }
+
     qemu_sem_wait(&multifd_send_state->channels_ready);
     for (i = next_channel;; i = (i + 1) % migrate_multifd_channels()) {
         p = &multifd_send_state->params[i];
@@ -1009,6 +1019,16 @@ static void multifd_send_terminate_threads(Error *err)
         }
     }
 
+    /*
+     * We don't want to exit each threads twice.  Depending on where
+     * we get the error, or if there are two independent errors in two
+     * threads at the same time, we can end calling this function
+     * twice.
+     */
+    if (atomic_xchg(&multifd_send_state->exiting, 1)) {
+        return;
+    }
+
     for (i = 0; i < migrate_multifd_channels(); i++) {
         MultiFDSendParams *p = &multifd_send_state->params[i];
 
@@ -1118,6 +1138,10 @@ static void *multifd_send_thread(void *opaque)
 
     while (true) {
         qemu_sem_wait(&p->sem);
+
+        if (atomic_read(&multifd_send_state->exiting)) {
+            break;
+        }
         qemu_mutex_lock(&p->mutex);
 
         if (p->pending_job) {
@@ -1224,6 +1248,7 @@ int multifd_save_setup(void)
     multifd_send_state->params = g_new0(MultiFDSendParams, thread_count);
     multifd_send_state->pages = multifd_pages_init(page_count);
     qemu_sem_init(&multifd_send_state->channels_ready, 0);
+    atomic_set(&multifd_send_state->exiting, 0);
 
     for (i = 0; i < thread_count; i++) {
         MultiFDSendParams *p = &multifd_send_state->params[i];
-- 
2.24.1

[PATCH v2 3/3] migration-test: Make sure that multifd and cancel works

[Juan Quintela]

Test that this sequerce works:

- launch source
- launch target
- start migration
- cancel migration
- relaunch target
- do migration again

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>

---

- Wait for 1st trhead to move to cancelled before launching second
  migration
- Add 'to2' parameter to diferentiate 1st and second target.
---
 tests/migration-test.c | 110 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 109 insertions(+), 1 deletion(-)

diff --git a/tests/migration-test.c b/tests/migration-test.c
index 1a8d06718e..7a833d65ce 100644
--- a/tests/migration-test.c
+++ b/tests/migration-test.c
@@ -387,6 +387,14 @@ static void migrate_recover(QTestState *who, const char *uri)
     qobject_unref(rsp);
 }
 
+static void migrate_cancel(QTestState *who)
+{
+    QDict *rsp;
+
+    rsp = wait_command(who, "{ 'execute': 'migrate_cancel' }");
+    qobject_unref(rsp);
+}
+
 static void migrate_set_capability(QTestState *who, const char *capability,
                                    bool value)
 {
@@ -419,6 +427,8 @@ static void migrate_postcopy_start(QTestState *from, QTestState *to)
 typedef struct {
     bool hide_stderr;
     bool use_shmem;
+    /* only launch the target process */
+    bool only_target;
     char *opts_source;
     char *opts_target;
 } MigrateStart;
@@ -534,7 +544,9 @@ static int test_migrate_start(QTestState **from, QTestState **to,
                                  arch_source, shmem_opts, args->opts_source,
                                  ignore_stderr);
     g_free(arch_source);
-    *from = qtest_init(cmd_source);
+    if (!args->only_target) {
+        *from = qtest_init(cmd_source);
+    }
     g_free(cmd_source);
 
     cmd_target = g_strdup_printf("-accel kvm -accel tcg%s%s "
@@ -1257,6 +1269,101 @@ static void test_multifd_tcp(void)
     free(uri);
 }
 
+/*
+ * This test does:
+ *  source               target
+ *                       migrate_incoming
+ *     migrate
+ *     migrate_cancel
+ *                       launch another target
+ *     migrate
+ *
+ *  And see that it works
+ */
+
+static void test_multifd_tcp_cancel(void)
+{
+    MigrateStart *args = migrate_start_new();
+    QTestState *from, *to, *to2;
+    QDict *rsp;
+    char *uri;
+
+    args->hide_stderr = true;
+
+    if (test_migrate_start(&from, &to, "defer", args)) {
+        return;
+    }
+
+    /*
+     * We want to pick a speed slow enough that the test completes
+     * quickly, but that it doesn't complete precopy even on a slow
+     * machine, so also set the downtime.
+     */
+    /* 1 ms should make it not converge*/
+    migrate_set_parameter_int(from, "downtime-limit", 1);
+    /* 1GB/s */
+    migrate_set_parameter_int(from, "max-bandwidth", 1000000000);
+
+    migrate_set_parameter_int(from, "multifd-channels", 16);
+    migrate_set_parameter_int(to, "multifd-channels", 16);
+
+    migrate_set_capability(from, "multifd", "true");
+    migrate_set_capability(to, "multifd", "true");
+
+    /* Start incoming migration from the 1st socket */
+    rsp = wait_command(to, "{ 'execute': 'migrate-incoming',"
+                           "  'arguments': { 'uri': 'tcp:127.0.0.1:0' }}");
+    qobject_unref(rsp);
+
+    /* Wait for the first serial output from the source */
+    wait_for_serial("src_serial");
+
+    uri = migrate_get_socket_address(to, "socket-address");
+
+    migrate_qmp(from, uri, "{}");
+
+    wait_for_migration_pass(from);
+
+    migrate_cancel(from);
+
+    args = migrate_start_new();
+    args->only_target = true;
+
+    if (test_migrate_start(&from, &to2, "defer", args)) {
+        return;
+    }
+
+    migrate_set_parameter_int(to2, "multifd-channels", 16);
+
+    migrate_set_capability(to2, "multifd", "true");
+
+    /* Start incoming migration from the 1st socket */
+    rsp = wait_command(to2, "{ 'execute': 'migrate-incoming',"
+                            "  'arguments': { 'uri': 'tcp:127.0.0.1:0' }}");
+    qobject_unref(rsp);
+
+    /* 300ms it should converge */
+    migrate_set_parameter_int(from, "downtime-limit", 300);
+
+    uri = migrate_get_socket_address(to2, "socket-address");
+
+    wait_for_migration_status(from, "cancelled", NULL);
+
+    migrate_qmp(from, uri, "{}");
+
+    wait_for_migration_pass(from);
+
+    if (!got_stop) {
+        qtest_qmp_eventwait(from, "STOP");
+    }
+    qtest_qmp_eventwait(to2, "RESUME");
+
+    wait_for_serial("dest_serial");
+    wait_for_migration_complete(from);
+    test_migrate_end(from, to2, true);
+    free(uri);
+}
+
 int main(int argc, char **argv)
 {
     char template[] = "/tmp/migration-test-XXXXXX";
@@ -1322,6 +1429,7 @@ int main(int argc, char **argv)
 
     qtest_add_func("/migration/auto_converge", test_migrate_auto_converge);
     qtest_add_func("/migration/multifd/tcp", test_multifd_tcp);
+    qtest_add_func("/migration/multifd/tcp/cancel", test_multifd_tcp_cancel);
 
     ret = g_test_run();
 
-- 
2.24.1

Re: [PATCH v2 1/3] migration-test: Add migration multifd test

[Dr. David Alan Gilbert]

* Juan Quintela (quintela@redhat.com) wrote:
> We set multifd-channels.
> 
> Signed-off-by: Juan Quintela <quintela@redhat.com>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> Tested-by: Wei Yang <richardw.yang@linux.intel.com>
> Signed-off-by: Juan Quintela <quintela@redhat.com>
> ---
>  tests/migration-test.c | 56 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 56 insertions(+)
> 
> diff --git a/tests/migration-test.c b/tests/migration-test.c
> index 53afec4395..1a8d06718e 100644
> --- a/tests/migration-test.c
> +++ b/tests/migration-test.c
> @@ -1202,6 +1202,61 @@ static void test_migrate_auto_converge(void)
>      test_migrate_end(from, to, true);
>  }
>  
> +static void test_multifd_tcp(void)
> +{
> +    MigrateStart *args = migrate_start_new();
> +    QTestState *from, *to;
> +    QDict *rsp;
> +    char *uri;
> +
> +    if (test_migrate_start(&from, &to, "defer", args)) {
> +        return;
> +    }
> +
> +    /*
> +     * We want to pick a speed slow enough that the test completes
> +     * quickly, but that it doesn't complete precopy even on a slow
> +     * machine, so also set the downtime.
> +     */
> +    /* 1 ms should make it not converge*/
> +    migrate_set_parameter_int(from, "downtime-limit", 1);
> +    /* 1GB/s */
> +    migrate_set_parameter_int(from, "max-bandwidth", 1000000000);
> +
> +    migrate_set_parameter_int(from, "multifd-channels", 16);
> +    migrate_set_parameter_int(to, "multifd-channels", 16);
> +
> +    migrate_set_capability(from, "multifd", "true");
> +    migrate_set_capability(to, "multifd", "true");
> +
> +    /* Start incoming migration from the 1st socket */
> +    rsp = wait_command(to, "{ 'execute': 'migrate-incoming',"
> +                           "  'arguments': { 'uri': 'tcp:127.0.0.1:0' }}");
> +    qobject_unref(rsp);
> +
> +    /* Wait for the first serial output from the source */
> +    wait_for_serial("src_serial");
> +
> +    uri = migrate_get_socket_address(to, "socket-address");
> +
> +    migrate_qmp(from, uri, "{}");
> +
> +    wait_for_migration_pass(from);
> +
> +    /* 300ms it should converge */
> +    migrate_set_parameter_int(from, "downtime-limit", 600);

Comment doesn't quite match here; but other than that;

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

> +    if (!got_stop) {
> +        qtest_qmp_eventwait(from, "STOP");
> +    }
> +    qtest_qmp_eventwait(to, "RESUME");
> +
> +    wait_for_serial("dest_serial");
> +    wait_for_migration_complete(from);
> +    test_migrate_end(from, to, true);
> +    free(uri);
> +}
> +
>  int main(int argc, char **argv)
>  {
>      char template[] = "/tmp/migration-test-XXXXXX";
> @@ -1266,6 +1321,7 @@ int main(int argc, char **argv)
>                     test_validate_uuid_dst_not_set);
>  
>      qtest_add_func("/migration/auto_converge", test_migrate_auto_converge);
> +    qtest_add_func("/migration/multifd/tcp", test_multifd_tcp);
>  
>      ret = g_test_run();
>  
> -- 
> 2.24.1
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [PATCH v2 2/3] migration: Make sure that we don't call write() in case of error

[Dr. David Alan Gilbert]

* Juan Quintela (quintela@redhat.com) wrote:
> If we are exiting due to an error/finish/.... Just don't try to even
> touch the channel with one IO operation.
> 
> Signed-off-by: Juan Quintela <quintela@redhat.com>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> Signed-off-by: Juan Quintela <quintela@redhat.com>
> ---
>  migration/ram.c | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
> 
> diff --git a/migration/ram.c b/migration/ram.c
> index 96feb4062c..6e678dbd2e 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -900,6 +900,12 @@ struct {
>      uint64_t packet_num;
>      /* send channels ready */
>      QemuSemaphore channels_ready;
> +    /*
> +     * Have we already run terminate threads.  There is a race when it
> +     * happens that we got one error while we are exiting.
> +     * We will use atomic operations.  Only valid values are 0 and 1.
> +     */
> +    int exiting;
>  } *multifd_send_state;
>  
>  /*
> @@ -928,6 +934,10 @@ static int multifd_send_pages(RAMState *rs)
>      MultiFDPages_t *pages = multifd_send_state->pages;
>      uint64_t transferred;
>  
> +    if (atomic_read(&multifd_send_state->exiting)) {
> +        return -1;
> +    }
> +
>      qemu_sem_wait(&multifd_send_state->channels_ready);
>      for (i = next_channel;; i = (i + 1) % migrate_multifd_channels()) {
>          p = &multifd_send_state->params[i];
> @@ -1009,6 +1019,16 @@ static void multifd_send_terminate_threads(Error *err)
>          }
>      }
>  
> +    /*
> +     * We don't want to exit each threads twice.  Depending on where
> +     * we get the error, or if there are two independent errors in two
> +     * threads at the same time, we can end calling this function
> +     * twice.
> +     */

Yeh that makes it cleaer.  Thanks

> +    if (atomic_xchg(&multifd_send_state->exiting, 1)) {
> +        return;
> +    }
> +
>      for (i = 0; i < migrate_multifd_channels(); i++) {
>          MultiFDSendParams *p = &multifd_send_state->params[i];
>  
> @@ -1118,6 +1138,10 @@ static void *multifd_send_thread(void *opaque)
>  
>      while (true) {
>          qemu_sem_wait(&p->sem);
> +
> +        if (atomic_read(&multifd_send_state->exiting)) {
> +            break;
> +        }
>          qemu_mutex_lock(&p->mutex);
>  
>          if (p->pending_job) {
> @@ -1224,6 +1248,7 @@ int multifd_save_setup(void)
>      multifd_send_state->params = g_new0(MultiFDSendParams, thread_count);
>      multifd_send_state->pages = multifd_pages_init(page_count);
>      qemu_sem_init(&multifd_send_state->channels_ready, 0);
> +    atomic_set(&multifd_send_state->exiting, 0);
>  
>      for (i = 0; i < thread_count; i++) {
>          MultiFDSendParams *p = &multifd_send_state->params[i];
> -- 
> 2.24.1
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [PATCH v2 3/3] migration-test: Make sure that multifd and cancel works

[Dr. David Alan Gilbert]

* Juan Quintela (quintela@redhat.com) wrote:
> Test that this sequerce works:
> 
> - launch source
> - launch target
> - start migration
> - cancel migration
> - relaunch target
> - do migration again
> 
> Signed-off-by: Juan Quintela <quintela@redhat.com>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> Signed-off-by: Juan Quintela <quintela@redhat.com>
> 
> ---
> 
> - Wait for 1st trhead to move to cancelled before launching second
>   migration
> - Add 'to2' parameter to diferentiate 1st and second target.
> ---
>  tests/migration-test.c | 110 ++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 109 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/migration-test.c b/tests/migration-test.c
> index 1a8d06718e..7a833d65ce 100644
> --- a/tests/migration-test.c
> +++ b/tests/migration-test.c
> @@ -387,6 +387,14 @@ static void migrate_recover(QTestState *who, const char *uri)
>      qobject_unref(rsp);
>  }
>  
> +static void migrate_cancel(QTestState *who)
> +{
> +    QDict *rsp;
> +
> +    rsp = wait_command(who, "{ 'execute': 'migrate_cancel' }");
> +    qobject_unref(rsp);
> +}
> +
>  static void migrate_set_capability(QTestState *who, const char *capability,
>                                     bool value)
>  {
> @@ -419,6 +427,8 @@ static void migrate_postcopy_start(QTestState *from, QTestState *to)
>  typedef struct {
>      bool hide_stderr;
>      bool use_shmem;
> +    /* only launch the target process */
> +    bool only_target;
>      char *opts_source;
>      char *opts_target;
>  } MigrateStart;
> @@ -534,7 +544,9 @@ static int test_migrate_start(QTestState **from, QTestState **to,
>                                   arch_source, shmem_opts, args->opts_source,
>                                   ignore_stderr);
>      g_free(arch_source);
> -    *from = qtest_init(cmd_source);
> +    if (!args->only_target) {
> +        *from = qtest_init(cmd_source);
> +    }
>      g_free(cmd_source);
>  
>      cmd_target = g_strdup_printf("-accel kvm -accel tcg%s%s "
> @@ -1257,6 +1269,101 @@ static void test_multifd_tcp(void)
>      free(uri);
>  }
>  
> +/*
> + * This test does:
> + *  source               target
> + *                       migrate_incoming
> + *     migrate
> + *     migrate_cancel
> + *                       launch another target
> + *     migrate
> + *
> + *  And see that it works
> + */
> +
> +static void test_multifd_tcp_cancel(void)
> +{
> +    MigrateStart *args = migrate_start_new();
> +    QTestState *from, *to, *to2;
> +    QDict *rsp;
> +    char *uri;
> +
> +    args->hide_stderr = true;
> +
> +    if (test_migrate_start(&from, &to, "defer", args)) {
> +        return;
> +    }
> +
> +    /*
> +     * We want to pick a speed slow enough that the test completes
> +     * quickly, but that it doesn't complete precopy even on a slow
> +     * machine, so also set the downtime.
> +     */
> +    /* 1 ms should make it not converge*/
> +    migrate_set_parameter_int(from, "downtime-limit", 1);
> +    /* 1GB/s */
> +    migrate_set_parameter_int(from, "max-bandwidth", 1000000000);

See previous comment about my update in the postcopy test, but OK.

> +    migrate_set_parameter_int(from, "multifd-channels", 16);
> +    migrate_set_parameter_int(to, "multifd-channels", 16);
> +
> +    migrate_set_capability(from, "multifd", "true");
> +    migrate_set_capability(to, "multifd", "true");
> +
> +    /* Start incoming migration from the 1st socket */
> +    rsp = wait_command(to, "{ 'execute': 'migrate-incoming',"
> +                           "  'arguments': { 'uri': 'tcp:127.0.0.1:0' }}");
> +    qobject_unref(rsp);
> +
> +    /* Wait for the first serial output from the source */
> +    wait_for_serial("src_serial");
> +
> +    uri = migrate_get_socket_address(to, "socket-address");
> +
> +    migrate_qmp(from, uri, "{}");
> +
> +    wait_for_migration_pass(from);
> +
> +    migrate_cancel(from);

You might want to add something here to validate that the cancel has
finished before starting the new one.

Dave

> +    args = migrate_start_new();
> +    args->only_target = true;
> +
> +    if (test_migrate_start(&from, &to2, "defer", args)) {
> +        return;
> +    }
> +
> +    migrate_set_parameter_int(to2, "multifd-channels", 16);
> +
> +    migrate_set_capability(to2, "multifd", "true");
> +
> +    /* Start incoming migration from the 1st socket */
> +    rsp = wait_command(to2, "{ 'execute': 'migrate-incoming',"
> +                            "  'arguments': { 'uri': 'tcp:127.0.0.1:0' }}");
> +    qobject_unref(rsp);
> +
> +    /* 300ms it should converge */
> +    migrate_set_parameter_int(from, "downtime-limit", 300);
> +
> +    uri = migrate_get_socket_address(to2, "socket-address");
> +
> +    wait_for_migration_status(from, "cancelled", NULL);
> +
> +    migrate_qmp(from, uri, "{}");
> +
> +    wait_for_migration_pass(from);
> +
> +    if (!got_stop) {
> +        qtest_qmp_eventwait(from, "STOP");
> +    }
> +    qtest_qmp_eventwait(to2, "RESUME");
> +
> +    wait_for_serial("dest_serial");
> +    wait_for_migration_complete(from);
> +    test_migrate_end(from, to2, true);
> +    free(uri);
> +}
> +
>  int main(int argc, char **argv)
>  {
>      char template[] = "/tmp/migration-test-XXXXXX";
> @@ -1322,6 +1429,7 @@ int main(int argc, char **argv)
>  
>      qtest_add_func("/migration/auto_converge", test_migrate_auto_converge);
>      qtest_add_func("/migration/multifd/tcp", test_multifd_tcp);
> +    qtest_add_func("/migration/multifd/tcp/cancel", test_multifd_tcp_cancel);
>  
>      ret = g_test_run();
>  
> -- 
> 2.24.1
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [PATCH v2 3/3] migration-test: Make sure that multifd and cancel works

[Juan Quintela]

"Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:
> * Juan Quintela (quintela@redhat.com) wrote:
>> Test that this sequerce works:
>> 
>> - launch source
>> - launch target
>> - start migration
>> - cancel migration
>> - relaunch target
>> - do migration again
>> 
>> Signed-off-by: Juan Quintela <quintela@redhat.com>
>> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>> Signed-off-by: Juan Quintela <quintela@redhat.com>
>> 
>> ---
>> 
>> - Wait for 1st trhead to move to cancelled before launching second
>>   migration
>> - Add 'to2' parameter to diferentiate 1st and second target.
>> ---
>>  tests/migration-test.c | 110 ++++++++++++++++++++++++++++++++++++++++-
>>  1 file changed, 109 insertions(+), 1 deletion(-)
>> 
>> diff --git a/tests/migration-test.c b/tests/migration-test.c
>> index 1a8d06718e..7a833d65ce 100644
>> --- a/tests/migration-test.c
>> +++ b/tests/migration-test.c
>> @@ -387,6 +387,14 @@ static void migrate_recover(QTestState *who, const char *uri)
>>      qobject_unref(rsp);
>>  }
>>  
>> +static void migrate_cancel(QTestState *who)
>> +{
>> +    QDict *rsp;
>> +
>> +    rsp = wait_command(who, "{ 'execute': 'migrate_cancel' }");
>> +    qobject_unref(rsp);
>> +}
>> +
>>  static void migrate_set_capability(QTestState *who, const char *capability,
>>                                     bool value)
>>  {
>> @@ -419,6 +427,8 @@ static void migrate_postcopy_start(QTestState *from, QTestState *to)
>>  typedef struct {
>>      bool hide_stderr;
>>      bool use_shmem;
>> +    /* only launch the target process */
>> +    bool only_target;
>>      char *opts_source;
>>      char *opts_target;
>>  } MigrateStart;
>> @@ -534,7 +544,9 @@ static int test_migrate_start(QTestState **from, QTestState **to,
>>                                   arch_source, shmem_opts, args->opts_source,
>>                                   ignore_stderr);
>>      g_free(arch_source);
>> -    *from = qtest_init(cmd_source);
>> +    if (!args->only_target) {
>> +        *from = qtest_init(cmd_source);
>> +    }
>>      g_free(cmd_source);
>>  
>>      cmd_target = g_strdup_printf("-accel kvm -accel tcg%s%s "
>> @@ -1257,6 +1269,101 @@ static void test_multifd_tcp(void)
>>      free(uri);
>>  }
>>  
>> +/*
>> + * This test does:
>> + *  source               target
>> + *                       migrate_incoming
>> + *     migrate
>> + *     migrate_cancel
>> + *                       launch another target
>> + *     migrate
>> + *
>> + *  And see that it works
>> + */
>> +
>> +static void test_multifd_tcp_cancel(void)
>> +{
>> +    MigrateStart *args = migrate_start_new();
>> +    QTestState *from, *to, *to2;
>> +    QDict *rsp;
>> +    char *uri;
>> +
>> +    args->hide_stderr = true;
>> +
>> +    if (test_migrate_start(&from, &to, "defer", args)) {
>> +        return;
>> +    }
>> +
>> +    /*
>> +     * We want to pick a speed slow enough that the test completes
>> +     * quickly, but that it doesn't complete precopy even on a slow
>> +     * machine, so also set the downtime.
>> +     */
>> +    /* 1 ms should make it not converge*/
>> +    migrate_set_parameter_int(from, "downtime-limit", 1);
>> +    /* 1GB/s */
>> +    migrate_set_parameter_int(from, "max-bandwidth", 1000000000);
>
> See previous comment about my update in the postcopy test, but OK.

Yes and no O:-)

function                 value
--------                 -----
postcopy_prepare            30 000 000
precopy_unix             1 000 000 000
xbzrle                   1 000 000 000
precpy_tcp               1 000 000 000
fd_proto                 1 000 000 000
auto-converge            1 000 000 000
multifd                  1 000 000 000
multifd_cancel           1 000 000 000

Can we agree to create three global constants with the following values:


const long long downtime_initial = 1;
const long long downtime_final = 300;  /* or ever 600 */
const long long bandwidth_initial = 30 000 000;

And substitute everywhere with the rigths constants?

Thanks, Juan.

>> +    migrate_set_parameter_int(from, "multifd-channels", 16);
>> +    migrate_set_parameter_int(to, "multifd-channels", 16);
>> +
>> +    migrate_set_capability(from, "multifd", "true");
>> +    migrate_set_capability(to, "multifd", "true");
>> +
>> +    /* Start incoming migration from the 1st socket */
>> +    rsp = wait_command(to, "{ 'execute': 'migrate-incoming',"
>> +                           "  'arguments': { 'uri': 'tcp:127.0.0.1:0' }}");
>> +    qobject_unref(rsp);
>> +
>> +    /* Wait for the first serial output from the source */
>> +    wait_for_serial("src_serial");
>> +
>> +    uri = migrate_get_socket_address(to, "socket-address");
>> +
>> +    migrate_qmp(from, uri, "{}");
>> +
>> +    wait_for_migration_pass(from);
>> +
>> +    migrate_cancel(from);
>
> You might want to add something here to validate that the cancel has
> finished before starting the new one.


>> +    args = migrate_start_new();
>> +    args->only_target = true;
>> +
>> +    if (test_migrate_start(&from, &to2, "defer", args)) {
>> +        return;
>> +    }
>> +
>> +    migrate_set_parameter_int(to2, "multifd-channels", 16);
>> +
>> +    migrate_set_capability(to2, "multifd", "true");
>> +
>> +    /* Start incoming migration from the 1st socket */
>> +    rsp = wait_command(to2, "{ 'execute': 'migrate-incoming',"
>> +                            "  'arguments': { 'uri': 'tcp:127.0.0.1:0' }}");
>> +    qobject_unref(rsp);
>> +
>> +    /* 300ms it should converge */
>> +    migrate_set_parameter_int(from, "downtime-limit", 300);
>> +
>> +    uri = migrate_get_socket_address(to2, "socket-address");
>> +
>> +    wait_for_migration_status(from, "cancelled", NULL);

That is exactly what this last line does, not?
The rest of things that we do here are for "to2" instance, that don't
affect us at all.

Once told that, I think that it makes sense to just move the set of
downtime-limit after the migration.

Thanks, Juan.


>> +
>> +    migrate_qmp(from, uri, "{}");
>> +
>> +    wait_for_migration_pass(from);
>> +
>> +    if (!got_stop) {
>> +        qtest_qmp_eventwait(from, "STOP");
>> +    }
>> +    qtest_qmp_eventwait(to2, "RESUME");
>> +
>> +    wait_for_serial("dest_serial");
>> +    wait_for_migration_complete(from);
>> +    test_migrate_end(from, to2, true);
>> +    free(uri);
>> +}
>> +
>>  int main(int argc, char **argv)
>>  {
>>      char template[] = "/tmp/migration-test-XXXXXX";
>> @@ -1322,6 +1429,7 @@ int main(int argc, char **argv)
>>  
>>      qtest_add_func("/migration/auto_converge", test_migrate_auto_converge);
>>      qtest_add_func("/migration/multifd/tcp", test_multifd_tcp);
>> +    qtest_add_func("/migration/multifd/tcp/cancel", test_multifd_tcp_cancel);
>>  
>>      ret = g_test_run();
>>  
>> -- 
>> 2.24.1
>> 
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [PATCH v2 3/3] migration-test: Make sure that multifd and cancel works

[Dr. David Alan Gilbert]

* Juan Quintela (quintela@redhat.com) wrote:
> "Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:
> > * Juan Quintela (quintela@redhat.com) wrote:
> >> Test that this sequerce works:
> >> 
> >> - launch source
> >> - launch target
> >> - start migration
> >> - cancel migration
> >> - relaunch target
> >> - do migration again
> >> 
> >> Signed-off-by: Juan Quintela <quintela@redhat.com>
> >> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> >> Signed-off-by: Juan Quintela <quintela@redhat.com>
> >> 
> >> ---
> >> 
> >> - Wait for 1st trhead to move to cancelled before launching second
> >>   migration
> >> - Add 'to2' parameter to diferentiate 1st and second target.
> >> ---
> >>  tests/migration-test.c | 110 ++++++++++++++++++++++++++++++++++++++++-
> >>  1 file changed, 109 insertions(+), 1 deletion(-)
> >> 
> >> diff --git a/tests/migration-test.c b/tests/migration-test.c
> >> index 1a8d06718e..7a833d65ce 100644
> >> --- a/tests/migration-test.c
> >> +++ b/tests/migration-test.c
> >> @@ -387,6 +387,14 @@ static void migrate_recover(QTestState *who, const char *uri)
> >>      qobject_unref(rsp);
> >>  }
> >>  
> >> +static void migrate_cancel(QTestState *who)
> >> +{
> >> +    QDict *rsp;
> >> +
> >> +    rsp = wait_command(who, "{ 'execute': 'migrate_cancel' }");
> >> +    qobject_unref(rsp);
> >> +}
> >> +
> >>  static void migrate_set_capability(QTestState *who, const char *capability,
> >>                                     bool value)
> >>  {
> >> @@ -419,6 +427,8 @@ static void migrate_postcopy_start(QTestState *from, QTestState *to)
> >>  typedef struct {
> >>      bool hide_stderr;
> >>      bool use_shmem;
> >> +    /* only launch the target process */
> >> +    bool only_target;
> >>      char *opts_source;
> >>      char *opts_target;
> >>  } MigrateStart;
> >> @@ -534,7 +544,9 @@ static int test_migrate_start(QTestState **from, QTestState **to,
> >>                                   arch_source, shmem_opts, args->opts_source,
> >>                                   ignore_stderr);
> >>      g_free(arch_source);
> >> -    *from = qtest_init(cmd_source);
> >> +    if (!args->only_target) {
> >> +        *from = qtest_init(cmd_source);
> >> +    }
> >>      g_free(cmd_source);
> >>  
> >>      cmd_target = g_strdup_printf("-accel kvm -accel tcg%s%s "
> >> @@ -1257,6 +1269,101 @@ static void test_multifd_tcp(void)
> >>      free(uri);
> >>  }
> >>  
> >> +/*
> >> + * This test does:
> >> + *  source               target
> >> + *                       migrate_incoming
> >> + *     migrate
> >> + *     migrate_cancel
> >> + *                       launch another target
> >> + *     migrate
> >> + *
> >> + *  And see that it works
> >> + */
> >> +
> >> +static void test_multifd_tcp_cancel(void)
> >> +{
> >> +    MigrateStart *args = migrate_start_new();
> >> +    QTestState *from, *to, *to2;
> >> +    QDict *rsp;
> >> +    char *uri;
> >> +
> >> +    args->hide_stderr = true;
> >> +
> >> +    if (test_migrate_start(&from, &to, "defer", args)) {
> >> +        return;
> >> +    }
> >> +
> >> +    /*
> >> +     * We want to pick a speed slow enough that the test completes
> >> +     * quickly, but that it doesn't complete precopy even on a slow
> >> +     * machine, so also set the downtime.
> >> +     */
> >> +    /* 1 ms should make it not converge*/
> >> +    migrate_set_parameter_int(from, "downtime-limit", 1);
> >> +    /* 1GB/s */
> >> +    migrate_set_parameter_int(from, "max-bandwidth", 1000000000);
> >
> > See previous comment about my update in the postcopy test, but OK.
> 
> Yes and no O:-)
> 
> function                 value
> --------                 -----
> postcopy_prepare            30 000 000
> precopy_unix             1 000 000 000
> xbzrle                   1 000 000 000
> precpy_tcp               1 000 000 000
> fd_proto                 1 000 000 000
> auto-converge            1 000 000 000
> multifd                  1 000 000 000
> multifd_cancel           1 000 000 000
> 
> Can we agree to create three global constants with the following values:
> 
> 
> const long long downtime_initial = 1;
> const long long downtime_final = 300;  /* or ever 600 */
> const long long bandwidth_initial = 30 000 000;
> 
> And substitute everywhere with the rigths constants?

Yeh, so we just have to make sure we only change it in the places
where we want to/dont want it to converge.

> Thanks, Juan.
> 
> >> +    migrate_set_parameter_int(from, "multifd-channels", 16);
> >> +    migrate_set_parameter_int(to, "multifd-channels", 16);
> >> +
> >> +    migrate_set_capability(from, "multifd", "true");
> >> +    migrate_set_capability(to, "multifd", "true");
> >> +
> >> +    /* Start incoming migration from the 1st socket */
> >> +    rsp = wait_command(to, "{ 'execute': 'migrate-incoming',"
> >> +                           "  'arguments': { 'uri': 'tcp:127.0.0.1:0' }}");
> >> +    qobject_unref(rsp);
> >> +
> >> +    /* Wait for the first serial output from the source */
> >> +    wait_for_serial("src_serial");
> >> +
> >> +    uri = migrate_get_socket_address(to, "socket-address");
> >> +
> >> +    migrate_qmp(from, uri, "{}");
> >> +
> >> +    wait_for_migration_pass(from);
> >> +
> >> +    migrate_cancel(from);
> >
> > You might want to add something here to validate that the cancel has
> > finished before starting the new one.
> 
> 
> >> +    args = migrate_start_new();
> >> +    args->only_target = true;
> >> +
> >> +    if (test_migrate_start(&from, &to2, "defer", args)) {
> >> +        return;
> >> +    }
> >> +
> >> +    migrate_set_parameter_int(to2, "multifd-channels", 16);
> >> +
> >> +    migrate_set_capability(to2, "multifd", "true");
> >> +
> >> +    /* Start incoming migration from the 1st socket */
> >> +    rsp = wait_command(to2, "{ 'execute': 'migrate-incoming',"
> >> +                            "  'arguments': { 'uri': 'tcp:127.0.0.1:0' }}");
> >> +    qobject_unref(rsp);
> >> +
> >> +    /* 300ms it should converge */
> >> +    migrate_set_parameter_int(from, "downtime-limit", 300);
> >> +
> >> +    uri = migrate_get_socket_address(to2, "socket-address");
> >> +
> >> +    wait_for_migration_status(from, "cancelled", NULL);
> 
> That is exactly what this last line does, not?
> The rest of things that we do here are for "to2" instance, that don't
> affect us at all.
> 
> Once told that, I think that it makes sense to just move the set of
> downtime-limit after the migration.

Ah OK, yes, that's fine.

Dave

> Thanks, Juan.
> 
> 
> >> +
> >> +    migrate_qmp(from, uri, "{}");
> >> +
> >> +    wait_for_migration_pass(from);
> >> +
> >> +    if (!got_stop) {
> >> +        qtest_qmp_eventwait(from, "STOP");
> >> +    }
> >> +    qtest_qmp_eventwait(to2, "RESUME");
> >> +
> >> +    wait_for_serial("dest_serial");
> >> +    wait_for_migration_complete(from);
> >> +    test_migrate_end(from, to2, true);
> >> +    free(uri);
> >> +}
> >> +
> >>  int main(int argc, char **argv)
> >>  {
> >>      char template[] = "/tmp/migration-test-XXXXXX";
> >> @@ -1322,6 +1429,7 @@ int main(int argc, char **argv)
> >>  
> >>      qtest_add_func("/migration/auto_converge", test_migrate_auto_converge);
> >>      qtest_add_func("/migration/multifd/tcp", test_multifd_tcp);
> >> +    qtest_add_func("/migration/multifd/tcp/cancel", test_multifd_tcp_cancel);
> >>  
> >>      ret = g_test_run();
> >>  
> >> -- 
> >> 2.24.1
> >> 
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [PATCH v2 1/3] migration-test: Add migration multifd test

[Juan Quintela]

"Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:
> * Juan Quintela (quintela@redhat.com) wrote:
>> We set multifd-channels.
>> 
>> Signed-off-by: Juan Quintela <quintela@redhat.com>
>> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>> Reviewed-by: Thomas Huth <thuth@redhat.com>
>> Tested-by: Wei Yang <richardw.yang@linux.intel.com>
>> Signed-off-by: Juan Quintela <quintela@redhat.com>
>> ---
>>  tests/migration-test.c | 56 ++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 56 insertions(+)
>> 
>> diff --git a/tests/migration-test.c b/tests/migration-test.c
>> index 53afec4395..1a8d06718e 100644
>> --- a/tests/migration-test.c
>> +++ b/tests/migration-test.c
>> @@ -1202,6 +1202,61 @@ static void test_migrate_auto_converge(void)
>>      test_migrate_end(from, to, true);
>>  }
>>  
>> +static void test_multifd_tcp(void)
>> +{
>> +    MigrateStart *args = migrate_start_new();
>> +    QTestState *from, *to;
>> +    QDict *rsp;
>> +    char *uri;
>> +
>> +    if (test_migrate_start(&from, &to, "defer", args)) {
>> +        return;
>> +    }
>> +
>> +    /*
>> +     * We want to pick a speed slow enough that the test completes
>> +     * quickly, but that it doesn't complete precopy even on a slow
>> +     * machine, so also set the downtime.
>> +     */
>> +    /* 1 ms should make it not converge*/
>> +    migrate_set_parameter_int(from, "downtime-limit", 1);
>> +    /* 1GB/s */
>> +    migrate_set_parameter_int(from, "max-bandwidth", 1000000000);
>> +
>> +    migrate_set_parameter_int(from, "multifd-channels", 16);
>> +    migrate_set_parameter_int(to, "multifd-channels", 16);
>> +
>> +    migrate_set_capability(from, "multifd", "true");
>> +    migrate_set_capability(to, "multifd", "true");
>> +
>> +    /* Start incoming migration from the 1st socket */
>> +    rsp = wait_command(to, "{ 'execute': 'migrate-incoming',"
>> +                           "  'arguments': { 'uri': 'tcp:127.0.0.1:0' }}");
>> +    qobject_unref(rsp);
>> +
>> +    /* Wait for the first serial output from the source */
>> +    wait_for_serial("src_serial");
>> +
>> +    uri = migrate_get_socket_address(to, "socket-address");
>> +
>> +    migrate_qmp(from, uri, "{}");
>> +
>> +    wait_for_migration_pass(from);
>> +
>> +    /* 300ms it should converge */
>> +    migrate_set_parameter_int(from, "downtime-limit", 600);
>
> Comment doesn't quite match here; but other than that;

Fixed code, thanks.

Re: [PATCH v2 0/3] Fix multifd + cancel + multifd

[no-reply]

Patchew URL: https://patchew.org/QEMU/20200107104914.1814-1-quintela@redhat.com/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.






The full log is available at
http://patchew.org/logs/20200107104914.1814-1-quintela@redhat.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [PATCH v2 0/3] Fix multifd + cancel + multifd

[Juan Quintela]

no-reply@patchew.org wrote:
> Patchew URL: https://patchew.org/QEMU/20200107104914.1814-1-quintela@redhat.com/
>
>
>
> Hi,
>
> This series failed the docker-mingw@fedora build test. Please find the
> testing commands and
> their output below. If you have Docker installed, you can probably reproduce it
> locally.
>
>
>
>
>

Hi

> The full log is available at
> http://patchew.org/logs/20200107104914.1814-1-quintela@redhat.com/testing.docker-mingw@fedora/?type=message.

N/A. Internal error while reading log file

So, I am going to assume that patchew is b0rken at the moment.

Later, Juan.