[Qemu-devel] [PULL 0/6] migration pull

[Qemu-devel] [PULL 0/6] migration pull

[Amit Shah]

The following changes since commit 0c6940d086f39bbf725d96104abe46da87429cb6:

  build: [bsd-user] Rename "syscall.h" to "target_syscall.h" in target directories (2016-02-25 16:41:08 +0000)

are available in the git repository at:

  https://git.kernel.org/pub/scm/virt/qemu/amit/migration.git tags/migration-for-2.6-4

for you to fetch changes up to aa81a236c8dc8fe18e26d9dadeb759dc53ee7e97:

  migration/vmstate: document VMStateFlags (2016-02-26 15:09:52 +0530)

----------------------------------------------------------------
migration pull
 - fix a qcow2 assert
 - fix for older distros (CentOS 5)
 - documentation for vmstate flags
 - minor code rearrangement

----------------------------------------------------------------

Denis V. Lunev (2):
  migration (ordinary): move bdrv_invalidate_cache_all of of coroutine
    context
  migration (postcopy): move bdrv_invalidate_cache_all of of coroutine
    context

Matthew Fortune (1):
  migration/postcopy-ram: Guard use of sys/eventfd.h with CONFIG_EVENTFD

Sascha Silbe (1):
  migration/vmstate: document VMStateFlags

Thomas Huth (1):
  MAINTAINERS: Add docs/migration.txt to the "Migration" section

Wei Yang (1):
  migration: reorder code to make it symmetric

 MAINTAINERS                   |   1 +
 include/migration/migration.h |   2 +
 include/migration/vmstate.h   | 100 +++++++++++++++++++++++++++++++++++++-----
 migration/migration.c         |  90 ++++++++++++++++++++-----------------
 migration/postcopy-ram.c      |   4 +-
 migration/savevm.c            |  34 ++++++++------
 6 files changed, 166 insertions(+), 65 deletions(-)

-- 
2.5.0

[Qemu-devel] [PULL 1/6] migration: reorder code to make it symmetric

[Amit Shah]

From: Wei Yang <richard.weiyang@gmail.com>

In qemu_savevm_state_complete_precopy(), it iterates on each device to add
a json object and transfer related status to destination, while the order
of the last two steps could be refined.

Current order:

    json_start_object()
    	save_section_header()
    	vmstate_save()
    json_end_object()
    	save_section_footer()

After the change:

    json_start_object()
    	save_section_header()
    	vmstate_save()
    	save_section_footer()
    json_end_object()

This patch reorder the code to to make it symmetric. No functional change.

Signed-off-by: Wei Yang <richard.weiyang@gmail.com>
Reviewed-by: Amit Shah <amit.shah@redhat.com>
Message-Id: <1454626230-16334-1-git-send-email-richard.weiyang@gmail.com>
Signed-off-by: Amit Shah <amit.shah@redhat.com>
---
 migration/savevm.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/migration/savevm.c b/migration/savevm.c
index 94f2894..02e8487 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1088,12 +1088,11 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only)
         json_prop_int(vmdesc, "instance_id", se->instance_id);
 
         save_section_header(f, se, QEMU_VM_SECTION_FULL);
-
         vmstate_save(f, se, vmdesc);
-
-        json_end_object(vmdesc);
         trace_savevm_section_end(se->idstr, se->section_id, 0);
         save_section_footer(f, se);
+
+        json_end_object(vmdesc);
     }
 
     if (!in_postcopy) {
-- 
2.5.0

[Qemu-devel] [PULL 2/6] migration/postcopy-ram: Guard use of sys/eventfd.h with CONFIG_EVENTFD

[Amit Shah]

From: Matthew Fortune <Matthew.Fortune@imgtec.com>

sys/eventfd.h was being guarded only by a check for linux but does
not exist on older distributions like CentOS 5. Move the include
into the code that uses it and add an appropriate guard.

Signed-off-by: Matthew Fortune <matthew.fortune@imgtec.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Message-Id: <6D39441BF12EF246A7ABCE6654B023536BB85DEB@hhmail02.hh.imgtec.org>
Signed-off-by: Amit Shah <amit.shah@redhat.com>
---
 migration/postcopy-ram.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
index 254c629..fbd0064 100644
--- a/migration/postcopy-ram.c
+++ b/migration/postcopy-ram.c
@@ -52,14 +52,14 @@ struct PostcopyDiscardState {
 #if defined(__linux__)
 
 #include <poll.h>
-#include <sys/eventfd.h>
 #include <sys/mman.h>
 #include <sys/ioctl.h>
 #include <sys/syscall.h>
 #include <asm/types.h> /* for __u64 */
 #endif
 
-#if defined(__linux__) && defined(__NR_userfaultfd)
+#if defined(__linux__) && defined(__NR_userfaultfd) && defined(CONFIG_EVENTFD)
+#include <sys/eventfd.h>
 #include <linux/userfaultfd.h>
 
 static bool ufd_version_check(int ufd)
-- 
2.5.0

[Qemu-devel] [PULL 3/6] migration (ordinary): move bdrv_invalidate_cache_all of of coroutine context

[Amit Shah]

From: "Denis V. Lunev" <den@openvz.org>

There is a possibility to hit an assert in qcow2_get_specific_info that
s->qcow_version is undefined. This happens when VM in starting from
suspended state, i.e. it processes incoming migration, and in the same
time 'info block' is called.

The problem is that qcow2_invalidate_cache() closes the image and
memset()s BDRVQcowState in the middle.

The patch moves processing of bdrv_invalidate_cache_all out of
coroutine context for standard migration to avoid that.

Signed-off-by: Denis V. Lunev <den@openvz.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
CC: Paolo Bonzini <pbonzini@redhat.com>
CC: Juan Quintela <quintela@redhat.com>
CC: Amit Shah <amit.shah@redhat.com>
Message-Id: <1456304019-10507-2-git-send-email-den@openvz.org>
Signed-off-by: Amit Shah <amit.shah@redhat.com>
---
 include/migration/migration.h |  2 +
 migration/migration.c         | 90 ++++++++++++++++++++++++-------------------
 2 files changed, 52 insertions(+), 40 deletions(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index 85b6026..ac2c12c 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -104,6 +104,8 @@ struct MigrationIncomingState {
     QemuMutex rp_mutex;    /* We send replies from multiple threads */
     void     *postcopy_tmp_page;
 
+    QEMUBH *bh;
+
     int state;
     /* See savevm.c */
     LoadStateEntry_Head loadvm_handlers;
diff --git a/migration/migration.c b/migration/migration.c
index fc5e50b..f82fdf6 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -323,10 +323,57 @@ void qemu_start_incoming_migration(const char *uri, Error **errp)
     }
 }
 
+static void process_incoming_migration_bh(void *opaque)
+{
+    Error *local_err = NULL;
+    MigrationIncomingState *mis = opaque;
+
+    /* Make sure all file formats flush their mutable metadata */
+    bdrv_invalidate_cache_all(&local_err);
+    if (local_err) {
+        migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
+                          MIGRATION_STATUS_FAILED);
+        error_report_err(local_err);
+        migrate_decompress_threads_join();
+        exit(EXIT_FAILURE);
+    }
+
+    /*
+     * This must happen after all error conditions are dealt with and
+     * we're sure the VM is going to be running on this host.
+     */
+    qemu_announce_self();
+
+    /* If global state section was not received or we are in running
+       state, we need to obey autostart. Any other state is set with
+       runstate_set. */
+
+    if (!global_state_received() ||
+        global_state_get_runstate() == RUN_STATE_RUNNING) {
+        if (autostart) {
+            vm_start();
+        } else {
+            runstate_set(RUN_STATE_PAUSED);
+        }
+    } else {
+        runstate_set(global_state_get_runstate());
+    }
+    migrate_decompress_threads_join();
+    /*
+     * This must happen after any state changes since as soon as an external
+     * observer sees this event they might start to prod at the VM assuming
+     * it's ready to use.
+     */
+    migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
+                      MIGRATION_STATUS_COMPLETED);
+    migration_incoming_state_destroy();
+
+    qemu_bh_delete(mis->bh);
+}
+
 static void process_incoming_migration_co(void *opaque)
 {
     QEMUFile *f = opaque;
-    Error *local_err = NULL;
     MigrationIncomingState *mis;
     PostcopyState ps;
     int ret;
@@ -369,45 +416,8 @@ static void process_incoming_migration_co(void *opaque)
         exit(EXIT_FAILURE);
     }
 
-    /* Make sure all file formats flush their mutable metadata */
-    bdrv_invalidate_cache_all(&local_err);
-    if (local_err) {
-        migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
-                          MIGRATION_STATUS_FAILED);
-        error_report_err(local_err);
-        migrate_decompress_threads_join();
-        exit(EXIT_FAILURE);
-    }
-
-    /*
-     * This must happen after all error conditions are dealt with and
-     * we're sure the VM is going to be running on this host.
-     */
-    qemu_announce_self();
-
-    /* If global state section was not received or we are in running
-       state, we need to obey autostart. Any other state is set with
-       runstate_set. */
-
-    if (!global_state_received() ||
-        global_state_get_runstate() == RUN_STATE_RUNNING) {
-        if (autostart) {
-            vm_start();
-        } else {
-            runstate_set(RUN_STATE_PAUSED);
-        }
-    } else {
-        runstate_set(global_state_get_runstate());
-    }
-    migrate_decompress_threads_join();
-    /*
-     * This must happen after any state changes since as soon as an external
-     * observer sees this event they might start to prod at the VM assuming
-     * it's ready to use.
-     */
-    migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
-                      MIGRATION_STATUS_COMPLETED);
-    migration_incoming_state_destroy();
+    mis->bh = qemu_bh_new(process_incoming_migration_bh, mis);
+    qemu_bh_schedule(mis->bh);
 }
 
 void process_incoming_migration(QEMUFile *f)
-- 
2.5.0

[Qemu-devel] [PULL 4/6] migration (postcopy): move bdrv_invalidate_cache_all of of coroutine context

[Amit Shah]

From: "Denis V. Lunev" <den@openvz.org>

There is a possibility to hit an assert in qcow2_get_specific_info that
s->qcow_version is undefined. This happens when VM in starting from
suspended state, i.e. it processes incoming migration, and in the same
time 'info block' is called.

The problem is that qcow2_invalidate_cache() closes the image and
memset()s BDRVQcowState in the middle.

The patch moves processing of bdrv_invalidate_cache_all out of
coroutine context for postcopy migration to avoid that. This function
is called with the following stack:
  process_incoming_migration_co
  qemu_loadvm_state
  qemu_loadvm_state_main
  loadvm_process_command
  loadvm_postcopy_handle_run

Signed-off-by: Denis V. Lunev <den@openvz.org>
Tested-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
CC: Paolo Bonzini <pbonzini@redhat.com>
CC: Juan Quintela <quintela@redhat.com>
CC: Amit Shah <amit.shah@redhat.com>
Message-Id: <1456304019-10507-3-git-send-email-den@openvz.org>
Signed-off-by: Amit Shah <amit.shah@redhat.com>
---
 migration/savevm.c | 29 +++++++++++++++++++----------
 1 file changed, 19 insertions(+), 10 deletions(-)

diff --git a/migration/savevm.c b/migration/savevm.c
index 02e8487..b459156 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1495,17 +1495,10 @@ static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)
     return 0;
 }
 
-/* After all discards we can start running and asking for pages */
-static int loadvm_postcopy_handle_run(MigrationIncomingState *mis)
+static void loadvm_postcopy_handle_run_bh(void *opaque)
 {
-    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_RUNNING);
     Error *local_err = NULL;
-
-    trace_loadvm_postcopy_handle_run();
-    if (ps != POSTCOPY_INCOMING_LISTENING) {
-        error_report("CMD_POSTCOPY_RUN in wrong postcopy state (%d)", ps);
-        return -1;
-    }
+    MigrationIncomingState *mis = opaque;
 
     /* TODO we should move all of this lot into postcopy_ram.c or a shared code
      * in migration.c
@@ -1518,7 +1511,6 @@ static int loadvm_postcopy_handle_run(MigrationIncomingState *mis)
     bdrv_invalidate_cache_all(&local_err);
     if (local_err) {
         error_report_err(local_err);
-        return -1;
     }
 
     trace_loadvm_postcopy_handle_run_cpu_sync();
@@ -1534,6 +1526,23 @@ static int loadvm_postcopy_handle_run(MigrationIncomingState *mis)
         runstate_set(RUN_STATE_PAUSED);
     }
 
+    qemu_bh_delete(mis->bh);
+}
+
+/* After all discards we can start running and asking for pages */
+static int loadvm_postcopy_handle_run(MigrationIncomingState *mis)
+{
+    PostcopyState ps = postcopy_state_set(POSTCOPY_INCOMING_RUNNING);
+
+    trace_loadvm_postcopy_handle_run();
+    if (ps != POSTCOPY_INCOMING_LISTENING) {
+        error_report("CMD_POSTCOPY_RUN in wrong postcopy state (%d)", ps);
+        return -1;
+    }
+
+    mis->bh = qemu_bh_new(loadvm_postcopy_handle_run_bh, NULL);
+    qemu_bh_schedule(mis->bh);
+
     /* We need to finish reading the stream from the package
      * and also stop reading anything more from the stream that loaded the
      * package (since it's now being read by the listener thread).
-- 
2.5.0

[Qemu-devel] [PULL 5/6] MAINTAINERS: Add docs/migration.txt to the "Migration" section

[Amit Shah]

From: Thomas Huth <thuth@redhat.com>

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Amit Shah <amit.shah@redhat.com>
Message-Id: <1456393669-20678-1-git-send-email-thuth@redhat.com>
Signed-off-by: Amit Shah <amit.shah@redhat.com>
---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 606d9c0..5fafa81 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1241,6 +1241,7 @@ F: include/migration/
 F: migration/
 F: scripts/vmstate-static-checker.py
 F: tests/vmstate-static-checker-data/
+F: docs/migration.txt
 
 Seccomp
 M: Eduardo Otubo <eduardo.otubo@profitbricks.com>
-- 
2.5.0

[Qemu-devel] [PULL 6/6] migration/vmstate: document VMStateFlags

[Amit Shah]

From: Sascha Silbe <silbe@linux.vnet.ibm.com>

The VMState API is rather sparsely documented. Start by describing the
meaning of all VMStateFlags.

Reviewed-by: Amit Shah <amit.shah@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
Message-Id: <1456474693-11662-1-git-send-email-silbe@linux.vnet.ibm.com>
Signed-off-by: Amit Shah <amit.shah@redhat.com>
---
 include/migration/vmstate.h | 100 +++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 90 insertions(+), 10 deletions(-)

diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index 7246f29..84ee355 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -88,21 +88,101 @@ struct VMStateInfo {
 };
 
 enum VMStateFlags {
+    /* Ignored */
     VMS_SINGLE           = 0x001,
+
+    /* The struct member at opaque + VMStateField.offset is a pointer
+     * to the actual field (e.g. struct a { uint8_t *b;
+     * }). Dereference the pointer before using it as basis for
+     * further pointer arithmetic (see e.g. VMS_ARRAY). Does not
+     * affect the meaning of VMStateField.num_offset or
+     * VMStateField.size_offset; see VMS_VARRAY* and VMS_VBUFFER for
+     * those. */
     VMS_POINTER          = 0x002,
+
+    /* The field is an array of fixed size. VMStateField.num contains
+     * the number of entries in the array. The size of each entry is
+     * given by VMStateField.size and / or opaque +
+     * VMStateField.size_offset; see VMS_VBUFFER and
+     * VMS_MULTIPLY. Each array entry will be processed individually
+     * (VMStateField.info.get()/put() if VMS_STRUCT is not set,
+     * recursion into VMStateField.vmsd if VMS_STRUCT is set). May not
+     * be combined with VMS_VARRAY*. */
     VMS_ARRAY            = 0x004,
+
+    /* The field is itself a struct, containing one or more
+     * fields. Recurse into VMStateField.vmsd. Most useful in
+     * combination with VMS_ARRAY / VMS_VARRAY*, recursing into each
+     * array entry. */
     VMS_STRUCT           = 0x008,
-    VMS_VARRAY_INT32     = 0x010,  /* Array with size in int32_t field*/
-    VMS_BUFFER           = 0x020,  /* static sized buffer */
+
+    /* The field is an array of variable size. The int32_t at opaque +
+     * VMStateField.num_offset contains the number of entries in the
+     * array. See the VMS_ARRAY description regarding array handling
+     * in general. May not be combined with VMS_ARRAY or any other
+     * VMS_VARRAY*. */
+    VMS_VARRAY_INT32     = 0x010,
+
+    /* Ignored */
+    VMS_BUFFER           = 0x020,
+
+    /* The field is a (fixed-size or variable-size) array of pointers
+     * (e.g. struct a { uint8_t *b[]; }). Dereference each array entry
+     * before using it. Note: Does not imply any one of VMS_ARRAY /
+     * VMS_VARRAY*; these need to be set explicitly. */
     VMS_ARRAY_OF_POINTER = 0x040,
-    VMS_VARRAY_UINT16    = 0x080,  /* Array with size in uint16_t field */
-    VMS_VBUFFER          = 0x100,  /* Buffer with size in int32_t field */
-    VMS_MULTIPLY         = 0x200,  /* multiply "size" field by field_size */
-    VMS_VARRAY_UINT8     = 0x400,  /* Array with size in uint8_t field*/
-    VMS_VARRAY_UINT32    = 0x800,  /* Array with size in uint32_t field*/
-    VMS_MUST_EXIST       = 0x1000, /* Field must exist in input */
-    VMS_ALLOC            = 0x2000, /* Alloc a buffer on the destination */
-    VMS_MULTIPLY_ELEMENTS = 0x4000, /* multiply varray size by field->num */
+
+    /* The field is an array of variable size. The uint16_t at opaque
+     * + VMStateField.num_offset (subject to VMS_MULTIPLY_ELEMENTS)
+     * contains the number of entries in the array. See the VMS_ARRAY
+     * description regarding array handling in general. May not be
+     * combined with VMS_ARRAY or any other VMS_VARRAY*. */
+    VMS_VARRAY_UINT16    = 0x080,
+
+    /* The size of the individual entries (a single array entry if
+     * VMS_ARRAY or any of VMS_VARRAY* are set, or the field itself if
+     * neither is set) is variable (i.e. not known at compile-time),
+     * but the same for all entries. Use the int32_t at opaque +
+     * VMStateField.size_offset (subject to VMS_MULTIPLY) to determine
+     * the size of each (and every) entry. */
+    VMS_VBUFFER          = 0x100,
+
+    /* Multiply the entry size given by the int32_t at opaque +
+     * VMStateField.size_offset (see VMS_VBUFFER description) with
+     * VMStateField.size to determine the number of bytes to be
+     * allocated. Only valid in combination with VMS_VBUFFER. */
+    VMS_MULTIPLY         = 0x200,
+
+    /* The field is an array of variable size. The uint8_t at opaque +
+     * VMStateField.num_offset (subject to VMS_MULTIPLY_ELEMENTS)
+     * contains the number of entries in the array. See the VMS_ARRAY
+     * description regarding array handling in general. May not be
+     * combined with VMS_ARRAY or any other VMS_VARRAY*. */
+    VMS_VARRAY_UINT8     = 0x400,
+
+    /* The field is an array of variable size. The uint32_t at opaque
+     * + VMStateField.num_offset (subject to VMS_MULTIPLY_ELEMENTS)
+     * contains the number of entries in the array. See the VMS_ARRAY
+     * description regarding array handling in general. May not be
+     * combined with VMS_ARRAY or any other VMS_VARRAY*. */
+    VMS_VARRAY_UINT32    = 0x800,
+
+    /* Fail loading the serialised VM state if this field is missing
+     * from the input. */
+    VMS_MUST_EXIST       = 0x1000,
+
+    /* When loading serialised VM state, allocate memory for the
+     * (entire) field. Only valid in combination with
+     * VMS_POINTER. Note: Not all combinations with other flags are
+     * currently supported, e.g. VMS_ALLOC|VMS_ARRAY_OF_POINTER won't
+     * cause the individual entries to be allocated. */
+    VMS_ALLOC            = 0x2000,
+
+    /* Multiply the number of entries given by the integer at opaque +
+     * VMStateField.num_offset (see VMS_VARRAY*) with VMStateField.num
+     * to determine the number of entries in the array. Only valid in
+     * combination with one of VMS_VARRAY*. */
+    VMS_MULTIPLY_ELEMENTS = 0x4000,
 };
 
 typedef struct {
-- 
2.5.0

Re: [Qemu-devel] [PULL 0/6] migration pull

[Peter Maydell]

On 26 February 2016 at 09:55, Amit Shah <amit.shah@redhat.com> wrote:
> The following changes since commit 0c6940d086f39bbf725d96104abe46da87429cb6:
>
>   build: [bsd-user] Rename "syscall.h" to "target_syscall.h" in target directories (2016-02-25 16:41:08 +0000)
>
> are available in the git repository at:
>
>   https://git.kernel.org/pub/scm/virt/qemu/amit/migration.git tags/migration-for-2.6-4
>
> for you to fetch changes up to aa81a236c8dc8fe18e26d9dadeb759dc53ee7e97:
>
>   migration/vmstate: document VMStateFlags (2016-02-26 15:09:52 +0530)
>
> ----------------------------------------------------------------
> migration pull
>  - fix a qcow2 assert
>  - fix for older distros (CentOS 5)
>  - documentation for vmstate flags
>  - minor code rearrangement
>

Hi. I'm afraid this causes make check to fail:

  /i386/ahci/flush/migrate:
Broken pipe
FAIL
GTester: last random seed: R02Se9947e491366b9d0992d3370999cd7b4
(pid=56749)
  /i386/ahci/migrate/sanity:
Broken pipe
FAIL
GTester: last random seed: R02S061c85bc582235ae136311cea53859ad
(pid=56763)
  /i386/ahci/migrate/dma/simple:
Broken pipe
FAIL
GTester: last random seed: R02S89869ed268c2e9d8eb479d6a01ec8000
(pid=56777)
  /i386/ahci/migrate/dma/halted:
Broken pipe
FAIL
GTester: last random seed: R02S217e72365f9791944c71ab3755893bb8
(pid=56791)
  /i386/ahci/migrate/ncq/simple:
Broken pipe
FAIL
GTester: last random seed: R02S31646b42a8c63966c6547de7ed50e191
(pid=56806)
  /i386/ahci/migrate/ncq/halted:
Broken pipe
FAIL
GTester: last random seed: R02S34da744548f394684dbb7b6d11e14043
(pid=56820)

(failure seen on ppc64be, x86-64 linux, aarch64, but not aarch32 or OSX).

thanks
-- PMM

Re: [Qemu-devel] [PULL 0/6] migration pull

[Amit Shah]

On (Fri) 26 Feb 2016 [12:23:01], Peter Maydell wrote:
> On 26 February 2016 at 09:55, Amit Shah <amit.shah@redhat.com> wrote:
> > The following changes since commit 0c6940d086f39bbf725d96104abe46da87429cb6:
> >
> >   build: [bsd-user] Rename "syscall.h" to "target_syscall.h" in target directories (2016-02-25 16:41:08 +0000)
> >
> > are available in the git repository at:
> >
> >   https://git.kernel.org/pub/scm/virt/qemu/amit/migration.git tags/migration-for-2.6-4
> >
> > for you to fetch changes up to aa81a236c8dc8fe18e26d9dadeb759dc53ee7e97:
> >
> >   migration/vmstate: document VMStateFlags (2016-02-26 15:09:52 +0530)
> >
> > ----------------------------------------------------------------
> > migration pull
> >  - fix a qcow2 assert
> >  - fix for older distros (CentOS 5)
> >  - documentation for vmstate flags
> >  - minor code rearrangement
> >
> 
> Hi. I'm afraid this causes make check to fail:

Oops, I should've caught that.

>   /i386/ahci/flush/migrate:

>   /i386/ahci/migrate/sanity:

>   /i386/ahci/migrate/dma/simple:

>   /i386/ahci/migrate/dma/halted:

>   /i386/ahci/migrate/ncq/simple:

>   /i386/ahci/migrate/ncq/halted:
> Broken pipe
> FAIL
> GTester: last random seed: R02S34da744548f394684dbb7b6d11e14043
> (pid=56820)
> 
> (failure seen on ppc64be, x86-64 linux, aarch64, but not aarch32 or OSX).

AHCI tests are failing due to Denis's patches.  I'm checking why.

		Amit

Re: [Qemu-devel] [PULL 0/6] migration pull

[Denis V. Lunev]

On 02/26/2016 04:15 PM, Amit Shah wrote:
> On (Fri) 26 Feb 2016 [12:23:01], Peter Maydell wrote:
>> On 26 February 2016 at 09:55, Amit Shah <amit.shah@redhat.com> wrote:
>>> The following changes since commit 0c6940d086f39bbf725d96104abe46da87429cb6:
>>>
>>>    build: [bsd-user] Rename "syscall.h" to "target_syscall.h" in target directories (2016-02-25 16:41:08 +0000)
>>>
>>> are available in the git repository at:
>>>
>>>    https://git.kernel.org/pub/scm/virt/qemu/amit/migration.git tags/migration-for-2.6-4
>>>
>>> for you to fetch changes up to aa81a236c8dc8fe18e26d9dadeb759dc53ee7e97:
>>>
>>>    migration/vmstate: document VMStateFlags (2016-02-26 15:09:52 +0530)
>>>
>>> ----------------------------------------------------------------
>>> migration pull
>>>   - fix a qcow2 assert
>>>   - fix for older distros (CentOS 5)
>>>   - documentation for vmstate flags
>>>   - minor code rearrangement
>>>
>> Hi. I'm afraid this causes make check to fail:
> Oops, I should've caught that.
>
>>    /i386/ahci/flush/migrate:
>>    /i386/ahci/migrate/sanity:
>>    /i386/ahci/migrate/dma/simple:
>>    /i386/ahci/migrate/dma/halted:
>>    /i386/ahci/migrate/ncq/simple:
>>    /i386/ahci/migrate/ncq/halted:
>> Broken pipe
>> FAIL
>> GTester: last random seed: R02S34da744548f394684dbb7b6d11e14043
>> (pid=56820)
>>
>> (failure seen on ppc64be, x86-64 linux, aarch64, but not aarch32 or OSX).
> AHCI tests are failing due to Denis's patches.  I'm checking why.
>
> 		Amit
oops. Interesting :(

Re: [Qemu-devel] [PULL 0/6] migration pull

[Amit Shah]

On (Fri) 26 Feb 2016 [17:58:34], Denis V. Lunev wrote:
> On 02/26/2016 04:15 PM, Amit Shah wrote:
> >On (Fri) 26 Feb 2016 [12:23:01], Peter Maydell wrote:

> >>Hi. I'm afraid this causes make check to fail:
> >Oops, I should've caught that.
> >
> >>   /i386/ahci/flush/migrate:
> >>   /i386/ahci/migrate/sanity:
> >>   /i386/ahci/migrate/dma/simple:
> >>   /i386/ahci/migrate/dma/halted:
> >>   /i386/ahci/migrate/ncq/simple:
> >>   /i386/ahci/migrate/ncq/halted:
> >>Broken pipe
> >>FAIL
> >>GTester: last random seed: R02S34da744548f394684dbb7b6d11e14043
> >>(pid=56820)
> >>
> >>(failure seen on ppc64be, x86-64 linux, aarch64, but not aarch32 or OSX).
> >AHCI tests are failing due to Denis's patches.  I'm checking why.
> >
> >		Amit
> oops. Interesting :(

Right - use-after-free; this on top of your patch fixes it.


diff --git a/migration/migration.c b/migration/migration.c
index f82fdf6..0129d9f 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -366,9 +366,8 @@ static void process_incoming_migration_bh(void *opaque)
      */
     migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
                       MIGRATION_STATUS_COMPLETED);
-    migration_incoming_state_destroy();
-
     qemu_bh_delete(mis->bh);
+    migration_incoming_state_destroy();
 }
 
 static void process_incoming_migration_co(void *opaque)


		Amit

Re: [Qemu-devel] [PULL 0/6] migration pull

[Denis V. Lunev]

On 02/26/2016 06:14 PM, Amit Shah wrote:
> On (Fri) 26 Feb 2016 [17:58:34], Denis V. Lunev wrote:
>> On 02/26/2016 04:15 PM, Amit Shah wrote:
>>> On (Fri) 26 Feb 2016 [12:23:01], Peter Maydell wrote:
>>>> Hi. I'm afraid this causes make check to fail:
>>> Oops, I should've caught that.
>>>
>>>>    /i386/ahci/flush/migrate:
>>>>    /i386/ahci/migrate/sanity:
>>>>    /i386/ahci/migrate/dma/simple:
>>>>    /i386/ahci/migrate/dma/halted:
>>>>    /i386/ahci/migrate/ncq/simple:
>>>>    /i386/ahci/migrate/ncq/halted:
>>>> Broken pipe
>>>> FAIL
>>>> GTester: last random seed: R02S34da744548f394684dbb7b6d11e14043
>>>> (pid=56820)
>>>>
>>>> (failure seen on ppc64be, x86-64 linux, aarch64, but not aarch32 or OSX).
>>> AHCI tests are failing due to Denis's patches.  I'm checking why.
>>>
>>> 		Amit
>> oops. Interesting :(
> Right - use-after-free; this on top of your patch fixes it.
>
>
> diff --git a/migration/migration.c b/migration/migration.c
> index f82fdf6..0129d9f 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -366,9 +366,8 @@ static void process_incoming_migration_bh(void *opaque)
>        */
>       migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
>                         MIGRATION_STATUS_COMPLETED);
> -    migration_incoming_state_destroy();
> -
>       qemu_bh_delete(mis->bh);
> +    migration_incoming_state_destroy();
>   }
>   
>   static void process_incoming_migration_co(void *opaque)
>
>
> 		Amit
sorry :( last minute changes are bad...

Re: [Qemu-devel] [PULL 0/6] Migration pull

[Peter Maydell]

On 8 February 2017 at 08:47, Thomas Huth <thuth@redhat.com> wrote:
> How fast is that aarch64 machine?

It's a juno board, so it's not very fast, but I wouldn't
have expected it to hit a 100 second timeout assuming
the test isn't trying to do a ludicrous amount of work.

> The timeout of the prom-env test is
> already 100 seconds, so that seems to be plenty to me (the test normally
> finishes here on my laptop within 15 seconds).
> OTOH, if it helps with the aarch64 machine, I'm also fine if we increase
> the timeout further - but what value would be appropriate then? Could you
> please run the following on that slow machine:
>
>  time QTEST_QEMU_BINARY=ppc64-softmmu/qemu-system-ppc64 tests/prom-env-test

/ppc64/prom-env/mac99: OK
/ppc64/prom-env/g3beige: OK
/ppc64/prom-env/pseries: OK

real    1m37.657s
user    1m36.748s
sys     0m1.440s

Breaking that down per testcase:

<tmmu/qemu-system-ppc64 tests/prom-env-test -v -p /ppc64/prom-env/mac99
/ppc64/prom-env/mac99: OK

real    0m9.836s
user    0m9.584s
sys     0m0.424s
<u/qemu-system-ppc64 tests/prom-env-test -v -p /ppc64/prom-env/g3beige
/ppc64/prom-env/g3beige: OK

real    0m9.099s
user    0m8.988s
sys     0m0.144s
<qemu-system-ppc64 tests/prom-env-test -v -p /ppc64/prom-env/pseries
/ppc64/prom-env/pseries: OK

real    1m24.362s
user    1m23.572s
sys     0m1.144s

pseries is much slower than the other two and at 85 seconds is
not all that much below the 100s. Something's odd here.

thanks
-- PMM

Re: [Qemu-devel] [PULL 0/6] Migration pull

[Thomas Huth]

On 07.02.2017 18:47, Peter Maydell wrote:
> On 7 February 2017 at 07:50, Juan Quintela <quintela@redhat.com> wrote:
>> Peter Maydell <peter.maydell@linaro.org> wrote:
>>> On 6 February 2017 at 16:51, Juan Quintela <quintela@redhat.com> wrote:
>>
>>> I get a test failure on aarch64 host on one of the ppc guest tests:
>>>
>>> TEST: tests/prom-env-test... (pid=26435)
>>>   /ppc64/prom-env/mac99:                                               OK
>>>   /ppc64/prom-env/g3beige:                                             OK
>>>   /ppc64/prom-env/pseries:                                             **
>>> ERROR:/home/petmay01/qemu/tests/prom-env-test.c:42:check_guest_memory:
>>> assertion failed (signature == MAGIC): (0x7c7f1b78 == 0xcafec0de)
>>> FAIL
>>> GTester: last random seed: R02Sbd97cf9617f5dd55ba90b08765cfa0d0
>>> (pid=26445)
>>> FAIL: tests/prom-env-test
>>>
>>>
>>> This might just be the test having an overzealous timeout (I
>>> think I've seen this before a while back, when I was trying
>>> to do a 'make -j2 check' on this slow board); but on the other
>>> hand it's been fine recently but it failed twice trying to test
>>> this pullreq.
>>
>> I don't have n aarch64 host, all I can say is that make check worked on
>> x86_64.
>>
>> I would expect that this didn't change anything for prom enviroment, but
>> one never knows :p
>>
>> I will wait to see if ppc folks can take a look at this.
> 
> I tried this pull again, and got two good test runs in a row,
> so I suspect this is just the host being a bit slow and the
> timeout being a bit enthusiastic.

How fast is that aarch64 machine? The timeout of the prom-env test is
already 100 seconds, so that seems to be plenty to me (the test normally
finishes here on my laptop within 15 seconds).
OTOH, if it helps with the aarch64 machine, I'm also fine if we increase
the timeout further - but what value would be appropriate then? Could you
please run the following on that slow machine:

 time QTEST_QEMU_BINARY=ppc64-softmmu/qemu-system-ppc64 tests/prom-env-test

Thomas

Re: [Qemu-devel] [PULL 0/6] Migration pull

[Peter Maydell]

On 7 February 2017 at 07:50, Juan Quintela <quintela@redhat.com> wrote:
> Peter Maydell <peter.maydell@linaro.org> wrote:
>> On 6 February 2017 at 16:51, Juan Quintela <quintela@redhat.com> wrote:
>
>
>> I get a test failure on aarch64 host on one of the ppc guest tests:
>>
>> TEST: tests/prom-env-test... (pid=26435)
>>   /ppc64/prom-env/mac99:                                               OK
>>   /ppc64/prom-env/g3beige:                                             OK
>>   /ppc64/prom-env/pseries:                                             **
>> ERROR:/home/petmay01/qemu/tests/prom-env-test.c:42:check_guest_memory:
>> assertion failed (signature == MAGIC): (0x7c7f1b78 == 0xcafec0de)
>> FAIL
>> GTester: last random seed: R02Sbd97cf9617f5dd55ba90b08765cfa0d0
>> (pid=26445)
>> FAIL: tests/prom-env-test
>>
>>
>> This might just be the test having an overzealous timeout (I
>> think I've seen this before a while back, when I was trying
>> to do a 'make -j2 check' on this slow board); but on the other
>> hand it's been fine recently but it failed twice trying to test
>> this pullreq.
>
> I don't have n aarch64 host, all I can say is that make check worked on
> x86_64.
>
> I would expect that this didn't change anything for prom enviroment, but
> one never knows :p
>
> I will wait to see if ppc folks can take a look at this.

I tried this pull again, and got two good test runs in a row,
so I suspect this is just the host being a bit slow and the
timeout being a bit enthusiastic.

Applied to master.

thanks
-- PMM

Re: [Qemu-devel] [PULL 0/6] Migration pull

[Juan Quintela]

Peter Maydell <peter.maydell@linaro.org> wrote:
> On 6 February 2017 at 16:51, Juan Quintela <quintela@redhat.com> wrote:


> I get a test failure on aarch64 host on one of the ppc guest tests:
>
> TEST: tests/prom-env-test... (pid=26435)
>   /ppc64/prom-env/mac99:                                               OK
>   /ppc64/prom-env/g3beige:                                             OK
>   /ppc64/prom-env/pseries:                                             **
> ERROR:/home/petmay01/qemu/tests/prom-env-test.c:42:check_guest_memory:
> assertion failed (signature == MAGIC): (0x7c7f1b78 == 0xcafec0de)
> FAIL
> GTester: last random seed: R02Sbd97cf9617f5dd55ba90b08765cfa0d0
> (pid=26445)
> FAIL: tests/prom-env-test
>
>
> This might just be the test having an overzealous timeout (I
> think I've seen this before a while back, when I was trying
> to do a 'make -j2 check' on this slow board); but on the other
> hand it's been fine recently but it failed twice trying to test
> this pullreq.

I don't have n aarch64 host, all I can say is that make check worked on
x86_64.

I would expect that this didn't change anything for prom enviroment, but
one never knows :p

I will wait to see if ppc folks can take a look at this.

Thanks, Juan.

Re: [Qemu-devel] [PULL 0/6] Migration pull

[Peter Maydell]

On 6 February 2017 at 16:51, Juan Quintela <quintela@redhat.com> wrote:
> Hi
>
> This are the current migration pull:
> - fix long id's for migration (David)
> - several postcopy fixes (David)
> - create the incomming state in a single place for Migraition (me)
>
> Please, apply.
>
> The following changes since commit 7d2c6c95511e42dffe2b263275e09957723d0ff4:
>
>   Merge remote-tracking branch 'remotes/kraxel/tags/pull-usb-20170206-1' into staging (2017-02-06 11:44:08 +0000)
>
> are available in the git repository at:
>
>   git://github.com/juanquintela/qemu.git tags/migration/20170206
>
> for you to fetch changes up to ef8d6488d2767fe81bb4bb9bcdc52af5ff718b56:
>
>   postcopy: Recover block devices on early failure (2017-02-06 13:36:49 +0100)
>
> ----------------------------------------------------------------
> migration/next for 20170206
>
> ----------------------------------------------------------------
> Dr. David Alan Gilbert (5):
>       vmstate_register_with_alias_id: Take an Error **
>       migration: Check for ID length
>       vmstate registration: check return values
>       Postcopy: Reset state to avoid cleanup assert
>       postcopy: Recover block devices on early failure
>
> Juan Quintela (1):
>       migration: create Migration Incoming State at init time

I get a test failure on aarch64 host on one of the ppc guest tests:

TEST: tests/prom-env-test... (pid=26435)
  /ppc64/prom-env/mac99:                                               OK
  /ppc64/prom-env/g3beige:                                             OK
  /ppc64/prom-env/pseries:                                             **
ERROR:/home/petmay01/qemu/tests/prom-env-test.c:42:check_guest_memory:
assertion failed (signature == MAGIC): (0x7c7f1b78 == 0xcafec0de)
FAIL
GTester: last random seed: R02Sbd97cf9617f5dd55ba90b08765cfa0d0
(pid=26445)
FAIL: tests/prom-env-test


This might just be the test having an overzealous timeout (I
think I've seen this before a while back, when I was trying
to do a 'make -j2 check' on this slow board); but on the other
hand it's been fine recently but it failed twice trying to test
this pullreq.

thanks
-- PMM

[Qemu-devel] [PULL 0/6] Migration pull

[Juan Quintela]

Hi

This are the current migration pull:
- fix long id's for migration (David)
- several postcopy fixes (David)
- create the incomming state in a single place for Migraition (me)

Please, apply.

The following changes since commit 7d2c6c95511e42dffe2b263275e09957723d0ff4:

  Merge remote-tracking branch 'remotes/kraxel/tags/pull-usb-20170206-1' into staging (2017-02-06 11:44:08 +0000)

are available in the git repository at:

  git://github.com/juanquintela/qemu.git tags/migration/20170206

for you to fetch changes up to ef8d6488d2767fe81bb4bb9bcdc52af5ff718b56:

  postcopy: Recover block devices on early failure (2017-02-06 13:36:49 +0100)

----------------------------------------------------------------
migration/next for 20170206

----------------------------------------------------------------
Dr. David Alan Gilbert (5):
      vmstate_register_with_alias_id: Take an Error **
      migration: Check for ID length
      vmstate registration: check return values
      Postcopy: Reset state to avoid cleanup assert
      postcopy: Recover block devices on early failure

Juan Quintela (1):
      migration: create Migration Incoming State at init time

 hw/core/qdev.c                |  9 ++++---
 hw/intc/apic_common.c         |  2 +-
 include/migration/migration.h |  1 -
 include/migration/vmstate.h   |  7 +++--
 migration/migration.c         | 63 ++++++++++++++++++++++++++++---------------
 migration/savevm.c            | 29 ++++++++++++++------
 stubs/vmstate.c               |  3 ++-
 7 files changed, 77 insertions(+), 37 deletions(-)