[Qemu-devel] [RFC PATCH 00/14] Block replication for continuous checkpoints

[Qemu-devel] [RFC PATCH 00/14] Block replication for continuous checkpoints

[Wen Congyang]

Block replication is a very important feature which is used for
continuous checkpoints(for example: COLO).

Usage:
Primary:
  -drive if=xxx,driver=quorum,read-pattern=first,\
         children.0.file.filename=1.raw,\
         children.0.driver=raw,\
         children.1.file.driver=nbd+colo,\
         children.1.file.host=xxx,\
         children.1.file.port=xxx,\
         children.1.file.export=xxx,\
         children.1.driver=raw
  Note:
  1. NBD Client should not be the first child of quorum.
  2. There should be only one NBD Client.
  3. host is the secondary physical machine's hostname or IP
  4. Each disk must have its own export name.

Secondary:
  -drive if=xxx,driver=blkcolo,export=xxx,\
         backing.file.filename=1.raw,\
         backing.driver=raw
  Then run qmp command:
    nbd_server_start host:port
  Note:
  1. The export name for the same disk must be the same in primary
     and secondary QEMU command line
  2. The qmp command nbd_server_start must be run before running the
     qmp command migrate on primary QEMU
  3. Don't use nbd_server_start's other options

You can get the patch here:
https://github.com/wencongyang/qemu-colo/commits/block-replication-v1

Wen Congyang (14):
  docs: block replication's description
  quorom: add a new read pattern
  quorum: ignore 0-length child
  Add new block driver interfaces to control disk replication
  quorom: implement block driver interfaces for block replication
  NBD client: connect to nbd server later
  NBD client: implement block driver interfaces for block replication
  block: add a new API to create a hidden BlockBackend
  block: give backing image its own BlockBackend
  allow the backing image access the origin BlockDriverState
  allow writing to the backing file
  Add disk buffer for block replication
  COW: move cow interfaces to a seperate file
  COLO: implement a new block driver

 Makefile.objs                  |   2 +-
 block.c                        |  53 +++++-
 block/Makefile.objs            |   1 +
 block/backup.c                 |  52 +-----
 block/blkcolo-buffer.c         | 324 ++++++++++++++++++++++++++++++++
 block/blkcolo.c                | 409 +++++++++++++++++++++++++++++++++++++++++
 block/blkcolo.h                |  35 ++++
 block/block-backend.c          |  29 ++-
 block/nbd.c                    | 155 ++++++++++++++--
 block/quorum.c                 |  79 +++++++-
 blockcow.c                     |  52 ++++++
 docs/block-replication.txt     | 129 +++++++++++++
 include/block/block.h          |  27 +++
 include/block/block_int.h      |  14 ++
 include/sysemu/block-backend.h |   2 +
 qapi/block-core.json           |   4 +-
 16 files changed, 1295 insertions(+), 72 deletions(-)
 create mode 100644 block/blkcolo-buffer.c
 create mode 100644 block/blkcolo.c
 create mode 100644 block/blkcolo.h
 create mode 100644 blockcow.c
 create mode 100644 docs/block-replication.txt

-- 
2.1.0

[Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
 docs/block-replication.txt | 129 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 129 insertions(+)
 create mode 100644 docs/block-replication.txt

diff --git a/docs/block-replication.txt b/docs/block-replication.txt
new file mode 100644
index 0000000..59150b8
--- /dev/null
+++ b/docs/block-replication.txt
@@ -0,0 +1,129 @@
+Block replication
+----------------------------------------
+Copyright Fujitsu, Corp. 2015
+Copyright (c) 2015 Intel Corporation
+Copyright (c) 2015 HUAWEI TECHNOLOGIES CO.,LTD.
+
+This work is licensed under the terms of the GNU GPL, version 2 or later.
+See the COPYING file in the top-level directory.
+
+The block replication is used for continuous checkpoints. It is designed
+for COLO that Secondary VM is running. It can also be applied for FT/HA
+scene that Secondary VM is not running.
+
+This document gives an overview of block replication's design.
+
+== Background ==
+High availability solutions such as micro checkpoint and COLO will do
+consecutive checkpoint. The VM state of Primary VM and Secondary VM is
+identical right after a VM checkpoint, but becomes different as the VM
+executes till the next checkpoint. To support disk contents checkpoint,
+the modified disk contents in the Secondary VM must be buffered, and are
+only dropped at next checkpoint time. To reduce the network transportation
+effort at the time of checkpoint, the disk modification operations of
+Primary disk are asynchronously forwarded to the Secondary node.
+
+== Workflow ==
+The following is the image of block replication workflow:
+
+        +----------------------+            +------------------------+
+        |Primary Write Requests|            |Secondary Write Requests|
+        +----------------------+            +------------------------+
+                  |                                       |
+                  |                                      (4)
+                  |                                       V
+                  |                              /-------------\
+                  |      Copy and Forward        |             |
+                  |---------(1)----------+       | Disk Buffer |
+                  |                      |       |             |
+                  |                     (3)      \-------------/
+                  |                 speculative      ^
+                  |                write through    (2)
+                  |                      |           |
+                  V                      V           |
+           +--------------+           +----------------+
+           | Primary Disk |           | Secondary Disk |
+           +--------------+           +----------------+
+
+    1) Primary write requests will be copied and forwarded to Secondary
+       QEMU.
+    2) Before Primary write requests are written to Secondary disk, the
+       original sector content will be read from Secondary disk and
+       buffered in the Disk buffer, but it will not overwrite the existing
+       sector content in the Disk buffer.
+    3) Primary write requests will be written to Secondary disk.
+    4) Secondary write requests will be buffered in the Disk buffer and it
+       will overwrite the existing sector content in the buffer.
+
+== Architecture ==
+We are going to implement COLO block replication from many basic
+blocks that are already in QEMU.
+
+         virtio-blk       ||
+             ^            ||                            .----------
+             |            ||                            | Secondary
+        1 Quorum          ||                            '----------
+         /      \         ||
+        /        \        ||
+   Primary      2 NBD  ------->  2 NBD
+     disk       client    ||     server                  virtio-blk
+                          ||        ^                         ^
+--------.                 ||        |                         |
+Primary |                 ||  Secondary disk <--------- COLO buffer 3
+--------'                 ||                   backing
+
+1) The disk on the primary is represented by a block device with two
+children, providing replication between a primary disk and the host that
+runs the secondary VM. The read pattern for quorum can be extended to
+make the primary always read from the local disk instead of going through
+NBD.
+
+2) The secondary disk receives writes from the primary VM through QEMU's
+embedded NBD server (speculative write-through).
+
+3) The disk on the secondary is represented by a custom block device
+("COLO buffer"). The disk buffer's backing image is the secondary disk,
+and the disk buffer uses bdrv_add_before_write_notifier to implement
+copy-on-write, similar to block/backup.c.
+
+== New block driver interface ==
+We add three block driver interfaces to control block replication:
+a. bdrv_start_replication()
+   Start block replication, called in migration/checkpoint thread.
+   We must call bdrv_start_replication() in secondary QEMU before
+   calling bdrv_start_replication() in primary QEMU.
+b. bdrv_do_checkpoint()
+   This interface is called after all VM state is transfered to
+   Secondary QEMU. The Disk buffer will be dropped in this interface.
+c. bdrv_stop_replication()
+   It is called when failover. We will flush the Disk buffer into
+   Secondary Disk and stop block replication.
+
+== Usage ==
+Primary:
+  -drive if=xxx,driver=quorum,read-pattern=first,\
+         children.0.file.filename=1.raw,\
+         children.0.driver=raw,\
+         children.1.file.driver=nbd+colo,\
+         children.1.file.host=xxx,\
+         children.1.file.port=xxx,\
+         children.1.file.export=xxx,\
+         children.1.driver=raw
+  Note:
+  1. NBD Client should not be the first child of quorum.
+  2. There should be only one NBD Client.
+  3. host is the secondary physical machine's hostname or IP
+  4. Each disk must have its own export name.
+
+Secondary:
+  -drive if=xxx,driver=blkcolo,export=xxx,\
+         backing.file.filename=1.raw,\
+         backing.driver=raw
+  Then run qmp command:
+    nbd_server_start host:port
+  Note:
+  1. The export name for the same disk must be the same in primary
+     and secondary QEMU command line
+  2. The qmp command nbd_server_start must be run before running the
+     qmp command migrate on primary QEMU
+  3. Don't use nbd_server_start's other options
-- 
2.1.0

[Qemu-devel] [RFC PATCH 02/14] quorom: add a new read pattern

[Wen Congyang]

To block replication, we only need to read from the first child.

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Cc: Luiz Capitulino <lcapitulino@redhat.com>
Cc: Michael Roth <mdroth@linux.vnet.ibm.com>
---
 block/quorum.c       | 5 +++--
 qapi/block-core.json | 4 +++-
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/block/quorum.c b/block/quorum.c
index 437b122..5ed1ff8 100644
--- a/block/quorum.c
+++ b/block/quorum.c
@@ -286,9 +286,10 @@ static void quorum_aio_cb(void *opaque, int ret)
     BDRVQuorumState *s = acb->common.bs->opaque;
     bool rewrite = false;
 
-    if (acb->is_read && s->read_pattern == QUORUM_READ_PATTERN_FIFO) {
+    if (acb->is_read && s->read_pattern != QUORUM_READ_PATTERN_QUORUM) {
         /* We try to read next child in FIFO order if we fail to read */
-        if (ret < 0 && ++acb->child_iter < s->num_children) {
+        if (s->read_pattern == QUORUM_READ_PATTERN_FIFO &&
+            ret < 0 && ++acb->child_iter < s->num_children) {
             read_fifo_child(acb);
             return;
         }
diff --git a/qapi/block-core.json b/qapi/block-core.json
index a3fdaf0..d6382e9 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1618,9 +1618,11 @@
 #
 # @fifo: read only from the first child that has not failed
 #
+# @first: read only from the first child
+#
 # Since: 2.2
 ##
-{ 'enum': 'QuorumReadPattern', 'data': [ 'quorum', 'fifo' ] }
+{ 'enum': 'QuorumReadPattern', 'data': [ 'quorum', 'fifo', 'first' ] }
 
 ##
 # @BlockdevOptionsQuorum
-- 
2.1.0

[Qemu-devel] [RFC PATCH 03/14] quorum: ignore 0-length child

[Wen Congyang]

We connect to NBD server when starting block replication, so
the length is 0 before starting block replication.

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
 block/quorum.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/block/quorum.c b/block/quorum.c
index 5ed1ff8..e6aff5f 100644
--- a/block/quorum.c
+++ b/block/quorum.c
@@ -734,6 +734,11 @@ static int64_t quorum_getlength(BlockDriverState *bs)
         if (value < 0) {
             return value;
         }
+
+        if (!value) {
+            continue;
+        }
+
         if (value != result) {
             return -EIO;
         }
-- 
2.1.0

[Qemu-devel] [RFC PATCH 04/14] Add new block driver interfaces to control disk replication

[Wen Congyang]

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
 block.c                   | 36 ++++++++++++++++++++++++++++++++++++
 include/block/block.h     | 10 ++++++++++
 include/block/block_int.h | 12 ++++++++++++
 3 files changed, 58 insertions(+)

diff --git a/block.c b/block.c
index 210fd5f..2335af1 100644
--- a/block.c
+++ b/block.c
@@ -6156,3 +6156,39 @@ BlockAcctStats *bdrv_get_stats(BlockDriverState *bs)
 {
     return &bs->stats;
 }
+
+int bdrv_start_replication(BlockDriverState *bs, int mode)
+{
+    BlockDriver *drv = bs->drv;
+    if (drv && drv->bdrv_start_replication) {
+        return drv->bdrv_start_replication(bs, mode);
+    } else if (bs->file) {
+        return bdrv_start_replication(bs->file, mode);
+    }
+
+    return -1;
+}
+
+int bdrv_do_checkpoint(BlockDriverState *bs)
+{
+    BlockDriver *drv = bs->drv;
+    if (drv && drv->bdrv_do_checkpoint) {
+        return drv->bdrv_do_checkpoint(bs);
+    } else if (bs->file) {
+        return bdrv_do_checkpoint(bs->file);
+    }
+
+    return -1;
+}
+
+int bdrv_stop_replication(BlockDriverState *bs)
+{
+    BlockDriver *drv = bs->drv;
+    if (drv && drv->bdrv_stop_replication) {
+        return drv->bdrv_stop_replication(bs);
+    } else if (bs->file) {
+        return bdrv_stop_replication(bs->file);
+    }
+
+    return -1;
+}
diff --git a/include/block/block.h b/include/block/block.h
index 321295e..632b9fc 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -557,4 +557,14 @@ void bdrv_flush_io_queue(BlockDriverState *bs);
 
 BlockAcctStats *bdrv_get_stats(BlockDriverState *bs);
 
+/* Checkpoint control, called in migration/checkpoint thread */
+enum {
+    COLO_UNPROTECTED_MODE = 0,
+    COLO_PRIMARY_MODE,
+    COLO_SECONDARY_MODE,
+};
+int bdrv_start_replication(BlockDriverState *bs, int mode);
+int bdrv_do_checkpoint(BlockDriverState *bs);
+int bdrv_stop_replication(BlockDriverState *bs);
+
 #endif
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 7ad1950..603f704 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -273,6 +273,18 @@ struct BlockDriver {
     void (*bdrv_io_unplug)(BlockDriverState *bs);
     void (*bdrv_flush_io_queue)(BlockDriverState *bs);
 
+
+    /* Checkpoint control, called in migration/checkpoint thread */
+    int (*bdrv_start_replication)(BlockDriverState *bs, int mode);
+    /*
+     * Drop Disk buffer when doing checkpoint.
+     */
+    int (*bdrv_do_checkpoint)(BlockDriverState *bs);
+    /* After failover, we should flush Disk buffer into secondary disk
+     * and stop block replication.
+     */
+    int (*bdrv_stop_replication)(BlockDriverState *bs);
+
     QLIST_ENTRY(BlockDriver) list;
 };
 
-- 
2.1.0

[Qemu-devel] [RFC PATCH 05/14] quorom: implement block driver interfaces for block replication

[Wen Congyang]

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
 block/quorum.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 69 insertions(+)

diff --git a/block/quorum.c b/block/quorum.c
index e6aff5f..c8479b4 100644
--- a/block/quorum.c
+++ b/block/quorum.c
@@ -1070,6 +1070,71 @@ static void quorum_refresh_filename(BlockDriverState *bs)
     bs->full_open_options = opts;
 }
 
+static int quorum_stop_replication(BlockDriverState *bs);
+static int quorum_start_replication(BlockDriverState *bs, int mode)
+{
+    BDRVQuorumState *s = bs->opaque;
+    int ret = -1, i;
+
+    /*
+     * TODO: support COLO_SECONDARY_MODE if we allow secondary
+     * QEMU becoming primary QEMU.
+     */
+    if (mode != COLO_PRIMARY_MODE) {
+        return -1;
+    }
+
+    if (s->read_pattern != QUORUM_READ_PATTERN_FIRST) {
+        return -1;
+    }
+
+    /* NBD client should not be the first child */
+    if (bdrv_start_replication(s->bs[0], mode) == 0) {
+        bdrv_stop_replication(s->bs[0]);
+        return -1;
+    }
+
+    for (i = 1; i < s->num_children; i++) {
+        if (bdrv_start_replication(s->bs[i], mode) == 0) {
+            ret++;
+        }
+    }
+
+    if (ret > 0) {
+        quorum_stop_replication(bs);
+    }
+
+    return ret ? -1 : 0;
+}
+
+static int quorum_do_checkpoint(BlockDriverState *bs)
+{
+    BDRVQuorumState *s = bs->opaque;
+    int i;
+
+    for (i = 1; i < s->num_children; i++) {
+        if (bdrv_do_checkpoint(s->bs[i]) == 0) {
+            return 0;
+        }
+    }
+
+    return -1;
+}
+
+static int quorum_stop_replication(BlockDriverState *bs)
+{
+    BDRVQuorumState *s = bs->opaque;
+    int ret = -1, i;
+
+    for (i = 0; i < s->num_children; i++) {
+        if (bdrv_stop_replication(s->bs[i]) == 0) {
+            ret++;
+        }
+    }
+
+    return ret ? -1 : 0;
+}
+
 static BlockDriver bdrv_quorum = {
     .format_name                        = "quorum",
     .protocol_name                      = "quorum",
@@ -1093,6 +1158,10 @@ static BlockDriver bdrv_quorum = {
 
     .is_filter                          = true,
     .bdrv_recurse_is_first_non_filter   = quorum_recurse_is_first_non_filter,
+
+    .bdrv_start_replication             = quorum_start_replication,
+    .bdrv_do_checkpoint                 = quorum_do_checkpoint,
+    .bdrv_stop_replication              = quorum_stop_replication,
 };
 
 static void bdrv_quorum_init(void)
-- 
2.1.0

[Qemu-devel] [RFC PATCH 06/14] NBD client: connect to nbd server later

[Wen Congyang]

The secondary qemu starts later than the primary qemu, so we
cannot connect to nbd server in bdrv_open().

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
 block/nbd.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 87 insertions(+), 13 deletions(-)

diff --git a/block/nbd.c b/block/nbd.c
index b05d1d0..19b9200 100644
--- a/block/nbd.c
+++ b/block/nbd.c
@@ -44,6 +44,8 @@
 typedef struct BDRVNBDState {
     NbdClientSession client;
     QemuOpts *socket_opts;
+    char *export;
+    bool connected;
 } BDRVNBDState;
 
 static int nbd_parse_uri(const char *filename, QDict *options)
@@ -247,20 +249,10 @@ static int nbd_establish_connection(BlockDriverState *bs, Error **errp)
     return sock;
 }
 
-static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
-                    Error **errp)
+static int nbd_connect_server(BlockDriverState *bs, Error **errp)
 {
     BDRVNBDState *s = bs->opaque;
-    char *export = NULL;
     int result, sock;
-    Error *local_err = NULL;
-
-    /* Pop the config into our state object. Exit if invalid. */
-    nbd_config(s, options, &export, &local_err);
-    if (local_err) {
-        error_propagate(errp, local_err);
-        return -EINVAL;
-    }
 
     /* establish TCP connection, return error if it fails
      * TODO: Configurable retry-until-timeout behaviour.
@@ -271,16 +263,57 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
     }
 
     /* NBD handshake */
-    result = nbd_client_session_init(&s->client, bs, sock, export, errp);
-    g_free(export);
+    result = nbd_client_session_init(&s->client, bs, sock, s->export, errp);
+    g_free(s->export);
+    s->export = NULL;
+    if (!result) {
+        s->connected = true;
+    }
+
     return result;
 }
 
+static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
+                    Error **errp)
+{
+    BDRVNBDState *s = bs->opaque;
+    Error *local_err = NULL;
+
+    /* Pop the config into our state object. Exit if invalid. */
+    nbd_config(s, options, &s->export, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return -EINVAL;
+    }
+
+    return nbd_connect_server(bs, errp);
+}
+
+static int nbd_open_colo(BlockDriverState *bs, QDict *options, int flags,
+                         Error **errp)
+{
+    BDRVNBDState *s = bs->opaque;
+    Error *local_err = NULL;
+
+    /* Pop the config into our state object. Exit if invalid. */
+    nbd_config(s, options, &s->export, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return -EINVAL;
+    }
+
+    return 0;
+}
+
 static int nbd_co_readv(BlockDriverState *bs, int64_t sector_num,
                         int nb_sectors, QEMUIOVector *qiov)
 {
     BDRVNBDState *s = bs->opaque;
 
+    if (!s->connected) {
+        return -EIO;
+    }
+
     return nbd_client_session_co_readv(&s->client, sector_num,
                                        nb_sectors, qiov);
 }
@@ -290,6 +323,10 @@ static int nbd_co_writev(BlockDriverState *bs, int64_t sector_num,
 {
     BDRVNBDState *s = bs->opaque;
 
+    if (!s->connected) {
+        return 0;
+    }
+
     return nbd_client_session_co_writev(&s->client, sector_num,
                                         nb_sectors, qiov);
 }
@@ -298,6 +335,10 @@ static int nbd_co_flush(BlockDriverState *bs)
 {
     BDRVNBDState *s = bs->opaque;
 
+    if (!s->connected) {
+        return 0;
+    }
+
     return nbd_client_session_co_flush(&s->client);
 }
 
@@ -312,6 +353,10 @@ static int nbd_co_discard(BlockDriverState *bs, int64_t sector_num,
 {
     BDRVNBDState *s = bs->opaque;
 
+    if (!s->connected) {
+        return 0;
+    }
+
     return nbd_client_session_co_discard(&s->client, sector_num,
                                          nb_sectors);
 }
@@ -322,6 +367,7 @@ static void nbd_close(BlockDriverState *bs)
 
     qemu_opts_del(s->socket_opts);
     nbd_client_session_close(&s->client);
+    s->connected = false;
 }
 
 static int64_t nbd_getlength(BlockDriverState *bs)
@@ -335,6 +381,10 @@ static void nbd_detach_aio_context(BlockDriverState *bs)
 {
     BDRVNBDState *s = bs->opaque;
 
+    if (!s->connected) {
+        return;
+    }
+
     nbd_client_session_detach_aio_context(&s->client);
 }
 
@@ -343,6 +393,10 @@ static void nbd_attach_aio_context(BlockDriverState *bs,
 {
     BDRVNBDState *s = bs->opaque;
 
+    if (!s->connected) {
+        return;
+    }
+
     nbd_client_session_attach_aio_context(&s->client, new_context);
 }
 
@@ -445,11 +499,31 @@ static BlockDriver bdrv_nbd_unix = {
     .bdrv_refresh_filename      = nbd_refresh_filename,
 };
 
+static BlockDriver bdrv_nbd_colo = {
+    .format_name                = "nbd+colo",
+    .protocol_name              = "nbd+colo",
+    .instance_size              = sizeof(BDRVNBDState),
+    .bdrv_parse_filename        = nbd_parse_filename,
+    .bdrv_file_open             = nbd_open_colo,
+    .bdrv_co_readv              = nbd_co_readv,
+    .bdrv_co_writev             = nbd_co_writev,
+    .bdrv_close                 = nbd_close,
+    .bdrv_co_flush_to_os        = nbd_co_flush,
+    .bdrv_co_discard            = nbd_co_discard,
+    .bdrv_getlength             = nbd_getlength,
+    .bdrv_detach_aio_context    = nbd_detach_aio_context,
+    .bdrv_attach_aio_context    = nbd_attach_aio_context,
+    .bdrv_refresh_filename      = nbd_refresh_filename,
+
+    .has_variable_length        = true,
+};
+
 static void bdrv_nbd_init(void)
 {
     bdrv_register(&bdrv_nbd);
     bdrv_register(&bdrv_nbd_tcp);
     bdrv_register(&bdrv_nbd_unix);
+    bdrv_register(&bdrv_nbd_colo);
 }
 
 block_init(bdrv_nbd_init);
-- 
2.1.0

[Qemu-devel] [RFC PATCH 07/14] NBD client: implement block driver interfaces for block replication

[Wen Congyang]

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
 block/nbd.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)

diff --git a/block/nbd.c b/block/nbd.c
index 19b9200..1ff6ecf 100644
--- a/block/nbd.c
+++ b/block/nbd.c
@@ -445,6 +445,58 @@ static void nbd_refresh_filename(BlockDriverState *bs)
     bs->full_open_options = opts;
 }
 
+static int nbd_start_replication(BlockDriverState *bs, int mode)
+{
+    BDRVNBDState *s = bs->opaque;
+    Error *local_err = NULL;
+    int ret;
+
+    /*
+     * TODO: support COLO_SECONDARY_MODE if we allow secondary
+     * QEMU becoming primary QEMU.
+     */
+    if (mode != COLO_PRIMARY_MODE) {
+        return -1;
+    }
+
+    if (s->connected) {
+        return -1;
+    }
+
+    /* TODO: NBD client should be one child of quorum, how to verify it? */
+    ret = nbd_connect_server(bs, &local_err);
+    if (local_err) {
+        error_free(local_err);
+    }
+
+    return ret;
+}
+
+static int nbd_do_checkpoint(BlockDriverState *bs)
+{
+    BDRVNBDState *s = bs->opaque;
+
+    if (!s->connected) {
+        return -1;
+    }
+
+    return 0;
+}
+
+static int nbd_stop_replication(BlockDriverState *bs)
+{
+    BDRVNBDState *s = bs->opaque;
+
+    if (!s->connected) {
+        return -1;
+    }
+
+    nbd_client_session_close(&s->client);
+    s->connected = false;
+
+    return 0;
+}
+
 static BlockDriver bdrv_nbd = {
     .format_name                = "nbd",
     .protocol_name              = "nbd",
@@ -514,6 +566,9 @@ static BlockDriver bdrv_nbd_colo = {
     .bdrv_detach_aio_context    = nbd_detach_aio_context,
     .bdrv_attach_aio_context    = nbd_attach_aio_context,
     .bdrv_refresh_filename      = nbd_refresh_filename,
+    .bdrv_start_replication     = nbd_start_replication,
+    .bdrv_do_checkpoint         = nbd_do_checkpoint,
+    .bdrv_stop_replication      = nbd_stop_replication,
 
     .has_variable_length        = true,
 };
-- 
2.1.0

[Qemu-devel] [RFC PATCH 08/14] block: add a new API to create a hidden BlockBackend

[Wen Congyang]

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
 block/block-backend.c          | 29 ++++++++++++++++++++++++++++-
 include/sysemu/block-backend.h |  2 ++
 2 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/block/block-backend.c b/block/block-backend.c
index c28e240..1bbc078 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -72,6 +72,19 @@ BlockBackend *blk_new(const char *name, Error **errp)
 }
 
 /*
+ * Create a new hidden BlockBackend, with a reference count of one.
+ * Return the new BlockBackend on success, null on failure.
+ */
+BlockBackend *blk_hide_new(void)
+{
+    BlockBackend *blk;
+
+    blk = g_new0(BlockBackend, 1);
+    blk->refcnt = 1;
+    return blk;
+}
+
+/*
  * Create a new BlockBackend with a new BlockDriverState attached.
  * Otherwise just like blk_new(), which see.
  */
@@ -91,6 +104,20 @@ BlockBackend *blk_new_with_bs(const char *name, Error **errp)
     return blk;
 }
 
+/*
+ * Create a new hidden BlockBackend with a new BlockDriverState attached.
+ * Otherwise just like blk_hide_new(), which see.
+ */
+BlockBackend *blk_hide_new_with_bs(void)
+{
+    BlockBackend *blk = blk_hide_new();
+    BlockDriverState *bs = bdrv_new();
+
+    blk->bs = bs;
+    bs->blk = blk;
+    return blk;
+}
+
 static void blk_delete(BlockBackend *blk)
 {
     assert(!blk->refcnt);
@@ -102,7 +129,7 @@ static void blk_delete(BlockBackend *blk)
         blk->bs = NULL;
     }
     /* Avoid double-remove after blk_hide_on_behalf_of_do_drive_del() */
-    if (blk->name[0]) {
+    if (blk->name && blk->name[0]) {
         QTAILQ_REMOVE(&blk_backends, blk, link);
     }
     g_free(blk->name);
diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h
index aab12b9..acc50f5 100644
--- a/include/sysemu/block-backend.h
+++ b/include/sysemu/block-backend.h
@@ -61,7 +61,9 @@ typedef struct BlockDevOps {
 } BlockDevOps;
 
 BlockBackend *blk_new(const char *name, Error **errp);
+BlockBackend *blk_hide_new(void);
 BlockBackend *blk_new_with_bs(const char *name, Error **errp);
+BlockBackend *blk_hide_new_with_bs(void);
 void blk_ref(BlockBackend *blk);
 void blk_unref(BlockBackend *blk);
 const char *blk_name(BlockBackend *blk);
-- 
2.1.0

[Qemu-devel] [RFC PATCH 09/14] block: give backing image its own BlockBackend

[Wen Congyang]

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
 block.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/block.c b/block.c
index 2335af1..a7a8932 100644
--- a/block.c
+++ b/block.c
@@ -1218,6 +1218,7 @@ int bdrv_open_backing_file(BlockDriverState *bs, QDict *options, Error **errp)
 {
     char *backing_filename = g_malloc0(PATH_MAX);
     int ret = 0;
+    BlockBackend *backing_blk;
     BlockDriverState *backing_hd;
     Error *local_err = NULL;
 
@@ -1255,7 +1256,8 @@ int bdrv_open_backing_file(BlockDriverState *bs, QDict *options, Error **errp)
         goto free_exit;
     }
 
-    backing_hd = bdrv_new();
+    backing_blk = blk_hide_new_with_bs();
+    backing_hd = blk_bs(backing_blk);
 
     if (bs->backing_format[0] != '\0' && !qdict_haskey(options, "driver")) {
         qdict_put(options, "driver", qstring_from_str(bs->backing_format));
@@ -1266,7 +1268,8 @@ int bdrv_open_backing_file(BlockDriverState *bs, QDict *options, Error **errp)
                     *backing_filename ? backing_filename : NULL, NULL, options,
                     bdrv_backing_flags(bs->open_flags), NULL, &local_err);
     if (ret < 0) {
-        bdrv_unref(backing_hd);
+        blk_unref(backing_blk);
+        backing_blk = NULL;
         backing_hd = NULL;
         bs->open_flags |= BDRV_O_NO_BACKING;
         error_setg(errp, "Could not open backing file: %s",
@@ -1870,9 +1873,9 @@ void bdrv_close(BlockDriverState *bs)
 
     if (bs->drv) {
         if (bs->backing_hd) {
-            BlockDriverState *backing_hd = bs->backing_hd;
+            BlockBackend *backing_blk = bs->backing_hd->blk;
             bdrv_set_backing_hd(bs, NULL);
-            bdrv_unref(backing_hd);
+            blk_unref(backing_blk);
         }
         bs->drv->bdrv_close(bs);
         g_free(bs->opaque);
-- 
2.1.0

[Qemu-devel] [RFC PATCH 10/14] allow the backing image access the origin BlockDriverState

[Wen Congyang]

Block replication needs this feature.

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
 block.c                   | 2 ++
 include/block/block_int.h | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/block.c b/block.c
index a7a8932..067c44b 100644
--- a/block.c
+++ b/block.c
@@ -1181,6 +1181,7 @@ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd)
     if (bs->backing_hd) {
         assert(bs->backing_blocker);
         bdrv_op_unblock_all(bs->backing_hd, bs->backing_blocker);
+        bs->backing_hd->origin_file = NULL;
     } else if (backing_hd) {
         error_setg(&bs->backing_blocker,
                    "device is used as backing hd of '%s'",
@@ -1193,6 +1194,7 @@ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd)
         bs->backing_blocker = NULL;
         goto out;
     }
+    backing_hd->origin_file = bs;
     bs->open_flags &= ~BDRV_O_NO_BACKING;
     pstrcpy(bs->backing_file, sizeof(bs->backing_file), backing_hd->filename);
     pstrcpy(bs->backing_format, sizeof(bs->backing_format),
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 603f704..9be13a8 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -360,6 +360,8 @@ struct BlockDriverState {
     char exact_filename[PATH_MAX];
 
     BlockDriverState *backing_hd;
+    /* used by backing image */
+    BlockDriverState *origin_file;
     BlockDriverState *file;
 
     NotifierList close_notifiers;
-- 
2.1.0

[Qemu-devel] [RFC PATCH 11/14] allow writing to the backing file

[Wen Congyang]

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
 block.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/block.c b/block.c
index 067c44b..96cf973 100644
--- a/block.c
+++ b/block.c
@@ -856,8 +856,8 @@ static int bdrv_inherited_flags(int flags)
  */
 static int bdrv_backing_flags(int flags)
 {
-    /* backing files always opened read-only */
-    flags &= ~(BDRV_O_RDWR | BDRV_O_COPY_ON_READ);
+    /* backing files are opened read-write for block replication */
+    flags &= ~BDRV_O_COPY_ON_READ;
 
     /* snapshot=on is handled on the top layer */
     flags &= ~(BDRV_O_SNAPSHOT | BDRV_O_TEMPORARY);
-- 
2.1.0

[Qemu-devel] [RFC PATCH 12/14] Add disk buffer for block replication

[Wen Congyang]

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
 block/Makefile.objs    |   1 +
 block/blkcolo-buffer.c | 324 +++++++++++++++++++++++++++++++++++++++++++++++++
 block/blkcolo.h        |  35 ++++++
 3 files changed, 360 insertions(+)
 create mode 100644 block/blkcolo-buffer.c
 create mode 100644 block/blkcolo.h

diff --git a/block/Makefile.objs b/block/Makefile.objs
index db2933e..1b7b458 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -21,6 +21,7 @@ block-obj-$(CONFIG_ARCHIPELAGO) += archipelago.o
 block-obj-$(CONFIG_LIBSSH2) += ssh.o
 block-obj-y += accounting.o
 block-obj-y += write-threshold.o
+block-obj-y += blkcolo-buffer.o
 
 common-obj-y += stream.o
 common-obj-y += commit.o
diff --git a/block/blkcolo-buffer.c b/block/blkcolo-buffer.c
new file mode 100644
index 0000000..1f64542
--- /dev/null
+++ b/block/blkcolo-buffer.c
@@ -0,0 +1,324 @@
+/*
+ * Block driver for COLO
+ *
+ * Copyright Fujitsu, Corp. 2015
+ * Copyright (c) 2015 Intel Corporation
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO.,LTD.
+ *
+ * Authors:
+ *     Wen Congyang <wency@cn.fujitsu.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "qemu/queue.h"
+#include "block/block.h"
+#include "block/blkcolo.h"
+
+typedef struct buffered_request_state {
+    uint64_t start_sector;
+    int nb_sectors;
+    void *data;
+    QSIMPLEQ_ENTRY(buffered_request_state) entry;
+} buffered_request_state;
+
+/* common functions */
+/*
+ * The buffered data may eat too much memory, and glibc cannot work
+ * very well in such case.
+ */
+static void *alloc_buffered_data(int nb_sectors)
+{
+    return g_malloc(nb_sectors * BDRV_SECTOR_SIZE);
+}
+
+static void free_buffered_data(void *data)
+{
+    g_free(data);
+}
+
+typedef struct search_brs_state {
+    uint64_t sector;
+    buffered_request_state *prev;
+} search_brs_state;
+
+static buffered_request_state *search_brs(disk_buffer *disk_buffer,
+                                           search_brs_state *sbs)
+{
+    buffered_request_state *brs;
+
+    QSIMPLEQ_FOREACH(brs, &disk_buffer->head, entry) {
+        if (sbs->sector < brs->start_sector) {
+            return NULL;
+        }
+
+        if (sbs->sector < brs->start_sector + brs->nb_sectors) {
+            return brs;
+        }
+
+        sbs->prev = brs;
+    }
+
+    return NULL;
+}
+
+static buffered_request_state *get_next_brs(buffered_request_state *brs)
+{
+    return QSIMPLEQ_NEXT(brs, entry);
+}
+
+static void add_brs_after(disk_buffer *disk_buffer,
+                          buffered_request_state *new_brs,
+                          buffered_request_state *prev)
+{
+    if (!prev) {
+        QSIMPLEQ_INSERT_HEAD(&disk_buffer->head, new_brs, entry);
+    } else {
+        QSIMPLEQ_INSERT_AFTER(&disk_buffer->head, prev, new_brs, entry);
+    }
+}
+
+static bool disk_buffer_empty(disk_buffer *disk_buffer)
+{
+    return QSIMPLEQ_EMPTY(&disk_buffer->head);
+}
+
+/* Disk buffer */
+static buffered_request_state *create_new_brs(QEMUIOVector *qiov,
+                                              uint64_t iov_sector,
+                                              uint64_t sector, int nb_sectors)
+{
+    buffered_request_state *brs;
+
+    brs = g_slice_new(buffered_request_state);
+    brs->start_sector = sector;
+    brs->nb_sectors = nb_sectors;
+    brs->data = alloc_buffered_data(nb_sectors);
+    qemu_iovec_to_buf(qiov, (sector - iov_sector) * BDRV_SECTOR_SIZE,
+                      brs->data, nb_sectors * BDRV_SECTOR_SIZE);
+
+    return brs;
+}
+
+static void free_brs(buffered_request_state *brs)
+{
+    free_buffered_data(brs->data);
+    g_slice_free(buffered_request_state, brs);
+}
+
+bool buffer_has_empty_range(disk_buffer *disk_buffer,
+                            uint64_t sector, int nb_sectors)
+{
+    buffered_request_state *brs;
+    search_brs_state sbs;
+    uint64_t cur_sector = sector;
+
+    if (nb_sectors <= 0) {
+        return false;
+    }
+
+    sbs.sector = sector;
+    sbs.prev = NULL;
+    brs = search_brs(disk_buffer, &sbs);
+    if (!brs) {
+        return true;
+    }
+
+    while (brs && cur_sector < sector + nb_sectors) {
+        if (cur_sector < brs->start_sector) {
+            return true;
+        }
+
+        if (brs->start_sector + brs->nb_sectors >= sector + nb_sectors) {
+            return false;
+        }
+
+        cur_sector = brs->start_sector + brs->nb_sectors;
+        brs = get_next_brs(brs);
+    }
+
+    if (cur_sector < sector + nb_sectors) {
+        return true;
+    } else {
+        return false;
+    }
+}
+
+/* Note: only the sector that exists in the buffer will be overwriten */
+void qiov_read_from_buffer(disk_buffer *disk_buffer, QEMUIOVector *qiov,
+                           uint64_t sector, int nb_sectors)
+{
+    search_brs_state sbs;
+    buffered_request_state *brs;
+    size_t offset, cur_nb_sectors;
+    uint64_t cur_sector = sector;
+    void *buf;
+
+    if (disk_buffer_empty(disk_buffer)) {
+        /* The disk buffer is empty */
+        return;
+    }
+
+    sbs.sector = sector;
+    sbs.prev = NULL;
+    brs = search_brs(disk_buffer, &sbs);
+    if (!brs) {
+        if (!sbs.prev) {
+            brs = QSIMPLEQ_FIRST(&disk_buffer->head);
+        } else {
+            brs = get_next_brs(sbs.prev);
+        }
+    }
+
+    while (brs && cur_sector < sector + nb_sectors) {
+        if (brs->start_sector >= sector + nb_sectors) {
+            break;
+        }
+
+        /* In the first loop, brs->start_sector can be less than sector */
+        if (brs->start_sector < cur_sector) {
+            offset = cur_sector - brs->start_sector;
+            buf = brs->data + offset * BDRV_SECTOR_SIZE;
+        } else {
+            cur_sector = brs->start_sector;
+            offset = 0;
+            buf = brs->data;
+        }
+        if (brs->start_sector + brs->nb_sectors >= sector + nb_sectors) {
+            cur_nb_sectors = sector + nb_sectors - cur_sector;
+        } else {
+            cur_nb_sectors = brs->nb_sectors - offset;
+        }
+        qemu_iovec_from_buf(qiov, (cur_sector - sector) * BDRV_SECTOR_SIZE,
+                            buf, cur_nb_sectors * BDRV_SECTOR_SIZE);
+
+        cur_sector = brs->start_sector + brs->nb_sectors;
+        brs = get_next_brs(brs);
+    }
+}
+
+void qiov_write_to_buffer(disk_buffer *disk_buffer, QEMUIOVector *qiov,
+                          uint64_t sector, int nb_sectors, bool overwrite)
+{
+    search_brs_state sbs;
+    buffered_request_state *brs, *new_brs, *prev;
+    uint64_t cur_sector = sector;
+    int cur_nb_sectors, offset;
+
+    if (disk_buffer_empty(disk_buffer)) {
+        /* The disk buffer is empty */
+        new_brs = create_new_brs(qiov, sector, cur_sector, nb_sectors);
+        add_brs_after(disk_buffer, new_brs, NULL);
+        return;
+    }
+
+    sbs.sector = sector;
+    sbs.prev = NULL;
+    brs = search_brs(disk_buffer, &sbs);
+    if (!sbs.prev) {
+        prev = NULL;
+        brs = QSIMPLEQ_FIRST(&disk_buffer->head);
+    } else {
+        prev = sbs.prev;
+        brs = get_next_brs(sbs.prev);
+    }
+
+    while (brs && cur_sector < sector + nb_sectors) {
+        if (cur_sector < brs->start_sector) {
+            if (sector + nb_sectors <= brs->start_sector) {
+                cur_nb_sectors = sector + nb_sectors - cur_sector;
+            } else {
+                cur_nb_sectors = brs->start_sector - cur_sector;
+            }
+            new_brs = create_new_brs(qiov, sector, cur_sector, cur_nb_sectors);
+            add_brs_after(disk_buffer, new_brs, prev);
+            cur_sector = brs->start_sector;
+        }
+
+        if (cur_sector >= sector + nb_sectors) {
+            break;
+        }
+
+        if (overwrite) {
+            offset = cur_sector - brs->start_sector;
+            if (sector + nb_sectors <= brs->start_sector + brs->nb_sectors) {
+                cur_nb_sectors = sector + nb_sectors - cur_sector;
+            } else {
+                cur_nb_sectors = brs->nb_sectors - offset;
+            }
+            qemu_iovec_to_buf(qiov, (cur_sector - sector) * BDRV_SECTOR_SIZE,
+                              brs->data + offset * BDRV_SECTOR_SIZE,
+                              cur_nb_sectors * BDRV_SECTOR_SIZE);
+        }
+
+        cur_sector = brs->start_sector + brs->nb_sectors;
+
+        prev = brs;
+        brs = get_next_brs(brs);
+    }
+
+    if (cur_sector < sector + nb_sectors) {
+        new_brs = create_new_brs(qiov, sector, cur_sector,
+                                 sector + nb_sectors - cur_sector);
+        add_brs_after(disk_buffer, new_brs, prev);
+    }
+}
+
+struct flushed_data {
+    QEMUIOVector qiov;
+    buffered_request_state *brs;
+};
+
+static void flush_buffered_data_complete(void *opaque, int ret)
+{
+    struct flushed_data *flushed_data = opaque;
+
+    /* We have reported the guest that this write ops successed */
+    assert(ret == 0);
+
+    qemu_iovec_destroy(&flushed_data->qiov);
+    free_brs(flushed_data->brs);
+    g_free(flushed_data);
+}
+
+void flush_buffered_data_to_disk(disk_buffer *disk_buffer,
+                                 BlockDriverState *bs)
+{
+    buffered_request_state *brs, *tmp;
+    struct flushed_data *flushed_data = NULL;
+
+    QSIMPLEQ_FOREACH_SAFE(brs, &disk_buffer->head, entry, tmp) {
+        /* brs is always the head */
+        QSIMPLEQ_REMOVE_HEAD(&disk_buffer->head, entry);
+
+        flushed_data = g_malloc(sizeof(struct flushed_data));
+        qemu_iovec_init(&flushed_data->qiov, 1);
+        qemu_iovec_add(&flushed_data->qiov, brs->data,
+                       brs->nb_sectors * BDRV_SECTOR_SIZE);
+        flushed_data->brs = brs;
+        bdrv_aio_writev(bs, brs->start_sector, &flushed_data->qiov,
+                        brs->nb_sectors, flush_buffered_data_complete,
+                        flushed_data);
+    }
+
+    bdrv_drain_all();
+}
+
+void init_disk_buffer(disk_buffer *disk_buffer)
+{
+    QSIMPLEQ_INIT(&disk_buffer->head);
+}
+
+void clear_all_buffered_data(disk_buffer *disk_buffer)
+{
+    buffered_request_state *brs, *tmp;
+
+    QSIMPLEQ_FOREACH_SAFE(brs, &disk_buffer->head, entry, tmp) {
+        /* brs is always the head */
+        QSIMPLEQ_REMOVE_HEAD(&disk_buffer->head, entry);
+        free_brs(brs);
+    }
+}
diff --git a/block/blkcolo.h b/block/blkcolo.h
new file mode 100644
index 0000000..d8e0d9a
--- /dev/null
+++ b/block/blkcolo.h
@@ -0,0 +1,35 @@
+/*
+ * Block driver for COLO
+ *
+ * Copyright Fujitsu, Corp. 2015
+ * Copyright (c) 2015 Intel Corporation
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO.,LTD.
+ *
+ * Authors:
+ *     Wen Congyang <wency@cn.fujitsu.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef BLOCK_BLKCOLO_H
+#define BLOCK_BLKCOLO_H
+
+typedef struct disk_buffer {
+    QSIMPLEQ_HEAD(, buffered_request_state) head;
+} disk_buffer;
+
+bool buffer_has_empty_range(disk_buffer *disk_buffer,
+                            uint64_t sector, int nb_sectors);
+void qiov_read_from_buffer(disk_buffer *disk_buffer, QEMUIOVector *qiov,
+                           uint64_t sector, int nb_sectors);
+void qiov_write_to_buffer(disk_buffer *disk_buffer, QEMUIOVector *qiov,
+                          uint64_t sector, int nb_sectors, bool overwrite);
+void flush_buffered_data_to_disk(disk_buffer *disk_buffer,
+                                 BlockDriverState *bs);
+
+void init_disk_buffer(disk_buffer *disk_buffer);
+void clear_all_buffered_data(disk_buffer *disk_buffer);
+
+#endif
-- 
2.1.0

[Qemu-devel] [RFC PATCH 13/14] COW: move cow interfaces to a seperate file

[Wen Congyang]

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
 Makefile.objs         |  2 +-
 block/backup.c        | 52 ++++-----------------------------------------------
 blockcow.c            | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++
 include/block/block.h | 17 +++++++++++++++++
 4 files changed, 74 insertions(+), 49 deletions(-)
 create mode 100644 blockcow.c

diff --git a/Makefile.objs b/Makefile.objs
index 28999d3..91bba07 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -7,7 +7,7 @@ util-obj-y = util/ qobject/ qapi/ qapi-types.o qapi-visit.o qapi-event.o
 # block-obj-y is code used by both qemu system emulation and qemu-img
 
 block-obj-y = async.o thread-pool.o
-block-obj-y += nbd.o block.o blockjob.o
+block-obj-y += nbd.o block.o blockjob.o blockcow.o
 block-obj-y += main-loop.o iohandler.o qemu-timer.o
 block-obj-$(CONFIG_POSIX) += aio-posix.o
 block-obj-$(CONFIG_WIN32) += aio-win32.o
diff --git a/block/backup.c b/block/backup.c
index 1c535b1..2816b9a 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -27,13 +27,6 @@
 
 #define SLICE_TIME 100000000ULL /* ns */
 
-typedef struct CowRequest {
-    int64_t start;
-    int64_t end;
-    QLIST_ENTRY(CowRequest) list;
-    CoQueue wait_queue; /* coroutines blocked on this request */
-} CowRequest;
-
 typedef struct BackupBlockJob {
     BlockJob common;
     BlockDriverState *target;
@@ -44,46 +37,9 @@ typedef struct BackupBlockJob {
     CoRwlock flush_rwlock;
     uint64_t sectors_read;
     HBitmap *bitmap;
-    QLIST_HEAD(, CowRequest) inflight_reqs;
+    CowJob cow_job;
 } BackupBlockJob;
 
-/* See if in-flight requests overlap and wait for them to complete */
-static void coroutine_fn wait_for_overlapping_requests(BackupBlockJob *job,
-                                                       int64_t start,
-                                                       int64_t end)
-{
-    CowRequest *req;
-    bool retry;
-
-    do {
-        retry = false;
-        QLIST_FOREACH(req, &job->inflight_reqs, list) {
-            if (end > req->start && start < req->end) {
-                qemu_co_queue_wait(&req->wait_queue);
-                retry = true;
-                break;
-            }
-        }
-    } while (retry);
-}
-
-/* Keep track of an in-flight request */
-static void cow_request_begin(CowRequest *req, BackupBlockJob *job,
-                                     int64_t start, int64_t end)
-{
-    req->start = start;
-    req->end = end;
-    qemu_co_queue_init(&req->wait_queue);
-    QLIST_INSERT_HEAD(&job->inflight_reqs, req, list);
-}
-
-/* Forget about a completed request */
-static void cow_request_end(CowRequest *req)
-{
-    QLIST_REMOVE(req, list);
-    qemu_co_queue_restart_all(&req->wait_queue);
-}
-
 static int coroutine_fn backup_do_cow(BlockDriverState *bs,
                                       int64_t sector_num, int nb_sectors,
                                       bool *error_is_read)
@@ -104,8 +60,8 @@ static int coroutine_fn backup_do_cow(BlockDriverState *bs,
 
     trace_backup_do_cow_enter(job, start, sector_num, nb_sectors);
 
-    wait_for_overlapping_requests(job, start, end);
-    cow_request_begin(&cow_request, job, start, end);
+    wait_for_overlapping_requests(&job->cow_job, start, end);
+    cow_request_begin(&cow_request, &job->cow_job, start, end);
 
     for (; start < end; start++) {
         if (hbitmap_get(job->bitmap, start)) {
@@ -255,7 +211,7 @@ static void coroutine_fn backup_run(void *opaque)
     int64_t start, end;
     int ret = 0;
 
-    QLIST_INIT(&job->inflight_reqs);
+    QLIST_INIT(&job->cow_job.inflight_reqs);
     qemu_co_rwlock_init(&job->flush_rwlock);
 
     start = 0;
diff --git a/blockcow.c b/blockcow.c
new file mode 100644
index 0000000..c070a62
--- /dev/null
+++ b/blockcow.c
@@ -0,0 +1,52 @@
+/*
+ * QEMU block WAW
+ *
+ * Copyright Fujitsu, Corp. 2015
+ * Copyright (c) 2015 Intel Corporation
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO.,LTD.
+ *
+ * Authors:
+ *  Wen Congyang (wency@cn.fujitsu.com)
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "block/block.h"
+
+/* See if in-flight requests overlap and wait for them to complete */
+void coroutine_fn
+wait_for_overlapping_requests(CowJob *job, int64_t start, int64_t end)
+{
+    CowRequest *req;
+    bool retry;
+
+    do {
+        retry = false;
+        QLIST_FOREACH(req, &job->inflight_reqs, list) {
+            if (end > req->start && start < req->end) {
+                qemu_co_queue_wait(&req->wait_queue);
+                retry = true;
+                break;
+            }
+        }
+    } while (retry);
+}
+
+/* Keep track of an in-flight request */
+void cow_request_begin(CowRequest *req, CowJob *job,
+                       int64_t start, int64_t end)
+{
+    req->start = start;
+    req->end = end;
+    qemu_co_queue_init(&req->wait_queue);
+    QLIST_INSERT_HEAD(&job->inflight_reqs, req, list);
+}
+
+/* Forget about a completed request */
+void cow_request_end(CowRequest *req)
+{
+    QLIST_REMOVE(req, list);
+    qemu_co_queue_restart_all(&req->wait_queue);
+}
diff --git a/include/block/block.h b/include/block/block.h
index 632b9fc..0a55373 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -567,4 +567,21 @@ int bdrv_start_replication(BlockDriverState *bs, int mode);
 int bdrv_do_checkpoint(BlockDriverState *bs);
 int bdrv_stop_replication(BlockDriverState *bs);
 
+typedef struct CowRequest {
+    int64_t start;
+    int64_t end;
+    QLIST_ENTRY(CowRequest) list;
+    CoQueue wait_queue; /* coroutines blocked on this request */
+} CowRequest;
+
+typedef struct CowJob {
+    QLIST_HEAD(, CowRequest) inflight_reqs;
+} CowJob;
+
+void coroutine_fn
+wait_for_overlapping_requests(CowJob *job, int64_t start, int64_t end);
+void cow_request_begin(CowRequest *req, CowJob *job,
+                       int64_t start, int64_t end);
+void cow_request_end(CowRequest *req);
+
 #endif
-- 
2.1.0

[Qemu-devel] [RFC PATCH 14/14] COLO: implement a new block driver

[Wen Congyang]

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
 block/Makefile.objs |   2 +-
 block/blkcolo.c     | 409 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 410 insertions(+), 1 deletion(-)
 create mode 100644 block/blkcolo.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 1b7b458..021e891 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -21,7 +21,7 @@ block-obj-$(CONFIG_ARCHIPELAGO) += archipelago.o
 block-obj-$(CONFIG_LIBSSH2) += ssh.o
 block-obj-y += accounting.o
 block-obj-y += write-threshold.o
-block-obj-y += blkcolo-buffer.o
+block-obj-y += blkcolo-buffer.o blkcolo.o
 
 common-obj-y += stream.o
 common-obj-y += commit.o
diff --git a/block/blkcolo.c b/block/blkcolo.c
new file mode 100644
index 0000000..2f73486
--- /dev/null
+++ b/block/blkcolo.c
@@ -0,0 +1,409 @@
+/*
+ * Block driver for block replication
+ *
+ * Copyright Fujitsu, Corp. 2015
+ * Copyright (c) 2015 Intel Corporation
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO.,LTD.
+ *
+ * Authors:
+ *     Wen Congyang <wency@cn.fujitsu.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "block/block_int.h"
+#include "sysemu/block-backend.h"
+#include "block/blkcolo.h"
+#include "block/nbd.h"
+
+#define COLO_OPT_EXPORT         "export"
+
+#define COLO_CLUSTER_BITS 16
+#define COLO_CLUSTER_SIZE (1 << COLO_CLUSTER_BITS)
+#define COLO_SECTORS_PER_CLUSTER (COLO_CLUSTER_SIZE / BDRV_SECTOR_SIZE)
+
+typedef struct BDRVBlkcoloState BDRVBlkcoloState;
+
+struct BDRVBlkcoloState {
+    BlockDriverState *bs;
+    char *export_name;
+    int mode;
+    disk_buffer disk_buffer;
+    NotifierWithReturn before_write;
+    NBDExport *exp;
+    CowJob cow_job;
+    bool error;
+};
+
+static void colo_svm_init(BDRVBlkcoloState *s);
+static void colo_svm_fini(BDRVBlkcoloState *s);
+
+static int switch_mode(BDRVBlkcoloState *s, int new_mode)
+{
+    if (s->mode == new_mode) {
+        return 0;
+    }
+
+    if (s->mode == COLO_SECONDARY_MODE) {
+        colo_svm_fini(s);
+    }
+
+    s->mode = new_mode;
+    if (s->mode == COLO_SECONDARY_MODE) {
+        colo_svm_init(s);
+    }
+
+    return 0;
+}
+
+/*
+ * Secondary mode functions
+ *
+ * All write requests are forwarded to secondary QEMU from primary QEMU.
+ * The secondary QEMU should do the following things:
+ * 1. Use NBD server to receive and handle the forwarded write requests
+ * 2. Buffer the secondary write requests
+ */
+
+static int coroutine_fn
+colo_svm_co_writev(BlockDriverState *bs, int64_t sector_num,
+                   int nb_sectors, QEMUIOVector *qiov)
+{
+    BDRVBlkcoloState *s = bs->opaque;
+
+    /*
+     * Write the request to the disk buffer. How to limit the
+     * write speed?
+     */
+    qiov_write_to_buffer(&s->disk_buffer, qiov, sector_num, nb_sectors, true);
+
+    return 0;
+}
+
+static int coroutine_fn
+colo_svm_co_readv(BlockDriverState *bs, int64_t sector_num,
+                  int nb_sectors, QEMUIOVector *qiov)
+{
+    BDRVBlkcoloState *s = bs->opaque;
+    int ret;
+
+    /*
+     * Read the sector content from secondary disk first. If the sector
+     * content is buffered, use the buffered content.
+     */
+    ret = bdrv_co_readv(bs->backing_hd, sector_num, nb_sectors, qiov);
+    if (ret) {
+        return ret;
+    }
+
+    /* Read from the buffer */
+    qiov_read_from_buffer(&s->disk_buffer, qiov, sector_num, nb_sectors);
+    return 0;
+}
+
+static int coroutine_fn
+colo_do_cow(BlockDriverState *bs, int64_t sector_num, int nb_sectors)
+{
+    BDRVBlkcoloState *s = bs->origin_file->opaque;
+    CowRequest cow_request;
+    struct iovec iov;
+    QEMUIOVector bounce_qiov;
+    void *bounce_buffer = NULL;
+    int ret = 0;
+    int64_t start, end;
+    int n;
+
+    start = sector_num / COLO_SECTORS_PER_CLUSTER;
+    end = DIV_ROUND_UP(sector_num + nb_sectors, COLO_SECTORS_PER_CLUSTER);
+
+    wait_for_overlapping_requests(&s->cow_job, start, end);
+    cow_request_begin(&cow_request, &s->cow_job, start, end);
+
+    nb_sectors = COLO_SECTORS_PER_CLUSTER;
+    for (; start < end; start++) {
+        sector_num = start * COLO_SECTORS_PER_CLUSTER;
+        if (!buffer_has_empty_range(&s->disk_buffer, sector_num, nb_sectors)) {
+            continue;
+        }
+
+        /* TODO */
+        n = COLO_SECTORS_PER_CLUSTER;
+
+        if (!bounce_buffer) {
+            bounce_buffer = qemu_blockalign(bs, COLO_CLUSTER_SIZE);
+        }
+        iov.iov_base = bounce_buffer;
+        iov.iov_len = n * BDRV_SECTOR_SIZE;
+        qemu_iovec_init_external(&bounce_qiov, &iov, 1);
+
+        ret = bdrv_co_readv(bs, sector_num, n, &bounce_qiov);
+        if (ret < 0) {
+            goto out;
+        }
+
+        qiov_write_to_buffer(&s->disk_buffer, &bounce_qiov,
+                             sector_num, n, false);
+    }
+
+out:
+    cow_request_end(&cow_request);
+    return ret;
+}
+
+static int coroutine_fn
+colo_before_write_notify(NotifierWithReturn *notifier, void *opaque)
+{
+    BdrvTrackedRequest *req = opaque;
+    BlockDriverState *bs = req->bs;
+    BDRVBlkcoloState *s = bs->origin_file->opaque;
+    int64_t sector_num = req->offset >> BDRV_SECTOR_BITS;
+    int nb_sectors = req->bytes >> BDRV_SECTOR_BITS;
+    int ret;
+
+    assert((req->offset & (BDRV_SECTOR_SIZE - 1)) == 0);
+    assert((req->bytes & (BDRV_SECTOR_SIZE - 1)) == 0);
+
+    ret = colo_do_cow(bs, sector_num, nb_sectors);
+    if (ret) {
+        s->error = true;
+    }
+
+    return ret;
+}
+
+/*
+ * It should be called in the migration/checkpoint thread, and the caller
+ * should be hold io thread lock
+ */
+static int svm_do_checkpoint(BDRVBlkcoloState *s)
+{
+    if (s->error) {
+        /* TODO: we should report the error more earlier */
+        return -1;
+    }
+
+    /* clear disk buffer */
+    clear_all_buffered_data(&s->disk_buffer);
+    return 0;
+}
+
+/* It should be called in the migration/checkpoint thread */
+static void svm_stop_replication(BDRVBlkcoloState *s)
+{
+    /* switch to unprotected mode */
+    switch_mode(s, COLO_UNPROTECTED_MODE);
+}
+
+static void colo_svm_init(BDRVBlkcoloState *s)
+{
+    BlockBackend *blk = s->bs->backing_hd->blk;
+
+    /* Init Disk Buffer */
+    init_disk_buffer(&s->disk_buffer);
+
+    s->before_write.notify = colo_before_write_notify;
+    bdrv_add_before_write_notifier(s->bs->backing_hd, &s->before_write);
+
+    /* start NBD server */
+    s->exp = nbd_export_new(blk, 0, -1, 0, NULL);
+    nbd_export_set_name(s->exp, s->export_name);
+
+    s->error = false;
+    QLIST_INIT(&s->cow_job.inflight_reqs);
+}
+
+static void colo_svm_fini(BDRVBlkcoloState *s)
+{
+    /* stop NBD server */
+    nbd_export_close(s->exp);
+    nbd_export_put(s->exp);
+
+    /* notifier_with_return_remove */
+    notifier_with_return_remove(&s->before_write);
+
+    /* TODO: All pvm write requests have been done? */
+
+    /* flush all buffered data to secondary disk */
+    flush_buffered_data_to_disk(&s->disk_buffer, s->bs->backing_hd);
+}
+
+/* block driver interfaces */
+static QemuOptsList colo_runtime_opts = {
+    .name = "colo",
+    .head = QTAILQ_HEAD_INITIALIZER(colo_runtime_opts.head),
+    .desc = {
+        {
+            .name = COLO_OPT_EXPORT,
+            .type = QEMU_OPT_STRING,
+            .help = "The NBD server name",
+        },
+        { /* end of list */ }
+    },
+};
+
+/*
+ * usage: -drive if=xxx,driver=colo,export=xxx,\
+ *        backing.file.filename=1.raw,\
+ *        backing.driver=raw
+ */
+static int blkcolo_open(BlockDriverState *bs, QDict *options, int flags,
+                        Error **errp)
+{
+    BDRVBlkcoloState *s = bs->opaque;
+    Error *local_err = NULL;
+    QemuOpts *opts = NULL;
+    int ret = 0;
+
+    s->bs = bs;
+
+    opts = qemu_opts_create(&colo_runtime_opts, NULL, 0, &error_abort);
+    qemu_opts_absorb_qdict(opts, options, &local_err);
+    if (local_err) {
+        ret = -EINVAL;
+        goto exit;
+    }
+
+    s->export_name = g_strdup(qemu_opt_get(opts, COLO_OPT_EXPORT));
+    if (!s->export_name) {
+        error_setg(&local_err, "Missing the option export");
+        ret = -EINVAL;
+        goto exit;
+    }
+
+exit:
+    qemu_opts_del(opts);
+    /* propagate error */
+    if (local_err) {
+        error_propagate(errp, local_err);
+    }
+    return ret;
+}
+
+static void blkcolo_close(BlockDriverState *bs)
+{
+    BDRVBlkcoloState *s = bs->opaque;
+
+    if (s->mode == COLO_SECONDARY_MODE) {
+        switch_mode(s, COLO_UNPROTECTED_MODE);
+    }
+
+    g_free(s->export_name);
+}
+
+static int64_t blkcolo_getlength(BlockDriverState *bs)
+{
+    if (!bs->backing_hd) {
+        return 0;
+    } else {
+        return bdrv_getlength(bs->backing_hd);
+    }
+}
+
+static int blkcolo_co_readv(BlockDriverState *bs, int64_t sector_num,
+                            int nb_sectors, QEMUIOVector *qiov)
+{
+    BDRVBlkcoloState *s = bs->opaque;
+
+    if (s->mode == COLO_SECONDARY_MODE) {
+        return colo_svm_co_readv(bs, sector_num, nb_sectors, qiov);
+    }
+
+    assert(s->mode == COLO_UNPROTECTED_MODE);
+
+    if (!bs->backing_hd) {
+        return -EIO;
+    } else {
+        return bdrv_co_readv(bs->backing_hd, sector_num, nb_sectors, qiov);
+    }
+}
+
+static int blkcolo_co_writev(BlockDriverState *bs, int64_t sector_num,
+                             int nb_sectors, QEMUIOVector *qiov)
+{
+    BDRVBlkcoloState *s = bs->opaque;
+
+    if (s->mode == COLO_SECONDARY_MODE) {
+        return colo_svm_co_writev(bs, sector_num, nb_sectors, qiov);
+    }
+
+    assert(s->mode == COLO_UNPROTECTED_MODE);
+
+    if (!bs->backing_hd) {
+        return -EIO;
+    } else {
+        return bdrv_co_writev(bs->backing_hd, sector_num, nb_sectors, qiov);
+    }
+}
+
+static int blkcolo_start_replication(BlockDriverState *bs, int mode)
+{
+    BDRVBlkcoloState *s = bs->opaque;
+
+    if (mode != COLO_SECONDARY_MODE ||
+        s->mode != COLO_UNPROTECTED_MODE ||
+        !bs->backing_hd) {
+        return -1;
+    }
+
+    if (!blk_is_inserted(bs->backing_hd->blk)) {
+        return -1;
+    }
+
+    if (blk_is_read_only(bs->backing_hd->blk)) {
+        return -1;
+    }
+
+    return switch_mode(s, mode);
+}
+
+static int blkcolo_do_checkpoint(BlockDriverState *bs)
+{
+    BDRVBlkcoloState *s = bs->opaque;
+
+    if (s->mode != COLO_SECONDARY_MODE) {
+        return -1;
+    }
+
+    return svm_do_checkpoint(s);
+}
+
+static int blkcolo_stop_replication(BlockDriverState *bs)
+{
+    BDRVBlkcoloState *s = bs->opaque;
+
+    if (s->mode != COLO_SECONDARY_MODE) {
+        return -1;
+    }
+
+    svm_stop_replication(s);
+    return 0;
+}
+
+static BlockDriver bdrv_blkcolo = {
+    .format_name                = "blkcolo",
+    .protocol_name              = "blkcolo",
+    .instance_size              = sizeof(BDRVBlkcoloState),
+
+    .bdrv_file_open             = blkcolo_open,
+    .bdrv_close                 = blkcolo_close,
+    .bdrv_getlength             = blkcolo_getlength,
+
+    .bdrv_co_readv              = blkcolo_co_readv,
+    .bdrv_co_writev             = blkcolo_co_writev,
+
+    .bdrv_start_replication     = blkcolo_start_replication,
+    .bdrv_do_checkpoint         = blkcolo_do_checkpoint,
+    .bdrv_stop_replication      = blkcolo_stop_replication,
+
+    .supports_backing           = true,
+    .has_variable_length        = true,
+};
+
+static void bdrv_blkcolo_init(void)
+{
+    bdrv_register(&bdrv_blkcolo);
+};
+
+block_init(bdrv_blkcolo_init);
-- 
2.1.0

Re: [Qemu-devel] [RFC PATCH 02/14] quorom: add a new read pattern

[Gonglei]

On 2015/2/12 11:07, Wen Congyang wrote:
> To block replication, we only need to read from the first child.
> 
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> Cc: Luiz Capitulino <lcapitulino@redhat.com>
> Cc: Michael Roth <mdroth@linux.vnet.ibm.com>

A little typo in title of patch 2 and patch 5:
s/quorom/quorum/

Regards,
-Gonglei

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Fam Zheng]

Hi Congyang,

On Thu, 02/12 11:07, Wen Congyang wrote:
> +== Workflow ==
> +The following is the image of block replication workflow:
> +
> +        +----------------------+            +------------------------+
> +        |Primary Write Requests|            |Secondary Write Requests|
> +        +----------------------+            +------------------------+
> +                  |                                       |
> +                  |                                      (4)
> +                  |                                       V
> +                  |                              /-------------\
> +                  |      Copy and Forward        |             |
> +                  |---------(1)----------+       | Disk Buffer |
> +                  |                      |       |             |
> +                  |                     (3)      \-------------/
> +                  |                 speculative      ^
> +                  |                write through    (2)
> +                  |                      |           |
> +                  V                      V           |
> +           +--------------+           +----------------+
> +           | Primary Disk |           | Secondary Disk |
> +           +--------------+           +----------------+
> +
> +    1) Primary write requests will be copied and forwarded to Secondary
> +       QEMU.
> +    2) Before Primary write requests are written to Secondary disk, the
> +       original sector content will be read from Secondary disk and
> +       buffered in the Disk buffer, but it will not overwrite the existing
> +       sector content in the Disk buffer.

I'm a little confused by the tenses ("will be" versus "are") and terms. I am
reading them as "s/will be/are/g"

Why do you need this buffer?

If both primary and secondary write to the same sector, what is saved in the
buffer?

Fam

> +    3) Primary write requests will be written to Secondary disk.
> +    4) Secondary write requests will be buffered in the Disk buffer and it
> +       will overwrite the existing sector content in the buffer.
> +

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 02/12/2015 03:21 PM, Fam Zheng wrote:
> Hi Congyang,
> 
> On Thu, 02/12 11:07, Wen Congyang wrote:
>> +== Workflow ==
>> +The following is the image of block replication workflow:
>> +
>> +        +----------------------+            +------------------------+
>> +        |Primary Write Requests|            |Secondary Write Requests|
>> +        +----------------------+            +------------------------+
>> +                  |                                       |
>> +                  |                                      (4)
>> +                  |                                       V
>> +                  |                              /-------------\
>> +                  |      Copy and Forward        |             |
>> +                  |---------(1)----------+       | Disk Buffer |
>> +                  |                      |       |             |
>> +                  |                     (3)      \-------------/
>> +                  |                 speculative      ^
>> +                  |                write through    (2)
>> +                  |                      |           |
>> +                  V                      V           |
>> +           +--------------+           +----------------+
>> +           | Primary Disk |           | Secondary Disk |
>> +           +--------------+           +----------------+
>> +
>> +    1) Primary write requests will be copied and forwarded to Secondary
>> +       QEMU.
>> +    2) Before Primary write requests are written to Secondary disk, the
>> +       original sector content will be read from Secondary disk and
>> +       buffered in the Disk buffer, but it will not overwrite the existing
>> +       sector content in the Disk buffer.
> 
> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
> reading them as "s/will be/are/g"
> 
> Why do you need this buffer?

We only sync the disk till next checkpoint. Before next checkpoint, secondary
vm write to the buffer.

> 
> If both primary and secondary write to the same sector, what is saved in the
> buffer?

The primary content will be written to the secondary disk, and the secondary content
is saved in the buffer.

Thanks
Wen Congyang

> 
> Fam
> 
>> +    3) Primary write requests will be written to Secondary disk.
>> +    4) Secondary write requests will be buffered in the Disk buffer and it
>> +       will overwrite the existing sector content in the buffer.
>> +
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Fam Zheng]

On Thu, 02/12 15:40, Wen Congyang wrote:
> On 02/12/2015 03:21 PM, Fam Zheng wrote:
> > Hi Congyang,
> > 
> > On Thu, 02/12 11:07, Wen Congyang wrote:
> >> +== Workflow ==
> >> +The following is the image of block replication workflow:
> >> +
> >> +        +----------------------+            +------------------------+
> >> +        |Primary Write Requests|            |Secondary Write Requests|
> >> +        +----------------------+            +------------------------+
> >> +                  |                                       |
> >> +                  |                                      (4)
> >> +                  |                                       V
> >> +                  |                              /-------------\
> >> +                  |      Copy and Forward        |             |
> >> +                  |---------(1)----------+       | Disk Buffer |
> >> +                  |                      |       |             |
> >> +                  |                     (3)      \-------------/
> >> +                  |                 speculative      ^
> >> +                  |                write through    (2)
> >> +                  |                      |           |
> >> +                  V                      V           |
> >> +           +--------------+           +----------------+
> >> +           | Primary Disk |           | Secondary Disk |
> >> +           +--------------+           +----------------+
> >> +
> >> +    1) Primary write requests will be copied and forwarded to Secondary
> >> +       QEMU.
> >> +    2) Before Primary write requests are written to Secondary disk, the
> >> +       original sector content will be read from Secondary disk and
> >> +       buffered in the Disk buffer, but it will not overwrite the existing
> >> +       sector content in the Disk buffer.
> > 
> > I'm a little confused by the tenses ("will be" versus "are") and terms. I am
> > reading them as "s/will be/are/g"
> > 
> > Why do you need this buffer?
> 
> We only sync the disk till next checkpoint. Before next checkpoint, secondary
> vm write to the buffer.
> 
> > 
> > If both primary and secondary write to the same sector, what is saved in the
> > buffer?
> 
> The primary content will be written to the secondary disk, and the secondary content
> is saved in the buffer.

I wonder if alternatively this is possible with an imaginary "writable backing
image" feature, as described below.

When we have a normal backing chain,

               {virtio-blk dev 'foo'}
                         |
                         |
                         |
    [base] <- [mid] <- (foo)

Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
to an existing image on top,

               {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
                         |                              |
                         |                              |
                         |                              |
    [base] <- [mid] <- (foo)  <---------------------- (bar)

It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
We can utilize an automatic hidden drive-backup target:

               {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
                         |                                                          |
                         |                                                          |
                         v                                                          v

    [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)

                         v                              ^
                         v                              ^
                         v                              ^
                         v                              ^
                         >>>> drive-backup sync=none >>>>

So when guest writes to 'foo', the old data is moved to (hidden target), which
remains unchanged from (bar)'s PoV.

The drive in the middle is called hidden because QEMU creates it automatically,
the naming is arbitrary.

It is interesting because it is a more generalized case of image fleecing,
where the (hidden target) is exposed via NBD server for data scanning (read
only) purpose.

More interestingly, with above facility, it is also possible to create a guest
visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
cheaply. Or call it shadow copy if you will.

Back to the COLO case, the configuration will be very similar:


                      {primary wr}                                                {secondary vm}
                            |                                                           |
                            |                                                           |
                            |                                                           |
                            v                                                           v

   [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)

                            v                              ^
                            v                              ^
                            v                              ^
                            v                              ^
                            >>>> drive-backup sync=none >>>>

The workflow analogue is:

> >> +    1) Primary write requests will be copied and forwarded to Secondary
> >> +       QEMU.

Primary write requests are forwarded to secondary QEMU as well.

> >> +    2) Before Primary write requests are written to Secondary disk, the
> >> +       original sector content will be read from Secondary disk and
> >> +       buffered in the Disk buffer, but it will not overwrite the existing
> >> +       sector content in the Disk buffer.

Before Primary write requests are written to (nbd target), aka the Secondary
disk, the orignal sector content is read from it and copied to (hidden buf
disk) by drive-backup. It obviously will not overwrite the data in (active
disk).

> >> +    3) Primary write requests will be written to Secondary disk.

Primary write requests are written to (nbd target).

> >> +    4) Secondary write requests will be buffered in the Disk buffer and it
> >> +       will overwrite the existing sector content in the buffer.

Secondary write request will be written in (active disk) as usual.

Finally, when checkpoint arrives, if you want to sync with primary, just drop
data in (hidden buf disk) and (active disk); when failover happends, if you
want to promote secondary vm, you can commit (active disk) to (nbd target), and
drop data in (hidden buf disk).

Fam

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 02/12/2015 04:44 PM, Fam Zheng wrote:
> On Thu, 02/12 15:40, Wen Congyang wrote:
>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
>>> Hi Congyang,
>>>
>>> On Thu, 02/12 11:07, Wen Congyang wrote:
>>>> +== Workflow ==
>>>> +The following is the image of block replication workflow:
>>>> +
>>>> +        +----------------------+            +------------------------+
>>>> +        |Primary Write Requests|            |Secondary Write Requests|
>>>> +        +----------------------+            +------------------------+
>>>> +                  |                                       |
>>>> +                  |                                      (4)
>>>> +                  |                                       V
>>>> +                  |                              /-------------\
>>>> +                  |      Copy and Forward        |             |
>>>> +                  |---------(1)----------+       | Disk Buffer |
>>>> +                  |                      |       |             |
>>>> +                  |                     (3)      \-------------/
>>>> +                  |                 speculative      ^
>>>> +                  |                write through    (2)
>>>> +                  |                      |           |
>>>> +                  V                      V           |
>>>> +           +--------------+           +----------------+
>>>> +           | Primary Disk |           | Secondary Disk |
>>>> +           +--------------+           +----------------+
>>>> +
>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>> +       QEMU.
>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>> +       original sector content will be read from Secondary disk and
>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>> +       sector content in the Disk buffer.
>>>
>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
>>> reading them as "s/will be/are/g"
>>>
>>> Why do you need this buffer?
>>
>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
>> vm write to the buffer.
>>
>>>
>>> If both primary and secondary write to the same sector, what is saved in the
>>> buffer?
>>
>> The primary content will be written to the secondary disk, and the secondary content
>> is saved in the buffer.
> 
> I wonder if alternatively this is possible with an imaginary "writable backing
> image" feature, as described below.
> 
> When we have a normal backing chain,
> 
>                {virtio-blk dev 'foo'}
>                          |
>                          |
>                          |
>     [base] <- [mid] <- (foo)
> 
> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
> to an existing image on top,
> 
>                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
>                          |                              |
>                          |                              |
>                          |                              |
>     [base] <- [mid] <- (foo)  <---------------------- (bar)
> 
> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
> We can utilize an automatic hidden drive-backup target:
> 
>                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
>                          |                                                          |
>                          |                                                          |
>                          v                                                          v
> 
>     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
> 
>                          v                              ^
>                          v                              ^
>                          v                              ^
>                          v                              ^
>                          >>>> drive-backup sync=none >>>>
> 
> So when guest writes to 'foo', the old data is moved to (hidden target), which
> remains unchanged from (bar)'s PoV.
> 
> The drive in the middle is called hidden because QEMU creates it automatically,
> the naming is arbitrary.
> 
> It is interesting because it is a more generalized case of image fleecing,
> where the (hidden target) is exposed via NBD server for data scanning (read
> only) purpose.
> 
> More interestingly, with above facility, it is also possible to create a guest
> visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
> cheaply. Or call it shadow copy if you will.
> 
> Back to the COLO case, the configuration will be very similar:
> 
> 
>                       {primary wr}                                                {secondary vm}
>                             |                                                           |
>                             |                                                           |
>                             |                                                           |
>                             v                                                           v
> 
>    [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)
> 
>                             v                              ^
>                             v                              ^
>                             v                              ^
>                             v                              ^
>                             >>>> drive-backup sync=none >>>>

What is active disk? There are two disk images?

Thanks
Wen Congyang

> 
> The workflow analogue is:
> 
>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>> +       QEMU.
> 
> Primary write requests are forwarded to secondary QEMU as well.
> 
>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>> +       original sector content will be read from Secondary disk and
>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>> +       sector content in the Disk buffer.
> 
> Before Primary write requests are written to (nbd target), aka the Secondary
> disk, the orignal sector content is read from it and copied to (hidden buf
> disk) by drive-backup. It obviously will not overwrite the data in (active
> disk).
> 
>>>> +    3) Primary write requests will be written to Secondary disk.
> 
> Primary write requests are written to (nbd target).
> 
>>>> +    4) Secondary write requests will be buffered in the Disk buffer and it
>>>> +       will overwrite the existing sector content in the buffer.
> 
> Secondary write request will be written in (active disk) as usual.
> 
> Finally, when checkpoint arrives, if you want to sync with primary, just drop
> data in (hidden buf disk) and (active disk); when failover happends, if you
> want to promote secondary vm, you can commit (active disk) to (nbd target), and
> drop data in (hidden buf disk).
> 
> Fam
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Hongyang Yang]

Hi Fam,

在 02/12/2015 04:44 PM, Fam Zheng 写道:
> On Thu, 02/12 15:40, Wen Congyang wrote:
>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
>>> Hi Congyang,
>>>
>>> On Thu, 02/12 11:07, Wen Congyang wrote:
>>>> +== Workflow ==
>>>> +The following is the image of block replication workflow:
>>>> +
>>>> +        +----------------------+            +------------------------+
>>>> +        |Primary Write Requests|            |Secondary Write Requests|
>>>> +        +----------------------+            +------------------------+
>>>> +                  |                                       |
>>>> +                  |                                      (4)
>>>> +                  |                                       V
>>>> +                  |                              /-------------\
>>>> +                  |      Copy and Forward        |             |
>>>> +                  |---------(1)----------+       | Disk Buffer |
>>>> +                  |                      |       |             |
>>>> +                  |                     (3)      \-------------/
>>>> +                  |                 speculative      ^
>>>> +                  |                write through    (2)
>>>> +                  |                      |           |
>>>> +                  V                      V           |
>>>> +           +--------------+           +----------------+
>>>> +           | Primary Disk |           | Secondary Disk |
>>>> +           +--------------+           +----------------+
>>>> +
>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>> +       QEMU.
>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>> +       original sector content will be read from Secondary disk and
>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>> +       sector content in the Disk buffer.
>>>
>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
>>> reading them as "s/will be/are/g"
>>>
>>> Why do you need this buffer?
>>
>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
>> vm write to the buffer.
>>
>>>
>>> If both primary and secondary write to the same sector, what is saved in the
>>> buffer?
>>
>> The primary content will be written to the secondary disk, and the secondary content
>> is saved in the buffer.
>
> I wonder if alternatively this is possible with an imaginary "writable backing
> image" feature, as described below.
>
> When we have a normal backing chain,
>
>                 {virtio-blk dev 'foo'}
>                           |
>                           |
>                           |
>      [base] <- [mid] <- (foo)
>
> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
> to an existing image on top,
>
>                 {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
>                           |                              |
>                           |                              |
>                           |                              |
>      [base] <- [mid] <- (foo)  <---------------------- (bar)
>
> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
> We can utilize an automatic hidden drive-backup target:
>
>                 {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
>                           |                                                          |
>                           |                                                          |
>                           v                                                          v
>
>      [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
>
>                           v                              ^
>                           v                              ^
>                           v                              ^
>                           v                              ^
>                           >>>> drive-backup sync=none >>>>
>
> So when guest writes to 'foo', the old data is moved to (hidden target), which
> remains unchanged from (bar)'s PoV.
>
> The drive in the middle is called hidden because QEMU creates it automatically,
> the naming is arbitrary.
>
> It is interesting because it is a more generalized case of image fleecing,
> where the (hidden target) is exposed via NBD server for data scanning (read
> only) purpose.
>
> More interestingly, with above facility, it is also possible to create a guest
> visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
> cheaply. Or call it shadow copy if you will.
>
> Back to the COLO case, the configuration will be very similar:
>
>
>                        {primary wr}                                                {secondary vm}
>                              |                                                           |
>                              |                                                           |
>                              |                                                           |
>                              v                                                           v
>
>     [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)
>
>                              v                              ^
>                              v                              ^
>                              v                              ^
>                              v                              ^
>                              >>>> drive-backup sync=none >>>>
>
> The workflow analogue is:
>
>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>> +       QEMU.
>
> Primary write requests are forwarded to secondary QEMU as well.
>
>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>> +       original sector content will be read from Secondary disk and
>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>> +       sector content in the Disk buffer.
>
> Before Primary write requests are written to (nbd target), aka the Secondary
> disk, the orignal sector content is read from it and copied to (hidden buf
> disk) by drive-backup. It obviously will not overwrite the data in (active
> disk).
>
>>>> +    3) Primary write requests will be written to Secondary disk.
>
> Primary write requests are written to (nbd target).
>
>>>> +    4) Secondary write requests will be buffered in the Disk buffer and it
>>>> +       will overwrite the existing sector content in the buffer.
>
> Secondary write request will be written in (active disk) as usual.
>
> Finally, when checkpoint arrives, if you want to sync with primary, just drop
> data in (hidden buf disk) and (active disk); when failover happends, if you
> want to promote secondary vm, you can commit (active disk) to (nbd target), and
> drop data in (hidden buf disk).

If I understand correctly, you split the Disk Buffer into a hidden buf disk +
an active disk. What we need to do is only to implement a buf disk(will be
used as hidden buf disk and active disk as mentioned), apart from this, we can
use the existing mechinism like backing-file/drive-backup?

>
> Fam
> .
>

-- 
Thanks,
Yang.

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Fam Zheng]

On Thu, 02/12 17:33, Wen Congyang wrote:
> On 02/12/2015 04:44 PM, Fam Zheng wrote:
> > On Thu, 02/12 15:40, Wen Congyang wrote:
> >> On 02/12/2015 03:21 PM, Fam Zheng wrote:
> >>> Hi Congyang,
> >>>
> >>> On Thu, 02/12 11:07, Wen Congyang wrote:
> >>>> +== Workflow ==
> >>>> +The following is the image of block replication workflow:
> >>>> +
> >>>> +        +----------------------+            +------------------------+
> >>>> +        |Primary Write Requests|            |Secondary Write Requests|
> >>>> +        +----------------------+            +------------------------+
> >>>> +                  |                                       |
> >>>> +                  |                                      (4)
> >>>> +                  |                                       V
> >>>> +                  |                              /-------------\
> >>>> +                  |      Copy and Forward        |             |
> >>>> +                  |---------(1)----------+       | Disk Buffer |
> >>>> +                  |                      |       |             |
> >>>> +                  |                     (3)      \-------------/
> >>>> +                  |                 speculative      ^
> >>>> +                  |                write through    (2)
> >>>> +                  |                      |           |
> >>>> +                  V                      V           |
> >>>> +           +--------------+           +----------------+
> >>>> +           | Primary Disk |           | Secondary Disk |
> >>>> +           +--------------+           +----------------+
> >>>> +
> >>>> +    1) Primary write requests will be copied and forwarded to Secondary
> >>>> +       QEMU.
> >>>> +    2) Before Primary write requests are written to Secondary disk, the
> >>>> +       original sector content will be read from Secondary disk and
> >>>> +       buffered in the Disk buffer, but it will not overwrite the existing
> >>>> +       sector content in the Disk buffer.
> >>>
> >>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
> >>> reading them as "s/will be/are/g"
> >>>
> >>> Why do you need this buffer?
> >>
> >> We only sync the disk till next checkpoint. Before next checkpoint, secondary
> >> vm write to the buffer.
> >>
> >>>
> >>> If both primary and secondary write to the same sector, what is saved in the
> >>> buffer?
> >>
> >> The primary content will be written to the secondary disk, and the secondary content
> >> is saved in the buffer.
> > 
> > I wonder if alternatively this is possible with an imaginary "writable backing
> > image" feature, as described below.
> > 
> > When we have a normal backing chain,
> > 
> >                {virtio-blk dev 'foo'}
> >                          |
> >                          |
> >                          |
> >     [base] <- [mid] <- (foo)
> > 
> > Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
> > to an existing image on top,
> > 
> >                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
> >                          |                              |
> >                          |                              |
> >                          |                              |
> >     [base] <- [mid] <- (foo)  <---------------------- (bar)
> > 
> > It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
> > We can utilize an automatic hidden drive-backup target:
> > 
> >                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
> >                          |                                                          |
> >                          |                                                          |
> >                          v                                                          v
> > 
> >     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
> > 
> >                          v                              ^
> >                          v                              ^
> >                          v                              ^
> >                          v                              ^
> >                          >>>> drive-backup sync=none >>>>
> > 
> > So when guest writes to 'foo', the old data is moved to (hidden target), which
> > remains unchanged from (bar)'s PoV.
> > 
> > The drive in the middle is called hidden because QEMU creates it automatically,
> > the naming is arbitrary.
> > 
> > It is interesting because it is a more generalized case of image fleecing,
> > where the (hidden target) is exposed via NBD server for data scanning (read
> > only) purpose.
> > 
> > More interestingly, with above facility, it is also possible to create a guest
> > visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
> > cheaply. Or call it shadow copy if you will.
> > 
> > Back to the COLO case, the configuration will be very similar:
> > 
> > 
> >                       {primary wr}                                                {secondary vm}
> >                             |                                                           |
> >                             |                                                           |
> >                             |                                                           |
> >                             v                                                           v
> > 
> >    [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)
> > 
> >                             v                              ^
> >                             v                              ^
> >                             v                              ^
> >                             v                              ^
> >                             >>>> drive-backup sync=none >>>>
> 
> What is active disk? There are two disk images?

It starts as an empty image with (hidden buf disk) as backing file, which in
turn has (nbd target) as backing file.

Fam

> 
> Thanks
> Wen Congyang
> 
> > 
> > The workflow analogue is:
> > 
> >>>> +    1) Primary write requests will be copied and forwarded to Secondary
> >>>> +       QEMU.
> > 
> > Primary write requests are forwarded to secondary QEMU as well.
> > 
> >>>> +    2) Before Primary write requests are written to Secondary disk, the
> >>>> +       original sector content will be read from Secondary disk and
> >>>> +       buffered in the Disk buffer, but it will not overwrite the existing
> >>>> +       sector content in the Disk buffer.
> > 
> > Before Primary write requests are written to (nbd target), aka the Secondary
> > disk, the orignal sector content is read from it and copied to (hidden buf
> > disk) by drive-backup. It obviously will not overwrite the data in (active
> > disk).
> > 
> >>>> +    3) Primary write requests will be written to Secondary disk.
> > 
> > Primary write requests are written to (nbd target).
> > 
> >>>> +    4) Secondary write requests will be buffered in the Disk buffer and it
> >>>> +       will overwrite the existing sector content in the buffer.
> > 
> > Secondary write request will be written in (active disk) as usual.
> > 
> > Finally, when checkpoint arrives, if you want to sync with primary, just drop
> > data in (hidden buf disk) and (active disk); when failover happends, if you
> > want to promote secondary vm, you can commit (active disk) to (nbd target), and
> > drop data in (hidden buf disk).
> > 
> > Fam
> > .
> > 
> 
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Fam Zheng]

On Thu, 02/12 17:36, Hongyang Yang wrote:
> Hi Fam,
> 
> 在 02/12/2015 04:44 PM, Fam Zheng 写道:
> >On Thu, 02/12 15:40, Wen Congyang wrote:
> >>On 02/12/2015 03:21 PM, Fam Zheng wrote:
> >>>Hi Congyang,
> >>>
> >>>On Thu, 02/12 11:07, Wen Congyang wrote:
> >>>>+== Workflow ==
> >>>>+The following is the image of block replication workflow:
> >>>>+
> >>>>+        +----------------------+            +------------------------+
> >>>>+        |Primary Write Requests|            |Secondary Write Requests|
> >>>>+        +----------------------+            +------------------------+
> >>>>+                  |                                       |
> >>>>+                  |                                      (4)
> >>>>+                  |                                       V
> >>>>+                  |                              /-------------\
> >>>>+                  |      Copy and Forward        |             |
> >>>>+                  |---------(1)----------+       | Disk Buffer |
> >>>>+                  |                      |       |             |
> >>>>+                  |                     (3)      \-------------/
> >>>>+                  |                 speculative      ^
> >>>>+                  |                write through    (2)
> >>>>+                  |                      |           |
> >>>>+                  V                      V           |
> >>>>+           +--------------+           +----------------+
> >>>>+           | Primary Disk |           | Secondary Disk |
> >>>>+           +--------------+           +----------------+
> >>>>+
> >>>>+    1) Primary write requests will be copied and forwarded to Secondary
> >>>>+       QEMU.
> >>>>+    2) Before Primary write requests are written to Secondary disk, the
> >>>>+       original sector content will be read from Secondary disk and
> >>>>+       buffered in the Disk buffer, but it will not overwrite the existing
> >>>>+       sector content in the Disk buffer.
> >>>
> >>>I'm a little confused by the tenses ("will be" versus "are") and terms. I am
> >>>reading them as "s/will be/are/g"
> >>>
> >>>Why do you need this buffer?
> >>
> >>We only sync the disk till next checkpoint. Before next checkpoint, secondary
> >>vm write to the buffer.
> >>
> >>>
> >>>If both primary and secondary write to the same sector, what is saved in the
> >>>buffer?
> >>
> >>The primary content will be written to the secondary disk, and the secondary content
> >>is saved in the buffer.
> >
> >I wonder if alternatively this is possible with an imaginary "writable backing
> >image" feature, as described below.
> >
> >When we have a normal backing chain,
> >
> >                {virtio-blk dev 'foo'}
> >                          |
> >                          |
> >                          |
> >     [base] <- [mid] <- (foo)
> >
> >Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
> >to an existing image on top,
> >
> >                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
> >                          |                              |
> >                          |                              |
> >                          |                              |
> >     [base] <- [mid] <- (foo)  <---------------------- (bar)
> >
> >It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
> >We can utilize an automatic hidden drive-backup target:
> >
> >                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
> >                          |                                                          |
> >                          |                                                          |
> >                          v                                                          v
> >
> >     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
> >
> >                          v                              ^
> >                          v                              ^
> >                          v                              ^
> >                          v                              ^
> >                          >>>> drive-backup sync=none >>>>
> >
> >So when guest writes to 'foo', the old data is moved to (hidden target), which
> >remains unchanged from (bar)'s PoV.
> >
> >The drive in the middle is called hidden because QEMU creates it automatically,
> >the naming is arbitrary.
> >
> >It is interesting because it is a more generalized case of image fleecing,
> >where the (hidden target) is exposed via NBD server for data scanning (read
> >only) purpose.
> >
> >More interestingly, with above facility, it is also possible to create a guest
> >visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
> >cheaply. Or call it shadow copy if you will.
> >
> >Back to the COLO case, the configuration will be very similar:
> >
> >
> >                       {primary wr}                                                {secondary vm}
> >                             |                                                           |
> >                             |                                                           |
> >                             |                                                           |
> >                             v                                                           v
> >
> >    [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)
> >
> >                             v                              ^
> >                             v                              ^
> >                             v                              ^
> >                             v                              ^
> >                             >>>> drive-backup sync=none >>>>
> >
> >The workflow analogue is:
> >
> >>>>+    1) Primary write requests will be copied and forwarded to Secondary
> >>>>+       QEMU.
> >
> >Primary write requests are forwarded to secondary QEMU as well.
> >
> >>>>+    2) Before Primary write requests are written to Secondary disk, the
> >>>>+       original sector content will be read from Secondary disk and
> >>>>+       buffered in the Disk buffer, but it will not overwrite the existing
> >>>>+       sector content in the Disk buffer.
> >
> >Before Primary write requests are written to (nbd target), aka the Secondary
> >disk, the orignal sector content is read from it and copied to (hidden buf
> >disk) by drive-backup. It obviously will not overwrite the data in (active
> >disk).
> >
> >>>>+    3) Primary write requests will be written to Secondary disk.
> >
> >Primary write requests are written to (nbd target).
> >
> >>>>+    4) Secondary write requests will be buffered in the Disk buffer and it
> >>>>+       will overwrite the existing sector content in the buffer.
> >
> >Secondary write request will be written in (active disk) as usual.
> >
> >Finally, when checkpoint arrives, if you want to sync with primary, just drop
> >data in (hidden buf disk) and (active disk); when failover happends, if you
> >want to promote secondary vm, you can commit (active disk) to (nbd target), and
> >drop data in (hidden buf disk).
> 
> If I understand correctly, you split the Disk Buffer into a hidden buf disk +
> an active disk. What we need to do is only to implement a buf disk(will be
> used as hidden buf disk and active disk as mentioned), apart from this, we can
> use the existing mechinism like backing-file/drive-backup?
> 

Yes, but you need a separate driver to take care of the buffer logic as
introduced in this series, which is less generic, but does the same thing we
will need in the image fleecing use case.

Fam

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 02/12/2015 05:44 PM, Fam Zheng wrote:
> On Thu, 02/12 17:33, Wen Congyang wrote:
>> On 02/12/2015 04:44 PM, Fam Zheng wrote:
>>> On Thu, 02/12 15:40, Wen Congyang wrote:
>>>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
>>>>> Hi Congyang,
>>>>>
>>>>> On Thu, 02/12 11:07, Wen Congyang wrote:
>>>>>> +== Workflow ==
>>>>>> +The following is the image of block replication workflow:
>>>>>> +
>>>>>> +        +----------------------+            +------------------------+
>>>>>> +        |Primary Write Requests|            |Secondary Write Requests|
>>>>>> +        +----------------------+            +------------------------+
>>>>>> +                  |                                       |
>>>>>> +                  |                                      (4)
>>>>>> +                  |                                       V
>>>>>> +                  |                              /-------------\
>>>>>> +                  |      Copy and Forward        |             |
>>>>>> +                  |---------(1)----------+       | Disk Buffer |
>>>>>> +                  |                      |       |             |
>>>>>> +                  |                     (3)      \-------------/
>>>>>> +                  |                 speculative      ^
>>>>>> +                  |                write through    (2)
>>>>>> +                  |                      |           |
>>>>>> +                  V                      V           |
>>>>>> +           +--------------+           +----------------+
>>>>>> +           | Primary Disk |           | Secondary Disk |
>>>>>> +           +--------------+           +----------------+
>>>>>> +
>>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>>>> +       QEMU.
>>>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>>>> +       original sector content will be read from Secondary disk and
>>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>>>> +       sector content in the Disk buffer.
>>>>>
>>>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
>>>>> reading them as "s/will be/are/g"
>>>>>
>>>>> Why do you need this buffer?
>>>>
>>>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
>>>> vm write to the buffer.
>>>>
>>>>>
>>>>> If both primary and secondary write to the same sector, what is saved in the
>>>>> buffer?
>>>>
>>>> The primary content will be written to the secondary disk, and the secondary content
>>>> is saved in the buffer.
>>>
>>> I wonder if alternatively this is possible with an imaginary "writable backing
>>> image" feature, as described below.
>>>
>>> When we have a normal backing chain,
>>>
>>>                {virtio-blk dev 'foo'}
>>>                          |
>>>                          |
>>>                          |
>>>     [base] <- [mid] <- (foo)
>>>
>>> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
>>> to an existing image on top,
>>>
>>>                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
>>>                          |                              |
>>>                          |                              |
>>>                          |                              |
>>>     [base] <- [mid] <- (foo)  <---------------------- (bar)
>>>
>>> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
>>> We can utilize an automatic hidden drive-backup target:
>>>
>>>                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
>>>                          |                                                          |
>>>                          |                                                          |
>>>                          v                                                          v
>>>
>>>     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
>>>
>>>                          v                              ^
>>>                          v                              ^
>>>                          v                              ^
>>>                          v                              ^
>>>                          >>>> drive-backup sync=none >>>>
>>>
>>> So when guest writes to 'foo', the old data is moved to (hidden target), which
>>> remains unchanged from (bar)'s PoV.
>>>
>>> The drive in the middle is called hidden because QEMU creates it automatically,
>>> the naming is arbitrary.
>>>
>>> It is interesting because it is a more generalized case of image fleecing,
>>> where the (hidden target) is exposed via NBD server for data scanning (read
>>> only) purpose.
>>>
>>> More interestingly, with above facility, it is also possible to create a guest
>>> visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
>>> cheaply. Or call it shadow copy if you will.
>>>
>>> Back to the COLO case, the configuration will be very similar:
>>>
>>>
>>>                       {primary wr}                                                {secondary vm}
>>>                             |                                                           |
>>>                             |                                                           |
>>>                             |                                                           |
>>>                             v                                                           v
>>>
>>>    [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)
>>>
>>>                             v                              ^
>>>                             v                              ^
>>>                             v                              ^
>>>                             v                              ^
>>>                             >>>> drive-backup sync=none >>>>
>>
>> What is active disk? There are two disk images?
> 
> It starts as an empty image with (hidden buf disk) as backing file, which in
> turn has (nbd target) as backing file.

It's too complicated..., and I don't understand it.
1. What is active disk? Use raw or a new block driver?
2. Hidden buf disk use new block driver?
3. nbd target is hidden buf disk's backing image? If it is opened read-only, we will
   export a nbd with read-only BlockDriverState, but nbd server needs to write it.

Thanks
Wen Congyang

> 
> Fam
> 
>>
>> Thanks
>> Wen Congyang
>>
>>>
>>> The workflow analogue is:
>>>
>>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>>>> +       QEMU.
>>>
>>> Primary write requests are forwarded to secondary QEMU as well.
>>>
>>>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>>>> +       original sector content will be read from Secondary disk and
>>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>>>> +       sector content in the Disk buffer.
>>>
>>> Before Primary write requests are written to (nbd target), aka the Secondary
>>> disk, the orignal sector content is read from it and copied to (hidden buf
>>> disk) by drive-backup. It obviously will not overwrite the data in (active
>>> disk).
>>>
>>>>>> +    3) Primary write requests will be written to Secondary disk.
>>>
>>> Primary write requests are written to (nbd target).
>>>
>>>>>> +    4) Secondary write requests will be buffered in the Disk buffer and it
>>>>>> +       will overwrite the existing sector content in the buffer.
>>>
>>> Secondary write request will be written in (active disk) as usual.
>>>
>>> Finally, when checkpoint arrives, if you want to sync with primary, just drop
>>> data in (hidden buf disk) and (active disk); when failover happends, if you
>>> want to promote secondary vm, you can commit (active disk) to (nbd target), and
>>> drop data in (hidden buf disk).
>>>
>>> Fam
>>> .
>>>
>>
>>
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[famz]

On Thu, 02/12 18:11, Wen Congyang wrote:
> On 02/12/2015 05:44 PM, Fam Zheng wrote:
> > On Thu, 02/12 17:33, Wen Congyang wrote:
> >> On 02/12/2015 04:44 PM, Fam Zheng wrote:
> >>> On Thu, 02/12 15:40, Wen Congyang wrote:
> >>>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
> >>>>> Hi Congyang,
> >>>>>
> >>>>> On Thu, 02/12 11:07, Wen Congyang wrote:
> >>>>>> +== Workflow ==
> >>>>>> +The following is the image of block replication workflow:
> >>>>>> +
> >>>>>> +        +----------------------+            +------------------------+
> >>>>>> +        |Primary Write Requests|            |Secondary Write Requests|
> >>>>>> +        +----------------------+            +------------------------+
> >>>>>> +                  |                                       |
> >>>>>> +                  |                                      (4)
> >>>>>> +                  |                                       V
> >>>>>> +                  |                              /-------------\
> >>>>>> +                  |      Copy and Forward        |             |
> >>>>>> +                  |---------(1)----------+       | Disk Buffer |
> >>>>>> +                  |                      |       |             |
> >>>>>> +                  |                     (3)      \-------------/
> >>>>>> +                  |                 speculative      ^
> >>>>>> +                  |                write through    (2)
> >>>>>> +                  |                      |           |
> >>>>>> +                  V                      V           |
> >>>>>> +           +--------------+           +----------------+
> >>>>>> +           | Primary Disk |           | Secondary Disk |
> >>>>>> +           +--------------+           +----------------+
> >>>>>> +
> >>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
> >>>>>> +       QEMU.
> >>>>>> +    2) Before Primary write requests are written to Secondary disk, the
> >>>>>> +       original sector content will be read from Secondary disk and
> >>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
> >>>>>> +       sector content in the Disk buffer.
> >>>>>
> >>>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
> >>>>> reading them as "s/will be/are/g"
> >>>>>
> >>>>> Why do you need this buffer?
> >>>>
> >>>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
> >>>> vm write to the buffer.
> >>>>
> >>>>>
> >>>>> If both primary and secondary write to the same sector, what is saved in the
> >>>>> buffer?
> >>>>
> >>>> The primary content will be written to the secondary disk, and the secondary content
> >>>> is saved in the buffer.
> >>>
> >>> I wonder if alternatively this is possible with an imaginary "writable backing
> >>> image" feature, as described below.
> >>>
> >>> When we have a normal backing chain,
> >>>
> >>>                {virtio-blk dev 'foo'}
> >>>                          |
> >>>                          |
> >>>                          |
> >>>     [base] <- [mid] <- (foo)
> >>>
> >>> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
> >>> to an existing image on top,
> >>>
> >>>                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
> >>>                          |                              |
> >>>                          |                              |
> >>>                          |                              |
> >>>     [base] <- [mid] <- (foo)  <---------------------- (bar)
> >>>
> >>> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
> >>> We can utilize an automatic hidden drive-backup target:
> >>>
> >>>                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
> >>>                          |                                                          |
> >>>                          |                                                          |
> >>>                          v                                                          v
> >>>
> >>>     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
> >>>
> >>>                          v                              ^
> >>>                          v                              ^
> >>>                          v                              ^
> >>>                          v                              ^
> >>>                          >>>> drive-backup sync=none >>>>
> >>>
> >>> So when guest writes to 'foo', the old data is moved to (hidden target), which
> >>> remains unchanged from (bar)'s PoV.
> >>>
> >>> The drive in the middle is called hidden because QEMU creates it automatically,
> >>> the naming is arbitrary.
> >>>
> >>> It is interesting because it is a more generalized case of image fleecing,
> >>> where the (hidden target) is exposed via NBD server for data scanning (read
> >>> only) purpose.
> >>>
> >>> More interestingly, with above facility, it is also possible to create a guest
> >>> visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
> >>> cheaply. Or call it shadow copy if you will.
> >>>
> >>> Back to the COLO case, the configuration will be very similar:
> >>>
> >>>
> >>>                       {primary wr}                                                {secondary vm}
> >>>                             |                                                           |
> >>>                             |                                                           |
> >>>                             |                                                           |
> >>>                             v                                                           v
> >>>
> >>>    [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)
> >>>
> >>>                             v                              ^
> >>>                             v                              ^
> >>>                             v                              ^
> >>>                             v                              ^
> >>>                             >>>> drive-backup sync=none >>>>
> >>
> >> What is active disk? There are two disk images?
> > 
> > It starts as an empty image with (hidden buf disk) as backing file, which in
> > turn has (nbd target) as backing file.
> 
> It's too complicated..., and I don't understand it.
> 1. What is active disk? Use raw or a new block driver?

It is an empty qcow2 image with the same lenght as your Secondary Disk.

> 2. Hidden buf disk use new block driver?

It is an empty qcow2 image with the same lenght as your Secondary Disk, too.

> 3. nbd target is hidden buf disk's backing image? If it is opened read-only, we will
>    export a nbd with read-only BlockDriverState, but nbd server needs to write it.

NBD target is your Secondary Disk. It is opened read-write.

The patches to enable opening it as read-write, and starting drive-backup
between it and hidden buf disk, are all work in progress (the core concept) of
image fleecing.

Fam

> >>>
> >>> The workflow analogue is:
> >>>
> >>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
> >>>>>> +       QEMU.
> >>>
> >>> Primary write requests are forwarded to secondary QEMU as well.
> >>>
> >>>>>> +    2) Before Primary write requests are written to Secondary disk, the
> >>>>>> +       original sector content will be read from Secondary disk and
> >>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
> >>>>>> +       sector content in the Disk buffer.
> >>>
> >>> Before Primary write requests are written to (nbd target), aka the Secondary
> >>> disk, the orignal sector content is read from it and copied to (hidden buf
> >>> disk) by drive-backup. It obviously will not overwrite the data in (active
> >>> disk).
> >>>
> >>>>>> +    3) Primary write requests will be written to Secondary disk.
> >>>
> >>> Primary write requests are written to (nbd target).
> >>>
> >>>>>> +    4) Secondary write requests will be buffered in the Disk buffer and it
> >>>>>> +       will overwrite the existing sector content in the buffer.
> >>>
> >>> Secondary write request will be written in (active disk) as usual.
> >>>
> >>> Finally, when checkpoint arrives, if you want to sync with primary, just drop
> >>> data in (hidden buf disk) and (active disk); when failover happends, if you
> >>> want to promote secondary vm, you can commit (active disk) to (nbd target), and
> >>> drop data in (hidden buf disk).
> >>>
> >>> Fam
> >>> .
> >>>
> >>
> >>
> > .
> > 
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 02/12/2015 06:26 PM, famz@redhat.com wrote:
> On Thu, 02/12 18:11, Wen Congyang wrote:
>> On 02/12/2015 05:44 PM, Fam Zheng wrote:
>>> On Thu, 02/12 17:33, Wen Congyang wrote:
>>>> On 02/12/2015 04:44 PM, Fam Zheng wrote:
>>>>> On Thu, 02/12 15:40, Wen Congyang wrote:
>>>>>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
>>>>>>> Hi Congyang,
>>>>>>>
>>>>>>> On Thu, 02/12 11:07, Wen Congyang wrote:
>>>>>>>> +== Workflow ==
>>>>>>>> +The following is the image of block replication workflow:
>>>>>>>> +
>>>>>>>> +        +----------------------+            +------------------------+
>>>>>>>> +        |Primary Write Requests|            |Secondary Write Requests|
>>>>>>>> +        +----------------------+            +------------------------+
>>>>>>>> +                  |                                       |
>>>>>>>> +                  |                                      (4)
>>>>>>>> +                  |                                       V
>>>>>>>> +                  |                              /-------------\
>>>>>>>> +                  |      Copy and Forward        |             |
>>>>>>>> +                  |---------(1)----------+       | Disk Buffer |
>>>>>>>> +                  |                      |       |             |
>>>>>>>> +                  |                     (3)      \-------------/
>>>>>>>> +                  |                 speculative      ^
>>>>>>>> +                  |                write through    (2)
>>>>>>>> +                  |                      |           |
>>>>>>>> +                  V                      V           |
>>>>>>>> +           +--------------+           +----------------+
>>>>>>>> +           | Primary Disk |           | Secondary Disk |
>>>>>>>> +           +--------------+           +----------------+
>>>>>>>> +
>>>>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>>>>>> +       QEMU.
>>>>>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>>>>>> +       original sector content will be read from Secondary disk and
>>>>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>>>>>> +       sector content in the Disk buffer.
>>>>>>>
>>>>>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
>>>>>>> reading them as "s/will be/are/g"
>>>>>>>
>>>>>>> Why do you need this buffer?
>>>>>>
>>>>>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
>>>>>> vm write to the buffer.
>>>>>>
>>>>>>>
>>>>>>> If both primary and secondary write to the same sector, what is saved in the
>>>>>>> buffer?
>>>>>>
>>>>>> The primary content will be written to the secondary disk, and the secondary content
>>>>>> is saved in the buffer.
>>>>>
>>>>> I wonder if alternatively this is possible with an imaginary "writable backing
>>>>> image" feature, as described below.
>>>>>
>>>>> When we have a normal backing chain,
>>>>>
>>>>>                {virtio-blk dev 'foo'}
>>>>>                          |
>>>>>                          |
>>>>>                          |
>>>>>     [base] <- [mid] <- (foo)
>>>>>
>>>>> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
>>>>> to an existing image on top,
>>>>>
>>>>>                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
>>>>>                          |                              |
>>>>>                          |                              |
>>>>>                          |                              |
>>>>>     [base] <- [mid] <- (foo)  <---------------------- (bar)
>>>>>
>>>>> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
>>>>> We can utilize an automatic hidden drive-backup target:
>>>>>
>>>>>                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
>>>>>                          |                                                          |
>>>>>                          |                                                          |
>>>>>                          v                                                          v
>>>>>
>>>>>     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
>>>>>
>>>>>                          v                              ^
>>>>>                          v                              ^
>>>>>                          v                              ^
>>>>>                          v                              ^
>>>>>                          >>>> drive-backup sync=none >>>>
>>>>>
>>>>> So when guest writes to 'foo', the old data is moved to (hidden target), which
>>>>> remains unchanged from (bar)'s PoV.
>>>>>
>>>>> The drive in the middle is called hidden because QEMU creates it automatically,
>>>>> the naming is arbitrary.
>>>>>
>>>>> It is interesting because it is a more generalized case of image fleecing,
>>>>> where the (hidden target) is exposed via NBD server for data scanning (read
>>>>> only) purpose.
>>>>>
>>>>> More interestingly, with above facility, it is also possible to create a guest
>>>>> visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
>>>>> cheaply. Or call it shadow copy if you will.
>>>>>
>>>>> Back to the COLO case, the configuration will be very similar:
>>>>>
>>>>>
>>>>>                       {primary wr}                                                {secondary vm}
>>>>>                             |                                                           |
>>>>>                             |                                                           |
>>>>>                             |                                                           |
>>>>>                             v                                                           v
>>>>>
>>>>>    [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)
>>>>>
>>>>>                             v                              ^
>>>>>                             v                              ^
>>>>>                             v                              ^
>>>>>                             v                              ^
>>>>>                             >>>> drive-backup sync=none >>>>
>>>>
>>>> What is active disk? There are two disk images?
>>>
>>> It starts as an empty image with (hidden buf disk) as backing file, which in
>>> turn has (nbd target) as backing file.
>>
>> It's too complicated..., and I don't understand it.
>> 1. What is active disk? Use raw or a new block driver?
> 
> It is an empty qcow2 image with the same lenght as your Secondary Disk.
> 
>> 2. Hidden buf disk use new block driver?
> 
> It is an empty qcow2 image with the same lenght as your Secondary Disk, too.
> 
>> 3. nbd target is hidden buf disk's backing image? If it is opened read-only, we will
>>    export a nbd with read-only BlockDriverState, but nbd server needs to write it.
> 
> NBD target is your Secondary Disk. It is opened read-write.
> 
> The patches to enable opening it as read-write, and starting drive-backup
> between it and hidden buf disk, are all work in progress (the core concept) of
> image fleecing.

What is image fleecing? Are you implementing it now?

Thanks
Wen Congyang

> 
> Fam
> 
>>>>>
>>>>> The workflow analogue is:
>>>>>
>>>>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>>>>>> +       QEMU.
>>>>>
>>>>> Primary write requests are forwarded to secondary QEMU as well.
>>>>>
>>>>>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>>>>>> +       original sector content will be read from Secondary disk and
>>>>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>>>>>> +       sector content in the Disk buffer.
>>>>>
>>>>> Before Primary write requests are written to (nbd target), aka the Secondary
>>>>> disk, the orignal sector content is read from it and copied to (hidden buf
>>>>> disk) by drive-backup. It obviously will not overwrite the data in (active
>>>>> disk).
>>>>>
>>>>>>>> +    3) Primary write requests will be written to Secondary disk.
>>>>>
>>>>> Primary write requests are written to (nbd target).
>>>>>
>>>>>>>> +    4) Secondary write requests will be buffered in the Disk buffer and it
>>>>>>>> +       will overwrite the existing sector content in the buffer.
>>>>>
>>>>> Secondary write request will be written in (active disk) as usual.
>>>>>
>>>>> Finally, when checkpoint arrives, if you want to sync with primary, just drop
>>>>> data in (hidden buf disk) and (active disk); when failover happends, if you
>>>>> want to promote secondary vm, you can commit (active disk) to (nbd target), and
>>>>> drop data in (hidden buf disk).
>>>>>
>>>>> Fam
>>>>> .
>>>>>
>>>>
>>>>
>>> .
>>>
>>
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Fam Zheng]

On Fri, 02/13 13:09, Wen Congyang wrote:
> What is image fleecing?
> 

It's the name of the feature which enables the built-in NBD server to exporting
a thin point-in-time snapshot created via drive-backup sync=none.

It's for host side data scanning tool to access a disk snapshot of running VM.
The workflow in theory is:

1. guest uses "disk0" as its virtio-blk device.

2. in qmp, use blockdev-add (drive-backup) to add an empty "target0" qcow2
image, that uses "disk0" as its backing file, and use nbd-server-add to export
this empty image with NBD. This way, all reads coming from NBD client will
produce data of "disk0".

3. in qmp, start blockdev-backup from "disk0" to "target0" with "sync=none".
After this point, all guest data written to "disk0" will COW the original data
to "target0", in other words, reading "target0" will effectively produce a
point-in-time snapshot of the time when blockdev-backup started.

4. after step 3, the disk data seen by the NBD client is the stable snapshot.
Because of the COW mechanism in blockdev-backup, "target0" is thin, and can be
dropped once the inspection process is done.

>  Are you implementing it now?

I worked on it. Most parts of the series is merged, the remaining part is
relatively small, namely to

1) enable adding "target0" in step 2 (currently in blockdev-add it's not
possible to reference an existing drive as backing file);

2) enable "blockdev-backup" from "disk0" to "target0", which is obviously not
possible because 1) is not done.

I do have the patches at my tree, just that they need to be refreshed. :)

https://github.com/famz/qemu/tree/image-fleecing

Fam

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[John Snow]

On 02/13/2015 02:01 AM, Fam Zheng wrote:
> On Fri, 02/13 13:09, Wen Congyang wrote:
>> What is image fleecing?
>>
>
> It's the name of the feature which enables the built-in NBD server to exporting
> a thin point-in-time snapshot created via drive-backup sync=none.
>
> It's for host side data scanning tool to access a disk snapshot of running VM.
> The workflow in theory is:
>
> 1. guest uses "disk0" as its virtio-blk device.
>
> 2. in qmp, use blockdev-add (drive-backup) to add an empty "target0" qcow2
> image, that uses "disk0" as its backing file, and use nbd-server-add to export
> this empty image with NBD. This way, all reads coming from NBD client will
> produce data of "disk0".
>
> 3. in qmp, start blockdev-backup from "disk0" to "target0" with "sync=none".
> After this point, all guest data written to "disk0" will COW the original data
> to "target0", in other words, reading "target0" will effectively produce a
> point-in-time snapshot of the time when blockdev-backup started.
>
> 4. after step 3, the disk data seen by the NBD client is the stable snapshot.
> Because of the COW mechanism in blockdev-backup, "target0" is thin, and can be
> dropped once the inspection process is done.
>
>>   Are you implementing it now?
>
> I worked on it. Most parts of the series is merged, the remaining part is
> relatively small, namely to
>
> 1) enable adding "target0" in step 2 (currently in blockdev-add it's not
> possible to reference an existing drive as backing file);
>
> 2) enable "blockdev-backup" from "disk0" to "target0", which is obviously not
> possible because 1) is not done.
>
> I do have the patches at my tree, just that they need to be refreshed. :)
>
> https://github.com/famz/qemu/tree/image-fleecing
>
> Fam
>

I had intended to pick up these patches after I got incremental backup 
working, as Fam had started both of these projects and I inherited them 
-- though I hadn't begun work in earnest on refining and testing this 
particular feature yet.

--js

Re: [Qemu-devel] [RFC PATCH 00/14] Block replication for continuous checkpoints

[Paolo Bonzini]

On 12/02/2015 04:07, Wen Congyang wrote:
> Wen Congyang (14):
>   docs: block replication's description
>   quorom: add a new read pattern
>   quorum: ignore 0-length child
>   Add new block driver interfaces to control disk replication
>   quorom: implement block driver interfaces for block replication
>   NBD client: connect to nbd server later
>   NBD client: implement block driver interfaces for block replication
>   block: add a new API to create a hidden BlockBackend
>   block: give backing image its own BlockBackend
>   allow the backing image access the origin BlockDriverState
>   allow writing to the backing file
>   Add disk buffer for block replication
>   COW: move cow interfaces to a seperate file
>   COLO: implement a new block driver

Hi Wen, sorry for the delay.

Kevin and Max need to review this series, as they are the most
comfortable with BlockDriverState vs. BlockBackend.

I suspect you cannot unconditionally give a separate BlockBackend to
each backing image, but they can hopefully suggest how to proceed.

Paolo

Re: [Qemu-devel] [RFC PATCH 02/14] quorom: add a new read pattern

[Max Reitz]

On 2015-02-11 at 22:07, Wen Congyang wrote:
> To block replication, we only need to read from the first child.
>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> Cc: Luiz Capitulino <lcapitulino@redhat.com>
> Cc: Michael Roth <mdroth@linux.vnet.ibm.com>
> ---
>   block/quorum.c       | 5 +++--
>   qapi/block-core.json | 4 +++-
>   2 files changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/block/quorum.c b/block/quorum.c
> index 437b122..5ed1ff8 100644
> --- a/block/quorum.c
> +++ b/block/quorum.c
> @@ -286,9 +286,10 @@ static void quorum_aio_cb(void *opaque, int ret)
>       BDRVQuorumState *s = acb->common.bs->opaque;
>       bool rewrite = false;
>   
> -    if (acb->is_read && s->read_pattern == QUORUM_READ_PATTERN_FIFO) {
> +    if (acb->is_read && s->read_pattern != QUORUM_READ_PATTERN_QUORUM) {

Maybe I'd prefer "&& (s->read_pattern == QUORUM_READ_PATTERN_FIFO || 
s->read_pattern == QUORUM_READ_PATTERN_FIRST)"; but it does fit with 
what quorum_aio_readv() does, so I'm fine with it.

>           /* We try to read next child in FIFO order if we fail to read */

However, I think this comment should be modified, because in fact we do 
not try to read the next child if s->read_pattern == 
QUORUM_READ_PATTERN_FIRST.

> -        if (ret < 0 && ++acb->child_iter < s->num_children) {
> +        if (s->read_pattern == QUORUM_READ_PATTERN_FIFO &&
> +            ret < 0 && ++acb->child_iter < s->num_children) {
>               read_fifo_child(acb);
>               return;
>           }
> diff --git a/qapi/block-core.json b/qapi/block-core.json
> index a3fdaf0..d6382e9 100644
> --- a/qapi/block-core.json
> +++ b/qapi/block-core.json
> @@ -1618,9 +1618,11 @@
>   #
>   # @fifo: read only from the first child that has not failed
>   #
> +# @first: read only from the first child

There should be version info here (like "(Since 2.3)").

Max

> +#
>   # Since: 2.2
>   ##
> -{ 'enum': 'QuorumReadPattern', 'data': [ 'quorum', 'fifo' ] }
> +{ 'enum': 'QuorumReadPattern', 'data': [ 'quorum', 'fifo', 'first' ] }
>   
>   ##
>   # @BlockdevOptionsQuorum

Re: [Qemu-devel] [RFC PATCH 03/14] quorum: ignore 0-length child

[Max Reitz]

On 2015-02-11 at 22:07, Wen Congyang wrote:
> We connect to NBD server when starting block replication, so
> the length is 0 before starting block replication.
>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
>   block/quorum.c | 5 +++++
>   1 file changed, 5 insertions(+)
>
> diff --git a/block/quorum.c b/block/quorum.c
> index 5ed1ff8..e6aff5f 100644
> --- a/block/quorum.c
> +++ b/block/quorum.c
> @@ -734,6 +734,11 @@ static int64_t quorum_getlength(BlockDriverState *bs)
>           if (value < 0) {
>               return value;
>           }
> +
> +        if (!value) {
> +            continue;
> +        }
> +
>           if (value != result) {
>               return -EIO;
>           }

Hm, what do you think about some specific error value returned by your 
delayed NBD implementation? Like -ENOTCONN or something like that? Then 
we'd be able to discern a real 0-length block device from a 
not-yet-connected NBD server.

Also, while you did write that one shouldn't be using the NBD client as 
the first quorum child, I think we should try to support that case 
anyway. For this patch, that means accepting that 
bdrv_getlength(s->bs[0]) may be off.

Max

Re: [Qemu-devel] [RFC PATCH 04/14] Add new block driver interfaces to control disk replication

[Max Reitz]

On 2015-02-11 at 22:07, Wen Congyang wrote:
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
>   block.c                   | 36 ++++++++++++++++++++++++++++++++++++
>   include/block/block.h     | 10 ++++++++++
>   include/block/block_int.h | 12 ++++++++++++
>   3 files changed, 58 insertions(+)
>
> diff --git a/block.c b/block.c
> index 210fd5f..2335af1 100644
> --- a/block.c
> +++ b/block.c
> @@ -6156,3 +6156,39 @@ BlockAcctStats *bdrv_get_stats(BlockDriverState *bs)
>   {
>       return &bs->stats;
>   }
> +
> +int bdrv_start_replication(BlockDriverState *bs, int mode)
> +{
> +    BlockDriver *drv = bs->drv;
> +    if (drv && drv->bdrv_start_replication) {
> +        return drv->bdrv_start_replication(bs, mode);
> +    } else if (bs->file) {
> +        return bdrv_start_replication(bs->file, mode);
> +    }
> +
> +    return -1;

I'd prefer returning -errno here (like -ENOTSUP). Alternatively, you may 
want to use Error objects (which would probably actually be the 
preferable way).

> +}
> +
> +int bdrv_do_checkpoint(BlockDriverState *bs)
> +{
> +    BlockDriver *drv = bs->drv;
> +    if (drv && drv->bdrv_do_checkpoint) {
> +        return drv->bdrv_do_checkpoint(bs);
> +    } else if (bs->file) {
> +        return bdrv_do_checkpoint(bs->file);
> +    }
> +
> +    return -1;

Same here.

> +}
> +
> +int bdrv_stop_replication(BlockDriverState *bs)
> +{
> +    BlockDriver *drv = bs->drv;
> +    if (drv && drv->bdrv_stop_replication) {
> +        return drv->bdrv_stop_replication(bs);
> +    } else if (bs->file) {
> +        return bdrv_stop_replication(bs->file);
> +    }
> +
> +    return -1;

And here.

> +}
> diff --git a/include/block/block.h b/include/block/block.h
> index 321295e..632b9fc 100644
> --- a/include/block/block.h
> +++ b/include/block/block.h
> @@ -557,4 +557,14 @@ void bdrv_flush_io_queue(BlockDriverState *bs);
>   
>   BlockAcctStats *bdrv_get_stats(BlockDriverState *bs);
>   
> +/* Checkpoint control, called in migration/checkpoint thread */
> +enum {
> +    COLO_UNPROTECTED_MODE = 0,
> +    COLO_PRIMARY_MODE,
> +    COLO_SECONDARY_MODE,
> +};

I have a feeling that you may want to define these values through QAPI...

There's nothing wrong with this patch, but I don't yet really know what 
you want to do with these functions (the doc didn't really help me with 
them), so I'll have to look into the rest of the series before I can 
really say something useful about it.

Max

> +int bdrv_start_replication(BlockDriverState *bs, int mode);
> +int bdrv_do_checkpoint(BlockDriverState *bs);
> +int bdrv_stop_replication(BlockDriverState *bs);
> +
>   #endif
> diff --git a/include/block/block_int.h b/include/block/block_int.h
> index 7ad1950..603f704 100644
> --- a/include/block/block_int.h
> +++ b/include/block/block_int.h
> @@ -273,6 +273,18 @@ struct BlockDriver {
>       void (*bdrv_io_unplug)(BlockDriverState *bs);
>       void (*bdrv_flush_io_queue)(BlockDriverState *bs);
>   
> +
> +    /* Checkpoint control, called in migration/checkpoint thread */
> +    int (*bdrv_start_replication)(BlockDriverState *bs, int mode);
> +    /*
> +     * Drop Disk buffer when doing checkpoint.
> +     */
> +    int (*bdrv_do_checkpoint)(BlockDriverState *bs);
> +    /* After failover, we should flush Disk buffer into secondary disk
> +     * and stop block replication.
> +     */
> +    int (*bdrv_stop_replication)(BlockDriverState *bs);
> +
>       QLIST_ENTRY(BlockDriver) list;
>   };
>   

Re: [Qemu-devel] [RFC PATCH 05/14] quorom: implement block driver interfaces for block replication

[Max Reitz]

On 2015-02-11 at 22:07, Wen Congyang wrote:
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
>   block/quorum.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 69 insertions(+)
>
> diff --git a/block/quorum.c b/block/quorum.c
> index e6aff5f..c8479b4 100644
> --- a/block/quorum.c
> +++ b/block/quorum.c
> @@ -1070,6 +1070,71 @@ static void quorum_refresh_filename(BlockDriverState *bs)
>       bs->full_open_options = opts;
>   }
>   
> +static int quorum_stop_replication(BlockDriverState *bs);
> +static int quorum_start_replication(BlockDriverState *bs, int mode)
> +{
> +    BDRVQuorumState *s = bs->opaque;
> +    int ret = -1, i;

Again, I'd prefer it if you used -errno (or the Error API).

> +
> +    /*
> +     * TODO: support COLO_SECONDARY_MODE if we allow secondary
> +     * QEMU becoming primary QEMU.
> +     */
> +    if (mode != COLO_PRIMARY_MODE) {
> +        return -1;
> +    }
> +
> +    if (s->read_pattern != QUORUM_READ_PATTERN_FIRST) {
> +        return -1;
> +    }
> +
> +    /* NBD client should not be the first child */
> +    if (bdrv_start_replication(s->bs[0], mode) == 0) {

If you allow the NBD client to be the first child you can probably drop 
this block (and start from "i = 0" in the for loop).

> +        bdrv_stop_replication(s->bs[0]);
> +        return -1;
> +    }
> +
> +    for (i = 1; i < s->num_children; i++) {
> +        if (bdrv_start_replication(s->bs[i], mode) == 0) {
> +            ret++;
> +        }
> +    }
> +
> +    if (ret > 0) {
> +        quorum_stop_replication(bs);
> +    }

I think it would be easier to read if you had an additional "count" 
variable which is set to 0 before the for loop and then incremented 
(instead of ret). This would then be "if (count > 1)".

> +
> +    return ret ? -1 : 0;

And this would be "return count == 0 ? 0 : -ENOTSUP" or something.

But apart from that, what's so bad about having multiple children which 
support bdrv_start_replication()? I mean, other than "It's not what we 
intended".

Max

> +}
> +
> +static int quorum_do_checkpoint(BlockDriverState *bs)
> +{
> +    BDRVQuorumState *s = bs->opaque;
> +    int i;
> +
> +    for (i = 1; i < s->num_children; i++) {
> +        if (bdrv_do_checkpoint(s->bs[i]) == 0) {
> +            return 0;
> +        }
> +    }
> +
> +    return -1;
> +}
> +
> +static int quorum_stop_replication(BlockDriverState *bs)
> +{
> +    BDRVQuorumState *s = bs->opaque;
> +    int ret = -1, i;
> +
> +    for (i = 0; i < s->num_children; i++) {
> +        if (bdrv_stop_replication(s->bs[i]) == 0) {
> +            ret++;
> +        }
> +    }
> +
> +    return ret ? -1 : 0;
> +}
> +
>   static BlockDriver bdrv_quorum = {
>       .format_name                        = "quorum",
>       .protocol_name                      = "quorum",
> @@ -1093,6 +1158,10 @@ static BlockDriver bdrv_quorum = {
>   
>       .is_filter                          = true,
>       .bdrv_recurse_is_first_non_filter   = quorum_recurse_is_first_non_filter,
> +
> +    .bdrv_start_replication             = quorum_start_replication,
> +    .bdrv_do_checkpoint                 = quorum_do_checkpoint,
> +    .bdrv_stop_replication              = quorum_stop_replication,
>   };
>   
>   static void bdrv_quorum_init(void)

Re: [Qemu-devel] [RFC PATCH 06/14] NBD client: connect to nbd server later

[Max Reitz]

On 2015-02-11 at 22:07, Wen Congyang wrote:
> The secondary qemu starts later than the primary qemu, so we
> cannot connect to nbd server in bdrv_open().
>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
>   block/nbd.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++++--------
>   1 file changed, 87 insertions(+), 13 deletions(-)
>
> diff --git a/block/nbd.c b/block/nbd.c
> index b05d1d0..19b9200 100644
> --- a/block/nbd.c
> +++ b/block/nbd.c
> @@ -44,6 +44,8 @@
>   typedef struct BDRVNBDState {
>       NbdClientSession client;
>       QemuOpts *socket_opts;
> +    char *export;
> +    bool connected;
>   } BDRVNBDState;
>   
>   static int nbd_parse_uri(const char *filename, QDict *options)
> @@ -247,20 +249,10 @@ static int nbd_establish_connection(BlockDriverState *bs, Error **errp)
>       return sock;
>   }
>   
> -static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
> -                    Error **errp)
> +static int nbd_connect_server(BlockDriverState *bs, Error **errp)
>   {
>       BDRVNBDState *s = bs->opaque;
> -    char *export = NULL;
>       int result, sock;
> -    Error *local_err = NULL;
> -
> -    /* Pop the config into our state object. Exit if invalid. */
> -    nbd_config(s, options, &export, &local_err);
> -    if (local_err) {
> -        error_propagate(errp, local_err);
> -        return -EINVAL;
> -    }
>   
>       /* establish TCP connection, return error if it fails
>        * TODO: Configurable retry-until-timeout behaviour.
> @@ -271,16 +263,57 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
>       }
>   
>       /* NBD handshake */
> -    result = nbd_client_session_init(&s->client, bs, sock, export, errp);
> -    g_free(export);
> +    result = nbd_client_session_init(&s->client, bs, sock, s->export, errp);
> +    g_free(s->export);
> +    s->export = NULL;
> +    if (!result) {
> +        s->connected = true;
> +    }
> +
>       return result;
>   }
>   
> +static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
> +                    Error **errp)
> +{
> +    BDRVNBDState *s = bs->opaque;
> +    Error *local_err = NULL;
> +
> +    /* Pop the config into our state object. Exit if invalid. */
> +    nbd_config(s, options, &s->export, &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return -EINVAL;
> +    }
> +
> +    return nbd_connect_server(bs, errp);
> +}
> +
> +static int nbd_open_colo(BlockDriverState *bs, QDict *options, int flags,
> +                         Error **errp)
> +{
> +    BDRVNBDState *s = bs->opaque;
> +    Error *local_err = NULL;
> +
> +    /* Pop the config into our state object. Exit if invalid. */
> +    nbd_config(s, options, &s->export, &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return -EINVAL;
> +    }
> +
> +    return 0;
> +}
> +
>   static int nbd_co_readv(BlockDriverState *bs, int64_t sector_num,
>                           int nb_sectors, QEMUIOVector *qiov)
>   {
>       BDRVNBDState *s = bs->opaque;
>   
> +    if (!s->connected) {
> +        return -EIO;
> +    }
> +
>       return nbd_client_session_co_readv(&s->client, sector_num,
>                                          nb_sectors, qiov);
>   }
> @@ -290,6 +323,10 @@ static int nbd_co_writev(BlockDriverState *bs, int64_t sector_num,
>   {
>       BDRVNBDState *s = bs->opaque;
>   
> +    if (!s->connected) {
> +        return 0;
> +    }

Would it break anything to return -EIO here as well? (And in all the 
following functions)

> +
>       return nbd_client_session_co_writev(&s->client, sector_num,
>                                           nb_sectors, qiov);
>   }
> @@ -298,6 +335,10 @@ static int nbd_co_flush(BlockDriverState *bs)
>   {
>       BDRVNBDState *s = bs->opaque;
>   
> +    if (!s->connected) {
> +        return 0;
> +    }
> +
>       return nbd_client_session_co_flush(&s->client);
>   }
>   
> @@ -312,6 +353,10 @@ static int nbd_co_discard(BlockDriverState *bs, int64_t sector_num,
>   {
>       BDRVNBDState *s = bs->opaque;
>   
> +    if (!s->connected) {
> +        return 0;
> +    }
> +
>       return nbd_client_session_co_discard(&s->client, sector_num,
>                                            nb_sectors);
>   }
> @@ -322,6 +367,7 @@ static void nbd_close(BlockDriverState *bs)
>   
>       qemu_opts_del(s->socket_opts);
>       nbd_client_session_close(&s->client);
> +    s->connected = false;
>   }
>   

As I proposed before, can you make nbd_getlength() return -ENOTCONN or 
something unique in case s->connected is false? I think that'd be better 
than returning 0 (which is a valid value).

Max

>   static int64_t nbd_getlength(BlockDriverState *bs)
> @@ -335,6 +381,10 @@ static void nbd_detach_aio_context(BlockDriverState *bs)
>   {
>       BDRVNBDState *s = bs->opaque;
>   
> +    if (!s->connected) {
> +        return;
> +    }
> +
>       nbd_client_session_detach_aio_context(&s->client);
>   }
>   
> @@ -343,6 +393,10 @@ static void nbd_attach_aio_context(BlockDriverState *bs,
>   {
>       BDRVNBDState *s = bs->opaque;
>   
> +    if (!s->connected) {
> +        return;
> +    }
> +
>       nbd_client_session_attach_aio_context(&s->client, new_context);
>   }
>   
> @@ -445,11 +499,31 @@ static BlockDriver bdrv_nbd_unix = {
>       .bdrv_refresh_filename      = nbd_refresh_filename,
>   };
>   
> +static BlockDriver bdrv_nbd_colo = {
> +    .format_name                = "nbd+colo",
> +    .protocol_name              = "nbd+colo",
> +    .instance_size              = sizeof(BDRVNBDState),
> +    .bdrv_parse_filename        = nbd_parse_filename,
> +    .bdrv_file_open             = nbd_open_colo,
> +    .bdrv_co_readv              = nbd_co_readv,
> +    .bdrv_co_writev             = nbd_co_writev,
> +    .bdrv_close                 = nbd_close,
> +    .bdrv_co_flush_to_os        = nbd_co_flush,
> +    .bdrv_co_discard            = nbd_co_discard,
> +    .bdrv_getlength             = nbd_getlength,
> +    .bdrv_detach_aio_context    = nbd_detach_aio_context,
> +    .bdrv_attach_aio_context    = nbd_attach_aio_context,
> +    .bdrv_refresh_filename      = nbd_refresh_filename,
> +
> +    .has_variable_length        = true,
> +};
> +
>   static void bdrv_nbd_init(void)
>   {
>       bdrv_register(&bdrv_nbd);
>       bdrv_register(&bdrv_nbd_tcp);
>       bdrv_register(&bdrv_nbd_unix);
> +    bdrv_register(&bdrv_nbd_colo);
>   }
>   
>   block_init(bdrv_nbd_init);

Re: [Qemu-devel] [RFC PATCH 07/14] NBD client: implement block driver interfaces for block replication

[Max Reitz]

On 2015-02-11 at 22:07, Wen Congyang wrote:
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
>   block/nbd.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 55 insertions(+)

So by now to me it looks like you're using bdrv_start_replication() on 
the primary VM to start transferring data through the NBD connection to 
the secondary VM.

I guess this is what you were discussing with Fam and John, whether 
there'd be a better way to do it by using functionality that is already 
(or is about to become) part of qemu, right?

> diff --git a/block/nbd.c b/block/nbd.c
> index 19b9200..1ff6ecf 100644
> --- a/block/nbd.c
> +++ b/block/nbd.c
> @@ -445,6 +445,58 @@ static void nbd_refresh_filename(BlockDriverState *bs)
>       bs->full_open_options = opts;
>   }
>   
> +static int nbd_start_replication(BlockDriverState *bs, int mode)
> +{
> +    BDRVNBDState *s = bs->opaque;
> +    Error *local_err = NULL;
> +    int ret;
> +
> +    /*
> +     * TODO: support COLO_SECONDARY_MODE if we allow secondary
> +     * QEMU becoming primary QEMU.
> +     */
> +    if (mode != COLO_PRIMARY_MODE) {
> +        return -1;

Once again, I'd like -ENOTSUP more (or -EINVAL or whatever you prefer).

> +    }
> +
> +    if (s->connected) {
> +        return -1;
> +    }
> +
> +    /* TODO: NBD client should be one child of quorum, how to verify it? */

Again, why would you care about that? Other than "It's how it's intended 
to be used".

> +    ret = nbd_connect_server(bs, &local_err);
> +    if (local_err) {
> +        error_free(local_err);
> +    }

If you'd use the Error API you'd be able to propagate the error.

Max

> +
> +    return ret;
> +}
> +
> +static int nbd_do_checkpoint(BlockDriverState *bs)
> +{
> +    BDRVNBDState *s = bs->opaque;
> +
> +    if (!s->connected) {
> +        return -1;
> +    }
> +
> +    return 0;
> +}
> +
> +static int nbd_stop_replication(BlockDriverState *bs)
> +{
> +    BDRVNBDState *s = bs->opaque;
> +
> +    if (!s->connected) {
> +        return -1;
> +    }
> +
> +    nbd_client_session_close(&s->client);
> +    s->connected = false;
> +
> +    return 0;
> +}
> +
>   static BlockDriver bdrv_nbd = {
>       .format_name                = "nbd",
>       .protocol_name              = "nbd",
> @@ -514,6 +566,9 @@ static BlockDriver bdrv_nbd_colo = {
>       .bdrv_detach_aio_context    = nbd_detach_aio_context,
>       .bdrv_attach_aio_context    = nbd_attach_aio_context,
>       .bdrv_refresh_filename      = nbd_refresh_filename,
> +    .bdrv_start_replication     = nbd_start_replication,
> +    .bdrv_do_checkpoint         = nbd_do_checkpoint,
> +    .bdrv_stop_replication      = nbd_stop_replication,
>   
>       .has_variable_length        = true,
>   };

Re: [Qemu-devel] [RFC PATCH 08/14] block: add a new API to create a hidden BlockBackend

[Max Reitz]

On 2015-02-11 at 22:07, Wen Congyang wrote:
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
>   block/block-backend.c          | 29 ++++++++++++++++++++++++++++-
>   include/sysemu/block-backend.h |  2 ++
>   2 files changed, 30 insertions(+), 1 deletion(-)

Hm, I'm currently working on a series that (among other things) separate 
the list of monitor-owned BlockBackends and the list of all BBs. That 
should be helpful here; but Paolo said it might not be necessary to 
create a BB at all, so... Maybe he's right. I'll have to look into the 
remaining patches first.

Max

Re: [Qemu-devel] [RFC PATCH 09/14] block: give backing image its own BlockBackend

[Max Reitz]

On 2015-02-11 at 22:07, Wen Congyang wrote:
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
>   block.c | 11 +++++++----
>   1 file changed, 7 insertions(+), 4 deletions(-)

Our current stance on BlockBackends is (as far as I know, anyway) that a 
BlockBackend always comes with a user. In case you're creating a 
BlockBackend through -drive or blockdev-add, the user is the monitor 
(which can use it to attach it to a device, for instance). In this case, 
there is no user and nobody holds a reference to the BB (other than the 
BDS, but that doesn't count).

Therefore, this patch doesn't look quite right.

Max

Re: [Qemu-devel] [RFC PATCH 02/14] quorom: add a new read pattern

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

On 02/11/2015 08:07 PM, Wen Congyang wrote:
> To block replication, we only need to read from the first child.

s/quorom/quorum/ in the subject line

s/To block/For block/

> 
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> Cc: Luiz Capitulino <lcapitulino@redhat.com>
> Cc: Michael Roth <mdroth@linux.vnet.ibm.com>
> ---
>  block/quorum.c       | 5 +++--
>  qapi/block-core.json | 4 +++-
>  2 files changed, 6 insertions(+), 3 deletions(-)

> +++ b/qapi/block-core.json
> @@ -1618,9 +1618,11 @@
>  #
>  # @fifo: read only from the first child that has not failed
>  #
> +# @first: read only from the first child
> +

Missing a 'since 2.3' designation.  Otherwise looks okay.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [RFC PATCH 04/14] Add new block driver interfaces to control disk replication

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1674 bytes --]

On 02/23/2015 01:57 PM, Max Reitz wrote:
> On 2015-02-11 at 22:07, Wen Congyang wrote:
>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
>> ---
>>   block.c                   | 36 ++++++++++++++++++++++++++++++++++++
>>   include/block/block.h     | 10 ++++++++++
>>   include/block/block_int.h | 12 ++++++++++++
>>   3 files changed, 58 insertions(+)
>>

>> +++ b/include/block/block.h
>> @@ -557,4 +557,14 @@ void bdrv_flush_io_queue(BlockDriverState *bs);
>>     BlockAcctStats *bdrv_get_stats(BlockDriverState *bs);
>>   +/* Checkpoint control, called in migration/checkpoint thread */
>> +enum {
>> +    COLO_UNPROTECTED_MODE = 0,
>> +    COLO_PRIMARY_MODE,
>> +    COLO_SECONDARY_MODE,
>> +};
> 
> I have a feeling that you may want to define these values through QAPI...

especially if you intend for a QMP command to output which mode a colo
disk is in.


>> +++ b/include/block/block_int.h
>> @@ -273,6 +273,18 @@ struct BlockDriver {
>>       void (*bdrv_io_unplug)(BlockDriverState *bs);
>>       void (*bdrv_flush_io_queue)(BlockDriverState *bs);
>>   +
>> +    /* Checkpoint control, called in migration/checkpoint thread */
>> +    int (*bdrv_start_replication)(BlockDriverState *bs, int mode);
>> +    /*
>> +     * Drop Disk buffer when doing checkpoint.
>> +     */
>> +    int (*bdrv_do_checkpoint)(BlockDriverState *bs);

Inconsistent comment style between one-line and multi-line.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [RFC PATCH 10/14] allow the backing image access the origin BlockDriverState

[Max Reitz]

On 2015-02-11 at 22:07, Wen Congyang wrote:
> Block replication needs this feature.
>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
>   block.c                   | 2 ++
>   include/block/block_int.h | 2 ++
>   2 files changed, 4 insertions(+)
>
> diff --git a/block.c b/block.c
> index a7a8932..067c44b 100644
> --- a/block.c
> +++ b/block.c
> @@ -1181,6 +1181,7 @@ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd)
>       if (bs->backing_hd) {
>           assert(bs->backing_blocker);
>           bdrv_op_unblock_all(bs->backing_hd, bs->backing_blocker);
> +        bs->backing_hd->origin_file = NULL;

Seems more like "backed_file" to me. Can you explain to me where "origin 
file" comes from?

Since apparently one BDS can be used as a backing file by only at most 
one other BDS, the patch seems fine to me (other than the naming issue).

Max

>       } else if (backing_hd) {
>           error_setg(&bs->backing_blocker,
>                      "device is used as backing hd of '%s'",
> @@ -1193,6 +1194,7 @@ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd)
>           bs->backing_blocker = NULL;
>           goto out;
>       }
> +    backing_hd->origin_file = bs;
>       bs->open_flags &= ~BDRV_O_NO_BACKING;
>       pstrcpy(bs->backing_file, sizeof(bs->backing_file), backing_hd->filename);
>       pstrcpy(bs->backing_format, sizeof(bs->backing_format),
> diff --git a/include/block/block_int.h b/include/block/block_int.h
> index 603f704..9be13a8 100644
> --- a/include/block/block_int.h
> +++ b/include/block/block_int.h
> @@ -360,6 +360,8 @@ struct BlockDriverState {
>       char exact_filename[PATH_MAX];
>   
>       BlockDriverState *backing_hd;
> +    /* used by backing image */
> +    BlockDriverState *origin_file;
>       BlockDriverState *file;
>   
>       NotifierList close_notifiers;

Re: [Qemu-devel] [RFC PATCH 11/14] allow writing to the backing file

[Max Reitz]

On 2015-02-11 at 22:07, Wen Congyang wrote:
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
>   block.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)

I don't think this is a good idea. With this patch, every time you open 
a COW file (with a backing file) R/W, the backing file will be writable. 
I'd rather like a way to explicitly overwrite the R/W mode of the 
backing file; but by default, in my opinion, it should stay read-only.

Max

> diff --git a/block.c b/block.c
> index 067c44b..96cf973 100644
> --- a/block.c
> +++ b/block.c
> @@ -856,8 +856,8 @@ static int bdrv_inherited_flags(int flags)
>    */
>   static int bdrv_backing_flags(int flags)
>   {
> -    /* backing files always opened read-only */
> -    flags &= ~(BDRV_O_RDWR | BDRV_O_COPY_ON_READ);
> +    /* backing files are opened read-write for block replication */
> +    flags &= ~BDRV_O_COPY_ON_READ;
>   
>       /* snapshot=on is handled on the top layer */
>       flags &= ~(BDRV_O_SNAPSHOT | BDRV_O_TEMPORARY);

Re: [Qemu-devel] [RFC PATCH 12/14] Add disk buffer for block replication

[Max Reitz]

On 2015-02-11 at 22:07, Wen Congyang wrote:
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
>   block/Makefile.objs    |   1 +
>   block/blkcolo-buffer.c | 324 +++++++++++++++++++++++++++++++++++++++++++++++++
>   block/blkcolo.h        |  35 ++++++
>   3 files changed, 360 insertions(+)
>   create mode 100644 block/blkcolo-buffer.c
>   create mode 100644 block/blkcolo.h

In general: Can you please add some prefix to the non-static functions, 
like colo_*?

As for the design questions regarding this block driver, I should 
probably leave that to Fam and John (because they seemed to have an idea 
how to approach the issue at hand using a different implementation, 
based on functionality that's already (or close to become) part of qemu).

Therefore, I'm hesitating to review this patch (and the following ones) 
until you've reached a conclusion on how to proceed.

Max

Re: [Qemu-devel] [RFC PATCH 14/14] COLO: implement a new block driver

[Max Reitz]

On 2015-02-11 at 22:07, Wen Congyang wrote:
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
>   block/Makefile.objs |   2 +-
>   block/blkcolo.c     | 409 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>   2 files changed, 410 insertions(+), 1 deletion(-)
>   create mode 100644 block/blkcolo.c

Seeing what you want to use the BlockBackend for (which is, set up an 
NBD server): I think it's best to create the BlockBackend the moment the 
NBD server is created, and destroy it the moment the NBD server is 
stopped (that is, create it in colo_svm_init() and destroy it in 
colo_svm_fini()).

Max

Re: [Qemu-devel] [RFC PATCH 03/14] quorum: ignore 0-length child

[Wen Congyang]

On 02/24/2015 04:43 AM, Max Reitz wrote:
> On 2015-02-11 at 22:07, Wen Congyang wrote:
>> We connect to NBD server when starting block replication, so
>> the length is 0 before starting block replication.
>>
>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
>> ---
>>   block/quorum.c | 5 +++++
>>   1 file changed, 5 insertions(+)
>>
>> diff --git a/block/quorum.c b/block/quorum.c
>> index 5ed1ff8..e6aff5f 100644
>> --- a/block/quorum.c
>> +++ b/block/quorum.c
>> @@ -734,6 +734,11 @@ static int64_t quorum_getlength(BlockDriverState *bs)
>>           if (value < 0) {
>>               return value;
>>           }
>> +
>> +        if (!value) {
>> +            continue;
>> +        }
>> +
>>           if (value != result) {
>>               return -EIO;
>>           }
> 
> Hm, what do you think about some specific error value returned by your delayed NBD implementation? Like -ENOTCONN or something like that? Then we'd be able to discern a real 0-length block device from a not-yet-connected NBD server.
> 
> Also, while you did write that one shouldn't be using the NBD client as the first quorum child, I think we should try to support that case anyway. For this patch, that means accepting that bdrv_getlength(s->bs[0]) may be off.

Good idea. I will try it.

Thanks
Wen Congyang

> 
> Max
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 02/12/2015 04:44 PM, Fam Zheng wrote:
> On Thu, 02/12 15:40, Wen Congyang wrote:
>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
>>> Hi Congyang,
>>>
>>> On Thu, 02/12 11:07, Wen Congyang wrote:
>>>> +== Workflow ==
>>>> +The following is the image of block replication workflow:
>>>> +
>>>> +        +----------------------+            +------------------------+
>>>> +        |Primary Write Requests|            |Secondary Write Requests|
>>>> +        +----------------------+            +------------------------+
>>>> +                  |                                       |
>>>> +                  |                                      (4)
>>>> +                  |                                       V
>>>> +                  |                              /-------------\
>>>> +                  |      Copy and Forward        |             |
>>>> +                  |---------(1)----------+       | Disk Buffer |
>>>> +                  |                      |       |             |
>>>> +                  |                     (3)      \-------------/
>>>> +                  |                 speculative      ^
>>>> +                  |                write through    (2)
>>>> +                  |                      |           |
>>>> +                  V                      V           |
>>>> +           +--------------+           +----------------+
>>>> +           | Primary Disk |           | Secondary Disk |
>>>> +           +--------------+           +----------------+
>>>> +
>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>> +       QEMU.
>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>> +       original sector content will be read from Secondary disk and
>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>> +       sector content in the Disk buffer.
>>>
>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
>>> reading them as "s/will be/are/g"
>>>
>>> Why do you need this buffer?
>>
>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
>> vm write to the buffer.
>>
>>>
>>> If both primary and secondary write to the same sector, what is saved in the
>>> buffer?
>>
>> The primary content will be written to the secondary disk, and the secondary content
>> is saved in the buffer.
> 
> I wonder if alternatively this is possible with an imaginary "writable backing
> image" feature, as described below.
> 
> When we have a normal backing chain,
> 
>                {virtio-blk dev 'foo'}
>                          |
>                          |
>                          |
>     [base] <- [mid] <- (foo)
> 
> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
> to an existing image on top,
> 
>                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
>                          |                              |
>                          |                              |
>                          |                              |
>     [base] <- [mid] <- (foo)  <---------------------- (bar)
> 
> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
> We can utilize an automatic hidden drive-backup target:
> 
>                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
>                          |                                                          |
>                          |                                                          |
>                          v                                                          v
> 
>     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
> 
>                          v                              ^
>                          v                              ^
>                          v                              ^
>                          v                              ^
>                          >>>> drive-backup sync=none >>>>
> 
> So when guest writes to 'foo', the old data is moved to (hidden target), which
> remains unchanged from (bar)'s PoV.
> 
> The drive in the middle is called hidden because QEMU creates it automatically,
> the naming is arbitrary.

I don't understand this. In which function, the hidden target is created automatically?

Thanks
Wen Congyang

> 
> It is interesting because it is a more generalized case of image fleecing,
> where the (hidden target) is exposed via NBD server for data scanning (read
> only) purpose.
> 
> More interestingly, with above facility, it is also possible to create a guest
> visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
> cheaply. Or call it shadow copy if you will.
> 
> Back to the COLO case, the configuration will be very similar:
> 
> 
>                       {primary wr}                                                {secondary vm}
>                             |                                                           |
>                             |                                                           |
>                             |                                                           |
>                             v                                                           v
> 
>    [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)
> 
>                             v                              ^
>                             v                              ^
>                             v                              ^
>                             v                              ^
>                             >>>> drive-backup sync=none >>>>
> 
> The workflow analogue is:
> 
>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>> +       QEMU.
> 
> Primary write requests are forwarded to secondary QEMU as well.
> 
>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>> +       original sector content will be read from Secondary disk and
>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>> +       sector content in the Disk buffer.
> 
> Before Primary write requests are written to (nbd target), aka the Secondary
> disk, the orignal sector content is read from it and copied to (hidden buf
> disk) by drive-backup. It obviously will not overwrite the data in (active
> disk).
> 
>>>> +    3) Primary write requests will be written to Secondary disk.
> 
> Primary write requests are written to (nbd target).
> 
>>>> +    4) Secondary write requests will be buffered in the Disk buffer and it
>>>> +       will overwrite the existing sector content in the buffer.
> 
> Secondary write request will be written in (active disk) as usual.
> 
> Finally, when checkpoint arrives, if you want to sync with primary, just drop
> data in (hidden buf disk) and (active disk); when failover happends, if you
> want to promote secondary vm, you can commit (active disk) to (nbd target), and
> drop data in (hidden buf disk).
> 
> Fam
> .
> 

Re: [Qemu-devel] [RFC PATCH 06/14] NBD client: connect to nbd server later

[Wen Congyang]

On 02/24/2015 05:31 AM, Max Reitz wrote:
> On 2015-02-11 at 22:07, Wen Congyang wrote:
>> The secondary qemu starts later than the primary qemu, so we
>> cannot connect to nbd server in bdrv_open().
>>
>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
>> ---
>>   block/nbd.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++++--------
>>   1 file changed, 87 insertions(+), 13 deletions(-)
>>
>> diff --git a/block/nbd.c b/block/nbd.c
>> index b05d1d0..19b9200 100644
>> --- a/block/nbd.c
>> +++ b/block/nbd.c
>> @@ -44,6 +44,8 @@
>>   typedef struct BDRVNBDState {
>>       NbdClientSession client;
>>       QemuOpts *socket_opts;
>> +    char *export;
>> +    bool connected;
>>   } BDRVNBDState;
>>     static int nbd_parse_uri(const char *filename, QDict *options)
>> @@ -247,20 +249,10 @@ static int nbd_establish_connection(BlockDriverState *bs, Error **errp)
>>       return sock;
>>   }
>>   -static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
>> -                    Error **errp)
>> +static int nbd_connect_server(BlockDriverState *bs, Error **errp)
>>   {
>>       BDRVNBDState *s = bs->opaque;
>> -    char *export = NULL;
>>       int result, sock;
>> -    Error *local_err = NULL;
>> -
>> -    /* Pop the config into our state object. Exit if invalid. */
>> -    nbd_config(s, options, &export, &local_err);
>> -    if (local_err) {
>> -        error_propagate(errp, local_err);
>> -        return -EINVAL;
>> -    }
>>         /* establish TCP connection, return error if it fails
>>        * TODO: Configurable retry-until-timeout behaviour.
>> @@ -271,16 +263,57 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
>>       }
>>         /* NBD handshake */
>> -    result = nbd_client_session_init(&s->client, bs, sock, export, errp);
>> -    g_free(export);
>> +    result = nbd_client_session_init(&s->client, bs, sock, s->export, errp);
>> +    g_free(s->export);
>> +    s->export = NULL;
>> +    if (!result) {
>> +        s->connected = true;
>> +    }
>> +
>>       return result;
>>   }
>>   +static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
>> +                    Error **errp)
>> +{
>> +    BDRVNBDState *s = bs->opaque;
>> +    Error *local_err = NULL;
>> +
>> +    /* Pop the config into our state object. Exit if invalid. */
>> +    nbd_config(s, options, &s->export, &local_err);
>> +    if (local_err) {
>> +        error_propagate(errp, local_err);
>> +        return -EINVAL;
>> +    }
>> +
>> +    return nbd_connect_server(bs, errp);
>> +}
>> +
>> +static int nbd_open_colo(BlockDriverState *bs, QDict *options, int flags,
>> +                         Error **errp)
>> +{
>> +    BDRVNBDState *s = bs->opaque;
>> +    Error *local_err = NULL;
>> +
>> +    /* Pop the config into our state object. Exit if invalid. */
>> +    nbd_config(s, options, &s->export, &local_err);
>> +    if (local_err) {
>> +        error_propagate(errp, local_err);
>> +        return -EINVAL;
>> +    }
>> +
>> +    return 0;
>> +}
>> +
>>   static int nbd_co_readv(BlockDriverState *bs, int64_t sector_num,
>>                           int nb_sectors, QEMUIOVector *qiov)
>>   {
>>       BDRVNBDState *s = bs->opaque;
>>   +    if (!s->connected) {
>> +        return -EIO;
>> +    }
>> +
>>       return nbd_client_session_co_readv(&s->client, sector_num,
>>                                          nb_sectors, qiov);
>>   }
>> @@ -290,6 +323,10 @@ static int nbd_co_writev(BlockDriverState *bs, int64_t sector_num,
>>   {
>>       BDRVNBDState *s = bs->opaque;
>>   +    if (!s->connected) {
>> +        return 0;
>> +    }
> 
> Would it break anything to return -EIO here as well? (And in all the following functions)

1. nbd_co_writev()
   If one child returns error, quorum will report it. There may be many write requests before
   we connect to nbd server, so there are too many qapi events...
2. nbd_co_flush()
   If quorum only have two children, and nbd client is the last one, quorum_co_flush()
   will return -EIO.
3. nbd_co_discard()
   quorum doens't call bdrv_co_discard(), so it is OK to return -EIO here.

So only nbd_co_discard() can return -EIO.

Thanks
Wen Congyang

> 
>> +
>>       return nbd_client_session_co_writev(&s->client, sector_num,
>>                                           nb_sectors, qiov);
>>   }
>> @@ -298,6 +335,10 @@ static int nbd_co_flush(BlockDriverState *bs)
>>   {
>>       BDRVNBDState *s = bs->opaque;
>>   +    if (!s->connected) {
>> +        return 0;
>> +    }
>> +
>>       return nbd_client_session_co_flush(&s->client);
>>   }
>>   @@ -312,6 +353,10 @@ static int nbd_co_discard(BlockDriverState *bs, int64_t sector_num,
>>   {
>>       BDRVNBDState *s = bs->opaque;
>>   +    if (!s->connected) {
>> +        return 0;
>> +    }
>> +
>>       return nbd_client_session_co_discard(&s->client, sector_num,
>>                                            nb_sectors);
>>   }
>> @@ -322,6 +367,7 @@ static void nbd_close(BlockDriverState *bs)
>>         qemu_opts_del(s->socket_opts);
>>       nbd_client_session_close(&s->client);
>> +    s->connected = false;
>>   }
>>   
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Fam Zheng]

On Tue, 02/24 15:50, Wen Congyang wrote:
> On 02/12/2015 04:44 PM, Fam Zheng wrote:
> > On Thu, 02/12 15:40, Wen Congyang wrote:
> >> On 02/12/2015 03:21 PM, Fam Zheng wrote:
> >>> Hi Congyang,
> >>>
> >>> On Thu, 02/12 11:07, Wen Congyang wrote:
> >>>> +== Workflow ==
> >>>> +The following is the image of block replication workflow:
> >>>> +
> >>>> +        +----------------------+            +------------------------+
> >>>> +        |Primary Write Requests|            |Secondary Write Requests|
> >>>> +        +----------------------+            +------------------------+
> >>>> +                  |                                       |
> >>>> +                  |                                      (4)
> >>>> +                  |                                       V
> >>>> +                  |                              /-------------\
> >>>> +                  |      Copy and Forward        |             |
> >>>> +                  |---------(1)----------+       | Disk Buffer |
> >>>> +                  |                      |       |             |
> >>>> +                  |                     (3)      \-------------/
> >>>> +                  |                 speculative      ^
> >>>> +                  |                write through    (2)
> >>>> +                  |                      |           |
> >>>> +                  V                      V           |
> >>>> +           +--------------+           +----------------+
> >>>> +           | Primary Disk |           | Secondary Disk |
> >>>> +           +--------------+           +----------------+
> >>>> +
> >>>> +    1) Primary write requests will be copied and forwarded to Secondary
> >>>> +       QEMU.
> >>>> +    2) Before Primary write requests are written to Secondary disk, the
> >>>> +       original sector content will be read from Secondary disk and
> >>>> +       buffered in the Disk buffer, but it will not overwrite the existing
> >>>> +       sector content in the Disk buffer.
> >>>
> >>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
> >>> reading them as "s/will be/are/g"
> >>>
> >>> Why do you need this buffer?
> >>
> >> We only sync the disk till next checkpoint. Before next checkpoint, secondary
> >> vm write to the buffer.
> >>
> >>>
> >>> If both primary and secondary write to the same sector, what is saved in the
> >>> buffer?
> >>
> >> The primary content will be written to the secondary disk, and the secondary content
> >> is saved in the buffer.
> > 
> > I wonder if alternatively this is possible with an imaginary "writable backing
> > image" feature, as described below.
> > 
> > When we have a normal backing chain,
> > 
> >                {virtio-blk dev 'foo'}
> >                          |
> >                          |
> >                          |
> >     [base] <- [mid] <- (foo)
> > 
> > Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
> > to an existing image on top,
> > 
> >                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
> >                          |                              |
> >                          |                              |
> >                          |                              |
> >     [base] <- [mid] <- (foo)  <---------------------- (bar)
> > 
> > It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
> > We can utilize an automatic hidden drive-backup target:
> > 
> >                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
> >                          |                                                          |
> >                          |                                                          |
> >                          v                                                          v
> > 
> >     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
> > 
> >                          v                              ^
> >                          v                              ^
> >                          v                              ^
> >                          v                              ^
> >                          >>>> drive-backup sync=none >>>>
> > 
> > So when guest writes to 'foo', the old data is moved to (hidden target), which
> > remains unchanged from (bar)'s PoV.
> > 
> > The drive in the middle is called hidden because QEMU creates it automatically,
> > the naming is arbitrary.
> 
> I don't understand this. In which function, the hidden target is created automatically?
> 

It's to be determined. This part is only in my mind :)

Fam

> 
> > 
> > It is interesting because it is a more generalized case of image fleecing,
> > where the (hidden target) is exposed via NBD server for data scanning (read
> > only) purpose.
> > 
> > More interestingly, with above facility, it is also possible to create a guest
> > visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
> > cheaply. Or call it shadow copy if you will.
> > 
> > Back to the COLO case, the configuration will be very similar:
> > 
> > 
> >                       {primary wr}                                                {secondary vm}
> >                             |                                                           |
> >                             |                                                           |
> >                             |                                                           |
> >                             v                                                           v
> > 
> >    [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)
> > 
> >                             v                              ^
> >                             v                              ^
> >                             v                              ^
> >                             v                              ^
> >                             >>>> drive-backup sync=none >>>>
> > 
> > The workflow analogue is:
> > 
> >>>> +    1) Primary write requests will be copied and forwarded to Secondary
> >>>> +       QEMU.
> > 
> > Primary write requests are forwarded to secondary QEMU as well.
> > 
> >>>> +    2) Before Primary write requests are written to Secondary disk, the
> >>>> +       original sector content will be read from Secondary disk and
> >>>> +       buffered in the Disk buffer, but it will not overwrite the existing
> >>>> +       sector content in the Disk buffer.
> > 
> > Before Primary write requests are written to (nbd target), aka the Secondary
> > disk, the orignal sector content is read from it and copied to (hidden buf
> > disk) by drive-backup. It obviously will not overwrite the data in (active
> > disk).
> > 
> >>>> +    3) Primary write requests will be written to Secondary disk.
> > 
> > Primary write requests are written to (nbd target).
> > 
> >>>> +    4) Secondary write requests will be buffered in the Disk buffer and it
> >>>> +       will overwrite the existing sector content in the buffer.
> > 
> > Secondary write request will be written in (active disk) as usual.
> > 
> > Finally, when checkpoint arrives, if you want to sync with primary, just drop
> > data in (hidden buf disk) and (active disk); when failover happends, if you
> > want to promote secondary vm, you can commit (active disk) to (nbd target), and
> > drop data in (hidden buf disk).
> > 
> > Fam
> > .
> > 
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 02/12/2015 04:44 PM, Fam Zheng wrote:
> On Thu, 02/12 15:40, Wen Congyang wrote:
>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
>>> Hi Congyang,
>>>
>>> On Thu, 02/12 11:07, Wen Congyang wrote:
>>>> +== Workflow ==
>>>> +The following is the image of block replication workflow:
>>>> +
>>>> +        +----------------------+            +------------------------+
>>>> +        |Primary Write Requests|            |Secondary Write Requests|
>>>> +        +----------------------+            +------------------------+
>>>> +                  |                                       |
>>>> +                  |                                      (4)
>>>> +                  |                                       V
>>>> +                  |                              /-------------\
>>>> +                  |      Copy and Forward        |             |
>>>> +                  |---------(1)----------+       | Disk Buffer |
>>>> +                  |                      |       |             |
>>>> +                  |                     (3)      \-------------/
>>>> +                  |                 speculative      ^
>>>> +                  |                write through    (2)
>>>> +                  |                      |           |
>>>> +                  V                      V           |
>>>> +           +--------------+           +----------------+
>>>> +           | Primary Disk |           | Secondary Disk |
>>>> +           +--------------+           +----------------+
>>>> +
>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>> +       QEMU.
>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>> +       original sector content will be read from Secondary disk and
>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>> +       sector content in the Disk buffer.
>>>
>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
>>> reading them as "s/will be/are/g"
>>>
>>> Why do you need this buffer?
>>
>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
>> vm write to the buffer.
>>
>>>
>>> If both primary and secondary write to the same sector, what is saved in the
>>> buffer?
>>
>> The primary content will be written to the secondary disk, and the secondary content
>> is saved in the buffer.
> 
> I wonder if alternatively this is possible with an imaginary "writable backing
> image" feature, as described below.
> 
> When we have a normal backing chain,
> 
>                {virtio-blk dev 'foo'}
>                          |
>                          |
>                          |
>     [base] <- [mid] <- (foo)

foo's backing is mid, and mid's backing is base?

The foo is a base's snapshot?

Thanks
Wen Congyang

> 
> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
> to an existing image on top,
> 
>                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
>                          |                              |
>                          |                              |
>                          |                              |
>     [base] <- [mid] <- (foo)  <---------------------- (bar)
> 
> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
> We can utilize an automatic hidden drive-backup target:
> 
>                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
>                          |                                                          |
>                          |                                                          |
>                          v                                                          v
> 
>     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
> 
>                          v                              ^
>                          v                              ^
>                          v                              ^
>                          v                              ^
>                          >>>> drive-backup sync=none >>>>
> 
> So when guest writes to 'foo', the old data is moved to (hidden target), which
> remains unchanged from (bar)'s PoV.
> 
> The drive in the middle is called hidden because QEMU creates it automatically,
> the naming is arbitrary.
> 
> It is interesting because it is a more generalized case of image fleecing,
> where the (hidden target) is exposed via NBD server for data scanning (read
> only) purpose.
> 
> More interestingly, with above facility, it is also possible to create a guest
> visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
> cheaply. Or call it shadow copy if you will.
> 
> Back to the COLO case, the configuration will be very similar:
> 
> 
>                       {primary wr}                                                {secondary vm}
>                             |                                                           |
>                             |                                                           |
>                             |                                                           |
>                             v                                                           v
> 
>    [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)
> 
>                             v                              ^
>                             v                              ^
>                             v                              ^
>                             v                              ^
>                             >>>> drive-backup sync=none >>>>
> 
> The workflow analogue is:
> 
>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>> +       QEMU.
> 
> Primary write requests are forwarded to secondary QEMU as well.
> 
>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>> +       original sector content will be read from Secondary disk and
>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>> +       sector content in the Disk buffer.
> 
> Before Primary write requests are written to (nbd target), aka the Secondary
> disk, the orignal sector content is read from it and copied to (hidden buf
> disk) by drive-backup. It obviously will not overwrite the data in (active
> disk).
> 
>>>> +    3) Primary write requests will be written to Secondary disk.
> 
> Primary write requests are written to (nbd target).
> 
>>>> +    4) Secondary write requests will be buffered in the Disk buffer and it
>>>> +       will overwrite the existing sector content in the buffer.
> 
> Secondary write request will be written in (active disk) as usual.
> 
> Finally, when checkpoint arrives, if you want to sync with primary, just drop
> data in (hidden buf disk) and (active disk); when failover happends, if you
> want to promote secondary vm, you can commit (active disk) to (nbd target), and
> drop data in (hidden buf disk).
> 
> Fam
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Fam Zheng]

On Wed, 02/25 16:11, Wen Congyang wrote:
> > 
> >                {virtio-blk dev 'foo'}
> >                          |
> >                          |
> >                          |
> >     [base] <- [mid] <- (foo)
> 
> foo's backing is mid, and mid's backing is base?

Yes.

Fam

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 02/25/2015 10:46 AM, Fam Zheng wrote:
> On Tue, 02/24 15:50, Wen Congyang wrote:
>> On 02/12/2015 04:44 PM, Fam Zheng wrote:
>>> On Thu, 02/12 15:40, Wen Congyang wrote:
>>>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
>>>>> Hi Congyang,
>>>>>
>>>>> On Thu, 02/12 11:07, Wen Congyang wrote:
>>>>>> +== Workflow ==
>>>>>> +The following is the image of block replication workflow:
>>>>>> +
>>>>>> +        +----------------------+            +------------------------+
>>>>>> +        |Primary Write Requests|            |Secondary Write Requests|
>>>>>> +        +----------------------+            +------------------------+
>>>>>> +                  |                                       |
>>>>>> +                  |                                      (4)
>>>>>> +                  |                                       V
>>>>>> +                  |                              /-------------\
>>>>>> +                  |      Copy and Forward        |             |
>>>>>> +                  |---------(1)----------+       | Disk Buffer |
>>>>>> +                  |                      |       |             |
>>>>>> +                  |                     (3)      \-------------/
>>>>>> +                  |                 speculative      ^
>>>>>> +                  |                write through    (2)
>>>>>> +                  |                      |           |
>>>>>> +                  V                      V           |
>>>>>> +           +--------------+           +----------------+
>>>>>> +           | Primary Disk |           | Secondary Disk |
>>>>>> +           +--------------+           +----------------+
>>>>>> +
>>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>>>> +       QEMU.
>>>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>>>> +       original sector content will be read from Secondary disk and
>>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>>>> +       sector content in the Disk buffer.
>>>>>
>>>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
>>>>> reading them as "s/will be/are/g"
>>>>>
>>>>> Why do you need this buffer?
>>>>
>>>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
>>>> vm write to the buffer.
>>>>
>>>>>
>>>>> If both primary and secondary write to the same sector, what is saved in the
>>>>> buffer?
>>>>
>>>> The primary content will be written to the secondary disk, and the secondary content
>>>> is saved in the buffer.
>>>
>>> I wonder if alternatively this is possible with an imaginary "writable backing
>>> image" feature, as described below.
>>>
>>> When we have a normal backing chain,
>>>
>>>                {virtio-blk dev 'foo'}
>>>                          |
>>>                          |
>>>                          |
>>>     [base] <- [mid] <- (foo)
>>>
>>> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
>>> to an existing image on top,
>>>
>>>                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
>>>                          |                              |
>>>                          |                              |
>>>                          |                              |
>>>     [base] <- [mid] <- (foo)  <---------------------- (bar)
>>>
>>> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
>>> We can utilize an automatic hidden drive-backup target:
>>>
>>>                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
>>>                          |                                                          |
>>>                          |                                                          |
>>>                          v                                                          v
>>>
>>>     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
>>>
>>>                          v                              ^
>>>                          v                              ^
>>>                          v                              ^
>>>                          v                              ^
>>>                          >>>> drive-backup sync=none >>>>
>>>
>>> So when guest writes to 'foo', the old data is moved to (hidden target), which
>>> remains unchanged from (bar)'s PoV.
>>>
>>> The drive in the middle is called hidden because QEMU creates it automatically,
>>> the naming is arbitrary.
>>
>> I don't understand this. In which function, the hidden target is created automatically?
>>
> 
> It's to be determined. This part is only in my mind :)

Does hidden target is only used for COLO?

Thanks
Wen Congyang

> 
> Fam
> 
>>
>>>
>>> It is interesting because it is a more generalized case of image fleecing,
>>> where the (hidden target) is exposed via NBD server for data scanning (read
>>> only) purpose.
>>>
>>> More interestingly, with above facility, it is also possible to create a guest
>>> visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
>>> cheaply. Or call it shadow copy if you will.
>>>
>>> Back to the COLO case, the configuration will be very similar:
>>>
>>>
>>>                       {primary wr}                                                {secondary vm}
>>>                             |                                                           |
>>>                             |                                                           |
>>>                             |                                                           |
>>>                             v                                                           v
>>>
>>>    [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)
>>>
>>>                             v                              ^
>>>                             v                              ^
>>>                             v                              ^
>>>                             v                              ^
>>>                             >>>> drive-backup sync=none >>>>
>>>
>>> The workflow analogue is:
>>>
>>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>>>> +       QEMU.
>>>
>>> Primary write requests are forwarded to secondary QEMU as well.
>>>
>>>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>>>> +       original sector content will be read from Secondary disk and
>>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>>>> +       sector content in the Disk buffer.
>>>
>>> Before Primary write requests are written to (nbd target), aka the Secondary
>>> disk, the orignal sector content is read from it and copied to (hidden buf
>>> disk) by drive-backup. It obviously will not overwrite the data in (active
>>> disk).
>>>
>>>>>> +    3) Primary write requests will be written to Secondary disk.
>>>
>>> Primary write requests are written to (nbd target).
>>>
>>>>>> +    4) Secondary write requests will be buffered in the Disk buffer and it
>>>>>> +       will overwrite the existing sector content in the buffer.
>>>
>>> Secondary write request will be written in (active disk) as usual.
>>>
>>> Finally, when checkpoint arrives, if you want to sync with primary, just drop
>>> data in (hidden buf disk) and (active disk); when failover happends, if you
>>> want to promote secondary vm, you can commit (active disk) to (nbd target), and
>>> drop data in (hidden buf disk).
>>>
>>> Fam
>>> .
>>>
>>
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Fam Zheng]

On Wed, 02/25 16:36, Wen Congyang wrote:
> On 02/25/2015 10:46 AM, Fam Zheng wrote:
> > On Tue, 02/24 15:50, Wen Congyang wrote:
> >> On 02/12/2015 04:44 PM, Fam Zheng wrote:
> >>> On Thu, 02/12 15:40, Wen Congyang wrote:
> >>>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
> >>>>> Hi Congyang,
> >>>>>
> >>>>> On Thu, 02/12 11:07, Wen Congyang wrote:
> >>>>>> +== Workflow ==
> >>>>>> +The following is the image of block replication workflow:
> >>>>>> +
> >>>>>> +        +----------------------+            +------------------------+
> >>>>>> +        |Primary Write Requests|            |Secondary Write Requests|
> >>>>>> +        +----------------------+            +------------------------+
> >>>>>> +                  |                                       |
> >>>>>> +                  |                                      (4)
> >>>>>> +                  |                                       V
> >>>>>> +                  |                              /-------------\
> >>>>>> +                  |      Copy and Forward        |             |
> >>>>>> +                  |---------(1)----------+       | Disk Buffer |
> >>>>>> +                  |                      |       |             |
> >>>>>> +                  |                     (3)      \-------------/
> >>>>>> +                  |                 speculative      ^
> >>>>>> +                  |                write through    (2)
> >>>>>> +                  |                      |           |
> >>>>>> +                  V                      V           |
> >>>>>> +           +--------------+           +----------------+
> >>>>>> +           | Primary Disk |           | Secondary Disk |
> >>>>>> +           +--------------+           +----------------+
> >>>>>> +
> >>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
> >>>>>> +       QEMU.
> >>>>>> +    2) Before Primary write requests are written to Secondary disk, the
> >>>>>> +       original sector content will be read from Secondary disk and
> >>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
> >>>>>> +       sector content in the Disk buffer.
> >>>>>
> >>>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
> >>>>> reading them as "s/will be/are/g"
> >>>>>
> >>>>> Why do you need this buffer?
> >>>>
> >>>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
> >>>> vm write to the buffer.
> >>>>
> >>>>>
> >>>>> If both primary and secondary write to the same sector, what is saved in the
> >>>>> buffer?
> >>>>
> >>>> The primary content will be written to the secondary disk, and the secondary content
> >>>> is saved in the buffer.
> >>>
> >>> I wonder if alternatively this is possible with an imaginary "writable backing
> >>> image" feature, as described below.
> >>>
> >>> When we have a normal backing chain,
> >>>
> >>>                {virtio-blk dev 'foo'}
> >>>                          |
> >>>                          |
> >>>                          |
> >>>     [base] <- [mid] <- (foo)
> >>>
> >>> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
> >>> to an existing image on top,
> >>>
> >>>                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
> >>>                          |                              |
> >>>                          |                              |
> >>>                          |                              |
> >>>     [base] <- [mid] <- (foo)  <---------------------- (bar)
> >>>
> >>> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
> >>> We can utilize an automatic hidden drive-backup target:
> >>>
> >>>                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
> >>>                          |                                                          |
> >>>                          |                                                          |
> >>>                          v                                                          v
> >>>
> >>>     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
> >>>
> >>>                          v                              ^
> >>>                          v                              ^
> >>>                          v                              ^
> >>>                          v                              ^
> >>>                          >>>> drive-backup sync=none >>>>
> >>>
> >>> So when guest writes to 'foo', the old data is moved to (hidden target), which
> >>> remains unchanged from (bar)'s PoV.
> >>>
> >>> The drive in the middle is called hidden because QEMU creates it automatically,
> >>> the naming is arbitrary.
> >>
> >> I don't understand this. In which function, the hidden target is created automatically?
> >>
> > 
> > It's to be determined. This part is only in my mind :)
> 
> Does hidden target is only used for COLO?
> 

I'm not sure I get your question.

In this case yes, this is a dedicate target that's only written to by COLO's
secondary VM.

In other general cases, this infrastructure could also be used for backup or
image fleecing.

Fam

> 
> > 
> > Fam
> > 
> >>
> >>>
> >>> It is interesting because it is a more generalized case of image fleecing,
> >>> where the (hidden target) is exposed via NBD server for data scanning (read
> >>> only) purpose.
> >>>
> >>> More interestingly, with above facility, it is also possible to create a guest
> >>> visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
> >>> cheaply. Or call it shadow copy if you will.
> >>>
> >>> Back to the COLO case, the configuration will be very similar:
> >>>
> >>>
> >>>                       {primary wr}                                                {secondary vm}
> >>>                             |                                                           |
> >>>                             |                                                           |
> >>>                             |                                                           |
> >>>                             v                                                           v
> >>>
> >>>    [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)
> >>>
> >>>                             v                              ^
> >>>                             v                              ^
> >>>                             v                              ^
> >>>                             v                              ^
> >>>                             >>>> drive-backup sync=none >>>>
> >>>
> >>> The workflow analogue is:
> >>>
> >>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
> >>>>>> +       QEMU.
> >>>
> >>> Primary write requests are forwarded to secondary QEMU as well.
> >>>
> >>>>>> +    2) Before Primary write requests are written to Secondary disk, the
> >>>>>> +       original sector content will be read from Secondary disk and
> >>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
> >>>>>> +       sector content in the Disk buffer.
> >>>
> >>> Before Primary write requests are written to (nbd target), aka the Secondary
> >>> disk, the orignal sector content is read from it and copied to (hidden buf
> >>> disk) by drive-backup. It obviously will not overwrite the data in (active
> >>> disk).
> >>>
> >>>>>> +    3) Primary write requests will be written to Secondary disk.
> >>>
> >>> Primary write requests are written to (nbd target).
> >>>
> >>>>>> +    4) Secondary write requests will be buffered in the Disk buffer and it
> >>>>>> +       will overwrite the existing sector content in the buffer.
> >>>
> >>> Secondary write request will be written in (active disk) as usual.
> >>>
> >>> Finally, when checkpoint arrives, if you want to sync with primary, just drop
> >>> data in (hidden buf disk) and (active disk); when failover happends, if you
> >>> want to promote secondary vm, you can commit (active disk) to (nbd target), and
> >>> drop data in (hidden buf disk).
> >>>
> >>> Fam
> >>> .
> >>>
> >>
> > .
> > 
> 
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 02/12/2015 04:44 PM, Fam Zheng wrote:
> On Thu, 02/12 15:40, Wen Congyang wrote:
>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
>>> Hi Congyang,
>>>
>>> On Thu, 02/12 11:07, Wen Congyang wrote:
>>>> +== Workflow ==
>>>> +The following is the image of block replication workflow:
>>>> +
>>>> +        +----------------------+            +------------------------+
>>>> +        |Primary Write Requests|            |Secondary Write Requests|
>>>> +        +----------------------+            +------------------------+
>>>> +                  |                                       |
>>>> +                  |                                      (4)
>>>> +                  |                                       V
>>>> +                  |                              /-------------\
>>>> +                  |      Copy and Forward        |             |
>>>> +                  |---------(1)----------+       | Disk Buffer |
>>>> +                  |                      |       |             |
>>>> +                  |                     (3)      \-------------/
>>>> +                  |                 speculative      ^
>>>> +                  |                write through    (2)
>>>> +                  |                      |           |
>>>> +                  V                      V           |
>>>> +           +--------------+           +----------------+
>>>> +           | Primary Disk |           | Secondary Disk |
>>>> +           +--------------+           +----------------+
>>>> +
>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>> +       QEMU.
>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>> +       original sector content will be read from Secondary disk and
>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>> +       sector content in the Disk buffer.
>>>
>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
>>> reading them as "s/will be/are/g"
>>>
>>> Why do you need this buffer?
>>
>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
>> vm write to the buffer.
>>
>>>
>>> If both primary and secondary write to the same sector, what is saved in the
>>> buffer?
>>
>> The primary content will be written to the secondary disk, and the secondary content
>> is saved in the buffer.
> 
> I wonder if alternatively this is possible with an imaginary "writable backing
> image" feature, as described below.
> 
> When we have a normal backing chain,
> 
>                {virtio-blk dev 'foo'}
>                          |
>                          |
>                          |
>     [base] <- [mid] <- (foo)
> 
> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
> to an existing image on top,
> 
>                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
>                          |                              |
>                          |                              |
>                          |                              |
>     [base] <- [mid] <- (foo)  <---------------------- (bar)
> 
> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
> We can utilize an automatic hidden drive-backup target:
> 
>                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
>                          |                                                          |
>                          |                                                          |
>                          v                                                          v
> 
>     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
> 
>                          v                              ^
>                          v                              ^
>                          v                              ^
>                          v                              ^
>                          >>>> drive-backup sync=none >>>>
> 
> So when guest writes to 'foo', the old data is moved to (hidden target), which
> remains unchanged from (bar)'s PoV.
> 
> The drive in the middle is called hidden because QEMU creates it automatically,
> the naming is arbitrary.
> 
> It is interesting because it is a more generalized case of image fleecing,
> where the (hidden target) is exposed via NBD server for data scanning (read
> only) purpose.
> 
> More interestingly, with above facility, it is also possible to create a guest
> visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
> cheaply. Or call it shadow copy if you will.
> 
> Back to the COLO case, the configuration will be very similar:
> 
> 
>                       {primary wr}                                                {secondary vm}
>                             |                                                           |
>                             |                                                           |
>                             |                                                           |
>                             v                                                           v
> 
>    [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)
> 
>                             v                              ^
>                             v                              ^
>                             v                              ^
>                             v                              ^
>                             >>>> drive-backup sync=none >>>>

Why nbd target has backing image ever?

> 
> The workflow analogue is:
> 
>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>> +       QEMU.
> 
> Primary write requests are forwarded to secondary QEMU as well.
> 
>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>> +       original sector content will be read from Secondary disk and
>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>> +       sector content in the Disk buffer.
> 
> Before Primary write requests are written to (nbd target), aka the Secondary
> disk, the orignal sector content is read from it and copied to (hidden buf
> disk) by drive-backup. It obviously will not overwrite the data in (active
> disk).
> 
>>>> +    3) Primary write requests will be written to Secondary disk.
> 
> Primary write requests are written to (nbd target).
> 
>>>> +    4) Secondary write requests will be buffered in the Disk buffer and it
>>>> +       will overwrite the existing sector content in the buffer.
> 
> Secondary write request will be written in (active disk) as usual.
> 
> Finally, when checkpoint arrives, if you want to sync with primary, just drop
> data in (hidden buf disk) and (active disk); when failover happends, if you
> want to promote secondary vm, you can commit (active disk) to (nbd target), and
> drop data in (hidden buf disk).

We cannot drop data in (hidden buf disk). Commit (hidden buf disk) to (nbd target)
first, and then commit (active disk) to (nbd target).

Thanks
Wen Congyang

> 
> Fam
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Fam Zheng]

On Wed, 02/25 17:10, Wen Congyang wrote:
> On 02/12/2015 04:44 PM, Fam Zheng wrote:
> > On Thu, 02/12 15:40, Wen Congyang wrote:
> >> On 02/12/2015 03:21 PM, Fam Zheng wrote:
> >>> Hi Congyang,
> >>>
> >>> On Thu, 02/12 11:07, Wen Congyang wrote:
> >>>> +== Workflow ==
> >>>> +The following is the image of block replication workflow:
> >>>> +
> >>>> +        +----------------------+            +------------------------+
> >>>> +        |Primary Write Requests|            |Secondary Write Requests|
> >>>> +        +----------------------+            +------------------------+
> >>>> +                  |                                       |
> >>>> +                  |                                      (4)
> >>>> +                  |                                       V
> >>>> +                  |                              /-------------\
> >>>> +                  |      Copy and Forward        |             |
> >>>> +                  |---------(1)----------+       | Disk Buffer |
> >>>> +                  |                      |       |             |
> >>>> +                  |                     (3)      \-------------/
> >>>> +                  |                 speculative      ^
> >>>> +                  |                write through    (2)
> >>>> +                  |                      |           |
> >>>> +                  V                      V           |
> >>>> +           +--------------+           +----------------+
> >>>> +           | Primary Disk |           | Secondary Disk |
> >>>> +           +--------------+           +----------------+
> >>>> +
> >>>> +    1) Primary write requests will be copied and forwarded to Secondary
> >>>> +       QEMU.
> >>>> +    2) Before Primary write requests are written to Secondary disk, the
> >>>> +       original sector content will be read from Secondary disk and
> >>>> +       buffered in the Disk buffer, but it will not overwrite the existing
> >>>> +       sector content in the Disk buffer.
> >>>
> >>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
> >>> reading them as "s/will be/are/g"
> >>>
> >>> Why do you need this buffer?
> >>
> >> We only sync the disk till next checkpoint. Before next checkpoint, secondary
> >> vm write to the buffer.
> >>
> >>>
> >>> If both primary and secondary write to the same sector, what is saved in the
> >>> buffer?
> >>
> >> The primary content will be written to the secondary disk, and the secondary content
> >> is saved in the buffer.
> > 
> > I wonder if alternatively this is possible with an imaginary "writable backing
> > image" feature, as described below.
> > 
> > When we have a normal backing chain,
> > 
> >                {virtio-blk dev 'foo'}
> >                          |
> >                          |
> >                          |
> >     [base] <- [mid] <- (foo)
> > 
> > Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
> > to an existing image on top,
> > 
> >                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
> >                          |                              |
> >                          |                              |
> >                          |                              |
> >     [base] <- [mid] <- (foo)  <---------------------- (bar)
> > 
> > It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
> > We can utilize an automatic hidden drive-backup target:
> > 
> >                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
> >                          |                                                          |
> >                          |                                                          |
> >                          v                                                          v
> > 
> >     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
> > 
> >                          v                              ^
> >                          v                              ^
> >                          v                              ^
> >                          v                              ^
> >                          >>>> drive-backup sync=none >>>>
> > 
> > So when guest writes to 'foo', the old data is moved to (hidden target), which
> > remains unchanged from (bar)'s PoV.
> > 
> > The drive in the middle is called hidden because QEMU creates it automatically,
> > the naming is arbitrary.
> > 
> > It is interesting because it is a more generalized case of image fleecing,
> > where the (hidden target) is exposed via NBD server for data scanning (read
> > only) purpose.
> > 
> > More interestingly, with above facility, it is also possible to create a guest
> > visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
> > cheaply. Or call it shadow copy if you will.
> > 
> > Back to the COLO case, the configuration will be very similar:
> > 
> > 
> >                       {primary wr}                                                {secondary vm}
> >                             |                                                           |
> >                             |                                                           |
> >                             |                                                           |
> >                             v                                                           v
> > 
> >    [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)
> > 
> >                             v                              ^
> >                             v                              ^
> >                             v                              ^
> >                             v                              ^
> >                             >>>> drive-backup sync=none >>>>
> 
> Why nbd target has backing image ever?

It's not strictly necessary, depending on your VM disk configuration. (for
example at the time of vm booting, your image already points to a backing file,
etc.

Fam

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 02/25/2015 04:58 PM, Fam Zheng wrote:
> On Wed, 02/25 16:36, Wen Congyang wrote:
>> On 02/25/2015 10:46 AM, Fam Zheng wrote:
>>> On Tue, 02/24 15:50, Wen Congyang wrote:
>>>> On 02/12/2015 04:44 PM, Fam Zheng wrote:
>>>>> On Thu, 02/12 15:40, Wen Congyang wrote:
>>>>>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
>>>>>>> Hi Congyang,
>>>>>>>
>>>>>>> On Thu, 02/12 11:07, Wen Congyang wrote:
>>>>>>>> +== Workflow ==
>>>>>>>> +The following is the image of block replication workflow:
>>>>>>>> +
>>>>>>>> +        +----------------------+            +------------------------+
>>>>>>>> +        |Primary Write Requests|            |Secondary Write Requests|
>>>>>>>> +        +----------------------+            +------------------------+
>>>>>>>> +                  |                                       |
>>>>>>>> +                  |                                      (4)
>>>>>>>> +                  |                                       V
>>>>>>>> +                  |                              /-------------\
>>>>>>>> +                  |      Copy and Forward        |             |
>>>>>>>> +                  |---------(1)----------+       | Disk Buffer |
>>>>>>>> +                  |                      |       |             |
>>>>>>>> +                  |                     (3)      \-------------/
>>>>>>>> +                  |                 speculative      ^
>>>>>>>> +                  |                write through    (2)
>>>>>>>> +                  |                      |           |
>>>>>>>> +                  V                      V           |
>>>>>>>> +           +--------------+           +----------------+
>>>>>>>> +           | Primary Disk |           | Secondary Disk |
>>>>>>>> +           +--------------+           +----------------+
>>>>>>>> +
>>>>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>>>>>> +       QEMU.
>>>>>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>>>>>> +       original sector content will be read from Secondary disk and
>>>>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>>>>>> +       sector content in the Disk buffer.
>>>>>>>
>>>>>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
>>>>>>> reading them as "s/will be/are/g"
>>>>>>>
>>>>>>> Why do you need this buffer?
>>>>>>
>>>>>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
>>>>>> vm write to the buffer.
>>>>>>
>>>>>>>
>>>>>>> If both primary and secondary write to the same sector, what is saved in the
>>>>>>> buffer?
>>>>>>
>>>>>> The primary content will be written to the secondary disk, and the secondary content
>>>>>> is saved in the buffer.
>>>>>
>>>>> I wonder if alternatively this is possible with an imaginary "writable backing
>>>>> image" feature, as described below.
>>>>>
>>>>> When we have a normal backing chain,
>>>>>
>>>>>                {virtio-blk dev 'foo'}
>>>>>                          |
>>>>>                          |
>>>>>                          |
>>>>>     [base] <- [mid] <- (foo)
>>>>>
>>>>> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
>>>>> to an existing image on top,
>>>>>
>>>>>                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
>>>>>                          |                              |
>>>>>                          |                              |
>>>>>                          |                              |
>>>>>     [base] <- [mid] <- (foo)  <---------------------- (bar)
>>>>>
>>>>> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
>>>>> We can utilize an automatic hidden drive-backup target:
>>>>>
>>>>>                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
>>>>>                          |                                                          |
>>>>>                          |                                                          |
>>>>>                          v                                                          v
>>>>>
>>>>>     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
>>>>>
>>>>>                          v                              ^
>>>>>                          v                              ^
>>>>>                          v                              ^
>>>>>                          v                              ^
>>>>>                          >>>> drive-backup sync=none >>>>
>>>>>
>>>>> So when guest writes to 'foo', the old data is moved to (hidden target), which
>>>>> remains unchanged from (bar)'s PoV.
>>>>>
>>>>> The drive in the middle is called hidden because QEMU creates it automatically,
>>>>> the naming is arbitrary.
>>>>
>>>> I don't understand this. In which function, the hidden target is created automatically?
>>>>
>>>
>>> It's to be determined. This part is only in my mind :)
>>
>> Does hidden target is only used for COLO?
>>
> 
> I'm not sure I get your question.
> 
> In this case yes, this is a dedicate target that's only written to by COLO's
> secondary VM.
> 
> In other general cases, this infrastructure could also be used for backup or
> image fleecing.

In COLO case, we can create (hidden buf disk) when starting block replication.
In other general cases, I don't know when to create (hidden buf disk)?

Thanks
Wen Congyang

> 
> Fam
> 
>>
>>>
>>> Fam
>>>
>>>>
>>>>>
>>>>> It is interesting because it is a more generalized case of image fleecing,
>>>>> where the (hidden target) is exposed via NBD server for data scanning (read
>>>>> only) purpose.
>>>>>
>>>>> More interestingly, with above facility, it is also possible to create a guest
>>>>> visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
>>>>> cheaply. Or call it shadow copy if you will.
>>>>>
>>>>> Back to the COLO case, the configuration will be very similar:
>>>>>
>>>>>
>>>>>                       {primary wr}                                                {secondary vm}
>>>>>                             |                                                           |
>>>>>                             |                                                           |
>>>>>                             |                                                           |
>>>>>                             v                                                           v
>>>>>
>>>>>    [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)
>>>>>
>>>>>                             v                              ^
>>>>>                             v                              ^
>>>>>                             v                              ^
>>>>>                             v                              ^
>>>>>                             >>>> drive-backup sync=none >>>>
>>>>>
>>>>> The workflow analogue is:
>>>>>
>>>>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>>>>>> +       QEMU.
>>>>>
>>>>> Primary write requests are forwarded to secondary QEMU as well.
>>>>>
>>>>>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>>>>>> +       original sector content will be read from Secondary disk and
>>>>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>>>>>> +       sector content in the Disk buffer.
>>>>>
>>>>> Before Primary write requests are written to (nbd target), aka the Secondary
>>>>> disk, the orignal sector content is read from it and copied to (hidden buf
>>>>> disk) by drive-backup. It obviously will not overwrite the data in (active
>>>>> disk).
>>>>>
>>>>>>>> +    3) Primary write requests will be written to Secondary disk.
>>>>>
>>>>> Primary write requests are written to (nbd target).
>>>>>
>>>>>>>> +    4) Secondary write requests will be buffered in the Disk buffer and it
>>>>>>>> +       will overwrite the existing sector content in the buffer.
>>>>>
>>>>> Secondary write request will be written in (active disk) as usual.
>>>>>
>>>>> Finally, when checkpoint arrives, if you want to sync with primary, just drop
>>>>> data in (hidden buf disk) and (active disk); when failover happends, if you
>>>>> want to promote secondary vm, you can commit (active disk) to (nbd target), and
>>>>> drop data in (hidden buf disk).
>>>>>
>>>>> Fam
>>>>> .
>>>>>
>>>>
>>> .
>>>
>>
>>
> .
> 

Re: [Qemu-devel] [RFC PATCH 06/14] NBD client: connect to nbd server later

[Max Reitz]

On 2015-02-24 at 21:23, Wen Congyang wrote:
> On 02/24/2015 05:31 AM, Max Reitz wrote:
>> On 2015-02-11 at 22:07, Wen Congyang wrote:
>>> The secondary qemu starts later than the primary qemu, so we
>>> cannot connect to nbd server in bdrv_open().
>>>
>>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
>>> ---
>>>    block/nbd.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++++--------
>>>    1 file changed, 87 insertions(+), 13 deletions(-)
>>>
>>> diff --git a/block/nbd.c b/block/nbd.c
>>> index b05d1d0..19b9200 100644
>>> --- a/block/nbd.c
>>> +++ b/block/nbd.c
>>> @@ -44,6 +44,8 @@
>>>    typedef struct BDRVNBDState {
>>>        NbdClientSession client;
>>>        QemuOpts *socket_opts;
>>> +    char *export;
>>> +    bool connected;
>>>    } BDRVNBDState;
>>>      static int nbd_parse_uri(const char *filename, QDict *options)
>>> @@ -247,20 +249,10 @@ static int nbd_establish_connection(BlockDriverState *bs, Error **errp)
>>>        return sock;
>>>    }
>>>    -static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
>>> -                    Error **errp)
>>> +static int nbd_connect_server(BlockDriverState *bs, Error **errp)
>>>    {
>>>        BDRVNBDState *s = bs->opaque;
>>> -    char *export = NULL;
>>>        int result, sock;
>>> -    Error *local_err = NULL;
>>> -
>>> -    /* Pop the config into our state object. Exit if invalid. */
>>> -    nbd_config(s, options, &export, &local_err);
>>> -    if (local_err) {
>>> -        error_propagate(errp, local_err);
>>> -        return -EINVAL;
>>> -    }
>>>          /* establish TCP connection, return error if it fails
>>>         * TODO: Configurable retry-until-timeout behaviour.
>>> @@ -271,16 +263,57 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
>>>        }
>>>          /* NBD handshake */
>>> -    result = nbd_client_session_init(&s->client, bs, sock, export, errp);
>>> -    g_free(export);
>>> +    result = nbd_client_session_init(&s->client, bs, sock, s->export, errp);
>>> +    g_free(s->export);
>>> +    s->export = NULL;
>>> +    if (!result) {
>>> +        s->connected = true;
>>> +    }
>>> +
>>>        return result;
>>>    }
>>>    +static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
>>> +                    Error **errp)
>>> +{
>>> +    BDRVNBDState *s = bs->opaque;
>>> +    Error *local_err = NULL;
>>> +
>>> +    /* Pop the config into our state object. Exit if invalid. */
>>> +    nbd_config(s, options, &s->export, &local_err);
>>> +    if (local_err) {
>>> +        error_propagate(errp, local_err);
>>> +        return -EINVAL;
>>> +    }
>>> +
>>> +    return nbd_connect_server(bs, errp);
>>> +}
>>> +
>>> +static int nbd_open_colo(BlockDriverState *bs, QDict *options, int flags,
>>> +                         Error **errp)
>>> +{
>>> +    BDRVNBDState *s = bs->opaque;
>>> +    Error *local_err = NULL;
>>> +
>>> +    /* Pop the config into our state object. Exit if invalid. */
>>> +    nbd_config(s, options, &s->export, &local_err);
>>> +    if (local_err) {
>>> +        error_propagate(errp, local_err);
>>> +        return -EINVAL;
>>> +    }
>>> +
>>> +    return 0;
>>> +}
>>> +
>>>    static int nbd_co_readv(BlockDriverState *bs, int64_t sector_num,
>>>                            int nb_sectors, QEMUIOVector *qiov)
>>>    {
>>>        BDRVNBDState *s = bs->opaque;
>>>    +    if (!s->connected) {
>>> +        return -EIO;
>>> +    }
>>> +
>>>        return nbd_client_session_co_readv(&s->client, sector_num,
>>>                                           nb_sectors, qiov);
>>>    }
>>> @@ -290,6 +323,10 @@ static int nbd_co_writev(BlockDriverState *bs, int64_t sector_num,
>>>    {
>>>        BDRVNBDState *s = bs->opaque;
>>>    +    if (!s->connected) {
>>> +        return 0;
>>> +    }
>> Would it break anything to return -EIO here as well? (And in all the following functions)
> 1. nbd_co_writev()
>     If one child returns error, quorum will report it. There may be many write requests before
>     we connect to nbd server, so there are too many qapi events...
> 2. nbd_co_flush()
>     If quorum only have two children, and nbd client is the last one, quorum_co_flush()
>     will return -EIO.
> 3. nbd_co_discard()
>     quorum doens't call bdrv_co_discard(), so it is OK to return -EIO here.
>
> So only nbd_co_discard() can return -EIO.

Hm, okay. How about adding an option to quorum for ignoring errors from 
a specific child? It's probably not possible to do something like 
"children.1.ignore-errors=true", but maybe you can just ignore errors in 
quorum from any but the first child if the read pattern is set to 
"first", that would make sense to me.

But if you don't want to do that, I guess just making NBD some kind of 
/dev/null before it's connected should be fine.

Max

>>> +
>>>        return nbd_client_session_co_writev(&s->client, sector_num,
>>>                                            nb_sectors, qiov);
>>>    }
>>> @@ -298,6 +335,10 @@ static int nbd_co_flush(BlockDriverState *bs)
>>>    {
>>>        BDRVNBDState *s = bs->opaque;
>>>    +    if (!s->connected) {
>>> +        return 0;
>>> +    }
>>> +
>>>        return nbd_client_session_co_flush(&s->client);
>>>    }
>>>    @@ -312,6 +353,10 @@ static int nbd_co_discard(BlockDriverState *bs, int64_t sector_num,
>>>    {
>>>        BDRVNBDState *s = bs->opaque;
>>>    +    if (!s->connected) {
>>> +        return 0;
>>> +    }
>>> +
>>>        return nbd_client_session_co_discard(&s->client, sector_num,
>>>                                             nb_sectors);
>>>    }
>>> @@ -322,6 +367,7 @@ static void nbd_close(BlockDriverState *bs)
>>>          qemu_opts_del(s->socket_opts);
>>>        nbd_client_session_close(&s->client);
>>> +    s->connected = false;
>>>    }
>>>    

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 02/25/2015 10:46 AM, Fam Zheng wrote:
> On Tue, 02/24 15:50, Wen Congyang wrote:
>> On 02/12/2015 04:44 PM, Fam Zheng wrote:
>>> On Thu, 02/12 15:40, Wen Congyang wrote:
>>>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
>>>>> Hi Congyang,
>>>>>
>>>>> On Thu, 02/12 11:07, Wen Congyang wrote:
>>>>>> +== Workflow ==
>>>>>> +The following is the image of block replication workflow:
>>>>>> +
>>>>>> +        +----------------------+            +------------------------+
>>>>>> +        |Primary Write Requests|            |Secondary Write Requests|
>>>>>> +        +----------------------+            +------------------------+
>>>>>> +                  |                                       |
>>>>>> +                  |                                      (4)
>>>>>> +                  |                                       V
>>>>>> +                  |                              /-------------\
>>>>>> +                  |      Copy and Forward        |             |
>>>>>> +                  |---------(1)----------+       | Disk Buffer |
>>>>>> +                  |                      |       |             |
>>>>>> +                  |                     (3)      \-------------/
>>>>>> +                  |                 speculative      ^
>>>>>> +                  |                write through    (2)
>>>>>> +                  |                      |           |
>>>>>> +                  V                      V           |
>>>>>> +           +--------------+           +----------------+
>>>>>> +           | Primary Disk |           | Secondary Disk |
>>>>>> +           +--------------+           +----------------+
>>>>>> +
>>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>>>> +       QEMU.
>>>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>>>> +       original sector content will be read from Secondary disk and
>>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>>>> +       sector content in the Disk buffer.
>>>>>
>>>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
>>>>> reading them as "s/will be/are/g"
>>>>>
>>>>> Why do you need this buffer?
>>>>
>>>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
>>>> vm write to the buffer.
>>>>
>>>>>
>>>>> If both primary and secondary write to the same sector, what is saved in the
>>>>> buffer?
>>>>
>>>> The primary content will be written to the secondary disk, and the secondary content
>>>> is saved in the buffer.
>>>
>>> I wonder if alternatively this is possible with an imaginary "writable backing
>>> image" feature, as described below.
>>>
>>> When we have a normal backing chain,
>>>
>>>                {virtio-blk dev 'foo'}
>>>                          |
>>>                          |
>>>                          |
>>>     [base] <- [mid] <- (foo)
>>>
>>> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
>>> to an existing image on top,
>>>
>>>                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
>>>                          |                              |
>>>                          |                              |
>>>                          |                              |
>>>     [base] <- [mid] <- (foo)  <---------------------- (bar)
>>>
>>> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
>>> We can utilize an automatic hidden drive-backup target:
>>>
>>>                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
>>>                          |                                                          |
>>>                          |                                                          |
>>>                          v                                                          v
>>>
>>>     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
>>>
>>>                          v                              ^
>>>                          v                              ^
>>>                          v                              ^
>>>                          v                              ^
>>>                          >>>> drive-backup sync=none >>>>
>>>
>>> So when guest writes to 'foo', the old data is moved to (hidden target), which
>>> remains unchanged from (bar)'s PoV.
>>>
>>> The drive in the middle is called hidden because QEMU creates it automatically,
>>> the naming is arbitrary.
>>
>> I don't understand this. In which function, the hidden target is created automatically?
>>
> 
> It's to be determined. This part is only in my mind :)

What about this:
-drive file=nbd-target,if=none,id=nbd-target0 \
-drive file=active-disk,if=virtio,driver=qcow2,backing.file.filename=hidden-disk,backing.driver=qcow2,backing.backing=nbd-target0

Thanks
Wen Congyang

> 
> Fam
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Fam Zheng]

On Thu, 02/26 14:38, Wen Congyang wrote:
> On 02/25/2015 10:46 AM, Fam Zheng wrote:
> > On Tue, 02/24 15:50, Wen Congyang wrote:
> >> On 02/12/2015 04:44 PM, Fam Zheng wrote:
> >>> On Thu, 02/12 15:40, Wen Congyang wrote:
> >>>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
> >>>>> Hi Congyang,
> >>>>>
> >>>>> On Thu, 02/12 11:07, Wen Congyang wrote:
> >>>>>> +== Workflow ==
> >>>>>> +The following is the image of block replication workflow:
> >>>>>> +
> >>>>>> +        +----------------------+            +------------------------+
> >>>>>> +        |Primary Write Requests|            |Secondary Write Requests|
> >>>>>> +        +----------------------+            +------------------------+
> >>>>>> +                  |                                       |
> >>>>>> +                  |                                      (4)
> >>>>>> +                  |                                       V
> >>>>>> +                  |                              /-------------\
> >>>>>> +                  |      Copy and Forward        |             |
> >>>>>> +                  |---------(1)----------+       | Disk Buffer |
> >>>>>> +                  |                      |       |             |
> >>>>>> +                  |                     (3)      \-------------/
> >>>>>> +                  |                 speculative      ^
> >>>>>> +                  |                write through    (2)
> >>>>>> +                  |                      |           |
> >>>>>> +                  V                      V           |
> >>>>>> +           +--------------+           +----------------+
> >>>>>> +           | Primary Disk |           | Secondary Disk |
> >>>>>> +           +--------------+           +----------------+
> >>>>>> +
> >>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
> >>>>>> +       QEMU.
> >>>>>> +    2) Before Primary write requests are written to Secondary disk, the
> >>>>>> +       original sector content will be read from Secondary disk and
> >>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
> >>>>>> +       sector content in the Disk buffer.
> >>>>>
> >>>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
> >>>>> reading them as "s/will be/are/g"
> >>>>>
> >>>>> Why do you need this buffer?
> >>>>
> >>>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
> >>>> vm write to the buffer.
> >>>>
> >>>>>
> >>>>> If both primary and secondary write to the same sector, what is saved in the
> >>>>> buffer?
> >>>>
> >>>> The primary content will be written to the secondary disk, and the secondary content
> >>>> is saved in the buffer.
> >>>
> >>> I wonder if alternatively this is possible with an imaginary "writable backing
> >>> image" feature, as described below.
> >>>
> >>> When we have a normal backing chain,
> >>>
> >>>                {virtio-blk dev 'foo'}
> >>>                          |
> >>>                          |
> >>>                          |
> >>>     [base] <- [mid] <- (foo)
> >>>
> >>> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
> >>> to an existing image on top,
> >>>
> >>>                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
> >>>                          |                              |
> >>>                          |                              |
> >>>                          |                              |
> >>>     [base] <- [mid] <- (foo)  <---------------------- (bar)
> >>>
> >>> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
> >>> We can utilize an automatic hidden drive-backup target:
> >>>
> >>>                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
> >>>                          |                                                          |
> >>>                          |                                                          |
> >>>                          v                                                          v
> >>>
> >>>     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
> >>>
> >>>                          v                              ^
> >>>                          v                              ^
> >>>                          v                              ^
> >>>                          v                              ^
> >>>                          >>>> drive-backup sync=none >>>>
> >>>
> >>> So when guest writes to 'foo', the old data is moved to (hidden target), which
> >>> remains unchanged from (bar)'s PoV.
> >>>
> >>> The drive in the middle is called hidden because QEMU creates it automatically,
> >>> the naming is arbitrary.
> >>
> >> I don't understand this. In which function, the hidden target is created automatically?
> >>
> > 
> > It's to be determined. This part is only in my mind :)
> 
> What about this:
> -drive file=nbd-target,if=none,id=nbd-target0 \
> -drive file=active-disk,if=virtio,driver=qcow2,backing.file.filename=hidden-disk,backing.driver=qcow2,backing.backing=nbd-target0
> 

It's close. I suppose backing.backing is referencing another drive as its
backing_hd, then you cannot have the other backing.file.* option - they
conflict. It would be something along:

-drive file=nbd-target,if=none,id=nbd-target0 \
-drive file=hidden-disk,if=none,id=hidden0,backing.backing=nbd-target0 \
-drive file=active-disk,if=virtio,driver=qcow2,backing.backing=hidden0

Or for simplicity, s/backing.backing=/backing=/g

Yes, adding these "backing=$drive_id" option is also exactly what we expect
in order to support image-fleecing, but we haven't figured how to allow that
without breaking other qmp operations like block jobs, etc.

Fam

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 02/26/2015 04:44 PM, Fam Zheng wrote:
> On Thu, 02/26 14:38, Wen Congyang wrote:
>> On 02/25/2015 10:46 AM, Fam Zheng wrote:
>>> On Tue, 02/24 15:50, Wen Congyang wrote:
>>>> On 02/12/2015 04:44 PM, Fam Zheng wrote:
>>>>> On Thu, 02/12 15:40, Wen Congyang wrote:
>>>>>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
>>>>>>> Hi Congyang,
>>>>>>>
>>>>>>> On Thu, 02/12 11:07, Wen Congyang wrote:
>>>>>>>> +== Workflow ==
>>>>>>>> +The following is the image of block replication workflow:
>>>>>>>> +
>>>>>>>> +        +----------------------+            +------------------------+
>>>>>>>> +        |Primary Write Requests|            |Secondary Write Requests|
>>>>>>>> +        +----------------------+            +------------------------+
>>>>>>>> +                  |                                       |
>>>>>>>> +                  |                                      (4)
>>>>>>>> +                  |                                       V
>>>>>>>> +                  |                              /-------------\
>>>>>>>> +                  |      Copy and Forward        |             |
>>>>>>>> +                  |---------(1)----------+       | Disk Buffer |
>>>>>>>> +                  |                      |       |             |
>>>>>>>> +                  |                     (3)      \-------------/
>>>>>>>> +                  |                 speculative      ^
>>>>>>>> +                  |                write through    (2)
>>>>>>>> +                  |                      |           |
>>>>>>>> +                  V                      V           |
>>>>>>>> +           +--------------+           +----------------+
>>>>>>>> +           | Primary Disk |           | Secondary Disk |
>>>>>>>> +           +--------------+           +----------------+
>>>>>>>> +
>>>>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>>>>>> +       QEMU.
>>>>>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>>>>>> +       original sector content will be read from Secondary disk and
>>>>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>>>>>> +       sector content in the Disk buffer.
>>>>>>>
>>>>>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
>>>>>>> reading them as "s/will be/are/g"
>>>>>>>
>>>>>>> Why do you need this buffer?
>>>>>>
>>>>>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
>>>>>> vm write to the buffer.
>>>>>>
>>>>>>>
>>>>>>> If both primary and secondary write to the same sector, what is saved in the
>>>>>>> buffer?
>>>>>>
>>>>>> The primary content will be written to the secondary disk, and the secondary content
>>>>>> is saved in the buffer.
>>>>>
>>>>> I wonder if alternatively this is possible with an imaginary "writable backing
>>>>> image" feature, as described below.
>>>>>
>>>>> When we have a normal backing chain,
>>>>>
>>>>>                {virtio-blk dev 'foo'}
>>>>>                          |
>>>>>                          |
>>>>>                          |
>>>>>     [base] <- [mid] <- (foo)
>>>>>
>>>>> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
>>>>> to an existing image on top,
>>>>>
>>>>>                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
>>>>>                          |                              |
>>>>>                          |                              |
>>>>>                          |                              |
>>>>>     [base] <- [mid] <- (foo)  <---------------------- (bar)
>>>>>
>>>>> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
>>>>> We can utilize an automatic hidden drive-backup target:
>>>>>
>>>>>                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
>>>>>                          |                                                          |
>>>>>                          |                                                          |
>>>>>                          v                                                          v
>>>>>
>>>>>     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
>>>>>
>>>>>                          v                              ^
>>>>>                          v                              ^
>>>>>                          v                              ^
>>>>>                          v                              ^
>>>>>                          >>>> drive-backup sync=none >>>>
>>>>>
>>>>> So when guest writes to 'foo', the old data is moved to (hidden target), which
>>>>> remains unchanged from (bar)'s PoV.
>>>>>
>>>>> The drive in the middle is called hidden because QEMU creates it automatically,
>>>>> the naming is arbitrary.
>>>>
>>>> I don't understand this. In which function, the hidden target is created automatically?
>>>>
>>>
>>> It's to be determined. This part is only in my mind :)
>>
>> What about this:
>> -drive file=nbd-target,if=none,id=nbd-target0 \
>> -drive file=active-disk,if=virtio,driver=qcow2,backing.file.filename=hidden-disk,backing.driver=qcow2,backing.backing=nbd-target0
>>
> 
> It's close. I suppose backing.backing is referencing another drive as its
> backing_hd, then you cannot have the other backing.file.* option - they
> conflict. It would be something along:
> 
> -drive file=nbd-target,if=none,id=nbd-target0 \
> -drive file=hidden-disk,if=none,id=hidden0,backing.backing=nbd-target0 \
> -drive file=active-disk,if=virtio,driver=qcow2,backing.backing=hidden0
> 
> Or for simplicity, s/backing.backing=/backing=/g

If using backing=drive_id, backing.backing and backing.file.* are not conflict.
backing.backing=$drive_id means that: backing file's backing file's id is $drive_id.

> 
> Yes, adding these "backing=$drive_id" option is also exactly what we expect
> in order to support image-fleecing, but we haven't figured how to allow that
> without breaking other qmp operations like block jobs, etc.

I don't understand this. In which case, qmp operations will be broken? Can you give
me some examples?

Thanks
Wen Congyang

> 
> Fam
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Fam Zheng]

On Thu, 02/26 17:07, Wen Congyang wrote:
> On 02/26/2015 04:44 PM, Fam Zheng wrote:
> > On Thu, 02/26 14:38, Wen Congyang wrote:
> >> On 02/25/2015 10:46 AM, Fam Zheng wrote:
> >>> On Tue, 02/24 15:50, Wen Congyang wrote:
> >>>> On 02/12/2015 04:44 PM, Fam Zheng wrote:
> >>>>> On Thu, 02/12 15:40, Wen Congyang wrote:
> >>>>>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
> >>>>>>> Hi Congyang,
> >>>>>>>
> >>>>>>> On Thu, 02/12 11:07, Wen Congyang wrote:
> >>>>>>>> +== Workflow ==
> >>>>>>>> +The following is the image of block replication workflow:
> >>>>>>>> +
> >>>>>>>> +        +----------------------+            +------------------------+
> >>>>>>>> +        |Primary Write Requests|            |Secondary Write Requests|
> >>>>>>>> +        +----------------------+            +------------------------+
> >>>>>>>> +                  |                                       |
> >>>>>>>> +                  |                                      (4)
> >>>>>>>> +                  |                                       V
> >>>>>>>> +                  |                              /-------------\
> >>>>>>>> +                  |      Copy and Forward        |             |
> >>>>>>>> +                  |---------(1)----------+       | Disk Buffer |
> >>>>>>>> +                  |                      |       |             |
> >>>>>>>> +                  |                     (3)      \-------------/
> >>>>>>>> +                  |                 speculative      ^
> >>>>>>>> +                  |                write through    (2)
> >>>>>>>> +                  |                      |           |
> >>>>>>>> +                  V                      V           |
> >>>>>>>> +           +--------------+           +----------------+
> >>>>>>>> +           | Primary Disk |           | Secondary Disk |
> >>>>>>>> +           +--------------+           +----------------+
> >>>>>>>> +
> >>>>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
> >>>>>>>> +       QEMU.
> >>>>>>>> +    2) Before Primary write requests are written to Secondary disk, the
> >>>>>>>> +       original sector content will be read from Secondary disk and
> >>>>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
> >>>>>>>> +       sector content in the Disk buffer.
> >>>>>>>
> >>>>>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
> >>>>>>> reading them as "s/will be/are/g"
> >>>>>>>
> >>>>>>> Why do you need this buffer?
> >>>>>>
> >>>>>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
> >>>>>> vm write to the buffer.
> >>>>>>
> >>>>>>>
> >>>>>>> If both primary and secondary write to the same sector, what is saved in the
> >>>>>>> buffer?
> >>>>>>
> >>>>>> The primary content will be written to the secondary disk, and the secondary content
> >>>>>> is saved in the buffer.
> >>>>>
> >>>>> I wonder if alternatively this is possible with an imaginary "writable backing
> >>>>> image" feature, as described below.
> >>>>>
> >>>>> When we have a normal backing chain,
> >>>>>
> >>>>>                {virtio-blk dev 'foo'}
> >>>>>                          |
> >>>>>                          |
> >>>>>                          |
> >>>>>     [base] <- [mid] <- (foo)
> >>>>>
> >>>>> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
> >>>>> to an existing image on top,
> >>>>>
> >>>>>                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
> >>>>>                          |                              |
> >>>>>                          |                              |
> >>>>>                          |                              |
> >>>>>     [base] <- [mid] <- (foo)  <---------------------- (bar)
> >>>>>
> >>>>> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
> >>>>> We can utilize an automatic hidden drive-backup target:
> >>>>>
> >>>>>                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
> >>>>>                          |                                                          |
> >>>>>                          |                                                          |
> >>>>>                          v                                                          v
> >>>>>
> >>>>>     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
> >>>>>
> >>>>>                          v                              ^
> >>>>>                          v                              ^
> >>>>>                          v                              ^
> >>>>>                          v                              ^
> >>>>>                          >>>> drive-backup sync=none >>>>
> >>>>>
> >>>>> So when guest writes to 'foo', the old data is moved to (hidden target), which
> >>>>> remains unchanged from (bar)'s PoV.
> >>>>>
> >>>>> The drive in the middle is called hidden because QEMU creates it automatically,
> >>>>> the naming is arbitrary.
> >>>>
> >>>> I don't understand this. In which function, the hidden target is created automatically?
> >>>>
> >>>
> >>> It's to be determined. This part is only in my mind :)
> >>
> >> What about this:
> >> -drive file=nbd-target,if=none,id=nbd-target0 \
> >> -drive file=active-disk,if=virtio,driver=qcow2,backing.file.filename=hidden-disk,backing.driver=qcow2,backing.backing=nbd-target0
> >>
> > 
> > It's close. I suppose backing.backing is referencing another drive as its
> > backing_hd, then you cannot have the other backing.file.* option - they
> > conflict. It would be something along:
> > 
> > -drive file=nbd-target,if=none,id=nbd-target0 \
> > -drive file=hidden-disk,if=none,id=hidden0,backing.backing=nbd-target0 \
> > -drive file=active-disk,if=virtio,driver=qcow2,backing.backing=hidden0
> > 
> > Or for simplicity, s/backing.backing=/backing=/g
> 
> If using backing=drive_id, backing.backing and backing.file.* are not conflict.
> backing.backing=$drive_id means that: backing file's backing file's id is $drive_id.

I see.

> 
> > 
> > Yes, adding these "backing=$drive_id" option is also exactly what we expect
> > in order to support image-fleecing, but we haven't figured how to allow that
> > without breaking other qmp operations like block jobs, etc.
> 
> I don't understand this. In which case, qmp operations will be broken? Can you give
> me some examples?
> 

I don't mean there is a fundamental stopper for this, but in order to relax the
assumption that "only top BDS can have a BlockBackend", we need to think
through the whole block layer, and add new finer checks/restrictions where it's
necessary, otherwise it will be a mess to allow arbitrary backing reference.

Some random questions I'm now aware of:

1. nbd-target0 is writable here, without the drive-backup, hidden0 could be
corrupted by writings to it. So there need to be a new convention and
invariance to follow.

2. in qmp, block-commit hidden0 to nbd-target0 or it's backing file, will
corrupt data (from nbd-target0's perspective).

3. unclear implications of "change" and "eject" when there is backing
reference.

4. can a drive be backing referenced by more than one other drives?

Just two cents, and I still need to think about it systematically.

Fam

Re: [Qemu-devel] [RFC PATCH 06/14] NBD client: connect to nbd server later

[Paolo Bonzini]

On 25/02/2015 15:22, Max Reitz wrote:
> 3. nbd_co_discard()
>     quorum doens't call bdrv_co_discard(), so it is OK to return -EIO here. 

That can change, so I think you should return -EIO either everywhere or
nowhere.  Which probably means nowhere.

> Hm, okay. How about adding an option to quorum for ignoring errors from
> a specific child? It's probably not possible to do something like
> "children.1.ignore-errors=true", but maybe you can just ignore errors in
> quorum from any but the first child if the read pattern is set to
> "first", that would make sense to me.
> 
> But if you don't want to do that, I guess just making NBD some kind of
> /dev/null before it's connected should be fine.

I think what Wen is doing is okay, especially since it's only the
special nbd+colo:// URIs that are acting as /dev/null.

Paolo

Re: [Qemu-devel] [RFC PATCH 07/14] NBD client: implement block driver interfaces for block replication

[Paolo Bonzini]

On 23/02/2015 22:41, Max Reitz wrote:
>>
>>   }
>>   +static int nbd_start_replication(BlockDriverState *bs, int mode)
>> +{
>> +    BDRVNBDState *s = bs->opaque;
>> +    Error *local_err = NULL;
>> +    int ret;
>> +
>> +    /*
>> +     * TODO: support COLO_SECONDARY_MODE if we allow secondary
>> +     * QEMU becoming primary QEMU.
>> +     */
>> +    if (mode != COLO_PRIMARY_MODE) {
>> +        return -1;
> 
> Once again, I'd like -ENOTSUP more (or -EINVAL or whatever you prefer).

Using the Error API is the right thing to do here.

Paolo

Re: [Qemu-devel] [RFC PATCH 11/14] allow writing to the backing file

[Paolo Bonzini]

On 23/02/2015 23:03, Max Reitz wrote:
> On 2015-02-11 at 22:07, Wen Congyang wrote:
>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
>> ---
>>   block.c | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> I don't think this is a good idea. With this patch, every time you open
> a COW file (with a backing file) R/W, the backing file will be writable.
> I'd rather like a way to explicitly overwrite the R/W mode of the
> backing file; but by default, in my opinion, it should stay read-only.

I agree.

Perhaps blkcolo_open or colo_svm_init can take care of setting
BDRV_O_RDWR on the backing file?  They could also use bdrv_reopen.

Paolo

> Max
> 
>> diff --git a/block.c b/block.c
>> index 067c44b..96cf973 100644
>> --- a/block.c
>> +++ b/block.c
>> @@ -856,8 +856,8 @@ static int bdrv_inherited_flags(int flags)
>>    */
>>   static int bdrv_backing_flags(int flags)
>>   {
>> -    /* backing files always opened read-only */
>> -    flags &= ~(BDRV_O_RDWR | BDRV_O_COPY_ON_READ);
>> +    /* backing files are opened read-write for block replication */
>> +    flags &= ~BDRV_O_COPY_ON_READ;
>>         /* snapshot=on is handled on the top layer */
>>       flags &= ~(BDRV_O_SNAPSHOT | BDRV_O_TEMPORARY);
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 02/26/2015 06:02 PM, Fam Zheng wrote:
> On Thu, 02/26 17:07, Wen Congyang wrote:
>> On 02/26/2015 04:44 PM, Fam Zheng wrote:
>>> On Thu, 02/26 14:38, Wen Congyang wrote:
>>>> On 02/25/2015 10:46 AM, Fam Zheng wrote:
>>>>> On Tue, 02/24 15:50, Wen Congyang wrote:
>>>>>> On 02/12/2015 04:44 PM, Fam Zheng wrote:
>>>>>>> On Thu, 02/12 15:40, Wen Congyang wrote:
>>>>>>>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
>>>>>>>>> Hi Congyang,
>>>>>>>>>
>>>>>>>>> On Thu, 02/12 11:07, Wen Congyang wrote:
>>>>>>>>>> +== Workflow ==
>>>>>>>>>> +The following is the image of block replication workflow:
>>>>>>>>>> +
>>>>>>>>>> +        +----------------------+            +------------------------+
>>>>>>>>>> +        |Primary Write Requests|            |Secondary Write Requests|
>>>>>>>>>> +        +----------------------+            +------------------------+
>>>>>>>>>> +                  |                                       |
>>>>>>>>>> +                  |                                      (4)
>>>>>>>>>> +                  |                                       V
>>>>>>>>>> +                  |                              /-------------\
>>>>>>>>>> +                  |      Copy and Forward        |             |
>>>>>>>>>> +                  |---------(1)----------+       | Disk Buffer |
>>>>>>>>>> +                  |                      |       |             |
>>>>>>>>>> +                  |                     (3)      \-------------/
>>>>>>>>>> +                  |                 speculative      ^
>>>>>>>>>> +                  |                write through    (2)
>>>>>>>>>> +                  |                      |           |
>>>>>>>>>> +                  V                      V           |
>>>>>>>>>> +           +--------------+           +----------------+
>>>>>>>>>> +           | Primary Disk |           | Secondary Disk |
>>>>>>>>>> +           +--------------+           +----------------+
>>>>>>>>>> +
>>>>>>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>>>>>>>> +       QEMU.
>>>>>>>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>>>>>>>> +       original sector content will be read from Secondary disk and
>>>>>>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>>>>>>>> +       sector content in the Disk buffer.
>>>>>>>>>
>>>>>>>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
>>>>>>>>> reading them as "s/will be/are/g"
>>>>>>>>>
>>>>>>>>> Why do you need this buffer?
>>>>>>>>
>>>>>>>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
>>>>>>>> vm write to the buffer.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> If both primary and secondary write to the same sector, what is saved in the
>>>>>>>>> buffer?
>>>>>>>>
>>>>>>>> The primary content will be written to the secondary disk, and the secondary content
>>>>>>>> is saved in the buffer.
>>>>>>>
>>>>>>> I wonder if alternatively this is possible with an imaginary "writable backing
>>>>>>> image" feature, as described below.
>>>>>>>
>>>>>>> When we have a normal backing chain,
>>>>>>>
>>>>>>>                {virtio-blk dev 'foo'}
>>>>>>>                          |
>>>>>>>                          |
>>>>>>>                          |
>>>>>>>     [base] <- [mid] <- (foo)
>>>>>>>
>>>>>>> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
>>>>>>> to an existing image on top,
>>>>>>>
>>>>>>>                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
>>>>>>>                          |                              |
>>>>>>>                          |                              |
>>>>>>>                          |                              |
>>>>>>>     [base] <- [mid] <- (foo)  <---------------------- (bar)
>>>>>>>
>>>>>>> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
>>>>>>> We can utilize an automatic hidden drive-backup target:
>>>>>>>
>>>>>>>                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
>>>>>>>                          |                                                          |
>>>>>>>                          |                                                          |
>>>>>>>                          v                                                          v
>>>>>>>
>>>>>>>     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
>>>>>>>
>>>>>>>                          v                              ^
>>>>>>>                          v                              ^
>>>>>>>                          v                              ^
>>>>>>>                          v                              ^
>>>>>>>                          >>>> drive-backup sync=none >>>>
>>>>>>>
>>>>>>> So when guest writes to 'foo', the old data is moved to (hidden target), which
>>>>>>> remains unchanged from (bar)'s PoV.
>>>>>>>
>>>>>>> The drive in the middle is called hidden because QEMU creates it automatically,
>>>>>>> the naming is arbitrary.
>>>>>>
>>>>>> I don't understand this. In which function, the hidden target is created automatically?
>>>>>>
>>>>>
>>>>> It's to be determined. This part is only in my mind :)
>>>>
>>>> What about this:
>>>> -drive file=nbd-target,if=none,id=nbd-target0 \
>>>> -drive file=active-disk,if=virtio,driver=qcow2,backing.file.filename=hidden-disk,backing.driver=qcow2,backing.backing=nbd-target0
>>>>
>>>
>>> It's close. I suppose backing.backing is referencing another drive as its
>>> backing_hd, then you cannot have the other backing.file.* option - they
>>> conflict. It would be something along:
>>>
>>> -drive file=nbd-target,if=none,id=nbd-target0 \
>>> -drive file=hidden-disk,if=none,id=hidden0,backing.backing=nbd-target0 \
>>> -drive file=active-disk,if=virtio,driver=qcow2,backing.backing=hidden0
>>>
>>> Or for simplicity, s/backing.backing=/backing=/g
>>
>> If using backing=drive_id, backing.backing and backing.file.* are not conflict.
>> backing.backing=$drive_id means that: backing file's backing file's id is $drive_id.
> 
> I see.
> 
>>
>>>
>>> Yes, adding these "backing=$drive_id" option is also exactly what we expect
>>> in order to support image-fleecing, but we haven't figured how to allow that
>>> without breaking other qmp operations like block jobs, etc.
>>
>> I don't understand this. In which case, qmp operations will be broken? Can you give
>> me some examples?
>>
> 
> I don't mean there is a fundamental stopper for this, but in order to relax the
> assumption that "only top BDS can have a BlockBackend", we need to think
> through the whole block layer, and add new finer checks/restrictions where it's
> necessary, otherwise it will be a mess to allow arbitrary backing reference.
> 
> Some random questions I'm now aware of:
> 
> 1. nbd-target0 is writable here, without the drive-backup, hidden0 could be
> corrupted by writings to it. So there need to be a new convention and
> invariance to follow.

Hmm, I understand while the hidden-disk should be opened automatically now.
If we use backing reference, I think we should open a hindden-disk, and set
drive backup automatically. Block any conflict operations(commit, change, eject?)

> 
> 2. in qmp, block-commit hidden0 to nbd-target0 or it's backing file, will
> corrupt data (from nbd-target0's perspective).
> 
> 3. unclear implications of "change" and "eject" when there is backing
> reference.
> 
> 4. can a drive be backing referenced by more than one other drives?

We can forbid it first.

Thanks
Wen Congyang

> 
> Just two cents, and I still need to think about it systematically.
> 
> Fam
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Fam Zheng]

On Fri, 02/27 10:27, Wen Congyang wrote:
> > 1. nbd-target0 is writable here, without the drive-backup, hidden0 could be
> > corrupted by writings to it. So there need to be a new convention and
> > invariance to follow.
> 
> Hmm, I understand while the hidden-disk should be opened automatically now.
> If we use backing reference, I think we should open a hindden-disk, and set
> drive backup automatically. Block any conflict operations(commit, change, eject?)

This might be a good idea.

> 
> > 
> > 2. in qmp, block-commit hidden0 to nbd-target0 or it's backing file, will
> > corrupt data (from nbd-target0's perspective).
> > 
> > 3. unclear implications of "change" and "eject" when there is backing
> > reference.
> > 
> > 4. can a drive be backing referenced by more than one other drives?
> 
> We can forbid it first.
> 

Yes, probably with a new op blocker type.

Fam

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 02/12/2015 06:26 PM, famz@redhat.com wrote:
> On Thu, 02/12 18:11, Wen Congyang wrote:
>> On 02/12/2015 05:44 PM, Fam Zheng wrote:
>>> On Thu, 02/12 17:33, Wen Congyang wrote:
>>>> On 02/12/2015 04:44 PM, Fam Zheng wrote:
>>>>> On Thu, 02/12 15:40, Wen Congyang wrote:
>>>>>> On 02/12/2015 03:21 PM, Fam Zheng wrote:
>>>>>>> Hi Congyang,
>>>>>>>
>>>>>>> On Thu, 02/12 11:07, Wen Congyang wrote:
>>>>>>>> +== Workflow ==
>>>>>>>> +The following is the image of block replication workflow:
>>>>>>>> +
>>>>>>>> +        +----------------------+            +------------------------+
>>>>>>>> +        |Primary Write Requests|            |Secondary Write Requests|
>>>>>>>> +        +----------------------+            +------------------------+
>>>>>>>> +                  |                                       |
>>>>>>>> +                  |                                      (4)
>>>>>>>> +                  |                                       V
>>>>>>>> +                  |                              /-------------\
>>>>>>>> +                  |      Copy and Forward        |             |
>>>>>>>> +                  |---------(1)----------+       | Disk Buffer |
>>>>>>>> +                  |                      |       |             |
>>>>>>>> +                  |                     (3)      \-------------/
>>>>>>>> +                  |                 speculative      ^
>>>>>>>> +                  |                write through    (2)
>>>>>>>> +                  |                      |           |
>>>>>>>> +                  V                      V           |
>>>>>>>> +           +--------------+           +----------------+
>>>>>>>> +           | Primary Disk |           | Secondary Disk |
>>>>>>>> +           +--------------+           +----------------+
>>>>>>>> +
>>>>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>>>>>> +       QEMU.
>>>>>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>>>>>> +       original sector content will be read from Secondary disk and
>>>>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>>>>>> +       sector content in the Disk buffer.
>>>>>>>
>>>>>>> I'm a little confused by the tenses ("will be" versus "are") and terms. I am
>>>>>>> reading them as "s/will be/are/g"
>>>>>>>
>>>>>>> Why do you need this buffer?
>>>>>>
>>>>>> We only sync the disk till next checkpoint. Before next checkpoint, secondary
>>>>>> vm write to the buffer.
>>>>>>
>>>>>>>
>>>>>>> If both primary and secondary write to the same sector, what is saved in the
>>>>>>> buffer?
>>>>>>
>>>>>> The primary content will be written to the secondary disk, and the secondary content
>>>>>> is saved in the buffer.
>>>>>
>>>>> I wonder if alternatively this is possible with an imaginary "writable backing
>>>>> image" feature, as described below.
>>>>>
>>>>> When we have a normal backing chain,
>>>>>
>>>>>                {virtio-blk dev 'foo'}
>>>>>                          |
>>>>>                          |
>>>>>                          |
>>>>>     [base] <- [mid] <- (foo)
>>>>>
>>>>> Where [base] and [mid] are read only, (foo) is writable. When we add an overlay
>>>>> to an existing image on top,
>>>>>
>>>>>                {virtio-blk dev 'foo'}        {virtio-blk dev 'bar'}
>>>>>                          |                              |
>>>>>                          |                              |
>>>>>                          |                              |
>>>>>     [base] <- [mid] <- (foo)  <---------------------- (bar)
>>>>>
>>>>> It's important to make sure that writes to 'foo' doesn't break data for 'bar'.
>>>>> We can utilize an automatic hidden drive-backup target:
>>>>>
>>>>>                {virtio-blk dev 'foo'}                                    {virtio-blk dev 'bar'}
>>>>>                          |                                                          |
>>>>>                          |                                                          |
>>>>>                          v                                                          v
>>>>>
>>>>>     [base] <- [mid] <- (foo)  <----------------- (hidden target) <--------------- (bar)
>>>>>
>>>>>                          v                              ^
>>>>>                          v                              ^
>>>>>                          v                              ^
>>>>>                          v                              ^
>>>>>                          >>>> drive-backup sync=none >>>>
>>>>>
>>>>> So when guest writes to 'foo', the old data is moved to (hidden target), which
>>>>> remains unchanged from (bar)'s PoV.
>>>>>
>>>>> The drive in the middle is called hidden because QEMU creates it automatically,
>>>>> the naming is arbitrary.
>>>>>
>>>>> It is interesting because it is a more generalized case of image fleecing,
>>>>> where the (hidden target) is exposed via NBD server for data scanning (read
>>>>> only) purpose.
>>>>>
>>>>> More interestingly, with above facility, it is also possible to create a guest
>>>>> visible live snapshot (disk 'bar') of an existing device (disk 'foo') very
>>>>> cheaply. Or call it shadow copy if you will.
>>>>>
>>>>> Back to the COLO case, the configuration will be very similar:
>>>>>
>>>>>
>>>>>                       {primary wr}                                                {secondary vm}
>>>>>                             |                                                           |
>>>>>                             |                                                           |
>>>>>                             |                                                           |
>>>>>                             v                                                           v
>>>>>
>>>>>    [what] <- [ever] <- (nbd target) <------------ (hidden buf disk) <------------- (active disk)
>>>>>
>>>>>                             v                              ^
>>>>>                             v                              ^
>>>>>                             v                              ^
>>>>>                             v                              ^
>>>>>                             >>>> drive-backup sync=none >>>>
>>>>
>>>> What is active disk? There are two disk images?
>>>
>>> It starts as an empty image with (hidden buf disk) as backing file, which in
>>> turn has (nbd target) as backing file.
>>
>> It's too complicated..., and I don't understand it.
>> 1. What is active disk? Use raw or a new block driver?
> 
> It is an empty qcow2 image with the same lenght as your Secondary Disk.

I test qcow2_make_empty()'s performance. The result shows that it may
take about 100ms(normal sata disk). It is not acceptable for COLO. So
I think disk buff is necessary(just use it to replace qcow2).

Thanks
Wen Congyang

> 
>> 2. Hidden buf disk use new block driver?
> 
> It is an empty qcow2 image with the same lenght as your Secondary Disk, too.
> 
>> 3. nbd target is hidden buf disk's backing image? If it is opened read-only, we will
>>    export a nbd with read-only BlockDriverState, but nbd server needs to write it.
> 
> NBD target is your Secondary Disk. It is opened read-write.
> 
> The patches to enable opening it as read-write, and starting drive-backup
> between it and hidden buf disk, are all work in progress (the core concept) of
> image fleecing.
> 
> Fam
> 
>>>>>
>>>>> The workflow analogue is:
>>>>>
>>>>>>>> +    1) Primary write requests will be copied and forwarded to Secondary
>>>>>>>> +       QEMU.
>>>>>
>>>>> Primary write requests are forwarded to secondary QEMU as well.
>>>>>
>>>>>>>> +    2) Before Primary write requests are written to Secondary disk, the
>>>>>>>> +       original sector content will be read from Secondary disk and
>>>>>>>> +       buffered in the Disk buffer, but it will not overwrite the existing
>>>>>>>> +       sector content in the Disk buffer.
>>>>>
>>>>> Before Primary write requests are written to (nbd target), aka the Secondary
>>>>> disk, the orignal sector content is read from it and copied to (hidden buf
>>>>> disk) by drive-backup. It obviously will not overwrite the data in (active
>>>>> disk).
>>>>>
>>>>>>>> +    3) Primary write requests will be written to Secondary disk.
>>>>>
>>>>> Primary write requests are written to (nbd target).
>>>>>
>>>>>>>> +    4) Secondary write requests will be buffered in the Disk buffer and it
>>>>>>>> +       will overwrite the existing sector content in the buffer.
>>>>>
>>>>> Secondary write request will be written in (active disk) as usual.
>>>>>
>>>>> Finally, when checkpoint arrives, if you want to sync with primary, just drop
>>>>> data in (hidden buf disk) and (active disk); when failover happends, if you
>>>>> want to promote secondary vm, you can commit (active disk) to (nbd target), and
>>>>> drop data in (hidden buf disk).
>>>>>
>>>>> Fam
>>>>> .
>>>>>
>>>>
>>>>
>>> .
>>>
>>
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Fam Zheng]

On Tue, 03/03 15:53, Wen Congyang wrote:
> I test qcow2_make_empty()'s performance. The result shows that it may
> take about 100ms(normal sata disk). It is not acceptable for COLO. So
> I think disk buff is necessary(just use it to replace qcow2).

Why not tmpfs or ramdisk?

Fam

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 03/03/2015 03:59 PM, Fam Zheng wrote:
> On Tue, 03/03 15:53, Wen Congyang wrote:
>> I test qcow2_make_empty()'s performance. The result shows that it may
>> take about 100ms(normal sata disk). It is not acceptable for COLO. So
>> I think disk buff is necessary(just use it to replace qcow2).
> 
> Why not tmpfs or ramdisk?

I test it, and it only takes 2-3ms.

Thanks
Wen Congyang

> 
> Fam
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Dr. David Alan Gilbert]

* Wen Congyang (wency@cn.fujitsu.com) wrote:
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>

Hi,

> ---
>  docs/block-replication.txt | 129 +++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 129 insertions(+)
>  create mode 100644 docs/block-replication.txt
> 
> diff --git a/docs/block-replication.txt b/docs/block-replication.txt
> new file mode 100644
> index 0000000..59150b8
> --- /dev/null
> +++ b/docs/block-replication.txt
> @@ -0,0 +1,129 @@
> +Block replication
> +----------------------------------------
> +Copyright Fujitsu, Corp. 2015
> +Copyright (c) 2015 Intel Corporation
> +Copyright (c) 2015 HUAWEI TECHNOLOGIES CO.,LTD.
> +
> +This work is licensed under the terms of the GNU GPL, version 2 or later.
> +See the COPYING file in the top-level directory.
> +
> +The block replication is used for continuous checkpoints. It is designed
> +for COLO that Secondary VM is running. It can also be applied for FT/HA
> +scene that Secondary VM is not running.
> +
> +This document gives an overview of block replication's design.
> +
> +== Background ==
> +High availability solutions such as micro checkpoint and COLO will do
> +consecutive checkpoint. The VM state of Primary VM and Secondary VM is
> +identical right after a VM checkpoint, but becomes different as the VM
> +executes till the next checkpoint. To support disk contents checkpoint,
> +the modified disk contents in the Secondary VM must be buffered, and are
> +only dropped at next checkpoint time. To reduce the network transportation
> +effort at the time of checkpoint, the disk modification operations of
> +Primary disk are asynchronously forwarded to the Secondary node.

Can you explain how the block data is synchronised with the main checkpoint
stream?  i.e. when the secondary receives a new checkpoint how does it know
it's received all of the block writes from the primary associated with that
checkpoint and that all the following writes that it receives are for the
next checkpoint period?

Dave

> +
> +== Workflow ==
> +The following is the image of block replication workflow:
> +
> +        +----------------------+            +------------------------+
> +        |Primary Write Requests|            |Secondary Write Requests|
> +        +----------------------+            +------------------------+
> +                  |                                       |
> +                  |                                      (4)
> +                  |                                       V
> +                  |                              /-------------\
> +                  |      Copy and Forward        |             |
> +                  |---------(1)----------+       | Disk Buffer |
> +                  |                      |       |             |
> +                  |                     (3)      \-------------/
> +                  |                 speculative      ^
> +                  |                write through    (2)
> +                  |                      |           |
> +                  V                      V           |
> +           +--------------+           +----------------+
> +           | Primary Disk |           | Secondary Disk |
> +           +--------------+           +----------------+
> +
> +    1) Primary write requests will be copied and forwarded to Secondary
> +       QEMU.
> +    2) Before Primary write requests are written to Secondary disk, the
> +       original sector content will be read from Secondary disk and
> +       buffered in the Disk buffer, but it will not overwrite the existing
> +       sector content in the Disk buffer.
> +    3) Primary write requests will be written to Secondary disk.
> +    4) Secondary write requests will be buffered in the Disk buffer and it
> +       will overwrite the existing sector content in the buffer.
> +
> +== Architecture ==
> +We are going to implement COLO block replication from many basic
> +blocks that are already in QEMU.
> +
> +         virtio-blk       ||
> +             ^            ||                            .----------
> +             |            ||                            | Secondary
> +        1 Quorum          ||                            '----------
> +         /      \         ||
> +        /        \        ||
> +   Primary      2 NBD  ------->  2 NBD
> +     disk       client    ||     server                  virtio-blk
> +                          ||        ^                         ^
> +--------.                 ||        |                         |
> +Primary |                 ||  Secondary disk <--------- COLO buffer 3
> +--------'                 ||                   backing
> +
> +1) The disk on the primary is represented by a block device with two
> +children, providing replication between a primary disk and the host that
> +runs the secondary VM. The read pattern for quorum can be extended to
> +make the primary always read from the local disk instead of going through
> +NBD.
> +
> +2) The secondary disk receives writes from the primary VM through QEMU's
> +embedded NBD server (speculative write-through).
> +
> +3) The disk on the secondary is represented by a custom block device
> +("COLO buffer"). The disk buffer's backing image is the secondary disk,
> +and the disk buffer uses bdrv_add_before_write_notifier to implement
> +copy-on-write, similar to block/backup.c.
> +
> +== New block driver interface ==
> +We add three block driver interfaces to control block replication:
> +a. bdrv_start_replication()
> +   Start block replication, called in migration/checkpoint thread.
> +   We must call bdrv_start_replication() in secondary QEMU before
> +   calling bdrv_start_replication() in primary QEMU.
> +b. bdrv_do_checkpoint()
> +   This interface is called after all VM state is transfered to
> +   Secondary QEMU. The Disk buffer will be dropped in this interface.
> +c. bdrv_stop_replication()
> +   It is called when failover. We will flush the Disk buffer into
> +   Secondary Disk and stop block replication.
> +
> +== Usage ==
> +Primary:
> +  -drive if=xxx,driver=quorum,read-pattern=first,\
> +         children.0.file.filename=1.raw,\
> +         children.0.driver=raw,\
> +         children.1.file.driver=nbd+colo,\
> +         children.1.file.host=xxx,\
> +         children.1.file.port=xxx,\
> +         children.1.file.export=xxx,\
> +         children.1.driver=raw
> +  Note:
> +  1. NBD Client should not be the first child of quorum.
> +  2. There should be only one NBD Client.
> +  3. host is the secondary physical machine's hostname or IP
> +  4. Each disk must have its own export name.
> +
> +Secondary:
> +  -drive if=xxx,driver=blkcolo,export=xxx,\
> +         backing.file.filename=1.raw,\
> +         backing.driver=raw
> +  Then run qmp command:
> +    nbd_server_start host:port
> +  Note:
> +  1. The export name for the same disk must be the same in primary
> +     and secondary QEMU command line
> +  2. The qmp command nbd_server_start must be run before running the
> +     qmp command migrate on primary QEMU
> +  3. Don't use nbd_server_start's other options
> -- 
> 2.1.0
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 03/05/2015 12:35 AM, Dr. David Alan Gilbert wrote:
> * Wen Congyang (wency@cn.fujitsu.com) wrote:
>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> 
> Hi,
> 
>> ---
>>  docs/block-replication.txt | 129 +++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 129 insertions(+)
>>  create mode 100644 docs/block-replication.txt
>>
>> diff --git a/docs/block-replication.txt b/docs/block-replication.txt
>> new file mode 100644
>> index 0000000..59150b8
>> --- /dev/null
>> +++ b/docs/block-replication.txt
>> @@ -0,0 +1,129 @@
>> +Block replication
>> +----------------------------------------
>> +Copyright Fujitsu, Corp. 2015
>> +Copyright (c) 2015 Intel Corporation
>> +Copyright (c) 2015 HUAWEI TECHNOLOGIES CO.,LTD.
>> +
>> +This work is licensed under the terms of the GNU GPL, version 2 or later.
>> +See the COPYING file in the top-level directory.
>> +
>> +The block replication is used for continuous checkpoints. It is designed
>> +for COLO that Secondary VM is running. It can also be applied for FT/HA
>> +scene that Secondary VM is not running.
>> +
>> +This document gives an overview of block replication's design.
>> +
>> +== Background ==
>> +High availability solutions such as micro checkpoint and COLO will do
>> +consecutive checkpoint. The VM state of Primary VM and Secondary VM is
>> +identical right after a VM checkpoint, but becomes different as the VM
>> +executes till the next checkpoint. To support disk contents checkpoint,
>> +the modified disk contents in the Secondary VM must be buffered, and are
>> +only dropped at next checkpoint time. To reduce the network transportation
>> +effort at the time of checkpoint, the disk modification operations of
>> +Primary disk are asynchronously forwarded to the Secondary node.
> 
> Can you explain how the block data is synchronised with the main checkpoint
> stream?  i.e. when the secondary receives a new checkpoint how does it know
> it's received all of the block writes from the primary associated with that
> checkpoint and that all the following writes that it receives are for the
> next checkpoint period?

NBD server will do it. Writing to NBD client will return after NBD server replies
the result(ACK or error).

Thanks
Wen Congyang

> 
> Dave
> 
>> +
>> +== Workflow ==
>> +The following is the image of block replication workflow:
>> +
>> +        +----------------------+            +------------------------+
>> +        |Primary Write Requests|            |Secondary Write Requests|
>> +        +----------------------+            +------------------------+
>> +                  |                                       |
>> +                  |                                      (4)
>> +                  |                                       V
>> +                  |                              /-------------\
>> +                  |      Copy and Forward        |             |
>> +                  |---------(1)----------+       | Disk Buffer |
>> +                  |                      |       |             |
>> +                  |                     (3)      \-------------/
>> +                  |                 speculative      ^
>> +                  |                write through    (2)
>> +                  |                      |           |
>> +                  V                      V           |
>> +           +--------------+           +----------------+
>> +           | Primary Disk |           | Secondary Disk |
>> +           +--------------+           +----------------+
>> +
>> +    1) Primary write requests will be copied and forwarded to Secondary
>> +       QEMU.
>> +    2) Before Primary write requests are written to Secondary disk, the
>> +       original sector content will be read from Secondary disk and
>> +       buffered in the Disk buffer, but it will not overwrite the existing
>> +       sector content in the Disk buffer.
>> +    3) Primary write requests will be written to Secondary disk.
>> +    4) Secondary write requests will be buffered in the Disk buffer and it
>> +       will overwrite the existing sector content in the buffer.
>> +
>> +== Architecture ==
>> +We are going to implement COLO block replication from many basic
>> +blocks that are already in QEMU.
>> +
>> +         virtio-blk       ||
>> +             ^            ||                            .----------
>> +             |            ||                            | Secondary
>> +        1 Quorum          ||                            '----------
>> +         /      \         ||
>> +        /        \        ||
>> +   Primary      2 NBD  ------->  2 NBD
>> +     disk       client    ||     server                  virtio-blk
>> +                          ||        ^                         ^
>> +--------.                 ||        |                         |
>> +Primary |                 ||  Secondary disk <--------- COLO buffer 3
>> +--------'                 ||                   backing
>> +
>> +1) The disk on the primary is represented by a block device with two
>> +children, providing replication between a primary disk and the host that
>> +runs the secondary VM. The read pattern for quorum can be extended to
>> +make the primary always read from the local disk instead of going through
>> +NBD.
>> +
>> +2) The secondary disk receives writes from the primary VM through QEMU's
>> +embedded NBD server (speculative write-through).
>> +
>> +3) The disk on the secondary is represented by a custom block device
>> +("COLO buffer"). The disk buffer's backing image is the secondary disk,
>> +and the disk buffer uses bdrv_add_before_write_notifier to implement
>> +copy-on-write, similar to block/backup.c.
>> +
>> +== New block driver interface ==
>> +We add three block driver interfaces to control block replication:
>> +a. bdrv_start_replication()
>> +   Start block replication, called in migration/checkpoint thread.
>> +   We must call bdrv_start_replication() in secondary QEMU before
>> +   calling bdrv_start_replication() in primary QEMU.
>> +b. bdrv_do_checkpoint()
>> +   This interface is called after all VM state is transfered to
>> +   Secondary QEMU. The Disk buffer will be dropped in this interface.
>> +c. bdrv_stop_replication()
>> +   It is called when failover. We will flush the Disk buffer into
>> +   Secondary Disk and stop block replication.
>> +
>> +== Usage ==
>> +Primary:
>> +  -drive if=xxx,driver=quorum,read-pattern=first,\
>> +         children.0.file.filename=1.raw,\
>> +         children.0.driver=raw,\
>> +         children.1.file.driver=nbd+colo,\
>> +         children.1.file.host=xxx,\
>> +         children.1.file.port=xxx,\
>> +         children.1.file.export=xxx,\
>> +         children.1.driver=raw
>> +  Note:
>> +  1. NBD Client should not be the first child of quorum.
>> +  2. There should be only one NBD Client.
>> +  3. host is the secondary physical machine's hostname or IP
>> +  4. Each disk must have its own export name.
>> +
>> +Secondary:
>> +  -drive if=xxx,driver=blkcolo,export=xxx,\
>> +         backing.file.filename=1.raw,\
>> +         backing.driver=raw
>> +  Then run qmp command:
>> +    nbd_server_start host:port
>> +  Note:
>> +  1. The export name for the same disk must be the same in primary
>> +     and secondary QEMU command line
>> +  2. The qmp command nbd_server_start must be run before running the
>> +     qmp command migrate on primary QEMU
>> +  3. Don't use nbd_server_start's other options
>> -- 
>> 2.1.0
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Dr. David Alan Gilbert]

* Wen Congyang (wency@cn.fujitsu.com) wrote:
> On 03/05/2015 12:35 AM, Dr. David Alan Gilbert wrote:
> > * Wen Congyang (wency@cn.fujitsu.com) wrote:
> >> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> >> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> >> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
> >> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> >> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> > 
> > Hi,
> > 
> >> ---
> >>  docs/block-replication.txt | 129 +++++++++++++++++++++++++++++++++++++++++++++
> >>  1 file changed, 129 insertions(+)
> >>  create mode 100644 docs/block-replication.txt
> >>
> >> diff --git a/docs/block-replication.txt b/docs/block-replication.txt
> >> new file mode 100644
> >> index 0000000..59150b8
> >> --- /dev/null
> >> +++ b/docs/block-replication.txt
> >> @@ -0,0 +1,129 @@
> >> +Block replication
> >> +----------------------------------------
> >> +Copyright Fujitsu, Corp. 2015
> >> +Copyright (c) 2015 Intel Corporation
> >> +Copyright (c) 2015 HUAWEI TECHNOLOGIES CO.,LTD.
> >> +
> >> +This work is licensed under the terms of the GNU GPL, version 2 or later.
> >> +See the COPYING file in the top-level directory.
> >> +
> >> +The block replication is used for continuous checkpoints. It is designed
> >> +for COLO that Secondary VM is running. It can also be applied for FT/HA
> >> +scene that Secondary VM is not running.
> >> +
> >> +This document gives an overview of block replication's design.
> >> +
> >> +== Background ==
> >> +High availability solutions such as micro checkpoint and COLO will do
> >> +consecutive checkpoint. The VM state of Primary VM and Secondary VM is
> >> +identical right after a VM checkpoint, but becomes different as the VM
> >> +executes till the next checkpoint. To support disk contents checkpoint,
> >> +the modified disk contents in the Secondary VM must be buffered, and are
> >> +only dropped at next checkpoint time. To reduce the network transportation
> >> +effort at the time of checkpoint, the disk modification operations of
> >> +Primary disk are asynchronously forwarded to the Secondary node.
> > 
> > Can you explain how the block data is synchronised with the main checkpoint
> > stream?  i.e. when the secondary receives a new checkpoint how does it know
> > it's received all of the block writes from the primary associated with that
> > checkpoint and that all the following writes that it receives are for the
> > next checkpoint period?
> 
> NBD server will do it. Writing to NBD client will return after NBD server replies
> the result(ACK or error).

Ah OK, so if the NBD client is synchronous then yes I can see that;
(I was confused by the word 'asynchronously' in your description above
but I guess that means asynchronous to the checkpoint stream).
I see that 'do_colo_transaction' keeps the primary stopped until after
the secondary does blk_do_checkpoint and then sends 'LOADED'.

I think yes that should work; although potentially you could make it faster;
since the primary doesn't need to know that it's write has been commited
until the next checkpoint, and if you could mark the separation in the two
checkpoints, then you could start the primary running again earlier.  But that's
all more complicated; this should work OK.

Thanks for the explanation,

Dave

> Thanks
> Wen Congyang
> 
> > 
> > Dave
> > 
> >> +
> >> +== Workflow ==
> >> +The following is the image of block replication workflow:
> >> +
> >> +        +----------------------+            +------------------------+
> >> +        |Primary Write Requests|            |Secondary Write Requests|
> >> +        +----------------------+            +------------------------+
> >> +                  |                                       |
> >> +                  |                                      (4)
> >> +                  |                                       V
> >> +                  |                              /-------------\
> >> +                  |      Copy and Forward        |             |
> >> +                  |---------(1)----------+       | Disk Buffer |
> >> +                  |                      |       |             |
> >> +                  |                     (3)      \-------------/
> >> +                  |                 speculative      ^
> >> +                  |                write through    (2)
> >> +                  |                      |           |
> >> +                  V                      V           |
> >> +           +--------------+           +----------------+
> >> +           | Primary Disk |           | Secondary Disk |
> >> +           +--------------+           +----------------+
> >> +
> >> +    1) Primary write requests will be copied and forwarded to Secondary
> >> +       QEMU.
> >> +    2) Before Primary write requests are written to Secondary disk, the
> >> +       original sector content will be read from Secondary disk and
> >> +       buffered in the Disk buffer, but it will not overwrite the existing
> >> +       sector content in the Disk buffer.
> >> +    3) Primary write requests will be written to Secondary disk.
> >> +    4) Secondary write requests will be buffered in the Disk buffer and it
> >> +       will overwrite the existing sector content in the buffer.
> >> +
> >> +== Architecture ==
> >> +We are going to implement COLO block replication from many basic
> >> +blocks that are already in QEMU.
> >> +
> >> +         virtio-blk       ||
> >> +             ^            ||                            .----------
> >> +             |            ||                            | Secondary
> >> +        1 Quorum          ||                            '----------
> >> +         /      \         ||
> >> +        /        \        ||
> >> +   Primary      2 NBD  ------->  2 NBD
> >> +     disk       client    ||     server                  virtio-blk
> >> +                          ||        ^                         ^
> >> +--------.                 ||        |                         |
> >> +Primary |                 ||  Secondary disk <--------- COLO buffer 3
> >> +--------'                 ||                   backing
> >> +
> >> +1) The disk on the primary is represented by a block device with two
> >> +children, providing replication between a primary disk and the host that
> >> +runs the secondary VM. The read pattern for quorum can be extended to
> >> +make the primary always read from the local disk instead of going through
> >> +NBD.
> >> +
> >> +2) The secondary disk receives writes from the primary VM through QEMU's
> >> +embedded NBD server (speculative write-through).
> >> +
> >> +3) The disk on the secondary is represented by a custom block device
> >> +("COLO buffer"). The disk buffer's backing image is the secondary disk,
> >> +and the disk buffer uses bdrv_add_before_write_notifier to implement
> >> +copy-on-write, similar to block/backup.c.
> >> +
> >> +== New block driver interface ==
> >> +We add three block driver interfaces to control block replication:
> >> +a. bdrv_start_replication()
> >> +   Start block replication, called in migration/checkpoint thread.
> >> +   We must call bdrv_start_replication() in secondary QEMU before
> >> +   calling bdrv_start_replication() in primary QEMU.
> >> +b. bdrv_do_checkpoint()
> >> +   This interface is called after all VM state is transfered to
> >> +   Secondary QEMU. The Disk buffer will be dropped in this interface.
> >> +c. bdrv_stop_replication()
> >> +   It is called when failover. We will flush the Disk buffer into
> >> +   Secondary Disk and stop block replication.
> >> +
> >> +== Usage ==
> >> +Primary:
> >> +  -drive if=xxx,driver=quorum,read-pattern=first,\
> >> +         children.0.file.filename=1.raw,\
> >> +         children.0.driver=raw,\
> >> +         children.1.file.driver=nbd+colo,\
> >> +         children.1.file.host=xxx,\
> >> +         children.1.file.port=xxx,\
> >> +         children.1.file.export=xxx,\
> >> +         children.1.driver=raw
> >> +  Note:
> >> +  1. NBD Client should not be the first child of quorum.
> >> +  2. There should be only one NBD Client.
> >> +  3. host is the secondary physical machine's hostname or IP
> >> +  4. Each disk must have its own export name.
> >> +
> >> +Secondary:
> >> +  -drive if=xxx,driver=blkcolo,export=xxx,\
> >> +         backing.file.filename=1.raw,\
> >> +         backing.driver=raw
> >> +  Then run qmp command:
> >> +    nbd_server_start host:port
> >> +  Note:
> >> +  1. The export name for the same disk must be the same in primary
> >> +     and secondary QEMU command line
> >> +  2. The qmp command nbd_server_start must be run before running the
> >> +     qmp command migrate on primary QEMU
> >> +  3. Don't use nbd_server_start's other options
> >> -- 
> >> 2.1.0
> >>
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > .
> > 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 03/03/2015 03:59 PM, Fam Zheng wrote:
> On Tue, 03/03 15:53, Wen Congyang wrote:
>> I test qcow2_make_empty()'s performance. The result shows that it may
>> take about 100ms(normal sata disk). It is not acceptable for COLO. So
>> I think disk buff is necessary(just use it to replace qcow2).
> 
> Why not tmpfs or ramdisk?

Another problem:
After failover, secondary write request will be written in (active disk)?
It is better to write request to (nbd target). Is there any feature can
be reused to implement it?

Thanks
Wen Congyang

> 
> Fam
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Fam Zheng]

On Wed, 03/11 14:44, Wen Congyang wrote:
> On 03/03/2015 03:59 PM, Fam Zheng wrote:
> > On Tue, 03/03 15:53, Wen Congyang wrote:
> >> I test qcow2_make_empty()'s performance. The result shows that it may
> >> take about 100ms(normal sata disk). It is not acceptable for COLO. So
> >> I think disk buff is necessary(just use it to replace qcow2).
> > 
> > Why not tmpfs or ramdisk?
> 
> Another problem:
> After failover, secondary write request will be written in (active disk)?
> It is better to write request to (nbd target). Is there any feature can
> be reused to implement it?

You can use block commit or stream to move the data.

Fam

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 03/11/2015 02:49 PM, Fam Zheng wrote:
> On Wed, 03/11 14:44, Wen Congyang wrote:
>> On 03/03/2015 03:59 PM, Fam Zheng wrote:
>>> On Tue, 03/03 15:53, Wen Congyang wrote:
>>>> I test qcow2_make_empty()'s performance. The result shows that it may
>>>> take about 100ms(normal sata disk). It is not acceptable for COLO. So
>>>> I think disk buff is necessary(just use it to replace qcow2).
>>>
>>> Why not tmpfs or ramdisk?
>>
>> Another problem:
>> After failover, secondary write request will be written in (active disk)?
>> It is better to write request to (nbd target). Is there any feature can
>> be reused to implement it?
> 
> You can use block commit or stream to move the data.

When doing failover, we can use it to move the data. After failover,
I need an endless job to move the data.

Thanks
Wen Congyang

> 
> Fam
> 
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Fam Zheng]

On Wed, 03/11 15:01, Wen Congyang wrote:
> On 03/11/2015 02:49 PM, Fam Zheng wrote:
> > On Wed, 03/11 14:44, Wen Congyang wrote:
> >> On 03/03/2015 03:59 PM, Fam Zheng wrote:
> >>> On Tue, 03/03 15:53, Wen Congyang wrote:
> >>>> I test qcow2_make_empty()'s performance. The result shows that it may
> >>>> take about 100ms(normal sata disk). It is not acceptable for COLO. So
> >>>> I think disk buff is necessary(just use it to replace qcow2).
> >>>
> >>> Why not tmpfs or ramdisk?
> >>
> >> Another problem:
> >> After failover, secondary write request will be written in (active disk)?
> >> It is better to write request to (nbd target). Is there any feature can
> >> be reused to implement it?
> > 
> > You can use block commit or stream to move the data.
> 
> When doing failover, we can use it to move the data. After failover,
> I need an endless job to move the data.
> 

I see what you mean. After failover, does the nbd server receive more data
(i.e. do you need a buffer to stash data from the other side)? If you commit
(active disk) to (nbd target), all the writes will go to a single image.

Fam

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 03/11/2015 03:04 PM, Fam Zheng wrote:
> On Wed, 03/11 15:01, Wen Congyang wrote:
>> On 03/11/2015 02:49 PM, Fam Zheng wrote:
>>> On Wed, 03/11 14:44, Wen Congyang wrote:
>>>> On 03/03/2015 03:59 PM, Fam Zheng wrote:
>>>>> On Tue, 03/03 15:53, Wen Congyang wrote:
>>>>>> I test qcow2_make_empty()'s performance. The result shows that it may
>>>>>> take about 100ms(normal sata disk). It is not acceptable for COLO. So
>>>>>> I think disk buff is necessary(just use it to replace qcow2).
>>>>>
>>>>> Why not tmpfs or ramdisk?
>>>>
>>>> Another problem:
>>>> After failover, secondary write request will be written in (active disk)?
>>>> It is better to write request to (nbd target). Is there any feature can
>>>> be reused to implement it?
>>>
>>> You can use block commit or stream to move the data.
>>
>> When doing failover, we can use it to move the data. After failover,
>> I need an endless job to move the data.
>>
> 
> I see what you mean. After failover, does the nbd server receive more data
> (i.e. do you need a buffer to stash data from the other side)? If you commit
> (active disk) to (nbd target), all the writes will go to a single image.

After failover(primary host downs), only secondary qemu works, and nbd server
doesn't receive any more data.

Thanks
Wen Congyang

> 
> Fam
> 
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 03/11/2015 02:49 PM, Fam Zheng wrote:
> On Wed, 03/11 14:44, Wen Congyang wrote:
>> On 03/03/2015 03:59 PM, Fam Zheng wrote:
>>> On Tue, 03/03 15:53, Wen Congyang wrote:
>>>> I test qcow2_make_empty()'s performance. The result shows that it may
>>>> take about 100ms(normal sata disk). It is not acceptable for COLO. So
>>>> I think disk buff is necessary(just use it to replace qcow2).
>>>
>>> Why not tmpfs or ramdisk?
>>
>> Another problem:
>> After failover, secondary write request will be written in (active disk)?
>> It is better to write request to (nbd target). Is there any feature can
>> be reused to implement it?
> 
> You can use block commit or stream to move the data.

Can the job stream move the data? I don't find the write ops in block/stream.c.

Thanks
Wen Congyang

> 
> Fam
> 
> .
> 

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Fam Zheng]

On Fri, 03/13 17:01, Wen Congyang wrote:
> On 03/11/2015 02:49 PM, Fam Zheng wrote:
> > On Wed, 03/11 14:44, Wen Congyang wrote:
> >> On 03/03/2015 03:59 PM, Fam Zheng wrote:
> >>> On Tue, 03/03 15:53, Wen Congyang wrote:
> >>>> I test qcow2_make_empty()'s performance. The result shows that it may
> >>>> take about 100ms(normal sata disk). It is not acceptable for COLO. So
> >>>> I think disk buff is necessary(just use it to replace qcow2).
> >>>
> >>> Why not tmpfs or ramdisk?
> >>
> >> Another problem:
> >> After failover, secondary write request will be written in (active disk)?
> >> It is better to write request to (nbd target). Is there any feature can
> >> be reused to implement it?
> > 
> > You can use block commit or stream to move the data.
> 
> Can the job stream move the data? I don't find the write ops in block/stream.c.

It is bdrv_co_copy_on_readv that moves data.

Fam

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Wen Congyang]

On 03/13/2015 05:05 PM, Fam Zheng wrote:
> On Fri, 03/13 17:01, Wen Congyang wrote:
>> On 03/11/2015 02:49 PM, Fam Zheng wrote:
>>> On Wed, 03/11 14:44, Wen Congyang wrote:
>>>> On 03/03/2015 03:59 PM, Fam Zheng wrote:
>>>>> On Tue, 03/03 15:53, Wen Congyang wrote:
>>>>>> I test qcow2_make_empty()'s performance. The result shows that it may
>>>>>> take about 100ms(normal sata disk). It is not acceptable for COLO. So
>>>>>> I think disk buff is necessary(just use it to replace qcow2).
>>>>>
>>>>> Why not tmpfs or ramdisk?
>>>>
>>>> Another problem:
>>>> After failover, secondary write request will be written in (active disk)?
>>>> It is better to write request to (nbd target). Is there any feature can
>>>> be reused to implement it?
>>>
>>> You can use block commit or stream to move the data.
>>
>> Can the job stream move the data? I don't find the write ops in block/stream.c.
> 
> It is bdrv_co_copy_on_readv that moves data.

Does the job stream move the data from base to top?

Thanks
Wen Congyang

> 
> Fam
> .
> 

Re: [Qemu-devel] [RFC PATCH 03/14] quorum: ignore 0-length child

[Wen Congyang]

On 02/24/2015 04:43 AM, Max Reitz wrote:
> On 2015-02-11 at 22:07, Wen Congyang wrote:
>> We connect to NBD server when starting block replication, so
>> the length is 0 before starting block replication.
>>
>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
>> ---
>>   block/quorum.c | 5 +++++
>>   1 file changed, 5 insertions(+)
>>
>> diff --git a/block/quorum.c b/block/quorum.c
>> index 5ed1ff8..e6aff5f 100644
>> --- a/block/quorum.c
>> +++ b/block/quorum.c
>> @@ -734,6 +734,11 @@ static int64_t quorum_getlength(BlockDriverState *bs)
>>           if (value < 0) {
>>               return value;
>>           }
>> +
>> +        if (!value) {
>> +            continue;
>> +        }
>> +
>>           if (value != result) {
>>               return -EIO;
>>           }
> 
> Hm, what do you think about some specific error value returned by your delayed NBD implementation? Like -ENOTCONN or something like that? Then we'd be able to discern a real 0-length block device from a not-yet-connected NBD server.

In my newest test, it cannot return -ENOTCONN, otherwise bdrv_open() will fail when we open the child.
Here is the backtrace:
(gdb) bt
#0  bdrv_open_common (bs=0x5555563b2230, file=0x0, options=0x5555563b55a0, flags=57410, drv=0x555555e90c00, errp=0x7fffffffd460) at block.c:1070
#1  0x000055555595ea72 in bdrv_open (pbs=0x7fffffffd5f8, filename=0x0, reference=0x0, options=0x5555563b55a0, flags=57410, drv=0x555555e90c00, errp=0x7fffffffd5e0) at block.c:1677
#2  0x000055555595e3a9 in bdrv_open_image (pbs=0x7fffffffd5f8, filename=0x0, options=0x5555563a6730, bdref_key=0x555555a86b4c "file", flags=57410, allow_none=true, errp=0x7fffffffd5e0) at block.c:1481
#3  0x000055555595e9ae in bdrv_open (pbs=0x555556388008, filename=0x0, reference=0x0, options=0x5555563a6730, flags=8258, drv=0x555555e8b800, errp=0x7fffffffd6b8) at block.c:1655
#4  0x00005555559b0058 in quorum_open (bs=0x55555639bd90, options=0x55555639f100, flags=8258, errp=0x7fffffffd758) at block/quorum.c:1000
#5  0x000055555595d0b8 in bdrv_open_common (bs=0x55555639bd90, file=0x0, options=0x55555639f100, flags=8258, drv=0x555555e8e5c0, errp=0x7fffffffd840) at block.c:1045
#6  0x000055555595ea72 in bdrv_open (pbs=0x55555639bd50, filename=0x0, reference=0x0, options=0x55555639f100, flags=8258, drv=0x555555e8e5c0, errp=0x7fffffffdb70) at block.c:1677
#7  0x00005555559b3bd3 in blk_new_open (name=0x55555639bc60 "virtio0", filename=0x0, reference=0x0, options=0x55555639a420, flags=66, errp=0x7fffffffdb70) at block/block-backend.c:129
#8  0x0000555555754f78 in blockdev_init (file=0x0, bs_opts=0x55555639a420, errp=0x7fffffffdb70) at blockdev.c:536
#9  0x0000555555755d90 in drive_new (all_opts=0x5555563777e0, block_default_type=IF_IDE) at blockdev.c:971
#10 0x000055555576b1f0 in drive_init_func (opts=0x5555563777e0, opaque=0x555556372b48) at vl.c:1104
#11 0x0000555555a1b019 in qemu_opts_foreach (list=0x555555e44060, func=0x55555576b1ba <drive_init_func>, opaque=0x555556372b48, abort_on_failure=1) at util/qemu-option.c:1059
#12 0x00005555557743dd in main (argc=25, argv=0x7fffffffe0c8, envp=0x7fffffffe198) at vl.c:4191

refresh_total_sectors() will fail if we return -ENOTCONN.

Thanks
Wen Congyang

> 
> Also, while you did write that one shouldn't be using the NBD client as the first quorum child, I think we should try to support that case anyway. For this patch, that means accepting that bdrv_getlength(s->bs[0]) may be off.
> 
> Max
> .
> 

Re: [Qemu-devel] [RFC PATCH 03/14] quorum: ignore 0-length child

[Max Reitz]

On 2015-03-18 at 01:29, Wen Congyang wrote:
> On 02/24/2015 04:43 AM, Max Reitz wrote:
>> On 2015-02-11 at 22:07, Wen Congyang wrote:
>>> We connect to NBD server when starting block replication, so
>>> the length is 0 before starting block replication.
>>>
>>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
>>> ---
>>>    block/quorum.c | 5 +++++
>>>    1 file changed, 5 insertions(+)
>>>
>>> diff --git a/block/quorum.c b/block/quorum.c
>>> index 5ed1ff8..e6aff5f 100644
>>> --- a/block/quorum.c
>>> +++ b/block/quorum.c
>>> @@ -734,6 +734,11 @@ static int64_t quorum_getlength(BlockDriverState *bs)
>>>            if (value < 0) {
>>>                return value;
>>>            }
>>> +
>>> +        if (!value) {
>>> +            continue;
>>> +        }
>>> +
>>>            if (value != result) {
>>>                return -EIO;
>>>            }
>> Hm, what do you think about some specific error value returned by your delayed NBD implementation? Like -ENOTCONN or something like that? Then we'd be able to discern a real 0-length block device from a not-yet-connected NBD server.
> In my newest test, it cannot return -ENOTCONN, otherwise bdrv_open() will fail when we open the child.
> Here is the backtrace:
> (gdb) bt
> #0  bdrv_open_common (bs=0x5555563b2230, file=0x0, options=0x5555563b55a0, flags=57410, drv=0x555555e90c00, errp=0x7fffffffd460) at block.c:1070
> #1  0x000055555595ea72 in bdrv_open (pbs=0x7fffffffd5f8, filename=0x0, reference=0x0, options=0x5555563b55a0, flags=57410, drv=0x555555e90c00, errp=0x7fffffffd5e0) at block.c:1677
> #2  0x000055555595e3a9 in bdrv_open_image (pbs=0x7fffffffd5f8, filename=0x0, options=0x5555563a6730, bdref_key=0x555555a86b4c "file", flags=57410, allow_none=true, errp=0x7fffffffd5e0) at block.c:1481
> #3  0x000055555595e9ae in bdrv_open (pbs=0x555556388008, filename=0x0, reference=0x0, options=0x5555563a6730, flags=8258, drv=0x555555e8b800, errp=0x7fffffffd6b8) at block.c:1655
> #4  0x00005555559b0058 in quorum_open (bs=0x55555639bd90, options=0x55555639f100, flags=8258, errp=0x7fffffffd758) at block/quorum.c:1000
> #5  0x000055555595d0b8 in bdrv_open_common (bs=0x55555639bd90, file=0x0, options=0x55555639f100, flags=8258, drv=0x555555e8e5c0, errp=0x7fffffffd840) at block.c:1045
> #6  0x000055555595ea72 in bdrv_open (pbs=0x55555639bd50, filename=0x0, reference=0x0, options=0x55555639f100, flags=8258, drv=0x555555e8e5c0, errp=0x7fffffffdb70) at block.c:1677
> #7  0x00005555559b3bd3 in blk_new_open (name=0x55555639bc60 "virtio0", filename=0x0, reference=0x0, options=0x55555639a420, flags=66, errp=0x7fffffffdb70) at block/block-backend.c:129
> #8  0x0000555555754f78 in blockdev_init (file=0x0, bs_opts=0x55555639a420, errp=0x7fffffffdb70) at blockdev.c:536
> #9  0x0000555555755d90 in drive_new (all_opts=0x5555563777e0, block_default_type=IF_IDE) at blockdev.c:971
> #10 0x000055555576b1f0 in drive_init_func (opts=0x5555563777e0, opaque=0x555556372b48) at vl.c:1104
> #11 0x0000555555a1b019 in qemu_opts_foreach (list=0x555555e44060, func=0x55555576b1ba <drive_init_func>, opaque=0x555556372b48, abort_on_failure=1) at util/qemu-option.c:1059
> #12 0x00005555557743dd in main (argc=25, argv=0x7fffffffe0c8, envp=0x7fffffffe198) at vl.c:4191
>
> refresh_total_sectors() will fail if we return -ENOTCONN.

Okay, then 0 will be fine, too.

Max

Re: [Qemu-devel] [RFC PATCH 01/14] docs: block replication's description

[Paolo Bonzini]

On 16/03/2015 07:19, Wen Congyang wrote:
> On 03/13/2015 05:05 PM, Fam Zheng wrote:
>> On Fri, 03/13 17:01, Wen Congyang wrote:
>>> On 03/11/2015 02:49 PM, Fam Zheng wrote:
>>>> On Wed, 03/11 14:44, Wen Congyang wrote:
>>>>> On 03/03/2015 03:59 PM, Fam Zheng wrote:
>>>>>> On Tue, 03/03 15:53, Wen Congyang wrote:
>>>>>>> I test qcow2_make_empty()'s performance. The result shows that it may
>>>>>>> take about 100ms(normal sata disk). It is not acceptable for COLO. So
>>>>>>> I think disk buff is necessary(just use it to replace qcow2).
>>>>>>
>>>>>> Why not tmpfs or ramdisk?
>>>>>
>>>>> Another problem:
>>>>> After failover, secondary write request will be written in (active disk)?
>>>>> It is better to write request to (nbd target). Is there any feature can
>>>>> be reused to implement it?
>>>>
>>>> You can use block commit or stream to move the data.
>>>
>>> Can the job stream move the data? I don't find the write ops in block/stream.c.
>>
>> It is bdrv_co_copy_on_readv that moves data.
> 
> Does the job stream move the data from base to top?

Yes.  block-commit goes in the other direction.

Paolo