All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 0/6] push backup with fleecing
@ 2021-05-21 17:49 Vladimir Sementsov-Ogievskiy
  2021-05-21 17:49 ` [PATCH 1/6] block/block-copy: use write-unchanged for fleecing scheme Vladimir Sementsov-Ogievskiy
                   ` (5 more replies)
  0 siblings, 6 replies; 7+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-21 17:49 UTC (permalink / raw)
  To: qemu-block; +Cc: qemu-devel, vsementsov, jsnow, mreitz, kwolf

Hi all!

Here is push-backup with fleecing. What is it:

1. Make fleecing scheme

guest blk
  |
  |root
  v
copy-before-write filter  -------> temp qcow2
  |                                  |
  |file                              | backing
  V                                  |
active disk <-------------------------

2. Start backup job from temp qcow2 to final remote target (NBD or
   something)

Benefit in comparison with simple backup job: for remote final target
write operations are not very fast. And guest have to wait for
copy-before-write operations. With fleecing scheme target for
copy-before-write operations is local qcow2 file with faster access than
actual backup target. So, guest is less disturbed by copy-before-write
operations.

Based-on: <20210520142205.607501-1-vsementsov@virtuozzo.com>
  ([PATCH v2 00/33] block: publish backup-top filter)

Vladimir Sementsov-Ogievskiy (6):
  block/block-copy: use write-unchanged for fleecing scheme
  block/copy-before-write: require BLK_PERM_WRITE_UNCHANGED for fleecing
  block: share writes on backing child of fleecing node
  block: blk_root(): return non-const pointer
  block/backup: don't insert filter if no writers
  iotests/image-fleecing: test push backup with fleecing

 block/copy-before-write.h                   |   1 +
 include/sysemu/block-backend.h              |   2 +-
 block.c                                     |   3 +-
 block/backup.c                              |  55 ++++++++++-
 block/block-backend.c                       |   2 +-
 block/block-copy.c                          |  14 ++-
 block/copy-before-write.c                   |  48 ++++++++-
 tests/qemu-iotests/tests/image-fleecing     | 104 +++++++++++++++-----
 tests/qemu-iotests/tests/image-fleecing.out |  60 +++++++++++
 9 files changed, 252 insertions(+), 37 deletions(-)

-- 
2.29.2



^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH 1/6] block/block-copy: use write-unchanged for fleecing scheme
  2021-05-21 17:49 [PATCH 0/6] push backup with fleecing Vladimir Sementsov-Ogievskiy
@ 2021-05-21 17:49 ` Vladimir Sementsov-Ogievskiy
  2021-05-21 17:49 ` [PATCH 2/6] block/copy-before-write: require BLK_PERM_WRITE_UNCHANGED for fleecing Vladimir Sementsov-Ogievskiy
                   ` (4 subsequent siblings)
  5 siblings, 0 replies; 7+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-21 17:49 UTC (permalink / raw)
  To: qemu-block; +Cc: qemu-devel, vsementsov, jsnow, mreitz, kwolf

We are going to use fleecing scheme for push-backup, so that
copy-before-write filter does copy before write operations to temporary
image and backup job copies data from (immutable from backup's point of
view) temporary image to actual backup target. For this to work
properly, backup job should unshare writes on immutable source node.
copy-before-write filter should do write-unchanged operations for this
(they are really unchanged, as source is a backing of temporary node).

So, let's detect fleecing scheme in block-copy and do write-unchanged
operations in this case.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 block/block-copy.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/block/block-copy.c b/block/block-copy.c
index 0a9c3692bf..1a9db5794a 100644
--- a/block/block-copy.c
+++ b/block/block-copy.c
@@ -418,6 +418,7 @@ static int coroutine_fn block_copy_do_copy(BlockCopyState *s,
     int ret;
     int64_t nbytes = MIN(offset + bytes, s->len) - offset;
     void *bounce_buffer = NULL;
+    BdrvRequestFlags write_flags = s->write_flags;
 
     assert(offset >= 0 && bytes > 0 && INT64_MAX - offset >= bytes);
     assert(QEMU_IS_ALIGNED(offset, s->cluster_size));
@@ -427,8 +428,15 @@ static int coroutine_fn block_copy_do_copy(BlockCopyState *s,
            offset + bytes == QEMU_ALIGN_UP(s->len, s->cluster_size));
     assert(nbytes < INT_MAX);
 
+    /* Detect fleecing scheme */
+    if (bdrv_backing_chain_next(s->target->bs) ==
+        bdrv_skip_filters(s->source->bs))
+    {
+        write_flags |= BDRV_REQ_WRITE_UNCHANGED;
+    }
+
     if (zeroes) {
-        ret = bdrv_co_pwrite_zeroes(s->target, offset, nbytes, s->write_flags &
+        ret = bdrv_co_pwrite_zeroes(s->target, offset, nbytes, write_flags &
                                     ~BDRV_REQ_WRITE_COMPRESSED);
         if (ret < 0) {
             trace_block_copy_write_zeroes_fail(s, offset, ret);
@@ -439,7 +447,7 @@ static int coroutine_fn block_copy_do_copy(BlockCopyState *s,
 
     if (s->use_copy_range) {
         ret = bdrv_co_copy_range(s->source, offset, s->target, offset, nbytes,
-                                 0, s->write_flags);
+                                 0, write_flags);
         if (ret < 0) {
             trace_block_copy_copy_range_fail(s, offset, ret);
             s->use_copy_range = false;
@@ -485,7 +493,7 @@ static int coroutine_fn block_copy_do_copy(BlockCopyState *s,
     }
 
     ret = bdrv_co_pwrite(s->target, offset, nbytes, bounce_buffer,
-                         s->write_flags);
+                         write_flags);
     if (ret < 0) {
         trace_block_copy_write_fail(s, offset, ret);
         *error_is_read = false;
-- 
2.29.2



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH 2/6] block/copy-before-write: require BLK_PERM_WRITE_UNCHANGED for fleecing
  2021-05-21 17:49 [PATCH 0/6] push backup with fleecing Vladimir Sementsov-Ogievskiy
  2021-05-21 17:49 ` [PATCH 1/6] block/block-copy: use write-unchanged for fleecing scheme Vladimir Sementsov-Ogievskiy
@ 2021-05-21 17:49 ` Vladimir Sementsov-Ogievskiy
  2021-05-21 17:49 ` [PATCH 3/6] block: share writes on backing child of fleecing node Vladimir Sementsov-Ogievskiy
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 7+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-21 17:49 UTC (permalink / raw)
  To: qemu-block; +Cc: qemu-devel, vsementsov, jsnow, mreitz, kwolf

Now block-copy detects fleecing scheme and do write-unchanged
operations if detected. So, let's require appropriate permissions.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 block/copy-before-write.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/block/copy-before-write.c b/block/copy-before-write.c
index 6ed5bce1f1..1b104545bd 100644
--- a/block/copy-before-write.c
+++ b/block/copy-before-write.c
@@ -113,6 +113,14 @@ static void cbw_refresh_filename(BlockDriverState *bs)
             bs->file->bs->filename);
 }
 
+static bool cbw_is_fleecing(BlockDriverState *bs)
+{
+    BDRVCopyBeforeWriteState *s = bs->opaque;
+
+    return bs->file && s->target &&
+        bdrv_skip_filters(bs) == bdrv_backing_chain_next(s->target->bs);
+}
+
 static void cbw_child_perm(BlockDriverState *bs, BdrvChild *c,
                            BdrvChildRole role,
                            BlockReopenQueue *reopen_queue,
@@ -129,7 +137,8 @@ static void cbw_child_perm(BlockDriverState *bs, BdrvChild *c,
          * only upfront.
          */
         *nshared = BLK_PERM_ALL & ~BLK_PERM_RESIZE;
-        *nperm = BLK_PERM_WRITE;
+        *nperm =
+            cbw_is_fleecing(bs) ? BLK_PERM_WRITE_UNCHANGED : BLK_PERM_WRITE;
     } else {
         /* Source child */
         bdrv_default_perms(bs, c, role, reopen_queue,
-- 
2.29.2



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH 3/6] block: share writes on backing child of fleecing node
  2021-05-21 17:49 [PATCH 0/6] push backup with fleecing Vladimir Sementsov-Ogievskiy
  2021-05-21 17:49 ` [PATCH 1/6] block/block-copy: use write-unchanged for fleecing scheme Vladimir Sementsov-Ogievskiy
  2021-05-21 17:49 ` [PATCH 2/6] block/copy-before-write: require BLK_PERM_WRITE_UNCHANGED for fleecing Vladimir Sementsov-Ogievskiy
@ 2021-05-21 17:49 ` Vladimir Sementsov-Ogievskiy
  2021-05-21 17:49 ` [PATCH 4/6] block: blk_root(): return non-const pointer Vladimir Sementsov-Ogievskiy
                   ` (2 subsequent siblings)
  5 siblings, 0 replies; 7+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-21 17:49 UTC (permalink / raw)
  To: qemu-block; +Cc: qemu-devel, vsementsov, jsnow, mreitz, kwolf

By default, we share writes on backing child only if our parents share
write permission on us.

Still, with fleecing scheme we want to be able to unshare writes on
fleecing node, which is a kind of immutable snapshot
(copy-before-write operations are write-unchanged). So, let's detect
fleecing node and share writes on its backing child. (we should share
them, otherwise copy-before-write filter can't write to its file
child).

With fleecing scheme we are sure, that writes to backing child goes
through copy-before-write filter, so we are safe to share them.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 block/copy-before-write.h |  1 +
 block.c                   |  3 ++-
 block/copy-before-write.c | 37 +++++++++++++++++++++++++++++++++++++
 3 files changed, 40 insertions(+), 1 deletion(-)

diff --git a/block/copy-before-write.h b/block/copy-before-write.h
index b386fd8f01..ca47af732a 100644
--- a/block/copy-before-write.h
+++ b/block/copy-before-write.h
@@ -36,5 +36,6 @@ BlockDriverState *bdrv_cbw_append(BlockDriverState *source,
                                   BlockCopyState **bcs,
                                   Error **errp);
 void bdrv_cbw_drop(BlockDriverState *bs);
+bool bdrv_is_fleecing_node(BlockDriverState *bs);
 
 #endif /* COPY_BEFORE_WRITE_H */
diff --git a/block.c b/block.c
index 39a5d4be90..c03810e8fa 100644
--- a/block.c
+++ b/block.c
@@ -50,6 +50,7 @@
 #include "qemu/cutils.h"
 #include "qemu/id.h"
 #include "block/coroutines.h"
+#include "block/copy-before-write.h"
 
 #ifdef CONFIG_BSD
 #include <sys/ioctl.h>
@@ -2502,7 +2503,7 @@ static void bdrv_default_perms_for_cow(BlockDriverState *bs, BdrvChild *c,
      * writable and resizable backing file.
      * TODO Require !(perm & BLK_PERM_CONSISTENT_READ), too?
      */
-    if (shared & BLK_PERM_WRITE) {
+    if (shared & BLK_PERM_WRITE || bdrv_is_fleecing_node(bs)) {
         shared = BLK_PERM_WRITE | BLK_PERM_RESIZE;
     } else {
         shared = 0;
diff --git a/block/copy-before-write.c b/block/copy-before-write.c
index 1b104545bd..88c9bb0a91 100644
--- a/block/copy-before-write.c
+++ b/block/copy-before-write.c
@@ -269,6 +269,43 @@ void bdrv_cbw_drop(BlockDriverState *bs)
     bdrv_unref(bs);
 }
 
+/*
+ * Detect is bs a fleecing node in some fleecing sceheme like:
+ *
+ * copy-before-write -- target --> fleecing-node
+ *   |                               |
+ *   | file                          | backing
+ * active-node  <---------------------
+ *
+ * In this case, fleecing-node can (and should) safely share writes on its
+ * backing child.
+ */
+bool bdrv_is_fleecing_node(BlockDriverState *bs)
+{
+    BdrvChild *parent;
+    BlockDriverState *parent_bs;
+    BDRVCopyBeforeWriteState *s;
+
+    QLIST_FOREACH(parent, &bs->parents, next_parent) {
+        if (parent->klass != &child_of_bds) {
+            continue;
+        }
+
+        parent_bs = parent->opaque;
+        if (parent_bs->drv != &bdrv_cbw_filter) {
+            continue;
+        }
+
+        s = parent_bs->opaque;
+
+        if (s->target->bs == bs && cbw_is_fleecing(parent_bs)) {
+            return true;
+        }
+    }
+
+    return false;
+}
+
 static void cbw_init(void)
 {
     bdrv_register(&bdrv_cbw_filter);
-- 
2.29.2



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH 4/6] block: blk_root(): return non-const pointer
  2021-05-21 17:49 [PATCH 0/6] push backup with fleecing Vladimir Sementsov-Ogievskiy
                   ` (2 preceding siblings ...)
  2021-05-21 17:49 ` [PATCH 3/6] block: share writes on backing child of fleecing node Vladimir Sementsov-Ogievskiy
@ 2021-05-21 17:49 ` Vladimir Sementsov-Ogievskiy
  2021-05-21 17:49 ` [PATCH 5/6] block/backup: don't insert filter if no writers Vladimir Sementsov-Ogievskiy
  2021-05-21 17:49 ` [PATCH 6/6] iotests/image-fleecing: test push backup with fleecing Vladimir Sementsov-Ogievskiy
  5 siblings, 0 replies; 7+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-21 17:49 UTC (permalink / raw)
  To: qemu-block; +Cc: qemu-devel, vsementsov, jsnow, mreitz, kwolf

In the following patch we'll want to pass blk children to block-copy.
Const pointers are not enough. So, return non const pointer from
blk_root().

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 include/sysemu/block-backend.h | 2 +-
 block/block-backend.c          | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h
index aec05ef0a0..9c755c38b1 100644
--- a/include/sysemu/block-backend.h
+++ b/include/sysemu/block-backend.h
@@ -266,7 +266,7 @@ int coroutine_fn blk_co_copy_range(BlockBackend *blk_in, int64_t off_in,
                                    int bytes, BdrvRequestFlags read_flags,
                                    BdrvRequestFlags write_flags);
 
-const BdrvChild *blk_root(BlockBackend *blk);
+BdrvChild *blk_root(BlockBackend *blk);
 
 int blk_make_empty(BlockBackend *blk, Error **errp);
 
diff --git a/block/block-backend.c b/block/block-backend.c
index b1abc6f3e6..758cd8507e 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -2452,7 +2452,7 @@ int coroutine_fn blk_co_copy_range(BlockBackend *blk_in, int64_t off_in,
                               bytes, read_flags, write_flags);
 }
 
-const BdrvChild *blk_root(BlockBackend *blk)
+BdrvChild *blk_root(BlockBackend *blk)
 {
     return blk->root;
 }
-- 
2.29.2



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH 5/6] block/backup: don't insert filter if no writers
  2021-05-21 17:49 [PATCH 0/6] push backup with fleecing Vladimir Sementsov-Ogievskiy
                   ` (3 preceding siblings ...)
  2021-05-21 17:49 ` [PATCH 4/6] block: blk_root(): return non-const pointer Vladimir Sementsov-Ogievskiy
@ 2021-05-21 17:49 ` Vladimir Sementsov-Ogievskiy
  2021-05-21 17:49 ` [PATCH 6/6] iotests/image-fleecing: test push backup with fleecing Vladimir Sementsov-Ogievskiy
  5 siblings, 0 replies; 7+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-21 17:49 UTC (permalink / raw)
  To: qemu-block; +Cc: qemu-devel, vsementsov, jsnow, mreitz, kwolf

If source is immutable and there no writers on it, we don't need to
insert a filter, so let's detect it and use simple blk's for
block-copy.

Note, that it's possible, that user will try to add writers on source
during backup. It will fail, as our source blk doesn't share write.

In future we can add a tri-state source-mode parameter for backup job
with the following values:

 immutable: got without filter. blockdev-backup command fails if there
            are writers on source. Adding writers during backup will
            fail.
 filtered: insert filter unconditionally. Writers are supported on
           start. User may add new writers above copy-before-write
           filter during backup [current behavior]
 auto: go "immutable" if there no writers on start, go "filtered"
       otherwise

And "auto" would be a default behavior. For now, let's just change a
default behavior to not create extra filter when it's not necessary.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 block/backup.c | 55 +++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 50 insertions(+), 5 deletions(-)

diff --git a/block/backup.c b/block/backup.c
index 14652ac98a..e856e4ad73 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -34,6 +34,8 @@ typedef struct BackupBlockJob {
     BlockDriverState *cbw;
     BlockDriverState *source_bs;
     BlockDriverState *target_bs;
+    BlockBackend *source_blk;
+    BlockBackend *target_blk;
 
     BdrvDirtyBitmap *sync_bitmap;
 
@@ -102,7 +104,17 @@ static void backup_clean(Job *job)
 {
     BackupBlockJob *s = container_of(job, BackupBlockJob, common.job);
     block_job_remove_all_bdrv(&s->common);
-    bdrv_cbw_drop(s->cbw);
+    if (s->cbw) {
+        assert(!s->source_blk && !s->target_blk);
+        bdrv_cbw_drop(s->cbw);
+    } else {
+        block_copy_state_free(s->bcs);
+        s->bcs = NULL;
+        blk_unref(s->source_blk);
+        s->source_blk = NULL;
+        blk_unref(s->target_blk);
+        s->target_blk = NULL;
+    }
 }
 
 void backup_do_checkpoint(BlockJob *job, Error **errp)
@@ -368,6 +380,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
     int64_t cluster_size;
     BlockDriverState *cbw = NULL;
     BlockCopyState *bcs = NULL;
+    BlockBackend *source_blk = NULL, *target_blk = NULL;
 
     assert(bs);
     assert(target);
@@ -450,9 +463,37 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
         goto error;
     }
 
-    cbw = bdrv_cbw_append(bs, target, filter_node_name, compress, &bcs, errp);
-    if (!cbw) {
-        goto error;
+    source_blk = blk_new_with_bs(bs, BLK_PERM_CONSISTENT_READ,
+                                    BLK_PERM_WRITE_UNCHANGED |
+                                    BLK_PERM_CONSISTENT_READ, NULL);
+    if (source_blk) {
+        BdrvDirtyBitmap *copy_bitmap;
+
+        target_blk  = blk_new_with_bs(target, BLK_PERM_WRITE,
+                                      BLK_PERM_CONSISTENT_READ, errp);
+        if (!target_blk) {
+            goto error;
+        }
+
+        bcs = block_copy_state_new(blk_root(source_blk), blk_root(target_blk),
+                                   false, compress, errp);
+        if (!bcs) {
+            goto error;
+        }
+
+        /*
+         * initalize bitmap in a way copy-before-write filter do it, to have
+         * same code path later.
+         */
+        copy_bitmap = block_copy_dirty_bitmap(bcs);
+        bdrv_set_dirty_bitmap(copy_bitmap, 0,
+                              bdrv_dirty_bitmap_size(copy_bitmap));
+    } else {
+        cbw = bdrv_cbw_append(bs, target, filter_node_name, compress, &bcs,
+                              errp);
+        if (!cbw) {
+            goto error;
+        }
     }
 
     cluster_size = block_copy_cluster_size(bcs);
@@ -464,7 +505,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
     }
 
     /* job->len is fixed, so we can't allow resize */
-    job = block_job_create(job_id, &backup_job_driver, txn, cbw,
+    job = block_job_create(job_id, &backup_job_driver, txn, cbw ?: bs,
                            0, BLK_PERM_ALL,
                            speed, creation_flags, cb, opaque, errp);
     if (!job) {
@@ -474,6 +515,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
     job->cbw = cbw;
     job->source_bs = bs;
     job->target_bs = target;
+    job->source_blk = source_blk;
+    job->target_blk = target_blk;
     job->on_source_error = on_source_error;
     job->on_target_error = on_target_error;
     job->sync_mode = sync_mode;
@@ -500,6 +543,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
     if (cbw) {
         bdrv_cbw_drop(cbw);
     }
+    blk_unref(source_blk);
+    blk_unref(target_blk);
 
     return NULL;
 }
-- 
2.29.2



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH 6/6] iotests/image-fleecing: test push backup with fleecing
  2021-05-21 17:49 [PATCH 0/6] push backup with fleecing Vladimir Sementsov-Ogievskiy
                   ` (4 preceding siblings ...)
  2021-05-21 17:49 ` [PATCH 5/6] block/backup: don't insert filter if no writers Vladimir Sementsov-Ogievskiy
@ 2021-05-21 17:49 ` Vladimir Sementsov-Ogievskiy
  5 siblings, 0 replies; 7+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2021-05-21 17:49 UTC (permalink / raw)
  To: qemu-block; +Cc: qemu-devel, vsementsov, jsnow, mreitz, kwolf

Add test for push backup with fleecing:

 - start fleecing with copy-before-write filter
 - start a backup job from temporary fleecing node to actual backup
   target

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 tests/qemu-iotests/tests/image-fleecing     | 104 +++++++++++++++-----
 tests/qemu-iotests/tests/image-fleecing.out |  60 +++++++++++
 2 files changed, 139 insertions(+), 25 deletions(-)

diff --git a/tests/qemu-iotests/tests/image-fleecing b/tests/qemu-iotests/tests/image-fleecing
index 404ebc00f1..4c0cf2a088 100755
--- a/tests/qemu-iotests/tests/image-fleecing
+++ b/tests/qemu-iotests/tests/image-fleecing
@@ -48,12 +48,20 @@ remainder = [('0xd5', '0x108000',  '32k'), # Right-end of partial-left [1]
              ('0xdc', '32M',       '32k'), # Left-end of partial-right [2]
              ('0xcd', '0x3ff0000', '64k')] # patterns[3]
 
-def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
+def do_test(vm, use_cbw, base_img_path, fleece_img_path, nbd_sock_path=None,
+            target_img_path=None):
+    push_backup = target_img_path is not None
+    assert (nbd_sock_path is not None) != push_backup
+    if push_backup:
+        assert use_cbw
+
     log('--- Setting up images ---')
     log('')
 
     assert qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M') == 0
     assert qemu_img('create', '-f', 'qcow2', fleece_img_path, '64M') == 0
+    if push_backup:
+        assert qemu_img('create', '-f', 'qcow2', target_img_path, '64M') == 0
 
     for p in patterns:
         qemu_io('-f', iotests.imgfmt,
@@ -108,25 +116,42 @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
                    target=tmp_node,
                    sync='none'))
 
-    log('')
-    log('--- Setting up NBD Export ---')
-    log('')
+    if push_backup:
+        log('')
+        log('--- Starting actual backup ---')
+        log('')
 
-    nbd_uri = 'nbd+unix:///%s?socket=%s' % (tmp_node, nbd_sock_path)
-    log(vm.qmp('nbd-server-start',
-               {'addr': { 'type': 'unix',
-                          'data': { 'path': nbd_sock_path } } }))
+        log(vm.qmp('blockdev-add', **{
+            'driver': iotests.imgfmt,
+            'node-name': 'target',
+            'file': {
+                'driver': 'file',
+                'filename': target_img_path
+            }
+        }))
+        log(vm.qmp('blockdev-backup', device=tmp_node,
+                   sync='full', target='target',
+                   job_id='push-backup', speed=1))
+    else:
+        log('')
+        log('--- Setting up NBD Export ---')
+        log('')
 
-    log(vm.qmp('nbd-server-add', device=tmp_node))
+        nbd_uri = 'nbd+unix:///%s?socket=%s' % (tmp_node, nbd_sock_path)
+        log(vm.qmp('nbd-server-start',
+                   {'addr': { 'type': 'unix',
+                              'data': { 'path': nbd_sock_path } } }))
 
-    log('')
-    log('--- Sanity Check ---')
-    log('')
+        log(vm.qmp('nbd-server-add', device=tmp_node))
 
-    for p in patterns + zeroes:
-        cmd = 'read -P%s %s %s' % p
-        log(cmd)
-        assert qemu_io_silent('-r', '-f', 'raw', '-c', cmd, nbd_uri) == 0
+        log('')
+        log('--- Sanity Check ---')
+        log('')
+
+        for p in patterns + zeroes:
+            cmd = 'read -P%s %s %s' % p
+            log(cmd)
+            assert qemu_io_silent('-r', '-f', 'raw', '-c', cmd, nbd_uri) == 0
 
     log('')
     log('--- Testing COW ---')
@@ -137,6 +162,20 @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
         log(cmd)
         log(vm.hmp_qemu_io(qom_path, cmd, qdev=True))
 
+    if push_backup:
+        # Check that previous operations were done during backup, not after
+        result = vm.qmp('query-block-jobs')
+        if len(result['return']) != 1:
+            log('Backup finished too fast, COW is not tested')
+
+        result = vm.qmp('block-job-set-speed', device='push-backup', speed=0)
+        assert result == {'return': {}}
+
+        log(vm.event_wait(name='BLOCK_JOB_COMPLETED',
+                          match={'data': {'device': 'push-backup'}}),
+                          filters=[iotests.filter_qmp_event])
+        log(vm.qmp('blockdev-del', node_name='target'))
+
     log('')
     log('--- Verifying Data ---')
     log('')
@@ -144,7 +183,10 @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
     for p in patterns + zeroes:
         cmd = 'read -P%s %s %s' % p
         log(cmd)
-        assert qemu_io_silent('-r', '-f', 'raw', '-c', cmd, nbd_uri) == 0
+        if push_backup:
+            assert qemu_io_silent('-r', '-c', cmd, target_img_path) == 0
+        else:
+            assert qemu_io_silent('-r', '-f', 'raw', '-c', cmd, nbd_uri) == 0
 
     log('')
     log('--- Cleanup ---')
@@ -159,7 +201,9 @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
         assert e is not None
         log(e, filters=[iotests.filter_qmp_event])
 
-    log(vm.qmp('nbd-server-stop'))
+    if not push_backup:
+        log(vm.qmp('nbd-server-stop'))
+
     log(vm.qmp('blockdev-del', node_name=tmp_node))
     vm.shutdown()
 
@@ -175,18 +219,28 @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm):
     log('')
     log('Done')
 
-
-def test(use_cbw):
+def test(use_cbw, nbd_sock_path=None, target_img_path=None):
     with iotests.FilePath('base.img') as base_img_path, \
          iotests.FilePath('fleece.img') as fleece_img_path, \
-         iotests.FilePath('nbd.sock',
-                          base_dir=iotests.sock_dir) as nbd_sock_path, \
          iotests.VM() as vm:
-        do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm)
+        do_test(vm, use_cbw, base_img_path, fleece_img_path, nbd_sock_path,
+                target_img_path)
+
+def test_pull(use_cbw):
+    with iotests.FilePath('nbd.sock',
+                          base_dir=iotests.sock_dir) as nbd_sock_path:
+        test(use_cbw, nbd_sock_path, None)
+
+def test_push():
+    with iotests.FilePath('target.img') as target_img_path:
+        test(True, None, target_img_path)
 
 
 log('=== Test backup(sync=none) based fleecing ===\n')
-test(False)
+test_pull(False)
 
 log('=== Test filter based fleecing ===\n')
-test(True)
+test_pull(True)
+
+log('=== Test push backup with fleecing ===\n')
+test_push()
diff --git a/tests/qemu-iotests/tests/image-fleecing.out b/tests/qemu-iotests/tests/image-fleecing.out
index e96d122a8b..24f97bf7f6 100644
--- a/tests/qemu-iotests/tests/image-fleecing.out
+++ b/tests/qemu-iotests/tests/image-fleecing.out
@@ -137,3 +137,63 @@ read -P0xdc 32M 32k
 read -P0xcd 0x3ff0000 64k
 
 Done
+=== Test push backup with fleecing ===
+--- Setting up images ---
+
+Done
+
+--- Launching VM ---
+
+Done
+
+--- Setting up Fleecing Graph ---
+
+{"return": {}}
+{"return": {}}
+{"return": {}}
+
+--- Starting actual backup ---
+
+{"return": {}}
+{"return": {}}
+
+--- Testing COW ---
+
+write -P0xab 0 64k
+{"return": ""}
+write -P0xad 0x00f8000 64k
+{"return": ""}
+write -P0x1d 0x2008000 64k
+{"return": ""}
+write -P0xea 0x3fe0000 64k
+{"return": ""}
+{"data": {"device": "push-backup", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": {}}
+
+--- Verifying Data ---
+
+read -P0x5d 0 64k
+read -P0xd5 1M 64k
+read -P0xdc 32M 64k
+read -P0xcd 0x3ff0000 64k
+read -P0 0x00f8000 32k
+read -P0 0x2010000 32k
+read -P0 0x3fe0000 64k
+
+--- Cleanup ---
+
+{"return": {}}
+{"return": {}}
+{"return": {}}
+
+--- Confirming writes ---
+
+read -P0xab 0 64k
+read -P0xad 0x00f8000 64k
+read -P0x1d 0x2008000 64k
+read -P0xea 0x3fe0000 64k
+read -P0xd5 0x108000 32k
+read -P0xdc 32M 32k
+read -P0xcd 0x3ff0000 64k
+
+Done
-- 
2.29.2



^ permalink raw reply related	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2021-05-21 17:56 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-21 17:49 [PATCH 0/6] push backup with fleecing Vladimir Sementsov-Ogievskiy
2021-05-21 17:49 ` [PATCH 1/6] block/block-copy: use write-unchanged for fleecing scheme Vladimir Sementsov-Ogievskiy
2021-05-21 17:49 ` [PATCH 2/6] block/copy-before-write: require BLK_PERM_WRITE_UNCHANGED for fleecing Vladimir Sementsov-Ogievskiy
2021-05-21 17:49 ` [PATCH 3/6] block: share writes on backing child of fleecing node Vladimir Sementsov-Ogievskiy
2021-05-21 17:49 ` [PATCH 4/6] block: blk_root(): return non-const pointer Vladimir Sementsov-Ogievskiy
2021-05-21 17:49 ` [PATCH 5/6] block/backup: don't insert filter if no writers Vladimir Sementsov-Ogievskiy
2021-05-21 17:49 ` [PATCH 6/6] iotests/image-fleecing: test push backup with fleecing Vladimir Sementsov-Ogievskiy

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.