[Qemu-devel] [PATCH v7 0/3] block/stream: get rid of the base

[Qemu-devel] [PATCH v7 0/3] block/stream: get rid of the base

[Andrey Shinkevich]

This series introduces a bottom intermediate node that eliminates the
dependency on the base that may change while stream job is running.
It happens when stream/commit parallel jobs are running on the same
backing chain. The base node of the stream job may be a top node of
the parallel commit job and can change before the stream job is
completed. We avoid that dependency by introducing the bottom node.

v7: [resend by Andrey]
  01: assert(intermediate) was inserted before the call to
      bdrv_is_allocated() in the intermediate node loop of the
      bdrv_is_allocated_above() as suggested by Max.
  02: The change of the intermediate node loop in the stream_start() was
      rolled back to its original design and the reassignment of the base
      node pointer was added as Vladimir and Max suggested. The relevant
      comment was amended.

v6: [resend by Vladimir]
  01: improve comment in block/io.c, suggested by Alberto

v5: [resend by Vladimir]
  01: use comment wording in block/io.c suggested by Alberto

v4:
trace_stream_start reverted to the base.
bdrv_is_allocated_above_inclusive() deleted and the new parameter
'bool include_base' was added to the bdrv_is_allocated_above().

Andrey Shinkevich (3):
  block: include base when checking image chain for block allocation
  block/stream: refactor stream_run: drop goto
  block/stream: introduce a bottom node

 block/commit.c         |  2 +-
 block/io.c             | 21 +++++++++++++------
 block/mirror.c         |  2 +-
 block/replication.c    |  2 +-
 block/stream.c         | 56 ++++++++++++++++++++++++--------------------------
 include/block/block.h  |  3 ++-
 tests/qemu-iotests/245 |  4 ++--
 7 files changed, 49 insertions(+), 41 deletions(-)

-- 
1.8.3.1

[Qemu-devel] [PATCH v7 1/3] block: include base when checking image chain for block allocation

[Andrey Shinkevich]

This patch is used in the 'block/stream: introduce a bottom node'
that is following. Instead of the base node, the caller may pass
the node that has the base as its backing image to the function
bdrv_is_allocated_above() with a new parameter include_base = true
and get rid of the dependency on the base that may change during
commit/stream parallel jobs. Now, if the specified base is not
found in the backing image chain, the QEMU will abort.

Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
---
 block/commit.c        |  2 +-
 block/io.c            | 21 +++++++++++++++------
 block/mirror.c        |  2 +-
 block/replication.c   |  2 +-
 block/stream.c        |  2 +-
 include/block/block.h |  3 ++-
 6 files changed, 21 insertions(+), 11 deletions(-)

diff --git a/block/commit.c b/block/commit.c
index 14e5bb3..62ca90b 100644
--- a/block/commit.c
+++ b/block/commit.c
@@ -176,7 +176,7 @@ static int coroutine_fn commit_run(Job *job, Error **errp)
             break;
         }
         /* Copy if allocated above the base */
-        ret = bdrv_is_allocated_above(blk_bs(s->top), blk_bs(s->base),
+        ret = bdrv_is_allocated_above(blk_bs(s->top), blk_bs(s->base), false,
                                       offset, COMMIT_BUFFER_SIZE, &n);
         copy = (ret == 1);
         trace_commit_one_iteration(s, offset, n, ret);
diff --git a/block/io.c b/block/io.c
index 3134a60..43cefc3 100644
--- a/block/io.c
+++ b/block/io.c
@@ -2287,10 +2287,11 @@ int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t offset,
 /*
  * Given an image chain: ... -> [BASE] -> [INTER1] -> [INTER2] -> [TOP]
  *
- * Return true if (a prefix of) the given range is allocated in any image
- * between BASE and TOP (inclusive).  BASE can be NULL to check if the given
- * offset is allocated in any image of the chain.  Return false otherwise,
- * or negative errno on failure.
+ * Return 1 if (a prefix of) the given range is allocated in any image
+ * between BASE and TOP (BASE is only included if include_base is set).
+ * BASE can be NULL to check if the given offset is allocated in any
+ * image of the chain.  Return 0 otherwise, or negative errno on
+ * failure.
  *
  * 'pnum' is set to the number of bytes (including and immediately
  * following the specified offset) that are known to be in the same
@@ -2302,17 +2303,21 @@ int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t offset,
  */
 int bdrv_is_allocated_above(BlockDriverState *top,
                             BlockDriverState *base,
-                            int64_t offset, int64_t bytes, int64_t *pnum)
+                            bool include_base, int64_t offset,
+                            int64_t bytes, int64_t *pnum)
 {
     BlockDriverState *intermediate;
     int ret;
     int64_t n = bytes;
 
+    assert(base || !include_base);
+
     intermediate = top;
-    while (intermediate && intermediate != base) {
+    while (include_base || intermediate != base) {
         int64_t pnum_inter;
         int64_t size_inter;
 
+        assert(intermediate);
         ret = bdrv_is_allocated(intermediate, offset, bytes, &pnum_inter);
         if (ret < 0) {
             return ret;
@@ -2331,6 +2336,10 @@ int bdrv_is_allocated_above(BlockDriverState *top,
             n = pnum_inter;
         }
 
+        if (intermediate == base) {
+            break;
+        }
+
         intermediate = backing_bs(intermediate);
     }
 
diff --git a/block/mirror.c b/block/mirror.c
index ec4bd9f..81c2967 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -807,7 +807,7 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
             return 0;
         }
 
-        ret = bdrv_is_allocated_above(bs, base, offset, bytes, &count);
+        ret = bdrv_is_allocated_above(bs, base, false, offset, bytes, &count);
         if (ret < 0) {
             return ret;
         }
diff --git a/block/replication.c b/block/replication.c
index 3d4dedd..fc8d2ad 100644
--- a/block/replication.c
+++ b/block/replication.c
@@ -272,7 +272,7 @@ static coroutine_fn int replication_co_writev(BlockDriverState *bs,
     while (remaining_sectors > 0) {
         int64_t count;
 
-        ret = bdrv_is_allocated_above(top->bs, base->bs,
+        ret = bdrv_is_allocated_above(top->bs, base->bs, false,
                                       sector_num * BDRV_SECTOR_SIZE,
                                       remaining_sectors * BDRV_SECTOR_SIZE,
                                       &count);
diff --git a/block/stream.c b/block/stream.c
index 1a906fd..97fddb2 100644
--- a/block/stream.c
+++ b/block/stream.c
@@ -160,7 +160,7 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
         } else if (ret >= 0) {
             /* Copy if allocated in the intermediate images.  Limit to the
              * known-unallocated area [offset, offset+n*BDRV_SECTOR_SIZE).  */
-            ret = bdrv_is_allocated_above(backing_bs(bs), base,
+            ret = bdrv_is_allocated_above(backing_bs(bs), base, false,
                                           offset, n, &n);
 
             /* Finish early if end of backing file has been reached */
diff --git a/include/block/block.h b/include/block/block.h
index 9b083e2..e19b972 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -443,7 +443,8 @@ int bdrv_block_status_above(BlockDriverState *bs, BlockDriverState *base,
 int bdrv_is_allocated(BlockDriverState *bs, int64_t offset, int64_t bytes,
                       int64_t *pnum);
 int bdrv_is_allocated_above(BlockDriverState *top, BlockDriverState *base,
-                            int64_t offset, int64_t bytes, int64_t *pnum);
+                            bool include_base, int64_t offset, int64_t bytes,
+                            int64_t *pnum);
 
 bool bdrv_is_read_only(BlockDriverState *bs);
 int bdrv_can_set_read_only(BlockDriverState *bs, bool read_only,
-- 
1.8.3.1

[Qemu-devel] [PATCH v7 2/3] block/stream: refactor stream_run: drop goto

[Andrey Shinkevich]

The goto is unnecessary in the stream_run() since the common exit
code was removed in the commit eb23654dbe43b549ea2a9ebff9d8e:
"jobs: utilize job_exit shim".

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
---
 block/stream.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/block/stream.c b/block/stream.c
index 97fddb2..65b13b2 100644
--- a/block/stream.c
+++ b/block/stream.c
@@ -120,13 +120,12 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
     void *buf;
 
     if (!bs->backing) {
-        goto out;
+        return 0;
     }
 
     len = bdrv_getlength(bs);
     if (len < 0) {
-        ret = len;
-        goto out;
+        return len;
     }
     job_progress_set_remaining(&s->common.job, len);
 
@@ -203,14 +202,10 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
         bdrv_disable_copy_on_read(bs);
     }
 
-    /* Do not remove the backing file if an error was there but ignored.  */
-    ret = error;
-
     qemu_vfree(buf);
 
-out:
-    /* Modify backing chain and close BDSes in main loop */
-    return ret;
+    /* Do not remove the backing file if an error was there but ignored. */
+    return error;
 }
 
 static const BlockJobDriver stream_job_driver = {
-- 
1.8.3.1

[Qemu-devel] [PATCH v7 3/3] block/stream: introduce a bottom node

[Andrey Shinkevich]

The bottom node is the intermediate block device that has the base as its
backing image. It is used instead of the base node while a block stream
job is running to avoid dependency on the base that may change due to the
parallel jobs. The change may take place due to a filter node as well that
is inserted between the base and the intermediate bottom node. It occurs
when the base node is the top one for another commit or stream job.
After the introduction of the bottom node, don't freeze its backing child,
that's the base, anymore.

Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
---
 block/stream.c         | 43 +++++++++++++++++++++++--------------------
 tests/qemu-iotests/245 |  4 ++--
 2 files changed, 25 insertions(+), 22 deletions(-)

diff --git a/block/stream.c b/block/stream.c
index 65b13b2..cd5e2ba 100644
--- a/block/stream.c
+++ b/block/stream.c
@@ -31,7 +31,7 @@ enum {
 
 typedef struct StreamBlockJob {
     BlockJob common;
-    BlockDriverState *base;
+    BlockDriverState *bottom;
     BlockdevOnError on_error;
     char *backing_file_str;
     bool bs_read_only;
@@ -54,7 +54,7 @@ static void stream_abort(Job *job)
 
     if (s->chain_frozen) {
         BlockJob *bjob = &s->common;
-        bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->base);
+        bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->bottom);
     }
 }
 
@@ -63,11 +63,11 @@ static int stream_prepare(Job *job)
     StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
     BlockJob *bjob = &s->common;
     BlockDriverState *bs = blk_bs(bjob->blk);
-    BlockDriverState *base = s->base;
+    BlockDriverState *base = backing_bs(s->bottom);
     Error *local_err = NULL;
     int ret = 0;
 
-    bdrv_unfreeze_backing_chain(bs, base);
+    bdrv_unfreeze_backing_chain(bs, s->bottom);
     s->chain_frozen = false;
 
     if (bs->backing) {
@@ -110,7 +110,7 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
     StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
     BlockBackend *blk = s->common.blk;
     BlockDriverState *bs = blk_bs(blk);
-    BlockDriverState *base = s->base;
+    bool enable_cor = !backing_bs(s->bottom);
     int64_t len;
     int64_t offset = 0;
     uint64_t delay_ns = 0;
@@ -119,7 +119,8 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
     int64_t n = 0; /* bytes */
     void *buf;
 
-    if (!bs->backing) {
+    if (bs == s->bottom) {
+        /* Nothing to stream */
         return 0;
     }
 
@@ -136,7 +137,7 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
      * backing chain since the copy-on-read operation does not take base into
      * account.
      */
-    if (!base) {
+    if (enable_cor) {
         bdrv_enable_copy_on_read(bs);
     }
 
@@ -159,9 +160,8 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
         } else if (ret >= 0) {
             /* Copy if allocated in the intermediate images.  Limit to the
              * known-unallocated area [offset, offset+n*BDRV_SECTOR_SIZE).  */
-            ret = bdrv_is_allocated_above(backing_bs(bs), base, false,
+            ret = bdrv_is_allocated_above(backing_bs(bs), s->bottom, true,
                                           offset, n, &n);
-
             /* Finish early if end of backing file has been reached */
             if (ret == 0 && n == 0) {
                 n = len - offset;
@@ -198,7 +198,7 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
         }
     }
 
-    if (!base) {
+    if (enable_cor) {
         bdrv_disable_copy_on_read(bs);
     }
 
@@ -230,8 +230,10 @@ void stream_start(const char *job_id, BlockDriverState *bs,
     StreamBlockJob *s;
     BlockDriverState *iter;
     bool bs_read_only;
+    int basic_flags = BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGED;
+    BlockDriverState *bottom = bdrv_find_overlay(bs, base);
 
-    if (bdrv_freeze_backing_chain(bs, base, errp) < 0) {
+    if (bdrv_freeze_backing_chain(bs, bottom, errp) < 0) {
         return;
     }
 
@@ -248,10 +250,8 @@ void stream_start(const char *job_id, BlockDriverState *bs,
      * already have our own plans. Also don't allow resize as the image size is
      * queried only at the job start and then cached. */
     s = block_job_create(job_id, &stream_job_driver, NULL, bs,
-                         BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGED |
-                         BLK_PERM_GRAPH_MOD,
-                         BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGED |
-                         BLK_PERM_WRITE,
+                         basic_flags | BLK_PERM_GRAPH_MOD,
+                         basic_flags | BLK_PERM_WRITE,
                          speed, creation_flags, NULL, NULL, errp);
     if (!s) {
         goto fail;
@@ -259,15 +259,18 @@ void stream_start(const char *job_id, BlockDriverState *bs,
 
     /* Block all intermediate nodes between bs and base, because they will
      * disappear from the chain after this operation. The streaming job reads
-     * every block only once, assuming that it doesn't change, so block writes
-     * and resizes. */
+     * every block only once, assuming that it doesn't change, so forbid writes
+     * and resizes. Reassign the base node pointer because the backing BS of the
+     * bottom node might change after the call to bdrv_reopen_set_read_only()
+     * due to parallel block jobs running.
+     */
+    base = backing_bs(bottom);
     for (iter = backing_bs(bs); iter && iter != base; iter = backing_bs(iter)) {
         block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                           BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGED,
-                           &error_abort);
+                           basic_flags, &error_abort);
     }
 
-    s->base = base;
+    s->bottom = bottom;
     s->backing_file_str = g_strdup(backing_file_str);
     s->bs_read_only = bs_read_only;
     s->chain_frozen = true;
diff --git a/tests/qemu-iotests/245 b/tests/qemu-iotests/245
index 349b94a..bc1ceb9 100644
--- a/tests/qemu-iotests/245
+++ b/tests/qemu-iotests/245
@@ -866,9 +866,9 @@ class TestBlockdevReopen(iotests.QMPTestCase):
                              auto_finalize = False)
         self.assert_qmp(result, 'return', {})
 
-        # We can't remove hd2 while the stream job is ongoing
+        # We can remove hd2 while the stream job is ongoing
         opts['backing']['backing'] = None
-        self.reopen(opts, {}, "Cannot change 'backing' link from 'hd1' to 'hd2'")
+        self.reopen(opts, {})
 
         # We can't remove hd1 while the stream job is ongoing
         opts['backing'] = None
-- 
1.8.3.1

Re: [Qemu-devel] [PATCH v7 1/3] block: include base when checking image chain for block allocation

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 2313 bytes --]

On 29.05.19 19:56, Andrey Shinkevich wrote:
> This patch is used in the 'block/stream: introduce a bottom node'
> that is following. Instead of the base node, the caller may pass
> the node that has the base as its backing image to the function
> bdrv_is_allocated_above() with a new parameter include_base = true
> and get rid of the dependency on the base that may change during
> commit/stream parallel jobs. Now, if the specified base is not
> found in the backing image chain, the QEMU will abort.
> 
> Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> Reviewed-by: Alberto Garcia <berto@igalia.com>
> ---
>  block/commit.c        |  2 +-
>  block/io.c            | 21 +++++++++++++++------
>  block/mirror.c        |  2 +-
>  block/replication.c   |  2 +-
>  block/stream.c        |  2 +-
>  include/block/block.h |  3 ++-
>  6 files changed, 21 insertions(+), 11 deletions(-)

This needs the following hunk squashed in so it still compiles:

(I can do that, if you agree.)

diff --git a/block/qcow2.c b/block/qcow2.c
index 9396d490d5..2a59eb27fe 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -2148,7 +2148,8 @@ static bool is_unallocated(BlockDriverState *bs,
int64_t offset, int64_t bytes)
 {
     int64_t nr;
     return !bytes ||
-        (!bdrv_is_allocated_above(bs, NULL, offset, bytes, &nr) && nr
== bytes);
+        (!bdrv_is_allocated_above(bs, NULL, false, offset, bytes, &nr) &&
+         nr == bytes);
 }

 static bool is_zero_cow(BlockDriverState *bs, QCowL2Meta *m)
diff --git a/qemu-img.c b/qemu-img.c
index 158b3a505f..79983772de 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -3518,7 +3518,7 @@ static int img_rebase(int argc, char **argv)
                  * to take action
                  */
                 ret = bdrv_is_allocated_above(backing_bs(bs),
prefix_chain_bs,
-                                              offset, n, &n);
+                                              false, offset, n, &n);
                 if (ret < 0) {
                     error_report("error while reading image metadata: %s",
                                  strerror(-ret));


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v7 0/3] block/stream: get rid of the base

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 2035 bytes --]

On 29.05.19 19:56, Andrey Shinkevich wrote:
> This series introduces a bottom intermediate node that eliminates the
> dependency on the base that may change while stream job is running.
> It happens when stream/commit parallel jobs are running on the same
> backing chain. The base node of the stream job may be a top node of
> the parallel commit job and can change before the stream job is
> completed. We avoid that dependency by introducing the bottom node.
> 
> v7: [resend by Andrey]
>   01: assert(intermediate) was inserted before the call to
>       bdrv_is_allocated() in the intermediate node loop of the
>       bdrv_is_allocated_above() as suggested by Max.
>   02: The change of the intermediate node loop in the stream_start() was
>       rolled back to its original design and the reassignment of the base
>       node pointer was added as Vladimir and Max suggested. The relevant
>       comment was amended.
> 
> v6: [resend by Vladimir]
>   01: improve comment in block/io.c, suggested by Alberto
> 
> v5: [resend by Vladimir]
>   01: use comment wording in block/io.c suggested by Alberto
> 
> v4:
> trace_stream_start reverted to the base.
> bdrv_is_allocated_above_inclusive() deleted and the new parameter
> 'bool include_base' was added to the bdrv_is_allocated_above().
> 
> Andrey Shinkevich (3):
>   block: include base when checking image chain for block allocation
>   block/stream: refactor stream_run: drop goto
>   block/stream: introduce a bottom node
> 
>  block/commit.c         |  2 +-
>  block/io.c             | 21 +++++++++++++------
>  block/mirror.c         |  2 +-
>  block/replication.c    |  2 +-
>  block/stream.c         | 56 ++++++++++++++++++++++++--------------------------
>  include/block/block.h  |  3 ++-
>  tests/qemu-iotests/245 |  4 ++--
>  7 files changed, 49 insertions(+), 41 deletions(-)

Reviewed-by: Max Reitz <mreitz@redhat.com>

Just needs some simple changes to patch 1 to rebase it on 863cc78f1b3
and c8bb23cbdbe.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v7 1/3] block: include base when checking image chain for block allocation

[Vladimir Sementsov-Ogievskiy]

19.06.2019 22:27, Max Reitz wrote:
> On 29.05.19 19:56, Andrey Shinkevich wrote:
>> This patch is used in the 'block/stream: introduce a bottom node'
>> that is following. Instead of the base node, the caller may pass
>> the node that has the base as its backing image to the function
>> bdrv_is_allocated_above() with a new parameter include_base = true
>> and get rid of the dependency on the base that may change during
>> commit/stream parallel jobs. Now, if the specified base is not
>> found in the backing image chain, the QEMU will abort.
>>
>> Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> Reviewed-by: Alberto Garcia <berto@igalia.com>
>> ---
>>   block/commit.c        |  2 +-
>>   block/io.c            | 21 +++++++++++++++------
>>   block/mirror.c        |  2 +-
>>   block/replication.c   |  2 +-
>>   block/stream.c        |  2 +-
>>   include/block/block.h |  3 ++-
>>   6 files changed, 21 insertions(+), 11 deletions(-)
> 
> This needs the following hunk squashed in so it still compiles:
> 
> (I can do that, if you agree.)

It will be great, thanks! (Andrey is on vocation now)

> 
> diff --git a/block/qcow2.c b/block/qcow2.c
> index 9396d490d5..2a59eb27fe 100644
> --- a/block/qcow2.c
> +++ b/block/qcow2.c
> @@ -2148,7 +2148,8 @@ static bool is_unallocated(BlockDriverState *bs,
> int64_t offset, int64_t bytes)
>   {
>       int64_t nr;
>       return !bytes ||
> -        (!bdrv_is_allocated_above(bs, NULL, offset, bytes, &nr) && nr
> == bytes);
> +        (!bdrv_is_allocated_above(bs, NULL, false, offset, bytes, &nr) &&
> +         nr == bytes);
>   }
> 
>   static bool is_zero_cow(BlockDriverState *bs, QCowL2Meta *m)
> diff --git a/qemu-img.c b/qemu-img.c
> index 158b3a505f..79983772de 100644
> --- a/qemu-img.c
> +++ b/qemu-img.c
> @@ -3518,7 +3518,7 @@ static int img_rebase(int argc, char **argv)
>                    * to take action
>                    */
>                   ret = bdrv_is_allocated_above(backing_bs(bs),
> prefix_chain_bs,
> -                                              offset, n, &n);
> +                                              false, offset, n, &n);
>                   if (ret < 0) {
>                       error_report("error while reading image metadata: %s",
>                                    strerror(-ret));
> 


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH v7 1/3] block: include base when checking image chain for block allocation

[Andrey Shinkevich]

On 19/06/2019 22:27, Max Reitz wrote:
> On 29.05.19 19:56, Andrey Shinkevich wrote:
>> This patch is used in the 'block/stream: introduce a bottom node'
>> that is following. Instead of the base node, the caller may pass
>> the node that has the base as its backing image to the function
>> bdrv_is_allocated_above() with a new parameter include_base = true
>> and get rid of the dependency on the base that may change during
>> commit/stream parallel jobs. Now, if the specified base is not
>> found in the backing image chain, the QEMU will abort.
>>
>> Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> Reviewed-by: Alberto Garcia <berto@igalia.com>
>> ---
>>   block/commit.c        |  2 +-
>>   block/io.c            | 21 +++++++++++++++------
>>   block/mirror.c        |  2 +-
>>   block/replication.c   |  2 +-
>>   block/stream.c        |  2 +-
>>   include/block/block.h |  3 ++-
>>   6 files changed, 21 insertions(+), 11 deletions(-)
> 
> This needs the following hunk squashed in so it still compiles:
> 
> (I can do that, if you agree.)

Yes, please!
Andrey

> 
> diff --git a/block/qcow2.c b/block/qcow2.c
> index 9396d490d5..2a59eb27fe 100644
> --- a/block/qcow2.c
> +++ b/block/qcow2.c
> @@ -2148,7 +2148,8 @@ static bool is_unallocated(BlockDriverState *bs,
> int64_t offset, int64_t bytes)
>   {
>       int64_t nr;
>       return !bytes ||
> -        (!bdrv_is_allocated_above(bs, NULL, offset, bytes, &nr) && nr
> == bytes);
> +        (!bdrv_is_allocated_above(bs, NULL, false, offset, bytes, &nr) &&
> +         nr == bytes);
>   }
> 
>   static bool is_zero_cow(BlockDriverState *bs, QCowL2Meta *m)
> diff --git a/qemu-img.c b/qemu-img.c
> index 158b3a505f..79983772de 100644
> --- a/qemu-img.c
> +++ b/qemu-img.c
> @@ -3518,7 +3518,7 @@ static int img_rebase(int argc, char **argv)
>                    * to take action
>                    */
>                   ret = bdrv_is_allocated_above(backing_bs(bs),
> prefix_chain_bs,
> -                                              offset, n, &n);
> +                                              false, offset, n, &n);
>                   if (ret < 0) {
>                       error_report("error while reading image metadata: %s",
>                                    strerror(-ret));
> 

-- 
With the best regards,
Andrey Shinkevich

Re: [Qemu-devel] [PATCH v7 0/3] block/stream: get rid of the base

[Andrey Shinkevich]

On 19/06/2019 22:29, Max Reitz wrote:
> On 29.05.19 19:56, Andrey Shinkevich wrote:
>> This series introduces a bottom intermediate node that eliminates the
>> dependency on the base that may change while stream job is running.
>> It happens when stream/commit parallel jobs are running on the same
>> backing chain. The base node of the stream job may be a top node of
>> the parallel commit job and can change before the stream job is
>> completed. We avoid that dependency by introducing the bottom node.
>>
>> v7: [resend by Andrey]
>>    01: assert(intermediate) was inserted before the call to
>>        bdrv_is_allocated() in the intermediate node loop of the
>>        bdrv_is_allocated_above() as suggested by Max.
>>    02: The change of the intermediate node loop in the stream_start() was
>>        rolled back to its original design and the reassignment of the base
>>        node pointer was added as Vladimir and Max suggested. The relevant
>>        comment was amended.
>>
>> v6: [resend by Vladimir]
>>    01: improve comment in block/io.c, suggested by Alberto
>>
>> v5: [resend by Vladimir]
>>    01: use comment wording in block/io.c suggested by Alberto
>>
>> v4:
>> trace_stream_start reverted to the base.
>> bdrv_is_allocated_above_inclusive() deleted and the new parameter
>> 'bool include_base' was added to the bdrv_is_allocated_above().
>>
>> Andrey Shinkevich (3):
>>    block: include base when checking image chain for block allocation
>>    block/stream: refactor stream_run: drop goto
>>    block/stream: introduce a bottom node
>>
>>   block/commit.c         |  2 +-
>>   block/io.c             | 21 +++++++++++++------
>>   block/mirror.c         |  2 +-
>>   block/replication.c    |  2 +-
>>   block/stream.c         | 56 ++++++++++++++++++++++++--------------------------
>>   include/block/block.h  |  3 ++-
>>   tests/qemu-iotests/245 |  4 ++--
>>   7 files changed, 49 insertions(+), 41 deletions(-)
> 
> Reviewed-by: Max Reitz <mreitz@redhat.com>
> 
> Just needs some simple changes to patch 1 to rebase it on 863cc78f1b3
> and c8bb23cbdbe.
> 
> Max
> 

Please let us know who would take this series for pull request?

Andrey
-- 
With the best regards,
Andrey Shinkevich

Re: [Qemu-devel] [PATCH v7 0/3] block/stream: get rid of the base

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 2769 bytes --]

On 25.06.19 16:40, Andrey Shinkevich wrote:
> 
> 
> On 19/06/2019 22:29, Max Reitz wrote:
>> On 29.05.19 19:56, Andrey Shinkevich wrote:
>>> This series introduces a bottom intermediate node that eliminates the
>>> dependency on the base that may change while stream job is running.
>>> It happens when stream/commit parallel jobs are running on the same
>>> backing chain. The base node of the stream job may be a top node of
>>> the parallel commit job and can change before the stream job is
>>> completed. We avoid that dependency by introducing the bottom node.
>>>
>>> v7: [resend by Andrey]
>>>    01: assert(intermediate) was inserted before the call to
>>>        bdrv_is_allocated() in the intermediate node loop of the
>>>        bdrv_is_allocated_above() as suggested by Max.
>>>    02: The change of the intermediate node loop in the stream_start() was
>>>        rolled back to its original design and the reassignment of the base
>>>        node pointer was added as Vladimir and Max suggested. The relevant
>>>        comment was amended.
>>>
>>> v6: [resend by Vladimir]
>>>    01: improve comment in block/io.c, suggested by Alberto
>>>
>>> v5: [resend by Vladimir]
>>>    01: use comment wording in block/io.c suggested by Alberto
>>>
>>> v4:
>>> trace_stream_start reverted to the base.
>>> bdrv_is_allocated_above_inclusive() deleted and the new parameter
>>> 'bool include_base' was added to the bdrv_is_allocated_above().
>>>
>>> Andrey Shinkevich (3):
>>>    block: include base when checking image chain for block allocation
>>>    block/stream: refactor stream_run: drop goto
>>>    block/stream: introduce a bottom node
>>>
>>>   block/commit.c         |  2 +-
>>>   block/io.c             | 21 +++++++++++++------
>>>   block/mirror.c         |  2 +-
>>>   block/replication.c    |  2 +-
>>>   block/stream.c         | 56 ++++++++++++++++++++++++--------------------------
>>>   include/block/block.h  |  3 ++-
>>>   tests/qemu-iotests/245 |  4 ++--
>>>   7 files changed, 49 insertions(+), 41 deletions(-)
>>
>> Reviewed-by: Max Reitz <mreitz@redhat.com>
>>
>> Just needs some simple changes to patch 1 to rebase it on 863cc78f1b3
>> and c8bb23cbdbe.
>>
>> Max
>>
> 
> Please let us know who would take this series for pull request?

Nothing, I was just waiting for my current pull request (lingering from
last week due to a build failure because of one of the patches) to be
processed.

I suppose I won’t wait longer and just take this series now and deal
with the fallout later if there’s something else in my last week’s pull
request that needs fixing...

So: Thanks, applied to my block branch:

https://git.xanclic.moe/XanClic/qemu/commits/branch/block

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v7 0/3] block/stream: get rid of the base

[Andrey Shinkevich]

On 25/06/2019 18:03, Max Reitz wrote:
> On 25.06.19 16:40, Andrey Shinkevich wrote:
>>
>>
>> On 19/06/2019 22:29, Max Reitz wrote:
>>> On 29.05.19 19:56, Andrey Shinkevich wrote:
>>>> This series introduces a bottom intermediate node that eliminates the
>>>> dependency on the base that may change while stream job is running.
>>>> It happens when stream/commit parallel jobs are running on the same
>>>> backing chain. The base node of the stream job may be a top node of
>>>> the parallel commit job and can change before the stream job is
>>>> completed. We avoid that dependency by introducing the bottom node.
>>>>
>>>> v7: [resend by Andrey]
>>>>     01: assert(intermediate) was inserted before the call to
>>>>         bdrv_is_allocated() in the intermediate node loop of the
>>>>         bdrv_is_allocated_above() as suggested by Max.
>>>>     02: The change of the intermediate node loop in the stream_start() was
>>>>         rolled back to its original design and the reassignment of the base
>>>>         node pointer was added as Vladimir and Max suggested. The relevant
>>>>         comment was amended.
>>>>
>>>> v6: [resend by Vladimir]
>>>>     01: improve comment in block/io.c, suggested by Alberto
>>>>
>>>> v5: [resend by Vladimir]
>>>>     01: use comment wording in block/io.c suggested by Alberto
>>>>
>>>> v4:
>>>> trace_stream_start reverted to the base.
>>>> bdrv_is_allocated_above_inclusive() deleted and the new parameter
>>>> 'bool include_base' was added to the bdrv_is_allocated_above().
>>>>
>>>> Andrey Shinkevich (3):
>>>>     block: include base when checking image chain for block allocation
>>>>     block/stream: refactor stream_run: drop goto
>>>>     block/stream: introduce a bottom node
>>>>
>>>>    block/commit.c         |  2 +-
>>>>    block/io.c             | 21 +++++++++++++------
>>>>    block/mirror.c         |  2 +-
>>>>    block/replication.c    |  2 +-
>>>>    block/stream.c         | 56 ++++++++++++++++++++++++--------------------------
>>>>    include/block/block.h  |  3 ++-
>>>>    tests/qemu-iotests/245 |  4 ++--
>>>>    7 files changed, 49 insertions(+), 41 deletions(-)
>>>
>>> Reviewed-by: Max Reitz <mreitz@redhat.com>
>>>
>>> Just needs some simple changes to patch 1 to rebase it on 863cc78f1b3
>>> and c8bb23cbdbe.
>>>
>>> Max
>>>
>>
>> Please let us know who would take this series for pull request?
> 
> Nothing, I was just waiting for my current pull request (lingering from
> last week due to a build failure because of one of the patches) to be
> processed.
> 
> I suppose I won’t wait longer and just take this series now and deal
> with the fallout later if there’s something else in my last week’s pull
> request that needs fixing...
> 
> So: Thanks, applied to my block branch:
> 
> https://git.xanclic.moe/XanClic/qemu/commits/branch/block
> 
> Max
> 

Thanks a lot, Max.

Andrey
-- 
With the best regards,
Andrey Shinkevich