[Qemu-devel] [PATCH v2 00/18] bitmaps: introduce 'bitmap' sync mode

[Qemu-devel] [PATCH v2 00/18] bitmaps: introduce 'bitmap' sync mode

[John Snow]

This series adds a new "BITMAP" sync mode that is meant to replace the
existing "INCREMENTAL" sync mode.

This mode can have its behavior modified by issuing any of three bitmap sync
modes, passed as arguments to the job.

The three bitmap sync modes are:
- CONDITIONAL: This is an alias for the old incremental mode. The bitmap is
               conditionally synchronized based on the return code of the job
               upon completion.
- NEVER: This is, effectively, the differential backup mode. It never clears
         the bitmap, as the name suggests.
- ALWAYS: Here is the new, exciting thing. The bitmap is always synchronized,
          even on failure. On success, this is identical to incremental, but
          on failure it clears only the bits that were copied successfully.
          This can be used to "resume" incremental backups from later points
          in times.

I wrote this series by accident on my way to implement incremental mode
for mirror, but this happened first -- the problem is that Mirror mode
uses its existing modes in a very particular way; and this was the best
way to add bitmap support into the mirror job properly.

Summary:
- 01-03: refactor blockdev-backup and drive-backup to share more interface code
- 04-05: add the new 'bitmap' sync mode with sync policy 'conditional',
         which is functionally identical to 'incremental' sync mode.
- 06:    add sync policy 'never' ("Differential" backups.)
- 07-11: rework some merging code to facilite patch 12;
- 12:    add sync policy 'always' ("Resumable" backups)
- 13-16: test infrastructure changes to support patch 16:
- 17:    new iotest!
- 18:    minor policy loosening as a QOL improvement

Future work:
 - Update bitmaps.rst to explain these. (WIP, it's hard, sorry!)
 - Add these modes to Mirror. (Done*, but needs tests.)
 - Allow the use of bitmaps and bitmap sync modes with non-BITMAP modes;
   This will allow for resumable/re-tryable full backups.

===
V2:
===

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/18:[down] 'qapi/block-core: Introduce BackupCommon'
002/18:[down] 'drive-backup: create do_backup_common'
003/18:[down] 'blockdev-backup: utilize do_backup_common'
004/18:[0006] [FC] 'qapi: add BitmapSyncMode enum'
005/18:[0076] [FC] 'block/backup: Add mirror sync mode 'bitmap''
006/18:[0003] [FC] 'block/backup: add 'never' policy to bitmap sync mode'
007/18:[0008] [FC] 'hbitmap: Fix merge when b is empty, and result is not an alias of a'
008/18:[0018] [FC] 'hbitmap: enable merging across granularities'
009/18:[down] 'block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal'
010/18:[down] 'block/dirty-bitmap: add bdrv_dirty_bitmap_get'
011/18:[down] 'block/backup: upgrade copy_bitmap to BdrvDirtyBitmap'
012/18:[0031] [FC] 'block/backup: add 'always' bitmap sync policy'
013/18:[----] [--] 'iotests: add testing shim for script-style python tests'
014/18:[----] [--] 'iotests: teach run_job to cancel pending jobs'
015/18:[0016] [FC] 'iotests: teach FilePath to produce multiple paths'
016/18:[down] 'iotests: Add virtio-scsi device helper'
017/18:[0319] [FC] 'iotests: add test 257 for bitmap-mode backups'
018/18:[0002] [FC] 'block/backup: loosen restriction on readonly bitmaps'

Changes:
004: Fixed typo
     Change @conditional docstring
005: Moved desugaring code into blockdev.c, facilitated by patches 1-3.
006: Change @never docstring slightly.
007: Merge will clear the target bitmap when both components bitmaps are empty,
     and the target bitmap is not an alias of either component bitmap.
008: Check orig_size (logical size) instead of size (actual size) to enable
         cross-granularity merging.
     Fix the sparse merge itself, based on the block/backup code.
     Clear the target bitmap before cross-granularity merge.
     Assert the size is itself equal when logical size and granularities are
         equal.
---: Dropped bdrv_dirty_bitmap_claim.
012: Rewrote the cleanup logic to hopefully be clearer.
     use merge intsead of dropped reclaim.
015: Changed docstring
     factored out filename pattern generation.
017: Fix mkpattern indent.
     Use integer division!!!
     Add parenthesis to boolean assignment
     Change test run ordering; update output to reflect this
     Use virtio-scsi-ccw when appropriate
     Update test output to reflect new test running order
018: Fallout from patches 1-3; restrictions only need loosened in one place
         instead of two.

Changes not made:
- Allowing 'cancel' to skip synchronization on cancel:
  Decided against it, opting for consistency. The user asked for a sync,
  and it's simpler logistically to execute on that desire.
  Use the new mode carefully, please!

John Snow (18):
  qapi/block-core: Introduce BackupCommon
  drive-backup: create do_backup_common
  blockdev-backup: utilize do_backup_common
  qapi: add BitmapSyncMode enum
  block/backup: Add mirror sync mode 'bitmap'
  block/backup: add 'never' policy to bitmap sync mode
  hbitmap: Fix merge when b is empty, and result is not an alias of a
  hbitmap: enable merging across granularities
  block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal
  block/dirty-bitmap: add bdrv_dirty_bitmap_get
  block/backup: upgrade copy_bitmap to BdrvDirtyBitmap
  block/backup: add 'always' bitmap sync policy
  iotests: add testing shim for script-style python tests
  iotests: teach run_job to cancel pending jobs
  iotests: teach FilePath to produce multiple paths
  iotests: Add virtio-scsi device helper
  iotests: add test 257 for bitmap-mode backups
  block/backup: loosen restriction on readonly bitmaps

 block/backup.c                |  126 +-
 block/dirty-bitmap.c          |   69 +-
 block/mirror.c                |    8 +-
 block/replication.c           |    2 +-
 blockdev.c                    |  226 ++--
 include/block/block_int.h     |    7 +-
 include/block/dirty-bitmap.h  |    4 +-
 migration/block.c             |    5 +-
 nbd/server.c                  |    2 +-
 qapi/block-core.json          |  137 +-
 tests/qemu-iotests/040        |    6 +-
 tests/qemu-iotests/093        |    6 +-
 tests/qemu-iotests/139        |    7 +-
 tests/qemu-iotests/238        |    5 +-
 tests/qemu-iotests/257        |  409 ++++++
 tests/qemu-iotests/257.out    | 2199 +++++++++++++++++++++++++++++++++
 tests/qemu-iotests/group      |    1 +
 tests/qemu-iotests/iotests.py |   99 +-
 util/hbitmap.c                |   49 +-
 19 files changed, 3041 insertions(+), 326 deletions(-)
 create mode 100755 tests/qemu-iotests/257
 create mode 100644 tests/qemu-iotests/257.out

-- 
2.21.0

[Qemu-devel] [PATCH v2 01/18] qapi/block-core: Introduce BackupCommon

[John Snow]

drive-backup and blockdev-backup have an awful lot of things in common
that are the same. Let's fix that.

I don't deduplicate 'target', because the semantics actually did change
between each structure. Leave that one alone so it can be documented
separately.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 qapi/block-core.json | 103 ++++++++++++++-----------------------------
 1 file changed, 33 insertions(+), 70 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index 0d43d4f37c..7b23efcf13 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1315,32 +1315,23 @@
   'data': { 'node': 'str', 'overlay': 'str' } }
 
 ##
-# @DriveBackup:
+# @BackupCommon:
 #
 # @job-id: identifier for the newly-created block job. If
 #          omitted, the device name will be used. (Since 2.7)
 #
 # @device: the device name or node-name of a root node which should be copied.
 #
-# @target: the target of the new image. If the file exists, or if it
-#          is a device, the existing file/device will be used as the new
-#          destination.  If it does not exist, a new file will be created.
-#
-# @format: the format of the new destination, default is to
-#          probe if @mode is 'existing', else the format of the source
-#
 # @sync: what parts of the disk image should be copied to the destination
 #        (all the disk, only the sectors allocated in the topmost image, from a
 #        dirty bitmap, or only new I/O).
 #
-# @mode: whether and how QEMU should create a new image, default is
-#        'absolute-paths'.
-#
-# @speed: the maximum speed, in bytes per second
+# @speed: the maximum speed, in bytes per second. The default is 0,
+#         for unlimited.
 #
 # @bitmap: the name of dirty bitmap if sync is "incremental".
 #          Must be present if sync is "incremental", must NOT be present
-#          otherwise. (Since 2.4)
+#          otherwise. (Since 2.4 (Drive), 3.1 (Blockdev))
 #
 # @compress: true to compress data, if the target format supports it.
 #            (default: false) (since 2.8)
@@ -1372,73 +1363,45 @@
 #
 # Since: 1.6
 ##
+{ 'struct': 'BackupCommon',
+  'data': { '*job-id': 'str', 'device': 'str',
+            'sync': 'MirrorSyncMode', '*speed': 'int',
+            '*bitmap': 'str', '*compress': 'bool',
+            '*on-source-error': 'BlockdevOnError',
+            '*on-target-error': 'BlockdevOnError',
+            '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
+
+##
+# @DriveBackup:
+#
+# @target: the target of the new image. If the file exists, or if it
+#          is a device, the existing file/device will be used as the new
+#          destination.  If it does not exist, a new file will be created.
+#
+# @format: the format of the new destination, default is to
+#          probe if @mode is 'existing', else the format of the source
+#
+# @mode: whether and how QEMU should create a new image, default is
+#        'absolute-paths'.
+#
+# Since: 1.6
+##
 { 'struct': 'DriveBackup',
-  'data': { '*job-id': 'str', 'device': 'str', 'target': 'str',
-            '*format': 'str', 'sync': 'MirrorSyncMode',
-            '*mode': 'NewImageMode', '*speed': 'int',
-            '*bitmap': 'str', '*compress': 'bool',
-            '*on-source-error': 'BlockdevOnError',
-            '*on-target-error': 'BlockdevOnError',
-            '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
+  'base': 'BackupCommon',
+  'data': { 'target': 'str',
+            '*format': 'str',
+            '*mode': 'NewImageMode' } }
 
 ##
 # @BlockdevBackup:
 #
-# @job-id: identifier for the newly-created block job. If
-#          omitted, the device name will be used. (Since 2.7)
-#
-# @device: the device name or node-name of a root node which should be copied.
-#
 # @target: the device name or node-name of the backup target node.
 #
-# @sync: what parts of the disk image should be copied to the destination
-#        (all the disk, only the sectors allocated in the topmost image, or
-#        only new I/O).
-#
-# @speed: the maximum speed, in bytes per second. The default is 0,
-#         for unlimited.
-#
-# @bitmap: the name of dirty bitmap if sync is "incremental".
-#          Must be present if sync is "incremental", must NOT be present
-#          otherwise. (Since 3.1)
-#
-# @compress: true to compress data, if the target format supports it.
-#            (default: false) (since 2.8)
-#
-# @on-source-error: the action to take on an error on the source,
-#                   default 'report'.  'stop' and 'enospc' can only be used
-#                   if the block device supports io-status (see BlockInfo).
-#
-# @on-target-error: the action to take on an error on the target,
-#                   default 'report' (no limitations, since this applies to
-#                   a different block device than @device).
-#
-# @auto-finalize: When false, this job will wait in a PENDING state after it has
-#                 finished its work, waiting for @block-job-finalize before
-#                 making any block graph changes.
-#                 When true, this job will automatically
-#                 perform its abort or commit actions.
-#                 Defaults to true. (Since 2.12)
-#
-# @auto-dismiss: When false, this job will wait in a CONCLUDED state after it
-#                has completely ceased all work, and awaits @block-job-dismiss.
-#                When true, this job will automatically disappear from the query
-#                list without user intervention.
-#                Defaults to true. (Since 2.12)
-#
-# Note: @on-source-error and @on-target-error only affect background
-# I/O.  If an error occurs during a guest write request, the device's
-# rerror/werror actions will be used.
-#
 # Since: 2.3
 ##
 { 'struct': 'BlockdevBackup',
-  'data': { '*job-id': 'str', 'device': 'str', 'target': 'str',
-            'sync': 'MirrorSyncMode', '*speed': 'int',
-            '*bitmap': 'str', '*compress': 'bool',
-            '*on-source-error': 'BlockdevOnError',
-            '*on-target-error': 'BlockdevOnError',
-            '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
+  'base': 'BackupCommon',
+  'data': { 'target': 'str' } }
 
 ##
 # @blockdev-snapshot-sync:
-- 
2.21.0

[Qemu-devel] [PATCH v2 02/18] drive-backup: create do_backup_common

[John Snow]

Create a common core that comprises the actual meat of what the backup API
boundary needs to do, and then switch drive-backup to use it.

Questions:
 - do_drive_backup now acquires and releases the aio_context in addition
   to do_backup_common doing the same. Can I drop this from drive_backup?
Signed-off-by: John Snow <jsnow@redhat.com>
---
 blockdev.c | 138 ++++++++++++++++++++++++++++++++---------------------
 1 file changed, 83 insertions(+), 55 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index 4d141e9a1f..5fd663a7e5 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -3425,6 +3425,86 @@ out:
     aio_context_release(aio_context);
 }
 
+static BlockJob *do_backup_common(BackupCommon *backup,
+                                  BlockDriverState *target_bs,
+                                  JobTxn *txn, Error **errp)
+{
+    BlockDriverState *bs;
+    BlockJob *job = NULL;
+    BdrvDirtyBitmap *bmap = NULL;
+    AioContext *aio_context;
+    Error *local_err = NULL;
+    int job_flags = JOB_DEFAULT;
+    int ret;
+
+    if (!backup->has_speed) {
+        backup->speed = 0;
+    }
+    if (!backup->has_on_source_error) {
+        backup->on_source_error = BLOCKDEV_ON_ERROR_REPORT;
+    }
+    if (!backup->has_on_target_error) {
+        backup->on_target_error = BLOCKDEV_ON_ERROR_REPORT;
+    }
+    if (!backup->has_job_id) {
+        backup->job_id = NULL;
+    }
+    if (!backup->has_auto_finalize) {
+        backup->auto_finalize = true;
+    }
+    if (!backup->has_auto_dismiss) {
+        backup->auto_dismiss = true;
+    }
+    if (!backup->has_compress) {
+        backup->compress = false;
+    }
+
+    bs = bdrv_lookup_bs(backup->device, backup->device, errp);
+    if (!bs) {
+        return NULL;
+    }
+
+    aio_context = bdrv_get_aio_context(bs);
+    aio_context_acquire(aio_context);
+
+    ret = bdrv_try_set_aio_context(target_bs, aio_context, errp);
+    if (ret < 0) {
+        bdrv_unref(target_bs);
+        goto out;
+    }
+
+    if (backup->has_bitmap) {
+        bmap = bdrv_find_dirty_bitmap(bs, backup->bitmap);
+        if (!bmap) {
+            error_setg(errp, "Bitmap '%s' could not be found", backup->bitmap);
+            goto out;
+        }
+        if (bdrv_dirty_bitmap_check(bmap, BDRV_BITMAP_DEFAULT, errp)) {
+            goto out;
+        }
+    }
+
+    if (!backup->auto_finalize) {
+        job_flags |= JOB_MANUAL_FINALIZE;
+    }
+    if (!backup->auto_dismiss) {
+        job_flags |= JOB_MANUAL_DISMISS;
+    }
+
+    job = backup_job_create(backup->job_id, bs, target_bs, backup->speed,
+                            backup->sync, bmap, backup->compress,
+                            backup->on_source_error, backup->on_target_error,
+                            job_flags, NULL, NULL, txn, &local_err);
+    if (local_err != NULL) {
+        error_propagate(errp, local_err);
+        goto out;
+    }
+
+out:
+    aio_context_release(aio_context);
+    return job;
+}
+
 static BlockJob *do_drive_backup(DriveBackup *backup, JobTxn *txn,
                                  Error **errp)
 {
@@ -3432,39 +3512,16 @@ static BlockJob *do_drive_backup(DriveBackup *backup, JobTxn *txn,
     BlockDriverState *target_bs;
     BlockDriverState *source = NULL;
     BlockJob *job = NULL;
-    BdrvDirtyBitmap *bmap = NULL;
     AioContext *aio_context;
     QDict *options = NULL;
     Error *local_err = NULL;
-    int flags, job_flags = JOB_DEFAULT;
+    int flags;
     int64_t size;
     bool set_backing_hd = false;
-    int ret;
 
-    if (!backup->has_speed) {
-        backup->speed = 0;
-    }
-    if (!backup->has_on_source_error) {
-        backup->on_source_error = BLOCKDEV_ON_ERROR_REPORT;
-    }
-    if (!backup->has_on_target_error) {
-        backup->on_target_error = BLOCKDEV_ON_ERROR_REPORT;
-    }
     if (!backup->has_mode) {
         backup->mode = NEW_IMAGE_MODE_ABSOLUTE_PATHS;
     }
-    if (!backup->has_job_id) {
-        backup->job_id = NULL;
-    }
-    if (!backup->has_auto_finalize) {
-        backup->auto_finalize = true;
-    }
-    if (!backup->has_auto_dismiss) {
-        backup->auto_dismiss = true;
-    }
-    if (!backup->has_compress) {
-        backup->compress = false;
-    }
 
     bs = bdrv_lookup_bs(backup->device, backup->device, errp);
     if (!bs) {
@@ -3541,12 +3598,6 @@ static BlockJob *do_drive_backup(DriveBackup *backup, JobTxn *txn,
         goto out;
     }
 
-    ret = bdrv_try_set_aio_context(target_bs, aio_context, errp);
-    if (ret < 0) {
-        bdrv_unref(target_bs);
-        goto out;
-    }
-
     if (set_backing_hd) {
         bdrv_set_backing_hd(target_bs, source, &local_err);
         if (local_err) {
@@ -3554,31 +3605,8 @@ static BlockJob *do_drive_backup(DriveBackup *backup, JobTxn *txn,
         }
     }
 
-    if (backup->has_bitmap) {
-        bmap = bdrv_find_dirty_bitmap(bs, backup->bitmap);
-        if (!bmap) {
-            error_setg(errp, "Bitmap '%s' could not be found", backup->bitmap);
-            goto unref;
-        }
-        if (bdrv_dirty_bitmap_check(bmap, BDRV_BITMAP_DEFAULT, errp)) {
-            goto unref;
-        }
-    }
-    if (!backup->auto_finalize) {
-        job_flags |= JOB_MANUAL_FINALIZE;
-    }
-    if (!backup->auto_dismiss) {
-        job_flags |= JOB_MANUAL_DISMISS;
-    }
-
-    job = backup_job_create(backup->job_id, bs, target_bs, backup->speed,
-                            backup->sync, bmap, backup->compress,
-                            backup->on_source_error, backup->on_target_error,
-                            job_flags, NULL, NULL, txn, &local_err);
-    if (local_err != NULL) {
-        error_propagate(errp, local_err);
-        goto unref;
-    }
+    job = do_backup_common(qapi_DriveBackup_base(backup),
+                           target_bs, txn, errp);
 
 unref:
     bdrv_unref(target_bs);
-- 
2.21.0

[Qemu-devel] [PATCH v2 03/18] blockdev-backup: utilize do_backup_common

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 blockdev.c | 73 +++---------------------------------------------------
 1 file changed, 3 insertions(+), 70 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index 5fd663a7e5..cad300d526 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -3638,82 +3638,15 @@ XDbgBlockGraph *qmp_x_debug_query_block_graph(Error **errp)
 BlockJob *do_blockdev_backup(BlockdevBackup *backup, JobTxn *txn,
                              Error **errp)
 {
-    BlockDriverState *bs;
     BlockDriverState *target_bs;
-    Error *local_err = NULL;
-    BdrvDirtyBitmap *bmap = NULL;
-    AioContext *aio_context;
-    BlockJob *job = NULL;
-    int job_flags = JOB_DEFAULT;
-    int ret;
-
-    if (!backup->has_speed) {
-        backup->speed = 0;
-    }
-    if (!backup->has_on_source_error) {
-        backup->on_source_error = BLOCKDEV_ON_ERROR_REPORT;
-    }
-    if (!backup->has_on_target_error) {
-        backup->on_target_error = BLOCKDEV_ON_ERROR_REPORT;
-    }
-    if (!backup->has_job_id) {
-        backup->job_id = NULL;
-    }
-    if (!backup->has_auto_finalize) {
-        backup->auto_finalize = true;
-    }
-    if (!backup->has_auto_dismiss) {
-        backup->auto_dismiss = true;
-    }
-    if (!backup->has_compress) {
-        backup->compress = false;
-    }
-
-    bs = bdrv_lookup_bs(backup->device, backup->device, errp);
-    if (!bs) {
-        return NULL;
-    }
-
-    aio_context = bdrv_get_aio_context(bs);
-    aio_context_acquire(aio_context);
 
     target_bs = bdrv_lookup_bs(backup->target, backup->target, errp);
     if (!target_bs) {
-        goto out;
+        return NULL;
     }
 
-    ret = bdrv_try_set_aio_context(target_bs, aio_context, errp);
-    if (ret < 0) {
-        goto out;
-    }
-
-    if (backup->has_bitmap) {
-        bmap = bdrv_find_dirty_bitmap(bs, backup->bitmap);
-        if (!bmap) {
-            error_setg(errp, "Bitmap '%s' could not be found", backup->bitmap);
-            goto out;
-        }
-        if (bdrv_dirty_bitmap_check(bmap, BDRV_BITMAP_DEFAULT, errp)) {
-            goto out;
-        }
-    }
-
-    if (!backup->auto_finalize) {
-        job_flags |= JOB_MANUAL_FINALIZE;
-    }
-    if (!backup->auto_dismiss) {
-        job_flags |= JOB_MANUAL_DISMISS;
-    }
-    job = backup_job_create(backup->job_id, bs, target_bs, backup->speed,
-                            backup->sync, bmap, backup->compress,
-                            backup->on_source_error, backup->on_target_error,
-                            job_flags, NULL, NULL, txn, &local_err);
-    if (local_err != NULL) {
-        error_propagate(errp, local_err);
-    }
-out:
-    aio_context_release(aio_context);
-    return job;
+    return do_backup_common(qapi_BlockdevBackup_base(backup),
+                            target_bs, txn, errp);
 }
 
 void qmp_blockdev_backup(BlockdevBackup *arg, Error **errp)
-- 
2.21.0

[Qemu-devel] [PATCH v2 04/18] qapi: add BitmapSyncMode enum

[John Snow]

Depending on what a user is trying to accomplish, there might be a few
bitmap cleanup actions that occur when an operation is finished that
could be useful.

I am proposing three:
- NEVER: The bitmap is never synchronized against what was copied.
- ALWAYS: The bitmap is always synchronized, even on failures.
- CONDITIONAL: The bitmap is synchronized only on success.

The existing incremental backup modes use 'conditional' semantics,
so add just that one for right now.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 qapi/block-core.json | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index 7b23efcf13..87eba5a5d9 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1134,6 +1134,20 @@
 { 'enum': 'MirrorSyncMode',
   'data': ['top', 'full', 'none', 'incremental'] }
 
+##
+# @BitmapSyncMode:
+#
+# An enumeration of possible behaviors for the synchronization of a bitmap
+# when used for data copy operations.
+#
+# @conditional: The bitmap is only synced when the operation is successful.
+#               This is the behavior always used for 'INCREMENTAL' backups.
+#
+# Since: 4.2
+##
+{ 'enum': 'BitmapSyncMode',
+  'data': ['conditional'] }
+
 ##
 # @MirrorCopyMode:
 #
-- 
2.21.0

[Qemu-devel] [PATCH v2 05/18] block/backup: Add mirror sync mode 'bitmap'

[John Snow]

We don't need or want a new sync mode for simple differences in
semantics.  Create a new mode simply named "BITMAP" that is designed to
make use of the new Bitmap Sync Mode field.

Because the only bitmap mode is 'conditional', this adds no new
functionality to the backup job (yet). The old incremental backup mode
is maintained as a syntactic sugar for sync=bitmap, mode=conditional.

Add all of the plumbing necessary to support this new instruction.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 block/backup.c            | 20 ++++++++++++--------
 block/mirror.c            |  6 ++++--
 block/replication.c       |  2 +-
 blockdev.c                | 19 +++++++++++++++++--
 include/block/block_int.h |  4 +++-
 qapi/block-core.json      | 20 ++++++++++++++------
 6 files changed, 51 insertions(+), 20 deletions(-)

diff --git a/block/backup.c b/block/backup.c
index 715e1d3be8..996941fa61 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -38,9 +38,9 @@ typedef struct CowRequest {
 typedef struct BackupBlockJob {
     BlockJob common;
     BlockBackend *target;
-    /* bitmap for sync=incremental */
     BdrvDirtyBitmap *sync_bitmap;
     MirrorSyncMode sync_mode;
+    BitmapSyncMode bitmap_mode;
     BlockdevOnError on_source_error;
     BlockdevOnError on_target_error;
     CoRwlock flush_rwlock;
@@ -452,7 +452,7 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
 
     job_progress_set_remaining(job, s->len);
 
-    if (s->sync_mode == MIRROR_SYNC_MODE_INCREMENTAL) {
+    if (s->sync_mode == MIRROR_SYNC_MODE_BITMAP) {
         backup_incremental_init_copy_bitmap(s);
     } else {
         hbitmap_set(s->copy_bitmap, 0, s->len);
@@ -536,6 +536,7 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target,
 BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
                   BlockDriverState *target, int64_t speed,
                   MirrorSyncMode sync_mode, BdrvDirtyBitmap *sync_bitmap,
+                  BitmapSyncMode bitmap_mode,
                   bool compress,
                   BlockdevOnError on_source_error,
                   BlockdevOnError on_target_error,
@@ -583,10 +584,13 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
         return NULL;
     }
 
-    if (sync_mode == MIRROR_SYNC_MODE_INCREMENTAL) {
+    /* QMP interface should have handled translating this to bitmap mode */
+    assert(sync_mode != MIRROR_SYNC_MODE_INCREMENTAL);
+
+    if (sync_mode == MIRROR_SYNC_MODE_BITMAP) {
         if (!sync_bitmap) {
             error_setg(errp, "must provide a valid bitmap name for "
-                             "\"incremental\" sync mode");
+                       "'%s' sync mode", MirrorSyncMode_str(sync_mode));
             return NULL;
         }
 
@@ -596,8 +600,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
         }
     } else if (sync_bitmap) {
         error_setg(errp,
-                   "a sync_bitmap was provided to backup_run, "
-                   "but received an incompatible sync_mode (%s)",
+                   "a bitmap was given to backup_job_create, "
+                   "but it received an incompatible sync_mode (%s)",
                    MirrorSyncMode_str(sync_mode));
         return NULL;
     }
@@ -639,8 +643,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
     job->on_source_error = on_source_error;
     job->on_target_error = on_target_error;
     job->sync_mode = sync_mode;
-    job->sync_bitmap = sync_mode == MIRROR_SYNC_MODE_INCREMENTAL ?
-                       sync_bitmap : NULL;
+    job->sync_bitmap = sync_bitmap;
+    job->bitmap_mode = bitmap_mode;
     job->compress = compress;
 
     /* Detect image-fleecing (and similar) schemes */
diff --git a/block/mirror.c b/block/mirror.c
index d17be4cdbc..42b3d9acd0 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -1717,8 +1717,10 @@ void mirror_start(const char *job_id, BlockDriverState *bs,
     bool is_none_mode;
     BlockDriverState *base;
 
-    if (mode == MIRROR_SYNC_MODE_INCREMENTAL) {
-        error_setg(errp, "Sync mode 'incremental' not supported");
+    if ((mode == MIRROR_SYNC_MODE_INCREMENTAL) ||
+        (mode == MIRROR_SYNC_MODE_BITMAP)) {
+        error_setg(errp, "Sync mode '%s' not supported",
+                   MirrorSyncMode_str(mode));
         return;
     }
     is_none_mode = mode == MIRROR_SYNC_MODE_NONE;
diff --git a/block/replication.c b/block/replication.c
index b41bc507c0..5475217c69 100644
--- a/block/replication.c
+++ b/block/replication.c
@@ -543,7 +543,7 @@ static void replication_start(ReplicationState *rs, ReplicationMode mode,
 
         s->backup_job = backup_job_create(
                                 NULL, s->secondary_disk->bs, s->hidden_disk->bs,
-                                0, MIRROR_SYNC_MODE_NONE, NULL, false,
+                                0, MIRROR_SYNC_MODE_NONE, NULL, 0, false,
                                 BLOCKDEV_ON_ERROR_REPORT,
                                 BLOCKDEV_ON_ERROR_REPORT, JOB_INTERNAL,
                                 backup_job_completed, bs, NULL, &local_err);
diff --git a/blockdev.c b/blockdev.c
index cad300d526..d5b089a446 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -3473,6 +3473,19 @@ static BlockJob *do_backup_common(BackupCommon *backup,
         goto out;
     }
 
+    if (backup->sync == MIRROR_SYNC_MODE_INCREMENTAL) {
+        if (backup->has_bitmap_mode &&
+            backup->bitmap_mode != BITMAP_SYNC_MODE_CONDITIONAL) {
+            error_setg(errp, "Bitmap sync mode must be 'conditional' "
+                       "when using sync mode '%s'",
+                       MirrorSyncMode_str(backup->sync));
+            goto out;
+        }
+        backup->has_bitmap_mode = true;
+        backup->sync = MIRROR_SYNC_MODE_BITMAP;
+        backup->bitmap_mode = BITMAP_SYNC_MODE_CONDITIONAL;
+    }
+
     if (backup->has_bitmap) {
         bmap = bdrv_find_dirty_bitmap(bs, backup->bitmap);
         if (!bmap) {
@@ -3492,8 +3505,10 @@ static BlockJob *do_backup_common(BackupCommon *backup,
     }
 
     job = backup_job_create(backup->job_id, bs, target_bs, backup->speed,
-                            backup->sync, bmap, backup->compress,
-                            backup->on_source_error, backup->on_target_error,
+                            backup->sync, bmap, backup->bitmap_mode,
+                            backup->compress,
+                            backup->on_source_error,
+                            backup->on_target_error,
                             job_flags, NULL, NULL, txn, &local_err);
     if (local_err != NULL) {
         error_propagate(errp, local_err);
diff --git a/include/block/block_int.h b/include/block/block_int.h
index d6415b53c1..e1f2aa627e 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -1132,7 +1132,8 @@ void mirror_start(const char *job_id, BlockDriverState *bs,
  * @target: Block device to write to.
  * @speed: The maximum speed, in bytes per second, or 0 for unlimited.
  * @sync_mode: What parts of the disk image should be copied to the destination.
- * @sync_bitmap: The dirty bitmap if sync_mode is MIRROR_SYNC_MODE_INCREMENTAL.
+ * @sync_bitmap: The dirty bitmap if sync_mode is 'bitmap' or 'incremental'
+ * @bitmap_mode: The bitmap synchronization policy to use.
  * @on_source_error: The action to take upon error reading from the source.
  * @on_target_error: The action to take upon error writing to the target.
  * @creation_flags: Flags that control the behavior of the Job lifetime.
@@ -1148,6 +1149,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
                             BlockDriverState *target, int64_t speed,
                             MirrorSyncMode sync_mode,
                             BdrvDirtyBitmap *sync_bitmap,
+                            BitmapSyncMode bitmap_mode,
                             bool compress,
                             BlockdevOnError on_source_error,
                             BlockdevOnError on_target_error,
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 87eba5a5d9..e2226ef3ea 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1127,12 +1127,15 @@
 #
 # @none: only copy data written from now on
 #
-# @incremental: only copy data described by the dirty bitmap. Since: 2.4
+# @incremental: only copy data described by the dirty bitmap. (since: 2.4)
+#
+# @bitmap: only copy data described by the dirty bitmap. (since: 4.2)
+#          Behavior on completion is determined by the BitmapSyncMode.
 #
 # Since: 1.3
 ##
 { 'enum': 'MirrorSyncMode',
-  'data': ['top', 'full', 'none', 'incremental'] }
+  'data': ['top', 'full', 'none', 'incremental', 'bitmap'] }
 
 ##
 # @BitmapSyncMode:
@@ -1343,9 +1346,13 @@
 # @speed: the maximum speed, in bytes per second. The default is 0,
 #         for unlimited.
 #
-# @bitmap: the name of dirty bitmap if sync is "incremental".
-#          Must be present if sync is "incremental", must NOT be present
-#          otherwise. (Since 2.4 (Drive), 3.1 (Blockdev))
+# @bitmap: the name of a dirty bitmap if sync is "bitmap" or "incremental".
+#          Must be present if sync is "bitmap" or "incremental".
+#          Must not be present otherwise. (Since 2.4 (Drive), 3.1 (Blockdev))
+#
+# @bitmap-mode: Specifies the type of data the bitmap should contain after
+#               the operation concludes. Must be present if sync is "bitmap".
+#               Must NOT be present otherwise. (Since 4.2)
 #
 # @compress: true to compress data, if the target format supports it.
 #            (default: false) (since 2.8)
@@ -1380,7 +1387,8 @@
 { 'struct': 'BackupCommon',
   'data': { '*job-id': 'str', 'device': 'str',
             'sync': 'MirrorSyncMode', '*speed': 'int',
-            '*bitmap': 'str', '*compress': 'bool',
+            '*bitmap': 'str', '*bitmap-mode': 'BitmapSyncMode',
+            '*compress': 'bool',
             '*on-source-error': 'BlockdevOnError',
             '*on-target-error': 'BlockdevOnError',
             '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
-- 
2.21.0

[Qemu-devel] [PATCH v2 06/18] block/backup: add 'never' policy to bitmap sync mode

[John Snow]

This adds a "never" policy for bitmap synchronization. Regardless of if
the job succeeds or fails, we never update the bitmap. This can be used
to perform differential backups, or simply to avoid the job modifying a
bitmap.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 block/backup.c       | 5 +++--
 qapi/block-core.json | 5 ++++-
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/block/backup.c b/block/backup.c
index 996941fa61..d7fdafebda 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -265,8 +265,9 @@ static void backup_cleanup_sync_bitmap(BackupBlockJob *job, int ret)
     BdrvDirtyBitmap *bm;
     BlockDriverState *bs = blk_bs(job->common.blk);
 
-    if (ret < 0) {
-        /* Merge the successor back into the parent, delete nothing. */
+    if (ret < 0 || job->bitmap_mode == BITMAP_SYNC_MODE_NEVER) {
+        /* Failure, or we don't want to synchronize the bitmap.
+         * Merge the successor back into the parent, delete nothing. */
         bm = bdrv_reclaim_dirty_bitmap(bs, job->sync_bitmap, NULL);
         assert(bm);
     } else {
diff --git a/qapi/block-core.json b/qapi/block-core.json
index e2226ef3ea..1007192655 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1146,10 +1146,13 @@
 # @conditional: The bitmap is only synced when the operation is successful.
 #               This is the behavior always used for 'INCREMENTAL' backups.
 #
+# @never: The bitmap is never synchronized with the operation, and is
+#         treated solely as a read-only manifest of blocks to copy.
+#
 # Since: 4.2
 ##
 { 'enum': 'BitmapSyncMode',
-  'data': ['conditional'] }
+  'data': ['conditional', 'never'] }
 
 ##
 # @MirrorCopyMode:
-- 
2.21.0

[Qemu-devel] [PATCH v2 07/18] hbitmap: Fix merge when b is empty, and result is not an alias of a

[John Snow]

Nobody calls the function like this currently, but we neither prohibit
or cope with this behavior. I decided to make the function cope with it.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 util/hbitmap.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/util/hbitmap.c b/util/hbitmap.c
index 7905212a8b..3b6acae42b 100644
--- a/util/hbitmap.c
+++ b/util/hbitmap.c
@@ -781,8 +781,9 @@ bool hbitmap_can_merge(const HBitmap *a, const HBitmap *b)
 }
 
 /**
- * Given HBitmaps A and B, let A := A (BITOR) B.
- * Bitmap B will not be modified.
+ * Given HBitmaps A and B, let R := A (BITOR) B.
+ * Bitmaps A and B will not be modified,
+ *     except when bitmap R is an alias of A or B.
  *
  * @return true if the merge was successful,
  *         false if it was not attempted.
@@ -797,7 +798,13 @@ bool hbitmap_merge(const HBitmap *a, const HBitmap *b, HBitmap *result)
     }
     assert(hbitmap_can_merge(b, result));
 
-    if (hbitmap_count(b) == 0) {
+    if ((!hbitmap_count(a) && result == b) ||
+        (!hbitmap_count(b) && result == a)) {
+        return true;
+    }
+
+    if (!hbitmap_count(a) && !hbitmap_count(b)) {
+        hbitmap_reset_all(result);
         return true;
     }
 
-- 
2.21.0

[Qemu-devel] [PATCH v2 08/18] hbitmap: enable merging across granularities

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 util/hbitmap.c | 36 +++++++++++++++++++++++++++++++++++-
 1 file changed, 35 insertions(+), 1 deletion(-)

diff --git a/util/hbitmap.c b/util/hbitmap.c
index 3b6acae42b..306bc4876d 100644
--- a/util/hbitmap.c
+++ b/util/hbitmap.c
@@ -777,7 +777,27 @@ void hbitmap_truncate(HBitmap *hb, uint64_t size)
 
 bool hbitmap_can_merge(const HBitmap *a, const HBitmap *b)
 {
-    return (a->size == b->size) && (a->granularity == b->granularity);
+    return (a->orig_size == b->orig_size);
+}
+
+/**
+ * hbitmap_sparse_merge: performs dst = dst | src
+ * works with differing granularities.
+ * best used when src is sparsely populated.
+ */
+static void hbitmap_sparse_merge(HBitmap *dst, const HBitmap *src)
+{
+    uint64_t offset = 0;
+    uint64_t count = src->orig_size;
+
+    while (hbitmap_next_dirty_area(src, &offset, &count)) {
+        hbitmap_set(dst, offset, count);
+        offset += count;
+        if (offset >= src->orig_size) {
+            break;
+        }
+        count = src->orig_size - offset;
+    }
 }
 
 /**
@@ -808,10 +828,24 @@ bool hbitmap_merge(const HBitmap *a, const HBitmap *b, HBitmap *result)
         return true;
     }
 
+    if (a->granularity != b->granularity) {
+        if ((a != result) && (b != result)) {
+            hbitmap_reset_all(result);
+        }
+        if (a != result) {
+            hbitmap_sparse_merge(result, a);
+        }
+        if (b != result) {
+            hbitmap_sparse_merge(result, b);
+        }
+        return true;
+    }
+
     /* This merge is O(size), as BITS_PER_LONG and HBITMAP_LEVELS are constant.
      * It may be possible to improve running times for sparsely populated maps
      * by using hbitmap_iter_next, but this is suboptimal for dense maps.
      */
+    assert(a->size == b->size);
     for (i = HBITMAP_LEVELS - 1; i >= 0; i--) {
         for (j = 0; j < a->sizes[i]; j++) {
             result->levels[i][j] = a->levels[i][j] | b->levels[i][j];
-- 
2.21.0

[Qemu-devel] [PATCH v2 09/18] block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal

[John Snow]

I'm surprised it didn't come up sooner, but sometimes we have a +busy
bitmap as a source. This is dangerous from the QMP API, but if we are
the owner that marked the bitmap busy, it's safe to merge it using it as
a read only source.

It is not safe in the general case to allow users to read from in-use
bitmaps, so create an internal variant that foregoes the safety
checking.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 block/dirty-bitmap.c      | 51 +++++++++++++++++++++++++++++++++------
 include/block/block_int.h |  3 +++
 2 files changed, 47 insertions(+), 7 deletions(-)

diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 95a9c2a5d8..b0f76826b3 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -810,11 +810,15 @@ bool bdrv_dirty_bitmap_next_dirty_area(BdrvDirtyBitmap *bitmap,
     return hbitmap_next_dirty_area(bitmap->bitmap, offset, bytes);
 }
 
+/**
+ * bdrv_merge_dirty_bitmap: merge src into dest.
+ * Ensures permissions on bitmaps are reasonable; use for public API.
+ *
+ * @backup: If provided, make a copy of dest here prior to merge.
+ */
 void bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
                              HBitmap **backup, Error **errp)
 {
-    bool ret;
-
     qemu_mutex_lock(dest->mutex);
     if (src->mutex != dest->mutex) {
         qemu_mutex_lock(src->mutex);
@@ -833,6 +837,37 @@ void bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
         goto out;
     }
 
+    assert(bdrv_dirty_bitmap_merge_internal(dest, src, backup, false));
+
+out:
+    qemu_mutex_unlock(dest->mutex);
+    if (src->mutex != dest->mutex) {
+        qemu_mutex_unlock(src->mutex);
+    }
+}
+
+/**
+ * bdrv_dirty_bitmap_merge_internal: merge src into dest.
+ * Does NOT check bitmap permissions; not suitable for use as public API.
+ *
+ * @backup: If provided, make a copy of dest here prior to merge.
+ * @lock: If true, lock and unlock bitmaps on the way in/out.
+ * returns true if the merge succeeded; false if unattempted.
+ */
+bool bdrv_dirty_bitmap_merge_internal(BdrvDirtyBitmap *dest,
+                                      const BdrvDirtyBitmap *src,
+                                      HBitmap **backup,
+                                      bool lock)
+{
+    bool ret;
+
+    if (lock) {
+        qemu_mutex_lock(dest->mutex);
+        if (src->mutex != dest->mutex) {
+            qemu_mutex_lock(src->mutex);
+        }
+    }
+
     if (backup) {
         *backup = dest->bitmap;
         dest->bitmap = hbitmap_alloc(dest->size, hbitmap_granularity(*backup));
@@ -840,11 +875,13 @@ void bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
     } else {
         ret = hbitmap_merge(dest->bitmap, src->bitmap, dest->bitmap);
     }
-    assert(ret);
 
-out:
-    qemu_mutex_unlock(dest->mutex);
-    if (src->mutex != dest->mutex) {
-        qemu_mutex_unlock(src->mutex);
+    if (lock) {
+        qemu_mutex_unlock(dest->mutex);
+        if (src->mutex != dest->mutex) {
+            qemu_mutex_unlock(src->mutex);
+        }
     }
+
+    return ret;
 }
diff --git a/include/block/block_int.h b/include/block/block_int.h
index e1f2aa627e..83ffdf4950 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -1238,6 +1238,9 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t offset, int64_t bytes);
 
 void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap **out);
 void bdrv_restore_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap *backup);
+bool bdrv_dirty_bitmap_merge_internal(BdrvDirtyBitmap *dest,
+                                      const BdrvDirtyBitmap *src,
+                                      HBitmap **backup, bool lock);
 
 void bdrv_inc_in_flight(BlockDriverState *bs);
 void bdrv_dec_in_flight(BlockDriverState *bs);
-- 
2.21.0

[Qemu-devel] [PATCH v2 10/18] block/dirty-bitmap: add bdrv_dirty_bitmap_get

[John Snow]

Add a public interface for get. While we're at it,
rename "bdrv_get_dirty_bitmap_locked" to "bdrv_dirty_bitmap_get_locked".

(There are more functions to rename to the bdrv_dirty_bitmap_VERB form,
but they will wait until the conclusion of this series.)

Signed-off-by: John Snow <jsnow@redhat.com>
---
 block/dirty-bitmap.c         | 18 +++++++++++-------
 block/mirror.c               |  2 +-
 include/block/dirty-bitmap.h |  4 ++--
 migration/block.c            |  5 ++---
 nbd/server.c                 |  2 +-
 5 files changed, 17 insertions(+), 14 deletions(-)

diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index b0f76826b3..97541521ab 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -509,14 +509,18 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
 }
 
 /* Called within bdrv_dirty_bitmap_lock..unlock */
-bool bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
-                           int64_t offset)
+bool bdrv_dirty_bitmap_get_locked(BdrvDirtyBitmap *bitmap, int64_t offset)
 {
-    if (bitmap) {
-        return hbitmap_get(bitmap->bitmap, offset);
-    } else {
-        return false;
-    }
+    return hbitmap_get(bitmap->bitmap, offset);
+}
+
+bool bdrv_dirty_bitmap_get(BdrvDirtyBitmap *bitmap, int64_t offset) {
+    bool ret;
+    bdrv_dirty_bitmap_lock(bitmap);
+    ret = bdrv_dirty_bitmap_get_locked(bitmap, offset);
+    bdrv_dirty_bitmap_unlock(bitmap);
+
+    return ret;
 }
 
 /**
diff --git a/block/mirror.c b/block/mirror.c
index 42b3d9acd0..1da57409f0 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -476,7 +476,7 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
         int64_t next_offset = offset + nb_chunks * s->granularity;
         int64_t next_chunk = next_offset / s->granularity;
         if (next_offset >= s->bdev_length ||
-            !bdrv_get_dirty_locked(source, s->dirty_bitmap, next_offset)) {
+            !bdrv_dirty_bitmap_get_locked(s->dirty_bitmap, next_offset)) {
             break;
         }
         if (test_bit(next_chunk, s->in_flight_bitmap)) {
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 62682eb865..0120ef3f05 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -84,12 +84,12 @@ void bdrv_dirty_bitmap_set_busy(BdrvDirtyBitmap *bitmap, bool busy);
 void bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
                              HBitmap **backup, Error **errp);
 void bdrv_dirty_bitmap_set_migration(BdrvDirtyBitmap *bitmap, bool migration);
+bool bdrv_dirty_bitmap_get(BdrvDirtyBitmap *bitmap, int64_t offset);
 
 /* Functions that require manual locking.  */
 void bdrv_dirty_bitmap_lock(BdrvDirtyBitmap *bitmap);
 void bdrv_dirty_bitmap_unlock(BdrvDirtyBitmap *bitmap);
-bool bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
-                           int64_t offset);
+bool bdrv_dirty_bitmap_get_locked(BdrvDirtyBitmap *bitmap, int64_t offset);
 void bdrv_set_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
                                   int64_t offset, int64_t bytes);
 void bdrv_reset_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
diff --git a/migration/block.c b/migration/block.c
index 91f98ef44a..a5b60456ae 100644
--- a/migration/block.c
+++ b/migration/block.c
@@ -520,7 +520,6 @@ static int mig_save_device_dirty(QEMUFile *f, BlkMigDevState *bmds,
                                  int is_async)
 {
     BlkMigBlock *blk;
-    BlockDriverState *bs = blk_bs(bmds->blk);
     int64_t total_sectors = bmds->total_sectors;
     int64_t sector;
     int nr_sectors;
@@ -535,8 +534,8 @@ static int mig_save_device_dirty(QEMUFile *f, BlkMigDevState *bmds,
             blk_mig_unlock();
         }
         bdrv_dirty_bitmap_lock(bmds->dirty_bitmap);
-        if (bdrv_get_dirty_locked(bs, bmds->dirty_bitmap,
-                                  sector * BDRV_SECTOR_SIZE)) {
+        if (bdrv_dirty_bitmap_get_locked(bmds->dirty_bitmap,
+                                         sector * BDRV_SECTOR_SIZE)) {
             if (total_sectors - sector < BDRV_SECTORS_PER_DIRTY_CHUNK) {
                 nr_sectors = total_sectors - sector;
             } else {
diff --git a/nbd/server.c b/nbd/server.c
index 10faedcfc5..fbd51b48a7 100644
--- a/nbd/server.c
+++ b/nbd/server.c
@@ -2003,7 +2003,7 @@ static unsigned int bitmap_to_extents(BdrvDirtyBitmap *bitmap, uint64_t offset,
     bdrv_dirty_bitmap_lock(bitmap);
 
     it = bdrv_dirty_iter_new(bitmap);
-    dirty = bdrv_get_dirty_locked(NULL, bitmap, offset);
+    dirty = bdrv_dirty_bitmap_get_locked(bitmap, offset);
 
     assert(begin < overall_end && nb_extents);
     while (begin < overall_end && i < nb_extents) {
-- 
2.21.0

[Qemu-devel] [PATCH v2 11/18] block/backup: upgrade copy_bitmap to BdrvDirtyBitmap

[John Snow]

This simplifies some interface matters; namely the initialization and
(later) merging the manifest back into the sync_bitmap if it was
provided.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 block/backup.c | 76 ++++++++++++++++++++++++--------------------------
 1 file changed, 37 insertions(+), 39 deletions(-)

diff --git a/block/backup.c b/block/backup.c
index d7fdafebda..9cc5a7f6ca 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -38,7 +38,10 @@ typedef struct CowRequest {
 typedef struct BackupBlockJob {
     BlockJob common;
     BlockBackend *target;
+
     BdrvDirtyBitmap *sync_bitmap;
+    BdrvDirtyBitmap *copy_bitmap;
+
     MirrorSyncMode sync_mode;
     BitmapSyncMode bitmap_mode;
     BlockdevOnError on_source_error;
@@ -51,7 +54,6 @@ typedef struct BackupBlockJob {
     NotifierWithReturn before_write;
     QLIST_HEAD(, CowRequest) inflight_reqs;
 
-    HBitmap *copy_bitmap;
     bool use_copy_range;
     int64_t copy_range_size;
 
@@ -113,7 +115,7 @@ static int coroutine_fn backup_cow_with_bounce_buffer(BackupBlockJob *job,
     int write_flags = job->serialize_target_writes ? BDRV_REQ_SERIALISING : 0;
 
     assert(QEMU_IS_ALIGNED(start, job->cluster_size));
-    hbitmap_reset(job->copy_bitmap, start, job->cluster_size);
+    bdrv_reset_dirty_bitmap(job->copy_bitmap, start, job->cluster_size);
     nbytes = MIN(job->cluster_size, job->len - start);
     if (!*bounce_buffer) {
         *bounce_buffer = blk_blockalign(blk, job->cluster_size);
@@ -146,7 +148,7 @@ static int coroutine_fn backup_cow_with_bounce_buffer(BackupBlockJob *job,
 
     return nbytes;
 fail:
-    hbitmap_set(job->copy_bitmap, start, job->cluster_size);
+    bdrv_set_dirty_bitmap(job->copy_bitmap, start, job->cluster_size);
     return ret;
 
 }
@@ -169,12 +171,12 @@ static int coroutine_fn backup_cow_with_offload(BackupBlockJob *job,
     assert(QEMU_IS_ALIGNED(start, job->cluster_size));
     nbytes = MIN(job->copy_range_size, end - start);
     nr_clusters = DIV_ROUND_UP(nbytes, job->cluster_size);
-    hbitmap_reset(job->copy_bitmap, start, job->cluster_size * nr_clusters);
+    bdrv_reset_dirty_bitmap(job->copy_bitmap, start, job->cluster_size * nr_clusters);
     ret = blk_co_copy_range(blk, start, job->target, start, nbytes,
                             read_flags, write_flags);
     if (ret < 0) {
         trace_backup_do_cow_copy_range_fail(job, start, ret);
-        hbitmap_set(job->copy_bitmap, start, job->cluster_size * nr_clusters);
+        bdrv_set_dirty_bitmap(job->copy_bitmap, start, job->cluster_size * nr_clusters);
         return ret;
     }
 
@@ -202,7 +204,7 @@ static int coroutine_fn backup_do_cow(BackupBlockJob *job,
     cow_request_begin(&cow_request, job, start, end);
 
     while (start < end) {
-        if (!hbitmap_get(job->copy_bitmap, start)) {
+        if (!bdrv_dirty_bitmap_get(job->copy_bitmap, start)) {
             trace_backup_do_cow_skip(job, start);
             start += job->cluster_size;
             continue; /* already copied */
@@ -296,14 +298,16 @@ static void backup_abort(Job *job)
 static void backup_clean(Job *job)
 {
     BackupBlockJob *s = container_of(job, BackupBlockJob, common.job);
+    BlockDriverState *bs = blk_bs(s->target);
+
+    if (s->copy_bitmap) {
+        bdrv_release_dirty_bitmap(bs, s->copy_bitmap);
+        s->copy_bitmap = NULL;
+    }
+
     assert(s->target);
     blk_unref(s->target);
     s->target = NULL;
-
-    if (s->copy_bitmap) {
-        hbitmap_free(s->copy_bitmap);
-        s->copy_bitmap = NULL;
-    }
 }
 
 void backup_do_checkpoint(BlockJob *job, Error **errp)
@@ -318,7 +322,7 @@ void backup_do_checkpoint(BlockJob *job, Error **errp)
         return;
     }
 
-    hbitmap_set(backup_job->copy_bitmap, 0, backup_job->len);
+    bdrv_set_dirty_bitmap(backup_job->copy_bitmap, 0, backup_job->len);
 }
 
 static void backup_drain(BlockJob *job)
@@ -387,59 +391,49 @@ static bool bdrv_is_unallocated_range(BlockDriverState *bs,
 
 static int coroutine_fn backup_loop(BackupBlockJob *job)
 {
-    int ret;
     bool error_is_read;
     int64_t offset;
-    HBitmapIter hbi;
+    BdrvDirtyBitmapIter *bdbi;
     BlockDriverState *bs = blk_bs(job->common.blk);
+    int ret = 0;
 
-    hbitmap_iter_init(&hbi, job->copy_bitmap, 0);
-    while ((offset = hbitmap_iter_next(&hbi)) != -1) {
+    bdbi = bdrv_dirty_iter_new(job->copy_bitmap);
+    while ((offset = bdrv_dirty_iter_next(bdbi)) != -1) {
         if (job->sync_mode == MIRROR_SYNC_MODE_TOP &&
             bdrv_is_unallocated_range(bs, offset, job->cluster_size))
         {
-            hbitmap_reset(job->copy_bitmap, offset, job->cluster_size);
+            bdrv_set_dirty_bitmap(job->copy_bitmap, offset, job->cluster_size);
             continue;
         }
 
         do {
             if (yield_and_check(job)) {
-                return 0;
+                goto out;
             }
             ret = backup_do_cow(job, offset,
                                 job->cluster_size, &error_is_read, false);
             if (ret < 0 && backup_error_action(job, error_is_read, -ret) ==
                            BLOCK_ERROR_ACTION_REPORT)
             {
-                return ret;
+                goto out;
             }
         } while (ret < 0);
     }
 
-    return 0;
+ out:
+    bdrv_dirty_iter_free(bdbi);
+    return ret;
 }
 
 /* init copy_bitmap from sync_bitmap */
 static void backup_incremental_init_copy_bitmap(BackupBlockJob *job)
 {
-    uint64_t offset = 0;
-    uint64_t bytes = job->len;
-
-    while (bdrv_dirty_bitmap_next_dirty_area(job->sync_bitmap,
-                                             &offset, &bytes))
-    {
-        hbitmap_set(job->copy_bitmap, offset, bytes);
-
-        offset += bytes;
-        if (offset >= job->len) {
-            break;
-        }
-        bytes = job->len - offset;
-    }
+    bdrv_dirty_bitmap_merge_internal(job->copy_bitmap, job->sync_bitmap,
+                                     NULL, true);
 
     /* TODO job_progress_set_remaining() would make more sense */
     job_progress_update(&job->common.job,
-        job->len - hbitmap_count(job->copy_bitmap));
+                        job->len - bdrv_get_dirty_count(job->copy_bitmap));
 }
 
 static int coroutine_fn backup_run(Job *job, Error **errp)
@@ -456,7 +450,7 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
     if (s->sync_mode == MIRROR_SYNC_MODE_BITMAP) {
         backup_incremental_init_copy_bitmap(s);
     } else {
-        hbitmap_set(s->copy_bitmap, 0, s->len);
+        bdrv_set_dirty_bitmap(s->copy_bitmap, 0, s->len);
     }
 
     s->before_write.notify = backup_before_write_notify;
@@ -549,7 +543,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
     BackupBlockJob *job = NULL;
     int ret;
     int64_t cluster_size;
-    HBitmap *copy_bitmap = NULL;
+    BdrvDirtyBitmap *copy_bitmap = NULL;
 
     assert(bs);
     assert(target);
@@ -619,7 +613,11 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
         goto error;
     }
 
-    copy_bitmap = hbitmap_alloc(len, ctz32(cluster_size));
+    copy_bitmap = bdrv_create_dirty_bitmap(target, cluster_size, NULL, errp);
+    if (!copy_bitmap) {
+        goto error;
+    }
+    bdrv_disable_dirty_bitmap(copy_bitmap);
 
     /* job->len is fixed, so we can't allow resize */
     job = block_job_create(job_id, &backup_job_driver, txn, bs,
@@ -670,7 +668,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
  error:
     if (copy_bitmap) {
         assert(!job || !job->copy_bitmap);
-        hbitmap_free(copy_bitmap);
+        bdrv_release_dirty_bitmap(bs, copy_bitmap);
     }
     if (sync_bitmap) {
         bdrv_reclaim_dirty_bitmap(bs, sync_bitmap, NULL);
-- 
2.21.0

[Qemu-devel] [PATCH v2 12/18] block/backup: add 'always' bitmap sync policy

[John Snow]

This adds an "always" policy for bitmap synchronization. Regardless of if
the job succeeds or fails, the bitmap is *always* synchronized. This means
that for backups that fail part-way through, the bitmap retains a record of
which sectors need to be copied out to accomplish a new backup using the
old, partial result.

In effect, this allows us to "resume" a failed backup; however the new backup
will be from the new point in time, so it isn't a "resume" as much as it is
an "incremental retry." This can be useful in the case of extremely large
backups that fail considerably through the operation and we'd like to not waste
the work that was already performed.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 block/backup.c       | 25 +++++++++++++++++--------
 qapi/block-core.json |  5 ++++-
 2 files changed, 21 insertions(+), 9 deletions(-)

diff --git a/block/backup.c b/block/backup.c
index 9cc5a7f6ca..495d8f71aa 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -266,16 +266,25 @@ static void backup_cleanup_sync_bitmap(BackupBlockJob *job, int ret)
 {
     BdrvDirtyBitmap *bm;
     BlockDriverState *bs = blk_bs(job->common.blk);
+    bool sync = (((ret == 0) || (job->bitmap_mode == BITMAP_SYNC_MODE_ALWAYS)) \
+                 && (job->bitmap_mode != BITMAP_SYNC_MODE_NEVER));
 
-    if (ret < 0 || job->bitmap_mode == BITMAP_SYNC_MODE_NEVER) {
-        /* Failure, or we don't want to synchronize the bitmap.
-         * Merge the successor back into the parent, delete nothing. */
-        bm = bdrv_reclaim_dirty_bitmap(bs, job->sync_bitmap, NULL);
-        assert(bm);
-    } else {
-        /* Everything is fine, delete this bitmap and install the backup. */
+    if (sync) {
+        /* We succeeded, or we always intended to sync the bitmap.
+         * Delete this bitmap and install the child. */
         bm = bdrv_dirty_bitmap_abdicate(bs, job->sync_bitmap, NULL);
-        assert(bm);
+    } else {
+        /* We failed, or we never intended to sync the bitmap anyway.
+         * Merge the successor back into the parent, keeping all data. */
+        bm = bdrv_reclaim_dirty_bitmap(bs, job->sync_bitmap, NULL);
+    }
+
+    assert(bm);
+
+    if (ret < 0 && job->bitmap_mode == BITMAP_SYNC_MODE_ALWAYS) {
+        /* If we failed and synced, merge in the bits we didn't copy: */
+        bdrv_dirty_bitmap_merge_internal(bm, job->copy_bitmap,
+                                         NULL, true);
     }
 }
 
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 1007192655..b527306d19 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1149,10 +1149,13 @@
 # @never: The bitmap is never synchronized with the operation, and is
 #         treated solely as a read-only manifest of blocks to copy.
 #
+# @always: The bitmap is always synchronized with the operation,
+#          regardless of whether or not the operation was successful.
+#
 # Since: 4.2
 ##
 { 'enum': 'BitmapSyncMode',
-  'data': ['conditional', 'never'] }
+  'data': ['conditional', 'never', 'always'] }
 
 ##
 # @MirrorCopyMode:
-- 
2.21.0

[Qemu-devel] [PATCH v2 13/18] iotests: add testing shim for script-style python tests

[John Snow]

Because the new-style python tests don't use the iotests.main() test
launcher, we don't turn on the debugger logging for these scripts
when invoked via ./check -d.

Refactor the launcher shim into new and old style shims so that they
share environmental configuration.

Two cleanup notes: debug was not actually used as a global, and there
was no reason to create a class in an inner scope just to achieve
default variables; we can simply create an instance of the runner with
the values we want instead.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/iotests.py | 40 +++++++++++++++++++++++------------
 1 file changed, 26 insertions(+), 14 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 3ecef5bc90..fcad957d63 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -61,7 +61,6 @@ cachemode = os.environ.get('CACHEMODE')
 qemu_default_machine = os.environ.get('QEMU_DEFAULT_MACHINE')
 
 socket_scm_helper = os.environ.get('SOCKET_SCM_HELPER', 'socket_scm_helper')
-debug = False
 
 luks_default_secret_object = 'secret,id=keysec0,data=' + \
                              os.environ.get('IMGKEYSECRET', '')
@@ -834,11 +833,22 @@ def skip_if_unsupported(required_formats=[], read_only=False):
         return func_wrapper
     return skip_test_decorator
 
-def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
-         unsupported_fmts=[]):
-    '''Run tests'''
+def execute_unittest(output, verbosity, debug):
+    runner = unittest.TextTestRunner(stream=output, descriptions=True,
+                                     verbosity=verbosity)
+    try:
+        # unittest.main() will use sys.exit(); so expect a SystemExit
+        # exception
+        unittest.main(testRunner=runner)
+    finally:
+        if not debug:
+            sys.stderr.write(re.sub(r'Ran (\d+) tests? in [\d.]+s',
+                                    r'Ran \1 tests', output.getvalue()))
 
-    global debug
+def execute_test(test_function=None,
+                 supported_fmts=[], supported_oses=['linux'],
+                 supported_cache_modes=[], unsupported_fmts=[]):
+    """Run either unittest or script-style tests."""
 
     # We are using TEST_DIR and QEMU_DEFAULT_MACHINE as proxies to
     # indicate that we're not being run via "check". There may be
@@ -870,13 +880,15 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
 
     logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
 
-    class MyTestRunner(unittest.TextTestRunner):
-        def __init__(self, stream=output, descriptions=True, verbosity=verbosity):
-            unittest.TextTestRunner.__init__(self, stream, descriptions, verbosity)
+    if not test_function:
+        execute_unittest(output, verbosity, debug)
+    else:
+        test_function()
 
-    # unittest.main() will use sys.exit() so expect a SystemExit exception
-    try:
-        unittest.main(testRunner=MyTestRunner)
-    finally:
-        if not debug:
-            sys.stderr.write(re.sub(r'Ran (\d+) tests? in [\d.]+s', r'Ran \1 tests', output.getvalue()))
+def script_main(test_function, *args, **kwargs):
+    """Run script-style tests outside of the unittest framework"""
+    execute_test(test_function, *args, **kwargs)
+
+def main(*args, **kwargs):
+    """Run tests using the unittest framework"""
+    execute_test(None, *args, **kwargs)
-- 
2.21.0

[Qemu-devel] [PATCH v2 14/18] iotests: teach run_job to cancel pending jobs

[John Snow]

run_job can cancel pending jobs to simulate failure. This lets us use
the pending callback to issue test commands while the job is open, but
then still have the job fail in the end.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/iotests.py | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index fcad957d63..c544659ecb 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -541,7 +541,22 @@ class VM(qtest.QEMUQtestMachine):
 
     # Returns None on success, and an error string on failure
     def run_job(self, job, auto_finalize=True, auto_dismiss=False,
-                pre_finalize=None, wait=60.0):
+                pre_finalize=None, cancel=False, wait=60.0):
+        """
+        run_job moves a job from creation through to dismissal.
+
+        :param job: String. ID of recently-launched job
+        :param auto_finalize: Bool. True if the job was launched with
+                              auto_finalize. Defaults to True.
+        :param auto_dismiss: Bool. True if the job was launched with
+                             auto_dismiss=True. Defaults to False.
+        :param pre_finalize: Callback. A callable that takes no arguments to be
+                             invoked prior to issuing job-finalize, if any.
+        :param cancel: Bool. When true, cancels the job after the pre_finalize
+                       callback.
+        :param wait: Float. Timeout value specifying how long to wait for any
+                     event, in seconds. Defaults to 60.0.
+        """
         match_device = {'data': {'device': job}}
         match_id = {'data': {'id': job}}
         events = [
@@ -568,7 +583,10 @@ class VM(qtest.QEMUQtestMachine):
             elif status == 'pending' and not auto_finalize:
                 if pre_finalize:
                     pre_finalize()
-                self.qmp_log('job-finalize', id=job)
+                if cancel:
+                    self.qmp_log('job-cancel', id=job)
+                else:
+                    self.qmp_log('job-finalize', id=job)
             elif status == 'concluded' and not auto_dismiss:
                 self.qmp_log('job-dismiss', id=job)
             elif status == 'null':
-- 
2.21.0

[Qemu-devel] [PATCH v2 15/18] iotests: teach FilePath to produce multiple paths

[John Snow]

Use "FilePaths" instead of "FilePath" to request multiple files be
cleaned up after we leave that object's scope.

This is not crucial; but it saves a little typing.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/iotests.py | 33 ++++++++++++++++++++++-----------
 1 file changed, 22 insertions(+), 11 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index c544659ecb..db00fd2adf 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -358,31 +358,43 @@ class Timeout:
     def timeout(self, signum, frame):
         raise Exception(self.errmsg)
 
+def file_pattern(name):
+    return "{0}-{1}".format(os.getpid(), name)
 
-class FilePath(object):
-    '''An auto-generated filename that cleans itself up.
+class FilePaths(object):
+    """
+    FilePaths is an auto-generated filename that cleans itself up.
 
     Use this context manager to generate filenames and ensure that the file
     gets deleted::
 
-        with TestFilePath('test.img') as img_path:
+        with FilePaths(['test.img']) as img_path:
             qemu_img('create', img_path, '1G')
         # migration_sock_path is automatically deleted
-    '''
-    def __init__(self, name):
-        filename = '{0}-{1}'.format(os.getpid(), name)
-        self.path = os.path.join(test_dir, filename)
+    """
+    def __init__(self, names):
+        self.paths = []
+        for name in names:
+            self.paths.append(os.path.join(test_dir, file_pattern(name)))
 
     def __enter__(self):
-        return self.path
+        return self.paths
 
     def __exit__(self, exc_type, exc_val, exc_tb):
         try:
-            os.remove(self.path)
+            for path in self.paths:
+                os.remove(path)
         except OSError:
             pass
         return False
 
+class FilePath(FilePaths):
+    """FilePath is a specialization of FilePaths that takes a single filename."""
+    def __init__(self, name):
+        super(FilePath, self).__init__([name])
+
+    def __enter__(self):
+        return self.paths[0]
 
 def file_path_remover():
     for path in reversed(file_path_remover.paths):
@@ -391,7 +403,6 @@ def file_path_remover():
         except OSError:
             pass
 
-
 def file_path(*names):
     ''' Another way to get auto-generated filename that cleans itself up.
 
@@ -407,7 +418,7 @@ def file_path(*names):
 
     paths = []
     for name in names:
-        filename = '{0}-{1}'.format(os.getpid(), name)
+        filename = file_pattern(name)
         path = os.path.join(test_dir, filename)
         file_path_remover.paths.append(path)
         paths.append(path)
-- 
2.21.0

[Qemu-devel] [PATCH v2 16/18] iotests: Add virtio-scsi device helper

[John Snow]

Seems that it comes up enough.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/040        | 6 +-----
 tests/qemu-iotests/093        | 6 ++----
 tests/qemu-iotests/139        | 7 ++-----
 tests/qemu-iotests/238        | 5 +----
 tests/qemu-iotests/iotests.py | 4 ++++
 5 files changed, 10 insertions(+), 18 deletions(-)

diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
index b81133a474..657b37103c 100755
--- a/tests/qemu-iotests/040
+++ b/tests/qemu-iotests/040
@@ -85,11 +85,7 @@ class TestSingleDrive(ImageCommitTestCase):
         qemu_io('-f', 'raw', '-c', 'write -P 0xab 0 524288', backing_img)
         qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0xef 524288 524288', mid_img)
         self.vm = iotests.VM().add_drive(test_img, "node-name=top,backing.node-name=mid,backing.backing.node-name=base", interface="none")
-        if iotests.qemu_default_machine == 's390-ccw-virtio':
-            self.vm.add_device("virtio-scsi-ccw")
-        else:
-            self.vm.add_device("virtio-scsi-pci")
-
+        self.vm.add_device(iotests.get_virtio_scsi_device())
         self.vm.add_device("scsi-hd,id=scsi0,drive=drive0")
         self.vm.launch()
 
diff --git a/tests/qemu-iotests/093 b/tests/qemu-iotests/093
index d88fbc182e..46153220f8 100755
--- a/tests/qemu-iotests/093
+++ b/tests/qemu-iotests/093
@@ -366,10 +366,8 @@ class ThrottleTestGroupNames(iotests.QMPTestCase):
 class ThrottleTestRemovableMedia(iotests.QMPTestCase):
     def setUp(self):
         self.vm = iotests.VM()
-        if iotests.qemu_default_machine == 's390-ccw-virtio':
-            self.vm.add_device("virtio-scsi-ccw,id=virtio-scsi")
-        else:
-            self.vm.add_device("virtio-scsi-pci,id=virtio-scsi")
+        self.vm.add_device("{},id=virtio-scsi".format(
+            iotests.get_virtio_scsi_device()))
         self.vm.launch()
 
     def tearDown(self):
diff --git a/tests/qemu-iotests/139 b/tests/qemu-iotests/139
index 933b45121a..2176ea51ba 100755
--- a/tests/qemu-iotests/139
+++ b/tests/qemu-iotests/139
@@ -35,11 +35,8 @@ class TestBlockdevDel(iotests.QMPTestCase):
     def setUp(self):
         iotests.qemu_img('create', '-f', iotests.imgfmt, base_img, '1M')
         self.vm = iotests.VM()
-        if iotests.qemu_default_machine == 's390-ccw-virtio':
-            self.vm.add_device("virtio-scsi-ccw,id=virtio-scsi")
-        else:
-            self.vm.add_device("virtio-scsi-pci,id=virtio-scsi")
-
+        self.vm.add_device("{},id=virtio-scsi".format(
+            iotests.get_virtio_scsi_device()))
         self.vm.launch()
 
     def tearDown(self):
diff --git a/tests/qemu-iotests/238 b/tests/qemu-iotests/238
index 1c0a46fa90..387a77b2cd 100755
--- a/tests/qemu-iotests/238
+++ b/tests/qemu-iotests/238
@@ -23,10 +23,7 @@ import os
 import iotests
 from iotests import log
 
-if iotests.qemu_default_machine == 's390-ccw-virtio':
-    virtio_scsi_device = 'virtio-scsi-ccw'
-else:
-    virtio_scsi_device = 'virtio-scsi-pci'
+virtio_scsi_device = iotests.get_virtio_scsi_device()
 
 vm = iotests.VM()
 vm.launch()
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index db00fd2adf..0b5278c5b4 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -164,6 +164,10 @@ def qemu_io_silent(*args):
                          (-exitcode, ' '.join(args)))
     return exitcode
 
+def get_virtio_scsi_device():
+    if qemu_default_machine == 's390-ccw-virtio':
+        return 'virtio-scsi-ccw'
+    return 'virtio-scsi-pci'
 
 class QemuIoInteractive:
     def __init__(self, *args):
-- 
2.21.0

[Qemu-devel] [PATCH v2 17/18] iotests: add test 257 for bitmap-mode backups

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 tests/qemu-iotests/257     |  409 +++++++
 tests/qemu-iotests/257.out | 2199 ++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/group   |    1 +
 3 files changed, 2609 insertions(+)
 create mode 100755 tests/qemu-iotests/257
 create mode 100644 tests/qemu-iotests/257.out

diff --git a/tests/qemu-iotests/257 b/tests/qemu-iotests/257
new file mode 100755
index 0000000000..169447ea97
--- /dev/null
+++ b/tests/qemu-iotests/257
@@ -0,0 +1,409 @@
+#!/usr/bin/env python
+#
+# Test bitmap-sync backups (incremental, differential, and partials)
+#
+# Copyright (c) 2019 John Snow for Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# owner=jsnow@redhat.com
+
+from collections import namedtuple
+import math
+import os
+
+import iotests
+from iotests import log, qemu_img
+
+SIZE = 64 * 1024 * 1024
+GRANULARITY = 64 * 1024
+
+Pattern = namedtuple('Pattern', ['byte', 'offset', 'size'])
+def mkpattern(byte, offset, size=GRANULARITY):
+    """Constructor for Pattern() with default size"""
+    return Pattern(byte, offset, size)
+
+class PatternGroup:
+    """Grouping of Pattern objects. Initialize with an iterable of Patterns."""
+    def __init__(self, patterns):
+        self.patterns = patterns
+
+    def bits(self, granularity):
+        """Calculate the unique bits dirtied by this pattern grouping"""
+        res = set()
+        for pattern in self.patterns:
+            lower = math.floor(pattern.offset / granularity)
+            upper = math.floor((pattern.offset + pattern.size - 1) / granularity)
+            res = res | set(range(lower, upper + 1))
+        return res
+
+GROUPS = [
+    PatternGroup([
+        # Batch 0: 4 clusters
+        mkpattern('0x49', 0x0000000),
+        mkpattern('0x6c', 0x0100000),   # 1M
+        mkpattern('0x6f', 0x2000000),   # 32M
+        mkpattern('0x76', 0x3ff0000)]), # 64M - 64K
+    PatternGroup([
+        # Batch 1: 6 clusters (3 new)
+        mkpattern('0x65', 0x0000000),   # Full overwrite
+        mkpattern('0x77', 0x00f8000),   # Partial-left (1M-32K)
+        mkpattern('0x72', 0x2008000),   # Partial-right (32M+32K)
+        mkpattern('0x69', 0x3fe0000)]), # Adjacent-left (64M - 128K)
+    PatternGroup([
+        # Batch 2: 7 clusters (3 new)
+        mkpattern('0x74', 0x0010000),   # Adjacent-right
+        mkpattern('0x69', 0x00e8000),   # Partial-left  (1M-96K)
+        mkpattern('0x6e', 0x2018000),   # Partial-right (32M+96K)
+        mkpattern('0x67', 0x3fe0000,
+                  2*GRANULARITY)]),     # Overwrite [(64M-128K)-64M)
+    PatternGroup([
+        # Batch 3: 8 clusters (5 new)
+        # Carefully chosen such that nothing re-dirties the one cluster
+        # that copies out successfully before failure in Group #1.
+        mkpattern('0xaa', 0x0010000,
+                  3*GRANULARITY),       # Overwrite and 2x Adjacent-right
+        mkpattern('0xbb', 0x00d8000),   # Partial-left (1M-160K)
+        mkpattern('0xcc', 0x2028000),   # Partial-right (32M+160K)
+        mkpattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right
+]
+
+class Drive:
+    """Represents, vaguely, a drive attached to a VM.
+    Includes format, graph, and device information."""
+
+    def __init__(self, path, vm=None):
+        self.path = path
+        self.vm = vm
+        self.fmt = None
+        self.size = None
+        self.node = None
+        self.device = None
+
+    @property
+    def name(self):
+        return self.node or self.device
+
+    def img_create(self, fmt, size):
+        self.fmt = fmt
+        self.size = size
+        iotests.qemu_img_create('-f', self.fmt, self.path, str(self.size))
+
+    def create_target(self, name, fmt, size):
+        basename = os.path.basename(self.path)
+        file_node_name = "file_{}".format(basename)
+        vm = self.vm
+
+        log(vm.command('blockdev-create', job_id='bdc-file-job',
+                       options={
+                           'driver': 'file',
+                           'filename': self.path,
+                           'size': 0,
+                       }))
+        vm.run_job('bdc-file-job')
+        log(vm.command('blockdev-add', driver='file',
+                       node_name=file_node_name, filename=self.path))
+
+        log(vm.command('blockdev-create', job_id='bdc-fmt-job',
+                       options={
+                           'driver': fmt,
+                           'file': file_node_name,
+                           'size': size,
+                       }))
+        vm.run_job('bdc-fmt-job')
+        log(vm.command('blockdev-add', driver=fmt,
+                       node_name=name,
+                       file=file_node_name))
+        self.fmt = fmt
+        self.size = size
+        self.node = name
+
+def query_bitmaps(vm):
+    res = vm.qmp("query-block")
+    return {"bitmaps": {device['device'] or device['qdev']:
+                        device.get('dirty-bitmaps', []) for
+                        device in res['return']}}
+
+def get_bitmap(bitmaps, drivename, name):
+    for bitmap in bitmaps['bitmaps'][drivename]:
+        if bitmap.get('name', '') == name:
+            return bitmap
+    return None
+
+def reference_backup(drive, n, filepath):
+    log("--- Reference Backup #{:d} ---\n".format(n))
+    target_id = "ref_target_{:d}".format(n)
+    job_id = "ref_backup_{:d}".format(n)
+    target_drive = Drive(filepath, vm=drive.vm)
+
+    target_drive.create_target(target_id, drive.fmt, drive.size)
+    drive.vm.qmp_log("blockdev-backup",
+                     job_id=job_id, device=drive.name,
+                     target=target_id, sync="full")
+    drive.vm.run_job(job_id, auto_dismiss=True)
+    log('')
+
+def bitmap_backup(drive, n, filepath, bitmap, bitmap_mode):
+    log("--- Bitmap Backup #{:d} ---\n".format(n))
+    target_id = "bitmap_target_{:d}".format(n)
+    job_id = "bitmap_backup_{:d}".format(n)
+    target_drive = Drive(filepath, vm=drive.vm)
+
+    target_drive.create_target(target_id, drive.fmt, drive.size)
+    drive.vm.qmp_log("blockdev-backup", job_id=job_id, device=drive.name,
+                     target=target_id, sync="bitmap",
+                     bitmap_mode=bitmap_mode,
+                     bitmap=bitmap,
+                     auto_finalize=False)
+    return job_id
+
+def perform_writes(drive, n):
+    log("--- Write #{:d} ---\n".format(n))
+    for pattern in GROUPS[n].patterns:
+        cmd = "write -P{:s} 0x{:07x} 0x{:x}".format(
+            pattern.byte,
+            pattern.offset,
+            pattern.size)
+        log(cmd)
+        log(drive.vm.hmp_qemu_io(drive.name, cmd))
+    bitmaps = query_bitmaps(drive.vm)
+    log(bitmaps, indent=2)
+    log('')
+    return bitmaps
+
+def calculate_bits(groups=None):
+    """Calculate how many bits we expect to see dirtied."""
+    if groups:
+        bits = set.union(*(GROUPS[group].bits(GRANULARITY) for group in groups))
+        return len(bits)
+    return 0
+
+def bitmap_comparison(bitmap, groups=None, want=0):
+    """
+    Print a nice human-readable message checking that this bitmap has as
+    many bits set as we expect it to.
+    """
+    log("= Checking Bitmap {:s} =".format(bitmap.get('name', '(anonymous)')))
+
+    if groups:
+        want = calculate_bits(groups)
+    have = bitmap['count'] // bitmap['granularity']
+
+    log("expecting {:d} dirty sectors; have {:d}. {:s}".format(
+        want, have, "OK!" if want == have else "ERROR!"))
+    log('')
+
+def compare_images(image, reference, baseimg=None, expected_match=True):
+    """
+    Print a nice human-readable message comparing these images.
+    """
+    expected_ret = 0 if expected_match else 1
+    if baseimg:
+        assert qemu_img("rebase", "-u", "-b", baseimg, image) == 0
+    ret = qemu_img("compare", image, reference)
+    log('qemu_img compare "{:s}" "{:s}" ==> {:s}, {:s}'.format(
+        image, reference,
+        "Identical" if ret == 0 else "Mismatch",
+        "OK!" if ret == expected_ret else "ERROR!"),
+        filters=[iotests.filter_testfiles])
+
+def test_bitmap_sync(bsync_mode, failure=None):
+    """
+    Test bitmap backup routines.
+
+    :param bsync_mode: Is the Bitmap Sync mode, and can be any of:
+        - conditional: This is the "incremental" style mode. Bitmaps are
+                       synchronized to what was copied out only on success.
+                       (Partial images must be discarded.)
+        - never:       This is the "differential" style mode.
+                       Bitmaps are never synchronized.
+        - always:      This is a "best effort" style mode.
+                       Bitmaps are always synchronized, regardless of failure.
+                       (Partial images must be kept.)
+
+    :param failure: Is the (optional) failure mode, and can be any of:
+        - None:         No failure. Test the normative path. Default.
+        - simulated:    Cancel the job right before it completes.
+                        This also tests writes "during" the job.
+        - intermediate: This tests a job that fails mid-process and produces
+                        an incomplete backup. Testing limitations prevent
+                        testing competing writes.
+    """
+    with iotests.FilePaths(['img', 'bsync1', 'bsync2',
+                            'fbackup0', 'fbackup1', 'fbackup2']) as \
+                            (img_path, bsync1, bsync2,
+                             fbackup0, fbackup1, fbackup2), \
+         iotests.VM() as vm:
+
+        mode = "Bitmap Sync Mode {:s}".format(bsync_mode)
+        preposition = "with" if failure else "without"
+        cond = "{:s} {:s}".format(preposition,
+                                  "{:s} failure".format(failure) if failure
+                                  else "failure")
+        log("\n=== {:s} {:s} ===\n".format(mode, cond))
+
+        log('--- Preparing image & VM ---\n')
+        drive0 = Drive(img_path, vm=vm)
+        drive0.img_create(iotests.imgfmt, SIZE)
+        vm.add_device("{},id=scsi0".format(iotests.get_virtio_scsi_device()))
+        vm.launch()
+
+        file_config = {
+            'driver': 'file',
+            'filename': drive0.path
+        }
+
+        if failure == 'intermediate':
+            file_config = {
+                'driver': 'blkdebug',
+                'image': file_config,
+                'set-state': [{
+                    'event': 'flush_to_disk',
+                    'state': 1,
+                    'new_state': 2
+                }, {
+                    'event': 'read_aio',
+                    'state': 2,
+                    'new_state': 3
+                }],
+                'inject-error': [{
+                    'event': 'read_aio',
+                    'errno': 5,
+                    'state': 3,
+                    'immediately': False,
+                    'once': True
+                }]
+            }
+
+        vm.qmp_log('blockdev-add',
+                   filters=[iotests.filter_qmp_testfiles],
+                   node_name="drive0",
+                   driver=drive0.fmt,
+                   file=file_config)
+        drive0.node = 'drive0'
+        drive0.device = 'device0'
+        # Use share-rw to allow writes directly to the node;
+        # The anonymous block-backend for this configuration prevents us
+        # from using HMP's qemu-io commands to address the device.
+        vm.qmp_log("device_add", id=drive0.device,
+                   drive=drive0.name, driver="scsi-hd",
+                   share_rw=True)
+        log('')
+
+        # 0 - Writes and Reference Backup
+        perform_writes(drive0, 0)
+        reference_backup(drive0, 0, fbackup0)
+        log('--- Add Bitmap ---\n')
+        vm.qmp_log("block-dirty-bitmap-add", node=drive0.name,
+                   name="bitmap0", granularity=GRANULARITY)
+        log('')
+
+        # 1 - Writes and Reference Backup
+        bitmaps = perform_writes(drive0, 1)
+        dirty_groups = {1}
+        bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0')
+        bitmap_comparison(bitmap, groups=dirty_groups)
+        reference_backup(drive0, 1, fbackup1)
+
+        # 1 - Bitmap Backup (Optional induced failure)
+        if failure == 'intermediate':
+            # Activate blkdebug induced failure for second-to-next read
+            log(vm.hmp_qemu_io(drive0.name, 'flush'))
+            log('')
+        job = bitmap_backup(drive0, 1, bsync1, "bitmap0", bsync_mode)
+
+        def _callback():
+            """Issue writes while the job is open to test bitmap divergence."""
+            # Note: when `failure` is 'intermediate', this isn't called.
+            log('')
+            bitmaps = perform_writes(drive0, 2)
+            # Named bitmap (static, should be unchanged)
+            bitmap_comparison(get_bitmap(bitmaps, drive0.device, 'bitmap0'),
+                              groups=dirty_groups)
+            # Anonymous bitmap (dynamic, shows new writes)
+            bitmap_comparison(get_bitmap(bitmaps, drive0.device, ''),
+                              groups={2})
+            dirty_groups.add(2)
+
+        vm.run_job(job, auto_dismiss=True, auto_finalize=False,
+                   pre_finalize=_callback,
+                   cancel=(failure == 'simulated'))
+        bitmaps = query_bitmaps(vm)
+        bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0')
+        log(bitmaps, indent=2)
+        log('')
+
+        if ((bsync_mode == 'conditional' and not failure) or
+                (bsync_mode == 'always' and failure != 'intermediate')):
+            dirty_groups.remove(1)
+
+        if bsync_mode == 'always' and failure == 'intermediate':
+            # We manage to copy one sector (one bit) before the error.
+            bitmap_comparison(bitmap,
+                              want=calculate_bits(groups=dirty_groups) - 1)
+        else:
+            bitmap_comparison(bitmap, groups=dirty_groups)
+
+        # 2 - Writes and Reference Backup
+        bitmaps = perform_writes(drive0, 3)
+        dirty_groups.add(3)
+        bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0')
+        if bsync_mode == 'always' and failure == 'intermediate':
+            # We're one bit short, still.
+            bitmap_comparison(bitmap,
+                              want=calculate_bits(groups=dirty_groups) - 1)
+        else:
+            bitmap_comparison(bitmap, groups=dirty_groups)
+        reference_backup(drive0, 2, fbackup2)
+
+        # 2 - Bitmap Backup (In failure modes, this is a recovery.)
+        job = bitmap_backup(drive0, 2, bsync2, "bitmap0", bsync_mode)
+        vm.run_job(job, auto_dismiss=True, auto_finalize=False)
+        bitmaps = query_bitmaps(vm)
+        bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0')
+        log(bitmaps, indent=2)
+        log('')
+        bitmap_comparison(bitmap, groups={}
+                          if bsync_mode != 'never'
+                          else dirty_groups)
+
+        log('--- Cleanup ---\n')
+        vm.qmp_log("block-dirty-bitmap-remove",
+                   node=drive0.name, name="bitmap0")
+        log(query_bitmaps(vm), indent=2)
+        vm.shutdown()
+        log('')
+
+        log('--- Verification ---\n')
+        # 'simulated' failures will actually all pass here because we canceled
+        # while "pending". This is actually undefined behavior,
+        # don't rely on this to be true!
+        compare_images(bsync1, fbackup1, baseimg=fbackup0,
+                       expected_match=failure != 'intermediate')
+        if not failure or bsync_mode == 'always':
+            # Always keep the last backup on success or when using 'always'
+            base = bsync1
+        else:
+            base = fbackup0
+        compare_images(bsync2, fbackup2, baseimg=base)
+        compare_images(img_path, fbackup2)
+        log('')
+
+def main():
+    for bsync_mode in ("never", "conditional", "always"):
+        for failure in ("simulated", "intermediate", None):
+            test_bitmap_sync(bsync_mode, failure)
+
+if __name__ == '__main__':
+    iotests.script_main(main, supported_fmts=['qcow2'])
diff --git a/tests/qemu-iotests/257.out b/tests/qemu-iotests/257.out
new file mode 100644
index 0000000000..bc541c836f
--- /dev/null
+++ b/tests/qemu-iotests/257.out
@@ -0,0 +1,2199 @@
+
+=== Bitmap Sync Mode never with simulated failure ===
+
+--- Preparing image & VM ---
+
+{"execute": "blockdev-add", "arguments": {"driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/PID-img"}, "node-name": "drive0"}}
+{"return": {}}
+{"execute": "device_add", "arguments": {"drive": "drive0", "driver": "scsi-hd", "id": "device0", "share-rw": true}}
+{"return": {}}
+
+--- Write #0 ---
+
+write -P0x49 0x0000000 0x10000
+{"return": ""}
+write -P0x6c 0x0100000 0x10000
+{"return": ""}
+write -P0x6f 0x2000000 0x10000
+{"return": ""}
+write -P0x76 0x3ff0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Reference Backup #0 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}}
+{"return": {}}
+{"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Add Bitmap ---
+
+{"execute": "block-dirty-bitmap-add", "arguments": {"granularity": 65536, "name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+
+--- Write #1 ---
+
+write -P0x65 0x0000000 0x10000
+{"return": ""}
+write -P0x77 0x00f8000 0x10000
+{"return": ""}
+write -P0x72 0x2008000 0x10000
+{"return": ""}
+write -P0x69 0x3fe0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+--- Reference Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}}
+{"return": {}}
+{"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Bitmap Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "job-id": "bitmap_backup_1", "sync": "bitmap", "target": "bitmap_target_1"}}
+{"return": {}}
+
+--- Write #2 ---
+
+write -P0x74 0x0010000 0x10000
+{"return": ""}
+write -P0x69 0x00e8000 0x10000
+{"return": ""}
+write -P0x6e 0x2018000 0x10000
+{"return": ""}
+write -P0x67 0x3fe0000 0x20000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      },
+      {
+        "busy": true,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "frozen"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+= Checking Bitmap (anonymous) =
+expecting 7 dirty sectors; have 7. OK!
+
+{"execute": "job-cancel", "arguments": {"id": "bitmap_backup_1"}}
+{"return": {}}
+{"data": {"id": "bitmap_backup_1", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 655360,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 10 dirty sectors; have 10. OK!
+
+--- Write #3 ---
+
+write -P0xaa 0x0010000 0x30000
+{"return": ""}
+write -P0xbb 0x00d8000 0x10000
+{"return": ""}
+write -P0xcc 0x2028000 0x10000
+{"return": ""}
+write -P0xdd 0x3fc0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 983040,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 15 dirty sectors; have 15. OK!
+
+--- Reference Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}}
+{"return": {}}
+{"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Bitmap Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "job-id": "bitmap_backup_2", "sync": "bitmap", "target": "bitmap_target_2"}}
+{"return": {}}
+{"execute": "job-finalize", "arguments": {"id": "bitmap_backup_2"}}
+{"return": {}}
+{"data": {"id": "bitmap_backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 983040,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 15 dirty sectors; have 15. OK!
+
+--- Cleanup ---
+
+{"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Verification ---
+
+qemu_img compare "TEST_DIR/PID-bsync1" "TEST_DIR/PID-fbackup1" ==> Identical, OK!
+qemu_img compare "TEST_DIR/PID-bsync2" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+qemu_img compare "TEST_DIR/PID-img" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+
+
+=== Bitmap Sync Mode never with intermediate failure ===
+
+--- Preparing image & VM ---
+
+{"execute": "blockdev-add", "arguments": {"driver": "qcow2", "file": {"driver": "blkdebug", "image": {"driver": "file", "filename": "TEST_DIR/PID-img"}, "inject-error": [{"errno": 5, "event": "read_aio", "immediately": false, "once": true, "state": 3}], "set-state": [{"event": "flush_to_disk", "new-state": 2, "state": 1}, {"event": "read_aio", "new-state": 3, "state": 2}]}, "node-name": "drive0"}}
+{"return": {}}
+{"execute": "device_add", "arguments": {"drive": "drive0", "driver": "scsi-hd", "id": "device0", "share-rw": true}}
+{"return": {}}
+
+--- Write #0 ---
+
+write -P0x49 0x0000000 0x10000
+{"return": ""}
+write -P0x6c 0x0100000 0x10000
+{"return": ""}
+write -P0x6f 0x2000000 0x10000
+{"return": ""}
+write -P0x76 0x3ff0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Reference Backup #0 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}}
+{"return": {}}
+{"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Add Bitmap ---
+
+{"execute": "block-dirty-bitmap-add", "arguments": {"granularity": 65536, "name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+
+--- Write #1 ---
+
+write -P0x65 0x0000000 0x10000
+{"return": ""}
+write -P0x77 0x00f8000 0x10000
+{"return": ""}
+write -P0x72 0x2008000 0x10000
+{"return": ""}
+write -P0x69 0x3fe0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+--- Reference Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}}
+{"return": {}}
+{"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+{"return": ""}
+
+--- Bitmap Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "job-id": "bitmap_backup_1", "sync": "bitmap", "target": "bitmap_target_1"}}
+{"return": {}}
+{"data": {"action": "report", "device": "bitmap_backup_1", "operation": "read"}, "event": "BLOCK_JOB_ERROR", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_1", "error": "Input/output error", "len": 67108864, "offset": 66781184, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+--- Write #3 ---
+
+write -P0xaa 0x0010000 0x30000
+{"return": ""}
+write -P0xbb 0x00d8000 0x10000
+{"return": ""}
+write -P0xcc 0x2028000 0x10000
+{"return": ""}
+write -P0xdd 0x3fc0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 917504,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 14 dirty sectors; have 14. OK!
+
+--- Reference Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}}
+{"return": {}}
+{"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Bitmap Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "job-id": "bitmap_backup_2", "sync": "bitmap", "target": "bitmap_target_2"}}
+{"return": {}}
+{"execute": "job-finalize", "arguments": {"id": "bitmap_backup_2"}}
+{"return": {}}
+{"data": {"id": "bitmap_backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 917504,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 14 dirty sectors; have 14. OK!
+
+--- Cleanup ---
+
+{"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Verification ---
+
+qemu_img compare "TEST_DIR/PID-bsync1" "TEST_DIR/PID-fbackup1" ==> Mismatch, OK!
+qemu_img compare "TEST_DIR/PID-bsync2" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+qemu_img compare "TEST_DIR/PID-img" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+
+
+=== Bitmap Sync Mode never without failure ===
+
+--- Preparing image & VM ---
+
+{"execute": "blockdev-add", "arguments": {"driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/PID-img"}, "node-name": "drive0"}}
+{"return": {}}
+{"execute": "device_add", "arguments": {"drive": "drive0", "driver": "scsi-hd", "id": "device0", "share-rw": true}}
+{"return": {}}
+
+--- Write #0 ---
+
+write -P0x49 0x0000000 0x10000
+{"return": ""}
+write -P0x6c 0x0100000 0x10000
+{"return": ""}
+write -P0x6f 0x2000000 0x10000
+{"return": ""}
+write -P0x76 0x3ff0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Reference Backup #0 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}}
+{"return": {}}
+{"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Add Bitmap ---
+
+{"execute": "block-dirty-bitmap-add", "arguments": {"granularity": 65536, "name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+
+--- Write #1 ---
+
+write -P0x65 0x0000000 0x10000
+{"return": ""}
+write -P0x77 0x00f8000 0x10000
+{"return": ""}
+write -P0x72 0x2008000 0x10000
+{"return": ""}
+write -P0x69 0x3fe0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+--- Reference Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}}
+{"return": {}}
+{"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Bitmap Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "job-id": "bitmap_backup_1", "sync": "bitmap", "target": "bitmap_target_1"}}
+{"return": {}}
+
+--- Write #2 ---
+
+write -P0x74 0x0010000 0x10000
+{"return": ""}
+write -P0x69 0x00e8000 0x10000
+{"return": ""}
+write -P0x6e 0x2018000 0x10000
+{"return": ""}
+write -P0x67 0x3fe0000 0x20000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      },
+      {
+        "busy": true,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "frozen"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+= Checking Bitmap (anonymous) =
+expecting 7 dirty sectors; have 7. OK!
+
+{"execute": "job-finalize", "arguments": {"id": "bitmap_backup_1"}}
+{"return": {}}
+{"data": {"id": "bitmap_backup_1", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 655360,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 10 dirty sectors; have 10. OK!
+
+--- Write #3 ---
+
+write -P0xaa 0x0010000 0x30000
+{"return": ""}
+write -P0xbb 0x00d8000 0x10000
+{"return": ""}
+write -P0xcc 0x2028000 0x10000
+{"return": ""}
+write -P0xdd 0x3fc0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 983040,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 15 dirty sectors; have 15. OK!
+
+--- Reference Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}}
+{"return": {}}
+{"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Bitmap Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "job-id": "bitmap_backup_2", "sync": "bitmap", "target": "bitmap_target_2"}}
+{"return": {}}
+{"execute": "job-finalize", "arguments": {"id": "bitmap_backup_2"}}
+{"return": {}}
+{"data": {"id": "bitmap_backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 983040,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 15 dirty sectors; have 15. OK!
+
+--- Cleanup ---
+
+{"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Verification ---
+
+qemu_img compare "TEST_DIR/PID-bsync1" "TEST_DIR/PID-fbackup1" ==> Identical, OK!
+qemu_img compare "TEST_DIR/PID-bsync2" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+qemu_img compare "TEST_DIR/PID-img" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+
+
+=== Bitmap Sync Mode conditional with simulated failure ===
+
+--- Preparing image & VM ---
+
+{"execute": "blockdev-add", "arguments": {"driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/PID-img"}, "node-name": "drive0"}}
+{"return": {}}
+{"execute": "device_add", "arguments": {"drive": "drive0", "driver": "scsi-hd", "id": "device0", "share-rw": true}}
+{"return": {}}
+
+--- Write #0 ---
+
+write -P0x49 0x0000000 0x10000
+{"return": ""}
+write -P0x6c 0x0100000 0x10000
+{"return": ""}
+write -P0x6f 0x2000000 0x10000
+{"return": ""}
+write -P0x76 0x3ff0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Reference Backup #0 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}}
+{"return": {}}
+{"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Add Bitmap ---
+
+{"execute": "block-dirty-bitmap-add", "arguments": {"granularity": 65536, "name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+
+--- Write #1 ---
+
+write -P0x65 0x0000000 0x10000
+{"return": ""}
+write -P0x77 0x00f8000 0x10000
+{"return": ""}
+write -P0x72 0x2008000 0x10000
+{"return": ""}
+write -P0x69 0x3fe0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+--- Reference Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}}
+{"return": {}}
+{"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Bitmap Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "conditional", "device": "drive0", "job-id": "bitmap_backup_1", "sync": "bitmap", "target": "bitmap_target_1"}}
+{"return": {}}
+
+--- Write #2 ---
+
+write -P0x74 0x0010000 0x10000
+{"return": ""}
+write -P0x69 0x00e8000 0x10000
+{"return": ""}
+write -P0x6e 0x2018000 0x10000
+{"return": ""}
+write -P0x67 0x3fe0000 0x20000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      },
+      {
+        "busy": true,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "frozen"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+= Checking Bitmap (anonymous) =
+expecting 7 dirty sectors; have 7. OK!
+
+{"execute": "job-cancel", "arguments": {"id": "bitmap_backup_1"}}
+{"return": {}}
+{"data": {"id": "bitmap_backup_1", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 655360,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 10 dirty sectors; have 10. OK!
+
+--- Write #3 ---
+
+write -P0xaa 0x0010000 0x30000
+{"return": ""}
+write -P0xbb 0x00d8000 0x10000
+{"return": ""}
+write -P0xcc 0x2028000 0x10000
+{"return": ""}
+write -P0xdd 0x3fc0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 983040,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 15 dirty sectors; have 15. OK!
+
+--- Reference Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}}
+{"return": {}}
+{"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Bitmap Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "conditional", "device": "drive0", "job-id": "bitmap_backup_2", "sync": "bitmap", "target": "bitmap_target_2"}}
+{"return": {}}
+{"execute": "job-finalize", "arguments": {"id": "bitmap_backup_2"}}
+{"return": {}}
+{"data": {"id": "bitmap_backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 0,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 0 dirty sectors; have 0. OK!
+
+--- Cleanup ---
+
+{"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Verification ---
+
+qemu_img compare "TEST_DIR/PID-bsync1" "TEST_DIR/PID-fbackup1" ==> Identical, OK!
+qemu_img compare "TEST_DIR/PID-bsync2" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+qemu_img compare "TEST_DIR/PID-img" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+
+
+=== Bitmap Sync Mode conditional with intermediate failure ===
+
+--- Preparing image & VM ---
+
+{"execute": "blockdev-add", "arguments": {"driver": "qcow2", "file": {"driver": "blkdebug", "image": {"driver": "file", "filename": "TEST_DIR/PID-img"}, "inject-error": [{"errno": 5, "event": "read_aio", "immediately": false, "once": true, "state": 3}], "set-state": [{"event": "flush_to_disk", "new-state": 2, "state": 1}, {"event": "read_aio", "new-state": 3, "state": 2}]}, "node-name": "drive0"}}
+{"return": {}}
+{"execute": "device_add", "arguments": {"drive": "drive0", "driver": "scsi-hd", "id": "device0", "share-rw": true}}
+{"return": {}}
+
+--- Write #0 ---
+
+write -P0x49 0x0000000 0x10000
+{"return": ""}
+write -P0x6c 0x0100000 0x10000
+{"return": ""}
+write -P0x6f 0x2000000 0x10000
+{"return": ""}
+write -P0x76 0x3ff0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Reference Backup #0 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}}
+{"return": {}}
+{"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Add Bitmap ---
+
+{"execute": "block-dirty-bitmap-add", "arguments": {"granularity": 65536, "name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+
+--- Write #1 ---
+
+write -P0x65 0x0000000 0x10000
+{"return": ""}
+write -P0x77 0x00f8000 0x10000
+{"return": ""}
+write -P0x72 0x2008000 0x10000
+{"return": ""}
+write -P0x69 0x3fe0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+--- Reference Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}}
+{"return": {}}
+{"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+{"return": ""}
+
+--- Bitmap Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "conditional", "device": "drive0", "job-id": "bitmap_backup_1", "sync": "bitmap", "target": "bitmap_target_1"}}
+{"return": {}}
+{"data": {"action": "report", "device": "bitmap_backup_1", "operation": "read"}, "event": "BLOCK_JOB_ERROR", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_1", "error": "Input/output error", "len": 67108864, "offset": 66781184, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+--- Write #3 ---
+
+write -P0xaa 0x0010000 0x30000
+{"return": ""}
+write -P0xbb 0x00d8000 0x10000
+{"return": ""}
+write -P0xcc 0x2028000 0x10000
+{"return": ""}
+write -P0xdd 0x3fc0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 917504,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 14 dirty sectors; have 14. OK!
+
+--- Reference Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}}
+{"return": {}}
+{"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Bitmap Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "conditional", "device": "drive0", "job-id": "bitmap_backup_2", "sync": "bitmap", "target": "bitmap_target_2"}}
+{"return": {}}
+{"execute": "job-finalize", "arguments": {"id": "bitmap_backup_2"}}
+{"return": {}}
+{"data": {"id": "bitmap_backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 0,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 0 dirty sectors; have 0. OK!
+
+--- Cleanup ---
+
+{"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Verification ---
+
+qemu_img compare "TEST_DIR/PID-bsync1" "TEST_DIR/PID-fbackup1" ==> Mismatch, OK!
+qemu_img compare "TEST_DIR/PID-bsync2" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+qemu_img compare "TEST_DIR/PID-img" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+
+
+=== Bitmap Sync Mode conditional without failure ===
+
+--- Preparing image & VM ---
+
+{"execute": "blockdev-add", "arguments": {"driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/PID-img"}, "node-name": "drive0"}}
+{"return": {}}
+{"execute": "device_add", "arguments": {"drive": "drive0", "driver": "scsi-hd", "id": "device0", "share-rw": true}}
+{"return": {}}
+
+--- Write #0 ---
+
+write -P0x49 0x0000000 0x10000
+{"return": ""}
+write -P0x6c 0x0100000 0x10000
+{"return": ""}
+write -P0x6f 0x2000000 0x10000
+{"return": ""}
+write -P0x76 0x3ff0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Reference Backup #0 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}}
+{"return": {}}
+{"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Add Bitmap ---
+
+{"execute": "block-dirty-bitmap-add", "arguments": {"granularity": 65536, "name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+
+--- Write #1 ---
+
+write -P0x65 0x0000000 0x10000
+{"return": ""}
+write -P0x77 0x00f8000 0x10000
+{"return": ""}
+write -P0x72 0x2008000 0x10000
+{"return": ""}
+write -P0x69 0x3fe0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+--- Reference Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}}
+{"return": {}}
+{"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Bitmap Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "conditional", "device": "drive0", "job-id": "bitmap_backup_1", "sync": "bitmap", "target": "bitmap_target_1"}}
+{"return": {}}
+
+--- Write #2 ---
+
+write -P0x74 0x0010000 0x10000
+{"return": ""}
+write -P0x69 0x00e8000 0x10000
+{"return": ""}
+write -P0x6e 0x2018000 0x10000
+{"return": ""}
+write -P0x67 0x3fe0000 0x20000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      },
+      {
+        "busy": true,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "frozen"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+= Checking Bitmap (anonymous) =
+expecting 7 dirty sectors; have 7. OK!
+
+{"execute": "job-finalize", "arguments": {"id": "bitmap_backup_1"}}
+{"return": {}}
+{"data": {"id": "bitmap_backup_1", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 7 dirty sectors; have 7. OK!
+
+--- Write #3 ---
+
+write -P0xaa 0x0010000 0x30000
+{"return": ""}
+write -P0xbb 0x00d8000 0x10000
+{"return": ""}
+write -P0xcc 0x2028000 0x10000
+{"return": ""}
+write -P0xdd 0x3fc0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 786432,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 12 dirty sectors; have 12. OK!
+
+--- Reference Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}}
+{"return": {}}
+{"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Bitmap Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "conditional", "device": "drive0", "job-id": "bitmap_backup_2", "sync": "bitmap", "target": "bitmap_target_2"}}
+{"return": {}}
+{"execute": "job-finalize", "arguments": {"id": "bitmap_backup_2"}}
+{"return": {}}
+{"data": {"id": "bitmap_backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 0,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 0 dirty sectors; have 0. OK!
+
+--- Cleanup ---
+
+{"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Verification ---
+
+qemu_img compare "TEST_DIR/PID-bsync1" "TEST_DIR/PID-fbackup1" ==> Identical, OK!
+qemu_img compare "TEST_DIR/PID-bsync2" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+qemu_img compare "TEST_DIR/PID-img" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+
+
+=== Bitmap Sync Mode always with simulated failure ===
+
+--- Preparing image & VM ---
+
+{"execute": "blockdev-add", "arguments": {"driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/PID-img"}, "node-name": "drive0"}}
+{"return": {}}
+{"execute": "device_add", "arguments": {"drive": "drive0", "driver": "scsi-hd", "id": "device0", "share-rw": true}}
+{"return": {}}
+
+--- Write #0 ---
+
+write -P0x49 0x0000000 0x10000
+{"return": ""}
+write -P0x6c 0x0100000 0x10000
+{"return": ""}
+write -P0x6f 0x2000000 0x10000
+{"return": ""}
+write -P0x76 0x3ff0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Reference Backup #0 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}}
+{"return": {}}
+{"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Add Bitmap ---
+
+{"execute": "block-dirty-bitmap-add", "arguments": {"granularity": 65536, "name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+
+--- Write #1 ---
+
+write -P0x65 0x0000000 0x10000
+{"return": ""}
+write -P0x77 0x00f8000 0x10000
+{"return": ""}
+write -P0x72 0x2008000 0x10000
+{"return": ""}
+write -P0x69 0x3fe0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+--- Reference Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}}
+{"return": {}}
+{"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Bitmap Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "job-id": "bitmap_backup_1", "sync": "bitmap", "target": "bitmap_target_1"}}
+{"return": {}}
+
+--- Write #2 ---
+
+write -P0x74 0x0010000 0x10000
+{"return": ""}
+write -P0x69 0x00e8000 0x10000
+{"return": ""}
+write -P0x6e 0x2018000 0x10000
+{"return": ""}
+write -P0x67 0x3fe0000 0x20000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      },
+      {
+        "busy": true,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "frozen"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+= Checking Bitmap (anonymous) =
+expecting 7 dirty sectors; have 7. OK!
+
+{"execute": "job-cancel", "arguments": {"id": "bitmap_backup_1"}}
+{"return": {}}
+{"data": {"id": "bitmap_backup_1", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 7 dirty sectors; have 7. OK!
+
+--- Write #3 ---
+
+write -P0xaa 0x0010000 0x30000
+{"return": ""}
+write -P0xbb 0x00d8000 0x10000
+{"return": ""}
+write -P0xcc 0x2028000 0x10000
+{"return": ""}
+write -P0xdd 0x3fc0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 786432,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 12 dirty sectors; have 12. OK!
+
+--- Reference Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}}
+{"return": {}}
+{"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Bitmap Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "job-id": "bitmap_backup_2", "sync": "bitmap", "target": "bitmap_target_2"}}
+{"return": {}}
+{"execute": "job-finalize", "arguments": {"id": "bitmap_backup_2"}}
+{"return": {}}
+{"data": {"id": "bitmap_backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 0,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 0 dirty sectors; have 0. OK!
+
+--- Cleanup ---
+
+{"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Verification ---
+
+qemu_img compare "TEST_DIR/PID-bsync1" "TEST_DIR/PID-fbackup1" ==> Identical, OK!
+qemu_img compare "TEST_DIR/PID-bsync2" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+qemu_img compare "TEST_DIR/PID-img" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+
+
+=== Bitmap Sync Mode always with intermediate failure ===
+
+--- Preparing image & VM ---
+
+{"execute": "blockdev-add", "arguments": {"driver": "qcow2", "file": {"driver": "blkdebug", "image": {"driver": "file", "filename": "TEST_DIR/PID-img"}, "inject-error": [{"errno": 5, "event": "read_aio", "immediately": false, "once": true, "state": 3}], "set-state": [{"event": "flush_to_disk", "new-state": 2, "state": 1}, {"event": "read_aio", "new-state": 3, "state": 2}]}, "node-name": "drive0"}}
+{"return": {}}
+{"execute": "device_add", "arguments": {"drive": "drive0", "driver": "scsi-hd", "id": "device0", "share-rw": true}}
+{"return": {}}
+
+--- Write #0 ---
+
+write -P0x49 0x0000000 0x10000
+{"return": ""}
+write -P0x6c 0x0100000 0x10000
+{"return": ""}
+write -P0x6f 0x2000000 0x10000
+{"return": ""}
+write -P0x76 0x3ff0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Reference Backup #0 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}}
+{"return": {}}
+{"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Add Bitmap ---
+
+{"execute": "block-dirty-bitmap-add", "arguments": {"granularity": 65536, "name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+
+--- Write #1 ---
+
+write -P0x65 0x0000000 0x10000
+{"return": ""}
+write -P0x77 0x00f8000 0x10000
+{"return": ""}
+write -P0x72 0x2008000 0x10000
+{"return": ""}
+write -P0x69 0x3fe0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+--- Reference Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}}
+{"return": {}}
+{"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+{"return": ""}
+
+--- Bitmap Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "job-id": "bitmap_backup_1", "sync": "bitmap", "target": "bitmap_target_1"}}
+{"return": {}}
+{"data": {"action": "report", "device": "bitmap_backup_1", "operation": "read"}, "event": "BLOCK_JOB_ERROR", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_1", "error": "Input/output error", "len": 67108864, "offset": 66781184, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 327680,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 5 dirty sectors; have 5. OK!
+
+--- Write #3 ---
+
+write -P0xaa 0x0010000 0x30000
+{"return": ""}
+write -P0xbb 0x00d8000 0x10000
+{"return": ""}
+write -P0xcc 0x2028000 0x10000
+{"return": ""}
+write -P0xdd 0x3fc0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 851968,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 13 dirty sectors; have 13. OK!
+
+--- Reference Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}}
+{"return": {}}
+{"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Bitmap Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "job-id": "bitmap_backup_2", "sync": "bitmap", "target": "bitmap_target_2"}}
+{"return": {}}
+{"execute": "job-finalize", "arguments": {"id": "bitmap_backup_2"}}
+{"return": {}}
+{"data": {"id": "bitmap_backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 0,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 0 dirty sectors; have 0. OK!
+
+--- Cleanup ---
+
+{"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Verification ---
+
+qemu_img compare "TEST_DIR/PID-bsync1" "TEST_DIR/PID-fbackup1" ==> Mismatch, OK!
+qemu_img compare "TEST_DIR/PID-bsync2" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+qemu_img compare "TEST_DIR/PID-img" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+
+
+=== Bitmap Sync Mode always without failure ===
+
+--- Preparing image & VM ---
+
+{"execute": "blockdev-add", "arguments": {"driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/PID-img"}, "node-name": "drive0"}}
+{"return": {}}
+{"execute": "device_add", "arguments": {"drive": "drive0", "driver": "scsi-hd", "id": "device0", "share-rw": true}}
+{"return": {}}
+
+--- Write #0 ---
+
+write -P0x49 0x0000000 0x10000
+{"return": ""}
+write -P0x6c 0x0100000 0x10000
+{"return": ""}
+write -P0x6f 0x2000000 0x10000
+{"return": ""}
+write -P0x76 0x3ff0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Reference Backup #0 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_0", "sync": "full", "target": "ref_target_0"}}
+{"return": {}}
+{"data": {"device": "ref_backup_0", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Add Bitmap ---
+
+{"execute": "block-dirty-bitmap-add", "arguments": {"granularity": 65536, "name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+
+--- Write #1 ---
+
+write -P0x65 0x0000000 0x10000
+{"return": ""}
+write -P0x77 0x00f8000 0x10000
+{"return": ""}
+write -P0x72 0x2008000 0x10000
+{"return": ""}
+write -P0x69 0x3fe0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+--- Reference Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_1", "sync": "full", "target": "ref_target_1"}}
+{"return": {}}
+{"data": {"device": "ref_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Bitmap Backup #1 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "job-id": "bitmap_backup_1", "sync": "bitmap", "target": "bitmap_target_1"}}
+{"return": {}}
+
+--- Write #2 ---
+
+write -P0x74 0x0010000 0x10000
+{"return": ""}
+write -P0x69 0x00e8000 0x10000
+{"return": ""}
+write -P0x6e 0x2018000 0x10000
+{"return": ""}
+write -P0x67 0x3fe0000 0x20000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      },
+      {
+        "busy": true,
+        "count": 393216,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "frozen"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 6 dirty sectors; have 6. OK!
+
+= Checking Bitmap (anonymous) =
+expecting 7 dirty sectors; have 7. OK!
+
+{"execute": "job-finalize", "arguments": {"id": "bitmap_backup_1"}}
+{"return": {}}
+{"data": {"id": "bitmap_backup_1", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 458752,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 7 dirty sectors; have 7. OK!
+
+--- Write #3 ---
+
+write -P0xaa 0x0010000 0x30000
+{"return": ""}
+write -P0xbb 0x00d8000 0x10000
+{"return": ""}
+write -P0xcc 0x2028000 0x10000
+{"return": ""}
+write -P0xdd 0x3fc0000 0x10000
+{"return": ""}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 786432,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 12 dirty sectors; have 12. OK!
+
+--- Reference Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"device": "drive0", "job-id": "ref_backup_2", "sync": "full", "target": "ref_target_2"}}
+{"return": {}}
+{"data": {"device": "ref_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+
+--- Bitmap Backup #2 ---
+
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-file-job"}}
+{"return": {}}
+{}
+{}
+{"execute": "job-dismiss", "arguments": {"id": "bdc-fmt-job"}}
+{"return": {}}
+{}
+{"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "job-id": "bitmap_backup_2", "sync": "bitmap", "target": "bitmap_target_2"}}
+{"return": {}}
+{"execute": "job-finalize", "arguments": {"id": "bitmap_backup_2"}}
+{"return": {}}
+{"data": {"id": "bitmap_backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "bitmap_backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{
+  "bitmaps": {
+    "device0": [
+      {
+        "busy": false,
+        "count": 0,
+        "granularity": 65536,
+        "name": "bitmap0",
+        "persistent": false,
+        "recording": true,
+        "status": "active"
+      }
+    ]
+  }
+}
+
+= Checking Bitmap bitmap0 =
+expecting 0 dirty sectors; have 0. OK!
+
+--- Cleanup ---
+
+{"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmap0", "node": "drive0"}}
+{"return": {}}
+{
+  "bitmaps": {
+    "device0": []
+  }
+}
+
+--- Verification ---
+
+qemu_img compare "TEST_DIR/PID-bsync1" "TEST_DIR/PID-fbackup1" ==> Identical, OK!
+qemu_img compare "TEST_DIR/PID-bsync2" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+qemu_img compare "TEST_DIR/PID-img" "TEST_DIR/PID-fbackup2" ==> Identical, OK!
+
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index b34c8e3c0c..b53afcd149 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -269,3 +269,4 @@
 254 rw auto backing quick
 255 rw auto quick
 256 rw auto quick
+257 rw auto
-- 
2.21.0

[Qemu-devel] [PATCH v2 18/18] block/backup: loosen restriction on readonly bitmaps

[John Snow]

With the "never" sync policy, we actually can utilize readonly bitmaps
now. Loosen the check at the QMP level, and tighten it based on
provided arguments down at the job creation level instead.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 block/backup.c | 6 ++++++
 blockdev.c     | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/block/backup.c b/block/backup.c
index 495d8f71aa..4abe90d462 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -598,6 +598,12 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
             return NULL;
         }
 
+        /* If we need to write to this bitmap, check that we can: */
+        if (bitmap_mode != BITMAP_SYNC_MODE_NEVER &&
+            bdrv_dirty_bitmap_check(sync_bitmap, BDRV_BITMAP_DEFAULT, errp)) {
+            return NULL;
+        }
+
         /* Create a new bitmap, and freeze/disable this one. */
         if (bdrv_dirty_bitmap_create_successor(bs, sync_bitmap, errp) < 0) {
             return NULL;
diff --git a/blockdev.c b/blockdev.c
index d5b089a446..020ce7e960 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -3492,7 +3492,7 @@ static BlockJob *do_backup_common(BackupCommon *backup,
             error_setg(errp, "Bitmap '%s' could not be found", backup->bitmap);
             goto out;
         }
-        if (bdrv_dirty_bitmap_check(bmap, BDRV_BITMAP_DEFAULT, errp)) {
+        if (bdrv_dirty_bitmap_check(bmap, BDRV_BITMAP_ALLOW_RO, errp)) {
             goto out;
         }
     }
-- 
2.21.0

Re: [Qemu-devel] [PATCH v2 00/18] bitmaps: introduce 'bitmap' sync mode

[no-reply]

Patchew URL: https://patchew.org/QEMU/20190703215542.16123-1-jsnow@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Subject: [Qemu-devel] [PATCH v2 00/18] bitmaps: introduce 'bitmap' sync mode
Message-id: 20190703215542.16123-1-jsnow@redhat.com

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Switched to a new branch 'test'
18acfee block/backup: loosen restriction on readonly bitmaps
4852213 iotests: add test 257 for bitmap-mode backups
0d47743 iotests: Add virtio-scsi device helper
9805af4 iotests: teach FilePath to produce multiple paths
b491804 iotests: teach run_job to cancel pending jobs
2925d97 iotests: add testing shim for script-style python tests
c7870c0 block/backup: add 'always' bitmap sync policy
1a5bc8c block/backup: upgrade copy_bitmap to BdrvDirtyBitmap
a7fd32c block/dirty-bitmap: add bdrv_dirty_bitmap_get
6ec8d60 block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal
8760b33 hbitmap: enable merging across granularities
919477d hbitmap: Fix merge when b is empty, and result is not an alias of a
dd0775a block/backup: add 'never' policy to bitmap sync mode
998d008 block/backup: Add mirror sync mode 'bitmap'
5e678f1 qapi: add BitmapSyncMode enum
7e58e56 blockdev-backup: utilize do_backup_common
e372afa drive-backup: create do_backup_common
677b04c qapi/block-core: Introduce BackupCommon

=== OUTPUT BEGIN ===
1/18 Checking commit 677b04c0b13d (qapi/block-core: Introduce BackupCommon)
2/18 Checking commit e372afac0fdc (drive-backup: create do_backup_common)
3/18 Checking commit 7e58e56768db (blockdev-backup: utilize do_backup_common)
4/18 Checking commit 5e678f15e446 (qapi: add BitmapSyncMode enum)
5/18 Checking commit 998d0084971b (block/backup: Add mirror sync mode 'bitmap')
6/18 Checking commit dd0775a4cf92 (block/backup: add 'never' policy to bitmap sync mode)
WARNING: Block comments use a leading /* on a separate line
#26: FILE: block/backup.c:269:
+        /* Failure, or we don't want to synchronize the bitmap.

WARNING: Block comments use a trailing */ on a separate line
#27: FILE: block/backup.c:270:
+         * Merge the successor back into the parent, delete nothing. */

total: 0 errors, 2 warnings, 25 lines checked

Patch 6/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
7/18 Checking commit 919477d56466 (hbitmap: Fix merge when b is empty, and result is not an alias of a)
8/18 Checking commit 8760b336946d (hbitmap: enable merging across granularities)
9/18 Checking commit 6ec8d60f0329 (block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal)
10/18 Checking commit a7fd32c4817d (block/dirty-bitmap: add bdrv_dirty_bitmap_get)
ERROR: open brace '{' following function declarations go on the next line
#36: FILE: block/dirty-bitmap.c:517:
+bool bdrv_dirty_bitmap_get(BdrvDirtyBitmap *bitmap, int64_t offset) {

total: 1 errors, 0 warnings, 72 lines checked

Patch 10/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

11/18 Checking commit 1a5bc8c73750 (block/backup: upgrade copy_bitmap to BdrvDirtyBitmap)
WARNING: line over 80 characters
#60: FILE: block/backup.c:174:
+    bdrv_reset_dirty_bitmap(job->copy_bitmap, start, job->cluster_size * nr_clusters);

WARNING: line over 80 characters
#66: FILE: block/backup.c:179:
+        bdrv_set_dirty_bitmap(job->copy_bitmap, start, job->cluster_size * nr_clusters);

total: 0 errors, 2 warnings, 192 lines checked

Patch 11/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
12/18 Checking commit c7870c05906a (block/backup: add 'always' bitmap sync policy)
WARNING: Block comments use a leading /* on a separate line
#41: FILE: block/backup.c:273:
+        /* We succeeded, or we always intended to sync the bitmap.

WARNING: Block comments use a trailing */ on a separate line
#42: FILE: block/backup.c:274:
+         * Delete this bitmap and install the child. */

WARNING: Block comments use a leading /* on a separate line
#46: FILE: block/backup.c:277:
+        /* We failed, or we never intended to sync the bitmap anyway.

WARNING: Block comments use a trailing */ on a separate line
#47: FILE: block/backup.c:278:
+         * Merge the successor back into the parent, keeping all data. */

total: 0 errors, 4 warnings, 47 lines checked

Patch 12/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
13/18 Checking commit 2925d97a5717 (iotests: add testing shim for script-style python tests)
14/18 Checking commit b491804bec4e (iotests: teach run_job to cancel pending jobs)
15/18 Checking commit 9805af48ccbb (iotests: teach FilePath to produce multiple paths)
WARNING: line over 80 characters
#63: FILE: tests/qemu-iotests/iotests.py:392:
+    """FilePath is a specialization of FilePaths that takes a single filename."""

total: 0 errors, 1 warnings, 67 lines checked

Patch 15/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
16/18 Checking commit 0d47743c9c1e (iotests: Add virtio-scsi device helper)
17/18 Checking commit 485221328b28 (iotests: add test 257 for bitmap-mode backups)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#11: 
new file mode 100755

total: 0 errors, 1 warnings, 2612 lines checked

Patch 17/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
18/18 Checking commit 18acfee9986a (block/backup: loosen restriction on readonly bitmaps)
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20190703215542.16123-1-jsnow@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [PATCH v2 00/18] bitmaps: introduce 'bitmap' sync mode

[no-reply]

Patchew URL: https://patchew.org/QEMU/20190703215542.16123-1-jsnow@redhat.com/



Hi,

This series failed the asan build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
=== TEST SCRIPT END ===

PASS 32 test-opts-visitor /visitor/opts/range/beyond
PASS 33 test-opts-visitor /visitor/opts/dict/unvisited
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-coroutine -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-coroutine" 
==10176==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-coroutine /basic/no-dangling-access
PASS 2 test-coroutine /basic/lifecycle
PASS 3 test-coroutine /basic/yield
==10176==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd760c6000; bottom 0x7f81455f8000; size: 0x007c30ace000 (533392580608)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 4 test-coroutine /basic/nesting
---
PASS 1 fdc-test /x86_64/fdc/cmos
PASS 2 fdc-test /x86_64/fdc/no_media_on_start
PASS 3 fdc-test /x86_64/fdc/read_without_media
==10185==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 fdc-test /x86_64/fdc/media_change
PASS 5 fdc-test /x86_64/fdc/sense_interrupt
PASS 6 fdc-test /x86_64/fdc/relative_seek
---
PASS 11 test-aio /aio/event/wait
PASS 12 test-aio /aio/event/flush
PASS 13 test-aio /aio/event/wait/no-flush-cb
==10200==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 14 test-aio /aio/timer/schedule
PASS 15 test-aio /aio/coroutine/queue-chaining
PASS 16 test-aio /aio-gsource/flush
---
PASS 28 test-aio /aio-gsource/timer/schedule
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-aio-multithread -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-aio-multithread" 
PASS 1 test-aio-multithread /aio/multi/lifecycle
==10206==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 test-aio-multithread /aio/multi/schedule
PASS 3 test-aio-multithread /aio/multi/mutex/contended
PASS 12 fdc-test /x86_64/fdc/read_no_dma_19
PASS 13 fdc-test /x86_64/fdc/fuzz-registers
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/ide-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="ide-test" 
==10234==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 test-aio-multithread /aio/multi/mutex/handoff
PASS 1 ide-test /x86_64/ide/identify
==10245==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 5 test-aio-multithread /aio/multi/mutex/mcs
PASS 2 ide-test /x86_64/ide/flush
==10256==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 6 test-aio-multithread /aio/multi/mutex/pthread
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-throttle -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-throttle" 
PASS 1 test-throttle /throttle/leak_bucket
---
PASS 13 test-throttle /throttle/config/ranges
PASS 14 test-throttle /throttle/config/max
PASS 15 test-throttle /throttle/config/iops_size
==10264==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 ide-test /x86_64/ide/bmdma/simple_rw
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-thread-pool -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-thread-pool" 
==10273==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-thread-pool /thread-pool/submit
PASS 2 test-thread-pool /thread-pool/submit-aio
PASS 3 test-thread-pool /thread-pool/submit-co
PASS 4 test-thread-pool /thread-pool/submit-many
==10268==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 5 test-thread-pool /thread-pool/cancel
PASS 4 ide-test /x86_64/ide/bmdma/trim
==10343==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 6 test-thread-pool /thread-pool/cancel-async
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-hbitmap -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-hbitmap" 
PASS 1 test-hbitmap /hbitmap/granularity
---
PASS 16 test-hbitmap /hbitmap/reset/empty
PASS 5 ide-test /x86_64/ide/bmdma/short_prdt
PASS 17 test-hbitmap /hbitmap/reset/general
==10354==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 18 test-hbitmap /hbitmap/reset/all
PASS 19 test-hbitmap /hbitmap/truncate/nop
PASS 20 test-hbitmap /hbitmap/truncate/grow/negligible
---
PASS 29 test-hbitmap /hbitmap/truncate/shrink/large
PASS 30 test-hbitmap /hbitmap/meta/zero
PASS 6 ide-test /x86_64/ide/bmdma/one_sector_short_prdt
==10360==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 7 ide-test /x86_64/ide/bmdma/long_prdt
==10366==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==10366==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe37420000; bottom 0x7f266f1fe000; size: 0x00d7c8222000 (926775648256)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 8 ide-test /x86_64/ide/bmdma/no_busmaster
PASS 9 ide-test /x86_64/ide/flush/nodev
==10377==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 31 test-hbitmap /hbitmap/meta/one
PASS 32 test-hbitmap /hbitmap/meta/byte
PASS 33 test-hbitmap /hbitmap/meta/word
PASS 10 ide-test /x86_64/ide/flush/empty_drive
==10382==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 34 test-hbitmap /hbitmap/meta/sector
PASS 35 test-hbitmap /hbitmap/serialize/align
PASS 11 ide-test /x86_64/ide/flush/retry_pci
==10388==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 12 ide-test /x86_64/ide/flush/retry_isa
PASS 36 test-hbitmap /hbitmap/serialize/basic
PASS 37 test-hbitmap /hbitmap/serialize/part
---
PASS 39 test-hbitmap /hbitmap/next_zero/next_zero_0
PASS 40 test-hbitmap /hbitmap/next_zero/next_zero_4
PASS 41 test-hbitmap /hbitmap/next_dirty_area/next_dirty_area_0
==10394==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 42 test-hbitmap /hbitmap/next_dirty_area/next_dirty_area_1
PASS 43 test-hbitmap /hbitmap/next_dirty_area/next_dirty_area_4
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-bdrv-drain -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-bdrv-drain" 
==10401==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-bdrv-drain /bdrv-drain/nested
PASS 2 test-bdrv-drain /bdrv-drain/multiparent
PASS 3 test-bdrv-drain /bdrv-drain/set_aio_context
---
PASS 28 test-bdrv-drain /bdrv-drain/blockjob/iothread/drain
PASS 29 test-bdrv-drain /bdrv-drain/blockjob/iothread/drain_subtree
PASS 30 test-bdrv-drain /bdrv-drain/blockjob/iothread/error/drain_all
==10419==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 31 test-bdrv-drain /bdrv-drain/blockjob/iothread/error/drain
PASS 32 test-bdrv-drain /bdrv-drain/blockjob/iothread/error/drain_subtree
PASS 33 test-bdrv-drain /bdrv-drain/deletion/drain
---
PASS 39 test-bdrv-drain /bdrv-drain/attach/drain
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-bdrv-graph-mod -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-bdrv-graph-mod" 
PASS 14 ide-test /x86_64/ide/cdrom/pio_large
==10447==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-bdrv-graph-mod /bdrv-graph-mod/update-perm-tree
PASS 2 test-bdrv-graph-mod /bdrv-graph-mod/should-update-child
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-blockjob -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-blockjob" 
==10455==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-blockjob /blockjob/ids
PASS 2 test-blockjob /blockjob/cancel/created
PASS 3 test-blockjob /blockjob/cancel/running
---
PASS 6 test-blockjob /blockjob/cancel/standby
PASS 7 test-blockjob /blockjob/cancel/pending
PASS 8 test-blockjob /blockjob/cancel/concluded
==10450==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-blockjob-txn -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-blockjob-txn" 
==10464==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-blockjob-txn /single/success
PASS 2 test-blockjob-txn /single/failure
PASS 3 test-blockjob-txn /single/cancel
---
PASS 15 ide-test /x86_64/ide/cdrom/dma
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/ahci-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="ahci-test" 
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-block-backend -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-block-backend" 
==10474==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-block-backend /block-backend/drain_aio_error
PASS 2 test-block-backend /block-backend/drain_all_aio_error
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-block-iothread -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-block-iothread" 
==10482==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-block-iothread /sync-op/pread
PASS 2 test-block-iothread /sync-op/pwrite
PASS 3 test-block-iothread /sync-op/load_vmstate
---
PASS 11 test-block-iothread /attach/blockjob
PASS 12 test-block-iothread /attach/second_node
PASS 13 test-block-iothread /attach/preserve_blk_ctx
==10484==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 14 test-block-iothread /propagate/basic
PASS 15 test-block-iothread /propagate/diamond
PASS 16 test-block-iothread /propagate/mirror
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-image-locking -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-image-locking" 
==10509==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-image-locking /image-locking/basic
PASS 2 test-image-locking /image-locking/set-perm-abort
PASS 1 ahci-test /x86_64/ahci/sanity
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-x86-cpuid -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-x86-cpuid" 
PASS 1 test-x86-cpuid /cpuid/topology/basic
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-xbzrle -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-xbzrle" 
==10513==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-xbzrle /xbzrle/uleb
PASS 2 test-xbzrle /xbzrle/encode_decode_zero
PASS 3 test-xbzrle /xbzrle/encode_decode_unchanged
---
PASS 1 test-shift128 /host-utils/test_lshift
PASS 2 test-shift128 /host-utils/test_rshift
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-mul64 -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-mul64" 
==10532==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-mul64 /host-utils/mulu64
PASS 2 test-mul64 /host-utils/muls64
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-int128 -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-int128" 
---
PASS 10 test-int128 /int128/int128_rshift
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/rcutorture -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="rcutorture" 
PASS 3 ahci-test /x86_64/ahci/pci_enable
==10565==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 rcutorture /rcu/torture/1reader
PASS 4 ahci-test /x86_64/ahci/hba_spec
==10587==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 rcutorture /rcu/torture/10readers
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-rcu-list -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-rcu-list" 
PASS 5 ahci-test /x86_64/ahci/hba_enable
==10600==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-rcu-list /rcu/qlist/single-threaded
PASS 6 ahci-test /x86_64/ahci/identify
==10612==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 test-rcu-list /rcu/qlist/short-few
PASS 7 ahci-test /x86_64/ahci/max
==10639==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 8 ahci-test /x86_64/ahci/reset
PASS 3 test-rcu-list /rcu/qlist/long-many
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-rcu-simpleq -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-rcu-simpleq" 
==10645==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==10645==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffda02e1000; bottom 0x7f76747fe000; size: 0x00872bae3000 (580553420800)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 1 test-rcu-simpleq /rcu/qsimpleq/single-threaded
PASS 9 ahci-test /x86_64/ahci/io/pio/lba28/simple/zero
==10664==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 test-rcu-simpleq /rcu/qsimpleq/short-few
==10664==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fff60a02000; bottom 0x7fcd2c7fe000; size: 0x003234204000 (215622893568)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 10 ahci-test /x86_64/ahci/io/pio/lba28/simple/low
==10691==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==10691==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe11e30000; bottom 0x7f7c735fe000; size: 0x00819e832000 (556710174720)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 11 ahci-test /x86_64/ahci/io/pio/lba28/simple/high
PASS 3 test-rcu-simpleq /rcu/qsimpleq/long-many
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-rcu-tailq -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-rcu-tailq" 
==10697==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==10697==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffcbffdf000; bottom 0x7f9368ffe000; size: 0x006956fe1000 (452431056896)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 12 ahci-test /x86_64/ahci/io/pio/lba28/double/zero
==10710==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-rcu-tailq /rcu/qtailq/single-threaded
==10710==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fff684b5000; bottom 0x7f03557fe000; size: 0x00fc12cb7000 (1082647080960)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 13 ahci-test /x86_64/ahci/io/pio/lba28/double/low
PASS 2 test-rcu-tailq /rcu/qtailq/short-few
==10722==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==10722==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd94c03000; bottom 0x7fe9b7ffe000; size: 0x0013dcc05000 (85307969536)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 14 ahci-test /x86_64/ahci/io/pio/lba28/double/high
==10749==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==10749==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fffc4c13000; bottom 0x7f96b137c000; size: 0x006913897000 (451299340288)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 3 test-rcu-tailq /rcu/qtailq/long-many
---
PASS 8 test-qdist /qdist/binning/shrink
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-qht -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-qht" 
PASS 15 ahci-test /x86_64/ahci/io/pio/lba28/long/zero
==10764==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==10764==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffdc832e000; bottom 0x7f1788f24000; size: 0x00e63f40a000 (988903677952)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 16 ahci-test /x86_64/ahci/io/pio/lba28/long/low
==10770==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==10770==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fffcf80d000; bottom 0x7f70849fe000; size: 0x008f4ae0f000 (615436578816)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 17 ahci-test /x86_64/ahci/io/pio/lba28/long/high
==10776==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 18 ahci-test /x86_64/ahci/io/pio/lba28/short/zero
==10782==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 19 ahci-test /x86_64/ahci/io/pio/lba28/short/low
==10788==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 20 ahci-test /x86_64/ahci/io/pio/lba28/short/high
==10794==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==10794==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe9d54b000; bottom 0x7fec331fe000; size: 0x00126a34d000 (79091257344)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 21 ahci-test /x86_64/ahci/io/pio/lba48/simple/zero
==10800==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==10800==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe12586000; bottom 0x7fddef1fe000; size: 0x002023388000 (138029858816)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 22 ahci-test /x86_64/ahci/io/pio/lba48/simple/low
==10806==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==10806==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fff19244000; bottom 0x7f71245fe000; size: 0x008df4c46000 (609696899072)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 23 ahci-test /x86_64/ahci/io/pio/lba48/simple/high
==10812==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==10812==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd8be65000; bottom 0x7f3a543fe000; size: 0x00c337a67000 (838452277248)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 1 test-qht /qht/mode/default
PASS 2 test-qht /qht/mode/resize
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-qht-par -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-qht-par" 
PASS 24 ahci-test /x86_64/ahci/io/pio/lba48/double/zero
==10824==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-qht-par /qht/parallel/2threads-0%updates-1s
==10824==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffdafa2e000; bottom 0x7f70cabfe000; size: 0x008ce4e30000 (605135503360)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 25 ahci-test /x86_64/ahci/io/pio/lba48/double/low
==10841==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 test-qht-par /qht/parallel/2threads-20%updates-1s
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-bitops -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-bitops" 
PASS 1 test-bitops /bitops/sextract32
---
PASS 5 test-bitops /bitops/half_unshuffle32
PASS 6 test-bitops /bitops/half_unshuffle64
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-bitcnt -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-bitcnt" 
==10841==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fffaadcd000; bottom 0x7f0dea3fe000; size: 0x00f1c09cf000 (1038318628864)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 1 test-bitcnt /bitcnt/ctpop8
---
PASS 7 check-qom-proplist /qom/proplist/class_iterator
PASS 8 check-qom-proplist /qom/proplist/delchild
PASS 9 check-qom-proplist /qom/resolve/partial
==10867==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-qemu-opts -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-qemu-opts" 
PASS 1 test-qemu-opts /qemu-opts/find_unknown_opts
PASS 2 test-qemu-opts /qemu-opts/find_opts
---
PASS 9 test-keyval /keyval/visit/alternate
PASS 10 test-keyval /keyval/visit/any
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-write-threshold -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-write-threshold" 
==10867==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd080a0000; bottom 0x7f4cca324000; size: 0x00b03dd7c000 (756951793664)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 1 test-write-threshold /write-threshold/not-set-on-init
---
PASS 16 test-crypto-secret /crypto/secret/crypt/badiv
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-crypto-tlscredsx509 -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-crypto-tlscredsx509" 
PASS 27 ahci-test /x86_64/ahci/io/pio/lba48/long/zero
==10918==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==10918==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fff5bfbd000; bottom 0x7f745d57c000; size: 0x008afea41000 (596977651712)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 1 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/perfectserver
PASS 2 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/perfectclient
PASS 28 ahci-test /x86_64/ahci/io/pio/lba48/long/low
PASS 3 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodca1
==10925==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodca2
==10925==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffff830b000; bottom 0x7ff5c23fe000; size: 0x000a35f0d000 (43854647296)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 5 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodca3
---
PASS 9 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodserver1
PASS 29 ahci-test /x86_64/ahci/io/pio/lba48/long/high
PASS 10 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodserver2
==10931==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 11 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodserver3
PASS 30 ahci-test /x86_64/ahci/io/pio/lba48/short/zero
PASS 12 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodserver4
PASS 13 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodserver5
==10937==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 14 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodserver6
PASS 15 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodserver7
PASS 16 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/badserver1
---
PASS 33 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/inactive2
PASS 34 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/inactive3
PASS 31 ahci-test /x86_64/ahci/io/pio/lba48/short/low
==10943==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 32 ahci-test /x86_64/ahci/io/pio/lba48/short/high
==10949==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 35 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/chain1
PASS 36 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/chain2
PASS 37 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/missingca
---
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-crypto-tlssession -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-crypto-tlssession" 
PASS 33 ahci-test /x86_64/ahci/io/dma/lba28/fragmented
PASS 1 test-crypto-tlssession /qcrypto/tlssession/psk
==10960==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 test-crypto-tlssession /qcrypto/tlssession/basicca
PASS 3 test-crypto-tlssession /qcrypto/tlssession/differentca
PASS 4 test-crypto-tlssession /qcrypto/tlssession/altname1
PASS 34 ahci-test /x86_64/ahci/io/dma/lba28/retry
==10966==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 5 test-crypto-tlssession /qcrypto/tlssession/altname2
PASS 6 test-crypto-tlssession /qcrypto/tlssession/altname3
PASS 7 test-crypto-tlssession /qcrypto/tlssession/altname4
PASS 35 ahci-test /x86_64/ahci/io/dma/lba28/simple/zero
==10972==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 8 test-crypto-tlssession /qcrypto/tlssession/altname5
PASS 9 test-crypto-tlssession /qcrypto/tlssession/altname6
PASS 36 ahci-test /x86_64/ahci/io/dma/lba28/simple/low
==10978==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 37 ahci-test /x86_64/ahci/io/dma/lba28/simple/high
==10984==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 10 test-crypto-tlssession /qcrypto/tlssession/wildcard1
PASS 11 test-crypto-tlssession /qcrypto/tlssession/wildcard2
PASS 38 ahci-test /x86_64/ahci/io/dma/lba28/double/zero
PASS 12 test-crypto-tlssession /qcrypto/tlssession/wildcard3
==10990==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 13 test-crypto-tlssession /qcrypto/tlssession/wildcard4
PASS 39 ahci-test /x86_64/ahci/io/dma/lba28/double/low
==10996==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 40 ahci-test /x86_64/ahci/io/dma/lba28/double/high
PASS 14 test-crypto-tlssession /qcrypto/tlssession/wildcard5
==11002==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 15 test-crypto-tlssession /qcrypto/tlssession/wildcard6
PASS 41 ahci-test /x86_64/ahci/io/dma/lba28/long/zero
==11008==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 16 test-crypto-tlssession /qcrypto/tlssession/cachain
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-qga -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-qga" 
PASS 42 ahci-test /x86_64/ahci/io/dma/lba28/long/low
==11020==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-qga /qga/sync-delimited
PASS 2 test-qga /qga/sync
PASS 3 test-qga /qga/ping
---
PASS 17 test-qga /qga/fsfreeze-status
PASS 43 ahci-test /x86_64/ahci/io/dma/lba28/long/high
PASS 18 test-qga /qga/blacklist
==11028==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 19 test-qga /qga/config
PASS 20 test-qga /qga/guest-exec
PASS 21 test-qga /qga/guest-exec-invalid
---
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-timed-average -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-timed-average" 
PASS 1 test-timed-average /timed-average/average
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-util-filemonitor -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-util-filemonitor" 
==11041==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-util-filemonitor /util/filemonitor
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-util-sockets -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-util-sockets" 
PASS 1 test-util-sockets /util/socket/is-socket/bad
---
PASS 4 test-authz-listfile /auth/list/explicit/deny
PASS 5 test-authz-listfile /auth/list/explicit/allow
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-io-task -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-io-task" 
==11075==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-io-task /crypto/task/complete
PASS 2 test-io-task /crypto/task/datafree
PASS 3 test-io-task /crypto/task/failure
---
PASS 5 test-io-channel-file /io/channel/pipe/async
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-io-channel-tls -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-io-channel-tls" 
PASS 46 ahci-test /x86_64/ahci/io/dma/lba28/short/high
==11147==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-io-channel-tls /qio/channel/tls/basic
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-io-channel-command -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-io-channel-command" 
PASS 1 test-io-channel-command /io/channel/command/fifo/sync
---
PASS 17 test-crypto-pbkdf /crypto/pbkdf/nonrfc/sha384/iter1200
PASS 18 test-crypto-pbkdf /crypto/pbkdf/nonrfc/ripemd160/iter1200
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-crypto-ivgen -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-crypto-ivgen" 
==11168==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-crypto-ivgen /crypto/ivgen/plain/1
PASS 2 test-crypto-ivgen /crypto/ivgen/plain/1f2e3d4c
PASS 3 test-crypto-ivgen /crypto/ivgen/plain/1f2e3d4c5b6a7988
---
PASS 1 test-logging /logging/parse_range
PASS 2 test-logging /logging/parse_path
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-replication -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-replication" 
==11209==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11207==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-replication /replication/primary/read
PASS 2 test-replication /replication/primary/write
PASS 49 ahci-test /x86_64/ahci/io/dma/lba48/simple/high
---
PASS 5 test-replication /replication/primary/do_checkpoint
PASS 6 test-replication /replication/primary/get_error_all
PASS 7 test-replication /replication/secondary/read
==11218==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 8 test-replication /replication/secondary/write
PASS 50 ahci-test /x86_64/ahci/io/dma/lba48/double/zero
==11224==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11209==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe308a7000; bottom 0x7f01cd6fc000; size: 0x00fc631ab000 (1083994451968)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 51 ahci-test /x86_64/ahci/io/dma/lba48/double/low
PASS 9 test-replication /replication/secondary/start
==11247==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 52 ahci-test /x86_64/ahci/io/dma/lba48/double/high
==11253==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 10 test-replication /replication/secondary/stop
PASS 53 ahci-test /x86_64/ahci/io/dma/lba48/long/zero
==11259==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 54 ahci-test /x86_64/ahci/io/dma/lba48/long/low
PASS 11 test-replication /replication/secondary/do_checkpoint
==11265==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 12 test-replication /replication/secondary/get_error_all
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-bufferiszero -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-bufferiszero" 
PASS 55 ahci-test /x86_64/ahci/io/dma/lba48/long/high
==11275==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 56 ahci-test /x86_64/ahci/io/dma/lba48/short/zero
==11282==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 57 ahci-test /x86_64/ahci/io/dma/lba48/short/low
==11288==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 58 ahci-test /x86_64/ahci/io/dma/lba48/short/high
==11294==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 59 ahci-test /x86_64/ahci/io/ncq/simple
==11300==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 60 ahci-test /x86_64/ahci/io/ncq/retry
==11306==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 61 ahci-test /x86_64/ahci/flush/simple
==11312==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 62 ahci-test /x86_64/ahci/flush/retry
==11318==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11323==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 63 ahci-test /x86_64/ahci/flush/migrate
==11332==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11337==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 64 ahci-test /x86_64/ahci/migrate/sanity
==11346==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11351==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 65 ahci-test /x86_64/ahci/migrate/dma/simple
==11360==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11365==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 66 ahci-test /x86_64/ahci/migrate/dma/halted
==11374==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11379==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-bufferiszero /cutils/bufferiszero
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  tests/test-uuid -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-uuid" 
PASS 1 test-uuid /uuid/is_null
---
PASS 22 test-qgraph /qgraph/test_test_in_path
PASS 23 test-qgraph /qgraph/test_double_edge
PASS 67 ahci-test /x86_64/ahci/migrate/ncq/simple
==11405==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11410==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 68 ahci-test /x86_64/ahci/migrate/ncq/halted
==11419==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 69 ahci-test /x86_64/ahci/cdrom/eject
==11424==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 70 ahci-test /x86_64/ahci/cdrom/dma/single
==11430==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 71 ahci-test /x86_64/ahci/cdrom/dma/multi
==11436==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 72 ahci-test /x86_64/ahci/cdrom/pio/single
==11442==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==11442==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd30fca000; bottom 0x7fba21ffe000; size: 0x00430efcc000 (288014254080)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 73 ahci-test /x86_64/ahci/cdrom/pio/multi
==11448==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 74 ahci-test /x86_64/ahci/cdrom/pio/bcl
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/hd-geo-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="hd-geo-test" 
PASS 1 hd-geo-test /x86_64/hd-geo/ide/none
==11462==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 hd-geo-test /x86_64/hd-geo/ide/drive/cd_0
==11468==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 hd-geo-test /x86_64/hd-geo/ide/drive/mbr/blank
==11474==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 hd-geo-test /x86_64/hd-geo/ide/drive/mbr/lba
==11480==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 5 hd-geo-test /x86_64/hd-geo/ide/drive/mbr/chs
==11486==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 6 hd-geo-test /x86_64/hd-geo/ide/device/mbr/blank
==11492==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 7 hd-geo-test /x86_64/hd-geo/ide/device/mbr/lba
==11498==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 8 hd-geo-test /x86_64/hd-geo/ide/device/mbr/chs
==11504==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 9 hd-geo-test /x86_64/hd-geo/ide/device/user/chs
==11509==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 10 hd-geo-test /x86_64/hd-geo/ide/device/user/chst
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/boot-order-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="boot-order-test" 
PASS 1 boot-order-test /x86_64/boot-order/pc
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==11577==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 bios-tables-test /x86_64/acpi/piix4
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==11583==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 bios-tables-test /x86_64/acpi/q35
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==11589==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 bios-tables-test /x86_64/acpi/piix4/bridge
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==11595==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 bios-tables-test /x86_64/acpi/piix4/ipmi
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==11601==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 5 bios-tables-test /x86_64/acpi/piix4/cpuhp
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==11608==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 6 bios-tables-test /x86_64/acpi/piix4/memhp
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==11614==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 7 bios-tables-test /x86_64/acpi/piix4/numamem
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==11620==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 8 bios-tables-test /x86_64/acpi/piix4/dimmpxm
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==11629==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 9 bios-tables-test /x86_64/acpi/q35/bridge
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==11635==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 10 bios-tables-test /x86_64/acpi/q35/mmio64
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==11641==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 11 bios-tables-test /x86_64/acpi/q35/ipmi
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==11647==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 12 bios-tables-test /x86_64/acpi/q35/cpuhp
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==11654==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 13 bios-tables-test /x86_64/acpi/q35/memhp
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==11660==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 14 bios-tables-test /x86_64/acpi/q35/numamem
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==11666==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 15 bios-tables-test /x86_64/acpi/q35/dimmpxm
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/boot-serial-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="boot-serial-test" 
PASS 1 boot-serial-test /x86_64/boot-serial/isapc
---
PASS 1 i440fx-test /x86_64/i440fx/defaults
PASS 2 i440fx-test /x86_64/i440fx/pam
PASS 3 i440fx-test /x86_64/i440fx/firmware/bios
==11750==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 i440fx-test /x86_64/i440fx/firmware/pflash
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/fw_cfg-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="fw_cfg-test" 
PASS 1 fw_cfg-test /x86_64/fw_cfg/signature
---
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/drive_del-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="drive_del-test" 
PASS 1 drive_del-test /x86_64/drive_del/without-dev
PASS 2 drive_del-test /x86_64/drive_del/after_failed_device_add
==11838==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 drive_del-test /x86_64/blockdev/drive_del_device_del
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/wdt_ib700-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="wdt_ib700-test" 
PASS 1 wdt_ib700-test /x86_64/wdt_ib700/pause
---
PASS 1 usb-hcd-uhci-test /x86_64/uhci/pci/init
PASS 2 usb-hcd-uhci-test /x86_64/uhci/pci/port1
PASS 3 usb-hcd-uhci-test /x86_64/uhci/pci/hotplug
==12033==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 usb-hcd-uhci-test /x86_64/uhci/pci/hotplug/usb-storage
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/usb-hcd-xhci-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="usb-hcd-xhci-test" 
PASS 1 usb-hcd-xhci-test /x86_64/xhci/pci/init
PASS 2 usb-hcd-xhci-test /x86_64/xhci/pci/hotplug
==12042==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 usb-hcd-xhci-test /x86_64/xhci/pci/hotplug/usb-uas
PASS 4 usb-hcd-xhci-test /x86_64/xhci/pci/hotplug/usb-ccid
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/cpu-plug-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="cpu-plug-test" 
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12148==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 vmgenid-test /x86_64/vmgenid/vmgenid/set-guid
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12154==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 vmgenid-test /x86_64/vmgenid/vmgenid/set-guid-auto
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12160==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 vmgenid-test /x86_64/vmgenid/vmgenid/query-monitor
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/tpm-crb-swtpm-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="tpm-crb-swtpm-test" 
SKIP 1 tpm-crb-swtpm-test /x86_64/tpm/crb-swtpm/test # SKIP swtpm not in PATH or missing --tpm2 support
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12265==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12270==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 migration-test /x86_64/migration/fd_proto
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12278==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12283==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 migration-test /x86_64/migration/postcopy/unix
PASS 5 migration-test /x86_64/migration/postcopy/recovery
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12313==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12318==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 6 migration-test /x86_64/migration/precopy/unix
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12327==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12332==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 7 migration-test /x86_64/migration/precopy/tcp
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12341==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
qemu-system-x86_64: Back to tcg accelerator
==12346==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 8 migration-test /x86_64/migration/xbzrle/unix
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/test-x86-cpuid-compat -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-x86-cpuid-compat" 
PASS 1 test-x86-cpuid-compat /x86/cpuid/parsing-plus-minus
---
PASS 6 numa-test /x86_64/numa/pc/dynamic/cpu
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}  QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/qmp-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="qmp-test" 
PASS 1 qmp-test /x86_64/qmp/protocol
==12675==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 qmp-test /x86_64/qmp/oob
PASS 3 qmp-test /x86_64/qmp/preconfig
PASS 4 qmp-test /x86_64/qmp/missing-any-arg
---
PASS 5 device-introspect-test /x86_64/device/introspect/abstract-interfaces

=================================================================
==12923==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x55d3975aeb6e in calloc (/tmp/qemu-test/build/x86_64-softmmu/qemu-system-x86_64+0x19f9b6e)
---

SUMMARY: AddressSanitizer: 64 byte(s) leaked in 2 allocation(s).
/tmp/qemu-test/src/tests/libqtest.c:137: kill_qemu() tried to terminate QEMU process but encountered exit status 1
ERROR - too few tests run (expected 6, got 5)
make: *** [/tmp/qemu-test/src/tests/Makefile.include:896: check-qtest-x86_64] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):


The full log is available at
http://patchew.org/logs/20190703215542.16123-1-jsnow@redhat.com/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [PATCH v2 00/18] bitmaps: introduce 'bitmap' sync mode

[John Snow]

I have clearly been lax in checking my coding style...

I'll fix these, but not until review.

On 7/3/19 9:50 PM, no-reply@patchew.org wrote:
> Patchew URL: https://patchew.org/QEMU/20190703215542.16123-1-jsnow@redhat.com/
> 
> 
> 
> Hi,
> 
> This series seems to have some coding style problems. See output below for
> more information:
> 
> Type: series
> Subject: [Qemu-devel] [PATCH v2 00/18] bitmaps: introduce 'bitmap' sync mode
> Message-id: 20190703215542.16123-1-jsnow@redhat.com
> 
> === TEST SCRIPT BEGIN ===
> #!/bin/bash
> git rev-parse base > /dev/null || exit 0
> git config --local diff.renamelimit 0
> git config --local diff.renames True
> git config --local diff.algorithm histogram
> ./scripts/checkpatch.pl --mailback base..
> === TEST SCRIPT END ===
> 
> Switched to a new branch 'test'
> 18acfee block/backup: loosen restriction on readonly bitmaps
> 4852213 iotests: add test 257 for bitmap-mode backups
> 0d47743 iotests: Add virtio-scsi device helper
> 9805af4 iotests: teach FilePath to produce multiple paths
> b491804 iotests: teach run_job to cancel pending jobs
> 2925d97 iotests: add testing shim for script-style python tests
> c7870c0 block/backup: add 'always' bitmap sync policy
> 1a5bc8c block/backup: upgrade copy_bitmap to BdrvDirtyBitmap
> a7fd32c block/dirty-bitmap: add bdrv_dirty_bitmap_get
> 6ec8d60 block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal
> 8760b33 hbitmap: enable merging across granularities
> 919477d hbitmap: Fix merge when b is empty, and result is not an alias of a
> dd0775a block/backup: add 'never' policy to bitmap sync mode
> 998d008 block/backup: Add mirror sync mode 'bitmap'
> 5e678f1 qapi: add BitmapSyncMode enum
> 7e58e56 blockdev-backup: utilize do_backup_common
> e372afa drive-backup: create do_backup_common
> 677b04c qapi/block-core: Introduce BackupCommon
> 
> === OUTPUT BEGIN ===
> 1/18 Checking commit 677b04c0b13d (qapi/block-core: Introduce BackupCommon)
> 2/18 Checking commit e372afac0fdc (drive-backup: create do_backup_common)
> 3/18 Checking commit 7e58e56768db (blockdev-backup: utilize do_backup_common)
> 4/18 Checking commit 5e678f15e446 (qapi: add BitmapSyncMode enum)
> 5/18 Checking commit 998d0084971b (block/backup: Add mirror sync mode 'bitmap')
> 6/18 Checking commit dd0775a4cf92 (block/backup: add 'never' policy to bitmap sync mode)
> WARNING: Block comments use a leading /* on a separate line
> #26: FILE: block/backup.c:269:
> +        /* Failure, or we don't want to synchronize the bitmap.
> 
> WARNING: Block comments use a trailing */ on a separate line
> #27: FILE: block/backup.c:270:
> +         * Merge the successor back into the parent, delete nothing. */
> 
> total: 0 errors, 2 warnings, 25 lines checked
> 
> Patch 6/18 has style problems, please review.  If any of these errors
> are false positives report them to the maintainer, see
> CHECKPATCH in MAINTAINERS.
> 7/18 Checking commit 919477d56466 (hbitmap: Fix merge when b is empty, and result is not an alias of a)
> 8/18 Checking commit 8760b336946d (hbitmap: enable merging across granularities)
> 9/18 Checking commit 6ec8d60f0329 (block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal)
> 10/18 Checking commit a7fd32c4817d (block/dirty-bitmap: add bdrv_dirty_bitmap_get)
> ERROR: open brace '{' following function declarations go on the next line
> #36: FILE: block/dirty-bitmap.c:517:
> +bool bdrv_dirty_bitmap_get(BdrvDirtyBitmap *bitmap, int64_t offset) {
> 
> total: 1 errors, 0 warnings, 72 lines checked
> 
> Patch 10/18 has style problems, please review.  If any of these errors
> are false positives report them to the maintainer, see
> CHECKPATCH in MAINTAINERS.
> 
> 11/18 Checking commit 1a5bc8c73750 (block/backup: upgrade copy_bitmap to BdrvDirtyBitmap)
> WARNING: line over 80 characters
> #60: FILE: block/backup.c:174:
> +    bdrv_reset_dirty_bitmap(job->copy_bitmap, start, job->cluster_size * nr_clusters);
> 
> WARNING: line over 80 characters
> #66: FILE: block/backup.c:179:
> +        bdrv_set_dirty_bitmap(job->copy_bitmap, start, job->cluster_size * nr_clusters);
> 
> total: 0 errors, 2 warnings, 192 lines checked
> 
> Patch 11/18 has style problems, please review.  If any of these errors
> are false positives report them to the maintainer, see
> CHECKPATCH in MAINTAINERS.
> 12/18 Checking commit c7870c05906a (block/backup: add 'always' bitmap sync policy)
> WARNING: Block comments use a leading /* on a separate line
> #41: FILE: block/backup.c:273:
> +        /* We succeeded, or we always intended to sync the bitmap.
> 
> WARNING: Block comments use a trailing */ on a separate line
> #42: FILE: block/backup.c:274:
> +         * Delete this bitmap and install the child. */
> 
> WARNING: Block comments use a leading /* on a separate line
> #46: FILE: block/backup.c:277:
> +        /* We failed, or we never intended to sync the bitmap anyway.
> 
> WARNING: Block comments use a trailing */ on a separate line
> #47: FILE: block/backup.c:278:
> +         * Merge the successor back into the parent, keeping all data. */
> 
> total: 0 errors, 4 warnings, 47 lines checked
> 
> Patch 12/18 has style problems, please review.  If any of these errors
> are false positives report them to the maintainer, see
> CHECKPATCH in MAINTAINERS.
> 13/18 Checking commit 2925d97a5717 (iotests: add testing shim for script-style python tests)
> 14/18 Checking commit b491804bec4e (iotests: teach run_job to cancel pending jobs)
> 15/18 Checking commit 9805af48ccbb (iotests: teach FilePath to produce multiple paths)
> WARNING: line over 80 characters
> #63: FILE: tests/qemu-iotests/iotests.py:392:
> +    """FilePath is a specialization of FilePaths that takes a single filename."""
> 
> total: 0 errors, 1 warnings, 67 lines checked
> 
> Patch 15/18 has style problems, please review.  If any of these errors
> are false positives report them to the maintainer, see
> CHECKPATCH in MAINTAINERS.
> 16/18 Checking commit 0d47743c9c1e (iotests: Add virtio-scsi device helper)
> 17/18 Checking commit 485221328b28 (iotests: add test 257 for bitmap-mode backups)
> WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
> #11: 
> new file mode 100755
> 
> total: 0 errors, 1 warnings, 2612 lines checked
> 
> Patch 17/18 has style problems, please review.  If any of these errors
> are false positives report them to the maintainer, see
> CHECKPATCH in MAINTAINERS.
> 18/18 Checking commit 18acfee9986a (block/backup: loosen restriction on readonly bitmaps)
> === OUTPUT END ===
> 
> Test command exited with code: 1
> 
> 
> The full log is available at
> http://patchew.org/logs/20190703215542.16123-1-jsnow@redhat.com/testing.checkpatch/?type=message.
> ---
> Email generated automatically by Patchew [https://patchew.org/].
> Please send your feedback to patchew-devel@redhat.com
> 

-- 
—js

Re: [Qemu-devel] [PATCH v2 01/18] qapi/block-core: Introduce BackupCommon

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 2254 bytes --]

On 03.07.19 23:55, John Snow wrote:
> drive-backup and blockdev-backup have an awful lot of things in common
> that are the same. Let's fix that.
> 
> I don't deduplicate 'target', because the semantics actually did change
> between each structure. Leave that one alone so it can be documented
> separately.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  qapi/block-core.json | 103 ++++++++++++++-----------------------------
>  1 file changed, 33 insertions(+), 70 deletions(-)
> 
> diff --git a/qapi/block-core.json b/qapi/block-core.json
> index 0d43d4f37c..7b23efcf13 100644
> --- a/qapi/block-core.json
> +++ b/qapi/block-core.json
> @@ -1315,32 +1315,23 @@
>    'data': { 'node': 'str', 'overlay': 'str' } }
>  
>  ##
> -# @DriveBackup:
> +# @BackupCommon:
>  #
>  # @job-id: identifier for the newly-created block job. If
>  #          omitted, the device name will be used. (Since 2.7)
>  #
>  # @device: the device name or node-name of a root node which should be copied.
>  #
> -# @target: the target of the new image. If the file exists, or if it
> -#          is a device, the existing file/device will be used as the new
> -#          destination.  If it does not exist, a new file will be created.
> -#
> -# @format: the format of the new destination, default is to
> -#          probe if @mode is 'existing', else the format of the source
> -#
>  # @sync: what parts of the disk image should be copied to the destination
>  #        (all the disk, only the sectors allocated in the topmost image, from a
>  #        dirty bitmap, or only new I/O).
>  #
> -# @mode: whether and how QEMU should create a new image, default is
> -#        'absolute-paths'.
> -#
> -# @speed: the maximum speed, in bytes per second
> +# @speed: the maximum speed, in bytes per second. The default is 0,
> +#         for unlimited.
>  #
>  # @bitmap: the name of dirty bitmap if sync is "incremental".
>  #          Must be present if sync is "incremental", must NOT be present
> -#          otherwise. (Since 2.4)
> +#          otherwise. (Since 2.4 (Drive), 3.1 (Blockdev))

s/Drive/drive-backup/, s/Blockdev/blockdev-backup/?

Other than that:

Reviewed-by: Max Reitz <mreitz@redhat.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 02/18] drive-backup: create do_backup_common

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 1768 bytes --]

On 03.07.19 23:55, John Snow wrote:
> Create a common core that comprises the actual meat of what the backup API
> boundary needs to do, and then switch drive-backup to use it.
> 
> Questions:
>  - do_drive_backup now acquires and releases the aio_context in addition
>    to do_backup_common doing the same. Can I drop this from drive_backup?

I wonder why you don’t just make it a requirement that
do_backup_common() is called with the context acquired?

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  blockdev.c | 138 ++++++++++++++++++++++++++++++++---------------------
>  1 file changed, 83 insertions(+), 55 deletions(-)
> 
> diff --git a/blockdev.c b/blockdev.c
> index 4d141e9a1f..5fd663a7e5 100644
> --- a/blockdev.c
> +++ b/blockdev.c
> @@ -3425,6 +3425,86 @@ out:
>      aio_context_release(aio_context);
>  }
>  
> +static BlockJob *do_backup_common(BackupCommon *backup,
> +                                  BlockDriverState *target_bs,
> +                                  JobTxn *txn, Error **errp)
> +{

[...]

> +    job = backup_job_create(backup->job_id, bs, target_bs, backup->speed,
> +                            backup->sync, bmap, backup->compress,
> +                            backup->on_source_error, backup->on_target_error,
> +                            job_flags, NULL, NULL, txn, &local_err);
> +    if (local_err != NULL) {
> +        error_propagate(errp, local_err);
> +        goto out;
> +    }

Below, you change do_drive_backup() to just pass errp instead of
local_err and not do error handling.  Why not do the same here?

Other than that:

Reviewed-by: Max Reitz <mreitz@redhat.com>

> +
> +out:
> +    aio_context_release(aio_context);
> +    return job;
> +}


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 00/18] bitmaps: introduce 'bitmap' sync mode

[no-reply]

Patchew URL: https://patchew.org/QEMU/20190703215542.16123-1-jsnow@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [Qemu-devel] [PATCH v2 00/18] bitmaps: introduce 'bitmap' sync mode
Type: series
Message-id: 20190703215542.16123-1-jsnow@redhat.com

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

From https://github.com/patchew-project/qemu
 - [tag update]      patchew/20190703215542.16123-1-jsnow@redhat.com -> patchew/20190703215542.16123-1-jsnow@redhat.com
Switched to a new branch 'test'
26e6479 block/backup: loosen restriction on readonly bitmaps
755e1b1 iotests: add test 257 for bitmap-mode backups
d48c9ee iotests: Add virtio-scsi device helper
b6e4226 iotests: teach FilePath to produce multiple paths
9b7ddb3 iotests: teach run_job to cancel pending jobs
78ed529 iotests: add testing shim for script-style python tests
78a4fa3 block/backup: add 'always' bitmap sync policy
edcb2f9 block/backup: upgrade copy_bitmap to BdrvDirtyBitmap
5691b3e block/dirty-bitmap: add bdrv_dirty_bitmap_get
3decdf3 block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal
1ee3ffe hbitmap: enable merging across granularities
0d187e6 hbitmap: Fix merge when b is empty, and result is not an alias of a
9fb7b80 block/backup: add 'never' policy to bitmap sync mode
3025abb block/backup: Add mirror sync mode 'bitmap'
78173da qapi: add BitmapSyncMode enum
8d6effc blockdev-backup: utilize do_backup_common
189a557 drive-backup: create do_backup_common
1aa2202 qapi/block-core: Introduce BackupCommon

=== OUTPUT BEGIN ===
1/18 Checking commit 1aa2202a91cb (qapi/block-core: Introduce BackupCommon)
2/18 Checking commit 189a557d32e9 (drive-backup: create do_backup_common)
3/18 Checking commit 8d6effca544a (blockdev-backup: utilize do_backup_common)
4/18 Checking commit 78173daac67b (qapi: add BitmapSyncMode enum)
5/18 Checking commit 3025abb79847 (block/backup: Add mirror sync mode 'bitmap')
6/18 Checking commit 9fb7b804caa8 (block/backup: add 'never' policy to bitmap sync mode)
WARNING: Block comments use a leading /* on a separate line
#26: FILE: block/backup.c:269:
+        /* Failure, or we don't want to synchronize the bitmap.

WARNING: Block comments use a trailing */ on a separate line
#27: FILE: block/backup.c:270:
+         * Merge the successor back into the parent, delete nothing. */

total: 0 errors, 2 warnings, 25 lines checked

Patch 6/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
7/18 Checking commit 0d187e69012e (hbitmap: Fix merge when b is empty, and result is not an alias of a)
8/18 Checking commit 1ee3ffe706d4 (hbitmap: enable merging across granularities)
9/18 Checking commit 3decdf358bb7 (block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal)
10/18 Checking commit 5691b3e06e73 (block/dirty-bitmap: add bdrv_dirty_bitmap_get)
ERROR: open brace '{' following function declarations go on the next line
#36: FILE: block/dirty-bitmap.c:517:
+bool bdrv_dirty_bitmap_get(BdrvDirtyBitmap *bitmap, int64_t offset) {

total: 1 errors, 0 warnings, 72 lines checked

Patch 10/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

11/18 Checking commit edcb2f9d526e (block/backup: upgrade copy_bitmap to BdrvDirtyBitmap)
WARNING: line over 80 characters
#60: FILE: block/backup.c:174:
+    bdrv_reset_dirty_bitmap(job->copy_bitmap, start, job->cluster_size * nr_clusters);

WARNING: line over 80 characters
#66: FILE: block/backup.c:179:
+        bdrv_set_dirty_bitmap(job->copy_bitmap, start, job->cluster_size * nr_clusters);

total: 0 errors, 2 warnings, 192 lines checked

Patch 11/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
12/18 Checking commit 78a4fa304c33 (block/backup: add 'always' bitmap sync policy)
WARNING: Block comments use a leading /* on a separate line
#41: FILE: block/backup.c:273:
+        /* We succeeded, or we always intended to sync the bitmap.

WARNING: Block comments use a trailing */ on a separate line
#42: FILE: block/backup.c:274:
+         * Delete this bitmap and install the child. */

WARNING: Block comments use a leading /* on a separate line
#46: FILE: block/backup.c:277:
+        /* We failed, or we never intended to sync the bitmap anyway.

WARNING: Block comments use a trailing */ on a separate line
#47: FILE: block/backup.c:278:
+         * Merge the successor back into the parent, keeping all data. */

total: 0 errors, 4 warnings, 47 lines checked

Patch 12/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
13/18 Checking commit 78ed5297c739 (iotests: add testing shim for script-style python tests)
14/18 Checking commit 9b7ddb390360 (iotests: teach run_job to cancel pending jobs)
15/18 Checking commit b6e42267d4bc (iotests: teach FilePath to produce multiple paths)
WARNING: line over 80 characters
#63: FILE: tests/qemu-iotests/iotests.py:392:
+    """FilePath is a specialization of FilePaths that takes a single filename."""

total: 0 errors, 1 warnings, 67 lines checked

Patch 15/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
16/18 Checking commit d48c9ee0bec9 (iotests: Add virtio-scsi device helper)
17/18 Checking commit 755e1b11478c (iotests: add test 257 for bitmap-mode backups)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#11: 
new file mode 100755

total: 0 errors, 1 warnings, 2612 lines checked

Patch 17/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
18/18 Checking commit 26e64793fc71 (block/backup: loosen restriction on readonly bitmaps)
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20190703215542.16123-1-jsnow@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [PATCH v2 03/18] blockdev-backup: utilize do_backup_common

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 267 bytes --]

On 03.07.19 23:55, John Snow wrote:
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  blockdev.c | 73 +++---------------------------------------------------
>  1 file changed, 3 insertions(+), 70 deletions(-)

Reviewed-by: Max Reitz <mreitz@redhat.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 04/18] qapi: add BitmapSyncMode enum

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 731 bytes --]

On 03.07.19 23:55, John Snow wrote:
> Depending on what a user is trying to accomplish, there might be a few
> bitmap cleanup actions that occur when an operation is finished that
> could be useful.
> 
> I am proposing three:
> - NEVER: The bitmap is never synchronized against what was copied.
> - ALWAYS: The bitmap is always synchronized, even on failures.
> - CONDITIONAL: The bitmap is synchronized only on success.
> 
> The existing incremental backup modes use 'conditional' semantics,
> so add just that one for right now.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  qapi/block-core.json | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)

Reviewed-by: Max Reitz <mreitz@redhat.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 05/18] block/backup: Add mirror sync mode 'bitmap'

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 2066 bytes --]

On 03.07.19 23:55, John Snow wrote:
> We don't need or want a new sync mode for simple differences in
> semantics.  Create a new mode simply named "BITMAP" that is designed to
> make use of the new Bitmap Sync Mode field.
> 
> Because the only bitmap mode is 'conditional', this adds no new
> functionality to the backup job (yet). The old incremental backup mode
> is maintained as a syntactic sugar for sync=bitmap, mode=conditional.
> 
> Add all of the plumbing necessary to support this new instruction.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  block/backup.c            | 20 ++++++++++++--------
>  block/mirror.c            |  6 ++++--
>  block/replication.c       |  2 +-
>  blockdev.c                | 19 +++++++++++++++++--
>  include/block/block_int.h |  4 +++-
>  qapi/block-core.json      | 20 ++++++++++++++------
>  6 files changed, 51 insertions(+), 20 deletions(-)

[...]

> diff --git a/blockdev.c b/blockdev.c
> index cad300d526..d5b089a446 100644
> --- a/blockdev.c
> +++ b/blockdev.c
> @@ -3473,6 +3473,19 @@ static BlockJob *do_backup_common(BackupCommon *backup,
>          goto out;
>      }
>  
> +    if (backup->sync == MIRROR_SYNC_MODE_INCREMENTAL) {
> +        if (backup->has_bitmap_mode &&
> +            backup->bitmap_mode != BITMAP_SYNC_MODE_CONDITIONAL) {
> +            error_setg(errp, "Bitmap sync mode must be 'conditional' "
> +                       "when using sync mode '%s'",
> +                       MirrorSyncMode_str(backup->sync));
> +            goto out;
> +        }
> +        backup->has_bitmap_mode = true;
> +        backup->sync = MIRROR_SYNC_MODE_BITMAP;
> +        backup->bitmap_mode = BITMAP_SYNC_MODE_CONDITIONAL;
> +    }
> +

A check for

  backup->sync == MIRROR_SYNC_MODE_BITMAP && !backup->has_bitmap_mode

or maybe even better

  backup->has_bitmap && !backup->has_bitmap_mode

is missing here.

Max

>      if (backup->has_bitmap) {
>          bmap = bdrv_find_dirty_bitmap(bs, backup->bitmap);
>          if (!bmap) {


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 06/18] block/backup: add 'never' policy to bitmap sync mode

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 502 bytes --]

On 03.07.19 23:55, John Snow wrote:
> This adds a "never" policy for bitmap synchronization. Regardless of if
> the job succeeds or fails, we never update the bitmap. This can be used
> to perform differential backups, or simply to avoid the job modifying a
> bitmap.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  block/backup.c       | 5 +++--
>  qapi/block-core.json | 5 ++++-
>  2 files changed, 7 insertions(+), 3 deletions(-)

Reviewed-by: Max Reitz <mreitz@redhat.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 07/18] hbitmap: Fix merge when b is empty, and result is not an alias of a

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 384 bytes --]

On 03.07.19 23:55, John Snow wrote:
> Nobody calls the function like this currently, but we neither prohibit
> or cope with this behavior. I decided to make the function cope with it.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  util/hbitmap.c | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)

Reviewed-by: Max Reitz <mreitz@redhat.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 08/18] hbitmap: enable merging across granularities

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 252 bytes --]

On 03.07.19 23:55, John Snow wrote:
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  util/hbitmap.c | 36 +++++++++++++++++++++++++++++++++++-
>  1 file changed, 35 insertions(+), 1 deletion(-)

Reviewed-by: Max Reitz <mreitz@redhat.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 09/18] block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 3059 bytes --]

On 03.07.19 23:55, John Snow wrote:
> I'm surprised it didn't come up sooner, but sometimes we have a +busy
> bitmap as a source. This is dangerous from the QMP API, but if we are
> the owner that marked the bitmap busy, it's safe to merge it using it as
> a read only source.
> 
> It is not safe in the general case to allow users to read from in-use
> bitmaps, so create an internal variant that foregoes the safety
> checking.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  block/dirty-bitmap.c      | 51 +++++++++++++++++++++++++++++++++------
>  include/block/block_int.h |  3 +++
>  2 files changed, 47 insertions(+), 7 deletions(-)
> 
> diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
> index 95a9c2a5d8..b0f76826b3 100644
> --- a/block/dirty-bitmap.c
> +++ b/block/dirty-bitmap.c
> @@ -810,11 +810,15 @@ bool bdrv_dirty_bitmap_next_dirty_area(BdrvDirtyBitmap *bitmap,
>      return hbitmap_next_dirty_area(bitmap->bitmap, offset, bytes);
>  }
>  
> +/**
> + * bdrv_merge_dirty_bitmap: merge src into dest.
> + * Ensures permissions on bitmaps are reasonable; use for public API.
> + *
> + * @backup: If provided, make a copy of dest here prior to merge.
> + */
>  void bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
>                               HBitmap **backup, Error **errp)
>  {
> -    bool ret;
> -
>      qemu_mutex_lock(dest->mutex);
>      if (src->mutex != dest->mutex) {
>          qemu_mutex_lock(src->mutex);
> @@ -833,6 +837,37 @@ void bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
>          goto out;
>      }
>  
> +    assert(bdrv_dirty_bitmap_merge_internal(dest, src, backup, false));

Please keep the explicit @ret.  We never define NDEBUG, but doing things
with side effects inside of assert() is bad style nonetheless.

> +
> +out:
> +    qemu_mutex_unlock(dest->mutex);
> +    if (src->mutex != dest->mutex) {
> +        qemu_mutex_unlock(src->mutex);
> +    }
> +}
> +
> +/**
> + * bdrv_dirty_bitmap_merge_internal: merge src into dest.
> + * Does NOT check bitmap permissions; not suitable for use as public API.
> + *
> + * @backup: If provided, make a copy of dest here prior to merge.
> + * @lock: If true, lock and unlock bitmaps on the way in/out.
> + * returns true if the merge succeeded; false if unattempted.
> + */
> +bool bdrv_dirty_bitmap_merge_internal(BdrvDirtyBitmap *dest,
> +                                      const BdrvDirtyBitmap *src,
> +                                      HBitmap **backup,
> +                                      bool lock)
> +{
> +    bool ret;
> +
> +    if (lock) {
> +        qemu_mutex_lock(dest->mutex);
> +        if (src->mutex != dest->mutex) {
> +            qemu_mutex_lock(src->mutex);
> +        }
> +    }
> +

Why not check for INCONSISTENT and RO still?

Max

>      if (backup) {
>          *backup = dest->bitmap;
>          dest->bitmap = hbitmap_alloc(dest->size, hbitmap_granularity(*backup));


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 10/18] block/dirty-bitmap: add bdrv_dirty_bitmap_get

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 1789 bytes --]

On 03.07.19 23:55, John Snow wrote:
> Add a public interface for get. While we're at it,
> rename "bdrv_get_dirty_bitmap_locked" to "bdrv_dirty_bitmap_get_locked".
> 
> (There are more functions to rename to the bdrv_dirty_bitmap_VERB form,
> but they will wait until the conclusion of this series.)
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  block/dirty-bitmap.c         | 18 +++++++++++-------
>  block/mirror.c               |  2 +-
>  include/block/dirty-bitmap.h |  4 ++--
>  migration/block.c            |  5 ++---
>  nbd/server.c                 |  2 +-
>  5 files changed, 17 insertions(+), 14 deletions(-)
> 
> diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
> index b0f76826b3..97541521ab 100644
> --- a/block/dirty-bitmap.c
> +++ b/block/dirty-bitmap.c
> @@ -509,14 +509,18 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
>  }
>  
>  /* Called within bdrv_dirty_bitmap_lock..unlock */
> -bool bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
> -                           int64_t offset)
> +bool bdrv_dirty_bitmap_get_locked(BdrvDirtyBitmap *bitmap, int64_t offset)
>  {
> -    if (bitmap) {
> -        return hbitmap_get(bitmap->bitmap, offset);
> -    } else {
> -        return false;
> -    }
> +    return hbitmap_get(bitmap->bitmap, offset);
> +}
> +
> +bool bdrv_dirty_bitmap_get(BdrvDirtyBitmap *bitmap, int64_t offset) {

I’m sure Patchew has told this already, but this is not Rust yet.

With that fixed:

Reviewed-by: Max Reitz <mreitz@redhat.com>

> +    bool ret;
> +    bdrv_dirty_bitmap_lock(bitmap);
> +    ret = bdrv_dirty_bitmap_get_locked(bitmap, offset);
> +    bdrv_dirty_bitmap_unlock(bitmap);
> +
> +    return ret;
>  }
>  
>  /**


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 11/18] block/backup: upgrade copy_bitmap to BdrvDirtyBitmap

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 4908 bytes --]

On 03.07.19 23:55, John Snow wrote:
> This simplifies some interface matters; namely the initialization and
> (later) merging the manifest back into the sync_bitmap if it was
> provided.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  block/backup.c | 76 ++++++++++++++++++++++++--------------------------
>  1 file changed, 37 insertions(+), 39 deletions(-)
> 
> diff --git a/block/backup.c b/block/backup.c
> index d7fdafebda..9cc5a7f6ca 100644
> --- a/block/backup.c
> +++ b/block/backup.c

[...]

> @@ -202,7 +204,7 @@ static int coroutine_fn backup_do_cow(BackupBlockJob *job,
>      cow_request_begin(&cow_request, job, start, end);
>  
>      while (start < end) {
> -        if (!hbitmap_get(job->copy_bitmap, start)) {
> +        if (!bdrv_dirty_bitmap_get(job->copy_bitmap, start)) {
>              trace_backup_do_cow_skip(job, start);
>              start += job->cluster_size;
>              continue; /* already copied */
> @@ -296,14 +298,16 @@ static void backup_abort(Job *job)
>  static void backup_clean(Job *job)
>  {
>      BackupBlockJob *s = container_of(job, BackupBlockJob, common.job);
> +    BlockDriverState *bs = blk_bs(s->target);

I’d prefer to call it target_bs, because “bs” is usually the source node.

Which brings me to a question: Why is the copy bitmap assigned to the
target in the first place?  Wouldn’t it make more sense on the source?

> +
> +    if (s->copy_bitmap) {
> +        bdrv_release_dirty_bitmap(bs, s->copy_bitmap);
> +        s->copy_bitmap = NULL;
> +    }
> +
>      assert(s->target);
>      blk_unref(s->target);
>      s->target = NULL;
> -
> -    if (s->copy_bitmap) {
> -        hbitmap_free(s->copy_bitmap);
> -        s->copy_bitmap = NULL;
> -    }
>  }
>  
>  void backup_do_checkpoint(BlockJob *job, Error **errp)

[...]

> @@ -387,59 +391,49 @@ static bool bdrv_is_unallocated_range(BlockDriverState *bs,
>  
>  static int coroutine_fn backup_loop(BackupBlockJob *job)
>  {
> -    int ret;
>      bool error_is_read;
>      int64_t offset;
> -    HBitmapIter hbi;
> +    BdrvDirtyBitmapIter *bdbi;
>      BlockDriverState *bs = blk_bs(job->common.blk);
> +    int ret = 0;
>  
> -    hbitmap_iter_init(&hbi, job->copy_bitmap, 0);
> -    while ((offset = hbitmap_iter_next(&hbi)) != -1) {
> +    bdbi = bdrv_dirty_iter_new(job->copy_bitmap);
> +    while ((offset = bdrv_dirty_iter_next(bdbi)) != -1) {
>          if (job->sync_mode == MIRROR_SYNC_MODE_TOP &&
>              bdrv_is_unallocated_range(bs, offset, job->cluster_size))
>          {
> -            hbitmap_reset(job->copy_bitmap, offset, job->cluster_size);
> +            bdrv_set_dirty_bitmap(job->copy_bitmap, offset, job->cluster_size);

Should be reset.

>              continue;
>          }
>  
>          do {
>              if (yield_and_check(job)) {
> -                return 0;
> +                goto out;
>              }
>              ret = backup_do_cow(job, offset,
>                                  job->cluster_size, &error_is_read, false);
>              if (ret < 0 && backup_error_action(job, error_is_read, -ret) ==
>                             BLOCK_ERROR_ACTION_REPORT)
>              {
> -                return ret;
> +                goto out;
>              }
>          } while (ret < 0);
>      }
>  
> -    return 0;
> + out:
> +    bdrv_dirty_iter_free(bdbi);
> +    return ret;
>  }
>  
>  /* init copy_bitmap from sync_bitmap */
>  static void backup_incremental_init_copy_bitmap(BackupBlockJob *job)
>  {
> -    uint64_t offset = 0;
> -    uint64_t bytes = job->len;
> -
> -    while (bdrv_dirty_bitmap_next_dirty_area(job->sync_bitmap,
> -                                             &offset, &bytes))
> -    {
> -        hbitmap_set(job->copy_bitmap, offset, bytes);
> -
> -        offset += bytes;
> -        if (offset >= job->len) {
> -            break;
> -        }
> -        bytes = job->len - offset;
> -    }
> +    bdrv_dirty_bitmap_merge_internal(job->copy_bitmap, job->sync_bitmap,
> +                                     NULL, true);

How about asserting that this was successful?

>  
>      /* TODO job_progress_set_remaining() would make more sense */
>      job_progress_update(&job->common.job,
> -        job->len - hbitmap_count(job->copy_bitmap));
> +                        job->len - bdrv_get_dirty_count(job->copy_bitmap));
>  }
>  
>  static int coroutine_fn backup_run(Job *job, Error **errp)

[...]

> @@ -670,7 +668,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
>   error:
>      if (copy_bitmap) {
>          assert(!job || !job->copy_bitmap);
> -        hbitmap_free(copy_bitmap);
> +        bdrv_release_dirty_bitmap(bs, copy_bitmap);

If you decide to keep the copy_bitmap on the target, s/bs/target/.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 12/18] block/backup: add 'always' bitmap sync policy

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 2810 bytes --]

On 03.07.19 23:55, John Snow wrote:
> This adds an "always" policy for bitmap synchronization. Regardless of if
> the job succeeds or fails, the bitmap is *always* synchronized. This means
> that for backups that fail part-way through, the bitmap retains a record of
> which sectors need to be copied out to accomplish a new backup using the
> old, partial result.
> 
> In effect, this allows us to "resume" a failed backup; however the new backup
> will be from the new point in time, so it isn't a "resume" as much as it is
> an "incremental retry." This can be useful in the case of extremely large
> backups that fail considerably through the operation and we'd like to not waste
> the work that was already performed.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  block/backup.c       | 25 +++++++++++++++++--------
>  qapi/block-core.json |  5 ++++-
>  2 files changed, 21 insertions(+), 9 deletions(-)
> 
> diff --git a/block/backup.c b/block/backup.c
> index 9cc5a7f6ca..495d8f71aa 100644
> --- a/block/backup.c
> +++ b/block/backup.c
> @@ -266,16 +266,25 @@ static void backup_cleanup_sync_bitmap(BackupBlockJob *job, int ret)
>  {
>      BdrvDirtyBitmap *bm;
>      BlockDriverState *bs = blk_bs(job->common.blk);
> +    bool sync = (((ret == 0) || (job->bitmap_mode == BITMAP_SYNC_MODE_ALWAYS)) \
> +                 && (job->bitmap_mode != BITMAP_SYNC_MODE_NEVER));
>  
> -    if (ret < 0 || job->bitmap_mode == BITMAP_SYNC_MODE_NEVER) {
> -        /* Failure, or we don't want to synchronize the bitmap.
> -         * Merge the successor back into the parent, delete nothing. */
> -        bm = bdrv_reclaim_dirty_bitmap(bs, job->sync_bitmap, NULL);
> -        assert(bm);
> -    } else {
> -        /* Everything is fine, delete this bitmap and install the backup. */
> +    if (sync) {
> +        /* We succeeded, or we always intended to sync the bitmap.
> +         * Delete this bitmap and install the child. */
>          bm = bdrv_dirty_bitmap_abdicate(bs, job->sync_bitmap, NULL);
> -        assert(bm);
> +    } else {
> +        /* We failed, or we never intended to sync the bitmap anyway.
> +         * Merge the successor back into the parent, keeping all data. */
> +        bm = bdrv_reclaim_dirty_bitmap(bs, job->sync_bitmap, NULL);
> +    }
> +
> +    assert(bm);
> +
> +    if (ret < 0 && job->bitmap_mode == BITMAP_SYNC_MODE_ALWAYS) {

“ret < 0 && sync” would be simpler – your choice.

> +        /* If we failed and synced, merge in the bits we didn't copy: */
> +        bdrv_dirty_bitmap_merge_internal(bm, job->copy_bitmap,
> +                                         NULL, true);

I presume this is for sync=full?

If so:

Reviewed-by: Max Reitz <mreitz@redhat.com>

>      }
>  }
>  


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 13/18] iotests: add testing shim for script-style python tests

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 810 bytes --]

On 03.07.19 23:55, John Snow wrote:
> Because the new-style python tests don't use the iotests.main() test
> launcher, we don't turn on the debugger logging for these scripts
> when invoked via ./check -d.
> 
> Refactor the launcher shim into new and old style shims so that they
> share environmental configuration.
> 
> Two cleanup notes: debug was not actually used as a global, and there
> was no reason to create a class in an inner scope just to achieve
> default variables; we can simply create an instance of the runner with
> the values we want instead.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  tests/qemu-iotests/iotests.py | 40 +++++++++++++++++++++++------------
>  1 file changed, 26 insertions(+), 14 deletions(-)

Reviewed-by: Max Reitz <mreitz@redhat.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 14/18] iotests: teach run_job to cancel pending jobs

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 449 bytes --]

On 03.07.19 23:55, John Snow wrote:
> run_job can cancel pending jobs to simulate failure. This lets us use
> the pending callback to issue test commands while the job is open, but
> then still have the job fail in the end.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  tests/qemu-iotests/iotests.py | 22 ++++++++++++++++++++--
>  1 file changed, 20 insertions(+), 2 deletions(-)

Reviewed-by: Max Reitz <mreitz@redhat.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 15/18] iotests: teach FilePath to produce multiple paths

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 912 bytes --]

On 03.07.19 23:55, John Snow wrote:
> Use "FilePaths" instead of "FilePath" to request multiple files be
> cleaned up after we leave that object's scope.
> 
> This is not crucial; but it saves a little typing.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  tests/qemu-iotests/iotests.py | 33 ++++++++++++++++++++++-----------
>  1 file changed, 22 insertions(+), 11 deletions(-)
> 
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index c544659ecb..db00fd2adf 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py

[...]

> @@ -391,7 +403,6 @@ def file_path_remover():
>          except OSError:
>              pass
>  
> -
>  def file_path(*names):
>      ''' Another way to get auto-generated filename that cleans itself up.
>  

Not sure about this hunk, but OK.

Reviewed-by: Max Reitz <mreitz@redhat.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 16/18] iotests: Add virtio-scsi device helper

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 455 bytes --]

On 03.07.19 23:55, John Snow wrote:
> Seems that it comes up enough.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  tests/qemu-iotests/040        | 6 +-----
>  tests/qemu-iotests/093        | 6 ++----
>  tests/qemu-iotests/139        | 7 ++-----
>  tests/qemu-iotests/238        | 5 +----
>  tests/qemu-iotests/iotests.py | 4 ++++
>  5 files changed, 10 insertions(+), 18 deletions(-)

Reviewed-by: Max Reitz <mreitz@redhat.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 17/18] iotests: add test 257 for bitmap-mode backups

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 436 bytes --]

On 03.07.19 23:55, John Snow wrote:
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  tests/qemu-iotests/257     |  409 +++++++
>  tests/qemu-iotests/257.out | 2199 ++++++++++++++++++++++++++++++++++++
>  tests/qemu-iotests/group   |    1 +
>  3 files changed, 2609 insertions(+)
>  create mode 100755 tests/qemu-iotests/257
>  create mode 100644 tests/qemu-iotests/257.out

Reviewed-by: Max Reitz <mreitz@redhat.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 18/18] block/backup: loosen restriction on readonly bitmaps

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 454 bytes --]

On 03.07.19 23:55, John Snow wrote:
> With the "never" sync policy, we actually can utilize readonly bitmaps
> now. Loosen the check at the QMP level, and tighten it based on
> provided arguments down at the job creation level instead.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  block/backup.c | 6 ++++++
>  blockdev.c     | 2 +-
>  2 files changed, 7 insertions(+), 1 deletion(-)

Reviewed-by: Max Reitz <mreitz@redhat.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 12/18] block/backup: add 'always' bitmap sync policy

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 1671 bytes --]

On 04.07.19 19:43, Max Reitz wrote:
> On 03.07.19 23:55, John Snow wrote:
>> This adds an "always" policy for bitmap synchronization. Regardless of if
>> the job succeeds or fails, the bitmap is *always* synchronized. This means
>> that for backups that fail part-way through, the bitmap retains a record of
>> which sectors need to be copied out to accomplish a new backup using the
>> old, partial result.
>>
>> In effect, this allows us to "resume" a failed backup; however the new backup
>> will be from the new point in time, so it isn't a "resume" as much as it is
>> an "incremental retry." This can be useful in the case of extremely large
>> backups that fail considerably through the operation and we'd like to not waste
>> the work that was already performed.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>  block/backup.c       | 25 +++++++++++++++++--------
>>  qapi/block-core.json |  5 ++++-
>>  2 files changed, 21 insertions(+), 9 deletions(-)
>>
>> diff --git a/block/backup.c b/block/backup.c
>> index 9cc5a7f6ca..495d8f71aa 100644
>> --- a/block/backup.c
>> +++ b/block/backup.c
>> @@ -266,16 +266,25 @@ static void backup_cleanup_sync_bitmap(BackupBlockJob *job, int ret)

[...]

>> +        /* If we failed and synced, merge in the bits we didn't copy: */
>> +        bdrv_dirty_bitmap_merge_internal(bm, job->copy_bitmap,
>> +                                         NULL, true);
> 
> I presume this is for sync=full?

Ah, no.  This is necessary because the original sync bitmap was
discarded by bdrv_dirty_bitmap_abdicate().  So yep, these bits need to
go back into the sync bitmap then.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 00/18] bitmaps: introduce 'bitmap' sync mode

[no-reply]

Patchew URL: https://patchew.org/QEMU/20190703215542.16123-1-jsnow@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [Qemu-devel] [PATCH v2 00/18] bitmaps: introduce 'bitmap' sync mode
Type: series
Message-id: 20190703215542.16123-1-jsnow@redhat.com

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Switched to a new branch 'test'
29d2e89 block/backup: loosen restriction on readonly bitmaps
3dac5e8 iotests: add test 257 for bitmap-mode backups
9b334f2 iotests: Add virtio-scsi device helper
76de918 iotests: teach FilePath to produce multiple paths
25e47cf iotests: teach run_job to cancel pending jobs
246979a iotests: add testing shim for script-style python tests
da9f5b2 block/backup: add 'always' bitmap sync policy
a0da4a4 block/backup: upgrade copy_bitmap to BdrvDirtyBitmap
c38beb0 block/dirty-bitmap: add bdrv_dirty_bitmap_get
e1f9101 block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal
8dcbb6e hbitmap: enable merging across granularities
f0a01f5 hbitmap: Fix merge when b is empty, and result is not an alias of a
cfddaac block/backup: add 'never' policy to bitmap sync mode
3d87c7e block/backup: Add mirror sync mode 'bitmap'
4bfeaea qapi: add BitmapSyncMode enum
aa153cb blockdev-backup: utilize do_backup_common
9b2ea6d drive-backup: create do_backup_common
f1dfe57 qapi/block-core: Introduce BackupCommon

=== OUTPUT BEGIN ===
1/18 Checking commit f1dfe5731cdd (qapi/block-core: Introduce BackupCommon)
2/18 Checking commit 9b2ea6d12ad1 (drive-backup: create do_backup_common)
3/18 Checking commit aa153cbc1d7b (blockdev-backup: utilize do_backup_common)
4/18 Checking commit 4bfeaeaea5c0 (qapi: add BitmapSyncMode enum)
5/18 Checking commit 3d87c7e95994 (block/backup: Add mirror sync mode 'bitmap')
6/18 Checking commit cfddaacb73e3 (block/backup: add 'never' policy to bitmap sync mode)
WARNING: Block comments use a leading /* on a separate line
#26: FILE: block/backup.c:269:
+        /* Failure, or we don't want to synchronize the bitmap.

WARNING: Block comments use a trailing */ on a separate line
#27: FILE: block/backup.c:270:
+         * Merge the successor back into the parent, delete nothing. */

total: 0 errors, 2 warnings, 25 lines checked

Patch 6/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
7/18 Checking commit f0a01f558ce5 (hbitmap: Fix merge when b is empty, and result is not an alias of a)
8/18 Checking commit 8dcbb6e9f205 (hbitmap: enable merging across granularities)
9/18 Checking commit e1f9101828dc (block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal)
10/18 Checking commit c38beb034ad8 (block/dirty-bitmap: add bdrv_dirty_bitmap_get)
ERROR: open brace '{' following function declarations go on the next line
#36: FILE: block/dirty-bitmap.c:517:
+bool bdrv_dirty_bitmap_get(BdrvDirtyBitmap *bitmap, int64_t offset) {

total: 1 errors, 0 warnings, 72 lines checked

Patch 10/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

11/18 Checking commit a0da4a40d744 (block/backup: upgrade copy_bitmap to BdrvDirtyBitmap)
WARNING: line over 80 characters
#60: FILE: block/backup.c:174:
+    bdrv_reset_dirty_bitmap(job->copy_bitmap, start, job->cluster_size * nr_clusters);

WARNING: line over 80 characters
#66: FILE: block/backup.c:179:
+        bdrv_set_dirty_bitmap(job->copy_bitmap, start, job->cluster_size * nr_clusters);

total: 0 errors, 2 warnings, 192 lines checked

Patch 11/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
12/18 Checking commit da9f5b248832 (block/backup: add 'always' bitmap sync policy)
WARNING: Block comments use a leading /* on a separate line
#41: FILE: block/backup.c:273:
+        /* We succeeded, or we always intended to sync the bitmap.

WARNING: Block comments use a trailing */ on a separate line
#42: FILE: block/backup.c:274:
+         * Delete this bitmap and install the child. */

WARNING: Block comments use a leading /* on a separate line
#46: FILE: block/backup.c:277:
+        /* We failed, or we never intended to sync the bitmap anyway.

WARNING: Block comments use a trailing */ on a separate line
#47: FILE: block/backup.c:278:
+         * Merge the successor back into the parent, keeping all data. */

total: 0 errors, 4 warnings, 47 lines checked

Patch 12/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
13/18 Checking commit 246979a623e7 (iotests: add testing shim for script-style python tests)
14/18 Checking commit 25e47cf37f92 (iotests: teach run_job to cancel pending jobs)
15/18 Checking commit 76de9180c629 (iotests: teach FilePath to produce multiple paths)
WARNING: line over 80 characters
#63: FILE: tests/qemu-iotests/iotests.py:392:
+    """FilePath is a specialization of FilePaths that takes a single filename."""

total: 0 errors, 1 warnings, 67 lines checked

Patch 15/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
16/18 Checking commit 9b334f2cf12b (iotests: Add virtio-scsi device helper)
17/18 Checking commit 3dac5e8f8422 (iotests: add test 257 for bitmap-mode backups)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#11: 
new file mode 100755

total: 0 errors, 1 warnings, 2612 lines checked

Patch 17/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
18/18 Checking commit 29d2e8995ec3 (block/backup: loosen restriction on readonly bitmaps)
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20190703215542.16123-1-jsnow@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [PATCH v2 01/18] qapi/block-core: Introduce BackupCommon

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> drive-backup and blockdev-backup have an awful lot of things in common
> that are the same. Let's fix that.
>
> I don't deduplicate 'target', because the semantics actually did change
> between each structure. Leave that one alone so it can be documented
> separately.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  qapi/block-core.json | 103 ++++++++++++++-----------------------------
>  1 file changed, 33 insertions(+), 70 deletions(-)
>
> diff --git a/qapi/block-core.json b/qapi/block-core.json
> index 0d43d4f37c..7b23efcf13 100644
> --- a/qapi/block-core.json
> +++ b/qapi/block-core.json
> @@ -1315,32 +1315,23 @@
>    'data': { 'node': 'str', 'overlay': 'str' } }
>  
>  ##
> -# @DriveBackup:
> +# @BackupCommon:
>  #
>  # @job-id: identifier for the newly-created block job. If
>  #          omitted, the device name will be used. (Since 2.7)
>  #
>  # @device: the device name or node-name of a root node which should be copied.
>  #
> -# @target: the target of the new image. If the file exists, or if it
> -#          is a device, the existing file/device will be used as the new
> -#          destination.  If it does not exist, a new file will be created.
> -#
> -# @format: the format of the new destination, default is to
> -#          probe if @mode is 'existing', else the format of the source
> -#
>  # @sync: what parts of the disk image should be copied to the destination
>  #        (all the disk, only the sectors allocated in the topmost image, from a
>  #        dirty bitmap, or only new I/O).

This is DriveBackup's wording.  Blockdev lacks "from a dirty bitmap, ".
Is this a doc fix?

>  #
> -# @mode: whether and how QEMU should create a new image, default is
> -#        'absolute-paths'.
> -#
> -# @speed: the maximum speed, in bytes per second
> +# @speed: the maximum speed, in bytes per second. The default is 0,
> +#         for unlimited.

This is Blockdev's wording.  DriveBackup lacks "the default is 0, for
unlimited."  Is this a doc fix?

>  #
>  # @bitmap: the name of dirty bitmap if sync is "incremental".
>  #          Must be present if sync is "incremental", must NOT be present
> -#          otherwise. (Since 2.4)
> +#          otherwise. (Since 2.4 (Drive), 3.1 (Blockdev))

I second Max's request to say (drive-backup) and (blockdev-backup),
strongly.

>  #
>  # @compress: true to compress data, if the target format supports it.
>  #            (default: false) (since 2.8)
> @@ -1372,73 +1363,45 @@
>  #
>  # Since: 1.6

BackupCommon is actually since 4.2.

The type doesn't appear in the external interface (only its extensions
DriveBackup and BlockdevBackup do), so documenting "since" is actually
pointless.  However, we blindly document "since" for *everything*,
simply because we don't want to waste time on deciding whether we have
to.

>  ##
> +{ 'struct': 'BackupCommon',
> +  'data': { '*job-id': 'str', 'device': 'str',
> +            'sync': 'MirrorSyncMode', '*speed': 'int',
> +            '*bitmap': 'str', '*compress': 'bool',
> +            '*on-source-error': 'BlockdevOnError',
> +            '*on-target-error': 'BlockdevOnError',
> +            '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
> +
> +##
> +# @DriveBackup:
> +#
> +# @target: the target of the new image. If the file exists, or if it
> +#          is a device, the existing file/device will be used as the new
> +#          destination.  If it does not exist, a new file will be created.
> +#
> +# @format: the format of the new destination, default is to
> +#          probe if @mode is 'existing', else the format of the source
> +#
> +# @mode: whether and how QEMU should create a new image, default is
> +#        'absolute-paths'.
> +#
> +# Since: 1.6
> +##
>  { 'struct': 'DriveBackup',
> -  'data': { '*job-id': 'str', 'device': 'str', 'target': 'str',
> -            '*format': 'str', 'sync': 'MirrorSyncMode',
> -            '*mode': 'NewImageMode', '*speed': 'int',
> -            '*bitmap': 'str', '*compress': 'bool',
> -            '*on-source-error': 'BlockdevOnError',
> -            '*on-target-error': 'BlockdevOnError',
> -            '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
> +  'base': 'BackupCommon',
> +  'data': { 'target': 'str',
> +            '*format': 'str',
> +            '*mode': 'NewImageMode' } }
>  
>  ##
>  # @BlockdevBackup:
>  #
> -# @job-id: identifier for the newly-created block job. If
> -#          omitted, the device name will be used. (Since 2.7)
> -#
> -# @device: the device name or node-name of a root node which should be copied.
> -#
>  # @target: the device name or node-name of the backup target node.
>  #
> -# @sync: what parts of the disk image should be copied to the destination
> -#        (all the disk, only the sectors allocated in the topmost image, or
> -#        only new I/O).
> -#
> -# @speed: the maximum speed, in bytes per second. The default is 0,
> -#         for unlimited.
> -#
> -# @bitmap: the name of dirty bitmap if sync is "incremental".
> -#          Must be present if sync is "incremental", must NOT be present
> -#          otherwise. (Since 3.1)
> -#
> -# @compress: true to compress data, if the target format supports it.
> -#            (default: false) (since 2.8)
> -#
> -# @on-source-error: the action to take on an error on the source,
> -#                   default 'report'.  'stop' and 'enospc' can only be used
> -#                   if the block device supports io-status (see BlockInfo).
> -#
> -# @on-target-error: the action to take on an error on the target,
> -#                   default 'report' (no limitations, since this applies to
> -#                   a different block device than @device).
> -#
> -# @auto-finalize: When false, this job will wait in a PENDING state after it has
> -#                 finished its work, waiting for @block-job-finalize before
> -#                 making any block graph changes.
> -#                 When true, this job will automatically
> -#                 perform its abort or commit actions.
> -#                 Defaults to true. (Since 2.12)
> -#
> -# @auto-dismiss: When false, this job will wait in a CONCLUDED state after it
> -#                has completely ceased all work, and awaits @block-job-dismiss.
> -#                When true, this job will automatically disappear from the query
> -#                list without user intervention.
> -#                Defaults to true. (Since 2.12)
> -#
> -# Note: @on-source-error and @on-target-error only affect background
> -# I/O.  If an error occurs during a guest write request, the device's
> -# rerror/werror actions will be used.
> -#
>  # Since: 2.3
>  ##
>  { 'struct': 'BlockdevBackup',
> -  'data': { '*job-id': 'str', 'device': 'str', 'target': 'str',
> -            'sync': 'MirrorSyncMode', '*speed': 'int',
> -            '*bitmap': 'str', '*compress': 'bool',
> -            '*on-source-error': 'BlockdevOnError',
> -            '*on-target-error': 'BlockdevOnError',
> -            '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
> +  'base': 'BackupCommon',
> +  'data': { 'target': 'str' } }
>  
>  ##
>  # @blockdev-snapshot-sync:

Re: [Qemu-devel] [PATCH v2 04/18] qapi: add BitmapSyncMode enum

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Depending on what a user is trying to accomplish, there might be a few
> bitmap cleanup actions that occur when an operation is finished that
> could be useful.
>
> I am proposing three:
> - NEVER: The bitmap is never synchronized against what was copied.
> - ALWAYS: The bitmap is always synchronized, even on failures.
> - CONDITIONAL: The bitmap is synchronized only on success.
>
> The existing incremental backup modes use 'conditional' semantics,
> so add just that one for right now.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  qapi/block-core.json | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
>
> diff --git a/qapi/block-core.json b/qapi/block-core.json
> index 7b23efcf13..87eba5a5d9 100644
> --- a/qapi/block-core.json
> +++ b/qapi/block-core.json
> @@ -1134,6 +1134,20 @@
>  { 'enum': 'MirrorSyncMode',
>    'data': ['top', 'full', 'none', 'incremental'] }
>  
> +##
> +# @BitmapSyncMode:
> +#
> +# An enumeration of possible behaviors for the synchronization of a bitmap
> +# when used for data copy operations.
> +#
> +# @conditional: The bitmap is only synced when the operation is successful.
> +#               This is the behavior always used for 'INCREMENTAL' backups.
> +#
> +# Since: 4.2
> +##
> +{ 'enum': 'BitmapSyncMode',
> +  'data': ['conditional'] }
> +
>  ##
>  # @MirrorCopyMode:
>  #

The name "conditional" makes me go "on what?".  What about "on-success"?

Re: [Qemu-devel] [PATCH v2 01/18] qapi/block-core: Introduce BackupCommon

[John Snow]

On 7/5/19 10:14 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> drive-backup and blockdev-backup have an awful lot of things in common
>> that are the same. Let's fix that.
>>
>> I don't deduplicate 'target', because the semantics actually did change
>> between each structure. Leave that one alone so it can be documented
>> separately.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>  qapi/block-core.json | 103 ++++++++++++++-----------------------------
>>  1 file changed, 33 insertions(+), 70 deletions(-)
>>
>> diff --git a/qapi/block-core.json b/qapi/block-core.json
>> index 0d43d4f37c..7b23efcf13 100644
>> --- a/qapi/block-core.json
>> +++ b/qapi/block-core.json
>> @@ -1315,32 +1315,23 @@
>>    'data': { 'node': 'str', 'overlay': 'str' } }
>>  
>>  ##
>> -# @DriveBackup:
>> +# @BackupCommon:
>>  #
>>  # @job-id: identifier for the newly-created block job. If
>>  #          omitted, the device name will be used. (Since 2.7)
>>  #
>>  # @device: the device name or node-name of a root node which should be copied.
>>  #
>> -# @target: the target of the new image. If the file exists, or if it
>> -#          is a device, the existing file/device will be used as the new
>> -#          destination.  If it does not exist, a new file will be created.
>> -#
>> -# @format: the format of the new destination, default is to
>> -#          probe if @mode is 'existing', else the format of the source
>> -#
>>  # @sync: what parts of the disk image should be copied to the destination
>>  #        (all the disk, only the sectors allocated in the topmost image, from a
>>  #        dirty bitmap, or only new I/O).
> 
> This is DriveBackup's wording.  Blockdev lacks "from a dirty bitmap, ".
> Is this a doc fix?
> 

Yes.

>>  #
>> -# @mode: whether and how QEMU should create a new image, default is
>> -#        'absolute-paths'.
>> -#
>> -# @speed: the maximum speed, in bytes per second
>> +# @speed: the maximum speed, in bytes per second. The default is 0,
>> +#         for unlimited.
> 
> This is Blockdev's wording.  DriveBackup lacks "the default is 0, for
> unlimited."  Is this a doc fix?
> 

Yes.

>>  #
>>  # @bitmap: the name of dirty bitmap if sync is "incremental".
>>  #          Must be present if sync is "incremental", must NOT be present
>> -#          otherwise. (Since 2.4)
>> +#          otherwise. (Since 2.4 (Drive), 3.1 (Blockdev))
> 
> I second Max's request to say (drive-backup) and (blockdev-backup),
> strongly.
> 

OK.

>>  #
>>  # @compress: true to compress data, if the target format supports it.
>>  #            (default: false) (since 2.8)
>> @@ -1372,73 +1363,45 @@
>>  #
>>  # Since: 1.6
> 
> BackupCommon is actually since 4.2.
> 
> The type doesn't appear in the external interface (only its extensions
> DriveBackup and BlockdevBackup do), so documenting "since" is actually
> pointless.  However, we blindly document "since" for *everything*,
> simply because we don't want to waste time on deciding whether we have
> to.
> 

Ah, sure.

>>  ##
>> +{ 'struct': 'BackupCommon',
>> +  'data': { '*job-id': 'str', 'device': 'str',
>> +            'sync': 'MirrorSyncMode', '*speed': 'int',
>> +            '*bitmap': 'str', '*compress': 'bool',
>> +            '*on-source-error': 'BlockdevOnError',
>> +            '*on-target-error': 'BlockdevOnError',
>> +            '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
>> +
>> +##
>> +# @DriveBackup:
>> +#
>> +# @target: the target of the new image. If the file exists, or if it
>> +#          is a device, the existing file/device will be used as the new
>> +#          destination.  If it does not exist, a new file will be created.
>> +#
>> +# @format: the format of the new destination, default is to
>> +#          probe if @mode is 'existing', else the format of the source
>> +#
>> +# @mode: whether and how QEMU should create a new image, default is
>> +#        'absolute-paths'.
>> +#
>> +# Since: 1.6
>> +##
>>  { 'struct': 'DriveBackup',
>> -  'data': { '*job-id': 'str', 'device': 'str', 'target': 'str',
>> -            '*format': 'str', 'sync': 'MirrorSyncMode',
>> -            '*mode': 'NewImageMode', '*speed': 'int',
>> -            '*bitmap': 'str', '*compress': 'bool',
>> -            '*on-source-error': 'BlockdevOnError',
>> -            '*on-target-error': 'BlockdevOnError',
>> -            '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
>> +  'base': 'BackupCommon',
>> +  'data': { 'target': 'str',
>> +            '*format': 'str',
>> +            '*mode': 'NewImageMode' } }
>>  
>>  ##
>>  # @BlockdevBackup:
>>  #
>> -# @job-id: identifier for the newly-created block job. If
>> -#          omitted, the device name will be used. (Since 2.7)
>> -#
>> -# @device: the device name or node-name of a root node which should be copied.
>> -#
>>  # @target: the device name or node-name of the backup target node.
>>  #
>> -# @sync: what parts of the disk image should be copied to the destination
>> -#        (all the disk, only the sectors allocated in the topmost image, or
>> -#        only new I/O).
>> -#
>> -# @speed: the maximum speed, in bytes per second. The default is 0,
>> -#         for unlimited.
>> -#
>> -# @bitmap: the name of dirty bitmap if sync is "incremental".
>> -#          Must be present if sync is "incremental", must NOT be present
>> -#          otherwise. (Since 3.1)
>> -#
>> -# @compress: true to compress data, if the target format supports it.
>> -#            (default: false) (since 2.8)
>> -#
>> -# @on-source-error: the action to take on an error on the source,
>> -#                   default 'report'.  'stop' and 'enospc' can only be used
>> -#                   if the block device supports io-status (see BlockInfo).
>> -#
>> -# @on-target-error: the action to take on an error on the target,
>> -#                   default 'report' (no limitations, since this applies to
>> -#                   a different block device than @device).
>> -#
>> -# @auto-finalize: When false, this job will wait in a PENDING state after it has
>> -#                 finished its work, waiting for @block-job-finalize before
>> -#                 making any block graph changes.
>> -#                 When true, this job will automatically
>> -#                 perform its abort or commit actions.
>> -#                 Defaults to true. (Since 2.12)
>> -#
>> -# @auto-dismiss: When false, this job will wait in a CONCLUDED state after it
>> -#                has completely ceased all work, and awaits @block-job-dismiss.
>> -#                When true, this job will automatically disappear from the query
>> -#                list without user intervention.
>> -#                Defaults to true. (Since 2.12)
>> -#
>> -# Note: @on-source-error and @on-target-error only affect background
>> -# I/O.  If an error occurs during a guest write request, the device's
>> -# rerror/werror actions will be used.
>> -#
>>  # Since: 2.3
>>  ##
>>  { 'struct': 'BlockdevBackup',
>> -  'data': { '*job-id': 'str', 'device': 'str', 'target': 'str',
>> -            'sync': 'MirrorSyncMode', '*speed': 'int',
>> -            '*bitmap': 'str', '*compress': 'bool',
>> -            '*on-source-error': 'BlockdevOnError',
>> -            '*on-target-error': 'BlockdevOnError',
>> -            '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } }
>> +  'base': 'BackupCommon',
>> +  'data': { 'target': 'str' } }
>>  
>>  ##
>>  # @blockdev-snapshot-sync:

Re: [Qemu-devel] [PATCH v2 04/18] qapi: add BitmapSyncMode enum

[John Snow]

On 7/5/19 10:18 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> Depending on what a user is trying to accomplish, there might be a few
>> bitmap cleanup actions that occur when an operation is finished that
>> could be useful.
>>
>> I am proposing three:
>> - NEVER: The bitmap is never synchronized against what was copied.
>> - ALWAYS: The bitmap is always synchronized, even on failures.
>> - CONDITIONAL: The bitmap is synchronized only on success.
>>
>> The existing incremental backup modes use 'conditional' semantics,
>> so add just that one for right now.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>  qapi/block-core.json | 14 ++++++++++++++
>>  1 file changed, 14 insertions(+)
>>
>> diff --git a/qapi/block-core.json b/qapi/block-core.json
>> index 7b23efcf13..87eba5a5d9 100644
>> --- a/qapi/block-core.json
>> +++ b/qapi/block-core.json
>> @@ -1134,6 +1134,20 @@
>>  { 'enum': 'MirrorSyncMode',
>>    'data': ['top', 'full', 'none', 'incremental'] }
>>  
>> +##
>> +# @BitmapSyncMode:
>> +#
>> +# An enumeration of possible behaviors for the synchronization of a bitmap
>> +# when used for data copy operations.
>> +#
>> +# @conditional: The bitmap is only synced when the operation is successful.
>> +#               This is the behavior always used for 'INCREMENTAL' backups.
>> +#
>> +# Since: 4.2
>> +##
>> +{ 'enum': 'BitmapSyncMode',
>> +  'data': ['conditional'] }
>> +
>>  ##
>>  # @MirrorCopyMode:
>>  #
> 
> The name "conditional" makes me go "on what?".  What about "on-success"?
> 

Good point. I do like that more.

Re: [Qemu-devel] [PATCH v2 09/18] block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal

[John Snow]

On 7/4/19 12:49 PM, Max Reitz wrote:
> On 03.07.19 23:55, John Snow wrote:
>> I'm surprised it didn't come up sooner, but sometimes we have a +busy
>> bitmap as a source. This is dangerous from the QMP API, but if we are
>> the owner that marked the bitmap busy, it's safe to merge it using it as
>> a read only source.
>>
>> It is not safe in the general case to allow users to read from in-use
>> bitmaps, so create an internal variant that foregoes the safety
>> checking.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>  block/dirty-bitmap.c      | 51 +++++++++++++++++++++++++++++++++------
>>  include/block/block_int.h |  3 +++
>>  2 files changed, 47 insertions(+), 7 deletions(-)
>>
>> diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
>> index 95a9c2a5d8..b0f76826b3 100644
>> --- a/block/dirty-bitmap.c
>> +++ b/block/dirty-bitmap.c
>> @@ -810,11 +810,15 @@ bool bdrv_dirty_bitmap_next_dirty_area(BdrvDirtyBitmap *bitmap,
>>      return hbitmap_next_dirty_area(bitmap->bitmap, offset, bytes);
>>  }
>>  
>> +/**
>> + * bdrv_merge_dirty_bitmap: merge src into dest.
>> + * Ensures permissions on bitmaps are reasonable; use for public API.
>> + *
>> + * @backup: If provided, make a copy of dest here prior to merge.
>> + */
>>  void bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
>>                               HBitmap **backup, Error **errp)
>>  {
>> -    bool ret;
>> -
>>      qemu_mutex_lock(dest->mutex);
>>      if (src->mutex != dest->mutex) {
>>          qemu_mutex_lock(src->mutex);
>> @@ -833,6 +837,37 @@ void bdrv_merge_dirty_bitmap(BdrvDirtyBitmap *dest, const BdrvDirtyBitmap *src,
>>          goto out;
>>      }
>>  
>> +    assert(bdrv_dirty_bitmap_merge_internal(dest, src, backup, false));
> 
> Please keep the explicit @ret.  We never define NDEBUG, but doing things
> with side effects inside of assert() is bad style nonetheless.
> 

Thank you for saving me from myself. I consistently forget this :(

>> +
>> +out:
>> +    qemu_mutex_unlock(dest->mutex);
>> +    if (src->mutex != dest->mutex) {
>> +        qemu_mutex_unlock(src->mutex);
>> +    }
>> +}
>> +
>> +/**
>> + * bdrv_dirty_bitmap_merge_internal: merge src into dest.
>> + * Does NOT check bitmap permissions; not suitable for use as public API.
>> + *
>> + * @backup: If provided, make a copy of dest here prior to merge.
>> + * @lock: If true, lock and unlock bitmaps on the way in/out.
>> + * returns true if the merge succeeded; false if unattempted.
>> + */
>> +bool bdrv_dirty_bitmap_merge_internal(BdrvDirtyBitmap *dest,
>> +                                      const BdrvDirtyBitmap *src,
>> +                                      HBitmap **backup,
>> +                                      bool lock)
>> +{
>> +    bool ret;
>> +
>> +    if (lock) {
>> +        qemu_mutex_lock(dest->mutex);
>> +        if (src->mutex != dest->mutex) {
>> +            qemu_mutex_lock(src->mutex);
>> +        }
>> +    }
>> +
> 
> Why not check for INCONSISTENT and RO still?
> 
> Max
> 

Well. "why", I guess. The user-facing API will always use the
non-internal version. This is the scary dangerous version that you don't
call unless you are Very Sure You Know What You're Doing.

I guess I just intended for the suitability checking to happen in
bdrv_dirty_bitmap_merge, and this is the implementation that you can
shoot yourself in the foot with if you'd like.

>>      if (backup) {
>>          *backup = dest->bitmap;
>>          dest->bitmap = hbitmap_alloc(dest->size, hbitmap_granularity(*backup));
> 

Re: [Qemu-devel] [PATCH v2 11/18] block/backup: upgrade copy_bitmap to BdrvDirtyBitmap

[John Snow]

On 7/4/19 1:29 PM, Max Reitz wrote:
> On 03.07.19 23:55, John Snow wrote:
>> This simplifies some interface matters; namely the initialization and
>> (later) merging the manifest back into the sync_bitmap if it was
>> provided.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>  block/backup.c | 76 ++++++++++++++++++++++++--------------------------
>>  1 file changed, 37 insertions(+), 39 deletions(-)
>>
>> diff --git a/block/backup.c b/block/backup.c
>> index d7fdafebda..9cc5a7f6ca 100644
>> --- a/block/backup.c
>> +++ b/block/backup.c
> 
> [...]
> 
>> @@ -202,7 +204,7 @@ static int coroutine_fn backup_do_cow(BackupBlockJob *job,
>>      cow_request_begin(&cow_request, job, start, end);
>>  
>>      while (start < end) {
>> -        if (!hbitmap_get(job->copy_bitmap, start)) {
>> +        if (!bdrv_dirty_bitmap_get(job->copy_bitmap, start)) {
>>              trace_backup_do_cow_skip(job, start);
>>              start += job->cluster_size;
>>              continue; /* already copied */
>> @@ -296,14 +298,16 @@ static void backup_abort(Job *job)
>>  static void backup_clean(Job *job)
>>  {
>>      BackupBlockJob *s = container_of(job, BackupBlockJob, common.job);
>> +    BlockDriverState *bs = blk_bs(s->target);
> 
> I’d prefer to call it target_bs, because “bs” is usually the source node.
> 

Sure;

> Which brings me to a question: Why is the copy bitmap assigned to the
> target in the first place?  Wouldn’t it make more sense on the source?
> 

*cough*;

the idea was that the target is *most likely* to be the temporary node
created for the purpose of this backup, even though backup does not
explicitly create the node.

So ... by creating it on the target, we avoid cluttering up the results
in block query; and otherwise it doesn't actually matter what node we
created it on, really.

I can move it over to the source, but the testing code has to get a
little smarter in order to find the "right" anonymous bitmap, which is
not impossible; but I thought this would actually be a convenience for
people.

>> +
>> +    if (s->copy_bitmap) {
>> +        bdrv_release_dirty_bitmap(bs, s->copy_bitmap);
>> +        s->copy_bitmap = NULL;
>> +    }
>> +
>>      assert(s->target);
>>      blk_unref(s->target);
>>      s->target = NULL;
>> -
>> -    if (s->copy_bitmap) {
>> -        hbitmap_free(s->copy_bitmap);
>> -        s->copy_bitmap = NULL;
>> -    }
>>  }
>>  
>>  void backup_do_checkpoint(BlockJob *job, Error **errp)
> 
> [...]
> 
>> @@ -387,59 +391,49 @@ static bool bdrv_is_unallocated_range(BlockDriverState *bs,
>>  
>>  static int coroutine_fn backup_loop(BackupBlockJob *job)
>>  {
>> -    int ret;
>>      bool error_is_read;
>>      int64_t offset;
>> -    HBitmapIter hbi;
>> +    BdrvDirtyBitmapIter *bdbi;
>>      BlockDriverState *bs = blk_bs(job->common.blk);
>> +    int ret = 0;
>>  
>> -    hbitmap_iter_init(&hbi, job->copy_bitmap, 0);
>> -    while ((offset = hbitmap_iter_next(&hbi)) != -1) {
>> +    bdbi = bdrv_dirty_iter_new(job->copy_bitmap);
>> +    while ((offset = bdrv_dirty_iter_next(bdbi)) != -1) {
>>          if (job->sync_mode == MIRROR_SYNC_MODE_TOP &&
>>              bdrv_is_unallocated_range(bs, offset, job->cluster_size))
>>          {
>> -            hbitmap_reset(job->copy_bitmap, offset, job->cluster_size);
>> +            bdrv_set_dirty_bitmap(job->copy_bitmap, offset, job->cluster_size);
> 
> Should be reset.
> 

Whoa, oops! I got through testing FULL but, clearly, not TOP. This also
doesn't trigger any other failures in our test suite, so I need to make
sure this is being covered.

Thank you.

>>              continue;
>>          }
>>  
>>          do {
>>              if (yield_and_check(job)) {
>> -                return 0;
>> +                goto out;
>>              }
>>              ret = backup_do_cow(job, offset,
>>                                  job->cluster_size, &error_is_read, false);
>>              if (ret < 0 && backup_error_action(job, error_is_read, -ret) ==
>>                             BLOCK_ERROR_ACTION_REPORT)
>>              {
>> -                return ret;
>> +                goto out;
>>              }
>>          } while (ret < 0);
>>      }
>>  
>> -    return 0;
>> + out:
>> +    bdrv_dirty_iter_free(bdbi);
>> +    return ret;
>>  }
>>  
>>  /* init copy_bitmap from sync_bitmap */
>>  static void backup_incremental_init_copy_bitmap(BackupBlockJob *job)
>>  {
>> -    uint64_t offset = 0;
>> -    uint64_t bytes = job->len;
>> -
>> -    while (bdrv_dirty_bitmap_next_dirty_area(job->sync_bitmap,
>> -                                             &offset, &bytes))
>> -    {
>> -        hbitmap_set(job->copy_bitmap, offset, bytes);
>> -
>> -        offset += bytes;
>> -        if (offset >= job->len) {
>> -            break;
>> -        }
>> -        bytes = job->len - offset;
>> -    }
>> +    bdrv_dirty_bitmap_merge_internal(job->copy_bitmap, job->sync_bitmap,
>> +                                     NULL, true);
> 
> How about asserting that this was successful?
> 

Sure.

>>  
>>      /* TODO job_progress_set_remaining() would make more sense */
>>      job_progress_update(&job->common.job,
>> -        job->len - hbitmap_count(job->copy_bitmap));
>> +                        job->len - bdrv_get_dirty_count(job->copy_bitmap));
>>  }
>>  
>>  static int coroutine_fn backup_run(Job *job, Error **errp)
> 
> [...]
> 
>> @@ -670,7 +668,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
>>   error:
>>      if (copy_bitmap) {
>>          assert(!job || !job->copy_bitmap);
>> -        hbitmap_free(copy_bitmap);
>> +        bdrv_release_dirty_bitmap(bs, copy_bitmap);
> 
> If you decide to keep the copy_bitmap on the target, s/bs/target/.

Ah, heck. Clearly I didn't test this error pathway either. I'll have to
add some more tests to make sure the error recovery works right.

> 
> Max
> 

Re: [Qemu-devel] [PATCH v2 12/18] block/backup: add 'always' bitmap sync policy

[John Snow]

On 7/4/19 2:05 PM, Max Reitz wrote:
> On 04.07.19 19:43, Max Reitz wrote:
>> On 03.07.19 23:55, John Snow wrote:
>>> This adds an "always" policy for bitmap synchronization. Regardless of if
>>> the job succeeds or fails, the bitmap is *always* synchronized. This means
>>> that for backups that fail part-way through, the bitmap retains a record of
>>> which sectors need to be copied out to accomplish a new backup using the
>>> old, partial result.
>>>
>>> In effect, this allows us to "resume" a failed backup; however the new backup
>>> will be from the new point in time, so it isn't a "resume" as much as it is
>>> an "incremental retry." This can be useful in the case of extremely large
>>> backups that fail considerably through the operation and we'd like to not waste
>>> the work that was already performed.
>>>
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> ---
>>>  block/backup.c       | 25 +++++++++++++++++--------
>>>  qapi/block-core.json |  5 ++++-
>>>  2 files changed, 21 insertions(+), 9 deletions(-)
>>>
>>> diff --git a/block/backup.c b/block/backup.c
>>> index 9cc5a7f6ca..495d8f71aa 100644
>>> --- a/block/backup.c
>>> +++ b/block/backup.c
>>> @@ -266,16 +266,25 @@ static void backup_cleanup_sync_bitmap(BackupBlockJob *job, int ret)
> 
> [...]
> 
>>> +        /* If we failed and synced, merge in the bits we didn't copy: */
>>> +        bdrv_dirty_bitmap_merge_internal(bm, job->copy_bitmap,
>>> +                                         NULL, true);
>>
>> I presume this is for sync=full?
> 

Well, we don't allow bitmaps for sync=full at this point in the series.

> Ah, no.  This is necessary because the original sync bitmap was
> discarded by bdrv_dirty_bitmap_abdicate().  So yep, these bits need to
> go back into the sync bitmap then.
> 
> Max
> 

Yuh -- we actually want to clear the original bitmap for the 'always'
case, which the "abdicate" handles for us. This ought to trigger only
for the always case, so I think the conditional here is correct and as
simple as it can be.

Does your R-B stand?

--js

Re: [Qemu-devel] [PATCH v2 02/18] drive-backup: create do_backup_common

[John Snow]

On 7/4/19 11:06 AM, Max Reitz wrote:
> On 03.07.19 23:55, John Snow wrote:
>> Create a common core that comprises the actual meat of what the backup API
>> boundary needs to do, and then switch drive-backup to use it.
>>
>> Questions:
>>  - do_drive_backup now acquires and releases the aio_context in addition
>>    to do_backup_common doing the same. Can I drop this from drive_backup?
> 
> I wonder why you don’t just make it a requirement that
> do_backup_common() is called with the context acquired?
> 

Ehm, it just honestly didn't occur to me that I could acquire the
context before doing the input sanitizing.

In this case, it is OK to do it, so I will hoist it back up into
do_blockdev_backup.

--js

>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>  blockdev.c | 138 ++++++++++++++++++++++++++++++++---------------------
>>  1 file changed, 83 insertions(+), 55 deletions(-)
>>
>> diff --git a/blockdev.c b/blockdev.c
>> index 4d141e9a1f..5fd663a7e5 100644
>> --- a/blockdev.c
>> +++ b/blockdev.c
>> @@ -3425,6 +3425,86 @@ out:
>>      aio_context_release(aio_context);
>>  }
>>  
>> +static BlockJob *do_backup_common(BackupCommon *backup,
>> +                                  BlockDriverState *target_bs,
>> +                                  JobTxn *txn, Error **errp)
>> +{
> 
> [...]
> 
>> +    job = backup_job_create(backup->job_id, bs, target_bs, backup->speed,
>> +                            backup->sync, bmap, backup->compress,
>> +                            backup->on_source_error, backup->on_target_error,
>> +                            job_flags, NULL, NULL, txn, &local_err);
>> +    if (local_err != NULL) {
>> +        error_propagate(errp, local_err);
>> +        goto out;
>> +    }
> 
> Below, you change do_drive_backup() to just pass errp instead of
> local_err and not do error handling.  Why not do the same here?
> 

Inertia.

> Other than that:
> 
> Reviewed-by: Max Reitz <mreitz@redhat.com>
> 

Suggestions applied, thank you.

Re: [Qemu-devel] [PATCH v2 02/18] drive-backup: create do_backup_common

[John Snow]

On 7/4/19 11:06 AM, Max Reitz wrote:
> On 03.07.19 23:55, John Snow wrote:
>> Create a common core that comprises the actual meat of what the backup API
>> boundary needs to do, and then switch drive-backup to use it.
>>
>> Questions:
>>  - do_drive_backup now acquires and releases the aio_context in addition
>>    to do_backup_common doing the same. Can I drop this from drive_backup?
> 
> I wonder why you don’t just make it a requirement that
> do_backup_common() is called with the context acquired?
> 

Oh, I remember: I wasn't actually sure if any of the calls that
drive-backup makes actually requires the AioContext. If that's true, no
reason to acquire it in two places and pass it in to the common function.

Probably bdrv_getlength and bdrv_refresh_filename both need it, though.
(Or at least, I can't promise they never wouldn't.)

--js

Re: [Qemu-devel] [PATCH v2 10/18] block/dirty-bitmap: add bdrv_dirty_bitmap_get

[John Snow]

On 7/4/19 1:01 PM, Max Reitz wrote:
> On 03.07.19 23:55, John Snow wrote:
>> Add a public interface for get. While we're at it,
>> rename "bdrv_get_dirty_bitmap_locked" to "bdrv_dirty_bitmap_get_locked".
>>
>> (There are more functions to rename to the bdrv_dirty_bitmap_VERB form,
>> but they will wait until the conclusion of this series.)
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>  block/dirty-bitmap.c         | 18 +++++++++++-------
>>  block/mirror.c               |  2 +-
>>  include/block/dirty-bitmap.h |  4 ++--
>>  migration/block.c            |  5 ++---
>>  nbd/server.c                 |  2 +-
>>  5 files changed, 17 insertions(+), 14 deletions(-)
>>
>> diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
>> index b0f76826b3..97541521ab 100644
>> --- a/block/dirty-bitmap.c
>> +++ b/block/dirty-bitmap.c
>> @@ -509,14 +509,18 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
>>  }
>>  
>>  /* Called within bdrv_dirty_bitmap_lock..unlock */
>> -bool bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
>> -                           int64_t offset)
>> +bool bdrv_dirty_bitmap_get_locked(BdrvDirtyBitmap *bitmap, int64_t offset)
>>  {
>> -    if (bitmap) {
>> -        return hbitmap_get(bitmap->bitmap, offset);
>> -    } else {
>> -        return false;
>> -    }
>> +    return hbitmap_get(bitmap->bitmap, offset);
>> +}
>> +
>> +bool bdrv_dirty_bitmap_get(BdrvDirtyBitmap *bitmap, int64_t offset) {
> 
> I’m sure Patchew has told this already, but this is not Rust yet.
> 
> With that fixed:
> 
> Reviewed-by: Max Reitz <mreitz@redhat.com>
> 

Why wait for rust to start living dangerously?

(Actually, just a thinko: I've been doing so much python lately that I
can only just barely remember to use a { instead of a :, and ...)

Re: [Qemu-devel] [PATCH v2 01/18] qapi/block-core: Introduce BackupCommon

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 7/5/19 10:14 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> drive-backup and blockdev-backup have an awful lot of things in common
>>> that are the same. Let's fix that.
>>>
>>> I don't deduplicate 'target', because the semantics actually did change
>>> between each structure. Leave that one alone so it can be documented
>>> separately.
>>>
>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>> ---
>>>  qapi/block-core.json | 103 ++++++++++++++-----------------------------
>>>  1 file changed, 33 insertions(+), 70 deletions(-)
>>>
>>> diff --git a/qapi/block-core.json b/qapi/block-core.json
>>> index 0d43d4f37c..7b23efcf13 100644
>>> --- a/qapi/block-core.json
>>> +++ b/qapi/block-core.json
>>> @@ -1315,32 +1315,23 @@
>>>    'data': { 'node': 'str', 'overlay': 'str' } }
>>>  
>>>  ##
>>> -# @DriveBackup:
>>> +# @BackupCommon:
>>>  #
>>>  # @job-id: identifier for the newly-created block job. If
>>>  #          omitted, the device name will be used. (Since 2.7)
>>>  #
>>>  # @device: the device name or node-name of a root node which should be copied.
>>>  #
>>> -# @target: the target of the new image. If the file exists, or if it
>>> -#          is a device, the existing file/device will be used as the new
>>> -#          destination.  If it does not exist, a new file will be created.
>>> -#
>>> -# @format: the format of the new destination, default is to
>>> -#          probe if @mode is 'existing', else the format of the source
>>> -#
>>>  # @sync: what parts of the disk image should be copied to the destination
>>>  #        (all the disk, only the sectors allocated in the topmost image, from a
>>>  #        dirty bitmap, or only new I/O).
>> 
>> This is DriveBackup's wording.  Blockdev lacks "from a dirty bitmap, ".
>> Is this a doc fix?
>
> Yes.

Worth mentioning in the commit message?

>>>  #
>>> -# @mode: whether and how QEMU should create a new image, default is
>>> -#        'absolute-paths'.
>>> -#
>>> -# @speed: the maximum speed, in bytes per second
>>> +# @speed: the maximum speed, in bytes per second. The default is 0,
>>> +#         for unlimited.
>> 
>> This is Blockdev's wording.  DriveBackup lacks "the default is 0, for
>> unlimited."  Is this a doc fix?
>
> Yes.

Worth mentioning in the commit message?

[...]

Re: [Qemu-devel] [PATCH v2 09/18] block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 1907 bytes --]

On 05.07.19 18:45, John Snow wrote:
> 
> 
> On 7/4/19 12:49 PM, Max Reitz wrote:
>> On 03.07.19 23:55, John Snow wrote:

[...]

>>> +
>>> +/**
>>> + * bdrv_dirty_bitmap_merge_internal: merge src into dest.
>>> + * Does NOT check bitmap permissions; not suitable for use as public API.
>>> + *
>>> + * @backup: If provided, make a copy of dest here prior to merge.
>>> + * @lock: If true, lock and unlock bitmaps on the way in/out.
>>> + * returns true if the merge succeeded; false if unattempted.
>>> + */
>>> +bool bdrv_dirty_bitmap_merge_internal(BdrvDirtyBitmap *dest,
>>> +                                      const BdrvDirtyBitmap *src,
>>> +                                      HBitmap **backup,
>>> +                                      bool lock)
>>> +{
>>> +    bool ret;
>>> +
>>> +    if (lock) {
>>> +        qemu_mutex_lock(dest->mutex);
>>> +        if (src->mutex != dest->mutex) {
>>> +            qemu_mutex_lock(src->mutex);
>>> +        }
>>> +    }
>>> +
>>
>> Why not check for INCONSISTENT and RO still?
>>
>> Max
>>
> 
> Well. "why", I guess. The user-facing API will always use the
> non-internal version. This is the scary dangerous version that you don't
> call unless you are Very Sure You Know What You're Doing.
> 
> I guess I just intended for the suitability checking to happen in
> bdrv_dirty_bitmap_merge, and this is the implementation that you can
> shoot yourself in the foot with if you'd like.

I’m asking because the reasoning behind being allowed to call this
function is that BUSY means someone who is not the monitor has exclusive
access to the bitmap, and that someone is the caller of this function.

There is no justification for why it should be allowed to call this
function for bitmaps that are inconsistent or read-only.  If someone
needs that, they should justify it with a patch, I think.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 11/18] block/backup: upgrade copy_bitmap to BdrvDirtyBitmap

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 1513 bytes --]

On 05.07.19 18:52, John Snow wrote:
> 
> 
> On 7/4/19 1:29 PM, Max Reitz wrote:

[...]

>> Which brings me to a question: Why is the copy bitmap assigned to the
>> target in the first place?  Wouldn’t it make more sense on the source?
>>
> 
> *cough*;
> 
> the idea was that the target is *most likely* to be the temporary node
> created for the purpose of this backup, even though backup does not
> explicitly create the node.
> 
> So ... by creating it on the target, we avoid cluttering up the results
> in block query; and otherwise it doesn't actually matter what node we
> created it on, really.
> 
> I can move it over to the source, but the testing code has to get a
> little smarter in order to find the "right" anonymous bitmap, which is
> not impossible; but I thought this would actually be a convenience for
> people.

You didn’t really say whether you think it makes more sense on the
source or on the target.

This is an internal bitmap, so from my understanding, the user better
not sees it at all.  It should be easy for them to ignore it, regardless
of which node it is on.  (I don’t consider “clutter” a strong point,
because, well, most of our current query interfaces are nothing but
clutter.)  Sure, that makes it a bit difficult for testing, but testing
shouldn’t really be the focus.

So for me, it comes down to where it makes more sense.  And I think it
makes more sense on the source, because it flags source clusters to copy.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 12/18] block/backup: add 'always' bitmap sync policy

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 2217 bytes --]

On 05.07.19 18:59, John Snow wrote:
> 
> 
> On 7/4/19 2:05 PM, Max Reitz wrote:
>> On 04.07.19 19:43, Max Reitz wrote:
>>> On 03.07.19 23:55, John Snow wrote:
>>>> This adds an "always" policy for bitmap synchronization. Regardless of if
>>>> the job succeeds or fails, the bitmap is *always* synchronized. This means
>>>> that for backups that fail part-way through, the bitmap retains a record of
>>>> which sectors need to be copied out to accomplish a new backup using the
>>>> old, partial result.
>>>>
>>>> In effect, this allows us to "resume" a failed backup; however the new backup
>>>> will be from the new point in time, so it isn't a "resume" as much as it is
>>>> an "incremental retry." This can be useful in the case of extremely large
>>>> backups that fail considerably through the operation and we'd like to not waste
>>>> the work that was already performed.
>>>>
>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>> ---
>>>>  block/backup.c       | 25 +++++++++++++++++--------
>>>>  qapi/block-core.json |  5 ++++-
>>>>  2 files changed, 21 insertions(+), 9 deletions(-)
>>>>
>>>> diff --git a/block/backup.c b/block/backup.c
>>>> index 9cc5a7f6ca..495d8f71aa 100644
>>>> --- a/block/backup.c
>>>> +++ b/block/backup.c
>>>> @@ -266,16 +266,25 @@ static void backup_cleanup_sync_bitmap(BackupBlockJob *job, int ret)
>>
>> [...]
>>
>>>> +        /* If we failed and synced, merge in the bits we didn't copy: */
>>>> +        bdrv_dirty_bitmap_merge_internal(bm, job->copy_bitmap,
>>>> +                                         NULL, true);
>>>
>>> I presume this is for sync=full?
>>
> 
> Well, we don't allow bitmaps for sync=full at this point in the series.
> 
>> Ah, no.  This is necessary because the original sync bitmap was
>> discarded by bdrv_dirty_bitmap_abdicate().  So yep, these bits need to
>> go back into the sync bitmap then.
>>
>> Max
>>
> 
> Yuh -- we actually want to clear the original bitmap for the 'always'
> case, which the "abdicate" handles for us. This ought to trigger only
> for the always case, so I think the conditional here is correct and as
> simple as it can be.
> 
> Does your R-B stand?

Sure.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 01/18] qapi/block-core: Introduce BackupCommon

[John Snow]

On 7/6/19 12:03 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 7/5/19 10:14 AM, Markus Armbruster wrote:
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> drive-backup and blockdev-backup have an awful lot of things in common
>>>> that are the same. Let's fix that.
>>>>
>>>> I don't deduplicate 'target', because the semantics actually did change
>>>> between each structure. Leave that one alone so it can be documented
>>>> separately.
>>>>
>>>> Signed-off-by: John Snow <jsnow@redhat.com>
>>>> ---
>>>>  qapi/block-core.json | 103 ++++++++++++++-----------------------------
>>>>  1 file changed, 33 insertions(+), 70 deletions(-)
>>>>
>>>> diff --git a/qapi/block-core.json b/qapi/block-core.json
>>>> index 0d43d4f37c..7b23efcf13 100644
>>>> --- a/qapi/block-core.json
>>>> +++ b/qapi/block-core.json
>>>> @@ -1315,32 +1315,23 @@
>>>>    'data': { 'node': 'str', 'overlay': 'str' } }
>>>>  
>>>>  ##
>>>> -# @DriveBackup:
>>>> +# @BackupCommon:
>>>>  #
>>>>  # @job-id: identifier for the newly-created block job. If
>>>>  #          omitted, the device name will be used. (Since 2.7)
>>>>  #
>>>>  # @device: the device name or node-name of a root node which should be copied.
>>>>  #
>>>> -# @target: the target of the new image. If the file exists, or if it
>>>> -#          is a device, the existing file/device will be used as the new
>>>> -#          destination.  If it does not exist, a new file will be created.
>>>> -#
>>>> -# @format: the format of the new destination, default is to
>>>> -#          probe if @mode is 'existing', else the format of the source
>>>> -#
>>>>  # @sync: what parts of the disk image should be copied to the destination
>>>>  #        (all the disk, only the sectors allocated in the topmost image, from a
>>>>  #        dirty bitmap, or only new I/O).
>>>
>>> This is DriveBackup's wording.  Blockdev lacks "from a dirty bitmap, ".
>>> Is this a doc fix?
>>
>> Yes.
> 
> Worth mentioning in the commit message?
> 
>>>>  #
>>>> -# @mode: whether and how QEMU should create a new image, default is
>>>> -#        'absolute-paths'.
>>>> -#
>>>> -# @speed: the maximum speed, in bytes per second
>>>> +# @speed: the maximum speed, in bytes per second. The default is 0,
>>>> +#         for unlimited.
>>>
>>> This is Blockdev's wording.  DriveBackup lacks "the default is 0, for
>>> unlimited."  Is this a doc fix?
>>
>> Yes.
> 
> Worth mentioning in the commit message?
> 
> [...]
> 

"eh,"

Both implementations already used the same backing implementation, so
it's really solely a doc update and it felt like a minor one, but I can
update the message.

Re: [Qemu-devel] [PATCH v2 09/18] block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal

[John Snow]

On 7/8/19 7:44 AM, Max Reitz wrote:
> On 05.07.19 18:45, John Snow wrote:
>>
>>
>> On 7/4/19 12:49 PM, Max Reitz wrote:
>>> On 03.07.19 23:55, John Snow wrote:
> 
> [...]
> 
>>>> +
>>>> +/**
>>>> + * bdrv_dirty_bitmap_merge_internal: merge src into dest.
>>>> + * Does NOT check bitmap permissions; not suitable for use as public API.
>>>> + *
>>>> + * @backup: If provided, make a copy of dest here prior to merge.
>>>> + * @lock: If true, lock and unlock bitmaps on the way in/out.
>>>> + * returns true if the merge succeeded; false if unattempted.
>>>> + */
>>>> +bool bdrv_dirty_bitmap_merge_internal(BdrvDirtyBitmap *dest,
>>>> +                                      const BdrvDirtyBitmap *src,
>>>> +                                      HBitmap **backup,
>>>> +                                      bool lock)
>>>> +{
>>>> +    bool ret;
>>>> +
>>>> +    if (lock) {
>>>> +        qemu_mutex_lock(dest->mutex);
>>>> +        if (src->mutex != dest->mutex) {
>>>> +            qemu_mutex_lock(src->mutex);
>>>> +        }
>>>> +    }
>>>> +
>>>
>>> Why not check for INCONSISTENT and RO still?
>>>
>>> Max
>>>
>>
>> Well. "why", I guess. The user-facing API will always use the
>> non-internal version. This is the scary dangerous version that you don't
>> call unless you are Very Sure You Know What You're Doing.
>>
>> I guess I just intended for the suitability checking to happen in
>> bdrv_dirty_bitmap_merge, and this is the implementation that you can
>> shoot yourself in the foot with if you'd like.
> 
> I’m asking because the reasoning behind being allowed to call this
> function is that BUSY means someone who is not the monitor has exclusive
> access to the bitmap, and that someone is the caller of this function.
> 

Unfortunately, there's no way mechanically to check that this is the
exact circumstance the function is being called in. I have named it
_internal and documented the source to explain when it's safe to use.

We do not keep track of who set +BUSY and therefore who is allowed to
call functions that normally prohibit that flag from being set.

I don't think it's worth implementing that, either.

> There is no justification for why it should be allowed to call this
> function for bitmaps that are inconsistent or read-only.  If someone
> needs that, they should justify it with a patch, I think.

The only caller here has already verified that this bitmap is not
inconsistent or read-only (because the caller MADE the bitmap). Do you
feel strongly enough that the check should be duplicated for this one
particular function?

There are many other dirty bitmap functions that, because they are
called as an interior function not directly invoked by the QMP API
layer, do not do any such checking.

GIGO, surely?

--js

Re: [Qemu-devel] [PATCH v2 11/18] block/backup: upgrade copy_bitmap to BdrvDirtyBitmap

[John Snow]

On 7/8/19 8:02 AM, Max Reitz wrote:
> On 05.07.19 18:52, John Snow wrote:
>>
>>
>> On 7/4/19 1:29 PM, Max Reitz wrote:
> 
> [...]
> 
>>> Which brings me to a question: Why is the copy bitmap assigned to the
>>> target in the first place?  Wouldn’t it make more sense on the source?
>>>
>>
>> *cough*;
>>
>> the idea was that the target is *most likely* to be the temporary node
>> created for the purpose of this backup, even though backup does not
>> explicitly create the node.
>>
>> So ... by creating it on the target, we avoid cluttering up the results
>> in block query; and otherwise it doesn't actually matter what node we
>> created it on, really.
>>
>> I can move it over to the source, but the testing code has to get a
>> little smarter in order to find the "right" anonymous bitmap, which is
>> not impossible; but I thought this would actually be a convenience for
>> people.
> 
> You didn’t really say whether you think it makes more sense on the
> source or on the target.
> 

Yeah, a bitmap that isn't recording writes seems to make about equal
sense on either to me; I chose the destination because it was more
likely to be temporary (in the drive-backup case) and I considered this
a temporary bitmap for use by the job.

Organizationally I felt that made sense. I realize it's also not
strictly true for the blockdev-backup case, but it also doesn't matter
terribly.

Semantically, it's a toss-up. Another coder could conceivably one day
decided to re-enable this bitmap and then it would make more sense on
the source. (That coder would be wrong to do that.)

> This is an internal bitmap, so from my understanding, the user better
> not sees it at all.  It should be easy for them to ignore it, regardless
> of which node it is on.  (I don’t consider “clutter” a strong point,
> because, well, most of our current query interfaces are nothing but
> clutter.)  Sure, that makes it a bit difficult for testing, but testing
> shouldn’t really be the focus.

We don't have an API to truly hide bitmaps. Vladimir wanted to add one
but I felt it complicated too much. I still don't really want one, I
don't think it's worth the time.

At the moment, anonymous bitmaps can be seen but because they have no
name, cannot be addressed. They have an implicit permission protection
that way.

(The complication is that Vladimir wanted to hide *named* bitmaps, which
has implications for the namespace that I didn't want to deal with. We
can hide anonymous bitmaps, but we can't do so automatically because
mirror and backup already use anonymous bitmaps that aren't hidden. I
hope people don't rely on seeing them in the query output, but they might.)

> 
> So for me, it comes down to where it makes more sense.  And I think it
> makes more sense on the source, because it flags source clusters to copy.
> 
> Max
> 

If you insist.

Re: [Qemu-devel] [PATCH v2 09/18] block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 3915 bytes --]

On 08.07.19 20:24, John Snow wrote:
> 
> 
> On 7/8/19 7:44 AM, Max Reitz wrote:
>> On 05.07.19 18:45, John Snow wrote:
>>>
>>>
>>> On 7/4/19 12:49 PM, Max Reitz wrote:
>>>> On 03.07.19 23:55, John Snow wrote:
>>
>> [...]
>>
>>>>> +
>>>>> +/**
>>>>> + * bdrv_dirty_bitmap_merge_internal: merge src into dest.
>>>>> + * Does NOT check bitmap permissions; not suitable for use as public API.
>>>>> + *
>>>>> + * @backup: If provided, make a copy of dest here prior to merge.
>>>>> + * @lock: If true, lock and unlock bitmaps on the way in/out.
>>>>> + * returns true if the merge succeeded; false if unattempted.
>>>>> + */
>>>>> +bool bdrv_dirty_bitmap_merge_internal(BdrvDirtyBitmap *dest,
>>>>> +                                      const BdrvDirtyBitmap *src,
>>>>> +                                      HBitmap **backup,
>>>>> +                                      bool lock)
>>>>> +{
>>>>> +    bool ret;
>>>>> +
>>>>> +    if (lock) {
>>>>> +        qemu_mutex_lock(dest->mutex);
>>>>> +        if (src->mutex != dest->mutex) {
>>>>> +            qemu_mutex_lock(src->mutex);
>>>>> +        }
>>>>> +    }
>>>>> +
>>>>
>>>> Why not check for INCONSISTENT and RO still?
>>>>
>>>> Max
>>>>
>>>
>>> Well. "why", I guess. The user-facing API will always use the
>>> non-internal version. This is the scary dangerous version that you don't
>>> call unless you are Very Sure You Know What You're Doing.
>>>
>>> I guess I just intended for the suitability checking to happen in
>>> bdrv_dirty_bitmap_merge, and this is the implementation that you can
>>> shoot yourself in the foot with if you'd like.
>>
>> I’m asking because the reasoning behind being allowed to call this
>> function is that BUSY means someone who is not the monitor has exclusive
>> access to the bitmap, and that someone is the caller of this function.
>>
> 
> Unfortunately, there's no way mechanically to check that this is the
> exact circumstance the function is being called in. I have named it
> _internal and documented the source to explain when it's safe to use.
> 
> We do not keep track of who set +BUSY and therefore who is allowed to
> call functions that normally prohibit that flag from being set.
> 
> I don't think it's worth implementing that, either.

Fully agreed.  I meant to say that calling this function on a busy
bitmap is completely fine, so I understand why there is no check and I
wouldn’t add one.

>> There is no justification for why it should be allowed to call this
>> function for bitmaps that are inconsistent or read-only.  If someone
>> needs that, they should justify it with a patch, I think.
> 
> The only caller here has already verified that this bitmap is not
> inconsistent or read-only (because the caller MADE the bitmap).

Which is why implementing it would be trivial, as the caller could just
pass &error_abort.

Well, or the function just asserts that !readonly && !inconsistent.
(Which would probably be better, because bdrv_dirty_bitmap_check() is
probably something better suited for external interfaces.  No need to
use it if all callers of bdrv_dirty_bitmap_merge_internal() only pass
&error_abort anyway.)

> Do you
> feel strongly enough that the check should be duplicated for this one
> particular function?

I don’t see a good reason not to.

I see a weak reason to add some form of a check (and be it just an
assertion), and that is that if someone needs to remove that check,
they’ll have to explicitly justify why that is OK.

Just like this patch justifies why it’s sometimes OK to call this
function on a busy bitmap.


So I feel weakly.

> There are many other dirty bitmap functions that, because they are
> called as an interior function not directly invoked by the QMP API
> layer, do not do any such checking.

Some still contain assertions, though.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 11/18] block/backup: upgrade copy_bitmap to BdrvDirtyBitmap

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 2464 bytes --]

On 08.07.19 20:32, John Snow wrote:
> 
> 
> On 7/8/19 8:02 AM, Max Reitz wrote:
>> On 05.07.19 18:52, John Snow wrote:
>>>
>>>
>>> On 7/4/19 1:29 PM, Max Reitz wrote:
>>
>> [...]
>>
>>>> Which brings me to a question: Why is the copy bitmap assigned to the
>>>> target in the first place?  Wouldn’t it make more sense on the source?
>>>>
>>>
>>> *cough*;
>>>
>>> the idea was that the target is *most likely* to be the temporary node
>>> created for the purpose of this backup, even though backup does not
>>> explicitly create the node.
>>>
>>> So ... by creating it on the target, we avoid cluttering up the results
>>> in block query; and otherwise it doesn't actually matter what node we
>>> created it on, really.
>>>
>>> I can move it over to the source, but the testing code has to get a
>>> little smarter in order to find the "right" anonymous bitmap, which is
>>> not impossible; but I thought this would actually be a convenience for
>>> people.
>>
>> You didn’t really say whether you think it makes more sense on the
>> source or on the target.
>>
> 
> Yeah, a bitmap that isn't recording writes seems to make about equal
> sense on either to me; I chose the destination because it was more
> likely to be temporary (in the drive-backup case) and I considered this
> a temporary bitmap for use by the job.
> 
> Organizationally I felt that made sense. I realize it's also not
> strictly true for the blockdev-backup case, but it also doesn't matter
> terribly.
> 
> Semantically, it's a toss-up. Another coder could conceivably one day
> decided to re-enable this bitmap and then it would make more sense on
> the source. (That coder would be wrong to do that.)

Hm.  If we had a filter node like we do for mirror, it should be there,
clearly.

...is what I wanted to say.  But then I looked it up and found out that
mirror actually still puts its bitmap on the source node.

Sure, one of the differences between backup and mirror is that backup’s
bitmap does not record writes and thus it functionally doesn’t matter
where it’s placed.  But why make mirror and backup behave more
differently than necessary?  They should be a single block job anyway.

>> So for me, it comes down to where it makes more sense.  And I think it
>> makes more sense on the source, because it flags source clusters to copy.
>>
>> Max
>>
> 
> If you insist.

I guess I kind of do, yes.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH v2 09/18] block/dirty-bitmap: add bdrv_dirty_bitmap_merge_internal

[John Snow]

On 7/8/19 2:33 PM, Max Reitz wrote:
> On 08.07.19 20:24, John Snow wrote:
>>
>>
>> On 7/8/19 7:44 AM, Max Reitz wrote:
>>> On 05.07.19 18:45, John Snow wrote:
>>>>
>>>>
>>>> On 7/4/19 12:49 PM, Max Reitz wrote:
>>>>> On 03.07.19 23:55, John Snow wrote:
>>>
>>> [...]
>>>
>>>>>> +
>>>>>> +/**
>>>>>> + * bdrv_dirty_bitmap_merge_internal: merge src into dest.
>>>>>> + * Does NOT check bitmap permissions; not suitable for use as public API.
>>>>>> + *
>>>>>> + * @backup: If provided, make a copy of dest here prior to merge.
>>>>>> + * @lock: If true, lock and unlock bitmaps on the way in/out.
>>>>>> + * returns true if the merge succeeded; false if unattempted.
>>>>>> + */
>>>>>> +bool bdrv_dirty_bitmap_merge_internal(BdrvDirtyBitmap *dest,
>>>>>> +                                      const BdrvDirtyBitmap *src,
>>>>>> +                                      HBitmap **backup,
>>>>>> +                                      bool lock)
>>>>>> +{
>>>>>> +    bool ret;
>>>>>> +
>>>>>> +    if (lock) {
>>>>>> +        qemu_mutex_lock(dest->mutex);
>>>>>> +        if (src->mutex != dest->mutex) {
>>>>>> +            qemu_mutex_lock(src->mutex);
>>>>>> +        }
>>>>>> +    }
>>>>>> +
>>>>>
>>>>> Why not check for INCONSISTENT and RO still?
>>>>>
>>>>> Max
>>>>>
>>>>
>>>> Well. "why", I guess. The user-facing API will always use the
>>>> non-internal version. This is the scary dangerous version that you don't
>>>> call unless you are Very Sure You Know What You're Doing.
>>>>
>>>> I guess I just intended for the suitability checking to happen in
>>>> bdrv_dirty_bitmap_merge, and this is the implementation that you can
>>>> shoot yourself in the foot with if you'd like.
>>>
>>> I’m asking because the reasoning behind being allowed to call this
>>> function is that BUSY means someone who is not the monitor has exclusive
>>> access to the bitmap, and that someone is the caller of this function.
>>>
>>
>> Unfortunately, there's no way mechanically to check that this is the
>> exact circumstance the function is being called in. I have named it
>> _internal and documented the source to explain when it's safe to use.
>>
>> We do not keep track of who set +BUSY and therefore who is allowed to
>> call functions that normally prohibit that flag from being set.
>>
>> I don't think it's worth implementing that, either.
> 
> Fully agreed.  I meant to say that calling this function on a busy
> bitmap is completely fine, so I understand why there is no check and I
> wouldn’t add one.
> 
>>> There is no justification for why it should be allowed to call this
>>> function for bitmaps that are inconsistent or read-only.  If someone
>>> needs that, they should justify it with a patch, I think.
>>
>> The only caller here has already verified that this bitmap is not
>> inconsistent or read-only (because the caller MADE the bitmap).
> 
> Which is why implementing it would be trivial, as the caller could just
> pass &error_abort.
> 
> Well, or the function just asserts that !readonly && !inconsistent.
> (Which would probably be better, because bdrv_dirty_bitmap_check() is
> probably something better suited for external interfaces.  No need to
> use it if all callers of bdrv_dirty_bitmap_merge_internal() only pass
> &error_abort anyway.)
> 
>> Do you
>> feel strongly enough that the check should be duplicated for this one
>> particular function?
> 
> I don’t see a good reason not to.
> 

"Fine," but I'll want the V3 changes eyed over first before I post a fixup.