[PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi

[PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi

[Patrik Janoušek]

Currently, QEMU doesn't support persistent dirty bitmaps for raw format
and also dirty bitmaps are for internal use only, and cannot be accessed
using third-party applications. These facts are very limiting
in case someone would like to develop their own backup tool becaouse
without access to the dirty bitmap it would be possible to implement
only full backups. And without persistent dirty bitmaps, it wouldn't
be possible to keep track of changed data after QEMU is restarted. And
this is exactly what I do as a part of my bachelor thesis. I've
developed a tool that is able to create incremental backups of drives
in raw format that are LVM volumes (ability to create snapshot is
required).

Please keep in mind that this is my first submission to such a large
project and also the first time when I send patch over the email.
So I hope I did it correctly.

Patrik Janoušek (2):
  block/raw: added support of persistent dirty bitmaps
  qapi: implementation of the block-dirty-bitmap-dump command

 block/meson.build               |   1 +
 block/monitor/bitmap-qmp-cmds.c |  61 ++++++++
 block/raw-format-bitmap.c       | 163 ++++++++++++++++++++
 block/raw-format.c              | 256 ++++++++++++++++++++++++++++++--
 block/raw-format.h              |  50 +++++++
 qapi/block-core.json            |  64 +++++++-
 6 files changed, 583 insertions(+), 12 deletions(-)
 create mode 100644 block/raw-format-bitmap.c
 create mode 100644 block/raw-format.h

-- 
2.31.0

[PATCH 1/2] block/raw: added support of persistent dirty bitmaps

[Patrik Janoušek]

Current implementation of dirty bitmaps for raw format is very
limited, because those bitmaps cannot be persistent. Basically it
makes sense, because the raw format doesn't have space where could
be dirty bitmap stored when QEMU is offline. This patch solves it
by storing content of every dirty bitmap in separate file on the
host filesystem.

However, this only solves one part of the problem. We also have to
store information about the existence of the dirty bitmap. This is
solved by adding custom options, that stores all required metadata
about dirty bitmap (filename where is the bitmap stored on the
host filesystem, granularity, persistence, etc.).

Signed-off-by: Patrik Janoušek <pj@patrikjanousek.cz>
---
 block/meson.build         |   1 +
 block/raw-format-bitmap.c | 163 ++++++++++++++++++++++++
 block/raw-format.c        | 256 ++++++++++++++++++++++++++++++++++++--
 block/raw-format.h        |  50 ++++++++
 4 files changed, 459 insertions(+), 11 deletions(-)
 create mode 100644 block/raw-format-bitmap.c
 create mode 100644 block/raw-format.h

diff --git a/block/meson.build b/block/meson.build
index 5dcc1e5cce..f163095574 100644
--- a/block/meson.build
+++ b/block/meson.build
@@ -30,6 +30,7 @@ block_ss.add(files(
   'qcow2.c',
   'quorum.c',
   'raw-format.c',
+  'raw-format-bitmap.c',
   'snapshot.c',
   'throttle-groups.c',
   'throttle.c',
diff --git a/block/raw-format-bitmap.c b/block/raw-format-bitmap.c
new file mode 100644
index 0000000000..f622f0a700
--- /dev/null
+++ b/block/raw-format-bitmap.c
@@ -0,0 +1,163 @@
+#include <stdio.h>
+
+#include "qemu/osdep.h"
+#include "block/block_int.h"
+
+#include "raw-format.h"
+#include "qapi/error.h"
+
+bool coroutine_fn raw_co_can_store_new_dirty_bitmap(BlockDriverState *bs,
+                                                    const char *name,
+                                                    uint32_t granularity,
+                                                    Error **errp)
+{
+    BDRVRawState *s = bs->opaque;
+
+    RawDirtyBitmapOpts *opts;
+    BdrvDirtyBitmap *bm;
+    FILE *fp;
+    bool file_exists;
+    for (int i = 0; i < s->rdbol.n; i++) {
+        opts = &s->rdbol.opts[i];
+        bm = bdrv_find_dirty_bitmap(bs, name);
+        if (!bm) {
+            error_setg(errp, "Dirty bitmap %s does not exists.", name);
+            return false;
+        }
+
+        if (!opts->has_filename) {
+            error_setg(errp, "Dirty bitmap does not have filename defined.");
+            return false;
+        }
+
+        if (!bdrv_dirty_bitmap_get_persistence(bm)) continue;
+        if (strcmp(opts->name, bdrv_dirty_bitmap_name(bm)) != 0)
+            continue;
+
+        file_exists = false;
+        fp = fopen(opts->filename, "r");
+        if (fp) {
+            file_exists = true;
+            fclose(false);
+        }
+
+        fp = fopen(opts->filename, "a");
+        if (!fp) {
+            error_setg(errp, "Unable to open file %s for read and write",
+                       opts->filename);
+            return false;
+        } else {
+            fclose(fp);
+            if (!file_exists) unlink(opts->filename);
+        }
+    }
+
+    return true;
+}
+
+bool raw_supports_persistent_dirty_bitmap(BlockDriverState *bs)
+{
+    return true;
+}
+
+int coroutine_fn raw_co_remove_persistent_dirty_bitmap(BlockDriverState *bs,
+                                                       const char *name,
+                                                       Error **errp)
+{
+    BDRVRawState *s = bs->opaque;
+
+    RawDirtyBitmapOpts *opts;
+    BdrvDirtyBitmap *bm;
+    for (int i = 0; i < s->rdbol.n; i++) {
+        opts = &s->rdbol.opts[i];
+        bm = bdrv_find_dirty_bitmap(bs, name);
+        if (!bm) {
+            error_setg(errp, "Dirty bitmap %s does not exists.", name);
+            return false;
+        }
+
+        if (!opts->has_filename) {
+            error_setg(errp, "Dirty bitmap does not have filename defined.");
+            return false;
+        }
+
+        if (!bdrv_dirty_bitmap_get_persistence(bm)) continue;
+        if (strcmp(opts->name, bdrv_dirty_bitmap_name(bm)) != 0)
+            continue;
+
+        return unlink(opts->filename);
+    }
+
+    return 0;
+}
+
+static void load_bitmap(BdrvDirtyBitmap *bm, FILE *fpbm) {
+    uint64_t bm_size = bdrv_dirty_bitmap_size(bm);
+    uint64_t tb_size = bdrv_dirty_bitmap_serialization_size(bm, 0, bm_size);
+    uint8_t *buf = g_malloc(tb_size);
+    size_t n = fread(buf, 1, tb_size, fpbm);
+
+    assert(n == tb_size);
+
+    bdrv_dirty_bitmap_deserialize_part(bm, buf, 0, bm_size, false);
+    bdrv_dirty_bitmap_deserialize_finish(bm);
+
+    g_free(buf);
+}
+
+void raw_load_persistent_dirty_bitmap(BlockDriverState *bs, FILE *fp,
+                                      uint32_t granularity, char* name,
+                                      bool persistent, bool disabled,
+                                      Error **errp)
+{
+    BdrvDirtyBitmap *bm;
+
+    bm = bdrv_create_dirty_bitmap(bs, granularity, name, errp);
+    if (bm) {
+        bdrv_dirty_bitmap_set_persistence(bm, persistent);
+
+        if (disabled)
+            bdrv_disable_dirty_bitmap(bm);
+    }
+
+    if (fp) {
+        load_bitmap(bm, fp);
+    }
+}
+
+static void store_bitmap(BdrvDirtyBitmap *bm, FILE *fpbm) {
+    uint64_t bm_size = bdrv_dirty_bitmap_size(bm);
+    uint64_t tb_size = bdrv_dirty_bitmap_serialization_size(bm, 0, bm_size);
+    uint8_t *buf = g_malloc(tb_size);
+
+    bdrv_dirty_bitmap_serialize_part(bm, buf, 0, bm_size);
+
+    fwrite(buf, 1, tb_size, fpbm);
+
+    g_free(buf);
+}
+
+void raw_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp)
+{
+    BDRVRawState *s = bs->opaque;
+
+    RawDirtyBitmapOpts *opts;
+    BdrvDirtyBitmap *bm;
+    for (int i = 0; i < s->rdbol.n; i++) {
+        opts = &s->rdbol.opts[i];
+
+        FOR_EACH_DIRTY_BITMAP(bs, bm) {
+            if (!bdrv_dirty_bitmap_get_persistence(bm)) continue;
+            assert(opts->has_filename);
+
+            if (strcmp(opts->filename, bdrv_dirty_bitmap_name(bm)) != 0)
+                continue;
+
+            FILE *fp = fopen(opts->filename, "w");
+            if(fp) {
+                store_bitmap(bm, fp);
+                fclose(fp);
+            }
+        }
+    }
+}
diff --git a/block/raw-format.c b/block/raw-format.c
index 42ec50802b..901f7a9c33 100644
--- a/block/raw-format.c
+++ b/block/raw-format.c
@@ -28,15 +28,13 @@
 
 #include "qemu/osdep.h"
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "qapi/error.h"
+#include "qapi/qmp/qlist.h"
 #include "qemu/module.h"
 #include "qemu/option.h"
 
-typedef struct BDRVRawState {
-    uint64_t offset;
-    uint64_t size;
-    bool has_size;
-} BDRVRawState;
+#include "raw-format.h"
 
 static const char *const mutable_opts[] = { "offset", "size", NULL };
 
@@ -58,6 +56,40 @@ static QemuOptsList raw_runtime_opts = {
     },
 };
 
+static QemuOptsList raw_dirty_bitmap_opts = {
+    .name = "raw-dirty-bitmap-opts",
+    .head = QTAILQ_HEAD_INITIALIZER(raw_dirty_bitmap_opts.head),
+    .desc = {
+        {
+            .name = "filename",
+            .type = QEMU_OPT_STRING,
+            .help = "path to the persistent dirty bitmap file",
+        },
+        {
+            .name = "name",
+            .type = QEMU_OPT_STRING,
+            .help = "name of the dirty bitmap",
+        },
+        {
+            .name = "granularity",
+            .type = QEMU_OPT_NUMBER,
+            .help = "granularity of the dirty bitmap",
+        },
+        {
+            .name = "persistent",
+            .type = QEMU_OPT_BOOL,
+            .help = "indicates whether the dirty bitmap is persistent"
+                    "(requires filename)",
+        },
+        {
+            .name = "disabled",
+            .type = QEMU_OPT_BOOL,
+            .help = "indicates whether the dirty bitmap is disabled",
+        },
+        { /* end of list */ }
+    },
+};
+
 static QemuOptsList raw_create_opts = {
     .name = "raw-create-opts",
     .head = QTAILQ_HEAD_INITIALIZER(raw_create_opts.head),
@@ -71,8 +103,144 @@ static QemuOptsList raw_create_opts = {
     }
 };
 
+static int raw_read_dirty_bitmap_options(QDict *options,
+                                         bool *has_filename, char **filename,
+                                         bool *has_name, char **name,
+                                         bool *has_granularity,
+                                         uint32_t *granularity,
+                                         bool *has_persistent, bool *persistent,
+                                         bool *has_disabled, bool *disabled,
+                                         Error **errp)
+{
+    QemuOpts *opts = NULL;
+    int ret;
+
+    opts = qemu_opts_create(&raw_dirty_bitmap_opts, NULL, 0, &error_abort);
+    if (!qemu_opts_absorb_qdict(opts, options, errp)) {
+        ret = -EINVAL;
+        goto end;
+    }
+
+    *has_filename = qemu_opt_find(opts, "filename");
+    *filename = g_strdup(qemu_opt_get(opts, "filename"));
+    *has_name = qemu_opt_find(opts, "name");
+    *name = g_strdup(qemu_opt_get(opts, "name"));
+    *has_granularity = qemu_opt_find(opts, "granularity");
+    *granularity = qemu_opt_get_number(opts, "granularity", 65536);
+    *has_persistent = qemu_opt_find(opts, "persistent");
+    *persistent = qemu_opt_get_bool(opts, "persistent", false);
+    *has_disabled = qemu_opt_find(opts, "disabled");
+    *disabled = qemu_opt_get_bool(opts, "disabled", false);
+
+    ret = 0;
+    end:
+    qemu_opts_del(opts);
+    return ret;
+}
+
+static int raw_read_dirty_bitmaps_options(QDict *options,
+                                          RawDirtyBitmapOptsList *rdbol,
+                                          Error **errp)
+{
+    int ret;
+    QDict *db_options = NULL;
+
+    QObject *qo;
+    char static_prefix[] = "dirty-bitmaps.";
+    uint static_prefix_len = strlen(static_prefix);
+    uint key_max_len = static_prefix_len;
+    char *key = g_malloc0(key_max_len + 1);
+    int i = 0, j;
+    char ibuff[16];
+    size_t nkey, nname;
+    *rdbol = (RawDirtyBitmapOptsList){
+        .n = 0,
+        .opts = NULL,
+    };
+    while (true) {
+        nkey = snprintf(ibuff, sizeof(ibuff), "%d.", i);
+
+        for (j = 0; raw_dirty_bitmap_opts.desc[j].name != NULL; j++) {
+            nname = strlen(raw_dirty_bitmap_opts.desc[j].name);
+            if (key_max_len < static_prefix_len + nkey + nname) {
+                key_max_len = static_prefix_len + nkey + nname;
+                key = g_realloc(key, key_max_len + 1);
+            }
+
+            strncpy(key, static_prefix, key_max_len);
+            strncpy(key + static_prefix_len, ibuff,
+                    key_max_len - static_prefix_len);
+            strncpy(key + static_prefix_len + nkey,
+                    raw_dirty_bitmap_opts.desc[j].name,
+                    key_max_len - static_prefix_len - nkey);
+            *(key + static_prefix_len + nkey + nname) = '\0';
+
+            qo = qdict_get(options, key);
+            if (qo != NULL) {
+                qobject_ref(qo);
+                qdict_del(options, key);
+
+                if (db_options == NULL) db_options = qdict_new();
+                qdict_put_obj(db_options, raw_dirty_bitmap_opts.desc[j].name,
+                              qo);
+            }
+        }
+
+        if (db_options != NULL && qdict_size(db_options) != 0) {
+            rdbol->n++;
+            if (rdbol->n == 1)
+                rdbol->opts = g_malloc0(sizeof(RawDirtyBitmapOpts) * rdbol->n);
+            else
+                rdbol->opts = g_realloc(rdbol->opts,
+                                        sizeof(RawDirtyBitmapOpts) * rdbol->n);
+
+            bool has_filename;
+            char* filename;
+            bool has_name;
+            char *name;
+            bool has_granularity;
+            uint32_t granularity;
+            bool has_persistent;
+            bool persistent;
+            bool has_disabled;
+            bool disabled;
+            ret = raw_read_dirty_bitmap_options(db_options,
+                                                &has_filename, &filename,
+                                                &has_name, &name,
+                                                &has_granularity, &granularity,
+                                                &has_persistent, &persistent,
+                                                &has_disabled, &disabled,
+                                                errp);
+            qobject_unref(db_options);
+            if (ret) {
+                goto end;
+            }
+
+            rdbol->opts[rdbol->n - 1] = (RawDirtyBitmapOpts) {
+                .has_filename = has_filename,
+                .filename = filename,
+                .has_name = has_name || has_filename,
+                .name = has_name ? name : filename,
+                .has_granularity = has_granularity,
+                .granularity = granularity,
+                .has_persistent = has_persistent,
+                .persistent = persistent,
+                .has_disabled = has_disabled,
+                .disabled = disabled,
+            };
+        } else {
+            break;
+        }
+    }
+
+    ret = 0;
+    end:
+    return ret;
+}
+
 static int raw_read_options(QDict *options, uint64_t *offset, bool *has_size,
-                            uint64_t *size, Error **errp)
+                            uint64_t *size, RawDirtyBitmapOptsList *rdbol,
+                            Error **errp)
 {
     QemuOpts *opts = NULL;
     int ret;
@@ -87,15 +255,35 @@ static int raw_read_options(QDict *options, uint64_t *offset, bool *has_size,
     *has_size = qemu_opt_find(opts, "size");
     *size = qemu_opt_get_size(opts, "size", 0);
 
+    ret = raw_read_dirty_bitmaps_options(options, rdbol, errp);
+    if (ret < 0) {
+        ret = -EINVAL;
+        goto end;
+    }
+
     ret = 0;
 end:
     qemu_opts_del(opts);
     return ret;
 }
 
+static void free_RawDirtyBitmapOptsList(RawDirtyBitmapOptsList *rdbol) {
+    if (rdbol->opts != NULL) {
+        for (int i = 0; i < rdbol->n; i++) {
+            if (rdbol->opts[i].filename != NULL)
+                g_free(rdbol->opts[i].filename);
+            if (rdbol->opts[i].name != NULL)
+                g_free(rdbol->opts[i].name);
+        }
+
+        g_free(rdbol->opts);
+        rdbol->opts = NULL;
+    }
+}
+
 static int raw_apply_options(BlockDriverState *bs, BDRVRawState *s,
                              uint64_t offset, bool has_size, uint64_t size,
-                             Error **errp)
+                             RawDirtyBitmapOptsList rdbol, Error **errp)
 {
     int64_t real_size = 0;
 
@@ -132,6 +320,29 @@ static int raw_apply_options(BlockDriverState *bs, BDRVRawState *s,
     s->offset = offset;
     s->has_size = has_size;
     s->size = has_size ? size : real_size - offset;
+    s->rdbol = rdbol;
+
+    RawDirtyBitmapOpts *opts;
+    for (int i = 0; i < rdbol.n; i++) {
+        opts = &rdbol.opts[i];
+
+        if (opts->has_persistent && opts->persistent && !opts->has_filename) {
+            error_setg(errp, "Filename is required for persistent bitmap");
+            return -EINVAL;
+        }
+
+        if (opts->has_filename) {
+            FILE *fp = fopen(opts->filename, "r");
+
+            raw_load_persistent_dirty_bitmap(bs, fp, opts->granularity,
+                                             opts->name, opts->persistent,
+                                             opts->disabled, errp);
+
+            if (fp) {
+                fclose(fp);
+            }
+        }
+    }
 
     return 0;
 }
@@ -141,6 +352,7 @@ static int raw_reopen_prepare(BDRVReopenState *reopen_state,
 {
     bool has_size;
     uint64_t offset, size;
+    RawDirtyBitmapOptsList rdbol = {0};
     int ret;
 
     assert(reopen_state != NULL);
@@ -149,13 +361,13 @@ static int raw_reopen_prepare(BDRVReopenState *reopen_state,
     reopen_state->opaque = g_new0(BDRVRawState, 1);
 
     ret = raw_read_options(reopen_state->options, &offset, &has_size, &size,
-                           errp);
+                           &rdbol, errp);
     if (ret < 0) {
         return ret;
     }
 
     ret = raw_apply_options(reopen_state->bs, reopen_state->opaque,
-                            offset, has_size, size, errp);
+                            offset, has_size, size, rdbol, errp);
     if (ret < 0) {
         return ret;
     }
@@ -170,12 +382,14 @@ static void raw_reopen_commit(BDRVReopenState *state)
 
     memcpy(s, new_s, sizeof(BDRVRawState));
 
+    free_RawDirtyBitmapOptsList(&((BDRVRawState *) state->opaque)->rdbol);
     g_free(state->opaque);
     state->opaque = NULL;
 }
 
 static void raw_reopen_abort(BDRVReopenState *state)
 {
+    free_RawDirtyBitmapOptsList(&((BDRVRawState *) state->opaque)->rdbol);
     g_free(state->opaque);
     state->opaque = NULL;
 }
@@ -438,10 +652,11 @@ static int raw_open(BlockDriverState *bs, QDict *options, int flags,
     BDRVRawState *s = bs->opaque;
     bool has_size;
     uint64_t offset, size;
+    RawDirtyBitmapOptsList rdbol = {0};
     BdrvChildRole file_role;
     int ret;
 
-    ret = raw_read_options(options, &offset, &has_size, &size, errp);
+    ret = raw_read_options(options, &offset, &has_size, &size, &rdbol, errp);
     if (ret < 0) {
         return ret;
     }
@@ -483,7 +698,7 @@ static int raw_open(BlockDriverState *bs, QDict *options, int flags,
                 bs->file->bs->filename);
     }
 
-    ret = raw_apply_options(bs, s, offset, has_size, size, errp);
+    ret = raw_apply_options(bs, s, offset, has_size, size, rdbol, errp);
     if (ret < 0) {
         return ret;
     }
@@ -496,6 +711,19 @@ static int raw_open(BlockDriverState *bs, QDict *options, int flags,
     return 0;
 }
 
+static void raw_close(BlockDriverState *bs)
+{
+    Error *local_err = NULL;
+
+    raw_store_persistent_dirty_bitmaps(bs, &local_err);
+
+    if (local_err != NULL) {
+        error_reportf_err(local_err, "Lost persistent bitmaps during "
+                          "inactivation of node '%s': ",
+                          bdrv_get_device_or_node_name(bs));
+    }
+}
+
 static int raw_probe(const uint8_t *buf, int buf_size, const char *filename)
 {
     /* smallest possible positive score so that raw is used if and only if no
@@ -583,6 +811,7 @@ BlockDriver bdrv_raw = {
     .bdrv_reopen_commit   = &raw_reopen_commit,
     .bdrv_reopen_abort    = &raw_reopen_abort,
     .bdrv_open            = &raw_open,
+    .bdrv_close           = &raw_close,
     .bdrv_child_perm      = bdrv_default_perms,
     .bdrv_co_create_opts  = &raw_co_create_opts,
     .bdrv_co_preadv       = &raw_co_preadv,
@@ -608,6 +837,11 @@ BlockDriver bdrv_raw = {
     .bdrv_has_zero_init   = &raw_has_zero_init,
     .strong_runtime_opts  = raw_strong_runtime_opts,
     .mutable_opts         = mutable_opts,
+    .bdrv_co_can_store_new_dirty_bitmap = raw_co_can_store_new_dirty_bitmap,
+    .bdrv_supports_persistent_dirty_bitmap =
+            raw_supports_persistent_dirty_bitmap,
+    .bdrv_co_remove_persistent_dirty_bitmap =
+            raw_co_remove_persistent_dirty_bitmap,
 };
 
 static void bdrv_raw_init(void)
diff --git a/block/raw-format.h b/block/raw-format.h
new file mode 100644
index 0000000000..1ac765255e
--- /dev/null
+++ b/block/raw-format.h
@@ -0,0 +1,50 @@
+#ifndef BLOCK_RAW_FORMAT_H
+#define BLOCK_RAW_FORMAT_H
+
+#include "qemu/coroutine.h"
+#include "qemu/osdep.h"
+
+typedef struct RawDirtyBitmapOpts {
+    bool has_filename;
+    char *filename;
+    bool has_name;
+    char *name;
+    bool has_granularity;
+    uint32_t granularity;
+    bool has_persistent;
+    bool persistent;
+    bool has_disabled;
+    bool disabled;
+} RawDirtyBitmapOpts;
+
+typedef struct RawDirtyBitmapOptsList {
+    size_t n;
+    RawDirtyBitmapOpts *opts;
+} RawDirtyBitmapOptsList;
+
+typedef struct BDRVRawState {
+    uint64_t offset;
+    uint64_t size;
+    bool has_size;
+    RawDirtyBitmapOptsList rdbol;
+} BDRVRawState;
+
+bool raw_co_can_store_new_dirty_bitmap(BlockDriverState *bs,
+                                       const char *name,
+                                       uint32_t granularity,
+                                       Error **errp);
+
+bool raw_supports_persistent_dirty_bitmap(BlockDriverState *bs);
+
+int raw_co_remove_persistent_dirty_bitmap(BlockDriverState *bs,
+                                          const char *name,
+                                          Error **errp);
+
+void raw_load_persistent_dirty_bitmap(BlockDriverState *bs, FILE *fp,
+                                      uint32_t granularity, char* name,
+                                      bool persistent, bool disabled,
+                                      Error **errp);
+
+void raw_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp);
+
+#endif
-- 
2.31.0

[PATCH 2/2] qapi: implementation of the block-dirty-bitmap-dump command

[Patrik Janoušek]

Currently, dirty bitmaps are for internal use only and there is
no support for accessing their content from third party-apps.
This patch implements new command block-dirty-bitmap-dump, which
returns content of the dirty bitmap encoded in base64. This is
very useful especially in combination with a drive that uses raw
format because third-party apps can easily use it to create
incremental or differential backup.

Signed-off-by: Patrik Janoušek <pj@patrikjanousek.cz>
---
 block/monitor/bitmap-qmp-cmds.c | 61 +++++++++++++++++++++++++++++++
 qapi/block-core.json            | 64 ++++++++++++++++++++++++++++++++-
 2 files changed, 124 insertions(+), 1 deletion(-)

diff --git a/block/monitor/bitmap-qmp-cmds.c b/block/monitor/bitmap-qmp-cmds.c
index 9f11deec64..7f296e9ba7 100644
--- a/block/monitor/bitmap-qmp-cmds.c
+++ b/block/monitor/bitmap-qmp-cmds.c
@@ -146,6 +146,67 @@ out:
     aio_context_release(aio_context);
 }
 
+BlockDirtyBitmapContent *qmp_block_dirty_bitmap_dump(const char *node,
+                                                     const char *name,
+                                                     bool has_clear, bool clear,
+                                                     Error **errp)
+{
+    BlockDriverState *bs;
+    BdrvDirtyBitmap *bitmap;
+    BlockDirtyBitmapContent *bdbc;
+    HBitmap *hb;
+    AioContext *aio_context;
+
+    if (!name || name[0] == '\0') {
+        error_setg(errp, "Bitmap name cannot be empty");
+        return NULL;
+    }
+
+    bs = bdrv_lookup_bs(node, node, errp);
+    if (!bs) {
+        return NULL;
+    }
+
+    aio_context = bdrv_get_aio_context(bs);
+    aio_context_acquire(aio_context);
+
+    bitmap = block_dirty_bitmap_lookup(node, name, &bs, errp);
+    if (!bitmap || !bs) {
+        return NULL;
+    }
+
+    if (has_clear && clear) {
+        /**
+         * Transactions cannot return value, so "clear" functionality must be
+         * implemented here while holding AiO context
+         */
+
+        bdrv_clear_dirty_bitmap(bitmap, &hb);
+
+        uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap);
+        uint64_t tb_size = hbitmap_serialization_size(hb, 0, bm_size);
+        uint8_t *buf = g_malloc(tb_size);
+
+        hbitmap_serialize_part(hb, buf, 0, bm_size);
+
+        bdbc = g_new0(BlockDirtyBitmapContent, 1);
+        bdbc->content = g_base64_encode((guchar *) buf, tb_size);
+    } else {
+        uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap);
+        uint64_t tb_size = bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_size);
+        uint8_t *buf = g_malloc(tb_size);
+
+        bdrv_dirty_bitmap_serialize_part(bitmap, buf, 0, bm_size);
+
+        bdbc = g_new0(BlockDirtyBitmapContent, 1);
+        bdbc->content = g_base64_encode((guchar *) buf, tb_size);
+    }
+
+    aio_context_release(aio_context);
+
+    return bdbc;
+}
+
 BdrvDirtyBitmap *block_dirty_bitmap_remove(const char *node, const char *name,
                                            bool release,
                                            BlockDriverState **bitmap_bs,
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 04ad80bc1e..cbe3dac384 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -2031,6 +2031,14 @@
 { 'struct': 'BlockDirtyBitmap',
   'data': { 'node': 'str', 'name': 'str' } }
 
+##
+# @BlockDirtyBitmapContent:
+#
+# @content: content of dirty bitmap (encoded in base64)
+##
+{ 'struct': 'BlockDirtyBitmapContent',
+  'data': { 'content': 'str' } }
+
 ##
 # @BlockDirtyBitmapAdd:
 #
@@ -2056,6 +2064,18 @@
   'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32',
             '*persistent': 'bool', '*disabled': 'bool' } }
 
+##
+# @BlockDirtyBitmapDump:
+#
+# @node: name of device/node which the bitmap is tracking
+#
+# @name: name of the dirty bitmap (must be less than 1024 bytes)
+#
+# @clear: true if bitmap should be cleared after dump
+##
+{ 'struct': 'BlockDirtyBitmapDump',
+  'data': { 'node': 'str', 'name': 'str', '*clear': 'bool' } }
+
 ##
 # @BlockDirtyBitmapMergeSource:
 #
@@ -2086,6 +2106,26 @@
   'data': { 'node': 'str', 'target': 'str',
             'bitmaps': ['BlockDirtyBitmapMergeSource'] } }
 
+##
+# @block-dirty-bitmap-dump:
+#
+# Dump a dirty bitmap with a name on the node.
+#
+# Returns: - nothing on success
+#          - If @node is not a valid block device or node, DeviceNotFound
+#          - If @name is already taken, GenericError with an explanation
+#
+# Example:
+#
+# -> { "execute": "block-dirty-bitmap-dump",
+#      "arguments": { "node": "drive0", "name": "bitmap0" } }
+# <- { "return": { "content": "TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFt... (trunc)" } }
+#
+##
+{ 'command': 'block-dirty-bitmap-dump',
+  'data': 'BlockDirtyBitmapDump',
+  'returns': 'BlockDirtyBitmapContent' }
+
 ##
 # @block-dirty-bitmap-add:
 #
@@ -3908,6 +3948,26 @@
             '*x-dirty-bitmap': 'str',
             '*reconnect-delay': 'uint32' } }
 
+##
+# @BlockdevOptionsRawDirtyBitmap:
+#
+# Dirty bitmap options for the raw driver.
+#
+# @name: the name of the dirty bitmap (Since 2.4)
+#
+# @filename: the filename of the dirty bitmap
+#
+# @granularity: granularity of the dirty bitmap in bytes (since 1.4)
+#
+# @persistent: true if the bitmap was stored on disk, is scheduled to be stored
+#              on disk, or both. (since 4.0)
+#
+# @disabled: true if the bitmap should not be loaded (and saved) automatically
+##
+{ 'struct': 'BlockdevOptionsRawDirtyBitmap',
+  'data': {'*name': 'str', 'filename': 'str', 'granularity': 'uint32',
+           'persistent': 'bool', '*disabled': 'bool' } }
+
 ##
 # @BlockdevOptionsRaw:
 #
@@ -3915,12 +3975,14 @@
 #
 # @offset: position where the block device starts
 # @size: the assumed size of the device
+# @dirty-bitmaps: dirty bitmaps of the raw block device
 #
 # Since: 2.9
 ##
 { 'struct': 'BlockdevOptionsRaw',
   'base': 'BlockdevOptionsGenericFormat',
-  'data': { '*offset': 'int', '*size': 'int' } }
+  'data': { '*offset': 'int', '*size': 'int' ,
+            '*dirty-bitmaps': ['BlockdevOptionsRawDirtyBitmap'] } }
 
 ##
 # @BlockdevOptionsThrottle:
-- 
2.31.0

Re: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi

[Vladimir Sementsov-Ogievskiy]

Hi Patrik!

20.03.2021 12:32, Patrik Janoušek wrote:
> Currently, QEMU doesn't support persistent dirty bitmaps for raw format

That's right, we don't have such feature now.

> and also dirty bitmaps are for internal use only, and cannot be accessed
> using third-party applications.

And that's is not. Bitmaps are accessible through bitmap QMP API

   block-dirty-bitmap-{add,remove,clear,merge}

And to retrieve the context of dirty bitmap you can export it through NBD protocol (see bitmaps argument in nbd specific options of block-export-add command)

> These facts are very limiting
> in case someone would like to develop their own backup tool becaouse
> without access to the dirty bitmap it would be possible to implement
> only full backups.

We do have external incremental backups, based on Qemu bitmap API. But it depends of course on qcow2 persistent bitmaps feature.

> And without persistent dirty bitmaps, it wouldn't
> be possible to keep track of changed data after QEMU is restarted. And
> this is exactly what I do as a part of my bachelor thesis. I've
> developed a tool that is able to create incremental backups of drives
> in raw format that are LVM volumes (ability to create snapshot is
> required).
> 
> Please keep in mind that this is my first submission to such a large
> project and also the first time when I send patch over the email.
> So I hope I did it correctly.
> 
> Patrik Janoušek (2):
>    block/raw: added support of persistent dirty bitmaps
>    qapi: implementation of the block-dirty-bitmap-dump command
> 
>   block/meson.build               |   1 +
>   block/monitor/bitmap-qmp-cmds.c |  61 ++++++++
>   block/raw-format-bitmap.c       | 163 ++++++++++++++++++++
>   block/raw-format.c              | 256 ++++++++++++++++++++++++++++++--
>   block/raw-format.h              |  50 +++++++
>   qapi/block-core.json            |  64 +++++++-
>   6 files changed, 583 insertions(+), 12 deletions(-)
>   create mode 100644 block/raw-format-bitmap.c
>   create mode 100644 block/raw-format.h
> 


-- 
Best regards,
Vladimir

Re: [PATCH 1/2] block/raw: added support of persistent dirty bitmaps

[Vladimir Sementsov-Ogievskiy]

20.03.2021 12:32, Patrik Janoušek wrote:
> Current implementation of dirty bitmaps for raw format is very
> limited, because those bitmaps cannot be persistent. Basically it
> makes sense, because the raw format doesn't have space where could
> be dirty bitmap stored when QEMU is offline. This patch solves it
> by storing content of every dirty bitmap in separate file on the
> host filesystem.
> 
> However, this only solves one part of the problem. We also have to
> store information about the existence of the dirty bitmap. This is
> solved by adding custom options, that stores all required metadata
> about dirty bitmap (filename where is the bitmap stored on the
> host filesystem, granularity, persistence, etc.).
> 
> Signed-off-by: Patrik Janoušek<pj@patrikjanousek.cz>


Hmm. Did you considered other ways? Honestly, I don't see a reason for yet another storing format for bitmaps.

The task could be simply solved with existing features:

1. We have extenal-data-file feature in qcow2 (read docs/interop/qcow2.txt). With this thing enabled, qcow2 file contains only metadata (persistent bitmaps for example) and data is stored in separate sequential raw file. I think you should start from it.

2. If for some reason [1] doesn't work for you, you can anyway use an empty qcow2 file to store bitmaps instead of inventing and implementing new format of bitmaps storing. (Same as your approach, you'll have a simple raw node, and additional options will say "load bitmaps from this qcow2 file". But for such options we'll need good reasons why [1] isn't enough.

-- 
Best regards,
Vladimir

Re: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi

[Patrik Janoušek]

[-- Attachment #1: Type: text/plain, Size: 2722 bytes --]

Hello, thank you for the quick reply.

On 3/22/21 9:29 AM, Vladimir Sementsov-Ogievskiy wrote:
> Hi Patrik!
>
> 20.03.2021 12:32, Patrik Janoušek wrote:
>> Currently, QEMU doesn't support persistent dirty bitmaps for raw format
>
> That's right, we don't have such feature now.
>
>> and also dirty bitmaps are for internal use only, and cannot be accessed
>> using third-party applications.
>
> And that's is not. Bitmaps are accessible through bitmap QMP API
>
>   block-dirty-bitmap-{add,remove,clear,merge}
I know about these commands, but they're useless to me without providing
access to the content of bitmap.
>
> And to retrieve the context of dirty bitmap you can export it through
> NBD protocol (see bitmaps argument in nbd specific options of
> block-export-add command)
Ok, I'll look at it.
>
>> These facts are very limiting
>> in case someone would like to develop their own backup tool becaouse
>> without access to the dirty bitmap it would be possible to implement
>> only full backups.
>
> We do have external incremental backups, based on Qemu bitmap API. But
> it depends of course on qcow2 persistent bitmaps feature.
Yes, I know. And that's the problem. The point of my bachelor thesis is
to implement a backup solution for the raw format.
>
>> And without persistent dirty bitmaps, it wouldn't
>> be possible to keep track of changed data after QEMU is restarted. And
>> this is exactly what I do as a part of my bachelor thesis. I've
>> developed a tool that is able to create incremental backups of drives
>> in raw format that are LVM volumes (ability to create snapshot is
>> required).
>>
>> Please keep in mind that this is my first submission to such a large
>> project and also the first time when I send patch over the email.
>> So I hope I did it correctly.
>>
>> Patrik Janoušek (2):
>>    block/raw: added support of persistent dirty bitmaps
>>    qapi: implementation of the block-dirty-bitmap-dump command
>>
>>   block/meson.build               |   1 +
>>   block/monitor/bitmap-qmp-cmds.c |  61 ++++++++
>>   block/raw-format-bitmap.c       | 163 ++++++++++++++++++++
>>   block/raw-format.c              | 256 ++++++++++++++++++++++++++++++--
>>   block/raw-format.h              |  50 +++++++
>>   qapi/block-core.json            |  64 +++++++-
>>   6 files changed, 583 insertions(+), 12 deletions(-)
>>   create mode 100644 block/raw-format-bitmap.c
>>   create mode 100644 block/raw-format.h
>>
>
>
Patrik Janoušek

IČO: 09291849


	+420 724 123 897 <tel:+420 724 123 897>

	pj@patrikjanousek.cz <mailto:pj@patrikjanousek.cz>


[-- Attachment #2.1: Type: text/html, Size: 10707 bytes --]

[-- Attachment #2.2: kaohbpgmkipimblo.png --]
[-- Type: image/png, Size: 177 bytes --]

[-- Attachment #2.3: biakegienjdedpnc.png --]
[-- Type: image/png, Size: 170 bytes --]

Re: [PATCH 2/2] qapi: implementation of the block-dirty-bitmap-dump command

[Vladimir Sementsov-Ogievskiy]

20.03.2021 12:32, Patrik Janoušek wrote:
> Currently, dirty bitmaps are for internal use only

As I said, that's not correct: for external use we have bitmap export through NBD protocol.

Protocol description is here https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md  (you need "Metadata querying" section)

And qemu bitmap export through NBD "metadata querying" feature is described here: docs/interop/nbd.txt


-- 
Best regards,
Vladimir

Re: [PATCH 1/2] block/raw: added support of persistent dirty bitmaps

[Patrik Janoušek]

On 3/22/21 9:41 AM, Vladimir Sementsov-Ogievskiy wrote:
> 20.03.2021 12:32, Patrik Janoušek wrote:
>> Current implementation of dirty bitmaps for raw format is very
>> limited, because those bitmaps cannot be persistent. Basically it
>> makes sense, because the raw format doesn't have space where could
>> be dirty bitmap stored when QEMU is offline. This patch solves it
>> by storing content of every dirty bitmap in separate file on the
>> host filesystem.
>>
>> However, this only solves one part of the problem. We also have to
>> store information about the existence of the dirty bitmap. This is
>> solved by adding custom options, that stores all required metadata
>> about dirty bitmap (filename where is the bitmap stored on the
>> host filesystem, granularity, persistence, etc.).
>>
>> Signed-off-by: Patrik Janoušek<pj@patrikjanousek.cz>
>
>
> Hmm. Did you considered other ways? Honestly, I don't see a reason for
> yet another storing format for bitmaps.
>
> The task could be simply solved with existing features:
>
> 1. We have extenal-data-file feature in qcow2 (read
> docs/interop/qcow2.txt). With this thing enabled, qcow2 file contains
> only metadata (persistent bitmaps for example) and data is stored in
> separate sequential raw file. I think you should start from it.

I didn't know about that feature. I'll look at it.

In case I use NBD to access the bitmap context and qcow2 as a solution
for persistent layer. Would the patch be acceptable? This is significant
change to my solution and I don't have enought time for it at the moment
(mainly due to other parts of my bachelor's thesis). I just want to know
if this kind of feature is interesting to you and its implementation is
worth my time.

>
> 2. If for some reason [1] doesn't work for you, you can anyway use an
> empty qcow2 file to store bitmaps instead of inventing and
> implementing new format of bitmaps storing. (Same as your approach,
> you'll have a simple raw node, and additional options will say "load
> bitmaps from this qcow2 file". But for such options we'll need good
> reasons why [1] isn't enough.
>

Re: [PATCH 1/2] block/raw: added support of persistent dirty bitmaps

[Vladimir Sementsov-Ogievskiy]

22.03.2021 13:18, Patrik Janoušek wrote:
> On 3/22/21 9:41 AM, Vladimir Sementsov-Ogievskiy wrote:
>> 20.03.2021 12:32, Patrik Janoušek wrote:
>>> Current implementation of dirty bitmaps for raw format is very
>>> limited, because those bitmaps cannot be persistent. Basically it
>>> makes sense, because the raw format doesn't have space where could
>>> be dirty bitmap stored when QEMU is offline. This patch solves it
>>> by storing content of every dirty bitmap in separate file on the
>>> host filesystem.
>>>
>>> However, this only solves one part of the problem. We also have to
>>> store information about the existence of the dirty bitmap. This is
>>> solved by adding custom options, that stores all required metadata
>>> about dirty bitmap (filename where is the bitmap stored on the
>>> host filesystem, granularity, persistence, etc.).
>>>
>>> Signed-off-by: Patrik Janoušek<pj@patrikjanousek.cz>
>>
>>
>> Hmm. Did you considered other ways? Honestly, I don't see a reason for
>> yet another storing format for bitmaps.
>>
>> The task could be simply solved with existing features:
>>
>> 1. We have extenal-data-file feature in qcow2 (read
>> docs/interop/qcow2.txt). With this thing enabled, qcow2 file contains
>> only metadata (persistent bitmaps for example) and data is stored in
>> separate sequential raw file. I think you should start from it.
> 
> I didn't know about that feature. I'll look at it.
> 
> In case I use NBD to access the bitmap context and qcow2 as a solution
> for persistent layer. Would the patch be acceptable? This is significant
> change to my solution and I don't have enought time for it at the moment
> (mainly due to other parts of my bachelor's thesis). I just want to know
> if this kind of feature is interesting to you and its implementation is
> worth my time.

Honestly, at this point I think it doesn't. If existing features satisfy your use-case, no reason to increase complexity of file-posix driver and QAPI.

And if existing features are not enough, we should start from understanding why they are not enough, and probably true way would be improving the existing features instead of implementing from scratch.

> 
>>
>> 2. If for some reason [1] doesn't work for you, you can anyway use an
>> empty qcow2 file to store bitmaps instead of inventing and
>> implementing new format of bitmaps storing. (Same as your approach,
>> you'll have a simple raw node, and additional options will say "load
>> bitmaps from this qcow2 file". But for such options we'll need good
>> reasons why [1] isn't enough.
>>
> 


-- 
Best regards,
Vladimir

Re: Fwd: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi

[Max Reitz]

Hi,

On 20.03.21 11:01, Patrik Janoušek wrote:
> I'm sorry, but I forgot to add you to the cc, so I'm forwarding the 
> patch to you additionally. I don't want to spam the mailing list 
> unnecessarily.

I think it’s better to still CC the list.  It’s so full of mail, one 
more won’t hurt. :)

(Re-adding qemu-block and qemu-devel, because the discussion belongs on 
the list(s).)

> -------- Forwarded Message --------
> Subject: 	[PATCH 0/2] block/raw: implemented persistent dirty bitmap and 
> ability to dump bitmap content via qapi
> Date: 	Sat, 20 Mar 2021 10:32:33 +0100
> From: 	Patrik Janoušek <pj@patrikjanousek.cz>
> To: 	qemu-devel@nongnu.org
> CC: 	Patrik Janoušek <pj@patrikjanousek.cz>, lmatejka@kiv.zcu.cz
> 
> 
> 
> Currently, QEMU doesn't support persistent dirty bitmaps for raw format
> and also dirty bitmaps are for internal use only, and cannot be accessed
> using third-party applications. These facts are very limiting
> in case someone would like to develop their own backup tool becaouse
> without access to the dirty bitmap it would be possible to implement
> only full backups. And without persistent dirty bitmaps, it wouldn't
> be possible to keep track of changed data after QEMU is restarted. And
> this is exactly what I do as a part of my bachelor thesis. I've
> developed a tool that is able to create incremental backups of drives
> in raw format that are LVM volumes (ability to create snapshot is
> required).

Similarly to what Vladimir has said already, the thing is that 
conceptually I can see no difference between having a raw image with the 
bitmaps stored in some other file, i.e.:

   { "driver": "raw",
     "dirty-bitmaps": [ {
       "filename": "sdc1.bitmap",
       "persistent": true
     } ],
     "file": {
       "driver": "file",
       "filename": "/dev/sdc1"
     } }

And having a qcow2 image with the raw data stored in some other file, i.e.:

   { "driver": "qcow2",
     "file": {
       "driver": "file",
       "filename": "sdc1.metadata"
     },
     "data-file": {
       "driver": "file",
       "filename": "/dev/sdc1"
     } }

(Where sdc1.metadata is a qcow2 file created with
“data-file=/dev/sdc1,data-file-raw=on”.)

To use persistent bitmaps with raw images, you need to add metadata 
(namely, the bitmaps).  Why not store that metadata in a qcow2 file?

Max

Re: [PATCH 1/2] block/raw: added support of persistent dirty bitmaps

[Vladimir Sementsov-Ogievskiy]

22.03.2021 13:46, Vladimir Sementsov-Ogievskiy wrote:
> 22.03.2021 13:18, Patrik Janoušek wrote:
>> On 3/22/21 9:41 AM, Vladimir Sementsov-Ogievskiy wrote:
>>> 20.03.2021 12:32, Patrik Janoušek wrote:
>>>> Current implementation of dirty bitmaps for raw format is very
>>>> limited, because those bitmaps cannot be persistent. Basically it
>>>> makes sense, because the raw format doesn't have space where could
>>>> be dirty bitmap stored when QEMU is offline. This patch solves it
>>>> by storing content of every dirty bitmap in separate file on the
>>>> host filesystem.
>>>>
>>>> However, this only solves one part of the problem. We also have to
>>>> store information about the existence of the dirty bitmap. This is
>>>> solved by adding custom options, that stores all required metadata
>>>> about dirty bitmap (filename where is the bitmap stored on the
>>>> host filesystem, granularity, persistence, etc.).
>>>>
>>>> Signed-off-by: Patrik Janoušek<pj@patrikjanousek.cz>
>>>
>>>
>>> Hmm. Did you considered other ways? Honestly, I don't see a reason for
>>> yet another storing format for bitmaps.
>>>
>>> The task could be simply solved with existing features:
>>>
>>> 1. We have extenal-data-file feature in qcow2 (read
>>> docs/interop/qcow2.txt). With this thing enabled, qcow2 file contains
>>> only metadata (persistent bitmaps for example) and data is stored in
>>> separate sequential raw file. I think you should start from it.
>>
>> I didn't know about that feature. I'll look at it.
>>
>> In case I use NBD to access the bitmap context and qcow2 as a solution
>> for persistent layer. Would the patch be acceptable? This is significant
>> change to my solution and I don't have enought time for it at the moment
>> (mainly due to other parts of my bachelor's thesis). I just want to know
>> if this kind of feature is interesting to you and its implementation is
>> worth my time.
> 
> Honestly, at this point I think it doesn't. If existing features satisfy your use-case, no reason to increase complexity of file-posix driver and QAPI.
> 

It's unpleasant to say this, keeping in mind that that's your first submission :(

I can still recommend in a connection with your bachelor's thesis to look at the videos at kvm-forum youtube channel, searching for backup:

   https://www.youtube.com/channel/UCRCSQmAOh7yzgheq-emy1xA/search?query=backup

You'll get a lot of information about current developments of external backup API.

Also note, that there is (or there will be ?) libvirt Backup API, which includes an API for external backup. I don't know the current status of it, but if your project is based on libvirt, it's better to use libvirt backup API instead of using qemu directly. About Libvirt Backup API it's better to ask Eric Blake (adding him to CC).


-- 
Best regards,
Vladimir

Re: Fwd: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi

[Patrik Janoušek]

On 3/22/21 11:48 AM, Max Reitz wrote:
> Hi,
>
> On 20.03.21 11:01, Patrik Janoušek wrote:
>> I'm sorry, but I forgot to add you to the cc, so I'm forwarding the
>> patch to you additionally. I don't want to spam the mailing list
>> unnecessarily.
>
> I think it’s better to still CC the list.  It’s so full of mail, one
> more won’t hurt. :)
>
> (Re-adding qemu-block and qemu-devel, because the discussion belongs
> on the list(s).)
>
>> -------- Forwarded Message --------
>> Subject:     [PATCH 0/2] block/raw: implemented persistent dirty
>> bitmap and ability to dump bitmap content via qapi
>> Date:     Sat, 20 Mar 2021 10:32:33 +0100
>> From:     Patrik Janoušek <pj@patrikjanousek.cz>
>> To:     qemu-devel@nongnu.org
>> CC:     Patrik Janoušek <pj@patrikjanousek.cz>, lmatejka@kiv.zcu.cz
>>
>>
>>
>> Currently, QEMU doesn't support persistent dirty bitmaps for raw format
>> and also dirty bitmaps are for internal use only, and cannot be accessed
>> using third-party applications. These facts are very limiting
>> in case someone would like to develop their own backup tool becaouse
>> without access to the dirty bitmap it would be possible to implement
>> only full backups. And without persistent dirty bitmaps, it wouldn't
>> be possible to keep track of changed data after QEMU is restarted. And
>> this is exactly what I do as a part of my bachelor thesis. I've
>> developed a tool that is able to create incremental backups of drives
>> in raw format that are LVM volumes (ability to create snapshot is
>> required).
>
> Similarly to what Vladimir has said already, the thing is that
> conceptually I can see no difference between having a raw image with
> the bitmaps stored in some other file, i.e.:
>
>   { "driver": "raw",
>     "dirty-bitmaps": [ {
>       "filename": "sdc1.bitmap",
>       "persistent": true
>     } ],
>     "file": {
>       "driver": "file",
>       "filename": "/dev/sdc1"
>     } }
>
> And having a qcow2 image with the raw data stored in some other file,
> i.e.:
>
>   { "driver": "qcow2",
>     "file": {
>       "driver": "file",
>       "filename": "sdc1.metadata"
>     },
>     "data-file": {
>       "driver": "file",
>       "filename": "/dev/sdc1"
>     } }
>
> (Where sdc1.metadata is a qcow2 file created with
> “data-file=/dev/sdc1,data-file-raw=on”.)
>
> To use persistent bitmaps with raw images, you need to add metadata
> (namely, the bitmaps).  Why not store that metadata in a qcow2 file?
>
> Max

So if I understand it correctly. I can configure dirty bitmaps in the
latest version of QEMU to be persistently stored in some other file.
Because even Proxmox Backup Server can't perform an incremental backup
after restarting QEMU, and that means something to me. I think they
would implement it if it was that simple.

Could you please send me simple example on how to configure (via command
line args) one raw format drive that can store dirty bitmaps
persistently in other qcow2 file? I may be missing something, but I
thought QEMU couldn't do it, because Proxmox community wants this
feature for a long time.

Patrik

Re: [PATCH 1/2] block/raw: added support of persistent dirty bitmaps

[Patrik Janoušek]

On 3/22/21 12:18 PM, Vladimir Sementsov-Ogievskiy wrote:
> 22.03.2021 13:46, Vladimir Sementsov-Ogievskiy wrote:
>> 22.03.2021 13:18, Patrik Janoušek wrote:
>>> On 3/22/21 9:41 AM, Vladimir Sementsov-Ogievskiy wrote:
>>>> 20.03.2021 12:32, Patrik Janoušek wrote:
>>>>> Current implementation of dirty bitmaps for raw format is very
>>>>> limited, because those bitmaps cannot be persistent. Basically it
>>>>> makes sense, because the raw format doesn't have space where could
>>>>> be dirty bitmap stored when QEMU is offline. This patch solves it
>>>>> by storing content of every dirty bitmap in separate file on the
>>>>> host filesystem.
>>>>>
>>>>> However, this only solves one part of the problem. We also have to
>>>>> store information about the existence of the dirty bitmap. This is
>>>>> solved by adding custom options, that stores all required metadata
>>>>> about dirty bitmap (filename where is the bitmap stored on the
>>>>> host filesystem, granularity, persistence, etc.).
>>>>>
>>>>> Signed-off-by: Patrik Janoušek<pj@patrikjanousek.cz>
>>>>
>>>>
>>>> Hmm. Did you considered other ways? Honestly, I don't see a reason for
>>>> yet another storing format for bitmaps.
>>>>
>>>> The task could be simply solved with existing features:
>>>>
>>>> 1. We have extenal-data-file feature in qcow2 (read
>>>> docs/interop/qcow2.txt). With this thing enabled, qcow2 file contains
>>>> only metadata (persistent bitmaps for example) and data is stored in
>>>> separate sequential raw file. I think you should start from it.
>>>
>>> I didn't know about that feature. I'll look at it.
>>>
>>> In case I use NBD to access the bitmap context and qcow2 as a solution
>>> for persistent layer. Would the patch be acceptable? This is
>>> significant
>>> change to my solution and I don't have enought time for it at the
>>> moment
>>> (mainly due to other parts of my bachelor's thesis). I just want to
>>> know
>>> if this kind of feature is interesting to you and its implementation is
>>> worth my time.
>>
>> Honestly, at this point I think it doesn't. If existing features
>> satisfy your use-case, no reason to increase complexity of file-posix
>> driver and QAPI.
>>
>
> It's unpleasant to say this, keeping in mind that that's your first
> submission :(
>
> I can still recommend in a connection with your bachelor's thesis to
> look at the videos at kvm-forum youtube channel, searching for backup:
>
>  
> https://www.youtube.com/channel/UCRCSQmAOh7yzgheq-emy1xA/search?query=backup
>
> You'll get a lot of information about current developments of external
> backup API.
>
> Also note, that there is (or there will be ?) libvirt Backup API,
> which includes an API for external backup. I don't know the current
> status of it, but if your project is based on libvirt, it's better to
> use libvirt backup API instead of using qemu directly. About Libvirt
> Backup API it's better to ask Eric Blake (adding him to CC).
Unfortunately, my solution is based on Proxmox so I can't use libvirt's
features. I know that a beta version of Proxmox Backup Server has been
released and it would be much better to improve their solution, but they
did it too late so I couldn't change assignment of my bachelor's thesis.

Re: Fwd: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi

[Max Reitz]

On 22.03.21 12:27, Patrik Janoušek wrote:
> On 3/22/21 11:48 AM, Max Reitz wrote:
>> Hi,
>>
>> On 20.03.21 11:01, Patrik Janoušek wrote:
>>> I'm sorry, but I forgot to add you to the cc, so I'm forwarding the
>>> patch to you additionally. I don't want to spam the mailing list
>>> unnecessarily.
>>
>> I think it’s better to still CC the list.  It’s so full of mail, one
>> more won’t hurt. :)
>>
>> (Re-adding qemu-block and qemu-devel, because the discussion belongs
>> on the list(s).)
>>
>>> -------- Forwarded Message --------
>>> Subject:     [PATCH 0/2] block/raw: implemented persistent dirty
>>> bitmap and ability to dump bitmap content via qapi
>>> Date:     Sat, 20 Mar 2021 10:32:33 +0100
>>> From:     Patrik Janoušek <pj@patrikjanousek.cz>
>>> To:     qemu-devel@nongnu.org
>>> CC:     Patrik Janoušek <pj@patrikjanousek.cz>, lmatejka@kiv.zcu.cz
>>>
>>>
>>>
>>> Currently, QEMU doesn't support persistent dirty bitmaps for raw format
>>> and also dirty bitmaps are for internal use only, and cannot be accessed
>>> using third-party applications. These facts are very limiting
>>> in case someone would like to develop their own backup tool becaouse
>>> without access to the dirty bitmap it would be possible to implement
>>> only full backups. And without persistent dirty bitmaps, it wouldn't
>>> be possible to keep track of changed data after QEMU is restarted. And
>>> this is exactly what I do as a part of my bachelor thesis. I've
>>> developed a tool that is able to create incremental backups of drives
>>> in raw format that are LVM volumes (ability to create snapshot is
>>> required).
>>
>> Similarly to what Vladimir has said already, the thing is that
>> conceptually I can see no difference between having a raw image with
>> the bitmaps stored in some other file, i.e.:
>>
>>    { "driver": "raw",
>>      "dirty-bitmaps": [ {
>>        "filename": "sdc1.bitmap",
>>        "persistent": true
>>      } ],
>>      "file": {
>>        "driver": "file",
>>        "filename": "/dev/sdc1"
>>      } }
>>
>> And having a qcow2 image with the raw data stored in some other file,
>> i.e.:
>>
>>    { "driver": "qcow2",
>>      "file": {
>>        "driver": "file",
>>        "filename": "sdc1.metadata"
>>      },
>>      "data-file": {
>>        "driver": "file",
>>        "filename": "/dev/sdc1"
>>      } }
>>
>> (Where sdc1.metadata is a qcow2 file created with
>> “data-file=/dev/sdc1,data-file-raw=on”.)
>>
>> To use persistent bitmaps with raw images, you need to add metadata
>> (namely, the bitmaps).  Why not store that metadata in a qcow2 file?
>>
>> Max
> 
> So if I understand it correctly. I can configure dirty bitmaps in the
> latest version of QEMU to be persistently stored in some other file.
> Because even Proxmox Backup Server can't perform an incremental backup
> after restarting QEMU, and that means something to me. I think they
> would implement it if it was that simple.
> 
> Could you please send me simple example on how to configure (via command
> line args) one raw format drive that can store dirty bitmaps
> persistently in other qcow2 file? I may be missing something, but I
> thought QEMU couldn't do it, because Proxmox community wants this
> feature for a long time.

One trouble is that if you use qemu-img create to create the qcow2 
image, it will always create an empty image, and so if use pass 
data_file to it, it will empty the existing raw image:

$ cp ~/tmp/arch.iso raw.img # Just some Arch Linux ISO

$ qemu-img create \
     -f qcow2 \
     -o data_file=raw.img,data_file_raw=on,preallocation=metadata \
     metadata.qcow2 \
     $(stat -c '%s' raw.img)
Formatting 'metadata.qcow2', fmt=qcow2 cluster_size=65536 
preallocation=metadata compression_type=zlib size=687865856 
data_file=raw.img data_file_raw=on lazy_refcounts=off refcount_bits=16

(If you check raw.img at this point, you’ll find that it’s empty, so you 
need to copy it from the source again:)

$ cp ~/tmp/arch.iso raw.img

Now if you use metadata.qcow2, the image data will actually all be 
stored in raw.img.


To get around the “creating metadata.qcow2 clears raw.img” problem, you 
can either create a temporary empty image of the same size as raw.img 
that you pass to qemu-img create, and then you use qemu-img amend to 
change the data-file pointer (which will not overwrite the new 
data-file’s contents):

$ qemu-img create -f raw tmp.raw $(stat -c '%s' raw.img)

$ qemu-img create \
     -f qcow2 \
     -o data_file=tmp.img,data_file_raw=on,preallocation=metadata \
     metadata.qcow2 \
     $(stat -c '%s' raw.img)
Formatting 'metadata.qcow2', fmt=qcow2 cluster_size=65536 
preallocation=metadata compression_type=zlib size=687865856 
data_file=tmp.img data_file_raw=on lazy_refcounts=off refcount_bits=16

$ qemu-img amend -o data_file=raw.img metadata.qcow2

$ rm tmp.img


Or you use the blockdev-create job to create the qcow2 image (because 
contrary to qemu-img create, that will not clear the data file):

$ touch metadata.qcow2

(Note that in the following QMP communication, what I sent and what qemu 
replies is mixed.  Everything that begins with '{ "execute"' is from me, 
everything else from qemu.  The number 687865856 is the size of raw.img 
in bytes.)

$ qemu-system-x86_64 -qmp stdio \
     -blockdev \
     '{ "node-name": "metadata-file",
        "driver": "file",
        "filename": "metadata.qcow2" }' \
     -blockdev \
     '{ "node-name": "data-file",
        "driver": "file",
        "filename": "raw.img" }'
{"QMP": {"version": {"qemu": {"micro": 0, "minor": 1, "major": 5}, 
"package": "qemu-5.1.0-9.fc33"}, "capabilities": ["oob"]}}

{ "execute": "qmp_capabilities" }
{"return": {}}

{ "execute": "blockdev-create",
   "arguments": {
     "job-id": "create",
     "options": {
       "driver": "qcow2",
       "file": "metadata-file",
       "data-file": "data-file",
       "data-file-raw": true,
       "preallocation": "metadata",
       "size": 687865856
     }
   }
}
{"timestamp": {"seconds": 1616414002, "microseconds": 836899}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "created", "id": "create"}}
{"timestamp": {"seconds": 1616414002, "microseconds": 837076}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "running", "id": "create"}}
{"return": {}}
{"timestamp": {"seconds": 1616414002, "microseconds": 870997}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "create"}}
{"timestamp": {"seconds": 1616414002, "microseconds": 871099}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "create"}}
{"timestamp": {"seconds": 1616414002, "microseconds": 871185}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "create"}}

{ "execute": "job-dismiss", "arguments": { "id": "create" } }
{"timestamp": {"seconds": 1616414022, "microseconds": 202880}, "event": 
"JOB_STATUS_CHANGE", "data": {"status": "null", "id": "create"}}
{"return": {}}

{ "execute": "quit" }
{"return": {}}
{"timestamp": {"seconds": 1616414028, "microseconds": 56457}, "event": 
"SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}


In any case, in the end you get a metadata.qcow2 that holds the metadata 
and points to raw.img for its data.  So you can use it like this:

$ qemu-system-x86_64 -enable-kvm -m 512 \
     -blockdev \
     '{ "node-name": "node0",
        "driver": "qcow2",
        "file": {
          "driver": "file",
          "filename": "metadata.qcow2"
        } }' \
     -device ide-cd,drive=node0 \
     -qmp stdio
{"QMP": {"version": {"qemu": {"micro": 0, "minor": 1, "major": 5}, 
"package": "qemu-5.1.0-9.fc33"}, "capabilities": ["oob"]}}

{ "execute": "qmp_capabilities" }
{"return": {}}

{ "execute": "block-dirty-bitmap-add",
   "arguments": {
     "node": "node0",
     "name": "bmap0",
     "persistent": true
   }
}
{"return": {}}

{ "execute": "quit" }
{"return": {}}
{"timestamp": {"seconds": 1616414627, "microseconds": 928250}, "event": 
"SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}

$ qemu-img info metadata.qcow2
image: metadata.qcow2
file format: qcow2
virtual size: 656 MiB (687865856 bytes)
disk size: 452 KiB
cluster_size: 65536
Format specific information:
     compat: 1.1
     compression type: zlib
     lazy refcounts: false
     bitmaps:
         [0]:
             flags:
                 [0]: auto
             name: bmap0
             granularity: 65536
     refcount bits: 16
     data file: raw.img
     data file raw: true
     corrupt: false

So the bitmap is now in metadata.qcow2, and as the disk size indicates, 
all the data is still in raw.img.


I hope the above helps you.  Sorry if it’s confusing, especially the 
first part where I’m like “The obvious way to create metadata.qcow2 will 
delete your data, so here are two alternatives that are weird but do 
what you want.”

Max

Re: Fwd: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi

[Fabian Grünbichler]

On March 22, 2021 12:27 pm, Patrik Janoušek wrote:
> On 3/22/21 11:48 AM, Max Reitz wrote:
>> Hi,
>>
>> On 20.03.21 11:01, Patrik Janoušek wrote:
>>> I'm sorry, but I forgot to add you to the cc, so I'm forwarding the
>>> patch to you additionally. I don't want to spam the mailing list
>>> unnecessarily.
>>
>> I think it’s better to still CC the list.  It’s so full of mail, one
>> more won’t hurt. :)
>>
>> (Re-adding qemu-block and qemu-devel, because the discussion belongs
>> on the list(s).)
>>
>>> -------- Forwarded Message --------
>>> Subject:     [PATCH 0/2] block/raw: implemented persistent dirty
>>> bitmap and ability to dump bitmap content via qapi
>>> Date:     Sat, 20 Mar 2021 10:32:33 +0100
>>> From:     Patrik Janoušek <pj@patrikjanousek.cz>
>>> To:     qemu-devel@nongnu.org
>>> CC:     Patrik Janoušek <pj@patrikjanousek.cz>, lmatejka@kiv.zcu.cz
>>>
>>>
>>>
>>> Currently, QEMU doesn't support persistent dirty bitmaps for raw format
>>> and also dirty bitmaps are for internal use only, and cannot be accessed
>>> using third-party applications. These facts are very limiting
>>> in case someone would like to develop their own backup tool becaouse
>>> without access to the dirty bitmap it would be possible to implement
>>> only full backups. And without persistent dirty bitmaps, it wouldn't
>>> be possible to keep track of changed data after QEMU is restarted. And
>>> this is exactly what I do as a part of my bachelor thesis. I've
>>> developed a tool that is able to create incremental backups of drives
>>> in raw format that are LVM volumes (ability to create snapshot is
>>> required).
>>
>> Similarly to what Vladimir has said already, the thing is that
>> conceptually I can see no difference between having a raw image with
>> the bitmaps stored in some other file, i.e.:
>>
>>   { "driver": "raw",
>>     "dirty-bitmaps": [ {
>>       "filename": "sdc1.bitmap",
>>       "persistent": true
>>     } ],
>>     "file": {
>>       "driver": "file",
>>       "filename": "/dev/sdc1"
>>     } }
>>
>> And having a qcow2 image with the raw data stored in some other file,
>> i.e.:
>>
>>   { "driver": "qcow2",
>>     "file": {
>>       "driver": "file",
>>       "filename": "sdc1.metadata"
>>     },
>>     "data-file": {
>>       "driver": "file",
>>       "filename": "/dev/sdc1"
>>     } }
>>
>> (Where sdc1.metadata is a qcow2 file created with
>> “data-file=/dev/sdc1,data-file-raw=on”.)
>>
>> To use persistent bitmaps with raw images, you need to add metadata
>> (namely, the bitmaps).  Why not store that metadata in a qcow2 file?
>>
>> Max
> 
> So if I understand it correctly. I can configure dirty bitmaps in the
> latest version of QEMU to be persistently stored in some other file.
> Because even Proxmox Backup Server can't perform an incremental backup
> after restarting QEMU, and that means something to me. I think they
> would implement it if it was that simple.

the main reason we haven't implemented something like that (yet) is not 
that it is technically difficult, but that we have no way to safeguard 
against external manipulation of the (raw) image:

- VM is running, backup happens, bitmap represents the delta since this 
  backup
- VM is stopped
- user/admin does something to VM disk image, believing such an action 
  is safe because VM is not running
- VM is started again - bitmap does not contain the manual changes
- VM is running, backup happens, backup is inconsistent with actual data 
  stored in image

because of persistence, this error is now carried forward indefinitely 
(it might self-correct at some point if all the invalid stuff has been 
dirtied by the guest).

while it is of course possible to argue that this is entirely the user's 
fault, it looks like it's the backup software/hypervisor's fault (no 
indication anything is wrong, backup data is bogus).

it might happen at some point as opt-in feature with all the warnings 
associated with potentially dangerous features, but for now we are okay 
with just carrying the bitmap as long as the VM instance is running 
(including migrations), and having to re-read and chunk/hash the disks 
for fresh instances. the latter is expensive, but less expensive than 
having to explain to users why their backups are bogus.

> 
> Could you please send me simple example on how to configure (via command
> line args) one raw format drive that can store dirty bitmaps
> persistently in other qcow2 file? I may be missing something, but I
> thought QEMU couldn't do it, because Proxmox community wants this
> feature for a long time.
> 
> Patrik
> 
> 
> 
> 
> 
> 

Re: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi

[Kevin Wolf]

Hi Patrik,

Am 22.03.2021 um 09:57 hat Patrik Janoušek geschrieben:
> On 3/22/21 9:29 AM, Vladimir Sementsov-Ogievskiy wrote:
> > We do have external incremental backups, based on Qemu bitmap API. But
> > it depends of course on qcow2 persistent bitmaps feature.
>
> Yes, I know. And that's the problem. The point of my bachelor thesis is
> to implement a backup solution for the raw format.

the problem with this is that raw isn't really a format, it's more the
absence of a format. You just have the content of the virtual disk in a
file and that's it. This means not having any metadata (apart from the
metadata stored in the filesystem, of course).

As soon as you add metadata in some way (in your case, by referencing
additional metadata files in runtime options), it's not raw any more. If
you write to the raw image file without updating the metadata, the
metadata becomes inconsistent with the content. In other words, both
files form a single disk image together and can only be used together or
you're breaking them.

This in turn means that you have just invented a new image format. It's
a bit unconventional in that it's spread across multiple files, and that
some of the metadata that brings everything together is even in command
line options instead of a file, but you have to combine these components
in the same way every time you start the VM, so it really is a new image
format.

We have gone through such discussions a while ago because obviously "raw
with dirty bitmaps" was a request that came soon after we discussed
persistent dirty maps. But as we came to the conclusion that any
addition to raw would create another new image format specific to QEMU,
we decided that we can as well use qcow2 for this, which is already the
fully featured QEMU image format.

I hope this background helps a bit to explain the reactions you have
received so far.

Kevin

Re: [PATCH 2/2] qapi: implementation of the block-dirty-bitmap-dump command

[Kevin Wolf]

Am 20.03.2021 um 10:32 hat Patrik Janoušek geschrieben:
> Currently, dirty bitmaps are for internal use only and there is
> no support for accessing their content from third party-apps.
> This patch implements new command block-dirty-bitmap-dump, which
> returns content of the dirty bitmap encoded in base64. This is
> very useful especially in combination with a drive that uses raw
> format because third-party apps can easily use it to create
> incremental or differential backup.
> 
> Signed-off-by: Patrik Janoušek <pj@patrikjanousek.cz>

People have already pointed you to NBD to get the block dirty status
(and they are right), but I think nobody has explained yet why we
decided against a QMP command to return bitmaps.

The problem is that disk images can be huge, and this means that dirty
bitmaps describing them get pretty large, too. So your new QMP command
ends up reading many megabytes from disk and sending it to the socket.
While it's doing all of this I/O, QEMU is blocked and the guest may
appear to be hanging until it completes.

The client would also have to download the whole bitmap even if it is
only interested in specific blocks. (This may or may not be relevant for
you specific use case.)

With the NBD export, the client can query a specific block range and its
request is processed asynchronously in the background while the guest
and the QMP monitor are still available.

Kevin

Re: [PATCH 1/2] block/raw: added support of persistent dirty bitmaps

[Kevin Wolf]

Am 20.03.2021 um 10:32 hat Patrik Janoušek geschrieben:
> Current implementation of dirty bitmaps for raw format is very
> limited, because those bitmaps cannot be persistent. Basically it
> makes sense, because the raw format doesn't have space where could
> be dirty bitmap stored when QEMU is offline. This patch solves it
> by storing content of every dirty bitmap in separate file on the
> host filesystem.
> 
> However, this only solves one part of the problem. We also have to
> store information about the existence of the dirty bitmap. This is
> solved by adding custom options, that stores all required metadata
> about dirty bitmap (filename where is the bitmap stored on the
> host filesystem, granularity, persistence, etc.).
> 
> Signed-off-by: Patrik Janoušek <pj@patrikjanousek.cz>

I'm not sure if you're going to try to change your thesis to use qcow2
with an external data file, but in case you're still using this patch
for your thesis, let's imagine for a moment that we all agreed on adding
the functionality to raw.

After skimming over the patch, I see two major things that we would ask
to change:

1. You need to change BlockdevOptions to the QAPI schema in
   qapi/block-core.json, because if you don't add new options there,
   they won't be accessible with -blockdev and QMP blockdev-add.

   Among others, this means you would describe RawDirtyBitmapOpts in the
   schema and would get the structure and some helpers automatically
   generated, simplifying your code.

   Instead of processing QemuOpts manually, I would then probably try to
   use the QAPI visitors to get RawDirtyBitmapOpts from the input, which
   could potentially further simplify the code.

2. Instead of having a 'filename': 'str' option and working on the
   bitmap files with stdio.h functions (which block the guest instead of
   allowing asynchronous operation in the background), we would probably
   want something like 'file': 'BlockdevRef', so you get another
   BlockDriverState for each bitmap file that you access with the normal
   QEMU block layer I/O functions.

   The advantage is not only that this isn't blocking, but it also
   allows you to configure more details than just the filename (think
   cache mode, AIO mode, locking, etc.). In fact, it would even allow
   you to store the metadata not in a file, but in any place that can be
   accessed with a protocol supported by QEMU (like NBD, iscsi, ssh,
   whatever).

Kevin

Re: [PATCH 1/2] block/raw: added support of persistent dirty bitmaps

[Lubos Matejka]

Kdy si muzem cinknout k dalsimu vyvoji?

Odesláno z iPhonu

> 22. 3. 2021 v 12:37, Patrik Janoušek <pj@patrikjanousek.cz>:
> 
> 
>> On 3/22/21 12:18 PM, Vladimir Sementsov-Ogievskiy wrote:
>> 22.03.2021 13:46, Vladimir Sementsov-Ogievskiy wrote:
>>> 22.03.2021 13:18, Patrik Janoušek wrote:
>>>> On 3/22/21 9:41 AM, Vladimir Sementsov-Ogievskiy wrote:
>>>>> 20.03.2021 12:32, Patrik Janoušek wrote:
>>>>>> Current implementation of dirty bitmaps for raw format is very
>>>>>> limited, because those bitmaps cannot be persistent. Basically it
>>>>>> makes sense, because the raw format doesn't have space where could
>>>>>> be dirty bitmap stored when QEMU is offline. This patch solves it
>>>>>> by storing content of every dirty bitmap in separate file on the
>>>>>> host filesystem.
>>>>>> 
>>>>>> However, this only solves one part of the problem. We also have to
>>>>>> store information about the existence of the dirty bitmap. This is
>>>>>> solved by adding custom options, that stores all required metadata
>>>>>> about dirty bitmap (filename where is the bitmap stored on the
>>>>>> host filesystem, granularity, persistence, etc.).
>>>>>> 
>>>>>> Signed-off-by: Patrik Janoušek<pj@patrikjanousek.cz>
>>>>> 
>>>>> 
>>>>> Hmm. Did you considered other ways? Honestly, I don't see a reason for
>>>>> yet another storing format for bitmaps.
>>>>> 
>>>>> The task could be simply solved with existing features:
>>>>> 
>>>>> 1. We have extenal-data-file feature in qcow2 (read
>>>>> docs/interop/qcow2.txt). With this thing enabled, qcow2 file contains
>>>>> only metadata (persistent bitmaps for example) and data is stored in
>>>>> separate sequential raw file. I think you should start from it.
>>>> 
>>>> I didn't know about that feature. I'll look at it.
>>>> 
>>>> In case I use NBD to access the bitmap context and qcow2 as a solution
>>>> for persistent layer. Would the patch be acceptable? This is
>>>> significant
>>>> change to my solution and I don't have enought time for it at the
>>>> moment
>>>> (mainly due to other parts of my bachelor's thesis). I just want to
>>>> know
>>>> if this kind of feature is interesting to you and its implementation is
>>>> worth my time.
>>> 
>>> Honestly, at this point I think it doesn't. If existing features
>>> satisfy your use-case, no reason to increase complexity of file-posix
>>> driver and QAPI.
>>> 
>> 
>> It's unpleasant to say this, keeping in mind that that's your first
>> submission :(
>> 
>> I can still recommend in a connection with your bachelor's thesis to
>> look at the videos at kvm-forum youtube channel, searching for backup:
>> 
>>  
>> https://www.youtube.com/channel/UCRCSQmAOh7yzgheq-emy1xA/search?query=backup
>> 
>> You'll get a lot of information about current developments of external
>> backup API.
>> 
>> Also note, that there is (or there will be ?) libvirt Backup API,
>> which includes an API for external backup. I don't know the current
>> status of it, but if your project is based on libvirt, it's better to
>> use libvirt backup API instead of using qemu directly. About Libvirt
>> Backup API it's better to ask Eric Blake (adding him to CC).
> Unfortunately, my solution is based on Proxmox so I can't use libvirt's
> features. I know that a beta version of Proxmox Backup Server has been
> released and it would be much better to improve their solution, but they
> did it too late so I couldn't change assignment of my bachelor's thesis.
> 

Re: Fwd: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi

[Patrik Janoušek]

On 3/22/21 1:06 PM, Max Reitz wrote:
> On 22.03.21 12:27, Patrik Janoušek wrote:
>> On 3/22/21 11:48 AM, Max Reitz wrote:
>>> Hi,
>>>
>>> On 20.03.21 11:01, Patrik Janoušek wrote:
>>>> I'm sorry, but I forgot to add you to the cc, so I'm forwarding the
>>>> patch to you additionally. I don't want to spam the mailing list
>>>> unnecessarily.
>>>
>>> I think it’s better to still CC the list.  It’s so full of mail, one
>>> more won’t hurt. :)
>>>
>>> (Re-adding qemu-block and qemu-devel, because the discussion belongs
>>> on the list(s).)
>>>
>>>> -------- Forwarded Message --------
>>>> Subject:     [PATCH 0/2] block/raw: implemented persistent dirty
>>>> bitmap and ability to dump bitmap content via qapi
>>>> Date:     Sat, 20 Mar 2021 10:32:33 +0100
>>>> From:     Patrik Janoušek <pj@patrikjanousek.cz>
>>>> To:     qemu-devel@nongnu.org
>>>> CC:     Patrik Janoušek <pj@patrikjanousek.cz>, lmatejka@kiv.zcu.cz
>>>>
>>>>
>>>>
>>>> Currently, QEMU doesn't support persistent dirty bitmaps for raw
>>>> format
>>>> and also dirty bitmaps are for internal use only, and cannot be
>>>> accessed
>>>> using third-party applications. These facts are very limiting
>>>> in case someone would like to develop their own backup tool becaouse
>>>> without access to the dirty bitmap it would be possible to implement
>>>> only full backups. And without persistent dirty bitmaps, it wouldn't
>>>> be possible to keep track of changed data after QEMU is restarted. And
>>>> this is exactly what I do as a part of my bachelor thesis. I've
>>>> developed a tool that is able to create incremental backups of drives
>>>> in raw format that are LVM volumes (ability to create snapshot is
>>>> required).
>>>
>>> Similarly to what Vladimir has said already, the thing is that
>>> conceptually I can see no difference between having a raw image with
>>> the bitmaps stored in some other file, i.e.:
>>>
>>>    { "driver": "raw",
>>>      "dirty-bitmaps": [ {
>>>        "filename": "sdc1.bitmap",
>>>        "persistent": true
>>>      } ],
>>>      "file": {
>>>        "driver": "file",
>>>        "filename": "/dev/sdc1"
>>>      } }
>>>
>>> And having a qcow2 image with the raw data stored in some other file,
>>> i.e.:
>>>
>>>    { "driver": "qcow2",
>>>      "file": {
>>>        "driver": "file",
>>>        "filename": "sdc1.metadata"
>>>      },
>>>      "data-file": {
>>>        "driver": "file",
>>>        "filename": "/dev/sdc1"
>>>      } }
>>>
>>> (Where sdc1.metadata is a qcow2 file created with
>>> “data-file=/dev/sdc1,data-file-raw=on”.)
>>>
>>> To use persistent bitmaps with raw images, you need to add metadata
>>> (namely, the bitmaps).  Why not store that metadata in a qcow2 file?
>>>
>>> Max
>>
>> So if I understand it correctly. I can configure dirty bitmaps in the
>> latest version of QEMU to be persistently stored in some other file.
>> Because even Proxmox Backup Server can't perform an incremental backup
>> after restarting QEMU, and that means something to me. I think they
>> would implement it if it was that simple.
>>
>> Could you please send me simple example on how to configure (via command
>> line args) one raw format drive that can store dirty bitmaps
>> persistently in other qcow2 file? I may be missing something, but I
>> thought QEMU couldn't do it, because Proxmox community wants this
>> feature for a long time.
>
> One trouble is that if you use qemu-img create to create the qcow2
> image, it will always create an empty image, and so if use pass
> data_file to it, it will empty the existing raw image:
>
> $ cp ~/tmp/arch.iso raw.img # Just some Arch Linux ISO
>
> $ qemu-img create \
>     -f qcow2 \
>     -o data_file=raw.img,data_file_raw=on,preallocation=metadata \
>     metadata.qcow2 \
>     $(stat -c '%s' raw.img)
> Formatting 'metadata.qcow2', fmt=qcow2 cluster_size=65536
> preallocation=metadata compression_type=zlib size=687865856
> data_file=raw.img data_file_raw=on lazy_refcounts=off refcount_bits=16
>
> (If you check raw.img at this point, you’ll find that it’s empty, so
> you need to copy it from the source again:)
>
> $ cp ~/tmp/arch.iso raw.img
>
> Now if you use metadata.qcow2, the image data will actually all be
> stored in raw.img.
>
>
> To get around the “creating metadata.qcow2 clears raw.img” problem,
> you can either create a temporary empty image of the same size as
> raw.img that you pass to qemu-img create, and then you use qemu-img
> amend to change the data-file pointer (which will not overwrite the
> new data-file’s contents):
>
> $ qemu-img create -f raw tmp.raw $(stat -c '%s' raw.img)
>
> $ qemu-img create \
>     -f qcow2 \
>     -o data_file=tmp.img,data_file_raw=on,preallocation=metadata \
>     metadata.qcow2 \
>     $(stat -c '%s' raw.img)
> Formatting 'metadata.qcow2', fmt=qcow2 cluster_size=65536
> preallocation=metadata compression_type=zlib size=687865856
> data_file=tmp.img data_file_raw=on lazy_refcounts=off refcount_bits=16
>
> $ qemu-img amend -o data_file=raw.img metadata.qcow2
>
> $ rm tmp.img
>
>
> Or you use the blockdev-create job to create the qcow2 image (because
> contrary to qemu-img create, that will not clear the data file):
>
> $ touch metadata.qcow2
>
> (Note that in the following QMP communication, what I sent and what
> qemu replies is mixed.  Everything that begins with '{ "execute"' is
> from me, everything else from qemu.  The number 687865856 is the size
> of raw.img in bytes.)
>
> $ qemu-system-x86_64 -qmp stdio \
>     -blockdev \
>     '{ "node-name": "metadata-file",
>        "driver": "file",
>        "filename": "metadata.qcow2" }' \
>     -blockdev \
>     '{ "node-name": "data-file",
>        "driver": "file",
>        "filename": "raw.img" }'
> {"QMP": {"version": {"qemu": {"micro": 0, "minor": 1, "major": 5},
> "package": "qemu-5.1.0-9.fc33"}, "capabilities": ["oob"]}}
>
> { "execute": "qmp_capabilities" }
> {"return": {}}
>
> { "execute": "blockdev-create",
>   "arguments": {
>     "job-id": "create",
>     "options": {
>       "driver": "qcow2",
>       "file": "metadata-file",
>       "data-file": "data-file",
>       "data-file-raw": true,
>       "preallocation": "metadata",
>       "size": 687865856
>     }
>   }
> }
> {"timestamp": {"seconds": 1616414002, "microseconds": 836899},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id":
> "create"}}
> {"timestamp": {"seconds": 1616414002, "microseconds": 837076},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id":
> "create"}}
> {"return": {}}
> {"timestamp": {"seconds": 1616414002, "microseconds": 870997},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id":
> "create"}}
> {"timestamp": {"seconds": 1616414002, "microseconds": 871099},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id":
> "create"}}
> {"timestamp": {"seconds": 1616414002, "microseconds": 871185},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id":
> "create"}}
>
> { "execute": "job-dismiss", "arguments": { "id": "create" } }
> {"timestamp": {"seconds": 1616414022, "microseconds": 202880},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "create"}}
> {"return": {}}
>
> { "execute": "quit" }
> {"return": {}}
> {"timestamp": {"seconds": 1616414028, "microseconds": 56457}, "event":
> "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
>
>
> In any case, in the end you get a metadata.qcow2 that holds the
> metadata and points to raw.img for its data.  So you can use it like
> this:
>
> $ qemu-system-x86_64 -enable-kvm -m 512 \
>     -blockdev \
>     '{ "node-name": "node0",
>        "driver": "qcow2",
>        "file": {
>          "driver": "file",
>          "filename": "metadata.qcow2"
>        } }' \
>     -device ide-cd,drive=node0 \
>     -qmp stdio
> {"QMP": {"version": {"qemu": {"micro": 0, "minor": 1, "major": 5},
> "package": "qemu-5.1.0-9.fc33"}, "capabilities": ["oob"]}}
>
> { "execute": "qmp_capabilities" }
> {"return": {}}
>
> { "execute": "block-dirty-bitmap-add",
>   "arguments": {
>     "node": "node0",
>     "name": "bmap0",
>     "persistent": true
>   }
> }
> {"return": {}}
>
> { "execute": "quit" }
> {"return": {}}
> {"timestamp": {"seconds": 1616414627, "microseconds": 928250},
> "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
>
> $ qemu-img info metadata.qcow2
> image: metadata.qcow2
> file format: qcow2
> virtual size: 656 MiB (687865856 bytes)
> disk size: 452 KiB
> cluster_size: 65536
> Format specific information:
>     compat: 1.1
>     compression type: zlib
>     lazy refcounts: false
>     bitmaps:
>         [0]:
>             flags:
>                 [0]: auto
>             name: bmap0
>             granularity: 65536
>     refcount bits: 16
>     data file: raw.img
>     data file raw: true
>     corrupt: false
>
> So the bitmap is now in metadata.qcow2, and as the disk size
> indicates, all the data is still in raw.img.
>
>
> I hope the above helps you.  Sorry if it’s confusing, especially the
> first part where I’m like “The obvious way to create metadata.qcow2
> will delete your data, so here are two alternatives that are weird but
> do what you want.”
>
> Max

Thank you for exhausting answer. I would also like to thank others for
their comments. I will try to rethink the developed solution and
consider its modifications. Maybe in the future I'll come up with
something you might like.

Regards,
Patrik