* [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi @ 2021-03-20 9:32 Patrik Janoušek 2021-03-20 9:32 ` [PATCH 1/2] block/raw: added support of persistent dirty bitmaps Patrik Janoušek ` (3 more replies) 0 siblings, 4 replies; 20+ messages in thread From: Patrik Janoušek @ 2021-03-20 9:32 UTC (permalink / raw) To: qemu-devel; +Cc: Patrik Janoušek, lmatejka Currently, QEMU doesn't support persistent dirty bitmaps for raw format and also dirty bitmaps are for internal use only, and cannot be accessed using third-party applications. These facts are very limiting in case someone would like to develop their own backup tool becaouse without access to the dirty bitmap it would be possible to implement only full backups. And without persistent dirty bitmaps, it wouldn't be possible to keep track of changed data after QEMU is restarted. And this is exactly what I do as a part of my bachelor thesis. I've developed a tool that is able to create incremental backups of drives in raw format that are LVM volumes (ability to create snapshot is required). Please keep in mind that this is my first submission to such a large project and also the first time when I send patch over the email. So I hope I did it correctly. Patrik Janoušek (2): block/raw: added support of persistent dirty bitmaps qapi: implementation of the block-dirty-bitmap-dump command block/meson.build | 1 + block/monitor/bitmap-qmp-cmds.c | 61 ++++++++ block/raw-format-bitmap.c | 163 ++++++++++++++++++++ block/raw-format.c | 256 ++++++++++++++++++++++++++++++-- block/raw-format.h | 50 +++++++ qapi/block-core.json | 64 +++++++- 6 files changed, 583 insertions(+), 12 deletions(-) create mode 100644 block/raw-format-bitmap.c create mode 100644 block/raw-format.h -- 2.31.0 ^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH 1/2] block/raw: added support of persistent dirty bitmaps 2021-03-20 9:32 [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi Patrik Janoušek @ 2021-03-20 9:32 ` Patrik Janoušek 2021-03-22 8:41 ` Vladimir Sementsov-Ogievskiy 2021-03-22 15:43 ` Kevin Wolf 2021-03-20 9:32 ` [PATCH 2/2] qapi: implementation of the block-dirty-bitmap-dump command Patrik Janoušek ` (2 subsequent siblings) 3 siblings, 2 replies; 20+ messages in thread From: Patrik Janoušek @ 2021-03-20 9:32 UTC (permalink / raw) To: qemu-devel; +Cc: Patrik Janoušek, lmatejka Current implementation of dirty bitmaps for raw format is very limited, because those bitmaps cannot be persistent. Basically it makes sense, because the raw format doesn't have space where could be dirty bitmap stored when QEMU is offline. This patch solves it by storing content of every dirty bitmap in separate file on the host filesystem. However, this only solves one part of the problem. We also have to store information about the existence of the dirty bitmap. This is solved by adding custom options, that stores all required metadata about dirty bitmap (filename where is the bitmap stored on the host filesystem, granularity, persistence, etc.). Signed-off-by: Patrik Janoušek <pj@patrikjanousek.cz> --- block/meson.build | 1 + block/raw-format-bitmap.c | 163 ++++++++++++++++++++++++ block/raw-format.c | 256 ++++++++++++++++++++++++++++++++++++-- block/raw-format.h | 50 ++++++++ 4 files changed, 459 insertions(+), 11 deletions(-) create mode 100644 block/raw-format-bitmap.c create mode 100644 block/raw-format.h diff --git a/block/meson.build b/block/meson.build index 5dcc1e5cce..f163095574 100644 --- a/block/meson.build +++ b/block/meson.build @@ -30,6 +30,7 @@ block_ss.add(files( 'qcow2.c', 'quorum.c', 'raw-format.c', + 'raw-format-bitmap.c', 'snapshot.c', 'throttle-groups.c', 'throttle.c', diff --git a/block/raw-format-bitmap.c b/block/raw-format-bitmap.c new file mode 100644 index 0000000000..f622f0a700 --- /dev/null +++ b/block/raw-format-bitmap.c @@ -0,0 +1,163 @@ +#include <stdio.h> + +#include "qemu/osdep.h" +#include "block/block_int.h" + +#include "raw-format.h" +#include "qapi/error.h" + +bool coroutine_fn raw_co_can_store_new_dirty_bitmap(BlockDriverState *bs, + const char *name, + uint32_t granularity, + Error **errp) +{ + BDRVRawState *s = bs->opaque; + + RawDirtyBitmapOpts *opts; + BdrvDirtyBitmap *bm; + FILE *fp; + bool file_exists; + for (int i = 0; i < s->rdbol.n; i++) { + opts = &s->rdbol.opts[i]; + bm = bdrv_find_dirty_bitmap(bs, name); + if (!bm) { + error_setg(errp, "Dirty bitmap %s does not exists.", name); + return false; + } + + if (!opts->has_filename) { + error_setg(errp, "Dirty bitmap does not have filename defined."); + return false; + } + + if (!bdrv_dirty_bitmap_get_persistence(bm)) continue; + if (strcmp(opts->name, bdrv_dirty_bitmap_name(bm)) != 0) + continue; + + file_exists = false; + fp = fopen(opts->filename, "r"); + if (fp) { + file_exists = true; + fclose(false); + } + + fp = fopen(opts->filename, "a"); + if (!fp) { + error_setg(errp, "Unable to open file %s for read and write", + opts->filename); + return false; + } else { + fclose(fp); + if (!file_exists) unlink(opts->filename); + } + } + + return true; +} + +bool raw_supports_persistent_dirty_bitmap(BlockDriverState *bs) +{ + return true; +} + +int coroutine_fn raw_co_remove_persistent_dirty_bitmap(BlockDriverState *bs, + const char *name, + Error **errp) +{ + BDRVRawState *s = bs->opaque; + + RawDirtyBitmapOpts *opts; + BdrvDirtyBitmap *bm; + for (int i = 0; i < s->rdbol.n; i++) { + opts = &s->rdbol.opts[i]; + bm = bdrv_find_dirty_bitmap(bs, name); + if (!bm) { + error_setg(errp, "Dirty bitmap %s does not exists.", name); + return false; + } + + if (!opts->has_filename) { + error_setg(errp, "Dirty bitmap does not have filename defined."); + return false; + } + + if (!bdrv_dirty_bitmap_get_persistence(bm)) continue; + if (strcmp(opts->name, bdrv_dirty_bitmap_name(bm)) != 0) + continue; + + return unlink(opts->filename); + } + + return 0; +} + +static void load_bitmap(BdrvDirtyBitmap *bm, FILE *fpbm) { + uint64_t bm_size = bdrv_dirty_bitmap_size(bm); + uint64_t tb_size = bdrv_dirty_bitmap_serialization_size(bm, 0, bm_size); + uint8_t *buf = g_malloc(tb_size); + size_t n = fread(buf, 1, tb_size, fpbm); + + assert(n == tb_size); + + bdrv_dirty_bitmap_deserialize_part(bm, buf, 0, bm_size, false); + bdrv_dirty_bitmap_deserialize_finish(bm); + + g_free(buf); +} + +void raw_load_persistent_dirty_bitmap(BlockDriverState *bs, FILE *fp, + uint32_t granularity, char* name, + bool persistent, bool disabled, + Error **errp) +{ + BdrvDirtyBitmap *bm; + + bm = bdrv_create_dirty_bitmap(bs, granularity, name, errp); + if (bm) { + bdrv_dirty_bitmap_set_persistence(bm, persistent); + + if (disabled) + bdrv_disable_dirty_bitmap(bm); + } + + if (fp) { + load_bitmap(bm, fp); + } +} + +static void store_bitmap(BdrvDirtyBitmap *bm, FILE *fpbm) { + uint64_t bm_size = bdrv_dirty_bitmap_size(bm); + uint64_t tb_size = bdrv_dirty_bitmap_serialization_size(bm, 0, bm_size); + uint8_t *buf = g_malloc(tb_size); + + bdrv_dirty_bitmap_serialize_part(bm, buf, 0, bm_size); + + fwrite(buf, 1, tb_size, fpbm); + + g_free(buf); +} + +void raw_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp) +{ + BDRVRawState *s = bs->opaque; + + RawDirtyBitmapOpts *opts; + BdrvDirtyBitmap *bm; + for (int i = 0; i < s->rdbol.n; i++) { + opts = &s->rdbol.opts[i]; + + FOR_EACH_DIRTY_BITMAP(bs, bm) { + if (!bdrv_dirty_bitmap_get_persistence(bm)) continue; + assert(opts->has_filename); + + if (strcmp(opts->filename, bdrv_dirty_bitmap_name(bm)) != 0) + continue; + + FILE *fp = fopen(opts->filename, "w"); + if(fp) { + store_bitmap(bm, fp); + fclose(fp); + } + } + } +} diff --git a/block/raw-format.c b/block/raw-format.c index 42ec50802b..901f7a9c33 100644 --- a/block/raw-format.c +++ b/block/raw-format.c @@ -28,15 +28,13 @@ #include "qemu/osdep.h" #include "block/block_int.h" +#include "block/qdict.h" #include "qapi/error.h" +#include "qapi/qmp/qlist.h" #include "qemu/module.h" #include "qemu/option.h" -typedef struct BDRVRawState { - uint64_t offset; - uint64_t size; - bool has_size; -} BDRVRawState; +#include "raw-format.h" static const char *const mutable_opts[] = { "offset", "size", NULL }; @@ -58,6 +56,40 @@ static QemuOptsList raw_runtime_opts = { }, }; +static QemuOptsList raw_dirty_bitmap_opts = { + .name = "raw-dirty-bitmap-opts", + .head = QTAILQ_HEAD_INITIALIZER(raw_dirty_bitmap_opts.head), + .desc = { + { + .name = "filename", + .type = QEMU_OPT_STRING, + .help = "path to the persistent dirty bitmap file", + }, + { + .name = "name", + .type = QEMU_OPT_STRING, + .help = "name of the dirty bitmap", + }, + { + .name = "granularity", + .type = QEMU_OPT_NUMBER, + .help = "granularity of the dirty bitmap", + }, + { + .name = "persistent", + .type = QEMU_OPT_BOOL, + .help = "indicates whether the dirty bitmap is persistent" + "(requires filename)", + }, + { + .name = "disabled", + .type = QEMU_OPT_BOOL, + .help = "indicates whether the dirty bitmap is disabled", + }, + { /* end of list */ } + }, +}; + static QemuOptsList raw_create_opts = { .name = "raw-create-opts", .head = QTAILQ_HEAD_INITIALIZER(raw_create_opts.head), @@ -71,8 +103,144 @@ static QemuOptsList raw_create_opts = { } }; +static int raw_read_dirty_bitmap_options(QDict *options, + bool *has_filename, char **filename, + bool *has_name, char **name, + bool *has_granularity, + uint32_t *granularity, + bool *has_persistent, bool *persistent, + bool *has_disabled, bool *disabled, + Error **errp) +{ + QemuOpts *opts = NULL; + int ret; + + opts = qemu_opts_create(&raw_dirty_bitmap_opts, NULL, 0, &error_abort); + if (!qemu_opts_absorb_qdict(opts, options, errp)) { + ret = -EINVAL; + goto end; + } + + *has_filename = qemu_opt_find(opts, "filename"); + *filename = g_strdup(qemu_opt_get(opts, "filename")); + *has_name = qemu_opt_find(opts, "name"); + *name = g_strdup(qemu_opt_get(opts, "name")); + *has_granularity = qemu_opt_find(opts, "granularity"); + *granularity = qemu_opt_get_number(opts, "granularity", 65536); + *has_persistent = qemu_opt_find(opts, "persistent"); + *persistent = qemu_opt_get_bool(opts, "persistent", false); + *has_disabled = qemu_opt_find(opts, "disabled"); + *disabled = qemu_opt_get_bool(opts, "disabled", false); + + ret = 0; + end: + qemu_opts_del(opts); + return ret; +} + +static int raw_read_dirty_bitmaps_options(QDict *options, + RawDirtyBitmapOptsList *rdbol, + Error **errp) +{ + int ret; + QDict *db_options = NULL; + + QObject *qo; + char static_prefix[] = "dirty-bitmaps."; + uint static_prefix_len = strlen(static_prefix); + uint key_max_len = static_prefix_len; + char *key = g_malloc0(key_max_len + 1); + int i = 0, j; + char ibuff[16]; + size_t nkey, nname; + *rdbol = (RawDirtyBitmapOptsList){ + .n = 0, + .opts = NULL, + }; + while (true) { + nkey = snprintf(ibuff, sizeof(ibuff), "%d.", i); + + for (j = 0; raw_dirty_bitmap_opts.desc[j].name != NULL; j++) { + nname = strlen(raw_dirty_bitmap_opts.desc[j].name); + if (key_max_len < static_prefix_len + nkey + nname) { + key_max_len = static_prefix_len + nkey + nname; + key = g_realloc(key, key_max_len + 1); + } + + strncpy(key, static_prefix, key_max_len); + strncpy(key + static_prefix_len, ibuff, + key_max_len - static_prefix_len); + strncpy(key + static_prefix_len + nkey, + raw_dirty_bitmap_opts.desc[j].name, + key_max_len - static_prefix_len - nkey); + *(key + static_prefix_len + nkey + nname) = '\0'; + + qo = qdict_get(options, key); + if (qo != NULL) { + qobject_ref(qo); + qdict_del(options, key); + + if (db_options == NULL) db_options = qdict_new(); + qdict_put_obj(db_options, raw_dirty_bitmap_opts.desc[j].name, + qo); + } + } + + if (db_options != NULL && qdict_size(db_options) != 0) { + rdbol->n++; + if (rdbol->n == 1) + rdbol->opts = g_malloc0(sizeof(RawDirtyBitmapOpts) * rdbol->n); + else + rdbol->opts = g_realloc(rdbol->opts, + sizeof(RawDirtyBitmapOpts) * rdbol->n); + + bool has_filename; + char* filename; + bool has_name; + char *name; + bool has_granularity; + uint32_t granularity; + bool has_persistent; + bool persistent; + bool has_disabled; + bool disabled; + ret = raw_read_dirty_bitmap_options(db_options, + &has_filename, &filename, + &has_name, &name, + &has_granularity, &granularity, + &has_persistent, &persistent, + &has_disabled, &disabled, + errp); + qobject_unref(db_options); + if (ret) { + goto end; + } + + rdbol->opts[rdbol->n - 1] = (RawDirtyBitmapOpts) { + .has_filename = has_filename, + .filename = filename, + .has_name = has_name || has_filename, + .name = has_name ? name : filename, + .has_granularity = has_granularity, + .granularity = granularity, + .has_persistent = has_persistent, + .persistent = persistent, + .has_disabled = has_disabled, + .disabled = disabled, + }; + } else { + break; + } + } + + ret = 0; + end: + return ret; +} + static int raw_read_options(QDict *options, uint64_t *offset, bool *has_size, - uint64_t *size, Error **errp) + uint64_t *size, RawDirtyBitmapOptsList *rdbol, + Error **errp) { QemuOpts *opts = NULL; int ret; @@ -87,15 +255,35 @@ static int raw_read_options(QDict *options, uint64_t *offset, bool *has_size, *has_size = qemu_opt_find(opts, "size"); *size = qemu_opt_get_size(opts, "size", 0); + ret = raw_read_dirty_bitmaps_options(options, rdbol, errp); + if (ret < 0) { + ret = -EINVAL; + goto end; + } + ret = 0; end: qemu_opts_del(opts); return ret; } +static void free_RawDirtyBitmapOptsList(RawDirtyBitmapOptsList *rdbol) { + if (rdbol->opts != NULL) { + for (int i = 0; i < rdbol->n; i++) { + if (rdbol->opts[i].filename != NULL) + g_free(rdbol->opts[i].filename); + if (rdbol->opts[i].name != NULL) + g_free(rdbol->opts[i].name); + } + + g_free(rdbol->opts); + rdbol->opts = NULL; + } +} + static int raw_apply_options(BlockDriverState *bs, BDRVRawState *s, uint64_t offset, bool has_size, uint64_t size, - Error **errp) + RawDirtyBitmapOptsList rdbol, Error **errp) { int64_t real_size = 0; @@ -132,6 +320,29 @@ static int raw_apply_options(BlockDriverState *bs, BDRVRawState *s, s->offset = offset; s->has_size = has_size; s->size = has_size ? size : real_size - offset; + s->rdbol = rdbol; + + RawDirtyBitmapOpts *opts; + for (int i = 0; i < rdbol.n; i++) { + opts = &rdbol.opts[i]; + + if (opts->has_persistent && opts->persistent && !opts->has_filename) { + error_setg(errp, "Filename is required for persistent bitmap"); + return -EINVAL; + } + + if (opts->has_filename) { + FILE *fp = fopen(opts->filename, "r"); + + raw_load_persistent_dirty_bitmap(bs, fp, opts->granularity, + opts->name, opts->persistent, + opts->disabled, errp); + + if (fp) { + fclose(fp); + } + } + } return 0; } @@ -141,6 +352,7 @@ static int raw_reopen_prepare(BDRVReopenState *reopen_state, { bool has_size; uint64_t offset, size; + RawDirtyBitmapOptsList rdbol = {0}; int ret; assert(reopen_state != NULL); @@ -149,13 +361,13 @@ static int raw_reopen_prepare(BDRVReopenState *reopen_state, reopen_state->opaque = g_new0(BDRVRawState, 1); ret = raw_read_options(reopen_state->options, &offset, &has_size, &size, - errp); + &rdbol, errp); if (ret < 0) { return ret; } ret = raw_apply_options(reopen_state->bs, reopen_state->opaque, - offset, has_size, size, errp); + offset, has_size, size, rdbol, errp); if (ret < 0) { return ret; } @@ -170,12 +382,14 @@ static void raw_reopen_commit(BDRVReopenState *state) memcpy(s, new_s, sizeof(BDRVRawState)); + free_RawDirtyBitmapOptsList(&((BDRVRawState *) state->opaque)->rdbol); g_free(state->opaque); state->opaque = NULL; } static void raw_reopen_abort(BDRVReopenState *state) { + free_RawDirtyBitmapOptsList(&((BDRVRawState *) state->opaque)->rdbol); g_free(state->opaque); state->opaque = NULL; } @@ -438,10 +652,11 @@ static int raw_open(BlockDriverState *bs, QDict *options, int flags, BDRVRawState *s = bs->opaque; bool has_size; uint64_t offset, size; + RawDirtyBitmapOptsList rdbol = {0}; BdrvChildRole file_role; int ret; - ret = raw_read_options(options, &offset, &has_size, &size, errp); + ret = raw_read_options(options, &offset, &has_size, &size, &rdbol, errp); if (ret < 0) { return ret; } @@ -483,7 +698,7 @@ static int raw_open(BlockDriverState *bs, QDict *options, int flags, bs->file->bs->filename); } - ret = raw_apply_options(bs, s, offset, has_size, size, errp); + ret = raw_apply_options(bs, s, offset, has_size, size, rdbol, errp); if (ret < 0) { return ret; } @@ -496,6 +711,19 @@ static int raw_open(BlockDriverState *bs, QDict *options, int flags, return 0; } +static void raw_close(BlockDriverState *bs) +{ + Error *local_err = NULL; + + raw_store_persistent_dirty_bitmaps(bs, &local_err); + + if (local_err != NULL) { + error_reportf_err(local_err, "Lost persistent bitmaps during " + "inactivation of node '%s': ", + bdrv_get_device_or_node_name(bs)); + } +} + static int raw_probe(const uint8_t *buf, int buf_size, const char *filename) { /* smallest possible positive score so that raw is used if and only if no @@ -583,6 +811,7 @@ BlockDriver bdrv_raw = { .bdrv_reopen_commit = &raw_reopen_commit, .bdrv_reopen_abort = &raw_reopen_abort, .bdrv_open = &raw_open, + .bdrv_close = &raw_close, .bdrv_child_perm = bdrv_default_perms, .bdrv_co_create_opts = &raw_co_create_opts, .bdrv_co_preadv = &raw_co_preadv, @@ -608,6 +837,11 @@ BlockDriver bdrv_raw = { .bdrv_has_zero_init = &raw_has_zero_init, .strong_runtime_opts = raw_strong_runtime_opts, .mutable_opts = mutable_opts, + .bdrv_co_can_store_new_dirty_bitmap = raw_co_can_store_new_dirty_bitmap, + .bdrv_supports_persistent_dirty_bitmap = + raw_supports_persistent_dirty_bitmap, + .bdrv_co_remove_persistent_dirty_bitmap = + raw_co_remove_persistent_dirty_bitmap, }; static void bdrv_raw_init(void) diff --git a/block/raw-format.h b/block/raw-format.h new file mode 100644 index 0000000000..1ac765255e --- /dev/null +++ b/block/raw-format.h @@ -0,0 +1,50 @@ +#ifndef BLOCK_RAW_FORMAT_H +#define BLOCK_RAW_FORMAT_H + +#include "qemu/coroutine.h" +#include "qemu/osdep.h" + +typedef struct RawDirtyBitmapOpts { + bool has_filename; + char *filename; + bool has_name; + char *name; + bool has_granularity; + uint32_t granularity; + bool has_persistent; + bool persistent; + bool has_disabled; + bool disabled; +} RawDirtyBitmapOpts; + +typedef struct RawDirtyBitmapOptsList { + size_t n; + RawDirtyBitmapOpts *opts; +} RawDirtyBitmapOptsList; + +typedef struct BDRVRawState { + uint64_t offset; + uint64_t size; + bool has_size; + RawDirtyBitmapOptsList rdbol; +} BDRVRawState; + +bool raw_co_can_store_new_dirty_bitmap(BlockDriverState *bs, + const char *name, + uint32_t granularity, + Error **errp); + +bool raw_supports_persistent_dirty_bitmap(BlockDriverState *bs); + +int raw_co_remove_persistent_dirty_bitmap(BlockDriverState *bs, + const char *name, + Error **errp); + +void raw_load_persistent_dirty_bitmap(BlockDriverState *bs, FILE *fp, + uint32_t granularity, char* name, + bool persistent, bool disabled, + Error **errp); + +void raw_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp); + +#endif -- 2.31.0 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH 1/2] block/raw: added support of persistent dirty bitmaps 2021-03-20 9:32 ` [PATCH 1/2] block/raw: added support of persistent dirty bitmaps Patrik Janoušek @ 2021-03-22 8:41 ` Vladimir Sementsov-Ogievskiy 2021-03-22 10:18 ` Patrik Janoušek 2021-03-22 15:43 ` Kevin Wolf 1 sibling, 1 reply; 20+ messages in thread From: Vladimir Sementsov-Ogievskiy @ 2021-03-22 8:41 UTC (permalink / raw) To: Patrik Janoušek, qemu-devel; +Cc: lmatejka 20.03.2021 12:32, Patrik Janoušek wrote: > Current implementation of dirty bitmaps for raw format is very > limited, because those bitmaps cannot be persistent. Basically it > makes sense, because the raw format doesn't have space where could > be dirty bitmap stored when QEMU is offline. This patch solves it > by storing content of every dirty bitmap in separate file on the > host filesystem. > > However, this only solves one part of the problem. We also have to > store information about the existence of the dirty bitmap. This is > solved by adding custom options, that stores all required metadata > about dirty bitmap (filename where is the bitmap stored on the > host filesystem, granularity, persistence, etc.). > > Signed-off-by: Patrik Janoušek<pj@patrikjanousek.cz> Hmm. Did you considered other ways? Honestly, I don't see a reason for yet another storing format for bitmaps. The task could be simply solved with existing features: 1. We have extenal-data-file feature in qcow2 (read docs/interop/qcow2.txt). With this thing enabled, qcow2 file contains only metadata (persistent bitmaps for example) and data is stored in separate sequential raw file. I think you should start from it. 2. If for some reason [1] doesn't work for you, you can anyway use an empty qcow2 file to store bitmaps instead of inventing and implementing new format of bitmaps storing. (Same as your approach, you'll have a simple raw node, and additional options will say "load bitmaps from this qcow2 file". But for such options we'll need good reasons why [1] isn't enough. -- Best regards, Vladimir ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/2] block/raw: added support of persistent dirty bitmaps 2021-03-22 8:41 ` Vladimir Sementsov-Ogievskiy @ 2021-03-22 10:18 ` Patrik Janoušek 2021-03-22 10:46 ` Vladimir Sementsov-Ogievskiy 0 siblings, 1 reply; 20+ messages in thread From: Patrik Janoušek @ 2021-03-22 10:18 UTC (permalink / raw) To: Vladimir Sementsov-Ogievskiy, qemu-devel; +Cc: lmatejka On 3/22/21 9:41 AM, Vladimir Sementsov-Ogievskiy wrote: > 20.03.2021 12:32, Patrik Janoušek wrote: >> Current implementation of dirty bitmaps for raw format is very >> limited, because those bitmaps cannot be persistent. Basically it >> makes sense, because the raw format doesn't have space where could >> be dirty bitmap stored when QEMU is offline. This patch solves it >> by storing content of every dirty bitmap in separate file on the >> host filesystem. >> >> However, this only solves one part of the problem. We also have to >> store information about the existence of the dirty bitmap. This is >> solved by adding custom options, that stores all required metadata >> about dirty bitmap (filename where is the bitmap stored on the >> host filesystem, granularity, persistence, etc.). >> >> Signed-off-by: Patrik Janoušek<pj@patrikjanousek.cz> > > > Hmm. Did you considered other ways? Honestly, I don't see a reason for > yet another storing format for bitmaps. > > The task could be simply solved with existing features: > > 1. We have extenal-data-file feature in qcow2 (read > docs/interop/qcow2.txt). With this thing enabled, qcow2 file contains > only metadata (persistent bitmaps for example) and data is stored in > separate sequential raw file. I think you should start from it. I didn't know about that feature. I'll look at it. In case I use NBD to access the bitmap context and qcow2 as a solution for persistent layer. Would the patch be acceptable? This is significant change to my solution and I don't have enought time for it at the moment (mainly due to other parts of my bachelor's thesis). I just want to know if this kind of feature is interesting to you and its implementation is worth my time. > > 2. If for some reason [1] doesn't work for you, you can anyway use an > empty qcow2 file to store bitmaps instead of inventing and > implementing new format of bitmaps storing. (Same as your approach, > you'll have a simple raw node, and additional options will say "load > bitmaps from this qcow2 file". But for such options we'll need good > reasons why [1] isn't enough. > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/2] block/raw: added support of persistent dirty bitmaps 2021-03-22 10:18 ` Patrik Janoušek @ 2021-03-22 10:46 ` Vladimir Sementsov-Ogievskiy 2021-03-22 11:18 ` Vladimir Sementsov-Ogievskiy 0 siblings, 1 reply; 20+ messages in thread From: Vladimir Sementsov-Ogievskiy @ 2021-03-22 10:46 UTC (permalink / raw) To: Patrik Janoušek, qemu-devel; +Cc: lmatejka 22.03.2021 13:18, Patrik Janoušek wrote: > On 3/22/21 9:41 AM, Vladimir Sementsov-Ogievskiy wrote: >> 20.03.2021 12:32, Patrik Janoušek wrote: >>> Current implementation of dirty bitmaps for raw format is very >>> limited, because those bitmaps cannot be persistent. Basically it >>> makes sense, because the raw format doesn't have space where could >>> be dirty bitmap stored when QEMU is offline. This patch solves it >>> by storing content of every dirty bitmap in separate file on the >>> host filesystem. >>> >>> However, this only solves one part of the problem. We also have to >>> store information about the existence of the dirty bitmap. This is >>> solved by adding custom options, that stores all required metadata >>> about dirty bitmap (filename where is the bitmap stored on the >>> host filesystem, granularity, persistence, etc.). >>> >>> Signed-off-by: Patrik Janoušek<pj@patrikjanousek.cz> >> >> >> Hmm. Did you considered other ways? Honestly, I don't see a reason for >> yet another storing format for bitmaps. >> >> The task could be simply solved with existing features: >> >> 1. We have extenal-data-file feature in qcow2 (read >> docs/interop/qcow2.txt). With this thing enabled, qcow2 file contains >> only metadata (persistent bitmaps for example) and data is stored in >> separate sequential raw file. I think you should start from it. > > I didn't know about that feature. I'll look at it. > > In case I use NBD to access the bitmap context and qcow2 as a solution > for persistent layer. Would the patch be acceptable? This is significant > change to my solution and I don't have enought time for it at the moment > (mainly due to other parts of my bachelor's thesis). I just want to know > if this kind of feature is interesting to you and its implementation is > worth my time. Honestly, at this point I think it doesn't. If existing features satisfy your use-case, no reason to increase complexity of file-posix driver and QAPI. And if existing features are not enough, we should start from understanding why they are not enough, and probably true way would be improving the existing features instead of implementing from scratch. > >> >> 2. If for some reason [1] doesn't work for you, you can anyway use an >> empty qcow2 file to store bitmaps instead of inventing and >> implementing new format of bitmaps storing. (Same as your approach, >> you'll have a simple raw node, and additional options will say "load >> bitmaps from this qcow2 file". But for such options we'll need good >> reasons why [1] isn't enough. >> > -- Best regards, Vladimir ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/2] block/raw: added support of persistent dirty bitmaps 2021-03-22 10:46 ` Vladimir Sementsov-Ogievskiy @ 2021-03-22 11:18 ` Vladimir Sementsov-Ogievskiy 2021-03-22 11:36 ` Patrik Janoušek 0 siblings, 1 reply; 20+ messages in thread From: Vladimir Sementsov-Ogievskiy @ 2021-03-22 11:18 UTC (permalink / raw) To: Patrik Janoušek, qemu-devel, qemu block; +Cc: lmatejka, Eric Blake 22.03.2021 13:46, Vladimir Sementsov-Ogievskiy wrote: > 22.03.2021 13:18, Patrik Janoušek wrote: >> On 3/22/21 9:41 AM, Vladimir Sementsov-Ogievskiy wrote: >>> 20.03.2021 12:32, Patrik Janoušek wrote: >>>> Current implementation of dirty bitmaps for raw format is very >>>> limited, because those bitmaps cannot be persistent. Basically it >>>> makes sense, because the raw format doesn't have space where could >>>> be dirty bitmap stored when QEMU is offline. This patch solves it >>>> by storing content of every dirty bitmap in separate file on the >>>> host filesystem. >>>> >>>> However, this only solves one part of the problem. We also have to >>>> store information about the existence of the dirty bitmap. This is >>>> solved by adding custom options, that stores all required metadata >>>> about dirty bitmap (filename where is the bitmap stored on the >>>> host filesystem, granularity, persistence, etc.). >>>> >>>> Signed-off-by: Patrik Janoušek<pj@patrikjanousek.cz> >>> >>> >>> Hmm. Did you considered other ways? Honestly, I don't see a reason for >>> yet another storing format for bitmaps. >>> >>> The task could be simply solved with existing features: >>> >>> 1. We have extenal-data-file feature in qcow2 (read >>> docs/interop/qcow2.txt). With this thing enabled, qcow2 file contains >>> only metadata (persistent bitmaps for example) and data is stored in >>> separate sequential raw file. I think you should start from it. >> >> I didn't know about that feature. I'll look at it. >> >> In case I use NBD to access the bitmap context and qcow2 as a solution >> for persistent layer. Would the patch be acceptable? This is significant >> change to my solution and I don't have enought time for it at the moment >> (mainly due to other parts of my bachelor's thesis). I just want to know >> if this kind of feature is interesting to you and its implementation is >> worth my time. > > Honestly, at this point I think it doesn't. If existing features satisfy your use-case, no reason to increase complexity of file-posix driver and QAPI. > It's unpleasant to say this, keeping in mind that that's your first submission :( I can still recommend in a connection with your bachelor's thesis to look at the videos at kvm-forum youtube channel, searching for backup: https://www.youtube.com/channel/UCRCSQmAOh7yzgheq-emy1xA/search?query=backup You'll get a lot of information about current developments of external backup API. Also note, that there is (or there will be ?) libvirt Backup API, which includes an API for external backup. I don't know the current status of it, but if your project is based on libvirt, it's better to use libvirt backup API instead of using qemu directly. About Libvirt Backup API it's better to ask Eric Blake (adding him to CC). -- Best regards, Vladimir ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/2] block/raw: added support of persistent dirty bitmaps 2021-03-22 11:18 ` Vladimir Sementsov-Ogievskiy @ 2021-03-22 11:36 ` Patrik Janoušek 2021-03-22 20:27 ` Lubos Matejka 0 siblings, 1 reply; 20+ messages in thread From: Patrik Janoušek @ 2021-03-22 11:36 UTC (permalink / raw) To: Vladimir Sementsov-Ogievskiy, qemu-devel, qemu block; +Cc: lmatejka On 3/22/21 12:18 PM, Vladimir Sementsov-Ogievskiy wrote: > 22.03.2021 13:46, Vladimir Sementsov-Ogievskiy wrote: >> 22.03.2021 13:18, Patrik Janoušek wrote: >>> On 3/22/21 9:41 AM, Vladimir Sementsov-Ogievskiy wrote: >>>> 20.03.2021 12:32, Patrik Janoušek wrote: >>>>> Current implementation of dirty bitmaps for raw format is very >>>>> limited, because those bitmaps cannot be persistent. Basically it >>>>> makes sense, because the raw format doesn't have space where could >>>>> be dirty bitmap stored when QEMU is offline. This patch solves it >>>>> by storing content of every dirty bitmap in separate file on the >>>>> host filesystem. >>>>> >>>>> However, this only solves one part of the problem. We also have to >>>>> store information about the existence of the dirty bitmap. This is >>>>> solved by adding custom options, that stores all required metadata >>>>> about dirty bitmap (filename where is the bitmap stored on the >>>>> host filesystem, granularity, persistence, etc.). >>>>> >>>>> Signed-off-by: Patrik Janoušek<pj@patrikjanousek.cz> >>>> >>>> >>>> Hmm. Did you considered other ways? Honestly, I don't see a reason for >>>> yet another storing format for bitmaps. >>>> >>>> The task could be simply solved with existing features: >>>> >>>> 1. We have extenal-data-file feature in qcow2 (read >>>> docs/interop/qcow2.txt). With this thing enabled, qcow2 file contains >>>> only metadata (persistent bitmaps for example) and data is stored in >>>> separate sequential raw file. I think you should start from it. >>> >>> I didn't know about that feature. I'll look at it. >>> >>> In case I use NBD to access the bitmap context and qcow2 as a solution >>> for persistent layer. Would the patch be acceptable? This is >>> significant >>> change to my solution and I don't have enought time for it at the >>> moment >>> (mainly due to other parts of my bachelor's thesis). I just want to >>> know >>> if this kind of feature is interesting to you and its implementation is >>> worth my time. >> >> Honestly, at this point I think it doesn't. If existing features >> satisfy your use-case, no reason to increase complexity of file-posix >> driver and QAPI. >> > > It's unpleasant to say this, keeping in mind that that's your first > submission :( > > I can still recommend in a connection with your bachelor's thesis to > look at the videos at kvm-forum youtube channel, searching for backup: > > > https://www.youtube.com/channel/UCRCSQmAOh7yzgheq-emy1xA/search?query=backup > > You'll get a lot of information about current developments of external > backup API. > > Also note, that there is (or there will be ?) libvirt Backup API, > which includes an API for external backup. I don't know the current > status of it, but if your project is based on libvirt, it's better to > use libvirt backup API instead of using qemu directly. About Libvirt > Backup API it's better to ask Eric Blake (adding him to CC). Unfortunately, my solution is based on Proxmox so I can't use libvirt's features. I know that a beta version of Proxmox Backup Server has been released and it would be much better to improve their solution, but they did it too late so I couldn't change assignment of my bachelor's thesis. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/2] block/raw: added support of persistent dirty bitmaps 2021-03-22 11:36 ` Patrik Janoušek @ 2021-03-22 20:27 ` Lubos Matejka 0 siblings, 0 replies; 20+ messages in thread From: Lubos Matejka @ 2021-03-22 20:27 UTC (permalink / raw) To: Patrik Janoušek Cc: Vladimir Sementsov-Ogievskiy, qemu-devel, qemu block, Lmatejka Kdy si muzem cinknout k dalsimu vyvoji? Odesláno z iPhonu > 22. 3. 2021 v 12:37, Patrik Janoušek <pj@patrikjanousek.cz>: > > >> On 3/22/21 12:18 PM, Vladimir Sementsov-Ogievskiy wrote: >> 22.03.2021 13:46, Vladimir Sementsov-Ogievskiy wrote: >>> 22.03.2021 13:18, Patrik Janoušek wrote: >>>> On 3/22/21 9:41 AM, Vladimir Sementsov-Ogievskiy wrote: >>>>> 20.03.2021 12:32, Patrik Janoušek wrote: >>>>>> Current implementation of dirty bitmaps for raw format is very >>>>>> limited, because those bitmaps cannot be persistent. Basically it >>>>>> makes sense, because the raw format doesn't have space where could >>>>>> be dirty bitmap stored when QEMU is offline. This patch solves it >>>>>> by storing content of every dirty bitmap in separate file on the >>>>>> host filesystem. >>>>>> >>>>>> However, this only solves one part of the problem. We also have to >>>>>> store information about the existence of the dirty bitmap. This is >>>>>> solved by adding custom options, that stores all required metadata >>>>>> about dirty bitmap (filename where is the bitmap stored on the >>>>>> host filesystem, granularity, persistence, etc.). >>>>>> >>>>>> Signed-off-by: Patrik Janoušek<pj@patrikjanousek.cz> >>>>> >>>>> >>>>> Hmm. Did you considered other ways? Honestly, I don't see a reason for >>>>> yet another storing format for bitmaps. >>>>> >>>>> The task could be simply solved with existing features: >>>>> >>>>> 1. We have extenal-data-file feature in qcow2 (read >>>>> docs/interop/qcow2.txt). With this thing enabled, qcow2 file contains >>>>> only metadata (persistent bitmaps for example) and data is stored in >>>>> separate sequential raw file. I think you should start from it. >>>> >>>> I didn't know about that feature. I'll look at it. >>>> >>>> In case I use NBD to access the bitmap context and qcow2 as a solution >>>> for persistent layer. Would the patch be acceptable? This is >>>> significant >>>> change to my solution and I don't have enought time for it at the >>>> moment >>>> (mainly due to other parts of my bachelor's thesis). I just want to >>>> know >>>> if this kind of feature is interesting to you and its implementation is >>>> worth my time. >>> >>> Honestly, at this point I think it doesn't. If existing features >>> satisfy your use-case, no reason to increase complexity of file-posix >>> driver and QAPI. >>> >> >> It's unpleasant to say this, keeping in mind that that's your first >> submission :( >> >> I can still recommend in a connection with your bachelor's thesis to >> look at the videos at kvm-forum youtube channel, searching for backup: >> >> >> https://www.youtube.com/channel/UCRCSQmAOh7yzgheq-emy1xA/search?query=backup >> >> You'll get a lot of information about current developments of external >> backup API. >> >> Also note, that there is (or there will be ?) libvirt Backup API, >> which includes an API for external backup. I don't know the current >> status of it, but if your project is based on libvirt, it's better to >> use libvirt backup API instead of using qemu directly. About Libvirt >> Backup API it's better to ask Eric Blake (adding him to CC). > Unfortunately, my solution is based on Proxmox so I can't use libvirt's > features. I know that a beta version of Proxmox Backup Server has been > released and it would be much better to improve their solution, but they > did it too late so I couldn't change assignment of my bachelor's thesis. > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/2] block/raw: added support of persistent dirty bitmaps 2021-03-20 9:32 ` [PATCH 1/2] block/raw: added support of persistent dirty bitmaps Patrik Janoušek 2021-03-22 8:41 ` Vladimir Sementsov-Ogievskiy @ 2021-03-22 15:43 ` Kevin Wolf 1 sibling, 0 replies; 20+ messages in thread From: Kevin Wolf @ 2021-03-22 15:43 UTC (permalink / raw) To: Patrik Janoušek; +Cc: qemu-devel, lmatejka Am 20.03.2021 um 10:32 hat Patrik Janoušek geschrieben: > Current implementation of dirty bitmaps for raw format is very > limited, because those bitmaps cannot be persistent. Basically it > makes sense, because the raw format doesn't have space where could > be dirty bitmap stored when QEMU is offline. This patch solves it > by storing content of every dirty bitmap in separate file on the > host filesystem. > > However, this only solves one part of the problem. We also have to > store information about the existence of the dirty bitmap. This is > solved by adding custom options, that stores all required metadata > about dirty bitmap (filename where is the bitmap stored on the > host filesystem, granularity, persistence, etc.). > > Signed-off-by: Patrik Janoušek <pj@patrikjanousek.cz> I'm not sure if you're going to try to change your thesis to use qcow2 with an external data file, but in case you're still using this patch for your thesis, let's imagine for a moment that we all agreed on adding the functionality to raw. After skimming over the patch, I see two major things that we would ask to change: 1. You need to change BlockdevOptions to the QAPI schema in qapi/block-core.json, because if you don't add new options there, they won't be accessible with -blockdev and QMP blockdev-add. Among others, this means you would describe RawDirtyBitmapOpts in the schema and would get the structure and some helpers automatically generated, simplifying your code. Instead of processing QemuOpts manually, I would then probably try to use the QAPI visitors to get RawDirtyBitmapOpts from the input, which could potentially further simplify the code. 2. Instead of having a 'filename': 'str' option and working on the bitmap files with stdio.h functions (which block the guest instead of allowing asynchronous operation in the background), we would probably want something like 'file': 'BlockdevRef', so you get another BlockDriverState for each bitmap file that you access with the normal QEMU block layer I/O functions. The advantage is not only that this isn't blocking, but it also allows you to configure more details than just the filename (think cache mode, AIO mode, locking, etc.). In fact, it would even allow you to store the metadata not in a file, but in any place that can be accessed with a protocol supported by QEMU (like NBD, iscsi, ssh, whatever). Kevin ^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH 2/2] qapi: implementation of the block-dirty-bitmap-dump command 2021-03-20 9:32 [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi Patrik Janoušek 2021-03-20 9:32 ` [PATCH 1/2] block/raw: added support of persistent dirty bitmaps Patrik Janoušek @ 2021-03-20 9:32 ` Patrik Janoušek 2021-03-22 9:02 ` Vladimir Sementsov-Ogievskiy 2021-03-22 15:12 ` Kevin Wolf 2021-03-22 8:29 ` [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi Vladimir Sementsov-Ogievskiy [not found] ` <856ca6ba-3871-068f-f821-269c40a5a4d5@patrikjanousek.cz> 3 siblings, 2 replies; 20+ messages in thread From: Patrik Janoušek @ 2021-03-20 9:32 UTC (permalink / raw) To: qemu-devel; +Cc: Patrik Janoušek, lmatejka Currently, dirty bitmaps are for internal use only and there is no support for accessing their content from third party-apps. This patch implements new command block-dirty-bitmap-dump, which returns content of the dirty bitmap encoded in base64. This is very useful especially in combination with a drive that uses raw format because third-party apps can easily use it to create incremental or differential backup. Signed-off-by: Patrik Janoušek <pj@patrikjanousek.cz> --- block/monitor/bitmap-qmp-cmds.c | 61 +++++++++++++++++++++++++++++++ qapi/block-core.json | 64 ++++++++++++++++++++++++++++++++- 2 files changed, 124 insertions(+), 1 deletion(-) diff --git a/block/monitor/bitmap-qmp-cmds.c b/block/monitor/bitmap-qmp-cmds.c index 9f11deec64..7f296e9ba7 100644 --- a/block/monitor/bitmap-qmp-cmds.c +++ b/block/monitor/bitmap-qmp-cmds.c @@ -146,6 +146,67 @@ out: aio_context_release(aio_context); } +BlockDirtyBitmapContent *qmp_block_dirty_bitmap_dump(const char *node, + const char *name, + bool has_clear, bool clear, + Error **errp) +{ + BlockDriverState *bs; + BdrvDirtyBitmap *bitmap; + BlockDirtyBitmapContent *bdbc; + HBitmap *hb; + AioContext *aio_context; + + if (!name || name[0] == '\0') { + error_setg(errp, "Bitmap name cannot be empty"); + return NULL; + } + + bs = bdrv_lookup_bs(node, node, errp); + if (!bs) { + return NULL; + } + + aio_context = bdrv_get_aio_context(bs); + aio_context_acquire(aio_context); + + bitmap = block_dirty_bitmap_lookup(node, name, &bs, errp); + if (!bitmap || !bs) { + return NULL; + } + + if (has_clear && clear) { + /** + * Transactions cannot return value, so "clear" functionality must be + * implemented here while holding AiO context + */ + + bdrv_clear_dirty_bitmap(bitmap, &hb); + + uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap); + uint64_t tb_size = hbitmap_serialization_size(hb, 0, bm_size); + uint8_t *buf = g_malloc(tb_size); + + hbitmap_serialize_part(hb, buf, 0, bm_size); + + bdbc = g_new0(BlockDirtyBitmapContent, 1); + bdbc->content = g_base64_encode((guchar *) buf, tb_size); + } else { + uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap); + uint64_t tb_size = bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_size); + uint8_t *buf = g_malloc(tb_size); + + bdrv_dirty_bitmap_serialize_part(bitmap, buf, 0, bm_size); + + bdbc = g_new0(BlockDirtyBitmapContent, 1); + bdbc->content = g_base64_encode((guchar *) buf, tb_size); + } + + aio_context_release(aio_context); + + return bdbc; +} + BdrvDirtyBitmap *block_dirty_bitmap_remove(const char *node, const char *name, bool release, BlockDriverState **bitmap_bs, diff --git a/qapi/block-core.json b/qapi/block-core.json index 04ad80bc1e..cbe3dac384 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -2031,6 +2031,14 @@ { 'struct': 'BlockDirtyBitmap', 'data': { 'node': 'str', 'name': 'str' } } +## +# @BlockDirtyBitmapContent: +# +# @content: content of dirty bitmap (encoded in base64) +## +{ 'struct': 'BlockDirtyBitmapContent', + 'data': { 'content': 'str' } } + ## # @BlockDirtyBitmapAdd: # @@ -2056,6 +2064,18 @@ 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32', '*persistent': 'bool', '*disabled': 'bool' } } +## +# @BlockDirtyBitmapDump: +# +# @node: name of device/node which the bitmap is tracking +# +# @name: name of the dirty bitmap (must be less than 1024 bytes) +# +# @clear: true if bitmap should be cleared after dump +## +{ 'struct': 'BlockDirtyBitmapDump', + 'data': { 'node': 'str', 'name': 'str', '*clear': 'bool' } } + ## # @BlockDirtyBitmapMergeSource: # @@ -2086,6 +2106,26 @@ 'data': { 'node': 'str', 'target': 'str', 'bitmaps': ['BlockDirtyBitmapMergeSource'] } } +## +# @block-dirty-bitmap-dump: +# +# Dump a dirty bitmap with a name on the node. +# +# Returns: - nothing on success +# - If @node is not a valid block device or node, DeviceNotFound +# - If @name is already taken, GenericError with an explanation +# +# Example: +# +# -> { "execute": "block-dirty-bitmap-dump", +# "arguments": { "node": "drive0", "name": "bitmap0" } } +# <- { "return": { "content": "TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFt... (trunc)" } } +# +## +{ 'command': 'block-dirty-bitmap-dump', + 'data': 'BlockDirtyBitmapDump', + 'returns': 'BlockDirtyBitmapContent' } + ## # @block-dirty-bitmap-add: # @@ -3908,6 +3948,26 @@ '*x-dirty-bitmap': 'str', '*reconnect-delay': 'uint32' } } +## +# @BlockdevOptionsRawDirtyBitmap: +# +# Dirty bitmap options for the raw driver. +# +# @name: the name of the dirty bitmap (Since 2.4) +# +# @filename: the filename of the dirty bitmap +# +# @granularity: granularity of the dirty bitmap in bytes (since 1.4) +# +# @persistent: true if the bitmap was stored on disk, is scheduled to be stored +# on disk, or both. (since 4.0) +# +# @disabled: true if the bitmap should not be loaded (and saved) automatically +## +{ 'struct': 'BlockdevOptionsRawDirtyBitmap', + 'data': {'*name': 'str', 'filename': 'str', 'granularity': 'uint32', + 'persistent': 'bool', '*disabled': 'bool' } } + ## # @BlockdevOptionsRaw: # @@ -3915,12 +3975,14 @@ # # @offset: position where the block device starts # @size: the assumed size of the device +# @dirty-bitmaps: dirty bitmaps of the raw block device # # Since: 2.9 ## { 'struct': 'BlockdevOptionsRaw', 'base': 'BlockdevOptionsGenericFormat', - 'data': { '*offset': 'int', '*size': 'int' } } + 'data': { '*offset': 'int', '*size': 'int' , + '*dirty-bitmaps': ['BlockdevOptionsRawDirtyBitmap'] } } ## # @BlockdevOptionsThrottle: -- 2.31.0 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] qapi: implementation of the block-dirty-bitmap-dump command 2021-03-20 9:32 ` [PATCH 2/2] qapi: implementation of the block-dirty-bitmap-dump command Patrik Janoušek @ 2021-03-22 9:02 ` Vladimir Sementsov-Ogievskiy 2021-03-22 15:12 ` Kevin Wolf 1 sibling, 0 replies; 20+ messages in thread From: Vladimir Sementsov-Ogievskiy @ 2021-03-22 9:02 UTC (permalink / raw) To: Patrik Janoušek, qemu-devel; +Cc: lmatejka 20.03.2021 12:32, Patrik Janoušek wrote: > Currently, dirty bitmaps are for internal use only As I said, that's not correct: for external use we have bitmap export through NBD protocol. Protocol description is here https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md (you need "Metadata querying" section) And qemu bitmap export through NBD "metadata querying" feature is described here: docs/interop/nbd.txt -- Best regards, Vladimir ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 2/2] qapi: implementation of the block-dirty-bitmap-dump command 2021-03-20 9:32 ` [PATCH 2/2] qapi: implementation of the block-dirty-bitmap-dump command Patrik Janoušek 2021-03-22 9:02 ` Vladimir Sementsov-Ogievskiy @ 2021-03-22 15:12 ` Kevin Wolf 1 sibling, 0 replies; 20+ messages in thread From: Kevin Wolf @ 2021-03-22 15:12 UTC (permalink / raw) To: Patrik Janoušek; +Cc: qemu-devel, lmatejka Am 20.03.2021 um 10:32 hat Patrik Janoušek geschrieben: > Currently, dirty bitmaps are for internal use only and there is > no support for accessing their content from third party-apps. > This patch implements new command block-dirty-bitmap-dump, which > returns content of the dirty bitmap encoded in base64. This is > very useful especially in combination with a drive that uses raw > format because third-party apps can easily use it to create > incremental or differential backup. > > Signed-off-by: Patrik Janoušek <pj@patrikjanousek.cz> People have already pointed you to NBD to get the block dirty status (and they are right), but I think nobody has explained yet why we decided against a QMP command to return bitmaps. The problem is that disk images can be huge, and this means that dirty bitmaps describing them get pretty large, too. So your new QMP command ends up reading many megabytes from disk and sending it to the socket. While it's doing all of this I/O, QEMU is blocked and the guest may appear to be hanging until it completes. The client would also have to download the whole bitmap even if it is only interested in specific blocks. (This may or may not be relevant for you specific use case.) With the NBD export, the client can query a specific block range and its request is processed asynchronously in the background while the guest and the QMP monitor are still available. Kevin ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi 2021-03-20 9:32 [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi Patrik Janoušek 2021-03-20 9:32 ` [PATCH 1/2] block/raw: added support of persistent dirty bitmaps Patrik Janoušek 2021-03-20 9:32 ` [PATCH 2/2] qapi: implementation of the block-dirty-bitmap-dump command Patrik Janoušek @ 2021-03-22 8:29 ` Vladimir Sementsov-Ogievskiy 2021-03-22 8:57 ` Patrik Janoušek [not found] ` <856ca6ba-3871-068f-f821-269c40a5a4d5@patrikjanousek.cz> 3 siblings, 1 reply; 20+ messages in thread From: Vladimir Sementsov-Ogievskiy @ 2021-03-22 8:29 UTC (permalink / raw) To: Patrik Janoušek, qemu-devel; +Cc: lmatejka Hi Patrik! 20.03.2021 12:32, Patrik Janoušek wrote: > Currently, QEMU doesn't support persistent dirty bitmaps for raw format That's right, we don't have such feature now. > and also dirty bitmaps are for internal use only, and cannot be accessed > using third-party applications. And that's is not. Bitmaps are accessible through bitmap QMP API block-dirty-bitmap-{add,remove,clear,merge} And to retrieve the context of dirty bitmap you can export it through NBD protocol (see bitmaps argument in nbd specific options of block-export-add command) > These facts are very limiting > in case someone would like to develop their own backup tool becaouse > without access to the dirty bitmap it would be possible to implement > only full backups. We do have external incremental backups, based on Qemu bitmap API. But it depends of course on qcow2 persistent bitmaps feature. > And without persistent dirty bitmaps, it wouldn't > be possible to keep track of changed data after QEMU is restarted. And > this is exactly what I do as a part of my bachelor thesis. I've > developed a tool that is able to create incremental backups of drives > in raw format that are LVM volumes (ability to create snapshot is > required). > > Please keep in mind that this is my first submission to such a large > project and also the first time when I send patch over the email. > So I hope I did it correctly. > > Patrik Janoušek (2): > block/raw: added support of persistent dirty bitmaps > qapi: implementation of the block-dirty-bitmap-dump command > > block/meson.build | 1 + > block/monitor/bitmap-qmp-cmds.c | 61 ++++++++ > block/raw-format-bitmap.c | 163 ++++++++++++++++++++ > block/raw-format.c | 256 ++++++++++++++++++++++++++++++-- > block/raw-format.h | 50 +++++++ > qapi/block-core.json | 64 +++++++- > 6 files changed, 583 insertions(+), 12 deletions(-) > create mode 100644 block/raw-format-bitmap.c > create mode 100644 block/raw-format.h > -- Best regards, Vladimir ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi 2021-03-22 8:29 ` [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi Vladimir Sementsov-Ogievskiy @ 2021-03-22 8:57 ` Patrik Janoušek 2021-03-22 14:53 ` Kevin Wolf 0 siblings, 1 reply; 20+ messages in thread From: Patrik Janoušek @ 2021-03-22 8:57 UTC (permalink / raw) To: Vladimir Sementsov-Ogievskiy, qemu-devel; +Cc: lmatejka [-- Attachment #1: Type: text/plain, Size: 2722 bytes --] Hello, thank you for the quick reply. On 3/22/21 9:29 AM, Vladimir Sementsov-Ogievskiy wrote: > Hi Patrik! > > 20.03.2021 12:32, Patrik Janoušek wrote: >> Currently, QEMU doesn't support persistent dirty bitmaps for raw format > > That's right, we don't have such feature now. > >> and also dirty bitmaps are for internal use only, and cannot be accessed >> using third-party applications. > > And that's is not. Bitmaps are accessible through bitmap QMP API > > block-dirty-bitmap-{add,remove,clear,merge} I know about these commands, but they're useless to me without providing access to the content of bitmap. > > And to retrieve the context of dirty bitmap you can export it through > NBD protocol (see bitmaps argument in nbd specific options of > block-export-add command) Ok, I'll look at it. > >> These facts are very limiting >> in case someone would like to develop their own backup tool becaouse >> without access to the dirty bitmap it would be possible to implement >> only full backups. > > We do have external incremental backups, based on Qemu bitmap API. But > it depends of course on qcow2 persistent bitmaps feature. Yes, I know. And that's the problem. The point of my bachelor thesis is to implement a backup solution for the raw format. > >> And without persistent dirty bitmaps, it wouldn't >> be possible to keep track of changed data after QEMU is restarted. And >> this is exactly what I do as a part of my bachelor thesis. I've >> developed a tool that is able to create incremental backups of drives >> in raw format that are LVM volumes (ability to create snapshot is >> required). >> >> Please keep in mind that this is my first submission to such a large >> project and also the first time when I send patch over the email. >> So I hope I did it correctly. >> >> Patrik Janoušek (2): >> block/raw: added support of persistent dirty bitmaps >> qapi: implementation of the block-dirty-bitmap-dump command >> >> block/meson.build | 1 + >> block/monitor/bitmap-qmp-cmds.c | 61 ++++++++ >> block/raw-format-bitmap.c | 163 ++++++++++++++++++++ >> block/raw-format.c | 256 ++++++++++++++++++++++++++++++-- >> block/raw-format.h | 50 +++++++ >> qapi/block-core.json | 64 +++++++- >> 6 files changed, 583 insertions(+), 12 deletions(-) >> create mode 100644 block/raw-format-bitmap.c >> create mode 100644 block/raw-format.h >> > > Patrik Janoušek IČO: 09291849 +420 724 123 897 <tel:+420 724 123 897> pj@patrikjanousek.cz <mailto:pj@patrikjanousek.cz> [-- Attachment #2.1: Type: text/html, Size: 10707 bytes --] [-- Attachment #2.2: kaohbpgmkipimblo.png --] [-- Type: image/png, Size: 177 bytes --] [-- Attachment #2.3: biakegienjdedpnc.png --] [-- Type: image/png, Size: 170 bytes --] ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi 2021-03-22 8:57 ` Patrik Janoušek @ 2021-03-22 14:53 ` Kevin Wolf 0 siblings, 0 replies; 20+ messages in thread From: Kevin Wolf @ 2021-03-22 14:53 UTC (permalink / raw) To: Patrik Janoušek; +Cc: Vladimir Sementsov-Ogievskiy, qemu-devel, lmatejka Hi Patrik, Am 22.03.2021 um 09:57 hat Patrik Janoušek geschrieben: > On 3/22/21 9:29 AM, Vladimir Sementsov-Ogievskiy wrote: > > We do have external incremental backups, based on Qemu bitmap API. But > > it depends of course on qcow2 persistent bitmaps feature. > > Yes, I know. And that's the problem. The point of my bachelor thesis is > to implement a backup solution for the raw format. the problem with this is that raw isn't really a format, it's more the absence of a format. You just have the content of the virtual disk in a file and that's it. This means not having any metadata (apart from the metadata stored in the filesystem, of course). As soon as you add metadata in some way (in your case, by referencing additional metadata files in runtime options), it's not raw any more. If you write to the raw image file without updating the metadata, the metadata becomes inconsistent with the content. In other words, both files form a single disk image together and can only be used together or you're breaking them. This in turn means that you have just invented a new image format. It's a bit unconventional in that it's spread across multiple files, and that some of the metadata that brings everything together is even in command line options instead of a file, but you have to combine these components in the same way every time you start the VM, so it really is a new image format. We have gone through such discussions a while ago because obviously "raw with dirty bitmaps" was a request that came soon after we discussed persistent dirty maps. But as we came to the conclusion that any addition to raw would create another new image format specific to QEMU, we decided that we can as well use qcow2 for this, which is already the fully featured QEMU image format. I hope this background helps a bit to explain the reactions you have received so far. Kevin ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <856ca6ba-3871-068f-f821-269c40a5a4d5@patrikjanousek.cz>]
* Re: Fwd: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi [not found] ` <856ca6ba-3871-068f-f821-269c40a5a4d5@patrikjanousek.cz> @ 2021-03-22 10:48 ` Max Reitz 2021-03-22 11:27 ` Patrik Janoušek 0 siblings, 1 reply; 20+ messages in thread From: Max Reitz @ 2021-03-22 10:48 UTC (permalink / raw) To: Patrik Janoušek, Kevin Wolf, Eric Blake, Vladimir Sementsov-Ogievskiy, John Snow Cc: qemu-devel, Qemu-block Hi, On 20.03.21 11:01, Patrik Janoušek wrote: > I'm sorry, but I forgot to add you to the cc, so I'm forwarding the > patch to you additionally. I don't want to spam the mailing list > unnecessarily. I think it’s better to still CC the list. It’s so full of mail, one more won’t hurt. :) (Re-adding qemu-block and qemu-devel, because the discussion belongs on the list(s).) > -------- Forwarded Message -------- > Subject: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and > ability to dump bitmap content via qapi > Date: Sat, 20 Mar 2021 10:32:33 +0100 > From: Patrik Janoušek <pj@patrikjanousek.cz> > To: qemu-devel@nongnu.org > CC: Patrik Janoušek <pj@patrikjanousek.cz>, lmatejka@kiv.zcu.cz > > > > Currently, QEMU doesn't support persistent dirty bitmaps for raw format > and also dirty bitmaps are for internal use only, and cannot be accessed > using third-party applications. These facts are very limiting > in case someone would like to develop their own backup tool becaouse > without access to the dirty bitmap it would be possible to implement > only full backups. And without persistent dirty bitmaps, it wouldn't > be possible to keep track of changed data after QEMU is restarted. And > this is exactly what I do as a part of my bachelor thesis. I've > developed a tool that is able to create incremental backups of drives > in raw format that are LVM volumes (ability to create snapshot is > required). Similarly to what Vladimir has said already, the thing is that conceptually I can see no difference between having a raw image with the bitmaps stored in some other file, i.e.: { "driver": "raw", "dirty-bitmaps": [ { "filename": "sdc1.bitmap", "persistent": true } ], "file": { "driver": "file", "filename": "/dev/sdc1" } } And having a qcow2 image with the raw data stored in some other file, i.e.: { "driver": "qcow2", "file": { "driver": "file", "filename": "sdc1.metadata" }, "data-file": { "driver": "file", "filename": "/dev/sdc1" } } (Where sdc1.metadata is a qcow2 file created with “data-file=/dev/sdc1,data-file-raw=on”.) To use persistent bitmaps with raw images, you need to add metadata (namely, the bitmaps). Why not store that metadata in a qcow2 file? Max ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Fwd: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi 2021-03-22 10:48 ` Fwd: " Max Reitz @ 2021-03-22 11:27 ` Patrik Janoušek 2021-03-22 12:06 ` Max Reitz 2021-03-22 12:44 ` Fabian Grünbichler 0 siblings, 2 replies; 20+ messages in thread From: Patrik Janoušek @ 2021-03-22 11:27 UTC (permalink / raw) To: Max Reitz, Kevin Wolf, Eric Blake, Vladimir Sementsov-Ogievskiy, John Snow Cc: qemu-devel, Qemu-block, lmatejka On 3/22/21 11:48 AM, Max Reitz wrote: > Hi, > > On 20.03.21 11:01, Patrik Janoušek wrote: >> I'm sorry, but I forgot to add you to the cc, so I'm forwarding the >> patch to you additionally. I don't want to spam the mailing list >> unnecessarily. > > I think it’s better to still CC the list. It’s so full of mail, one > more won’t hurt. :) > > (Re-adding qemu-block and qemu-devel, because the discussion belongs > on the list(s).) > >> -------- Forwarded Message -------- >> Subject: [PATCH 0/2] block/raw: implemented persistent dirty >> bitmap and ability to dump bitmap content via qapi >> Date: Sat, 20 Mar 2021 10:32:33 +0100 >> From: Patrik Janoušek <pj@patrikjanousek.cz> >> To: qemu-devel@nongnu.org >> CC: Patrik Janoušek <pj@patrikjanousek.cz>, lmatejka@kiv.zcu.cz >> >> >> >> Currently, QEMU doesn't support persistent dirty bitmaps for raw format >> and also dirty bitmaps are for internal use only, and cannot be accessed >> using third-party applications. These facts are very limiting >> in case someone would like to develop their own backup tool becaouse >> without access to the dirty bitmap it would be possible to implement >> only full backups. And without persistent dirty bitmaps, it wouldn't >> be possible to keep track of changed data after QEMU is restarted. And >> this is exactly what I do as a part of my bachelor thesis. I've >> developed a tool that is able to create incremental backups of drives >> in raw format that are LVM volumes (ability to create snapshot is >> required). > > Similarly to what Vladimir has said already, the thing is that > conceptually I can see no difference between having a raw image with > the bitmaps stored in some other file, i.e.: > > { "driver": "raw", > "dirty-bitmaps": [ { > "filename": "sdc1.bitmap", > "persistent": true > } ], > "file": { > "driver": "file", > "filename": "/dev/sdc1" > } } > > And having a qcow2 image with the raw data stored in some other file, > i.e.: > > { "driver": "qcow2", > "file": { > "driver": "file", > "filename": "sdc1.metadata" > }, > "data-file": { > "driver": "file", > "filename": "/dev/sdc1" > } } > > (Where sdc1.metadata is a qcow2 file created with > “data-file=/dev/sdc1,data-file-raw=on”.) > > To use persistent bitmaps with raw images, you need to add metadata > (namely, the bitmaps). Why not store that metadata in a qcow2 file? > > Max So if I understand it correctly. I can configure dirty bitmaps in the latest version of QEMU to be persistently stored in some other file. Because even Proxmox Backup Server can't perform an incremental backup after restarting QEMU, and that means something to me. I think they would implement it if it was that simple. Could you please send me simple example on how to configure (via command line args) one raw format drive that can store dirty bitmaps persistently in other qcow2 file? I may be missing something, but I thought QEMU couldn't do it, because Proxmox community wants this feature for a long time. Patrik ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Fwd: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi 2021-03-22 11:27 ` Patrik Janoušek @ 2021-03-22 12:06 ` Max Reitz 2021-03-22 21:45 ` Patrik Janoušek 2021-03-22 12:44 ` Fabian Grünbichler 1 sibling, 1 reply; 20+ messages in thread From: Max Reitz @ 2021-03-22 12:06 UTC (permalink / raw) To: Patrik Janoušek, Kevin Wolf, Eric Blake, Vladimir Sementsov-Ogievskiy, John Snow Cc: qemu-devel, Qemu-block, lmatejka On 22.03.21 12:27, Patrik Janoušek wrote: > On 3/22/21 11:48 AM, Max Reitz wrote: >> Hi, >> >> On 20.03.21 11:01, Patrik Janoušek wrote: >>> I'm sorry, but I forgot to add you to the cc, so I'm forwarding the >>> patch to you additionally. I don't want to spam the mailing list >>> unnecessarily. >> >> I think it’s better to still CC the list. It’s so full of mail, one >> more won’t hurt. :) >> >> (Re-adding qemu-block and qemu-devel, because the discussion belongs >> on the list(s).) >> >>> -------- Forwarded Message -------- >>> Subject: [PATCH 0/2] block/raw: implemented persistent dirty >>> bitmap and ability to dump bitmap content via qapi >>> Date: Sat, 20 Mar 2021 10:32:33 +0100 >>> From: Patrik Janoušek <pj@patrikjanousek.cz> >>> To: qemu-devel@nongnu.org >>> CC: Patrik Janoušek <pj@patrikjanousek.cz>, lmatejka@kiv.zcu.cz >>> >>> >>> >>> Currently, QEMU doesn't support persistent dirty bitmaps for raw format >>> and also dirty bitmaps are for internal use only, and cannot be accessed >>> using third-party applications. These facts are very limiting >>> in case someone would like to develop their own backup tool becaouse >>> without access to the dirty bitmap it would be possible to implement >>> only full backups. And without persistent dirty bitmaps, it wouldn't >>> be possible to keep track of changed data after QEMU is restarted. And >>> this is exactly what I do as a part of my bachelor thesis. I've >>> developed a tool that is able to create incremental backups of drives >>> in raw format that are LVM volumes (ability to create snapshot is >>> required). >> >> Similarly to what Vladimir has said already, the thing is that >> conceptually I can see no difference between having a raw image with >> the bitmaps stored in some other file, i.e.: >> >> { "driver": "raw", >> "dirty-bitmaps": [ { >> "filename": "sdc1.bitmap", >> "persistent": true >> } ], >> "file": { >> "driver": "file", >> "filename": "/dev/sdc1" >> } } >> >> And having a qcow2 image with the raw data stored in some other file, >> i.e.: >> >> { "driver": "qcow2", >> "file": { >> "driver": "file", >> "filename": "sdc1.metadata" >> }, >> "data-file": { >> "driver": "file", >> "filename": "/dev/sdc1" >> } } >> >> (Where sdc1.metadata is a qcow2 file created with >> “data-file=/dev/sdc1,data-file-raw=on”.) >> >> To use persistent bitmaps with raw images, you need to add metadata >> (namely, the bitmaps). Why not store that metadata in a qcow2 file? >> >> Max > > So if I understand it correctly. I can configure dirty bitmaps in the > latest version of QEMU to be persistently stored in some other file. > Because even Proxmox Backup Server can't perform an incremental backup > after restarting QEMU, and that means something to me. I think they > would implement it if it was that simple. > > Could you please send me simple example on how to configure (via command > line args) one raw format drive that can store dirty bitmaps > persistently in other qcow2 file? I may be missing something, but I > thought QEMU couldn't do it, because Proxmox community wants this > feature for a long time. One trouble is that if you use qemu-img create to create the qcow2 image, it will always create an empty image, and so if use pass data_file to it, it will empty the existing raw image: $ cp ~/tmp/arch.iso raw.img # Just some Arch Linux ISO $ qemu-img create \ -f qcow2 \ -o data_file=raw.img,data_file_raw=on,preallocation=metadata \ metadata.qcow2 \ $(stat -c '%s' raw.img) Formatting 'metadata.qcow2', fmt=qcow2 cluster_size=65536 preallocation=metadata compression_type=zlib size=687865856 data_file=raw.img data_file_raw=on lazy_refcounts=off refcount_bits=16 (If you check raw.img at this point, you’ll find that it’s empty, so you need to copy it from the source again:) $ cp ~/tmp/arch.iso raw.img Now if you use metadata.qcow2, the image data will actually all be stored in raw.img. To get around the “creating metadata.qcow2 clears raw.img” problem, you can either create a temporary empty image of the same size as raw.img that you pass to qemu-img create, and then you use qemu-img amend to change the data-file pointer (which will not overwrite the new data-file’s contents): $ qemu-img create -f raw tmp.raw $(stat -c '%s' raw.img) $ qemu-img create \ -f qcow2 \ -o data_file=tmp.img,data_file_raw=on,preallocation=metadata \ metadata.qcow2 \ $(stat -c '%s' raw.img) Formatting 'metadata.qcow2', fmt=qcow2 cluster_size=65536 preallocation=metadata compression_type=zlib size=687865856 data_file=tmp.img data_file_raw=on lazy_refcounts=off refcount_bits=16 $ qemu-img amend -o data_file=raw.img metadata.qcow2 $ rm tmp.img Or you use the blockdev-create job to create the qcow2 image (because contrary to qemu-img create, that will not clear the data file): $ touch metadata.qcow2 (Note that in the following QMP communication, what I sent and what qemu replies is mixed. Everything that begins with '{ "execute"' is from me, everything else from qemu. The number 687865856 is the size of raw.img in bytes.) $ qemu-system-x86_64 -qmp stdio \ -blockdev \ '{ "node-name": "metadata-file", "driver": "file", "filename": "metadata.qcow2" }' \ -blockdev \ '{ "node-name": "data-file", "driver": "file", "filename": "raw.img" }' {"QMP": {"version": {"qemu": {"micro": 0, "minor": 1, "major": 5}, "package": "qemu-5.1.0-9.fc33"}, "capabilities": ["oob"]}} { "execute": "qmp_capabilities" } {"return": {}} { "execute": "blockdev-create", "arguments": { "job-id": "create", "options": { "driver": "qcow2", "file": "metadata-file", "data-file": "data-file", "data-file-raw": true, "preallocation": "metadata", "size": 687865856 } } } {"timestamp": {"seconds": 1616414002, "microseconds": 836899}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "create"}} {"timestamp": {"seconds": 1616414002, "microseconds": 837076}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "create"}} {"return": {}} {"timestamp": {"seconds": 1616414002, "microseconds": 870997}, "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": "create"}} {"timestamp": {"seconds": 1616414002, "microseconds": 871099}, "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": "create"}} {"timestamp": {"seconds": 1616414002, "microseconds": 871185}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "create"}} { "execute": "job-dismiss", "arguments": { "id": "create" } } {"timestamp": {"seconds": 1616414022, "microseconds": 202880}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "create"}} {"return": {}} { "execute": "quit" } {"return": {}} {"timestamp": {"seconds": 1616414028, "microseconds": 56457}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} In any case, in the end you get a metadata.qcow2 that holds the metadata and points to raw.img for its data. So you can use it like this: $ qemu-system-x86_64 -enable-kvm -m 512 \ -blockdev \ '{ "node-name": "node0", "driver": "qcow2", "file": { "driver": "file", "filename": "metadata.qcow2" } }' \ -device ide-cd,drive=node0 \ -qmp stdio {"QMP": {"version": {"qemu": {"micro": 0, "minor": 1, "major": 5}, "package": "qemu-5.1.0-9.fc33"}, "capabilities": ["oob"]}} { "execute": "qmp_capabilities" } {"return": {}} { "execute": "block-dirty-bitmap-add", "arguments": { "node": "node0", "name": "bmap0", "persistent": true } } {"return": {}} { "execute": "quit" } {"return": {}} {"timestamp": {"seconds": 1616414627, "microseconds": 928250}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} $ qemu-img info metadata.qcow2 image: metadata.qcow2 file format: qcow2 virtual size: 656 MiB (687865856 bytes) disk size: 452 KiB cluster_size: 65536 Format specific information: compat: 1.1 compression type: zlib lazy refcounts: false bitmaps: [0]: flags: [0]: auto name: bmap0 granularity: 65536 refcount bits: 16 data file: raw.img data file raw: true corrupt: false So the bitmap is now in metadata.qcow2, and as the disk size indicates, all the data is still in raw.img. I hope the above helps you. Sorry if it’s confusing, especially the first part where I’m like “The obvious way to create metadata.qcow2 will delete your data, so here are two alternatives that are weird but do what you want.” Max ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Fwd: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi 2021-03-22 12:06 ` Max Reitz @ 2021-03-22 21:45 ` Patrik Janoušek 0 siblings, 0 replies; 20+ messages in thread From: Patrik Janoušek @ 2021-03-22 21:45 UTC (permalink / raw) To: Max Reitz, Kevin Wolf, Eric Blake, Vladimir Sementsov-Ogievskiy, John Snow Cc: qemu-devel, Qemu-block, lmatejka On 3/22/21 1:06 PM, Max Reitz wrote: > On 22.03.21 12:27, Patrik Janoušek wrote: >> On 3/22/21 11:48 AM, Max Reitz wrote: >>> Hi, >>> >>> On 20.03.21 11:01, Patrik Janoušek wrote: >>>> I'm sorry, but I forgot to add you to the cc, so I'm forwarding the >>>> patch to you additionally. I don't want to spam the mailing list >>>> unnecessarily. >>> >>> I think it’s better to still CC the list. It’s so full of mail, one >>> more won’t hurt. :) >>> >>> (Re-adding qemu-block and qemu-devel, because the discussion belongs >>> on the list(s).) >>> >>>> -------- Forwarded Message -------- >>>> Subject: [PATCH 0/2] block/raw: implemented persistent dirty >>>> bitmap and ability to dump bitmap content via qapi >>>> Date: Sat, 20 Mar 2021 10:32:33 +0100 >>>> From: Patrik Janoušek <pj@patrikjanousek.cz> >>>> To: qemu-devel@nongnu.org >>>> CC: Patrik Janoušek <pj@patrikjanousek.cz>, lmatejka@kiv.zcu.cz >>>> >>>> >>>> >>>> Currently, QEMU doesn't support persistent dirty bitmaps for raw >>>> format >>>> and also dirty bitmaps are for internal use only, and cannot be >>>> accessed >>>> using third-party applications. These facts are very limiting >>>> in case someone would like to develop their own backup tool becaouse >>>> without access to the dirty bitmap it would be possible to implement >>>> only full backups. And without persistent dirty bitmaps, it wouldn't >>>> be possible to keep track of changed data after QEMU is restarted. And >>>> this is exactly what I do as a part of my bachelor thesis. I've >>>> developed a tool that is able to create incremental backups of drives >>>> in raw format that are LVM volumes (ability to create snapshot is >>>> required). >>> >>> Similarly to what Vladimir has said already, the thing is that >>> conceptually I can see no difference between having a raw image with >>> the bitmaps stored in some other file, i.e.: >>> >>> { "driver": "raw", >>> "dirty-bitmaps": [ { >>> "filename": "sdc1.bitmap", >>> "persistent": true >>> } ], >>> "file": { >>> "driver": "file", >>> "filename": "/dev/sdc1" >>> } } >>> >>> And having a qcow2 image with the raw data stored in some other file, >>> i.e.: >>> >>> { "driver": "qcow2", >>> "file": { >>> "driver": "file", >>> "filename": "sdc1.metadata" >>> }, >>> "data-file": { >>> "driver": "file", >>> "filename": "/dev/sdc1" >>> } } >>> >>> (Where sdc1.metadata is a qcow2 file created with >>> “data-file=/dev/sdc1,data-file-raw=on”.) >>> >>> To use persistent bitmaps with raw images, you need to add metadata >>> (namely, the bitmaps). Why not store that metadata in a qcow2 file? >>> >>> Max >> >> So if I understand it correctly. I can configure dirty bitmaps in the >> latest version of QEMU to be persistently stored in some other file. >> Because even Proxmox Backup Server can't perform an incremental backup >> after restarting QEMU, and that means something to me. I think they >> would implement it if it was that simple. >> >> Could you please send me simple example on how to configure (via command >> line args) one raw format drive that can store dirty bitmaps >> persistently in other qcow2 file? I may be missing something, but I >> thought QEMU couldn't do it, because Proxmox community wants this >> feature for a long time. > > One trouble is that if you use qemu-img create to create the qcow2 > image, it will always create an empty image, and so if use pass > data_file to it, it will empty the existing raw image: > > $ cp ~/tmp/arch.iso raw.img # Just some Arch Linux ISO > > $ qemu-img create \ > -f qcow2 \ > -o data_file=raw.img,data_file_raw=on,preallocation=metadata \ > metadata.qcow2 \ > $(stat -c '%s' raw.img) > Formatting 'metadata.qcow2', fmt=qcow2 cluster_size=65536 > preallocation=metadata compression_type=zlib size=687865856 > data_file=raw.img data_file_raw=on lazy_refcounts=off refcount_bits=16 > > (If you check raw.img at this point, you’ll find that it’s empty, so > you need to copy it from the source again:) > > $ cp ~/tmp/arch.iso raw.img > > Now if you use metadata.qcow2, the image data will actually all be > stored in raw.img. > > > To get around the “creating metadata.qcow2 clears raw.img” problem, > you can either create a temporary empty image of the same size as > raw.img that you pass to qemu-img create, and then you use qemu-img > amend to change the data-file pointer (which will not overwrite the > new data-file’s contents): > > $ qemu-img create -f raw tmp.raw $(stat -c '%s' raw.img) > > $ qemu-img create \ > -f qcow2 \ > -o data_file=tmp.img,data_file_raw=on,preallocation=metadata \ > metadata.qcow2 \ > $(stat -c '%s' raw.img) > Formatting 'metadata.qcow2', fmt=qcow2 cluster_size=65536 > preallocation=metadata compression_type=zlib size=687865856 > data_file=tmp.img data_file_raw=on lazy_refcounts=off refcount_bits=16 > > $ qemu-img amend -o data_file=raw.img metadata.qcow2 > > $ rm tmp.img > > > Or you use the blockdev-create job to create the qcow2 image (because > contrary to qemu-img create, that will not clear the data file): > > $ touch metadata.qcow2 > > (Note that in the following QMP communication, what I sent and what > qemu replies is mixed. Everything that begins with '{ "execute"' is > from me, everything else from qemu. The number 687865856 is the size > of raw.img in bytes.) > > $ qemu-system-x86_64 -qmp stdio \ > -blockdev \ > '{ "node-name": "metadata-file", > "driver": "file", > "filename": "metadata.qcow2" }' \ > -blockdev \ > '{ "node-name": "data-file", > "driver": "file", > "filename": "raw.img" }' > {"QMP": {"version": {"qemu": {"micro": 0, "minor": 1, "major": 5}, > "package": "qemu-5.1.0-9.fc33"}, "capabilities": ["oob"]}} > > { "execute": "qmp_capabilities" } > {"return": {}} > > { "execute": "blockdev-create", > "arguments": { > "job-id": "create", > "options": { > "driver": "qcow2", > "file": "metadata-file", > "data-file": "data-file", > "data-file-raw": true, > "preallocation": "metadata", > "size": 687865856 > } > } > } > {"timestamp": {"seconds": 1616414002, "microseconds": 836899}, > "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": > "create"}} > {"timestamp": {"seconds": 1616414002, "microseconds": 837076}, > "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": > "create"}} > {"return": {}} > {"timestamp": {"seconds": 1616414002, "microseconds": 870997}, > "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id": > "create"}} > {"timestamp": {"seconds": 1616414002, "microseconds": 871099}, > "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id": > "create"}} > {"timestamp": {"seconds": 1616414002, "microseconds": 871185}, > "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": > "create"}} > > { "execute": "job-dismiss", "arguments": { "id": "create" } } > {"timestamp": {"seconds": 1616414022, "microseconds": 202880}, > "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "create"}} > {"return": {}} > > { "execute": "quit" } > {"return": {}} > {"timestamp": {"seconds": 1616414028, "microseconds": 56457}, "event": > "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} > > > In any case, in the end you get a metadata.qcow2 that holds the > metadata and points to raw.img for its data. So you can use it like > this: > > $ qemu-system-x86_64 -enable-kvm -m 512 \ > -blockdev \ > '{ "node-name": "node0", > "driver": "qcow2", > "file": { > "driver": "file", > "filename": "metadata.qcow2" > } }' \ > -device ide-cd,drive=node0 \ > -qmp stdio > {"QMP": {"version": {"qemu": {"micro": 0, "minor": 1, "major": 5}, > "package": "qemu-5.1.0-9.fc33"}, "capabilities": ["oob"]}} > > { "execute": "qmp_capabilities" } > {"return": {}} > > { "execute": "block-dirty-bitmap-add", > "arguments": { > "node": "node0", > "name": "bmap0", > "persistent": true > } > } > {"return": {}} > > { "execute": "quit" } > {"return": {}} > {"timestamp": {"seconds": 1616414627, "microseconds": 928250}, > "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}} > > $ qemu-img info metadata.qcow2 > image: metadata.qcow2 > file format: qcow2 > virtual size: 656 MiB (687865856 bytes) > disk size: 452 KiB > cluster_size: 65536 > Format specific information: > compat: 1.1 > compression type: zlib > lazy refcounts: false > bitmaps: > [0]: > flags: > [0]: auto > name: bmap0 > granularity: 65536 > refcount bits: 16 > data file: raw.img > data file raw: true > corrupt: false > > So the bitmap is now in metadata.qcow2, and as the disk size > indicates, all the data is still in raw.img. > > > I hope the above helps you. Sorry if it’s confusing, especially the > first part where I’m like “The obvious way to create metadata.qcow2 > will delete your data, so here are two alternatives that are weird but > do what you want.” > > Max Thank you for exhausting answer. I would also like to thank others for their comments. I will try to rethink the developed solution and consider its modifications. Maybe in the future I'll come up with something you might like. Regards, Patrik ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Fwd: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi 2021-03-22 11:27 ` Patrik Janoušek 2021-03-22 12:06 ` Max Reitz @ 2021-03-22 12:44 ` Fabian Grünbichler 1 sibling, 0 replies; 20+ messages in thread From: Fabian Grünbichler @ 2021-03-22 12:44 UTC (permalink / raw) To: Eric Blake, John Snow, Kevin Wolf, Max Reitz, Patrik Janoušek, Vladimir Sementsov-Ogievskiy Cc: lmatejka, Qemu-block, qemu-devel On March 22, 2021 12:27 pm, Patrik Janoušek wrote: > On 3/22/21 11:48 AM, Max Reitz wrote: >> Hi, >> >> On 20.03.21 11:01, Patrik Janoušek wrote: >>> I'm sorry, but I forgot to add you to the cc, so I'm forwarding the >>> patch to you additionally. I don't want to spam the mailing list >>> unnecessarily. >> >> I think it’s better to still CC the list. It’s so full of mail, one >> more won’t hurt. :) >> >> (Re-adding qemu-block and qemu-devel, because the discussion belongs >> on the list(s).) >> >>> -------- Forwarded Message -------- >>> Subject: [PATCH 0/2] block/raw: implemented persistent dirty >>> bitmap and ability to dump bitmap content via qapi >>> Date: Sat, 20 Mar 2021 10:32:33 +0100 >>> From: Patrik Janoušek <pj@patrikjanousek.cz> >>> To: qemu-devel@nongnu.org >>> CC: Patrik Janoušek <pj@patrikjanousek.cz>, lmatejka@kiv.zcu.cz >>> >>> >>> >>> Currently, QEMU doesn't support persistent dirty bitmaps for raw format >>> and also dirty bitmaps are for internal use only, and cannot be accessed >>> using third-party applications. These facts are very limiting >>> in case someone would like to develop their own backup tool becaouse >>> without access to the dirty bitmap it would be possible to implement >>> only full backups. And without persistent dirty bitmaps, it wouldn't >>> be possible to keep track of changed data after QEMU is restarted. And >>> this is exactly what I do as a part of my bachelor thesis. I've >>> developed a tool that is able to create incremental backups of drives >>> in raw format that are LVM volumes (ability to create snapshot is >>> required). >> >> Similarly to what Vladimir has said already, the thing is that >> conceptually I can see no difference between having a raw image with >> the bitmaps stored in some other file, i.e.: >> >> { "driver": "raw", >> "dirty-bitmaps": [ { >> "filename": "sdc1.bitmap", >> "persistent": true >> } ], >> "file": { >> "driver": "file", >> "filename": "/dev/sdc1" >> } } >> >> And having a qcow2 image with the raw data stored in some other file, >> i.e.: >> >> { "driver": "qcow2", >> "file": { >> "driver": "file", >> "filename": "sdc1.metadata" >> }, >> "data-file": { >> "driver": "file", >> "filename": "/dev/sdc1" >> } } >> >> (Where sdc1.metadata is a qcow2 file created with >> “data-file=/dev/sdc1,data-file-raw=on”.) >> >> To use persistent bitmaps with raw images, you need to add metadata >> (namely, the bitmaps). Why not store that metadata in a qcow2 file? >> >> Max > > So if I understand it correctly. I can configure dirty bitmaps in the > latest version of QEMU to be persistently stored in some other file. > Because even Proxmox Backup Server can't perform an incremental backup > after restarting QEMU, and that means something to me. I think they > would implement it if it was that simple. the main reason we haven't implemented something like that (yet) is not that it is technically difficult, but that we have no way to safeguard against external manipulation of the (raw) image: - VM is running, backup happens, bitmap represents the delta since this backup - VM is stopped - user/admin does something to VM disk image, believing such an action is safe because VM is not running - VM is started again - bitmap does not contain the manual changes - VM is running, backup happens, backup is inconsistent with actual data stored in image because of persistence, this error is now carried forward indefinitely (it might self-correct at some point if all the invalid stuff has been dirtied by the guest). while it is of course possible to argue that this is entirely the user's fault, it looks like it's the backup software/hypervisor's fault (no indication anything is wrong, backup data is bogus). it might happen at some point as opt-in feature with all the warnings associated with potentially dangerous features, but for now we are okay with just carrying the bitmap as long as the VM instance is running (including migrations), and having to re-read and chunk/hash the disks for fresh instances. the latter is expensive, but less expensive than having to explain to users why their backups are bogus. > > Could you please send me simple example on how to configure (via command > line args) one raw format drive that can store dirty bitmaps > persistently in other qcow2 file? I may be missing something, but I > thought QEMU couldn't do it, because Proxmox community wants this > feature for a long time. > > Patrik > > > > > > ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2021-03-22 23:33 UTC | newest] Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-03-20 9:32 [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi Patrik Janoušek 2021-03-20 9:32 ` [PATCH 1/2] block/raw: added support of persistent dirty bitmaps Patrik Janoušek 2021-03-22 8:41 ` Vladimir Sementsov-Ogievskiy 2021-03-22 10:18 ` Patrik Janoušek 2021-03-22 10:46 ` Vladimir Sementsov-Ogievskiy 2021-03-22 11:18 ` Vladimir Sementsov-Ogievskiy 2021-03-22 11:36 ` Patrik Janoušek 2021-03-22 20:27 ` Lubos Matejka 2021-03-22 15:43 ` Kevin Wolf 2021-03-20 9:32 ` [PATCH 2/2] qapi: implementation of the block-dirty-bitmap-dump command Patrik Janoušek 2021-03-22 9:02 ` Vladimir Sementsov-Ogievskiy 2021-03-22 15:12 ` Kevin Wolf 2021-03-22 8:29 ` [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi Vladimir Sementsov-Ogievskiy 2021-03-22 8:57 ` Patrik Janoušek 2021-03-22 14:53 ` Kevin Wolf [not found] ` <856ca6ba-3871-068f-f821-269c40a5a4d5@patrikjanousek.cz> 2021-03-22 10:48 ` Fwd: " Max Reitz 2021-03-22 11:27 ` Patrik Janoušek 2021-03-22 12:06 ` Max Reitz 2021-03-22 21:45 ` Patrik Janoušek 2021-03-22 12:44 ` Fabian Grünbichler
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.