* [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps
@ 2017-05-03 12:25 Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 01/25] specs/qcow2: fix bitmap granularity qemu-specific note Vladimir Sementsov-Ogievskiy
` (25 more replies)
0 siblings, 26 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Hi all!
There is a new update of qcow2-bitmap series - v18.
web: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=qcow2-bitmap-v18
git: https://src.openvz.org/scm/~vsementsov/qemu.git (tag qcow2-bitmap-v18)
v18:
rebased on master (sorry for v17)
08: contextual: qcow2_do_open is changed instead of qcow2_open
rename s/need_update_header/update_header/ in qcow2_do_open, to not do it in 10
save r-b's by Max and John
09: new patch
10: load_bitmap_data: do not clear bitmap parameter - it should be already cleared
(it actually created before single load_bitmap_data() call)
if some bitmaps are loaded, but we can't write the image (it is readonly
or inactive), so we can't mark bitmaps "in use" in the image, mark
corresponding BdrvDirtyBitmap read-only.
change error_setg to error_setg_errno for "Can't update bitmap directory"
no needs to rename s/need_update_header/update_header/ here, as it done in 08
13: function bdrv_has_persistent_bitmaps becomes bdrv_has_changed_persistent_bitmaps,
to handle readonly field.
14: declaration moved to the bottom of .h, save r-b's
15: firstly check bdrv_has_changed_persistent_bitmaps and only then fail on !can_write, and then QSIMPLEQ_INIT(&drop_tables)
skip readonly bitmaps in saving loop
18: remove '#optional', 2.9 -> 2.10, save r-b's
19: remove '#optional', 2.9 -> 2.10, save r-b's
20: 2.9 -> 2.10, save r-b's
21: add check of read-only image open, drop r-b's
24: add comment to qapi/block-core.json, that block-dirty-bitmap-add removes bitmap
from storage. r-b's by Max and John saved
v17:
08: add r-b's by Max and John
09: clear unknown autoclear features from BDRVQcow2State before calling
qcow2_load_autoloading_dirty_bitmaps(), and also do not extra update
header if it is updated by qcow2_load_autoloading_dirty_bitmaps().
11: new patch, splitted out from 16
12: rewrite commit message
14: changing bdrv_close moved to separate new patch 11
s/1/1ULL/ ; s/%u/%llu/ for two errors
16: s/No/Not/
add Max's r-b
24: new patch
v16:
mostly by Kevin's comments:
07: just moved here, as preparation for merging refcounts to 08
08: move "qcow2-bitmap: refcounts" staff to this patch, to not break qcow2-img check
drop r-b's
move necessary supporting static functions to this patch too (to not break compilation)
fprintf -> error_report
other small changes with error messages and comments
code style
for qcow2-bitmap.c:
'include "exec/log.h"' was dropped
s/1/(1U << 0) for BME_FLAG_IN_USE
add BME_TABLE_ENTRY_FLAG_ALL_ONES to replace magic 1
don't check 'cluster_size <= 0' in check_table_entry
old "[PATCH v15 08/25] block: introduce auto-loading bitmaps" was dropped
09: was "[PATCH v15 09/25] qcow2: add .bdrv_load_autoloading_dirty_bitmaps"
drop r-b's
some staff was moved to 08
update_header_sync - error on bdrv_flush fail
rename disk_sectors_in_bitmap_cluster to sectors_covered_by_bitmap_cluster
and adjust comment.
so, variable for storing this function result: s/dsc/sbc/
s/1/BME_TABLE_ENTRY_FLAG_ALL_ONES/
also, as Qcow2BitmapTable already introduced in 08,
s/table_offset/table.offset/ and s/table_size/table.size, etc..
update_ext_header_and_dir_in_place: add comments, add additional
update_header_sync, to reduce indeterminacy in case of error.
call qcow2_load_autoloading_dirty_bitmaps directly from qcow2_open
no .bdrv_load_autoloading_dirty_bitmaps
11: drop bdrv_store_persistent_dirty_bitmaps at all.
drop r-b's
13: rename patch, rewrite commit msg
drop r-b's
move bdrv_release_named_dirty_bitmaps in bdrv_close() after drv->bdrv_close()
Qcow2BitmapTable is already introduced, so no
s/table_offset/table.offset/ and s/table_size/table.size, etc.. here
old 25 with check_constraints_on_bitmap() improvements merged here (Eric)
like in 09, s/dsc/sbc/ and s/disk_sectors_in_bitmap_cluster/sectors_covered_by_bitmap_cluster/
no .bdrv_store_persistent_dirty_bitmaps
call qcow2_store_persistent_dirty_bitmaps directly from qcow2_inactivate
15: corresponding part of 25 merged here. Add John's r-b, as 25 was also reviewed by John.
16: add John's r-b
v15:
13,14: add John's r-b
15: qcow2_can_store_new_dirty_bitmap:
- check max dirty bitmaps and bitmap directory overhead
- switch to error_prepend
rm Max's r-b
not add John's r-b
17-24: add John's r-b
25: changed because 15 changed,
not add John's r-b
v14:
07: use '|=' to update need_update_header
add John's r-b
add Max's r-b
09: remove unused bitmap_table_to_cpu()
left Max's r-b, hope it's ok
add John's r-b
10: remove extra new line
add John's r-b
11: add John's r-b
12: add John's r-b
13: small fixes by John's review:
- remove weird g_free of NULL pointer from
if (tb == NULL) {
g_free(tb);
return -EINVAL;
}
- remove extra comment "/* errp is already set */"
- s/"Too large bitmap directory"/"Bitmap directory is too large"/
left Max's r-b, hope you don't mind
22: add Max's r-b
23: add Max's r-b
24: add Max's r-b
25: new patch to improve error message on check_constraints_on_bitmap fail
v13: Just a fix for style checker.
13: line over 80
14: line over 80
22: s/if () \n{/if () {/
v12:
07: do not update header in qcow2_read_extensions, instead do it in the
end of qcow2_open, where it is updated also to clear unknown
autoclear features.
Wrong bdrv_is_root_node used instead of bdrv_is_read_only is fixed
automatically.
08: add Max's r-b
09: contextual: s/raw_bsd/raw-format/
qcow2-bitmap.c: copyright s/2016/2017/
fix compilation: move update_header_sync() to this patch
add Max's r-b
13: update_header_sync() is moved to 09
add Max's r-b
15: add Max's r-b
16: As qmp-commands.txt is removed from master, remove it here too.
Sentence "For now only Qcow2 disks support persistent bitmaps.
Default is false." moved to qapi/block-core.json.
Hope that is OK, Max's r-b is not dropped.
17: qmp-commands.txt deleted, r-b is not dropped too.
18: s/is failed/has failed/, add Max's r-b
19: copyright: s/2016/2017/
21: drop "res->corruptions > 0 => ret = -EINVAL", add Max's r-b
22: actually, patch is replaced with a new one, however some parts are the same.
instead of changing bdrv_release_dirty_bitmap behaviour, create new
bdrv_remove_persistent_dirty_bitmap
23: improve comment, about not-exists is not an error
24: new patch
Old 24 (qcow2-bitmap: cache bitmap list in BDRVQcow2State) will be sent
separately, to be applied after these series.
Also: about bdrv_dirty_bitmap_set_persistance: I've decided not change its behaviour
for now and just call bdrv_remove_persistent_dirty_bitmap from
qmp_block_dirty_bitmap_remove. Reasons:
1. Do not change reviewed part.
2. I'm not sure, that this complication of BdrvDirtyBitmap.persistent semantics
is good (current semantics are just: .persistent means that bitmap should be
saved on disk close). .persistent actually is not very related to "is there
stored version of the bitmap in the image".
3. It may be done later.
4. It is not actually needed for now, as the only usage is dropping bitmap in
bdrv_remove_persistent_dirty_bitmap. May be in future it will be needed,
when we will have qmp interfaces for finer control of persistent dirty bitmaps.
v11:
Fix automatic build test fail.
18: wording - "ASCII representation of SHA256 ..."
24: fix redeclaration error. (This is still RFC, may be not needed patch,
see description in v10 below).
v10:
07: rm John's r-b
not add Max's r-b
fix subject s/dirty bitmaps/bitmaps
improve WARNING message
remove header ext if autoclear bit is unset, through qcow2_updata_header() - r-b blocking change
08: move bdrv_load_autoloading_dirty_bitmaps under asserts block (Max)
fix typo in commit message
move bdrv_load_autoloading_dirty_bitmaps after asserts
John's r-b
09: rewrite check_dir_entry
remove extra feature cluster_size=0 from check_table_entry
which clears autoclear bit before trying to update bitmap directory
bitmap_list_store - do not free old clusters if in_place=true
changes in comments, fix indents
add functions
bitmap_list_count
update_ext_header_and_dir_in_place (unset autoclear bit before trying to
modify bitmap directory)
(for usage in qcow2_load_autoloading_dirty_bitmaps)
11: add node name to error report
Max's r-b
13: add type Qcow2BitmapTable
move from bm.table_* to bm.table.*
rewrite check_constraints_on_bitmap
flush(bs->file) instead of flush(bs) in update_ext_header_and_dir
assert(DIV_ROUND_UP(bm_size, dsc) == tb_size);
and assert(write_size <= s->cluster_size); in store_bitmap_data
fix: s/tb_size/tb_size * sizeof(tb[0]) in store_bitmap
in qcow2_store_persistent_dirty_bitmaps:
fail if !can_write
fix counting of new_nb_bitmaps and new_dir_size
free old clusters _after_ successful directory update (aggregate old bitmap
tables into drop_tables)
2 14: rename to can_store_new_dirty_bitmap
Max's r-b
15: rename to can_store_new_dirty_bitmap
rm extra !!
16: Max's r-b
rename to can_store_new_dirty_bitmap
indenting
Since s/2.8/2.9
17: Max's r-b
Since s/2.8/2.9
18: hashing can fail in comment for qapi
Since s/2.8/2.9
remove dead comment
19: indentation
Max's r-b
20: Max's r-b
21: fix error handling
return 0 if nb_bitmaps == 0
new patches:
22-23: remove bitmap from file on bdrv_release_dirty_bitmap
24: experimental, RFC: cache bitmap list to bs, to not reload it from file.
I'm not sure that is needed now, but if you want I can merge it to earlier
patches.
v9:
rebase on master!
01-04,06,07: John's r-b
03: rewording in comment ("The concurrent resetting ..."), John's r-b
07: reword error messages
commit message: dirty bitmap -> bitmap
09: fix uninitialized ret in bitmap_list_store() (fix compilation warning)
18: fix compilation of tests/test-hbitmap
also, change constants names for qcow2: DIRTY_BITMAP -> BITMAP (as it is not
dirty bitmaps extension, but just bitmaps extension), QCOW -> QCOW2 (bitmaps
are only for qcow2)
v8:
Patches 01-06 are mostly unchanged, except for spelling and wording.
02 - add Eric's r-b
03,04 - add Max's r-b
07-09 is refactored "bitmap-read" part of the feature. Main changes:
- introduce bitmap list - normal list, which represents bitmap directory.
Function bitmap_list_load loads bitmap directory, checks everything and
creates normal list.
- introduce .bdrv_load_autoloading_dirty_bitmaps : instead of loading bitmaps
in qcow2_open, lets load them only in bdrv_open, to avoid reloading in
bdrv_snapshot_goto
- do not delete bitmaps after read, but mark them IN_USE
10 has contextual changes and rewording of comment. I've added Max's r-b, hope it's ok.
11: add error_report("Persistent bitmaps are lost") in case of failed bitmap store
12: add Max's r-b
13 is refactored "bitmap-store" part of the feature. see 07-09 description
- for now I just free old clusters and allocate new. This will be improved
with a separate patch.
patch about "delete bitmaps on truncate" is removed. This case will be handled later.
IN_USE, autoclear, check-constraints things are merged into other patches.
14-15 are mew patches, to early check possibility of creating persistent bitmap with
specified name and granularity in specified BDS
16-17: spelling, rewording, indenting, tiny code simplifying, check can_store (by 14-15),
s/if (autoload && !persistent)/if (has_autoload && !persistent)/,
rebase (deleted qmp-commands.hx)
18: alternative to md5 in query-block:
- separated qmp command -x-debug-block-dirty-bitmap-sha256
- as adviced by Daniel P. Berrange in my parallel thread:
- sha256 instead of md5
- use qcrypto_hash_* instead of GChecksum
- fix bug =) (size was wrong in hbitmap_md5)
19: s/3999/3fff/, use x-debug-block-dirty-bitmap-sha256
20: new patch to rename and publish inc_refcounts
21: some fixes and refactoring mostyly by Max's comments.
Max, Eric, great tanks for your review!
I hope, I've covered most of your comments by this update.
TODO:
- handle reopening image RO->RW and incoming migration, set IN_USE for existing loaded bitmaps
in these cases.
- reuse old, already allocated data clusters for bitmaps storing
- truncate bitmaps in the image on truncate
v7:
https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fqcow2-bitmap-v7
based on block-next (https://github.com/XanClic/qemu/commits/block-next)
- a lot of refactoring and reordering of patches.
- dead code removed (bdrv_dirty_bitmap_load, etc.)
- do not maintain extra data for now
- do not store dirty bitmap directory in memory
(as we use it seldom, we can just reread if needed)
By Kevin's review:
01 - commit message changed: fix->improvement (as it was not a bug)
03 - r-b
04 - r-b
05 - add 21 patch to fix spec, also, removed all (I hope) mentions of
"Bitmap Header", switch to one unified name for it - "Bitmap
Directory Entry", to avoid misunderstanding with Qcow2 header.
(also, add patch 22, to fix it in spec)
v6.06 - improve for_each_dir_entry loop, reorder patches, other small fixes
v6.07 ~> v7.09 - dead code removed, I've moved to one function
.bdrv_store_persistent_bitmaps and have wrapper and callback in one
patch (with also some other staff. If it not ok, I can split them)
v6.08 - about keeping bitmap directory instead of bitmap list: no I don't keep
it at all.
v6.16 - old bdrv_ bitmap-storing related functions are removed. The new one is
bdrv_store_persistent_bitmaps.
v6:
https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fqcow2-bitmap-v6
based on block-next (https://github.com/XanClic/qemu/commits/block-next)
There are a lot of changes, reorderings and additions in comparement with v5.
One principal thing: now bitmaps are removed from image after loading instead
of marking them in_use. It is simpler and we do not need to store superfluous data.
Also, we are no more interested in command line interface to dirty bitmaps.
So it is dropped. If someone needs it I can add it later.
Vladimir Sementsov-Ogievskiy (25):
specs/qcow2: fix bitmap granularity qemu-specific note
specs/qcow2: do not use wording 'bitmap header'
hbitmap: improve dirty iter
tests: add hbitmap iter test
block: fix bdrv_dirty_bitmap_granularity signature
block/dirty-bitmap: add deserialize_ones func
qcow2-refcount: rename inc_refcounts() and make it public
qcow2: add bitmaps extension
block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
qcow2: autoloading dirty bitmaps
block/dirty-bitmap: add autoload field to BdrvDirtyBitmap
block: bdrv_close: release bitmaps after drv->bdrv_close
block: introduce persistent dirty bitmaps
block/dirty-bitmap: add bdrv_dirty_bitmap_next()
qcow2: add persistent dirty bitmaps support
block: add bdrv_can_store_new_dirty_bitmap
qcow2: add .bdrv_can_store_new_dirty_bitmap
qmp: add persistent flag to block-dirty-bitmap-add
qmp: add autoload parameter to block-dirty-bitmap-add
qmp: add x-debug-block-dirty-bitmap-sha256
iotests: test qcow2 persistent dirty bitmap
block/dirty-bitmap: add bdrv_remove_persistent_dirty_bitmap
qcow2: add .bdrv_remove_persistent_dirty_bitmap
qmp: block-dirty-bitmap-remove: remove persistent
block: release persistent bitmaps on inactivate
block.c | 32 +-
block/Makefile.objs | 2 +-
block/dirty-bitmap.c | 125 +++-
block/qcow2-bitmap.c | 1396 ++++++++++++++++++++++++++++++++++++++++++
block/qcow2-refcount.c | 59 +-
block/qcow2.c | 149 ++++-
block/qcow2.h | 41 ++
blockdev.c | 71 ++-
docs/specs/qcow2.txt | 8 +-
include/block/block.h | 3 +
include/block/block_int.h | 8 +
include/block/dirty-bitmap.h | 25 +-
include/qemu/hbitmap.h | 49 +-
qapi/block-core.json | 42 +-
tests/Makefile.include | 2 +-
tests/qemu-iotests/165 | 105 ++++
tests/qemu-iotests/165.out | 5 +
tests/qemu-iotests/group | 1 +
tests/test-hbitmap.c | 19 +
util/hbitmap.c | 51 +-
20 files changed, 2119 insertions(+), 74 deletions(-)
create mode 100644 block/qcow2-bitmap.c
create mode 100755 tests/qemu-iotests/165
create mode 100644 tests/qemu-iotests/165.out
--
2.11.1
^ permalink raw reply [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 01/25] specs/qcow2: fix bitmap granularity qemu-specific note
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 02/25] specs/qcow2: do not use wording 'bitmap header' Vladimir Sementsov-Ogievskiy
` (24 subsequent siblings)
25 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
docs/specs/qcow2.txt | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
index 80cdfd0e91..dda53dd2a3 100644
--- a/docs/specs/qcow2.txt
+++ b/docs/specs/qcow2.txt
@@ -472,8 +472,7 @@ Structure of a bitmap directory entry:
17: granularity_bits
Granularity bits. Valid values: 0 - 63.
- Note: Qemu currently doesn't support granularity_bits
- greater than 31.
+ Note: Qemu currently supports only values 9 - 31.
Granularity is calculated as
granularity = 1 << granularity_bits
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 02/25] specs/qcow2: do not use wording 'bitmap header'
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 01/25] specs/qcow2: fix bitmap granularity qemu-specific note Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 03/25] hbitmap: improve dirty iter Vladimir Sementsov-Ogievskiy
` (23 subsequent siblings)
25 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
A bitmap directory entry is sometimes called a 'bitmap header'. This
patch leaves only one name - 'bitmap directory entry'. The name 'bitmap
header' creates misunderstandings with 'qcow2 header' and 'qcow2 bitmap
header extension' (which is extension of qcow2 header)
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
docs/specs/qcow2.txt | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
index dda53dd2a3..8874e8c774 100644
--- a/docs/specs/qcow2.txt
+++ b/docs/specs/qcow2.txt
@@ -201,7 +201,7 @@ The fields of the bitmaps extension are:
8 - 15: bitmap_directory_size
Size of the bitmap directory in bytes. It is the cumulative
- size of all (nb_bitmaps) bitmap headers.
+ size of all (nb_bitmaps) bitmap directory entries.
16 - 23: bitmap_directory_offset
Offset into the image file at which the bitmap directory
@@ -426,8 +426,7 @@ Each bitmap saved in the image is described in a bitmap directory entry. The
bitmap directory is a contiguous area in the image file, whose starting offset
and length are given by the header extension fields bitmap_directory_offset and
bitmap_directory_size. The entries of the bitmap directory have variable
-length, depending on the lengths of the bitmap name and extra data. These
-entries are also called bitmap headers.
+length, depending on the lengths of the bitmap name and extra data.
Structure of a bitmap directory entry:
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 03/25] hbitmap: improve dirty iter
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 01/25] specs/qcow2: fix bitmap granularity qemu-specific note Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 02/25] specs/qcow2: do not use wording 'bitmap header' Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 04/25] tests: add hbitmap iter test Vladimir Sementsov-Ogievskiy
` (22 subsequent siblings)
25 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Make dirty iter resistant to resetting bits in corresponding HBitmap.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
include/qemu/hbitmap.h | 26 ++++----------------------
util/hbitmap.c | 23 ++++++++++++++++++++++-
2 files changed, 26 insertions(+), 23 deletions(-)
diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
index 9239fe515e..6b04391266 100644
--- a/include/qemu/hbitmap.h
+++ b/include/qemu/hbitmap.h
@@ -256,10 +256,9 @@ void hbitmap_free(HBitmap *hb);
* the lowest-numbered bit that is set in @hb, starting at @first.
*
* Concurrent setting of bits is acceptable, and will at worst cause the
- * iteration to miss some of those bits. Resetting bits before the current
- * position of the iterator is also okay. However, concurrent resetting of
- * bits can lead to unexpected behavior if the iterator has not yet reached
- * those bits.
+ * iteration to miss some of those bits.
+ *
+ * The concurrent resetting of bits is OK.
*/
void hbitmap_iter_init(HBitmapIter *hbi, const HBitmap *hb, uint64_t first);
@@ -298,24 +297,7 @@ void hbitmap_free_meta(HBitmap *hb);
* Return the next bit that is set in @hbi's associated HBitmap,
* or -1 if all remaining bits are zero.
*/
-static inline int64_t hbitmap_iter_next(HBitmapIter *hbi)
-{
- unsigned long cur = hbi->cur[HBITMAP_LEVELS - 1];
- int64_t item;
-
- if (cur == 0) {
- cur = hbitmap_iter_skip_words(hbi);
- if (cur == 0) {
- return -1;
- }
- }
-
- /* The next call will resume work from the next bit. */
- hbi->cur[HBITMAP_LEVELS - 1] = cur & (cur - 1);
- item = ((uint64_t)hbi->pos << BITS_PER_LEVEL) + ctzl(cur);
-
- return item << hbi->granularity;
-}
+int64_t hbitmap_iter_next(HBitmapIter *hbi);
/**
* hbitmap_iter_next_word:
diff --git a/util/hbitmap.c b/util/hbitmap.c
index 35088e19c4..0b38817505 100644
--- a/util/hbitmap.c
+++ b/util/hbitmap.c
@@ -106,8 +106,9 @@ unsigned long hbitmap_iter_skip_words(HBitmapIter *hbi)
unsigned long cur;
do {
- cur = hbi->cur[--i];
+ i--;
pos >>= BITS_PER_LEVEL;
+ cur = hbi->cur[i] & hb->levels[i][pos];
} while (cur == 0);
/* Check for end of iteration. We always use fewer than BITS_PER_LONG
@@ -139,6 +140,26 @@ unsigned long hbitmap_iter_skip_words(HBitmapIter *hbi)
return cur;
}
+int64_t hbitmap_iter_next(HBitmapIter *hbi)
+{
+ unsigned long cur = hbi->cur[HBITMAP_LEVELS - 1] &
+ hbi->hb->levels[HBITMAP_LEVELS - 1][hbi->pos];
+ int64_t item;
+
+ if (cur == 0) {
+ cur = hbitmap_iter_skip_words(hbi);
+ if (cur == 0) {
+ return -1;
+ }
+ }
+
+ /* The next call will resume work from the next bit. */
+ hbi->cur[HBITMAP_LEVELS - 1] = cur & (cur - 1);
+ item = ((uint64_t)hbi->pos << BITS_PER_LEVEL) + ctzl(cur);
+
+ return item << hbi->granularity;
+}
+
void hbitmap_iter_init(HBitmapIter *hbi, const HBitmap *hb, uint64_t first)
{
unsigned i, bit;
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 04/25] tests: add hbitmap iter test
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (2 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 03/25] hbitmap: improve dirty iter Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 05/25] block: fix bdrv_dirty_bitmap_granularity signature Vladimir Sementsov-Ogievskiy
` (21 subsequent siblings)
25 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Test that hbitmap iter is resistant to bitmap resetting.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
tests/test-hbitmap.c | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/tests/test-hbitmap.c b/tests/test-hbitmap.c
index 23773d2051..1acb353889 100644
--- a/tests/test-hbitmap.c
+++ b/tests/test-hbitmap.c
@@ -909,6 +909,22 @@ static void hbitmap_test_add(const char *testpath,
hbitmap_test_teardown);
}
+static void test_hbitmap_iter_and_reset(TestHBitmapData *data,
+ const void *unused)
+{
+ HBitmapIter hbi;
+
+ hbitmap_test_init(data, L1 * 2, 0);
+ hbitmap_set(data->hb, 0, data->size);
+
+ hbitmap_iter_init(&hbi, data->hb, BITS_PER_LONG - 1);
+
+ hbitmap_iter_next(&hbi);
+
+ hbitmap_reset_all(data->hb);
+ hbitmap_iter_next(&hbi);
+}
+
int main(int argc, char **argv)
{
g_test_init(&argc, &argv, NULL);
@@ -966,6 +982,9 @@ int main(int argc, char **argv)
test_hbitmap_serialize_part);
hbitmap_test_add("/hbitmap/serialize/zeroes",
test_hbitmap_serialize_zeroes);
+
+ hbitmap_test_add("/hbitmap/iter/iter_and_reset",
+ test_hbitmap_iter_and_reset);
g_test_run();
return 0;
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 05/25] block: fix bdrv_dirty_bitmap_granularity signature
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (3 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 04/25] tests: add hbitmap iter test Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 06/25] block/dirty-bitmap: add deserialize_ones func Vladimir Sementsov-Ogievskiy
` (20 subsequent siblings)
25 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Make getter signature const-correct. This allows other functions with
const dirty bitmap parameter use bdrv_dirty_bitmap_granularity().
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
block/dirty-bitmap.c | 2 +-
include/block/dirty-bitmap.h | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 519737c8d3..186941cfc3 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -388,7 +388,7 @@ uint32_t bdrv_get_default_bitmap_granularity(BlockDriverState *bs)
return granularity;
}
-uint32_t bdrv_dirty_bitmap_granularity(BdrvDirtyBitmap *bitmap)
+uint32_t bdrv_dirty_bitmap_granularity(const BdrvDirtyBitmap *bitmap)
{
return BDRV_SECTOR_SIZE << hbitmap_granularity(bitmap->bitmap);
}
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 9dea14ba03..7cbe623ba7 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -29,7 +29,7 @@ void bdrv_disable_dirty_bitmap(BdrvDirtyBitmap *bitmap);
void bdrv_enable_dirty_bitmap(BdrvDirtyBitmap *bitmap);
BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs);
uint32_t bdrv_get_default_bitmap_granularity(BlockDriverState *bs);
-uint32_t bdrv_dirty_bitmap_granularity(BdrvDirtyBitmap *bitmap);
+uint32_t bdrv_dirty_bitmap_granularity(const BdrvDirtyBitmap *bitmap);
uint32_t bdrv_dirty_bitmap_meta_granularity(BdrvDirtyBitmap *bitmap);
bool bdrv_dirty_bitmap_enabled(BdrvDirtyBitmap *bitmap);
bool bdrv_dirty_bitmap_frozen(BdrvDirtyBitmap *bitmap);
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 06/25] block/dirty-bitmap: add deserialize_ones func
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (4 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 05/25] block: fix bdrv_dirty_bitmap_granularity signature Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 07/25] qcow2-refcount: rename inc_refcounts() and make it public Vladimir Sementsov-Ogievskiy
` (19 subsequent siblings)
25 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Add bdrv_dirty_bitmap_deserialize_ones() function, which is needed for
qcow2 bitmap loading, to handle unallocated bitmap parts, marked as
all-ones.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
block/dirty-bitmap.c | 7 +++++++
include/block/dirty-bitmap.h | 3 +++
include/qemu/hbitmap.h | 15 +++++++++++++++
util/hbitmap.c | 17 +++++++++++++++++
4 files changed, 42 insertions(+)
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 186941cfc3..90af37287f 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -499,6 +499,13 @@ void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap,
hbitmap_deserialize_zeroes(bitmap->bitmap, start, count, finish);
}
+void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap,
+ uint64_t start, uint64_t count,
+ bool finish)
+{
+ hbitmap_deserialize_ones(bitmap->bitmap, start, count, finish);
+}
+
void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap)
{
hbitmap_deserialize_finish(bitmap->bitmap);
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 7cbe623ba7..1e17729ac2 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -70,6 +70,9 @@ void bdrv_dirty_bitmap_deserialize_part(BdrvDirtyBitmap *bitmap,
void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap,
uint64_t start, uint64_t count,
bool finish);
+void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap,
+ uint64_t start, uint64_t count,
+ bool finish);
void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
#endif
diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
index 6b04391266..b52304ac29 100644
--- a/include/qemu/hbitmap.h
+++ b/include/qemu/hbitmap.h
@@ -229,6 +229,21 @@ void hbitmap_deserialize_zeroes(HBitmap *hb, uint64_t start, uint64_t count,
bool finish);
/**
+ * hbitmap_deserialize_ones
+ * @hb: HBitmap to operate on.
+ * @start: First bit to restore.
+ * @count: Number of bits to restore.
+ * @finish: Whether to call hbitmap_deserialize_finish automatically.
+ *
+ * Fills the bitmap with ones.
+ *
+ * If @finish is false, caller must call hbitmap_serialize_finish before using
+ * the bitmap.
+ */
+void hbitmap_deserialize_ones(HBitmap *hb, uint64_t start, uint64_t count,
+ bool finish);
+
+/**
* hbitmap_deserialize_finish
* @hb: HBitmap to operate on.
*
diff --git a/util/hbitmap.c b/util/hbitmap.c
index 0b38817505..0c1591a594 100644
--- a/util/hbitmap.c
+++ b/util/hbitmap.c
@@ -551,6 +551,23 @@ void hbitmap_deserialize_zeroes(HBitmap *hb, uint64_t start, uint64_t count,
}
}
+void hbitmap_deserialize_ones(HBitmap *hb, uint64_t start, uint64_t count,
+ bool finish)
+{
+ uint64_t el_count;
+ unsigned long *first;
+
+ if (!count) {
+ return;
+ }
+ serialization_chunk(hb, start, count, &first, &el_count);
+
+ memset(first, 0xff, el_count * sizeof(unsigned long));
+ if (finish) {
+ hbitmap_deserialize_finish(hb);
+ }
+}
+
void hbitmap_deserialize_finish(HBitmap *bitmap)
{
int64_t i, size, prev_size;
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 07/25] qcow2-refcount: rename inc_refcounts() and make it public
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (5 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 06/25] block/dirty-bitmap: add deserialize_ones func Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 08/25] qcow2: add bitmaps extension Vladimir Sementsov-Ogievskiy
` (18 subsequent siblings)
25 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
This is needed for the following patch, which will introduce refcounts
checking for qcow2 bitmaps.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
block/qcow2-refcount.c | 53 ++++++++++++++++++++++++++------------------------
block/qcow2.h | 4 ++++
2 files changed, 32 insertions(+), 25 deletions(-)
diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c
index 9e96f64c8b..3562697527 100644
--- a/block/qcow2-refcount.c
+++ b/block/qcow2-refcount.c
@@ -1325,11 +1325,10 @@ static int realloc_refcount_array(BDRVQcow2State *s, void **array,
*
* Modifies the number of errors in res.
*/
-static int inc_refcounts(BlockDriverState *bs,
- BdrvCheckResult *res,
- void **refcount_table,
- int64_t *refcount_table_size,
- int64_t offset, int64_t size)
+int qcow2_inc_refcounts_imrt(BlockDriverState *bs, BdrvCheckResult *res,
+ void **refcount_table,
+ int64_t *refcount_table_size,
+ int64_t offset, int64_t size)
{
BDRVQcow2State *s = bs->opaque;
uint64_t start, last, cluster_offset, k, refcount;
@@ -1422,8 +1421,9 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res,
nb_csectors = ((l2_entry >> s->csize_shift) &
s->csize_mask) + 1;
l2_entry &= s->cluster_offset_mask;
- ret = inc_refcounts(bs, res, refcount_table, refcount_table_size,
- l2_entry & ~511, nb_csectors * 512);
+ ret = qcow2_inc_refcounts_imrt(bs, res,
+ refcount_table, refcount_table_size,
+ l2_entry & ~511, nb_csectors * 512);
if (ret < 0) {
goto fail;
}
@@ -1461,8 +1461,9 @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res,
}
/* Mark cluster as used */
- ret = inc_refcounts(bs, res, refcount_table, refcount_table_size,
- offset, s->cluster_size);
+ ret = qcow2_inc_refcounts_imrt(bs, res,
+ refcount_table, refcount_table_size,
+ offset, s->cluster_size);
if (ret < 0) {
goto fail;
}
@@ -1514,8 +1515,8 @@ static int check_refcounts_l1(BlockDriverState *bs,
l1_size2 = l1_size * sizeof(uint64_t);
/* Mark L1 table as used */
- ret = inc_refcounts(bs, res, refcount_table, refcount_table_size,
- l1_table_offset, l1_size2);
+ ret = qcow2_inc_refcounts_imrt(bs, res, refcount_table, refcount_table_size,
+ l1_table_offset, l1_size2);
if (ret < 0) {
goto fail;
}
@@ -1544,8 +1545,9 @@ static int check_refcounts_l1(BlockDriverState *bs,
if (l2_offset) {
/* Mark L2 table as used */
l2_offset &= L1E_OFFSET_MASK;
- ret = inc_refcounts(bs, res, refcount_table, refcount_table_size,
- l2_offset, s->cluster_size);
+ ret = qcow2_inc_refcounts_imrt(bs, res,
+ refcount_table, refcount_table_size,
+ l2_offset, s->cluster_size);
if (ret < 0) {
goto fail;
}
@@ -1760,14 +1762,15 @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res,
}
res->corruptions_fixed++;
- ret = inc_refcounts(bs, res, refcount_table, nb_clusters,
- offset, s->cluster_size);
+ ret = qcow2_inc_refcounts_imrt(bs, res,
+ refcount_table, nb_clusters,
+ offset, s->cluster_size);
if (ret < 0) {
return ret;
}
/* No need to check whether the refcount is now greater than 1:
* This area was just allocated and zeroed, so it can only be
- * exactly 1 after inc_refcounts() */
+ * exactly 1 after qcow2_inc_refcounts_imrt() */
continue;
resize_fail:
@@ -1782,8 +1785,8 @@ resize_fail:
}
if (offset != 0) {
- ret = inc_refcounts(bs, res, refcount_table, nb_clusters,
- offset, s->cluster_size);
+ ret = qcow2_inc_refcounts_imrt(bs, res, refcount_table, nb_clusters,
+ offset, s->cluster_size);
if (ret < 0) {
return ret;
}
@@ -1823,8 +1826,8 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
}
/* header */
- ret = inc_refcounts(bs, res, refcount_table, nb_clusters,
- 0, s->cluster_size);
+ ret = qcow2_inc_refcounts_imrt(bs, res, refcount_table, nb_clusters,
+ 0, s->cluster_size);
if (ret < 0) {
return ret;
}
@@ -1845,16 +1848,16 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
return ret;
}
}
- ret = inc_refcounts(bs, res, refcount_table, nb_clusters,
- s->snapshots_offset, s->snapshots_size);
+ ret = qcow2_inc_refcounts_imrt(bs, res, refcount_table, nb_clusters,
+ s->snapshots_offset, s->snapshots_size);
if (ret < 0) {
return ret;
}
/* refcount data */
- ret = inc_refcounts(bs, res, refcount_table, nb_clusters,
- s->refcount_table_offset,
- s->refcount_table_size * sizeof(uint64_t));
+ ret = qcow2_inc_refcounts_imrt(bs, res, refcount_table, nb_clusters,
+ s->refcount_table_offset,
+ s->refcount_table_size * sizeof(uint64_t));
if (ret < 0) {
return ret;
}
diff --git a/block/qcow2.h b/block/qcow2.h
index f8aeb08794..5d1763dba1 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -520,6 +520,10 @@ int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset,
int64_t size);
int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset,
int64_t size);
+int qcow2_inc_refcounts_imrt(BlockDriverState *bs, BdrvCheckResult *res,
+ void **refcount_table,
+ int64_t *refcount_table_size,
+ int64_t offset, int64_t size);
int qcow2_change_refcount_order(BlockDriverState *bs, int refcount_order,
BlockDriverAmendStatusCB *status_cb,
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 08/25] qcow2: add bitmaps extension
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (6 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 07/25] qcow2-refcount: rename inc_refcounts() and make it public Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-29 15:34 ` Max Reitz
2017-05-03 12:25 ` [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap Vladimir Sementsov-Ogievskiy
` (17 subsequent siblings)
25 siblings, 1 reply; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Add bitmap extension as specified in docs/specs/qcow2.txt.
For now, just mirror extension header into Qcow2 state and check
constraints. Also, calculate refcounts for qcow2 bitmaps, to not break
qemu-img check.
For now, disable image resize if it has bitmaps. It will be fixed later.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
block/Makefile.objs | 2 +-
block/qcow2-bitmap.c | 439 +++++++++++++++++++++++++++++++++++++++++++++++++
block/qcow2-refcount.c | 6 +
block/qcow2.c | 124 +++++++++++++-
block/qcow2.h | 27 +++
5 files changed, 592 insertions(+), 6 deletions(-)
create mode 100644 block/qcow2-bitmap.c
diff --git a/block/Makefile.objs b/block/Makefile.objs
index ea955302c8..9efc6c49ea 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -1,5 +1,5 @@
block-obj-y += raw-format.o qcow.o vdi.o vmdk.o cloop.o bochs.o vpc.o vvfat.o dmg.o
-block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o
+block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o qcow2-bitmap.o
block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o
block-obj-y += qed-check.o
block-obj-y += vhdx.o vhdx-endian.o vhdx-log.o
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
new file mode 100644
index 0000000000..b8e472b3e8
--- /dev/null
+++ b/block/qcow2-bitmap.c
@@ -0,0 +1,439 @@
+/*
+ * Bitmaps for the QCOW version 2 format
+ *
+ * Copyright (c) 2014-2017 Vladimir Sementsov-Ogievskiy
+ *
+ * This file is derived from qcow2-snapshot.c, original copyright:
+ * Copyright (c) 2004-2006 Fabrice Bellard
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+
+#include "block/block_int.h"
+#include "block/qcow2.h"
+
+/* NOTICE: BME here means Bitmaps Extension and used as a namespace for
+ * _internal_ constants. Please do not use this _internal_ abbreviation for
+ * other needs and/or outside of this file. */
+
+/* Bitmap directory entry constraints */
+#define BME_MAX_TABLE_SIZE 0x8000000
+#define BME_MAX_PHYS_SIZE 0x20000000 /* restrict BdrvDirtyBitmap size in RAM */
+#define BME_MAX_GRANULARITY_BITS 31
+#define BME_MIN_GRANULARITY_BITS 9
+#define BME_MAX_NAME_SIZE 1023
+
+/* Bitmap directory entry flags */
+#define BME_RESERVED_FLAGS 0xfffffffcU
+
+/* bits [1, 8] U [56, 63] are reserved */
+#define BME_TABLE_ENTRY_RESERVED_MASK 0xff000000000001feULL
+#define BME_TABLE_ENTRY_OFFSET_MASK 0x00fffffffffffe00ULL
+#define BME_TABLE_ENTRY_FLAG_ALL_ONES (1ULL << 0)
+
+typedef struct QEMU_PACKED Qcow2BitmapDirEntry {
+ /* header is 8 byte aligned */
+ uint64_t bitmap_table_offset;
+
+ uint32_t bitmap_table_size;
+ uint32_t flags;
+
+ uint8_t type;
+ uint8_t granularity_bits;
+ uint16_t name_size;
+ uint32_t extra_data_size;
+ /* extra data follows */
+ /* name follows */
+} Qcow2BitmapDirEntry;
+
+typedef struct Qcow2BitmapTable {
+ uint64_t offset;
+ uint32_t size; /* number of 64bit entries */
+ QSIMPLEQ_ENTRY(Qcow2BitmapTable) entry;
+} Qcow2BitmapTable;
+
+typedef struct Qcow2Bitmap {
+ Qcow2BitmapTable table;
+ uint32_t flags;
+ uint8_t granularity_bits;
+ char *name;
+
+ QSIMPLEQ_ENTRY(Qcow2Bitmap) entry;
+} Qcow2Bitmap;
+typedef QSIMPLEQ_HEAD(Qcow2BitmapList, Qcow2Bitmap) Qcow2BitmapList;
+
+typedef enum BitmapType {
+ BT_DIRTY_TRACKING_BITMAP = 1
+} BitmapType;
+
+static int check_table_entry(uint64_t entry, int cluster_size)
+{
+ uint64_t offset;
+
+ if (entry & BME_TABLE_ENTRY_RESERVED_MASK) {
+ return -EINVAL;
+ }
+
+ offset = entry & BME_TABLE_ENTRY_OFFSET_MASK;
+ if (offset != 0) {
+ /* if offset specified, bit 0 is reserved */
+ if (entry & BME_TABLE_ENTRY_FLAG_ALL_ONES) {
+ return -EINVAL;
+ }
+
+ if (offset % cluster_size != 0) {
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+static int bitmap_table_load(BlockDriverState *bs, Qcow2BitmapTable *tb,
+ uint64_t **bitmap_table)
+{
+ int ret;
+ BDRVQcow2State *s = bs->opaque;
+ uint32_t i;
+ uint64_t *table;
+
+ assert(tb->size != 0);
+ table = g_try_new(uint64_t, tb->size);
+ if (table == NULL) {
+ return -ENOMEM;
+ }
+
+ assert(tb->size <= BME_MAX_TABLE_SIZE);
+ ret = bdrv_pread(bs->file, tb->offset,
+ table, tb->size * sizeof(uint64_t));
+ if (ret < 0) {
+ goto fail;
+ }
+
+ for (i = 0; i < tb->size; ++i) {
+ be64_to_cpus(&table[i]);
+ ret = check_table_entry(table[i], s->cluster_size);
+ if (ret < 0) {
+ goto fail;
+ }
+ }
+
+ *bitmap_table = table;
+ return 0;
+
+fail:
+ g_free(table);
+
+ return ret;
+}
+
+/*
+ * Bitmap List
+ */
+
+/*
+ * Bitmap List private functions
+ * Only Bitmap List knows about bitmap directory structure in Qcow2.
+ */
+
+static inline void bitmap_dir_entry_to_cpu(Qcow2BitmapDirEntry *entry)
+{
+ be64_to_cpus(&entry->bitmap_table_offset);
+ be32_to_cpus(&entry->bitmap_table_size);
+ be32_to_cpus(&entry->flags);
+ be16_to_cpus(&entry->name_size);
+ be32_to_cpus(&entry->extra_data_size);
+}
+
+static inline int calc_dir_entry_size(size_t name_size, size_t extra_data_size)
+{
+ return align_offset(sizeof(Qcow2BitmapDirEntry) +
+ name_size + extra_data_size, 8);
+}
+
+static inline int dir_entry_size(Qcow2BitmapDirEntry *entry)
+{
+ return calc_dir_entry_size(entry->name_size, entry->extra_data_size);
+}
+
+static inline const char *dir_entry_name_field(Qcow2BitmapDirEntry *entry)
+{
+ return (const char *)(entry + 1) + entry->extra_data_size;
+}
+
+static inline char *dir_entry_copy_name(Qcow2BitmapDirEntry *entry)
+{
+ const char *name_field = dir_entry_name_field(entry);
+ return g_strndup(name_field, entry->name_size);
+}
+
+static inline Qcow2BitmapDirEntry *next_dir_entry(Qcow2BitmapDirEntry *entry)
+{
+ return (Qcow2BitmapDirEntry *)((uint8_t *)entry + dir_entry_size(entry));
+}
+
+static int check_dir_entry(BlockDriverState *bs, Qcow2BitmapDirEntry *entry)
+{
+ BDRVQcow2State *s = bs->opaque;
+ uint64_t phys_bitmap_bytes;
+ int64_t len;
+
+ bool fail = (entry->bitmap_table_size == 0) ||
+ (entry->bitmap_table_offset == 0) ||
+ (entry->bitmap_table_offset % s->cluster_size) ||
+ (entry->bitmap_table_size > BME_MAX_TABLE_SIZE) ||
+ (entry->granularity_bits > BME_MAX_GRANULARITY_BITS) ||
+ (entry->granularity_bits < BME_MIN_GRANULARITY_BITS) ||
+ (entry->flags & BME_RESERVED_FLAGS) ||
+ (entry->name_size > BME_MAX_NAME_SIZE) ||
+ (entry->type != BT_DIRTY_TRACKING_BITMAP);
+
+ if (fail) {
+ return -EINVAL;
+ }
+
+ phys_bitmap_bytes = (uint64_t)entry->bitmap_table_size * s->cluster_size;
+ len = bdrv_getlength(bs);
+
+ if (len < 0) {
+ return len;
+ }
+
+ fail = (phys_bitmap_bytes > BME_MAX_PHYS_SIZE) ||
+ (len > ((phys_bitmap_bytes * 8) << entry->granularity_bits));
+
+ return fail ? -EINVAL : 0;
+}
+
+/*
+ * Bitmap List public functions
+ */
+
+static void bitmap_free(Qcow2Bitmap *bm)
+{
+ g_free(bm->name);
+ g_free(bm);
+}
+
+static void bitmap_list_free(Qcow2BitmapList *bm_list)
+{
+ Qcow2Bitmap *bm;
+
+ if (bm_list == NULL) {
+ return;
+ }
+
+ while ((bm = QSIMPLEQ_FIRST(bm_list)) != NULL) {
+ QSIMPLEQ_REMOVE_HEAD(bm_list, entry);
+ bitmap_free(bm);
+ }
+
+ g_free(bm_list);
+}
+
+static Qcow2BitmapList *bitmap_list_new(void)
+{
+ Qcow2BitmapList *bm_list = g_new(Qcow2BitmapList, 1);
+ QSIMPLEQ_INIT(bm_list);
+
+ return bm_list;
+}
+
+/* bitmap_list_load
+ * Get bitmap list from qcow2 image. Actually reads bitmap directory,
+ * checks it and convert to bitmap list.
+ */
+static Qcow2BitmapList *bitmap_list_load(BlockDriverState *bs, uint64_t offset,
+ uint64_t size, Error **errp)
+{
+ int ret;
+ BDRVQcow2State *s = bs->opaque;
+ uint8_t *dir, *dir_end;
+ Qcow2BitmapDirEntry *e;
+ uint32_t nb_dir_entries = 0;
+ Qcow2BitmapList *bm_list = NULL;
+
+ if (size == 0) {
+ error_setg(errp, "Requested bitmap directory size is zero");
+ return NULL;
+ }
+
+ if (size > QCOW2_MAX_BITMAP_DIRECTORY_SIZE) {
+ error_setg(errp, "Requested bitmap directory size is too big");
+ return NULL;
+ }
+
+ dir = g_try_malloc(size);
+ if (dir == NULL) {
+ error_setg(errp, "Failed to allocate space for bitmap directory");
+ return NULL;
+ }
+ dir_end = dir + size;
+
+ ret = bdrv_pread(bs->file, offset, dir, size);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "Failed to read bitmap directory");
+ goto fail;
+ }
+
+ bm_list = bitmap_list_new();
+ for (e = (Qcow2BitmapDirEntry *)dir;
+ e < (Qcow2BitmapDirEntry *)dir_end;
+ e = next_dir_entry(e))
+ {
+ Qcow2Bitmap *bm;
+
+ if ((uint8_t *)(e + 1) > dir_end) {
+ goto broken_dir;
+ }
+
+ if (++nb_dir_entries > s->nb_bitmaps) {
+ error_setg(errp, "More bitmaps found than specified in header"
+ " extension");
+ goto fail;
+ }
+ bitmap_dir_entry_to_cpu(e);
+
+ if ((uint8_t *)next_dir_entry(e) > dir_end) {
+ goto broken_dir;
+ }
+
+ if (e->extra_data_size != 0) {
+ error_setg(errp, "Bitmap extra data is not supported");
+ goto fail;
+ }
+
+ ret = check_dir_entry(bs, e);
+ if (ret < 0) {
+ error_setg(errp, "Bitmap '%.*s' doesn't satisfy the constraints",
+ e->name_size, dir_entry_name_field(e));
+ goto fail;
+ }
+
+ bm = g_new(Qcow2Bitmap, 1);
+ bm->table.offset = e->bitmap_table_offset;
+ bm->table.size = e->bitmap_table_size;
+ bm->flags = e->flags;
+ bm->granularity_bits = e->granularity_bits;
+ bm->name = dir_entry_copy_name(e);
+ QSIMPLEQ_INSERT_TAIL(bm_list, bm, entry);
+ }
+
+ if (nb_dir_entries != s->nb_bitmaps) {
+ error_setg(errp, "Less bitmaps found than specified in header"
+ " extension");
+ goto fail;
+ }
+
+ if ((uint8_t *)e != dir_end) {
+ goto broken_dir;
+ }
+
+ g_free(dir);
+ return bm_list;
+
+broken_dir:
+ ret = -EINVAL;
+ error_setg(errp, "Broken bitmap directory");
+
+fail:
+ g_free(dir);
+ bitmap_list_free(bm_list);
+
+ return NULL;
+}
+
+int qcow2_check_bitmaps_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
+ void **refcount_table,
+ int64_t *refcount_table_size)
+{
+ int ret;
+ BDRVQcow2State *s = bs->opaque;
+ Qcow2BitmapList *bm_list;
+ Qcow2Bitmap *bm;
+
+ if (s->nb_bitmaps == 0) {
+ return 0;
+ }
+
+ ret = qcow2_inc_refcounts_imrt(bs, res, refcount_table, refcount_table_size,
+ s->bitmap_directory_offset,
+ s->bitmap_directory_size);
+ if (ret < 0) {
+ return ret;
+ }
+
+ bm_list = bitmap_list_load(bs, s->bitmap_directory_offset,
+ s->bitmap_directory_size, NULL);
+ if (bm_list == NULL) {
+ res->corruptions++;
+ return -EINVAL;
+ }
+
+ QSIMPLEQ_FOREACH(bm, bm_list, entry) {
+ uint64_t *bitmap_table = NULL;
+ int i;
+
+ ret = qcow2_inc_refcounts_imrt(bs, res,
+ refcount_table, refcount_table_size,
+ bm->table.offset,
+ bm->table.size * sizeof(uint64_t));
+ if (ret < 0) {
+ goto out;
+ }
+
+ ret = bitmap_table_load(bs, &bm->table, &bitmap_table);
+ if (ret < 0) {
+ res->corruptions++;
+ goto out;
+ }
+
+ for (i = 0; i < bm->table.size; ++i) {
+ uint64_t entry = bitmap_table[i];
+ uint64_t offset = entry & BME_TABLE_ENTRY_OFFSET_MASK;
+
+ if (check_table_entry(entry, s->cluster_size) < 0) {
+ res->corruptions++;
+ continue;
+ }
+
+ if (offset == 0) {
+ continue;
+ }
+
+ ret = qcow2_inc_refcounts_imrt(bs, res,
+ refcount_table, refcount_table_size,
+ offset, s->cluster_size);
+ if (ret < 0) {
+ g_free(bitmap_table);
+ goto out;
+ }
+ }
+
+ g_free(bitmap_table);
+ }
+
+out:
+ bitmap_list_free(bm_list);
+
+ return ret;
+}
diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c
index 3562697527..a8e942b63e 100644
--- a/block/qcow2-refcount.c
+++ b/block/qcow2-refcount.c
@@ -1862,6 +1862,12 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
return ret;
}
+ /* bitmaps */
+ ret = qcow2_check_bitmaps_refcounts(bs, res, refcount_table, nb_clusters);
+ if (ret < 0) {
+ return ret;
+ }
+
return check_refblocks(bs, res, fix, rebuild, refcount_table, nb_clusters);
}
diff --git a/block/qcow2.c b/block/qcow2.c
index 6a92d2ef3f..6028e9a149 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -63,6 +63,7 @@ typedef struct {
#define QCOW2_EXT_MAGIC_END 0
#define QCOW2_EXT_MAGIC_BACKING_FORMAT 0xE2792ACA
#define QCOW2_EXT_MAGIC_FEATURE_TABLE 0x6803f857
+#define QCOW2_EXT_MAGIC_BITMAPS 0x23852875
static int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename)
{
@@ -86,12 +87,17 @@ static int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename)
*/
static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset,
uint64_t end_offset, void **p_feature_table,
- Error **errp)
+ bool *need_update_header, Error **errp)
{
BDRVQcow2State *s = bs->opaque;
QCowExtension ext;
uint64_t offset;
int ret;
+ Qcow2BitmapHeaderExt bitmaps_ext;
+
+ if (need_update_header != NULL) {
+ *need_update_header = false;
+ }
#ifdef DEBUG_EXT
printf("qcow2_read_extensions: start=%ld end=%ld\n", start_offset, end_offset);
@@ -162,6 +168,85 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset,
}
break;
+ case QCOW2_EXT_MAGIC_BITMAPS:
+ if (ext.len != sizeof(bitmaps_ext)) {
+ error_setg_errno(errp, -ret, "bitmaps_ext: "
+ "Invalid extension length");
+ return -EINVAL;
+ }
+
+ if (!(s->autoclear_features & QCOW2_AUTOCLEAR_BITMAPS)) {
+ error_report("WARNING: a program lacking bitmap support "
+ "modified this file, so all bitmaps are now "
+ "considered inconsistent. Some clusters may be "
+ "leaked, run 'qemu-img check -r' on the image "
+ "file to fix.");
+ if (need_update_header != NULL) {
+ /* Updating is needed to drop invalid bitmap extension. */
+ *need_update_header = true;
+ }
+ break;
+ }
+
+ ret = bdrv_pread(bs->file, offset, &bitmaps_ext, ext.len);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "bitmaps_ext: "
+ "Could not read ext header");
+ return ret;
+ }
+
+ if (bitmaps_ext.reserved32 != 0) {
+ error_setg_errno(errp, -ret, "bitmaps_ext: "
+ "Reserved field is not zero");
+ return -EINVAL;
+ }
+
+ be32_to_cpus(&bitmaps_ext.nb_bitmaps);
+ be64_to_cpus(&bitmaps_ext.bitmap_directory_size);
+ be64_to_cpus(&bitmaps_ext.bitmap_directory_offset);
+
+ if (bitmaps_ext.nb_bitmaps > QCOW2_MAX_BITMAPS) {
+ error_setg(errp,
+ "bitmaps_ext: Image has %" PRIu32 " bitmaps, "
+ "exceeding the QEMU supported maximum of %d",
+ bitmaps_ext.nb_bitmaps, QCOW2_MAX_BITMAPS);
+ return -EINVAL;
+ }
+
+ if (bitmaps_ext.nb_bitmaps == 0) {
+ error_setg(errp, "found bitmaps extension with zero bitmaps");
+ return -EINVAL;
+ }
+
+ if (bitmaps_ext.bitmap_directory_offset & (s->cluster_size - 1)) {
+ error_setg(errp, "bitmaps_ext: "
+ "invalid bitmap directory offset");
+ return -EINVAL;
+ }
+
+ if (bitmaps_ext.bitmap_directory_size >
+ QCOW2_MAX_BITMAP_DIRECTORY_SIZE) {
+ error_setg(errp, "bitmaps_ext: "
+ "bitmap directory size (%" PRIu64 ") exceeds "
+ "the maximum supported size (%d)",
+ bitmaps_ext.bitmap_directory_size,
+ QCOW2_MAX_BITMAP_DIRECTORY_SIZE);
+ return -EINVAL;
+ }
+
+ s->nb_bitmaps = bitmaps_ext.nb_bitmaps;
+ s->bitmap_directory_offset =
+ bitmaps_ext.bitmap_directory_offset;
+ s->bitmap_directory_size =
+ bitmaps_ext.bitmap_directory_size;
+
+#ifdef DEBUG_EXT
+ printf("Qcow2: Got bitmaps extension: "
+ "offset=%" PRIu64 " nb_bitmaps=%" PRIu32 "\n",
+ s->bitmap_directory_offset, s->nb_bitmaps);
+#endif
+ break;
+
default:
/* unknown magic - save it in case we need to rewrite the header */
{
@@ -824,6 +909,7 @@ static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags,
Error *local_err = NULL;
uint64_t ext_end;
uint64_t l1_vm_state_index;
+ bool update_header = false;
ret = bdrv_pread(bs->file, 0, &header, sizeof(header));
if (ret < 0) {
@@ -929,7 +1015,7 @@ static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags,
if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
void *feature_table = NULL;
qcow2_read_extensions(bs, header.header_length, ext_end,
- &feature_table, NULL);
+ &feature_table, NULL, NULL);
report_unsupported_feature(errp, feature_table,
s->incompatible_features &
~QCOW2_INCOMPAT_MASK);
@@ -1116,7 +1202,7 @@ static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags,
/* read qcow2 extensions */
if (qcow2_read_extensions(bs, header.header_length, ext_end, NULL,
- &local_err)) {
+ &update_header, &local_err)) {
error_propagate(errp, local_err);
ret = -EINVAL;
goto fail;
@@ -1152,8 +1238,10 @@ static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags,
}
/* Clear unknown autoclear feature bits */
- if (!bs->read_only && !(flags & BDRV_O_INACTIVE) && s->autoclear_features) {
- s->autoclear_features = 0;
+ update_header |= s->autoclear_features & ~QCOW2_AUTOCLEAR_MASK;
+
+ if (update_header && !bs->read_only && !(flags & BDRV_O_INACTIVE)) {
+ s->autoclear_features &= QCOW2_AUTOCLEAR_MASK;
ret = qcow2_update_header(bs);
if (ret < 0) {
error_setg_errno(errp, -ret, "Could not update qcow2 header");
@@ -1965,6 +2053,25 @@ int qcow2_update_header(BlockDriverState *bs)
buflen -= ret;
}
+ /* Bitmap extension */
+ if (s->nb_bitmaps > 0) {
+ Qcow2BitmapHeaderExt bitmaps_header = {
+ .nb_bitmaps = cpu_to_be32(s->nb_bitmaps),
+ .bitmap_directory_size =
+ cpu_to_be64(s->bitmap_directory_size),
+ .bitmap_directory_offset =
+ cpu_to_be64(s->bitmap_directory_offset)
+ };
+ ret = header_ext_add(buf, QCOW2_EXT_MAGIC_BITMAPS,
+ &bitmaps_header, sizeof(bitmaps_header),
+ buflen);
+ if (ret < 0) {
+ goto fail;
+ }
+ buf += ret;
+ buflen -= ret;
+ }
+
/* Keep unknown header extensions */
QLIST_FOREACH(uext, &s->unknown_header_ext, next) {
ret = header_ext_add(buf, uext->magic, uext->data, uext->len, buflen);
@@ -2542,6 +2649,13 @@ static int qcow2_truncate(BlockDriverState *bs, int64_t offset)
return -ENOTSUP;
}
+ /* cannot proceed if image has bitmaps */
+ if (s->nb_bitmaps) {
+ /* TODO: resize bitmaps in the image */
+ error_report("Can't resize an image which has bitmaps");
+ return -ENOTSUP;
+ }
+
/* shrinking is currently not supported */
if (offset < bs->total_sectors * 512) {
error_report("qcow2 doesn't support shrinking images yet");
diff --git a/block/qcow2.h b/block/qcow2.h
index 5d1763dba1..fbe6caefce 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -52,6 +52,10 @@
* space for snapshot names and IDs */
#define QCOW_MAX_SNAPSHOTS_SIZE (1024 * QCOW_MAX_SNAPSHOTS)
+/* Bitmap header extension constraints */
+#define QCOW2_MAX_BITMAPS 65535
+#define QCOW2_MAX_BITMAP_DIRECTORY_SIZE (1024 * QCOW2_MAX_BITMAPS)
+
/* indicate that the refcount of the referenced cluster is exactly one. */
#define QCOW_OFLAG_COPIED (1ULL << 63)
/* indicate that the cluster is compressed (they never have the copied flag) */
@@ -195,6 +199,14 @@ enum {
QCOW2_COMPAT_FEAT_MASK = QCOW2_COMPAT_LAZY_REFCOUNTS,
};
+/* Autoclear feature bits */
+enum {
+ QCOW2_AUTOCLEAR_BITMAPS_BITNR = 0,
+ QCOW2_AUTOCLEAR_BITMAPS = 1 << QCOW2_AUTOCLEAR_BITMAPS_BITNR,
+
+ QCOW2_AUTOCLEAR_MASK = QCOW2_AUTOCLEAR_BITMAPS,
+};
+
enum qcow2_discard_type {
QCOW2_DISCARD_NEVER = 0,
QCOW2_DISCARD_ALWAYS,
@@ -222,6 +234,13 @@ typedef uint64_t Qcow2GetRefcountFunc(const void *refcount_array,
typedef void Qcow2SetRefcountFunc(void *refcount_array,
uint64_t index, uint64_t value);
+typedef struct Qcow2BitmapHeaderExt {
+ uint32_t nb_bitmaps;
+ uint32_t reserved32;
+ uint64_t bitmap_directory_size;
+ uint64_t bitmap_directory_offset;
+} QEMU_PACKED Qcow2BitmapHeaderExt;
+
typedef struct BDRVQcow2State {
int cluster_bits;
int cluster_size;
@@ -264,6 +283,10 @@ typedef struct BDRVQcow2State {
unsigned int nb_snapshots;
QCowSnapshot *snapshots;
+ uint32_t nb_bitmaps;
+ uint64_t bitmap_directory_size;
+ uint64_t bitmap_directory_offset;
+
int flags;
int qcow_version;
bool use_lazy_refcounts;
@@ -594,4 +617,8 @@ int qcow2_cache_get_empty(BlockDriverState *bs, Qcow2Cache *c, uint64_t offset,
void **table);
void qcow2_cache_put(BlockDriverState *bs, Qcow2Cache *c, void **table);
+/* qcow2-bitmap.c functions */
+int qcow2_check_bitmaps_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
+ void **refcount_table,
+ int64_t *refcount_table_size);
#endif
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (7 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 08/25] qcow2: add bitmaps extension Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-19 23:02 ` John Snow
` (2 more replies)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 10/25] qcow2: autoloading dirty bitmaps Vladimir Sementsov-Ogievskiy
` (16 subsequent siblings)
25 siblings, 3 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
It will be needed in following commits for persistent bitmaps.
If bitmap is loaded from read-only storage (and we can't mark it
"in use" in this storage) corresponding BdrvDirtyBitmap should be
read-only.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
block/dirty-bitmap.c | 16 ++++++++++++++++
include/block/dirty-bitmap.h | 3 +++
2 files changed, 19 insertions(+)
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 90af37287f..ab6a95cf41 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -44,6 +44,8 @@ struct BdrvDirtyBitmap {
int64_t size; /* Size of the bitmap (Number of sectors) */
bool disabled; /* Bitmap is read-only */
int active_iterators; /* How many iterators are active */
+ bool readonly; /* Bitmap is read-only and may be changed only
+ by deserialize* functions */
QLIST_ENTRY(BdrvDirtyBitmap) list;
};
@@ -436,6 +438,7 @@ void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
int64_t cur_sector, int64_t nr_sectors)
{
assert(bdrv_dirty_bitmap_enabled(bitmap));
+ assert(!bdrv_dirty_bitmap_readonly(bitmap));
hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
}
@@ -443,12 +446,14 @@ void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
int64_t cur_sector, int64_t nr_sectors)
{
assert(bdrv_dirty_bitmap_enabled(bitmap));
+ assert(!bdrv_dirty_bitmap_readonly(bitmap));
hbitmap_reset(bitmap->bitmap, cur_sector, nr_sectors);
}
void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap **out)
{
assert(bdrv_dirty_bitmap_enabled(bitmap));
+ assert(!bdrv_dirty_bitmap_readonly(bitmap));
if (!out) {
hbitmap_reset_all(bitmap->bitmap);
} else {
@@ -519,6 +524,7 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
if (!bdrv_dirty_bitmap_enabled(bitmap)) {
continue;
}
+ assert(!bdrv_dirty_bitmap_readonly(bitmap));
hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
}
}
@@ -540,3 +546,13 @@ int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap)
{
return hbitmap_count(bitmap->meta);
}
+
+bool bdrv_dirty_bitmap_readonly(const BdrvDirtyBitmap *bitmap)
+{
+ return bitmap->readonly;
+}
+
+void bdrv_dirty_bitmap_set_readonly(BdrvDirtyBitmap *bitmap)
+{
+ bitmap->readonly = true;
+}
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 1e17729ac2..0aab5841f5 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -75,4 +75,7 @@ void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap,
bool finish);
void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
+bool bdrv_dirty_bitmap_readonly(const BdrvDirtyBitmap *bitmap);
+void bdrv_dirty_bitmap_set_readonly(BdrvDirtyBitmap *bitmap);
+
#endif
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 10/25] qcow2: autoloading dirty bitmaps
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (8 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-29 16:17 ` Max Reitz
2017-05-03 12:25 ` [Qemu-devel] [PATCH 11/25] block/dirty-bitmap: add autoload field to BdrvDirtyBitmap Vladimir Sementsov-Ogievskiy
` (15 subsequent siblings)
25 siblings, 1 reply; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Auto loading bitmaps are bitmaps in Qcow2, with the AUTO flag set. They
are loaded when the image is opened and become BdrvDirtyBitmaps for the
corresponding drive.
Extra data in bitmaps is not supported for now.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
block/qcow2-bitmap.c | 387 +++++++++++++++++++++++++++++++++++++++++++++++++++
block/qcow2.c | 17 ++-
block/qcow2.h | 2 +
3 files changed, 404 insertions(+), 2 deletions(-)
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index b8e472b3e8..a2522d7ccf 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -44,6 +44,8 @@
/* Bitmap directory entry flags */
#define BME_RESERVED_FLAGS 0xfffffffcU
+#define BME_FLAG_IN_USE (1U << 0)
+#define BME_FLAG_AUTO (1U << 1)
/* bits [1, 8] U [56, 63] are reserved */
#define BME_TABLE_ENTRY_RESERVED_MASK 0xff000000000001feULL
@@ -85,6 +87,23 @@ typedef enum BitmapType {
BT_DIRTY_TRACKING_BITMAP = 1
} BitmapType;
+static inline bool can_write(BlockDriverState *bs)
+{
+ return !bdrv_is_read_only(bs) && !(bdrv_get_flags(bs) & BDRV_O_INACTIVE);
+}
+
+static int update_header_sync(BlockDriverState *bs)
+{
+ int ret;
+
+ ret = qcow2_update_header(bs);
+ if (ret < 0) {
+ return ret;
+ }
+
+ return bdrv_flush(bs);
+}
+
static int check_table_entry(uint64_t entry, int cluster_size)
{
uint64_t offset;
@@ -146,6 +165,120 @@ fail:
return ret;
}
+/* This function returns the number of disk sectors covered by a single qcow2
+ * cluster of bitmap data. */
+static uint64_t sectors_covered_by_bitmap_cluster(const BDRVQcow2State *s,
+ const BdrvDirtyBitmap *bitmap)
+{
+ uint32_t sector_granularity =
+ bdrv_dirty_bitmap_granularity(bitmap) >> BDRV_SECTOR_BITS;
+
+ return (uint64_t)sector_granularity * (s->cluster_size << 3);
+}
+
+/* load_bitmap_data
+ * @bitmap_table entries must satisfy specification constraints.
+ * @bitmap must be cleared */
+static int load_bitmap_data(BlockDriverState *bs,
+ const uint64_t *bitmap_table,
+ uint32_t bitmap_table_size,
+ BdrvDirtyBitmap *bitmap)
+{
+ int ret = 0;
+ BDRVQcow2State *s = bs->opaque;
+ uint64_t sector, sbc;
+ uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap);
+ uint8_t *buf = NULL;
+ uint64_t i, tab_size =
+ size_to_clusters(s,
+ bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_size));
+
+ if (tab_size != bitmap_table_size || tab_size > BME_MAX_TABLE_SIZE) {
+ return -EINVAL;
+ }
+
+ buf = g_malloc(s->cluster_size);
+ sbc = sectors_covered_by_bitmap_cluster(s, bitmap);
+ for (i = 0, sector = 0; i < tab_size; ++i, sector += sbc) {
+ uint64_t count = MIN(bm_size - sector, sbc);
+ uint64_t entry = bitmap_table[i];
+ uint64_t offset = entry & BME_TABLE_ENTRY_OFFSET_MASK;
+
+ assert(check_table_entry(entry, s->cluster_size) == 0);
+
+ if (offset == 0) {
+ if (entry & BME_TABLE_ENTRY_FLAG_ALL_ONES) {
+ bdrv_dirty_bitmap_deserialize_ones(bitmap, sector, count,
+ false);
+ } else {
+ /* No need to deserialize zeros because the dirty bitmap is
+ * already cleared */
+ }
+ } else {
+ ret = bdrv_pread(bs->file, offset, buf, s->cluster_size);
+ if (ret < 0) {
+ goto finish;
+ }
+ bdrv_dirty_bitmap_deserialize_part(bitmap, buf, sector, count,
+ false);
+ }
+ }
+ ret = 0;
+
+ bdrv_dirty_bitmap_deserialize_finish(bitmap);
+
+finish:
+ g_free(buf);
+
+ return ret;
+}
+
+static BdrvDirtyBitmap *load_bitmap(BlockDriverState *bs,
+ Qcow2Bitmap *bm, Error **errp)
+{
+ int ret;
+ uint64_t *bitmap_table = NULL;
+ uint32_t granularity;
+ BdrvDirtyBitmap *bitmap = NULL;
+
+ if (bm->flags & BME_FLAG_IN_USE) {
+ error_setg(errp, "Bitmap '%s' is in use", bm->name);
+ goto fail;
+ }
+
+ ret = bitmap_table_load(bs, &bm->table, &bitmap_table);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret,
+ "Could not read bitmap_table table from image for "
+ "bitmap '%s'", bm->name);
+ goto fail;
+ }
+
+ granularity = 1U << bm->granularity_bits;
+ bitmap = bdrv_create_dirty_bitmap(bs, granularity, bm->name, errp);
+ if (bitmap == NULL) {
+ goto fail;
+ }
+
+ ret = load_bitmap_data(bs, bitmap_table, bm->table.size, bitmap);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "Could not read bitmap '%s' from image",
+ bm->name);
+ goto fail;
+ }
+
+ g_free(bitmap_table);
+ return bitmap;
+
+fail:
+ g_free(bitmap_table);
+ if (bitmap != NULL) {
+ bdrv_release_dirty_bitmap(bs, bitmap);
+ }
+
+ return NULL;
+}
+
/*
* Bitmap List
*/
@@ -164,6 +297,15 @@ static inline void bitmap_dir_entry_to_cpu(Qcow2BitmapDirEntry *entry)
be32_to_cpus(&entry->extra_data_size);
}
+static inline void bitmap_dir_entry_to_be(Qcow2BitmapDirEntry *entry)
+{
+ cpu_to_be64s(&entry->bitmap_table_offset);
+ cpu_to_be32s(&entry->bitmap_table_size);
+ cpu_to_be32s(&entry->flags);
+ cpu_to_be16s(&entry->name_size);
+ cpu_to_be32s(&entry->extra_data_size);
+}
+
static inline int calc_dir_entry_size(size_t name_size, size_t extra_data_size)
{
return align_offset(sizeof(Qcow2BitmapDirEntry) +
@@ -224,6 +366,17 @@ static int check_dir_entry(BlockDriverState *bs, Qcow2BitmapDirEntry *entry)
return fail ? -EINVAL : 0;
}
+static inline void bitmap_directory_to_be(uint8_t *dir, size_t size)
+{
+ uint8_t *end = dir + size;
+ while (dir < end) {
+ Qcow2BitmapDirEntry *e = (Qcow2BitmapDirEntry *)dir;
+ dir += dir_entry_size(e);
+
+ bitmap_dir_entry_to_be(e);
+ }
+}
+
/*
* Bitmap List public functions
*/
@@ -258,6 +411,18 @@ static Qcow2BitmapList *bitmap_list_new(void)
return bm_list;
}
+static uint32_t bitmap_list_count(Qcow2BitmapList *bm_list)
+{
+ Qcow2Bitmap *bm;
+ uint32_t nb_bitmaps = 0;
+
+ QSIMPLEQ_FOREACH(bm, bm_list, entry) {
+ nb_bitmaps++;
+ }
+
+ return nb_bitmaps;
+}
+
/* bitmap_list_load
* Get bitmap list from qcow2 image. Actually reads bitmap directory,
* checks it and convert to bitmap list.
@@ -437,3 +602,225 @@ out:
return ret;
}
+
+/* bitmap_list_store
+ * Store bitmap list to qcow2 image as a bitmap directory.
+ * Everything is checked.
+ */
+static int bitmap_list_store(BlockDriverState *bs, Qcow2BitmapList *bm_list,
+ uint64_t *offset, uint64_t *size, bool in_place)
+{
+ int ret;
+ uint8_t *dir;
+ int64_t dir_offset = 0;
+ uint64_t dir_size = 0;
+ Qcow2Bitmap *bm;
+ Qcow2BitmapDirEntry *e;
+
+ QSIMPLEQ_FOREACH(bm, bm_list, entry) {
+ dir_size += calc_dir_entry_size(strlen(bm->name), 0);
+ }
+
+ if (dir_size == 0 || dir_size > QCOW2_MAX_BITMAP_DIRECTORY_SIZE) {
+ return -EINVAL;
+ }
+
+ if (in_place) {
+ if (*size != dir_size || *offset == 0) {
+ return -EINVAL;
+ }
+
+ dir_offset = *offset;
+ }
+
+ dir = g_try_malloc(dir_size);
+ if (dir == NULL) {
+ return -ENOMEM;
+ }
+
+ e = (Qcow2BitmapDirEntry *)dir;
+ QSIMPLEQ_FOREACH(bm, bm_list, entry) {
+ e->bitmap_table_offset = bm->table.offset;
+ e->bitmap_table_size = bm->table.size;
+ e->flags = bm->flags;
+ e->type = BT_DIRTY_TRACKING_BITMAP;
+ e->granularity_bits = bm->granularity_bits;
+ e->name_size = strlen(bm->name);
+ e->extra_data_size = 0;
+ memcpy(e + 1, bm->name, e->name_size);
+
+ if (check_dir_entry(bs, e) < 0) {
+ ret = -EINVAL;
+ goto fail;
+ }
+
+ e = next_dir_entry(e);
+ }
+
+ bitmap_directory_to_be(dir, dir_size);
+
+ if (!in_place) {
+ dir_offset = qcow2_alloc_clusters(bs, dir_size);
+ if (dir_offset < 0) {
+ ret = dir_offset;
+ goto fail;
+ }
+ }
+
+ ret = qcow2_pre_write_overlap_check(bs, 0, dir_offset, dir_size);
+ if (ret < 0) {
+ goto fail;
+ }
+
+ ret = bdrv_pwrite(bs->file, dir_offset, dir, dir_size);
+ if (ret < 0) {
+ goto fail;
+ }
+
+ g_free(dir);
+
+ if (!in_place) {
+ *size = dir_size;
+ *offset = dir_offset;
+ }
+
+ return 0;
+
+fail:
+ g_free(dir);
+
+ if (!in_place && dir_offset > 0) {
+ qcow2_free_clusters(bs, dir_offset, dir_size, QCOW2_DISCARD_OTHER);
+ }
+
+ return ret;
+}
+
+/*
+ * Bitmap List end
+ */
+
+static int update_ext_header_and_dir_in_place(BlockDriverState *bs,
+ Qcow2BitmapList *bm_list)
+{
+ BDRVQcow2State *s = bs->opaque;
+ int ret;
+
+ if (!(s->autoclear_features & QCOW2_AUTOCLEAR_BITMAPS) ||
+ bm_list == NULL || QSIMPLEQ_EMPTY(bm_list) ||
+ bitmap_list_count(bm_list) != s->nb_bitmaps)
+ {
+ return -EINVAL;
+ }
+
+ s->autoclear_features &= ~(uint64_t)QCOW2_AUTOCLEAR_BITMAPS;
+ ret = update_header_sync(bs);
+ if (ret < 0) {
+ /* Two variants are possible here:
+ * 1. Autoclear flag is dropped, all bitmaps will be lost.
+ * 2. Autoclear flag is not dropped, old state is left.
+ */
+ return ret;
+ }
+
+ /* autoclear bit is not set, so we can safely update bitmap directory */
+
+ ret = bitmap_list_store(bs, bm_list, &s->bitmap_directory_offset,
+ &s->bitmap_directory_size, true);
+ if (ret < 0) {
+ /* autoclear bit is cleared, so all leaked clusters would be removed on
+ * qemu-img check */
+ return ret;
+ }
+
+ ret = update_header_sync(bs);
+ if (ret < 0) {
+ /* autoclear bit is cleared, so all leaked clusters would be removed on
+ * qemu-img check */
+ return ret;
+ }
+
+ s->autoclear_features |= QCOW2_AUTOCLEAR_BITMAPS;
+ return update_header_sync(bs);
+ /* If final update_header_sync() fails, two variants are possible:
+ * 1. Autoclear flag is not set, all bitmaps will be lost.
+ * 2. Autoclear flag is set, header and directory are successfully updated.
+ */
+}
+
+/* for g_slist_foreach for GSList of BdrvDirtyBitmap* elements */
+static void release_dirty_bitmap_helper(gpointer bitmap,
+ gpointer bs)
+{
+ bdrv_release_dirty_bitmap(bs, bitmap);
+}
+
+/* for g_slist_foreach for GSList of BdrvDirtyBitmap* elements */
+static void set_readonly_helper(gpointer bitmap, gpointer unused)
+{
+ bdrv_dirty_bitmap_set_readonly(bitmap);
+}
+
+/* qcow2_load_autoloading_dirty_bitmaps()
+ * Return value is a hint for caller: true means that Qcow2 header was updated.
+ * (false doesn't mean that header should be updated by caller, it just means
+ * that updating was not needed or image can not be written).
+ * On fail the function returns false.
+ */
+bool qcow2_load_autoloading_dirty_bitmaps(BlockDriverState *bs, Error **errp)
+{
+ BDRVQcow2State *s = bs->opaque;
+ Qcow2BitmapList *bm_list;
+ Qcow2Bitmap *bm;
+ GSList *created_dirty_bitmaps = NULL;
+ bool header_updated = false;
+
+ if (s->nb_bitmaps == 0) {
+ /* No bitmaps - nothing to do */
+ return false;
+ }
+
+ bm_list = bitmap_list_load(bs, s->bitmap_directory_offset,
+ s->bitmap_directory_size, errp);
+ if (bm_list == NULL) {
+ return false;
+ }
+
+ QSIMPLEQ_FOREACH(bm, bm_list, entry) {
+ if ((bm->flags & BME_FLAG_AUTO) && !(bm->flags & BME_FLAG_IN_USE)) {
+ BdrvDirtyBitmap *bitmap = load_bitmap(bs, bm, errp);
+ if (bitmap == NULL) {
+ goto fail;
+ }
+ bm->flags |= BME_FLAG_IN_USE;
+ created_dirty_bitmaps =
+ g_slist_append(created_dirty_bitmaps, bitmap);
+ }
+ }
+
+ if (created_dirty_bitmaps != NULL) {
+ if (can_write(bs)) {
+ /* in_use flags must be updated */
+ int ret = update_ext_header_and_dir_in_place(bs, bm_list);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "Can't update bitmap directory");
+ goto fail;
+ }
+ header_updated = true;
+ } else {
+ g_slist_foreach(created_dirty_bitmaps, set_readonly_helper, NULL);
+ }
+ }
+
+ g_slist_free(created_dirty_bitmaps);
+ bitmap_list_free(bm_list);
+
+ return header_updated;
+
+fail:
+ g_slist_foreach(created_dirty_bitmaps, release_dirty_bitmap_helper, bs);
+ g_slist_free(created_dirty_bitmaps);
+ bitmap_list_free(bm_list);
+
+ return false;
+}
diff --git a/block/qcow2.c b/block/qcow2.c
index 6028e9a149..4d3654d606 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1239,9 +1239,22 @@ static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags,
/* Clear unknown autoclear feature bits */
update_header |= s->autoclear_features & ~QCOW2_AUTOCLEAR_MASK;
-
- if (update_header && !bs->read_only && !(flags & BDRV_O_INACTIVE)) {
+ update_header =
+ update_header && !bs->read_only && !(flags & BDRV_O_INACTIVE);
+ if (update_header) {
s->autoclear_features &= QCOW2_AUTOCLEAR_MASK;
+ }
+
+ if (qcow2_load_autoloading_dirty_bitmaps(bs, &local_err)) {
+ update_header = false;
+ }
+ if (local_err != NULL) {
+ error_propagate(errp, local_err);
+ ret = -EINVAL;
+ goto fail;
+ }
+
+ if (update_header) {
ret = qcow2_update_header(bs);
if (ret < 0) {
error_setg_errno(errp, -ret, "Could not update qcow2 header");
diff --git a/block/qcow2.h b/block/qcow2.h
index fbe6caefce..15e9791faa 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -621,4 +621,6 @@ void qcow2_cache_put(BlockDriverState *bs, Qcow2Cache *c, void **table);
int qcow2_check_bitmaps_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
void **refcount_table,
int64_t *refcount_table_size);
+bool qcow2_load_autoloading_dirty_bitmaps(BlockDriverState *bs, Error **errp);
+
#endif
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 11/25] block/dirty-bitmap: add autoload field to BdrvDirtyBitmap
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (9 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 10/25] qcow2: autoloading dirty bitmaps Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 12/25] block: bdrv_close: release bitmaps after drv->bdrv_close Vladimir Sementsov-Ogievskiy
` (14 subsequent siblings)
25 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Mirror AUTO flag from Qcow2 bitmap in BdrvDirtyBitmap. This will be
needed in future, to save this flag back to Qcow2 for persistent
bitmaps.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
block/dirty-bitmap.c | 15 +++++++++++++++
block/qcow2-bitmap.c | 2 ++
include/block/dirty-bitmap.h | 3 +++
3 files changed, 20 insertions(+)
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index ab6a95cf41..bdbfb01cd7 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -46,6 +46,8 @@ struct BdrvDirtyBitmap {
int active_iterators; /* How many iterators are active */
bool readonly; /* Bitmap is read-only and may be changed only
by deserialize* functions */
+ bool autoload; /* For persistent bitmaps: bitmap must be
+ autoloaded on image opening */
QLIST_ENTRY(BdrvDirtyBitmap) list;
};
@@ -72,6 +74,7 @@ void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap)
assert(!bdrv_dirty_bitmap_frozen(bitmap));
g_free(bitmap->name);
bitmap->name = NULL;
+ bitmap->autoload = false;
}
BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
@@ -240,6 +243,8 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs,
bitmap->name = NULL;
successor->name = name;
bitmap->successor = NULL;
+ successor->autoload = bitmap->autoload;
+ bitmap->autoload = false;
bdrv_release_dirty_bitmap(bs, bitmap);
return successor;
@@ -556,3 +561,13 @@ void bdrv_dirty_bitmap_set_readonly(BdrvDirtyBitmap *bitmap)
{
bitmap->readonly = true;
}
+
+void bdrv_dirty_bitmap_set_autoload(BdrvDirtyBitmap *bitmap, bool autoload)
+{
+ bitmap->autoload = autoload;
+}
+
+bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap)
+{
+ return bitmap->autoload;
+}
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index a2522d7ccf..ecaac57778 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -792,6 +792,8 @@ bool qcow2_load_autoloading_dirty_bitmaps(BlockDriverState *bs, Error **errp)
if (bitmap == NULL) {
goto fail;
}
+
+ bdrv_dirty_bitmap_set_autoload(bitmap, true);
bm->flags |= BME_FLAG_IN_USE;
created_dirty_bitmaps =
g_slist_append(created_dirty_bitmaps, bitmap);
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 0aab5841f5..60e2853d48 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -78,4 +78,7 @@ void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
bool bdrv_dirty_bitmap_readonly(const BdrvDirtyBitmap *bitmap);
void bdrv_dirty_bitmap_set_readonly(BdrvDirtyBitmap *bitmap);
+void bdrv_dirty_bitmap_set_autoload(BdrvDirtyBitmap *bitmap, bool autoload);
+bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap);
+
#endif
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 12/25] block: bdrv_close: release bitmaps after drv->bdrv_close
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (10 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 11/25] block/dirty-bitmap: add autoload field to BdrvDirtyBitmap Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-29 16:34 ` Max Reitz
2017-05-03 12:25 ` [Qemu-devel] [PATCH 13/25] block: introduce persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (13 subsequent siblings)
25 siblings, 1 reply; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Release bitmaps after 'if (bs->drv) { ... }' block. This will allow
format driver to save persistent bitmaps, which will appear in following
commits.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
block.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/block.c b/block.c
index 5db266be21..17c853f3ee 100644
--- a/block.c
+++ b/block.c
@@ -2949,9 +2949,6 @@ static void bdrv_close(BlockDriverState *bs)
bdrv_flush(bs);
bdrv_drain(bs); /* in case flush left pending I/O */
- bdrv_release_named_dirty_bitmaps(bs);
- assert(QLIST_EMPTY(&bs->dirty_bitmaps));
-
if (bs->drv) {
BdrvChild *child, *next;
@@ -2990,6 +2987,9 @@ static void bdrv_close(BlockDriverState *bs)
bs->full_open_options = NULL;
}
+ bdrv_release_named_dirty_bitmaps(bs);
+ assert(QLIST_EMPTY(&bs->dirty_bitmaps));
+
QLIST_FOREACH_SAFE(ban, &bs->aio_notifiers, list, ban_next) {
g_free(ban);
}
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 13/25] block: introduce persistent dirty bitmaps
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (11 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 12/25] block: bdrv_close: release bitmaps after drv->bdrv_close Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-29 16:49 ` Max Reitz
2017-05-03 12:25 ` [Qemu-devel] [PATCH 14/25] block/dirty-bitmap: add bdrv_dirty_bitmap_next() Vladimir Sementsov-Ogievskiy
` (12 subsequent siblings)
25 siblings, 1 reply; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
New field BdrvDirtyBitmap.persistent means, that bitmap should be saved
by format driver in .bdrv_close and .bdrv_inactivate. No format driver
supports it for now.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
block/dirty-bitmap.c | 26 ++++++++++++++++++++++++++
block/qcow2-bitmap.c | 1 +
include/block/dirty-bitmap.h | 5 +++++
3 files changed, 32 insertions(+)
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index bdbfb01cd7..bac4fe3008 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -48,6 +48,7 @@ struct BdrvDirtyBitmap {
by deserialize* functions */
bool autoload; /* For persistent bitmaps: bitmap must be
autoloaded on image opening */
+ bool persistent; /* bitmap must be saved to owner disk image */
QLIST_ENTRY(BdrvDirtyBitmap) list;
};
@@ -74,6 +75,7 @@ void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap)
assert(!bdrv_dirty_bitmap_frozen(bitmap));
g_free(bitmap->name);
bitmap->name = NULL;
+ bitmap->persistent = false;
bitmap->autoload = false;
}
@@ -243,6 +245,8 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs,
bitmap->name = NULL;
successor->name = name;
bitmap->successor = NULL;
+ successor->persistent = bitmap->persistent;
+ bitmap->persistent = false;
successor->autoload = bitmap->autoload;
bitmap->autoload = false;
bdrv_release_dirty_bitmap(bs, bitmap);
@@ -571,3 +575,25 @@ bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap)
{
return bitmap->autoload;
}
+
+void bdrv_dirty_bitmap_set_persistance(BdrvDirtyBitmap *bitmap, bool persistent)
+{
+ bitmap->persistent = persistent;
+}
+
+bool bdrv_dirty_bitmap_get_persistance(BdrvDirtyBitmap *bitmap)
+{
+ return bitmap->persistent;
+}
+
+bool bdrv_has_changed_persistent_bitmaps(BlockDriverState *bs)
+{
+ BdrvDirtyBitmap *bm;
+ QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) {
+ if (bm->persistent && !bm->readonly) {
+ return true;
+ }
+ }
+
+ return false;
+}
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index ecaac57778..f0887cc8ff 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -793,6 +793,7 @@ bool qcow2_load_autoloading_dirty_bitmaps(BlockDriverState *bs, Error **errp)
goto fail;
}
+ bdrv_dirty_bitmap_set_persistance(bitmap, true);
bdrv_dirty_bitmap_set_autoload(bitmap, true);
bm->flags |= BME_FLAG_IN_USE;
created_dirty_bitmaps =
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 60e2853d48..a48e4df246 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -81,4 +81,9 @@ void bdrv_dirty_bitmap_set_readonly(BdrvDirtyBitmap *bitmap);
void bdrv_dirty_bitmap_set_autoload(BdrvDirtyBitmap *bitmap, bool autoload);
bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap);
+void bdrv_dirty_bitmap_set_persistance(BdrvDirtyBitmap *bitmap,
+ bool persistent);
+bool bdrv_dirty_bitmap_get_persistance(BdrvDirtyBitmap *bitmap);
+bool bdrv_has_changed_persistent_bitmaps(BlockDriverState *bs);
+
#endif
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 14/25] block/dirty-bitmap: add bdrv_dirty_bitmap_next()
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (12 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 13/25] block: introduce persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 15/25] qcow2: add persistent dirty bitmaps support Vladimir Sementsov-Ogievskiy
` (11 subsequent siblings)
25 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
block/dirty-bitmap.c | 7 +++++++
include/block/dirty-bitmap.h | 3 +++
2 files changed, 10 insertions(+)
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index bac4fe3008..df0d3cd954 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -597,3 +597,10 @@ bool bdrv_has_changed_persistent_bitmaps(BlockDriverState *bs)
return false;
}
+
+BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,
+ BdrvDirtyBitmap *bitmap)
+{
+ return bitmap == NULL ? QLIST_FIRST(&bs->dirty_bitmaps) :
+ QLIST_NEXT(bitmap, list);
+}
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index a48e4df246..741573be97 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -86,4 +86,7 @@ void bdrv_dirty_bitmap_set_persistance(BdrvDirtyBitmap *bitmap,
bool bdrv_dirty_bitmap_get_persistance(BdrvDirtyBitmap *bitmap);
bool bdrv_has_changed_persistent_bitmaps(BlockDriverState *bs);
+BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,
+ BdrvDirtyBitmap *bitmap);
+
#endif
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 15/25] qcow2: add persistent dirty bitmaps support
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (13 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 14/25] block/dirty-bitmap: add bdrv_dirty_bitmap_next() Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-29 17:07 ` Max Reitz
2017-05-03 12:25 ` [Qemu-devel] [PATCH 16/25] block: add bdrv_can_store_new_dirty_bitmap Vladimir Sementsov-Ogievskiy
` (10 subsequent siblings)
25 siblings, 1 reply; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Store persistent dirty bitmaps in qcow2 image.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
block/qcow2-bitmap.c | 475 +++++++++++++++++++++++++++++++++++++++++++++++++++
block/qcow2.c | 9 +
block/qcow2.h | 1 +
3 files changed, 485 insertions(+)
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index f0887cc8ff..663c5b2ac7 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -27,6 +27,7 @@
#include "qemu/osdep.h"
#include "qapi/error.h"
+#include "qemu/cutils.h"
#include "block/block_int.h"
#include "block/qcow2.h"
@@ -42,6 +43,10 @@
#define BME_MIN_GRANULARITY_BITS 9
#define BME_MAX_NAME_SIZE 1023
+#if BME_MAX_TABLE_SIZE * 8ULL > INT_MAX
+#error In the code bitmap table physical size assumed to fit into int
+#endif
+
/* Bitmap directory entry flags */
#define BME_RESERVED_FLAGS 0xfffffffcU
#define BME_FLAG_IN_USE (1U << 0)
@@ -72,6 +77,8 @@ typedef struct Qcow2BitmapTable {
uint32_t size; /* number of 64bit entries */
QSIMPLEQ_ENTRY(Qcow2BitmapTable) entry;
} Qcow2BitmapTable;
+typedef QSIMPLEQ_HEAD(Qcow2BitmapTableList, Qcow2BitmapTable)
+ Qcow2BitmapTableList;
typedef struct Qcow2Bitmap {
Qcow2BitmapTable table;
@@ -79,6 +86,8 @@ typedef struct Qcow2Bitmap {
uint8_t granularity_bits;
char *name;
+ BdrvDirtyBitmap *dirty_bitmap;
+
QSIMPLEQ_ENTRY(Qcow2Bitmap) entry;
} Qcow2Bitmap;
typedef QSIMPLEQ_HEAD(Qcow2BitmapList, Qcow2Bitmap) Qcow2BitmapList;
@@ -104,6 +113,15 @@ static int update_header_sync(BlockDriverState *bs)
return bdrv_flush(bs);
}
+static inline void bitmap_table_to_be(uint64_t *bitmap_table, size_t size)
+{
+ size_t i;
+
+ for (i = 0; i < size; ++i) {
+ cpu_to_be64s(&bitmap_table[i]);
+ }
+}
+
static int check_table_entry(uint64_t entry, int cluster_size)
{
uint64_t offset;
@@ -127,6 +145,70 @@ static int check_table_entry(uint64_t entry, int cluster_size)
return 0;
}
+static int check_constraints_on_bitmap(BlockDriverState *bs,
+ const char *name,
+ uint32_t granularity,
+ Error **errp)
+{
+ BDRVQcow2State *s = bs->opaque;
+ int granularity_bits = ctz32(granularity);
+ int64_t len = bdrv_getlength(bs);
+
+ assert(granularity > 0);
+ assert((granularity & (granularity - 1)) == 0);
+
+ if (len < 0) {
+ error_setg_errno(errp, -len, "Failed to get size of '%s'",
+ bdrv_get_device_or_node_name(bs));
+ return len;
+ }
+
+ if (granularity_bits > BME_MAX_GRANULARITY_BITS) {
+ error_setg(errp, "Granularity exceeds maximum (%llu bytes)",
+ 1ULL << BME_MAX_GRANULARITY_BITS);
+ return -EINVAL;
+ }
+ if (granularity_bits < BME_MIN_GRANULARITY_BITS) {
+ error_setg(errp, "Granularity is under minimum (%llu bytes)",
+ 1ULL << BME_MIN_GRANULARITY_BITS);
+ return -EINVAL;
+ }
+
+ if ((len > (uint64_t)BME_MAX_PHYS_SIZE << granularity_bits) ||
+ (len > (uint64_t)BME_MAX_TABLE_SIZE * s->cluster_size <<
+ granularity_bits))
+ {
+ error_setg(errp, "Too much space will be occupied by the bitmap. "
+ "Use larger granularity");
+ return -EINVAL;
+ }
+
+ if (strlen(name) > BME_MAX_NAME_SIZE) {
+ error_setg(errp, "Name length exceeds maximum (%u characters)",
+ BME_MAX_NAME_SIZE);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static void clear_bitmap_table(BlockDriverState *bs, uint64_t *bitmap_table,
+ uint32_t bitmap_table_size)
+{
+ BDRVQcow2State *s = bs->opaque;
+ int i;
+
+ for (i = 0; i < bitmap_table_size; ++i) {
+ uint64_t addr = bitmap_table[i] & BME_TABLE_ENTRY_OFFSET_MASK;
+ if (!addr) {
+ continue;
+ }
+
+ qcow2_free_clusters(bs, addr, s->cluster_size, QCOW2_DISCARD_OTHER);
+ bitmap_table[i] = 0;
+ }
+}
+
static int bitmap_table_load(BlockDriverState *bs, Qcow2BitmapTable *tb,
uint64_t **bitmap_table)
{
@@ -165,6 +247,28 @@ fail:
return ret;
}
+static int free_bitmap_clusters(BlockDriverState *bs, Qcow2BitmapTable *tb)
+{
+ int ret;
+ uint64_t *bitmap_table;
+
+ ret = bitmap_table_load(bs, tb, &bitmap_table);
+ if (ret < 0) {
+ assert(bitmap_table == NULL);
+ return ret;
+ }
+
+ clear_bitmap_table(bs, bitmap_table, tb->size);
+ qcow2_free_clusters(bs, tb->offset, tb->size * sizeof(uint64_t),
+ QCOW2_DISCARD_OTHER);
+ g_free(bitmap_table);
+
+ tb->offset = 0;
+ tb->size = 0;
+
+ return 0;
+}
+
/* This function returns the number of disk sectors covered by a single qcow2
* cluster of bitmap data. */
static uint64_t sectors_covered_by_bitmap_cluster(const BDRVQcow2State *s,
@@ -748,6 +852,69 @@ static int update_ext_header_and_dir_in_place(BlockDriverState *bs,
*/
}
+static int update_ext_header_and_dir(BlockDriverState *bs,
+ Qcow2BitmapList *bm_list)
+{
+ BDRVQcow2State *s = bs->opaque;
+ int ret;
+ uint64_t new_offset = 0;
+ uint64_t new_size = 0;
+ uint32_t new_nb_bitmaps = 0;
+ uint64_t old_offset = s->bitmap_directory_offset;
+ uint64_t old_size = s->bitmap_directory_size;
+ uint32_t old_nb_bitmaps = s->nb_bitmaps;
+ uint64_t old_autocl = s->autoclear_features;
+
+ if (bm_list != NULL && !QSIMPLEQ_EMPTY(bm_list)) {
+ new_nb_bitmaps = bitmap_list_count(bm_list);
+
+ if (new_nb_bitmaps > QCOW2_MAX_BITMAPS) {
+ return -EINVAL;
+ }
+
+ ret = bitmap_list_store(bs, bm_list, &new_offset, &new_size, false);
+ if (ret < 0) {
+ return ret;
+ }
+
+ ret = bdrv_flush(bs->file->bs);
+ if (ret < 0) {
+ goto fail;
+ }
+
+ s->autoclear_features |= QCOW2_AUTOCLEAR_BITMAPS;
+ } else {
+ s->autoclear_features &= ~(uint64_t)QCOW2_AUTOCLEAR_BITMAPS;
+ }
+
+ s->bitmap_directory_offset = new_offset;
+ s->bitmap_directory_size = new_size;
+ s->nb_bitmaps = new_nb_bitmaps;
+
+ ret = update_header_sync(bs);
+ if (ret < 0) {
+ goto fail;
+ }
+
+ if (old_size > 0) {
+ qcow2_free_clusters(bs, old_offset, old_size, QCOW2_DISCARD_OTHER);
+ }
+
+ return 0;
+
+fail:
+ if (new_offset > 0) {
+ qcow2_free_clusters(bs, new_offset, new_size, QCOW2_DISCARD_OTHER);
+ }
+
+ s->bitmap_directory_offset = old_offset;
+ s->bitmap_directory_size = old_size;
+ s->nb_bitmaps = old_nb_bitmaps;
+ s->autoclear_features = old_autocl;
+
+ return ret;
+}
+
/* for g_slist_foreach for GSList of BdrvDirtyBitmap* elements */
static void release_dirty_bitmap_helper(gpointer bitmap,
gpointer bs)
@@ -827,3 +994,311 @@ fail:
return false;
}
+
+/* store_bitmap_data()
+ * Store bitmap to image, filling bitmap table accordingly.
+ */
+static uint64_t *store_bitmap_data(BlockDriverState *bs,
+ BdrvDirtyBitmap *bitmap,
+ uint32_t *bitmap_table_size, Error **errp)
+{
+ int ret;
+ BDRVQcow2State *s = bs->opaque;
+ int64_t sector;
+ uint64_t sbc;
+ uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap);
+ const char *bm_name = bdrv_dirty_bitmap_name(bitmap);
+ uint8_t *buf = NULL;
+ BdrvDirtyBitmapIter *dbi;
+ uint64_t *tb;
+ uint64_t tb_size =
+ size_to_clusters(s,
+ bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_size));
+
+ if (tb_size > BME_MAX_TABLE_SIZE ||
+ tb_size * s->cluster_size > BME_MAX_PHYS_SIZE)
+ {
+ error_setg(errp, "Bitmap '%s' is too big", bm_name);
+ return NULL;
+ }
+
+ tb = g_try_new0(uint64_t, tb_size);
+ if (tb == NULL) {
+ error_setg(errp, "No memory");
+ return NULL;
+ }
+
+ dbi = bdrv_dirty_iter_new(bitmap, 0);
+ buf = g_malloc(s->cluster_size);
+ sbc = sectors_covered_by_bitmap_cluster(s, bitmap);
+ assert(DIV_ROUND_UP(bm_size, sbc) == tb_size);
+
+ while ((sector = bdrv_dirty_iter_next(dbi)) != -1) {
+ uint64_t cluster = sector / sbc;
+ uint64_t end, write_size;
+ int64_t off;
+
+ sector = cluster * sbc;
+ end = MIN(bm_size, sector + sbc);
+ write_size =
+ bdrv_dirty_bitmap_serialization_size(bitmap, sector, end - sector);
+ assert(write_size <= s->cluster_size);
+
+ off = qcow2_alloc_clusters(bs, s->cluster_size);
+ if (off < 0) {
+ error_setg_errno(errp, -off,
+ "Failed to allocate clusters for bitmap '%s'",
+ bm_name);
+ goto fail;
+ }
+ tb[cluster] = off;
+
+ bdrv_dirty_bitmap_serialize_part(bitmap, buf, sector, end - sector);
+ if (write_size < s->cluster_size) {
+ memset(buf + write_size, 0, s->cluster_size - write_size);
+ }
+
+ ret = qcow2_pre_write_overlap_check(bs, 0, off, s->cluster_size);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "Qcow2 overlap check failed");
+ goto fail;
+ }
+
+ ret = bdrv_pwrite(bs->file, off, buf, s->cluster_size);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "Failed to write bitmap '%s' to file",
+ bm_name);
+ goto fail;
+ }
+
+ if (end >= bm_size) {
+ break;
+ }
+
+ bdrv_set_dirty_iter(dbi, end);
+ }
+
+ *bitmap_table_size = tb_size;
+ g_free(buf);
+ bdrv_dirty_iter_free(dbi);
+
+ return tb;
+
+fail:
+ clear_bitmap_table(bs, tb, tb_size);
+ g_free(buf);
+ bdrv_dirty_iter_free(dbi);
+ g_free(tb);
+
+ return NULL;
+}
+
+/* store_bitmap()
+ * Store bm->dirty_bitmap to qcow2.
+ * Set bm->table_offset and bm->table_size accordingly.
+ */
+static int store_bitmap(BlockDriverState *bs, Qcow2Bitmap *bm, Error **errp)
+{
+ int ret;
+ uint64_t *tb;
+ int64_t tb_offset;
+ uint32_t tb_size;
+ BdrvDirtyBitmap *bitmap = bm->dirty_bitmap;
+ const char *bm_name;
+
+ assert(bitmap != NULL);
+
+ bm_name = bdrv_dirty_bitmap_name(bitmap);
+
+ tb = store_bitmap_data(bs, bitmap, &tb_size, errp);
+ if (tb == NULL) {
+ return -EINVAL;
+ }
+
+ assert(tb_size <= BME_MAX_TABLE_SIZE);
+ tb_offset = qcow2_alloc_clusters(bs, tb_size * sizeof(tb[0]));
+ if (tb_offset < 0) {
+ error_setg_errno(errp, -tb_offset,
+ "Failed to allocate clusters for bitmap '%s'",
+ bm_name);
+ goto fail;
+ }
+
+ ret = qcow2_pre_write_overlap_check(bs, 0, tb_offset,
+ tb_size * sizeof(tb[0]));
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "Qcow2 overlap check failed");
+ goto fail;
+ }
+
+ bitmap_table_to_be(tb, tb_size);
+ ret = bdrv_pwrite(bs->file, tb_offset, tb, tb_size * sizeof(tb[0]));
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "Failed to write bitmap '%s' to file",
+ bm_name);
+ goto fail;
+ }
+
+ g_free(tb);
+
+ bm->table.offset = tb_offset;
+ bm->table.size = tb_size;
+
+ return 0;
+
+fail:
+ clear_bitmap_table(bs, tb, tb_size);
+
+ if (tb_offset > 0) {
+ qcow2_free_clusters(bs, tb_offset, tb_size * sizeof(tb[0]),
+ QCOW2_DISCARD_OTHER);
+ }
+
+ g_free(tb);
+
+ return ret;
+}
+
+static Qcow2Bitmap *find_bitmap_by_name(Qcow2BitmapList *bm_list,
+ const char *name)
+{
+ Qcow2Bitmap *bm;
+
+ QSIMPLEQ_FOREACH(bm, bm_list, entry) {
+ if (strcmp(name, bm->name) == 0) {
+ return bm;
+ }
+ }
+
+ return NULL;
+}
+
+void qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp)
+{
+ BdrvDirtyBitmap *bitmap;
+ BDRVQcow2State *s = bs->opaque;
+ uint32_t new_nb_bitmaps = s->nb_bitmaps;
+ uint64_t new_dir_size = s->bitmap_directory_size;
+ int ret;
+ Qcow2BitmapList *bm_list;
+ Qcow2Bitmap *bm;
+ Qcow2BitmapTableList drop_tables;
+ Qcow2BitmapTable *tb, *tb_next;
+
+ if (!bdrv_has_changed_persistent_bitmaps(bs)) {
+ /* nothing to do */
+ return;
+ }
+
+ if (!can_write(bs)) {
+ error_setg(errp, "No write access");
+ return;
+ }
+
+ QSIMPLEQ_INIT(&drop_tables);
+
+ if (s->nb_bitmaps == 0) {
+ bm_list = bitmap_list_new();
+ } else {
+ bm_list = bitmap_list_load(bs, s->bitmap_directory_offset,
+ s->bitmap_directory_size, errp);
+ if (bm_list == NULL) {
+ return;
+ }
+ }
+
+ /* check constraints and names */
+ for (bitmap = bdrv_dirty_bitmap_next(bs, NULL); bitmap != NULL;
+ bitmap = bdrv_dirty_bitmap_next(bs, bitmap))
+ {
+ const char *name = bdrv_dirty_bitmap_name(bitmap);
+ uint32_t granularity = bdrv_dirty_bitmap_granularity(bitmap);
+ Qcow2Bitmap *bm;
+
+ if (!bdrv_dirty_bitmap_get_persistance(bitmap) ||
+ bdrv_dirty_bitmap_readonly(bitmap))
+ {
+ continue;
+ }
+
+ if (check_constraints_on_bitmap(bs, name, granularity, errp) < 0) {
+ error_prepend(errp, "Bitmap '%s' doesn't satisfy the constraints: ",
+ name);
+ goto fail;
+ }
+
+ bm = find_bitmap_by_name(bm_list, name);
+ if (bm == NULL) {
+ if (++new_nb_bitmaps > QCOW2_MAX_BITMAPS) {
+ error_setg(errp, "Too many persistent bitmaps");
+ goto fail;
+ }
+
+ new_dir_size += calc_dir_entry_size(strlen(name), 0);
+ if (new_dir_size > QCOW2_MAX_BITMAP_DIRECTORY_SIZE) {
+ error_setg(errp, "Bitmap directory is too large");
+ goto fail;
+ }
+
+ bm = g_new0(Qcow2Bitmap, 1);
+ bm->name = g_strdup(name);
+ QSIMPLEQ_INSERT_TAIL(bm_list, bm, entry);
+ } else {
+ if (!(bm->flags & BME_FLAG_IN_USE)) {
+ error_setg(errp, "Bitmap '%s' already exists in the image",
+ name);
+ goto fail;
+ }
+ tb = g_memdup(&bm->table, sizeof(bm->table));
+ bm->table.offset = 0;
+ bm->table.size = 0;
+ QSIMPLEQ_INSERT_TAIL(&drop_tables, tb, entry);
+ }
+ bm->flags = bdrv_dirty_bitmap_get_autoload(bitmap) ? BME_FLAG_AUTO : 0;
+ bm->granularity_bits = ctz32(bdrv_dirty_bitmap_granularity(bitmap));
+ bm->dirty_bitmap = bitmap;
+ }
+
+ /* allocate clusters and store bitmaps */
+ QSIMPLEQ_FOREACH(bm, bm_list, entry) {
+ if (bm->dirty_bitmap == NULL) {
+ continue;
+ }
+
+ ret = store_bitmap(bs, bm, errp);
+ if (ret < 0) {
+ goto fail;
+ }
+ }
+
+ ret = update_ext_header_and_dir(bs, bm_list);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "Failed to update bitmap extension");
+ goto fail;
+ }
+
+ /* Bitmap directory was successfully updated, so, old data can be dropped.
+ * TODO it is better to reuse these clusters */
+ QSIMPLEQ_FOREACH_SAFE(tb, &drop_tables, entry, tb_next) {
+ free_bitmap_clusters(bs, tb);
+ g_free(tb);
+ }
+
+ bitmap_list_free(bm_list);
+ return;
+
+fail:
+ QSIMPLEQ_FOREACH(bm, bm_list, entry) {
+ if (bm->dirty_bitmap == NULL || bm->table.offset == 0) {
+ continue;
+ }
+
+ free_bitmap_clusters(bs, &bm->table);
+ }
+
+ QSIMPLEQ_FOREACH_SAFE(tb, &drop_tables, entry, tb_next) {
+ g_free(tb);
+ }
+
+ bitmap_list_free(bm_list);
+}
diff --git a/block/qcow2.c b/block/qcow2.c
index 4d3654d606..f3bda2345d 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1823,6 +1823,7 @@ static int qcow2_inactivate(BlockDriverState *bs)
{
BDRVQcow2State *s = bs->opaque;
int ret, result = 0;
+ Error *local_err = NULL;
ret = qcow2_cache_flush(bs, s->l2_table_cache);
if (ret) {
@@ -1838,6 +1839,14 @@ static int qcow2_inactivate(BlockDriverState *bs)
strerror(-ret));
}
+ qcow2_store_persistent_dirty_bitmaps(bs, &local_err);
+ if (local_err != NULL) {
+ result = -EINVAL;
+ error_report_err(local_err);
+ error_report("Persistent bitmaps are lost for node '%s'",
+ bdrv_get_device_or_node_name(bs));
+ }
+
if (result == 0) {
qcow2_mark_clean(bs);
}
diff --git a/block/qcow2.h b/block/qcow2.h
index 15e9791faa..b79f780c02 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -622,5 +622,6 @@ int qcow2_check_bitmaps_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
void **refcount_table,
int64_t *refcount_table_size);
bool qcow2_load_autoloading_dirty_bitmaps(BlockDriverState *bs, Error **errp);
+void qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp);
#endif
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 16/25] block: add bdrv_can_store_new_dirty_bitmap
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (14 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 15/25] qcow2: add persistent dirty bitmaps support Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 17/25] qcow2: add .bdrv_can_store_new_dirty_bitmap Vladimir Sementsov-Ogievskiy
` (9 subsequent siblings)
25 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
This will be needed to check some restrictions before making bitmap
persistent in qmp-block-dirty-bitmap-add (this functionality will be
added by future patch)
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
block.c | 22 ++++++++++++++++++++++
include/block/block.h | 3 +++
include/block/block_int.h | 5 +++++
3 files changed, 30 insertions(+)
diff --git a/block.c b/block.c
index 17c853f3ee..795d36bb64 100644
--- a/block.c
+++ b/block.c
@@ -4804,3 +4804,25 @@ void bdrv_del_child(BlockDriverState *parent_bs, BdrvChild *child, Error **errp)
parent_bs->drv->bdrv_del_child(parent_bs, child, errp);
}
+
+bool bdrv_can_store_new_dirty_bitmap(BlockDriverState *bs, const char *name,
+ uint32_t granularity, Error **errp)
+{
+ BlockDriver *drv = bs->drv;
+
+ if (!drv) {
+ error_setg_errno(errp, ENOMEDIUM,
+ "Can't store persistent bitmaps to %s",
+ bdrv_get_device_or_node_name(bs));
+ return false;
+ }
+
+ if (!drv->bdrv_can_store_new_dirty_bitmap) {
+ error_setg_errno(errp, ENOTSUP,
+ "Can't store persistent bitmaps to %s",
+ bdrv_get_device_or_node_name(bs));
+ return false;
+ }
+
+ return drv->bdrv_can_store_new_dirty_bitmap(bs, name, granularity, errp);
+}
diff --git a/include/block/block.h b/include/block/block.h
index 144df0ddfb..f92529a4e8 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -624,4 +624,7 @@ void bdrv_add_child(BlockDriverState *parent, BlockDriverState *child,
Error **errp);
void bdrv_del_child(BlockDriverState *parent, BdrvChild *child, Error **errp);
+bool bdrv_can_store_new_dirty_bitmap(BlockDriverState *bs, const char *name,
+ uint32_t granularity, Error **errp);
+
#endif
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 4f8cd29ae4..9092e78ade 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -373,6 +373,11 @@ struct BlockDriver {
uint64_t parent_perm, uint64_t parent_shared,
uint64_t *nperm, uint64_t *nshared);
+ bool (*bdrv_can_store_new_dirty_bitmap)(BlockDriverState *bs,
+ const char *name,
+ uint32_t granularity,
+ Error **errp);
+
QLIST_ENTRY(BlockDriver) list;
};
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 17/25] qcow2: add .bdrv_can_store_new_dirty_bitmap
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (15 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 16/25] block: add bdrv_can_store_new_dirty_bitmap Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 18/25] qmp: add persistent flag to block-dirty-bitmap-add Vladimir Sementsov-Ogievskiy
` (8 subsequent siblings)
25 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Realize .bdrv_can_store_new_dirty_bitmap interface.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
---
block/qcow2-bitmap.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++
block/qcow2.c | 2 ++
block/qcow2.h | 4 ++++
3 files changed, 57 insertions(+)
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index 663c5b2ac7..d0b7c576a2 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -1302,3 +1302,54 @@ fail:
bitmap_list_free(bm_list);
}
+
+bool qcow2_can_store_new_dirty_bitmap(BlockDriverState *bs,
+ const char *name,
+ uint32_t granularity,
+ Error **errp)
+{
+ BDRVQcow2State *s = bs->opaque;
+ bool found;
+ Qcow2BitmapList *bm_list;
+
+ if (check_constraints_on_bitmap(bs, name, granularity, errp) != 0) {
+ goto fail;
+ }
+
+ if (s->nb_bitmaps == 0) {
+ return true;
+ }
+
+ if (s->nb_bitmaps >= QCOW2_MAX_BITMAPS) {
+ error_setg(errp,
+ "Maximum number of persistent bitmaps is already reached");
+ goto fail;
+ }
+
+ if (s->bitmap_directory_size + calc_dir_entry_size(strlen(name), 0) >
+ QCOW2_MAX_BITMAP_DIRECTORY_SIZE)
+ {
+ error_setg(errp, "Not enough space in the bitmap directory");
+ goto fail;
+ }
+
+ bm_list = bitmap_list_load(bs, s->bitmap_directory_offset,
+ s->bitmap_directory_size, errp);
+ if (bm_list == NULL) {
+ goto fail;
+ }
+
+ found = find_bitmap_by_name(bm_list, name);
+ bitmap_list_free(bm_list);
+ if (found) {
+ error_setg(errp, "Bitmap with the same name is already stored");
+ goto fail;
+ }
+
+ return true;
+
+fail:
+ error_prepend(errp, "Can't make bitmap '%s' persistent in '%s': ",
+ name, bdrv_get_device_or_node_name(bs));
+ return false;
+}
diff --git a/block/qcow2.c b/block/qcow2.c
index f3bda2345d..e813944bf2 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -3587,6 +3587,8 @@ BlockDriver bdrv_qcow2 = {
.bdrv_detach_aio_context = qcow2_detach_aio_context,
.bdrv_attach_aio_context = qcow2_attach_aio_context,
+
+ .bdrv_can_store_new_dirty_bitmap = qcow2_can_store_new_dirty_bitmap,
};
static void bdrv_qcow2_init(void)
diff --git a/block/qcow2.h b/block/qcow2.h
index b79f780c02..a054bc1dbf 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -623,5 +623,9 @@ int qcow2_check_bitmaps_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
int64_t *refcount_table_size);
bool qcow2_load_autoloading_dirty_bitmaps(BlockDriverState *bs, Error **errp);
void qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp);
+bool qcow2_can_store_new_dirty_bitmap(BlockDriverState *bs,
+ const char *name,
+ uint32_t granularity,
+ Error **errp);
#endif
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 18/25] qmp: add persistent flag to block-dirty-bitmap-add
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (16 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 17/25] qcow2: add .bdrv_can_store_new_dirty_bitmap Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 19/25] qmp: add autoload parameter " Vladimir Sementsov-Ogievskiy
` (7 subsequent siblings)
25 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Add optional 'persistent' flag to qmp command block-dirty-bitmap-add.
Default is false.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
blockdev.c | 18 +++++++++++++++++-
qapi/block-core.json | 8 +++++++-
2 files changed, 24 insertions(+), 2 deletions(-)
diff --git a/blockdev.c b/blockdev.c
index 64282065d8..9e5206867c 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1985,6 +1985,7 @@ static void block_dirty_bitmap_add_prepare(BlkActionState *common,
/* AIO context taken and released within qmp_block_dirty_bitmap_add */
qmp_block_dirty_bitmap_add(action->node, action->name,
action->has_granularity, action->granularity,
+ action->has_persistent, action->persistent,
&local_err);
if (!local_err) {
@@ -2731,10 +2732,12 @@ out:
void qmp_block_dirty_bitmap_add(const char *node, const char *name,
bool has_granularity, uint32_t granularity,
+ bool has_persistent, bool persistent,
Error **errp)
{
AioContext *aio_context;
BlockDriverState *bs;
+ BdrvDirtyBitmap *bitmap;
if (!name || name[0] == '\0') {
error_setg(errp, "Bitmap name cannot be empty");
@@ -2760,7 +2763,20 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name,
granularity = bdrv_get_default_bitmap_granularity(bs);
}
- bdrv_create_dirty_bitmap(bs, granularity, name, errp);
+ if (!has_persistent) {
+ persistent = false;
+ }
+
+ if (persistent &&
+ !bdrv_can_store_new_dirty_bitmap(bs, name, granularity, errp))
+ {
+ goto out;
+ }
+
+ bitmap = bdrv_create_dirty_bitmap(bs, granularity, name, errp);
+ if (bitmap != NULL) {
+ bdrv_dirty_bitmap_set_persistance(bitmap, persistent);
+ }
out:
aio_context_release(aio_context);
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 87fb747ab6..499e0e32aa 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1560,10 +1560,16 @@
# @granularity: the bitmap granularity, default is 64k for
# block-dirty-bitmap-add
#
+# @persistent: the bitmap is persistent, i.e. it will be saved to the
+# corresponding block device image file on its close. For now only
+# Qcow2 disks support persistent bitmaps. Default is false for
+# block-dirty-bitmap-add. (Since: 2.10)
+#
# Since: 2.4
##
{ 'struct': 'BlockDirtyBitmapAdd',
- 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32' } }
+ 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32',
+ '*persistent': 'bool' } }
##
# @block-dirty-bitmap-add:
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 19/25] qmp: add autoload parameter to block-dirty-bitmap-add
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (17 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 18/25] qmp: add persistent flag to block-dirty-bitmap-add Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 20/25] qmp: add x-debug-block-dirty-bitmap-sha256 Vladimir Sementsov-Ogievskiy
` (6 subsequent siblings)
25 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Optional. Default is false.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
blockdev.c | 18 ++++++++++++++++--
qapi/block-core.json | 6 +++++-
2 files changed, 21 insertions(+), 3 deletions(-)
diff --git a/blockdev.c b/blockdev.c
index 9e5206867c..cec6cfcbc4 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1986,6 +1986,7 @@ static void block_dirty_bitmap_add_prepare(BlkActionState *common,
qmp_block_dirty_bitmap_add(action->node, action->name,
action->has_granularity, action->granularity,
action->has_persistent, action->persistent,
+ action->has_autoload, action->autoload,
&local_err);
if (!local_err) {
@@ -2733,6 +2734,7 @@ out:
void qmp_block_dirty_bitmap_add(const char *node, const char *name,
bool has_granularity, uint32_t granularity,
bool has_persistent, bool persistent,
+ bool has_autoload, bool autoload,
Error **errp)
{
AioContext *aio_context;
@@ -2766,6 +2768,15 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name,
if (!has_persistent) {
persistent = false;
}
+ if (!has_autoload) {
+ autoload = false;
+ }
+
+ if (has_autoload && !persistent) {
+ error_setg(errp, "Autoload flag must be used only for persistent "
+ "bitmaps");
+ goto out;
+ }
if (persistent &&
!bdrv_can_store_new_dirty_bitmap(bs, name, granularity, errp))
@@ -2774,10 +2785,13 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name,
}
bitmap = bdrv_create_dirty_bitmap(bs, granularity, name, errp);
- if (bitmap != NULL) {
- bdrv_dirty_bitmap_set_persistance(bitmap, persistent);
+ if (bitmap == NULL) {
+ goto out;
}
+ bdrv_dirty_bitmap_set_persistance(bitmap, persistent);
+ bdrv_dirty_bitmap_set_autoload(bitmap, autoload);
+
out:
aio_context_release(aio_context);
}
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 499e0e32aa..3abc894ecb 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1565,11 +1565,15 @@
# Qcow2 disks support persistent bitmaps. Default is false for
# block-dirty-bitmap-add. (Since: 2.10)
#
+# @autoload: the bitmap will be automatically loaded when the image it is stored
+# in is opened. This flag may only be specified for persistent
+# bitmaps. Default is false for block-dirty-bitmap-add. (Since: 2.10)
+#
# Since: 2.4
##
{ 'struct': 'BlockDirtyBitmapAdd',
'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32',
- '*persistent': 'bool' } }
+ '*persistent': 'bool', '*autoload': 'bool' } }
##
# @block-dirty-bitmap-add:
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 20/25] qmp: add x-debug-block-dirty-bitmap-sha256
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (18 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 19/25] qmp: add autoload parameter " Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 21/25] iotests: test qcow2 persistent dirty bitmap Vladimir Sementsov-Ogievskiy
` (5 subsequent siblings)
25 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
block/dirty-bitmap.c | 5 +++++
blockdev.c | 29 +++++++++++++++++++++++++++++
include/block/dirty-bitmap.h | 2 ++
include/qemu/hbitmap.h | 8 ++++++++
qapi/block-core.json | 27 +++++++++++++++++++++++++++
tests/Makefile.include | 2 +-
util/hbitmap.c | 11 +++++++++++
7 files changed, 83 insertions(+), 1 deletion(-)
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index df0d3cd954..9fc3c0cb54 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -604,3 +604,8 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,
return bitmap == NULL ? QLIST_FIRST(&bs->dirty_bitmaps) :
QLIST_NEXT(bitmap, list);
}
+
+char *bdrv_dirty_bitmap_sha256(const BdrvDirtyBitmap *bitmap, Error **errp)
+{
+ return hbitmap_sha256(bitmap->bitmap, errp);
+}
diff --git a/blockdev.c b/blockdev.c
index cec6cfcbc4..c49d414891 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -2855,6 +2855,35 @@ void qmp_block_dirty_bitmap_clear(const char *node, const char *name,
aio_context_release(aio_context);
}
+BlockDirtyBitmapSha256 *qmp_x_debug_block_dirty_bitmap_sha256(const char *node,
+ const char *name,
+ Error **errp)
+{
+ AioContext *aio_context;
+ BdrvDirtyBitmap *bitmap;
+ BlockDriverState *bs;
+ BlockDirtyBitmapSha256 *ret = NULL;
+ char *sha256;
+
+ bitmap = block_dirty_bitmap_lookup(node, name, &bs, &aio_context, errp);
+ if (!bitmap || !bs) {
+ return NULL;
+ }
+
+ sha256 = bdrv_dirty_bitmap_sha256(bitmap, errp);
+ if (sha256 == NULL) {
+ goto out;
+ }
+
+ ret = g_new(BlockDirtyBitmapSha256, 1);
+ ret->sha256 = sha256;
+
+out:
+ aio_context_release(aio_context);
+
+ return ret;
+}
+
void hmp_drive_del(Monitor *mon, const QDict *qdict)
{
const char *id = qdict_get_str(qdict, "id");
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 741573be97..8aac54278c 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -89,4 +89,6 @@ bool bdrv_has_changed_persistent_bitmaps(BlockDriverState *bs);
BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,
BdrvDirtyBitmap *bitmap);
+char *bdrv_dirty_bitmap_sha256(const BdrvDirtyBitmap *bitmap, Error **errp);
+
#endif
diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
index b52304ac29..d3a74a21fc 100644
--- a/include/qemu/hbitmap.h
+++ b/include/qemu/hbitmap.h
@@ -253,6 +253,14 @@ void hbitmap_deserialize_ones(HBitmap *hb, uint64_t start, uint64_t count,
void hbitmap_deserialize_finish(HBitmap *hb);
/**
+ * hbitmap_sha256:
+ * @bitmap: HBitmap to operate on.
+ *
+ * Returns SHA256 hash of the last level.
+ */
+char *hbitmap_sha256(const HBitmap *bitmap, Error **errp);
+
+/**
* hbitmap_free:
* @hb: HBitmap to operate on.
*
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 3abc894ecb..e9523b4b8c 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1643,6 +1643,33 @@
'data': 'BlockDirtyBitmap' }
##
+# @BlockDirtyBitmapSha256:
+#
+# SHA256 hash of dirty bitmap data
+#
+# @sha256: ASCII representation of SHA256 bitmap hash
+#
+# Since: 2.10
+##
+ { 'struct': 'BlockDirtyBitmapSha256',
+ 'data': {'sha256': 'str'} }
+
+##
+# @x-debug-block-dirty-bitmap-sha256:
+#
+# Get bitmap SHA256
+#
+# Returns: BlockDirtyBitmapSha256 on success
+# If @node is not a valid block device, DeviceNotFound
+# If @name is not found or if hashing has failed, GenericError with an
+# explanation
+#
+# Since: 2.10
+##
+ { 'command': 'x-debug-block-dirty-bitmap-sha256',
+ 'data': 'BlockDirtyBitmap', 'returns': 'BlockDirtyBitmapSha256' }
+
+##
# @blockdev-mirror:
#
# Start mirroring a block device's writes to a new destination.
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 31931c0d77..38686f7f86 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -548,7 +548,7 @@ tests/test-blockjob$(EXESUF): tests/test-blockjob.o $(test-block-obj-y) $(test-u
tests/test-blockjob-txn$(EXESUF): tests/test-blockjob-txn.o $(test-block-obj-y) $(test-util-obj-y)
tests/test-thread-pool$(EXESUF): tests/test-thread-pool.o $(test-block-obj-y)
tests/test-iov$(EXESUF): tests/test-iov.o $(test-util-obj-y)
-tests/test-hbitmap$(EXESUF): tests/test-hbitmap.o $(test-util-obj-y)
+tests/test-hbitmap$(EXESUF): tests/test-hbitmap.o $(test-util-obj-y) $(test-crypto-obj-y)
tests/test-x86-cpuid$(EXESUF): tests/test-x86-cpuid.o
tests/test-xbzrle$(EXESUF): tests/test-xbzrle.o migration/xbzrle.o page_cache.o $(test-util-obj-y)
tests/test-cutils$(EXESUF): tests/test-cutils.o util/cutils.o
diff --git a/util/hbitmap.c b/util/hbitmap.c
index 0c1591a594..21535cc90b 100644
--- a/util/hbitmap.c
+++ b/util/hbitmap.c
@@ -13,6 +13,7 @@
#include "qemu/hbitmap.h"
#include "qemu/host-utils.h"
#include "trace.h"
+#include "crypto/hash.h"
/* HBitmaps provides an array of bits. The bits are stored as usual in an
* array of unsigned longs, but HBitmap is also optimized to provide fast
@@ -727,3 +728,13 @@ void hbitmap_free_meta(HBitmap *hb)
hbitmap_free(hb->meta);
hb->meta = NULL;
}
+
+char *hbitmap_sha256(const HBitmap *bitmap, Error **errp)
+{
+ size_t size = bitmap->sizes[HBITMAP_LEVELS - 1] * sizeof(unsigned long);
+ char *data = (char *)bitmap->levels[HBITMAP_LEVELS - 1];
+ char *hash = NULL;
+ qcrypto_hash_digest(QCRYPTO_HASH_ALG_SHA256, data, size, &hash, errp);
+
+ return hash;
+}
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 21/25] iotests: test qcow2 persistent dirty bitmap
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (19 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 20/25] qmp: add x-debug-block-dirty-bitmap-sha256 Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-29 17:31 ` Max Reitz
2017-05-03 12:25 ` [Qemu-devel] [PATCH 22/25] block/dirty-bitmap: add bdrv_remove_persistent_dirty_bitmap Vladimir Sementsov-Ogievskiy
` (4 subsequent siblings)
25 siblings, 1 reply; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/165 | 105 +++++++++++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/165.out | 5 +++
tests/qemu-iotests/group | 1 +
3 files changed, 111 insertions(+)
create mode 100755 tests/qemu-iotests/165
create mode 100644 tests/qemu-iotests/165.out
diff --git a/tests/qemu-iotests/165 b/tests/qemu-iotests/165
new file mode 100755
index 0000000000..74d7b79a0b
--- /dev/null
+++ b/tests/qemu-iotests/165
@@ -0,0 +1,105 @@
+#!/usr/bin/env python
+#
+# Tests for persistent dirty bitmaps.
+#
+# Copyright: Vladimir Sementsov-Ogievskiy 2015-2017
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import re
+import iotests
+from iotests import qemu_img
+
+disk = os.path.join(iotests.test_dir, 'disk')
+disk_size = 0x40000000 # 1G
+
+# regions for qemu_io: (start, count) in bytes
+regions1 = ((0, 0x100000),
+ (0x200000, 0x100000))
+
+regions2 = ((0x10000000, 0x20000),
+ (0x3fff0000, 0x10000))
+
+class TestPersistentDirtyBitmap(iotests.QMPTestCase):
+
+ def setUp(self):
+ qemu_img('create', '-f', iotests.imgfmt, disk, str(disk_size))
+
+ def tearDown(self):
+ os.remove(disk)
+
+ def mkVm(self):
+ return iotests.VM().add_drive(disk)
+
+ def mkVmRo(self):
+ return iotests.VM().add_drive(disk, opts='readonly=on')
+
+ def getSha256(self):
+ result = self.vm.qmp('x-debug-block-dirty-bitmap-sha256',
+ node='drive0', name='bitmap0')
+ return result['return']['sha256']
+
+ def checkBitmap(self, sha256):
+ result = self.vm.qmp('x-debug-block-dirty-bitmap-sha256',
+ node='drive0', name='bitmap0')
+ self.assert_qmp(result, 'return/sha256', sha256);
+
+ def writeRegions(self, regions):
+ for r in regions:
+ self.vm.hmp_qemu_io('drive0',
+ 'write %d %d' % r)
+
+ def qmpAddBitmap(self):
+ self.vm.qmp('block-dirty-bitmap-add', node='drive0',
+ name='bitmap0', persistent=True, autoload=True)
+
+ def test_persistent(self):
+ self.vm = self.mkVm()
+ self.vm.launch()
+ self.qmpAddBitmap()
+
+ self.writeRegions(regions1)
+ sha256 = self.getSha256()
+
+ self.vm.shutdown()
+
+ self.vm = self.mkVmRo()
+ self.vm.launch()
+ self.vm.shutdown()
+
+ #catch 'Persistent bitmaps are lost' possible error
+ log = self.vm.get_log()
+ log = re.sub(r'^\[I \d+\.\d+\] OPENED\n', '', log)
+ log = re.sub(r'\[I \+\d+\.\d+\] CLOSED\n?$', '', log)
+ if log:
+ print log
+
+ self.vm = self.mkVm()
+ self.vm.launch()
+
+ self.checkBitmap(sha256)
+ self.writeRegions(regions2)
+ sha256 = self.getSha256()
+
+ self.vm.shutdown()
+ self.vm.launch()
+
+ self.checkBitmap(sha256)
+
+ self.vm.shutdown()
+
+if __name__ == '__main__':
+ iotests.main(supported_fmts=['qcow2'])
diff --git a/tests/qemu-iotests/165.out b/tests/qemu-iotests/165.out
new file mode 100644
index 0000000000..ae1213e6f8
--- /dev/null
+++ b/tests/qemu-iotests/165.out
@@ -0,0 +1,5 @@
+.
+----------------------------------------------------------------------
+Ran 1 tests
+
+OK
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 43142ddfcf..60fcb282f9 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -162,6 +162,7 @@
159 rw auto quick
160 rw auto quick
162 auto quick
+165 rw auto quick
170 rw auto quick
171 rw auto quick
172 auto
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 22/25] block/dirty-bitmap: add bdrv_remove_persistent_dirty_bitmap
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (20 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 21/25] iotests: test qcow2 persistent dirty bitmap Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 23/25] qcow2: add .bdrv_remove_persistent_dirty_bitmap Vladimir Sementsov-Ogievskiy
` (3 subsequent siblings)
25 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Interface for removing persistent bitmap from its storage.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
block/dirty-bitmap.c | 18 ++++++++++++++++++
include/block/block_int.h | 3 +++
include/block/dirty-bitmap.h | 3 +++
3 files changed, 24 insertions(+)
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 9fc3c0cb54..659617a3f8 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -329,12 +329,30 @@ void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
/**
* Release all named dirty bitmaps attached to a BDS (for use in bdrv_close()).
* There must not be any frozen bitmaps attached.
+ * This function does not remove persistent bitmaps from the storage.
*/
void bdrv_release_named_dirty_bitmaps(BlockDriverState *bs)
{
bdrv_do_release_matching_dirty_bitmap(bs, NULL, true);
}
+/**
+ * Remove persistent dirty bitmap from the storage if it exists.
+ * Absence of bitmap is not an error, because we have the following scenario:
+ * BdrvDirtyBitmap can have .persistent = true but not yet saved and have no
+ * stored version. For such bitmap bdrv_remove_persistent_dirty_bitmap() should
+ * not fail.
+ * This function doesn't release corresponding BdrvDirtyBitmap.
+ */
+void bdrv_remove_persistent_dirty_bitmap(BlockDriverState *bs,
+ const char *name,
+ Error **errp)
+{
+ if (bs->drv && bs->drv->bdrv_remove_persistent_dirty_bitmap) {
+ bs->drv->bdrv_remove_persistent_dirty_bitmap(bs, name, errp);
+ }
+}
+
void bdrv_disable_dirty_bitmap(BdrvDirtyBitmap *bitmap)
{
assert(!bdrv_dirty_bitmap_frozen(bitmap));
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 9092e78ade..d1550fda0b 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -377,6 +377,9 @@ struct BlockDriver {
const char *name,
uint32_t granularity,
Error **errp);
+ void (*bdrv_remove_persistent_dirty_bitmap)(BlockDriverState *bs,
+ const char *name,
+ Error **errp);
QLIST_ENTRY(BlockDriver) list;
};
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 8aac54278c..2ee63c46e9 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -25,6 +25,9 @@ BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs,
void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap);
void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
void bdrv_release_named_dirty_bitmaps(BlockDriverState *bs);
+void bdrv_remove_persistent_dirty_bitmap(BlockDriverState *bs,
+ const char *name,
+ Error **errp);
void bdrv_disable_dirty_bitmap(BdrvDirtyBitmap *bitmap);
void bdrv_enable_dirty_bitmap(BdrvDirtyBitmap *bitmap);
BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs);
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 23/25] qcow2: add .bdrv_remove_persistent_dirty_bitmap
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (21 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 22/25] block/dirty-bitmap: add bdrv_remove_persistent_dirty_bitmap Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 24/25] qmp: block-dirty-bitmap-remove: remove persistent Vladimir Sementsov-Ogievskiy
` (2 subsequent siblings)
25 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Realize .bdrv_remove_persistent_dirty_bitmap interface.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
block/qcow2-bitmap.c | 41 +++++++++++++++++++++++++++++++++++++++++
block/qcow2.c | 1 +
block/qcow2.h | 3 +++
3 files changed, 45 insertions(+)
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index d0b7c576a2..0d47b9fdb4 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -1173,6 +1173,47 @@ static Qcow2Bitmap *find_bitmap_by_name(Qcow2BitmapList *bm_list,
return NULL;
}
+void qcow2_remove_persistent_dirty_bitmap(BlockDriverState *bs,
+ const char *name,
+ Error **errp)
+{
+ int ret;
+ BDRVQcow2State *s = bs->opaque;
+ Qcow2Bitmap *bm;
+ Qcow2BitmapList *bm_list;
+
+ if (s->nb_bitmaps == 0) {
+ /* Absence of the bitmap is not an error: see explanation above
+ * bdrv_remove_persistent_dirty_bitmap() definition. */
+ return;
+ }
+
+ bm_list = bitmap_list_load(bs, s->bitmap_directory_offset,
+ s->bitmap_directory_size, errp);
+ if (bm_list == NULL) {
+ return;
+ }
+
+ bm = find_bitmap_by_name(bm_list, name);
+ if (bm == NULL) {
+ goto fail;
+ }
+
+ QSIMPLEQ_REMOVE(bm_list, bm, Qcow2Bitmap, entry);
+
+ ret = update_ext_header_and_dir(bs, bm_list);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "Failed to update bitmap extension");
+ goto fail;
+ }
+
+ free_bitmap_clusters(bs, &bm->table);
+
+fail:
+ bitmap_free(bm);
+ bitmap_list_free(bm_list);
+}
+
void qcow2_store_persistent_dirty_bitmaps(BlockDriverState *bs, Error **errp)
{
BdrvDirtyBitmap *bitmap;
diff --git a/block/qcow2.c b/block/qcow2.c
index e813944bf2..d29fb564b5 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -3589,6 +3589,7 @@ BlockDriver bdrv_qcow2 = {
.bdrv_attach_aio_context = qcow2_attach_aio_context,
.bdrv_can_store_new_dirty_bitmap = qcow2_can_store_new_dirty_bitmap,
+ .bdrv_remove_persistent_dirty_bitmap = qcow2_remove_persistent_dirty_bitmap,
};
static void bdrv_qcow2_init(void)
diff --git a/block/qcow2.h b/block/qcow2.h
index a054bc1dbf..0b3fa98e6e 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -627,5 +627,8 @@ bool qcow2_can_store_new_dirty_bitmap(BlockDriverState *bs,
const char *name,
uint32_t granularity,
Error **errp);
+void qcow2_remove_persistent_dirty_bitmap(BlockDriverState *bs,
+ const char *name,
+ Error **errp);
#endif
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 24/25] qmp: block-dirty-bitmap-remove: remove persistent
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (22 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 23/25] qcow2: add .bdrv_remove_persistent_dirty_bitmap Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-29 17:38 ` Max Reitz
2017-05-03 12:25 ` [Qemu-devel] [PATCH 25/25] block: release persistent bitmaps on inactivate Vladimir Sementsov-Ogievskiy
2017-05-15 9:51 ` [Qemu-devel] ping Re: [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
25 siblings, 1 reply; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Remove persistent bitmap from the storage on block-dirty-bitmap-remove.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
blockdev.c | 10 ++++++++++
qapi/block-core.json | 3 ++-
2 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/blockdev.c b/blockdev.c
index c49d414891..0ad1a658b9 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -2802,6 +2802,7 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name,
AioContext *aio_context;
BlockDriverState *bs;
BdrvDirtyBitmap *bitmap;
+ Error *local_err = NULL;
bitmap = block_dirty_bitmap_lookup(node, name, &bs, &aio_context, errp);
if (!bitmap || !bs) {
@@ -2814,6 +2815,15 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name,
name);
goto out;
}
+
+ if (bdrv_dirty_bitmap_get_persistance(bitmap)) {
+ bdrv_remove_persistent_dirty_bitmap(bs, name, &local_err);
+ if (local_err != NULL) {
+ error_propagate(errp, local_err);
+ goto out;
+ }
+ }
+
bdrv_dirty_bitmap_make_anon(bitmap);
bdrv_release_dirty_bitmap(bs, bitmap);
diff --git a/qapi/block-core.json b/qapi/block-core.json
index e9523b4b8c..933b1a5fe5 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1600,7 +1600,8 @@
# @block-dirty-bitmap-remove:
#
# Stop write tracking and remove the dirty bitmap that was created
-# with block-dirty-bitmap-add.
+# with block-dirty-bitmap-add. If bitmap is persistent, remove it from it's
+# storage too.
#
# Returns: nothing on success
# If @node is not a valid block device or node, DeviceNotFound
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 25/25] block: release persistent bitmaps on inactivate
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (23 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 24/25] qmp: block-dirty-bitmap-remove: remove persistent Vladimir Sementsov-Ogievskiy
@ 2017-05-03 12:25 ` Vladimir Sementsov-Ogievskiy
2017-05-29 17:54 ` Max Reitz
2017-05-15 9:51 ` [Qemu-devel] ping Re: [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
25 siblings, 1 reply; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-03 12:25 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
We should release them here to reload on invalidate cache.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
block.c | 4 ++++
block/dirty-bitmap.c | 29 +++++++++++++++++++++++------
include/block/dirty-bitmap.h | 1 +
3 files changed, 28 insertions(+), 6 deletions(-)
diff --git a/block.c b/block.c
index 795d36bb64..14896c65fa 100644
--- a/block.c
+++ b/block.c
@@ -4001,6 +4001,10 @@ static int bdrv_inactivate_recurse(BlockDriverState *bs,
if (setting_flag) {
bs->open_flags |= BDRV_O_INACTIVE;
}
+
+ /* At this point persistent bitmaps should be stored by format driver */
+ bdrv_release_persistent_dirty_bitmaps(bs);
+
return 0;
}
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 659617a3f8..30fe2b4355 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -296,13 +296,18 @@ void bdrv_dirty_bitmap_truncate(BlockDriverState *bs)
}
}
-static void bdrv_do_release_matching_dirty_bitmap(BlockDriverState *bs,
- BdrvDirtyBitmap *bitmap,
- bool only_named)
+static bool bdrv_dirty_bitmap_has_name(BdrvDirtyBitmap *bitmap)
+{
+ return !!bdrv_dirty_bitmap_name(bitmap);
+}
+
+static void bdrv_do_release_matching_dirty_bitmap(
+ BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
+ bool (*cond)(BdrvDirtyBitmap *bitmap))
{
BdrvDirtyBitmap *bm, *next;
QLIST_FOREACH_SAFE(bm, &bs->dirty_bitmaps, list, next) {
- if ((!bitmap || bm == bitmap) && (!only_named || bm->name)) {
+ if ((!bitmap || bm == bitmap) && (!cond || cond(bm))) {
assert(!bm->active_iterators);
assert(!bdrv_dirty_bitmap_frozen(bm));
assert(!bm->meta);
@@ -323,7 +328,7 @@ static void bdrv_do_release_matching_dirty_bitmap(BlockDriverState *bs,
void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
{
- bdrv_do_release_matching_dirty_bitmap(bs, bitmap, false);
+ bdrv_do_release_matching_dirty_bitmap(bs, bitmap, NULL);
}
/**
@@ -333,7 +338,19 @@ void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
*/
void bdrv_release_named_dirty_bitmaps(BlockDriverState *bs)
{
- bdrv_do_release_matching_dirty_bitmap(bs, NULL, true);
+ bdrv_do_release_matching_dirty_bitmap(bs, NULL, bdrv_dirty_bitmap_has_name);
+}
+
+/**
+ * Release all persistent dirty bitmaps attached to a BDS (for use in
+ * bdrv_inactivate_recurse()).
+ * There must not be any frozen bitmaps attached.
+ * This function does not remove persistent bitmaps from the storage.
+ */
+void bdrv_release_persistent_dirty_bitmaps(BlockDriverState *bs)
+{
+ bdrv_do_release_matching_dirty_bitmap(bs, NULL,
+ bdrv_dirty_bitmap_get_persistance);
}
/**
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 2ee63c46e9..23f669bd2e 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -25,6 +25,7 @@ BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs,
void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap);
void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap);
void bdrv_release_named_dirty_bitmaps(BlockDriverState *bs);
+void bdrv_release_persistent_dirty_bitmaps(BlockDriverState *bs);
void bdrv_remove_persistent_dirty_bitmap(BlockDriverState *bs,
const char *name,
Error **errp);
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
* [Qemu-devel] ping Re: [PATCH v18 00/25] qcow2: persistent dirty bitmaps
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
` (24 preceding siblings ...)
2017-05-03 12:25 ` [Qemu-devel] [PATCH 25/25] block: release persistent bitmaps on inactivate Vladimir Sementsov-Ogievskiy
@ 2017-05-15 9:51 ` Vladimir Sementsov-Ogievskiy
2017-05-27 9:05 ` [Qemu-devel] ping2 " Vladimir Sementsov-Ogievskiy
25 siblings, 1 reply; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-15 9:51 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
ping...
What about this?
03.05.2017 15:25, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
>
> There is a new update of qcow2-bitmap series - v18.
>
> web: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=qcow2-bitmap-v18
> git: https://src.openvz.org/scm/~vsementsov/qemu.git (tag qcow2-bitmap-v18)
>
> v18:
>
> rebased on master (sorry for v17)
>
> 08: contextual: qcow2_do_open is changed instead of qcow2_open
> rename s/need_update_header/update_header/ in qcow2_do_open, to not do it in 10
> save r-b's by Max and John
> 09: new patch
> 10: load_bitmap_data: do not clear bitmap parameter - it should be already cleared
> (it actually created before single load_bitmap_data() call)
> if some bitmaps are loaded, but we can't write the image (it is readonly
> or inactive), so we can't mark bitmaps "in use" in the image, mark
> corresponding BdrvDirtyBitmap read-only.
> change error_setg to error_setg_errno for "Can't update bitmap directory"
> no needs to rename s/need_update_header/update_header/ here, as it done in 08
> 13: function bdrv_has_persistent_bitmaps becomes bdrv_has_changed_persistent_bitmaps,
> to handle readonly field.
> 14: declaration moved to the bottom of .h, save r-b's
> 15: firstly check bdrv_has_changed_persistent_bitmaps and only then fail on !can_write, and then QSIMPLEQ_INIT(&drop_tables)
> skip readonly bitmaps in saving loop
> 18: remove '#optional', 2.9 -> 2.10, save r-b's
> 19: remove '#optional', 2.9 -> 2.10, save r-b's
> 20: 2.9 -> 2.10, save r-b's
> 21: add check of read-only image open, drop r-b's
> 24: add comment to qapi/block-core.json, that block-dirty-bitmap-add removes bitmap
> from storage. r-b's by Max and John saved
>
>
> v17:
> 08: add r-b's by Max and John
> 09: clear unknown autoclear features from BDRVQcow2State before calling
> qcow2_load_autoloading_dirty_bitmaps(), and also do not extra update
> header if it is updated by qcow2_load_autoloading_dirty_bitmaps().
> 11: new patch, splitted out from 16
> 12: rewrite commit message
> 14: changing bdrv_close moved to separate new patch 11
> s/1/1ULL/ ; s/%u/%llu/ for two errors
> 16: s/No/Not/
> add Max's r-b
> 24: new patch
>
>
> v16:
>
> mostly by Kevin's comments:
> 07: just moved here, as preparation for merging refcounts to 08
> 08: move "qcow2-bitmap: refcounts" staff to this patch, to not break qcow2-img check
> drop r-b's
> move necessary supporting static functions to this patch too (to not break compilation)
> fprintf -> error_report
> other small changes with error messages and comments
> code style
> for qcow2-bitmap.c:
> 'include "exec/log.h"' was dropped
> s/1/(1U << 0) for BME_FLAG_IN_USE
> add BME_TABLE_ENTRY_FLAG_ALL_ONES to replace magic 1
> don't check 'cluster_size <= 0' in check_table_entry
> old "[PATCH v15 08/25] block: introduce auto-loading bitmaps" was dropped
> 09: was "[PATCH v15 09/25] qcow2: add .bdrv_load_autoloading_dirty_bitmaps"
> drop r-b's
> some staff was moved to 08
> update_header_sync - error on bdrv_flush fail
> rename disk_sectors_in_bitmap_cluster to sectors_covered_by_bitmap_cluster
> and adjust comment.
> so, variable for storing this function result: s/dsc/sbc/
> s/1/BME_TABLE_ENTRY_FLAG_ALL_ONES/
> also, as Qcow2BitmapTable already introduced in 08,
> s/table_offset/table.offset/ and s/table_size/table.size, etc..
> update_ext_header_and_dir_in_place: add comments, add additional
> update_header_sync, to reduce indeterminacy in case of error.
> call qcow2_load_autoloading_dirty_bitmaps directly from qcow2_open
> no .bdrv_load_autoloading_dirty_bitmaps
> 11: drop bdrv_store_persistent_dirty_bitmaps at all.
> drop r-b's
> 13: rename patch, rewrite commit msg
> drop r-b's
> move bdrv_release_named_dirty_bitmaps in bdrv_close() after drv->bdrv_close()
> Qcow2BitmapTable is already introduced, so no
> s/table_offset/table.offset/ and s/table_size/table.size, etc.. here
> old 25 with check_constraints_on_bitmap() improvements merged here (Eric)
> like in 09, s/dsc/sbc/ and s/disk_sectors_in_bitmap_cluster/sectors_covered_by_bitmap_cluster/
> no .bdrv_store_persistent_dirty_bitmaps
> call qcow2_store_persistent_dirty_bitmaps directly from qcow2_inactivate
> 15: corresponding part of 25 merged here. Add John's r-b, as 25 was also reviewed by John.
> 16: add John's r-b
>
>
> v15:
> 13,14: add John's r-b
> 15: qcow2_can_store_new_dirty_bitmap:
> - check max dirty bitmaps and bitmap directory overhead
> - switch to error_prepend
> rm Max's r-b
> not add John's r-b
> 17-24: add John's r-b
> 25: changed because 15 changed,
> not add John's r-b
>
>
> v14:
>
> 07: use '|=' to update need_update_header
> add John's r-b
> add Max's r-b
> 09: remove unused bitmap_table_to_cpu()
> left Max's r-b, hope it's ok
> add John's r-b
> 10: remove extra new line
> add John's r-b
> 11: add John's r-b
> 12: add John's r-b
> 13: small fixes by John's review:
> - remove weird g_free of NULL pointer from
> if (tb == NULL) {
> g_free(tb);
> return -EINVAL;
> }
> - remove extra comment "/* errp is already set */"
> - s/"Too large bitmap directory"/"Bitmap directory is too large"/
> left Max's r-b, hope you don't mind
> 22: add Max's r-b
> 23: add Max's r-b
> 24: add Max's r-b
> 25: new patch to improve error message on check_constraints_on_bitmap fail
>
>
> v13: Just a fix for style checker.
> 13: line over 80
> 14: line over 80
> 22: s/if () \n{/if () {/
>
> v12:
> 07: do not update header in qcow2_read_extensions, instead do it in the
> end of qcow2_open, where it is updated also to clear unknown
> autoclear features.
> Wrong bdrv_is_root_node used instead of bdrv_is_read_only is fixed
> automatically.
> 08: add Max's r-b
> 09: contextual: s/raw_bsd/raw-format/
> qcow2-bitmap.c: copyright s/2016/2017/
> fix compilation: move update_header_sync() to this patch
> add Max's r-b
> 13: update_header_sync() is moved to 09
> add Max's r-b
> 15: add Max's r-b
> 16: As qmp-commands.txt is removed from master, remove it here too.
> Sentence "For now only Qcow2 disks support persistent bitmaps.
> Default is false." moved to qapi/block-core.json.
> Hope that is OK, Max's r-b is not dropped.
> 17: qmp-commands.txt deleted, r-b is not dropped too.
> 18: s/is failed/has failed/, add Max's r-b
> 19: copyright: s/2016/2017/
> 21: drop "res->corruptions > 0 => ret = -EINVAL", add Max's r-b
> 22: actually, patch is replaced with a new one, however some parts are the same.
> instead of changing bdrv_release_dirty_bitmap behaviour, create new
> bdrv_remove_persistent_dirty_bitmap
> 23: improve comment, about not-exists is not an error
> 24: new patch
>
> Old 24 (qcow2-bitmap: cache bitmap list in BDRVQcow2State) will be sent
> separately, to be applied after these series.
>
> Also: about bdrv_dirty_bitmap_set_persistance: I've decided not change its behaviour
> for now and just call bdrv_remove_persistent_dirty_bitmap from
> qmp_block_dirty_bitmap_remove. Reasons:
> 1. Do not change reviewed part.
> 2. I'm not sure, that this complication of BdrvDirtyBitmap.persistent semantics
> is good (current semantics are just: .persistent means that bitmap should be
> saved on disk close). .persistent actually is not very related to "is there
> stored version of the bitmap in the image".
> 3. It may be done later.
> 4. It is not actually needed for now, as the only usage is dropping bitmap in
> bdrv_remove_persistent_dirty_bitmap. May be in future it will be needed,
> when we will have qmp interfaces for finer control of persistent dirty bitmaps.
>
>
> v11:
>
> Fix automatic build test fail.
>
> 18: wording - "ASCII representation of SHA256 ..."
> 24: fix redeclaration error. (This is still RFC, may be not needed patch,
> see description in v10 below).
>
> v10:
>
> 07: rm John's r-b
> not add Max's r-b
> fix subject s/dirty bitmaps/bitmaps
> improve WARNING message
> remove header ext if autoclear bit is unset, through qcow2_updata_header() - r-b blocking change
> 08: move bdrv_load_autoloading_dirty_bitmaps under asserts block (Max)
> fix typo in commit message
> move bdrv_load_autoloading_dirty_bitmaps after asserts
> John's r-b
> 09: rewrite check_dir_entry
> remove extra feature cluster_size=0 from check_table_entry
> which clears autoclear bit before trying to update bitmap directory
> bitmap_list_store - do not free old clusters if in_place=true
> changes in comments, fix indents
> add functions
> bitmap_list_count
> update_ext_header_and_dir_in_place (unset autoclear bit before trying to
> modify bitmap directory)
> (for usage in qcow2_load_autoloading_dirty_bitmaps)
> 11: add node name to error report
> Max's r-b
> 13: add type Qcow2BitmapTable
> move from bm.table_* to bm.table.*
> rewrite check_constraints_on_bitmap
> flush(bs->file) instead of flush(bs) in update_ext_header_and_dir
> assert(DIV_ROUND_UP(bm_size, dsc) == tb_size);
> and assert(write_size <= s->cluster_size); in store_bitmap_data
> fix: s/tb_size/tb_size * sizeof(tb[0]) in store_bitmap
> in qcow2_store_persistent_dirty_bitmaps:
> fail if !can_write
> fix counting of new_nb_bitmaps and new_dir_size
> free old clusters _after_ successful directory update (aggregate old bitmap
> tables into drop_tables)
> 2 14: rename to can_store_new_dirty_bitmap
> Max's r-b
> 15: rename to can_store_new_dirty_bitmap
> rm extra !!
> 16: Max's r-b
> rename to can_store_new_dirty_bitmap
> indenting
> Since s/2.8/2.9
> 17: Max's r-b
> Since s/2.8/2.9
> 18: hashing can fail in comment for qapi
> Since s/2.8/2.9
> remove dead comment
> 19: indentation
> Max's r-b
> 20: Max's r-b
> 21: fix error handling
> return 0 if nb_bitmaps == 0
> new patches:
> 22-23: remove bitmap from file on bdrv_release_dirty_bitmap
> 24: experimental, RFC: cache bitmap list to bs, to not reload it from file.
> I'm not sure that is needed now, but if you want I can merge it to earlier
> patches.
>
> v9:
>
> rebase on master!
>
> 01-04,06,07: John's r-b
> 03: rewording in comment ("The concurrent resetting ..."), John's r-b
> 07: reword error messages
> commit message: dirty bitmap -> bitmap
> 09: fix uninitialized ret in bitmap_list_store() (fix compilation warning)
> 18: fix compilation of tests/test-hbitmap
>
> also, change constants names for qcow2: DIRTY_BITMAP -> BITMAP (as it is not
> dirty bitmaps extension, but just bitmaps extension), QCOW -> QCOW2 (bitmaps
> are only for qcow2)
>
> v8:
>
> Patches 01-06 are mostly unchanged, except for spelling and wording.
> 02 - add Eric's r-b
> 03,04 - add Max's r-b
>
> 07-09 is refactored "bitmap-read" part of the feature. Main changes:
> - introduce bitmap list - normal list, which represents bitmap directory.
> Function bitmap_list_load loads bitmap directory, checks everything and
> creates normal list.
> - introduce .bdrv_load_autoloading_dirty_bitmaps : instead of loading bitmaps
> in qcow2_open, lets load them only in bdrv_open, to avoid reloading in
> bdrv_snapshot_goto
> - do not delete bitmaps after read, but mark them IN_USE
>
> 10 has contextual changes and rewording of comment. I've added Max's r-b, hope it's ok.
>
> 11: add error_report("Persistent bitmaps are lost") in case of failed bitmap store
>
> 12: add Max's r-b
>
> 13 is refactored "bitmap-store" part of the feature. see 07-09 description
> - for now I just free old clusters and allocate new. This will be improved
> with a separate patch.
>
> patch about "delete bitmaps on truncate" is removed. This case will be handled later.
> IN_USE, autoclear, check-constraints things are merged into other patches.
>
> 14-15 are mew patches, to early check possibility of creating persistent bitmap with
> specified name and granularity in specified BDS
>
> 16-17: spelling, rewording, indenting, tiny code simplifying, check can_store (by 14-15),
> s/if (autoload && !persistent)/if (has_autoload && !persistent)/,
> rebase (deleted qmp-commands.hx)
>
> 18: alternative to md5 in query-block:
> - separated qmp command -x-debug-block-dirty-bitmap-sha256
> - as adviced by Daniel P. Berrange in my parallel thread:
> - sha256 instead of md5
> - use qcrypto_hash_* instead of GChecksum
> - fix bug =) (size was wrong in hbitmap_md5)
>
> 19: s/3999/3fff/, use x-debug-block-dirty-bitmap-sha256
>
> 20: new patch to rename and publish inc_refcounts
>
> 21: some fixes and refactoring mostyly by Max's comments.
>
> Max, Eric, great tanks for your review!
> I hope, I've covered most of your comments by this update.
>
> TODO:
> - handle reopening image RO->RW and incoming migration, set IN_USE for existing loaded bitmaps
> in these cases.
> - reuse old, already allocated data clusters for bitmaps storing
> - truncate bitmaps in the image on truncate
>
>
> v7:
>
> https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fqcow2-bitmap-v7
> based on block-next (https://github.com/XanClic/qemu/commits/block-next)
>
> - a lot of refactoring and reordering of patches.
> - dead code removed (bdrv_dirty_bitmap_load, etc.)
> - do not maintain extra data for now
> - do not store dirty bitmap directory in memory
> (as we use it seldom, we can just reread if needed)
>
> By Kevin's review:
> 01 - commit message changed: fix->improvement (as it was not a bug)
> 03 - r-b
> 04 - r-b
> 05 - add 21 patch to fix spec, also, removed all (I hope) mentions of
> "Bitmap Header", switch to one unified name for it - "Bitmap
> Directory Entry", to avoid misunderstanding with Qcow2 header.
> (also, add patch 22, to fix it in spec)
> v6.06 - improve for_each_dir_entry loop, reorder patches, other small fixes
> v6.07 ~> v7.09 - dead code removed, I've moved to one function
> .bdrv_store_persistent_bitmaps and have wrapper and callback in one
> patch (with also some other staff. If it not ok, I can split them)
> v6.08 - about keeping bitmap directory instead of bitmap list: no I don't keep
> it at all.
> v6.16 - old bdrv_ bitmap-storing related functions are removed. The new one is
> bdrv_store_persistent_bitmaps.
>
>
> v6:
> https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fqcow2-bitmap-v6
> based on block-next (https://github.com/XanClic/qemu/commits/block-next)
>
> There are a lot of changes, reorderings and additions in comparement with v5.
> One principal thing: now bitmaps are removed from image after loading instead
> of marking them in_use. It is simpler and we do not need to store superfluous data.
> Also, we are no more interested in command line interface to dirty bitmaps.
> So it is dropped. If someone needs it I can add it later.
>
> Vladimir Sementsov-Ogievskiy (25):
> specs/qcow2: fix bitmap granularity qemu-specific note
> specs/qcow2: do not use wording 'bitmap header'
> hbitmap: improve dirty iter
> tests: add hbitmap iter test
> block: fix bdrv_dirty_bitmap_granularity signature
> block/dirty-bitmap: add deserialize_ones func
> qcow2-refcount: rename inc_refcounts() and make it public
> qcow2: add bitmaps extension
> block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
> qcow2: autoloading dirty bitmaps
> block/dirty-bitmap: add autoload field to BdrvDirtyBitmap
> block: bdrv_close: release bitmaps after drv->bdrv_close
> block: introduce persistent dirty bitmaps
> block/dirty-bitmap: add bdrv_dirty_bitmap_next()
> qcow2: add persistent dirty bitmaps support
> block: add bdrv_can_store_new_dirty_bitmap
> qcow2: add .bdrv_can_store_new_dirty_bitmap
> qmp: add persistent flag to block-dirty-bitmap-add
> qmp: add autoload parameter to block-dirty-bitmap-add
> qmp: add x-debug-block-dirty-bitmap-sha256
> iotests: test qcow2 persistent dirty bitmap
> block/dirty-bitmap: add bdrv_remove_persistent_dirty_bitmap
> qcow2: add .bdrv_remove_persistent_dirty_bitmap
> qmp: block-dirty-bitmap-remove: remove persistent
> block: release persistent bitmaps on inactivate
>
> block.c | 32 +-
> block/Makefile.objs | 2 +-
> block/dirty-bitmap.c | 125 +++-
> block/qcow2-bitmap.c | 1396 ++++++++++++++++++++++++++++++++++++++++++
> block/qcow2-refcount.c | 59 +-
> block/qcow2.c | 149 ++++-
> block/qcow2.h | 41 ++
> blockdev.c | 71 ++-
> docs/specs/qcow2.txt | 8 +-
> include/block/block.h | 3 +
> include/block/block_int.h | 8 +
> include/block/dirty-bitmap.h | 25 +-
> include/qemu/hbitmap.h | 49 +-
> qapi/block-core.json | 42 +-
> tests/Makefile.include | 2 +-
> tests/qemu-iotests/165 | 105 ++++
> tests/qemu-iotests/165.out | 5 +
> tests/qemu-iotests/group | 1 +
> tests/test-hbitmap.c | 19 +
> util/hbitmap.c | 51 +-
> 20 files changed, 2119 insertions(+), 74 deletions(-)
> create mode 100644 block/qcow2-bitmap.c
> create mode 100755 tests/qemu-iotests/165
> create mode 100644 tests/qemu-iotests/165.out
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-03 12:25 ` [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap Vladimir Sementsov-Ogievskiy
@ 2017-05-19 23:02 ` John Snow
2017-05-25 9:34 ` Vladimir Sementsov-Ogievskiy
2017-05-31 22:58 ` John Snow
2017-05-29 15:49 ` Max Reitz
2017-05-29 18:35 ` Max Reitz
2 siblings, 2 replies; 56+ messages in thread
From: John Snow @ 2017-05-19 23:02 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, famz, armbru, mreitz, stefanha, pbonzini, den
On 05/03/2017 08:25 AM, Vladimir Sementsov-Ogievskiy wrote:
> It will be needed in following commits for persistent bitmaps.
> If bitmap is loaded from read-only storage (and we can't mark it
> "in use" in this storage) corresponding BdrvDirtyBitmap should be
> read-only.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> block/dirty-bitmap.c | 16 ++++++++++++++++
> include/block/dirty-bitmap.h | 3 +++
> 2 files changed, 19 insertions(+)
>
> diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
> index 90af37287f..ab6a95cf41 100644
> --- a/block/dirty-bitmap.c
> +++ b/block/dirty-bitmap.c
> @@ -44,6 +44,8 @@ struct BdrvDirtyBitmap {
> int64_t size; /* Size of the bitmap (Number of sectors) */
> bool disabled; /* Bitmap is read-only */
> int active_iterators; /* How many iterators are active */
> + bool readonly; /* Bitmap is read-only and may be changed only
> + by deserialize* functions */
> QLIST_ENTRY(BdrvDirtyBitmap) list;
> };
>
> @@ -436,6 +438,7 @@ void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
> int64_t cur_sector, int64_t nr_sectors)
> {
> assert(bdrv_dirty_bitmap_enabled(bitmap));
> + assert(!bdrv_dirty_bitmap_readonly(bitmap));
Doesn't this change the nature of the assertion?
We're going from
return !(bitmap->disabled || bitmap->successor);
to
!bitmap->readonly
That makes me a little nervous to ACK this patch, because the
correctness depends on how readonly is updated and manipulated in the
future, which makes it more prone to error.
> hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
> }
>
> @@ -443,12 +446,14 @@ void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
> int64_t cur_sector, int64_t nr_sectors)
> {
> assert(bdrv_dirty_bitmap_enabled(bitmap));
> + assert(!bdrv_dirty_bitmap_readonly(bitmap));
> hbitmap_reset(bitmap->bitmap, cur_sector, nr_sectors);
> }
>
> void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap **out)
> {
> assert(bdrv_dirty_bitmap_enabled(bitmap));
> + assert(!bdrv_dirty_bitmap_readonly(bitmap));
> if (!out) {
> hbitmap_reset_all(bitmap->bitmap);
> } else {
> @@ -519,6 +524,7 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
> if (!bdrv_dirty_bitmap_enabled(bitmap)) {
> continue;
> }
> + assert(!bdrv_dirty_bitmap_readonly(bitmap));
> hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
> }
> }
> @@ -540,3 +546,13 @@ int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap)
> {
> return hbitmap_count(bitmap->meta);
> }
> +
> +bool bdrv_dirty_bitmap_readonly(const BdrvDirtyBitmap *bitmap)
> +{
> + return bitmap->readonly;
> +}
> +
> +void bdrv_dirty_bitmap_set_readonly(BdrvDirtyBitmap *bitmap)
> +{
> + bitmap->readonly = true;
> +}
> diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
> index 1e17729ac2..0aab5841f5 100644
> --- a/include/block/dirty-bitmap.h
> +++ b/include/block/dirty-bitmap.h
> @@ -75,4 +75,7 @@ void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap,
> bool finish);
> void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
>
> +bool bdrv_dirty_bitmap_readonly(const BdrvDirtyBitmap *bitmap);
> +void bdrv_dirty_bitmap_set_readonly(BdrvDirtyBitmap *bitmap);
> +
> #endif
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-19 23:02 ` John Snow
@ 2017-05-25 9:34 ` Vladimir Sementsov-Ogievskiy
2017-05-31 22:58 ` John Snow
1 sibling, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-25 9:34 UTC (permalink / raw)
To: John Snow, qemu-block, qemu-devel
Cc: kwolf, famz, armbru, mreitz, stefanha, pbonzini, den
20.05.2017 02:02, John Snow wrote:
>
> On 05/03/2017 08:25 AM, Vladimir Sementsov-Ogievskiy wrote:
>> It will be needed in following commits for persistent bitmaps.
>> If bitmap is loaded from read-only storage (and we can't mark it
>> "in use" in this storage) corresponding BdrvDirtyBitmap should be
>> read-only.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>> block/dirty-bitmap.c | 16 ++++++++++++++++
>> include/block/dirty-bitmap.h | 3 +++
>> 2 files changed, 19 insertions(+)
>>
>> diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
>> index 90af37287f..ab6a95cf41 100644
>> --- a/block/dirty-bitmap.c
>> +++ b/block/dirty-bitmap.c
>> @@ -44,6 +44,8 @@ struct BdrvDirtyBitmap {
>> int64_t size; /* Size of the bitmap (Number of sectors) */
>> bool disabled; /* Bitmap is read-only */
>> int active_iterators; /* How many iterators are active */
>> + bool readonly; /* Bitmap is read-only and may be changed only
>> + by deserialize* functions */
>> QLIST_ENTRY(BdrvDirtyBitmap) list;
>> };
>>
>> @@ -436,6 +438,7 @@ void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
>> int64_t cur_sector, int64_t nr_sectors)
>> {
>> assert(bdrv_dirty_bitmap_enabled(bitmap));
>> + assert(!bdrv_dirty_bitmap_readonly(bitmap));
> Doesn't this change the nature of the assertion?
>
> We're going from
> return !(bitmap->disabled || bitmap->successor);
> to
> !bitmap->readonly
>
> That makes me a little nervous to ACK this patch, because the
> correctness depends on how readonly is updated and manipulated in the
> future, which makes it more prone to error.
Sorry for late answer, I was ill.
Hmm, old assert is still here. I just add another one. Readonly means:
bitmap must not be changed by _get _set and _clear at all - from loading
up to releasing. It is needed to maintain the case: "bitmap was loaded
from read-only storage, so that it was not marked as 'in_use' in that
storge". 'disabled' field is not appropriate because it is managed by user.
>
>> hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
>> }
>>
>> @@ -443,12 +446,14 @@ void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
>> int64_t cur_sector, int64_t nr_sectors)
>> {
>> assert(bdrv_dirty_bitmap_enabled(bitmap));
>> + assert(!bdrv_dirty_bitmap_readonly(bitmap));
>> hbitmap_reset(bitmap->bitmap, cur_sector, nr_sectors);
>> }
>>
>> void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap **out)
>> {
>> assert(bdrv_dirty_bitmap_enabled(bitmap));
>> + assert(!bdrv_dirty_bitmap_readonly(bitmap));
>> if (!out) {
>> hbitmap_reset_all(bitmap->bitmap);
>> } else {
>> @@ -519,6 +524,7 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
>> if (!bdrv_dirty_bitmap_enabled(bitmap)) {
>> continue;
>> }
>> + assert(!bdrv_dirty_bitmap_readonly(bitmap));
>> hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
>> }
>> }
>> @@ -540,3 +546,13 @@ int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap)
>> {
>> return hbitmap_count(bitmap->meta);
>> }
>> +
>> +bool bdrv_dirty_bitmap_readonly(const BdrvDirtyBitmap *bitmap)
>> +{
>> + return bitmap->readonly;
>> +}
>> +
>> +void bdrv_dirty_bitmap_set_readonly(BdrvDirtyBitmap *bitmap)
>> +{
>> + bitmap->readonly = true;
>> +}
>> diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
>> index 1e17729ac2..0aab5841f5 100644
>> --- a/include/block/dirty-bitmap.h
>> +++ b/include/block/dirty-bitmap.h
>> @@ -75,4 +75,7 @@ void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap,
>> bool finish);
>> void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
>>
>> +bool bdrv_dirty_bitmap_readonly(const BdrvDirtyBitmap *bitmap);
>> +void bdrv_dirty_bitmap_set_readonly(BdrvDirtyBitmap *bitmap);
>> +
>> #endif
>>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 56+ messages in thread
* [Qemu-devel] ping2 Re: [PATCH v18 00/25] qcow2: persistent dirty bitmaps
2017-05-15 9:51 ` [Qemu-devel] ping Re: [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
@ 2017-05-27 9:05 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-27 9:05 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
ping...
What about this?
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 08/25] qcow2: add bitmaps extension
2017-05-03 12:25 ` [Qemu-devel] [PATCH 08/25] qcow2: add bitmaps extension Vladimir Sementsov-Ogievskiy
@ 2017-05-29 15:34 ` Max Reitz
0 siblings, 0 replies; 56+ messages in thread
From: Max Reitz @ 2017-05-29 15:34 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
[-- Attachment #1: Type: text/plain, Size: 1746 bytes --]
On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
> Add bitmap extension as specified in docs/specs/qcow2.txt.
> For now, just mirror extension header into Qcow2 state and check
> constraints. Also, calculate refcounts for qcow2 bitmaps, to not break
> qemu-img check.
>
> For now, disable image resize if it has bitmaps. It will be fixed later.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> Reviewed-by: Max Reitz <mreitz@redhat.com>
> Reviewed-by: John Snow <jsnow@redhat.com>
> ---
> block/Makefile.objs | 2 +-
> block/qcow2-bitmap.c | 439 +++++++++++++++++++++++++++++++++++++++++++++++++
> block/qcow2-refcount.c | 6 +
> block/qcow2.c | 124 +++++++++++++-
> block/qcow2.h | 27 +++
> 5 files changed, 592 insertions(+), 6 deletions(-)
> create mode 100644 block/qcow2-bitmap.c
[...]
> diff --git a/block/qcow2.c b/block/qcow2.c
> index 6a92d2ef3f..6028e9a149 100644
> --- a/block/qcow2.c
> +++ b/block/qcow2.c
[...]
> @@ -2542,6 +2649,13 @@ static int qcow2_truncate(BlockDriverState *bs, int64_t offset)
> return -ENOTSUP;
> }
>
> + /* cannot proceed if image has bitmaps */
> + if (s->nb_bitmaps) {
> + /* TODO: resize bitmaps in the image */
> + error_report("Can't resize an image which has bitmaps");
Minor note: This needs to be error_setg(errp, ...); now...
> + return -ENOTSUP;
> + }
> +
> /* shrinking is currently not supported */
> if (offset < bs->total_sectors * 512) {
> error_report("qcow2 doesn't support shrinking images yet");
...as does this (which blocks the patch from being applicable).
(But please keep my R-b.)
Max
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 498 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-03 12:25 ` [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap Vladimir Sementsov-Ogievskiy
2017-05-19 23:02 ` John Snow
@ 2017-05-29 15:49 ` Max Reitz
2017-05-29 18:35 ` Max Reitz
2 siblings, 0 replies; 56+ messages in thread
From: Max Reitz @ 2017-05-29 15:49 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
[-- Attachment #1: Type: text/plain, Size: 543 bytes --]
On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
> It will be needed in following commits for persistent bitmaps.
> If bitmap is loaded from read-only storage (and we can't mark it
> "in use" in this storage) corresponding BdrvDirtyBitmap should be
> read-only.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> block/dirty-bitmap.c | 16 ++++++++++++++++
> include/block/dirty-bitmap.h | 3 +++
> 2 files changed, 19 insertions(+)
Reviewed-by: Max Reitz <mreitz@redhat.com>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 498 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 10/25] qcow2: autoloading dirty bitmaps
2017-05-03 12:25 ` [Qemu-devel] [PATCH 10/25] qcow2: autoloading dirty bitmaps Vladimir Sementsov-Ogievskiy
@ 2017-05-29 16:17 ` Max Reitz
0 siblings, 0 replies; 56+ messages in thread
From: Max Reitz @ 2017-05-29 16:17 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
[-- Attachment #1: Type: text/plain, Size: 1542 bytes --]
On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
> Auto loading bitmaps are bitmaps in Qcow2, with the AUTO flag set. They
> are loaded when the image is opened and become BdrvDirtyBitmaps for the
> corresponding drive.
>
> Extra data in bitmaps is not supported for now.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> block/qcow2-bitmap.c | 387 +++++++++++++++++++++++++++++++++++++++++++++++++++
> block/qcow2.c | 17 ++-
> block/qcow2.h | 2 +
> 3 files changed, 404 insertions(+), 2 deletions(-)
>
> diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
> index b8e472b3e8..a2522d7ccf 100644
> --- a/block/qcow2-bitmap.c
> +++ b/block/qcow2-bitmap.c
[...]
> @@ -437,3 +602,225 @@ out:
[...]
> +/* for g_slist_foreach for GSList of BdrvDirtyBitmap* elements */
> +static void set_readonly_helper(gpointer bitmap, gpointer unused)
> +{
> + bdrv_dirty_bitmap_set_readonly(bitmap);
> +}
> +
> +/* qcow2_load_autoloading_dirty_bitmaps()
> + * Return value is a hint for caller: true means that Qcow2 header was updated.
"...that the Qcow2 header..."
> + * (false doesn't mean that header should be updated by caller, it just means
"...that the header...", "...by the caller, ..."
> + * that updating was not needed or image can not be written).
"...or the image cannot be written [to]"
> + * On fail the function returns false.
"On failure..."
With these fixed:
Reviewed-by: Max Reitz <mreitz@redhat.com>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 498 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 12/25] block: bdrv_close: release bitmaps after drv->bdrv_close
2017-05-03 12:25 ` [Qemu-devel] [PATCH 12/25] block: bdrv_close: release bitmaps after drv->bdrv_close Vladimir Sementsov-Ogievskiy
@ 2017-05-29 16:34 ` Max Reitz
0 siblings, 0 replies; 56+ messages in thread
From: Max Reitz @ 2017-05-29 16:34 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
[-- Attachment #1: Type: text/plain, Size: 425 bytes --]
On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
> Release bitmaps after 'if (bs->drv) { ... }' block. This will allow
> format driver to save persistent bitmaps, which will appear in following
> commits.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> block.c | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
Reviewed-by: Max Reitz <mreitz@redhat.com>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 498 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 13/25] block: introduce persistent dirty bitmaps
2017-05-03 12:25 ` [Qemu-devel] [PATCH 13/25] block: introduce persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
@ 2017-05-29 16:49 ` Max Reitz
0 siblings, 0 replies; 56+ messages in thread
From: Max Reitz @ 2017-05-29 16:49 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
[-- Attachment #1: Type: text/plain, Size: 551 bytes --]
On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
> New field BdrvDirtyBitmap.persistent means, that bitmap should be saved
> by format driver in .bdrv_close and .bdrv_inactivate. No format driver
> supports it for now.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> block/dirty-bitmap.c | 26 ++++++++++++++++++++++++++
> block/qcow2-bitmap.c | 1 +
> include/block/dirty-bitmap.h | 5 +++++
> 3 files changed, 32 insertions(+)
Reviewed-by: Max Reitz <mreitz@redhat.com>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 498 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 15/25] qcow2: add persistent dirty bitmaps support
2017-05-03 12:25 ` [Qemu-devel] [PATCH 15/25] qcow2: add persistent dirty bitmaps support Vladimir Sementsov-Ogievskiy
@ 2017-05-29 17:07 ` Max Reitz
0 siblings, 0 replies; 56+ messages in thread
From: Max Reitz @ 2017-05-29 17:07 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
[-- Attachment #1: Type: text/plain, Size: 429 bytes --]
On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
> Store persistent dirty bitmaps in qcow2 image.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> block/qcow2-bitmap.c | 475 +++++++++++++++++++++++++++++++++++++++++++++++++++
> block/qcow2.c | 9 +
> block/qcow2.h | 1 +
> 3 files changed, 485 insertions(+)
Reviewed-by: Max Reitz <mreitz@redhat.com>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 498 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 21/25] iotests: test qcow2 persistent dirty bitmap
2017-05-03 12:25 ` [Qemu-devel] [PATCH 21/25] iotests: test qcow2 persistent dirty bitmap Vladimir Sementsov-Ogievskiy
@ 2017-05-29 17:31 ` Max Reitz
0 siblings, 0 replies; 56+ messages in thread
From: Max Reitz @ 2017-05-29 17:31 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
[-- Attachment #1: Type: text/plain, Size: 485 bytes --]
On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> tests/qemu-iotests/165 | 105 +++++++++++++++++++++++++++++++++++++++++++++
> tests/qemu-iotests/165.out | 5 +++
> tests/qemu-iotests/group | 1 +
> 3 files changed, 111 insertions(+)
> create mode 100755 tests/qemu-iotests/165
> create mode 100644 tests/qemu-iotests/165.out
Reviewed-by: Max Reitz <mreitz@redhat.com>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 498 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 24/25] qmp: block-dirty-bitmap-remove: remove persistent
2017-05-03 12:25 ` [Qemu-devel] [PATCH 24/25] qmp: block-dirty-bitmap-remove: remove persistent Vladimir Sementsov-Ogievskiy
@ 2017-05-29 17:38 ` Max Reitz
0 siblings, 0 replies; 56+ messages in thread
From: Max Reitz @ 2017-05-29 17:38 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
[-- Attachment #1: Type: text/plain, Size: 2037 bytes --]
On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
> Remove persistent bitmap from the storage on block-dirty-bitmap-remove.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> Reviewed-by: Max Reitz <mreitz@redhat.com>
> Reviewed-by: John Snow <jsnow@redhat.com>
> ---
> blockdev.c | 10 ++++++++++
> qapi/block-core.json | 3 ++-
> 2 files changed, 12 insertions(+), 1 deletion(-)
>
> diff --git a/blockdev.c b/blockdev.c
> index c49d414891..0ad1a658b9 100644
> --- a/blockdev.c
> +++ b/blockdev.c
> @@ -2802,6 +2802,7 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name,
> AioContext *aio_context;
> BlockDriverState *bs;
> BdrvDirtyBitmap *bitmap;
> + Error *local_err = NULL;
>
> bitmap = block_dirty_bitmap_lookup(node, name, &bs, &aio_context, errp);
> if (!bitmap || !bs) {
> @@ -2814,6 +2815,15 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name,
> name);
> goto out;
> }
> +
> + if (bdrv_dirty_bitmap_get_persistance(bitmap)) {
> + bdrv_remove_persistent_dirty_bitmap(bs, name, &local_err);
> + if (local_err != NULL) {
> + error_propagate(errp, local_err);
> + goto out;
> + }
> + }
> +
> bdrv_dirty_bitmap_make_anon(bitmap);
> bdrv_release_dirty_bitmap(bs, bitmap);
>
> diff --git a/qapi/block-core.json b/qapi/block-core.json
> index e9523b4b8c..933b1a5fe5 100644
> --- a/qapi/block-core.json
> +++ b/qapi/block-core.json
> @@ -1600,7 +1600,8 @@
> # @block-dirty-bitmap-remove:
> #
> # Stop write tracking and remove the dirty bitmap that was created
> -# with block-dirty-bitmap-add.
> +# with block-dirty-bitmap-add. If bitmap is persistent, remove it from it's
s/If bitmap/If the bitmap/, s/it's/its/
Max
> +# storage too.
> #
> # Returns: nothing on success
> # If @node is not a valid block device or node, DeviceNotFound
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 498 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 25/25] block: release persistent bitmaps on inactivate
2017-05-03 12:25 ` [Qemu-devel] [PATCH 25/25] block: release persistent bitmaps on inactivate Vladimir Sementsov-Ogievskiy
@ 2017-05-29 17:54 ` Max Reitz
2017-05-30 6:30 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 56+ messages in thread
From: Max Reitz @ 2017-05-29 17:54 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
[-- Attachment #1: Type: text/plain, Size: 1111 bytes --]
On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
> We should release them here to reload on invalidate cache.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> block.c | 4 ++++
> block/dirty-bitmap.c | 29 +++++++++++++++++++++++------
> include/block/dirty-bitmap.h | 1 +
> 3 files changed, 28 insertions(+), 6 deletions(-)
>
> diff --git a/block.c b/block.c
> index 795d36bb64..14896c65fa 100644
> --- a/block.c
> +++ b/block.c
> @@ -4001,6 +4001,10 @@ static int bdrv_inactivate_recurse(BlockDriverState *bs,
> if (setting_flag) {
> bs->open_flags |= BDRV_O_INACTIVE;
> }
> +
> + /* At this point persistent bitmaps should be stored by format driver */
s/by format driver/by the format driver/
> + bdrv_release_persistent_dirty_bitmaps(bs);
Also, as far as I can see, this doesn't store the bitmaps but just
releases them (without storing them). I'm not sure whether that is
right, but it definitely contradicts the comment above.
Max
> +
> return 0;
> }
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 498 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-03 12:25 ` [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap Vladimir Sementsov-Ogievskiy
2017-05-19 23:02 ` John Snow
2017-05-29 15:49 ` Max Reitz
@ 2017-05-29 18:35 ` Max Reitz
2017-05-30 6:50 ` Vladimir Sementsov-Ogievskiy
2 siblings, 1 reply; 56+ messages in thread
From: Max Reitz @ 2017-05-29 18:35 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
[-- Attachment #1: Type: text/plain, Size: 2727 bytes --]
On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
> It will be needed in following commits for persistent bitmaps.
> If bitmap is loaded from read-only storage (and we can't mark it
> "in use" in this storage) corresponding BdrvDirtyBitmap should be
> read-only.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> block/dirty-bitmap.c | 16 ++++++++++++++++
> include/block/dirty-bitmap.h | 3 +++
> 2 files changed, 19 insertions(+)
Revisiting this again after the whole series: So you never really make
sure that the read-only bitmaps are not written to (except for these
assertions). The idea is that you only set it for read-only BDS and
read-only BDS are never written to. But that assumption is not true,
generally, and can be broken e.g. using a commit job:
$ ./qemu-img create -f qcow2 backing.qcow2 64M
Formatting 'backing.qcow2', fmt=qcow2 size=67108864 encryption=off
cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ ./qemu-img create -f qcow2 -b backing.qcow2 top.qcow2
Formatting 'top.qcow2', fmt=qcow2 size=67108864
backing_file=backing.qcow2 encryption=off cluster_size=65536
lazy_refcounts=off refcount_bits=16
$ x86_64-softmmu/qemu-system-x86_64 -qmp stdio
{"QMP": {"version": {"qemu": {"micro": 50, "minor": 9, "major": 2},
"package": " (v2.9.0-632-g4a52d43-dirty)"}, "capabilities": []}}
{'execute': 'qmp_capabilities'}
{"return": {}}
{'execute': 'blockdev-add',
'arguments': {'node-name': 'backing-node', 'driver': 'qcow2',
'file': {'driver': 'file', 'filename': 'backing.qcow2'}}}
{"return": {}}
{'execute': 'block-dirty-bitmap-add',
'arguments': {'node': 'backing-node', 'name': 'foo',
'persistent': true, 'autoload': true}}
{"return": {}}
{'execute': 'blockdev-del', 'arguments': {'node-name': 'backing-node'}}
{"return": {}}
{'execute': 'blockdev-add',
'arguments': {'node-name': 'top-node', 'driver': 'qcow2',
'file': {'driver': 'file', 'filename': 'top.qcow2'}}}
{"return": {}}
{'execute': 'human-monitor-command',
'arguments': {'command-line': 'qemu-io top-node "write 0 64k"'}}
wrote 65536/65536 bytes at offset 0
64 KiB, 1 ops; 0.0079 sec (7.852 MiB/sec and 125.6281 ops/sec)
{"return": ""}
{'execute': 'block-commit',
'arguments': {'device': 'top-node', 'job-id': 'commit-job'}}
{"return": {}}
qemu-system-x86_64: block/dirty-bitmap.c:571: bdrv_set_dirty: Assertion
`!bdrv_dirty_bitmap_readonly(bitmap)' failed.
[1] 10872 abort (core dumped) x86_64-softmmu/qemu-system-x86_64 -qmp
stdio
So there needs to be something else than just assertions to make sure
that read-only bitmaps are never written to.
Max
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 498 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 25/25] block: release persistent bitmaps on inactivate
2017-05-29 17:54 ` Max Reitz
@ 2017-05-30 6:30 ` Vladimir Sementsov-Ogievskiy
2017-05-31 13:37 ` Max Reitz
0 siblings, 1 reply; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-30 6:30 UTC (permalink / raw)
To: Max Reitz, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
29.05.2017 20:54, Max Reitz wrote:
> On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
>> We should release them here to reload on invalidate cache.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>> block.c | 4 ++++
>> block/dirty-bitmap.c | 29 +++++++++++++++++++++++------
>> include/block/dirty-bitmap.h | 1 +
>> 3 files changed, 28 insertions(+), 6 deletions(-)
>>
>> diff --git a/block.c b/block.c
>> index 795d36bb64..14896c65fa 100644
>> --- a/block.c
>> +++ b/block.c
>> @@ -4001,6 +4001,10 @@ static int bdrv_inactivate_recurse(BlockDriverState *bs,
>> if (setting_flag) {
>> bs->open_flags |= BDRV_O_INACTIVE;
>> }
>> +
>> + /* At this point persistent bitmaps should be stored by format driver */
> s/by format driver/by the format driver/
>
>> + bdrv_release_persistent_dirty_bitmaps(bs);
> Also, as far as I can see, this doesn't store the bitmaps but just
> releases them (without storing them). I'm not sure whether that is
> right, but it definitely contradicts the comment above.
I mean they should be _already_ stored.. They actually are stored in
qcow2_inactivate.
>
> Max
>
>> +
>> return 0;
>> }
>>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-29 18:35 ` Max Reitz
@ 2017-05-30 6:50 ` Vladimir Sementsov-Ogievskiy
2017-05-30 7:31 ` Vladimir Sementsov-Ogievskiy
2017-05-31 13:43 ` Max Reitz
0 siblings, 2 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-30 6:50 UTC (permalink / raw)
To: Max Reitz, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
Thank you for this scenario. Hmm.
So, as I need guarantee that image and bitmap are unchanged,
bdrv_set_dirty should return error and fail the whole write. Ok?
29.05.2017 21:35, Max Reitz wrote:
> On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
>> It will be needed in following commits for persistent bitmaps.
>> If bitmap is loaded from read-only storage (and we can't mark it
>> "in use" in this storage) corresponding BdrvDirtyBitmap should be
>> read-only.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>> block/dirty-bitmap.c | 16 ++++++++++++++++
>> include/block/dirty-bitmap.h | 3 +++
>> 2 files changed, 19 insertions(+)
> Revisiting this again after the whole series: So you never really make
> sure that the read-only bitmaps are not written to (except for these
> assertions). The idea is that you only set it for read-only BDS and
> read-only BDS are never written to. But that assumption is not true,
> generally, and can be broken e.g. using a commit job:
>
> $ ./qemu-img create -f qcow2 backing.qcow2 64M
> Formatting 'backing.qcow2', fmt=qcow2 size=67108864 encryption=off
> cluster_size=65536 lazy_refcounts=off refcount_bits=16
> $ ./qemu-img create -f qcow2 -b backing.qcow2 top.qcow2
> Formatting 'top.qcow2', fmt=qcow2 size=67108864
> backing_file=backing.qcow2 encryption=off cluster_size=65536
> lazy_refcounts=off refcount_bits=16
> $ x86_64-softmmu/qemu-system-x86_64 -qmp stdio
> {"QMP": {"version": {"qemu": {"micro": 50, "minor": 9, "major": 2},
> "package": " (v2.9.0-632-g4a52d43-dirty)"}, "capabilities": []}}
> {'execute': 'qmp_capabilities'}
> {"return": {}}
> {'execute': 'blockdev-add',
> 'arguments': {'node-name': 'backing-node', 'driver': 'qcow2',
> 'file': {'driver': 'file', 'filename': 'backing.qcow2'}}}
> {"return": {}}
> {'execute': 'block-dirty-bitmap-add',
> 'arguments': {'node': 'backing-node', 'name': 'foo',
> 'persistent': true, 'autoload': true}}
> {"return": {}}
> {'execute': 'blockdev-del', 'arguments': {'node-name': 'backing-node'}}
> {"return": {}}
> {'execute': 'blockdev-add',
> 'arguments': {'node-name': 'top-node', 'driver': 'qcow2',
> 'file': {'driver': 'file', 'filename': 'top.qcow2'}}}
> {"return": {}}
> {'execute': 'human-monitor-command',
> 'arguments': {'command-line': 'qemu-io top-node "write 0 64k"'}}
> wrote 65536/65536 bytes at offset 0
> 64 KiB, 1 ops; 0.0079 sec (7.852 MiB/sec and 125.6281 ops/sec)
> {"return": ""}
> {'execute': 'block-commit',
> 'arguments': {'device': 'top-node', 'job-id': 'commit-job'}}
> {"return": {}}
> qemu-system-x86_64: block/dirty-bitmap.c:571: bdrv_set_dirty: Assertion
> `!bdrv_dirty_bitmap_readonly(bitmap)' failed.
> [1] 10872 abort (core dumped) x86_64-softmmu/qemu-system-x86_64 -qmp
> stdio
>
> So there needs to be something else than just assertions to make sure
> that read-only bitmaps are never written to.
>
> Max
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-30 6:50 ` Vladimir Sementsov-Ogievskiy
@ 2017-05-30 7:31 ` Vladimir Sementsov-Ogievskiy
2017-05-30 7:47 ` Vladimir Sementsov-Ogievskiy
2017-05-31 13:43 ` Max Reitz
1 sibling, 1 reply; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-30 7:31 UTC (permalink / raw)
To: Max Reitz, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
30.05.2017 09:50, Vladimir Sementsov-Ogievskiy wrote:
> Thank you for this scenario. Hmm.
>
> So, as I need guarantee that image and bitmap are unchanged,
> bdrv_set_dirty should return error and fail the whole write. Ok?
No, bad idea. As bdrv_set_dirty is called after write completed.
>
> 29.05.2017 21:35, Max Reitz wrote:
>> On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
>>> It will be needed in following commits for persistent bitmaps.
>>> If bitmap is loaded from read-only storage (and we can't mark it
>>> "in use" in this storage) corresponding BdrvDirtyBitmap should be
>>> read-only.
>>>
>>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>> ---
>>> block/dirty-bitmap.c | 16 ++++++++++++++++
>>> include/block/dirty-bitmap.h | 3 +++
>>> 2 files changed, 19 insertions(+)
>> Revisiting this again after the whole series: So you never really make
>> sure that the read-only bitmaps are not written to (except for these
>> assertions). The idea is that you only set it for read-only BDS and
>> read-only BDS are never written to. But that assumption is not true,
>> generally, and can be broken e.g. using a commit job:
>>
>> $ ./qemu-img create -f qcow2 backing.qcow2 64M
>> Formatting 'backing.qcow2', fmt=qcow2 size=67108864 encryption=off
>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> $ ./qemu-img create -f qcow2 -b backing.qcow2 top.qcow2
>> Formatting 'top.qcow2', fmt=qcow2 size=67108864
>> backing_file=backing.qcow2 encryption=off cluster_size=65536
>> lazy_refcounts=off refcount_bits=16
>> $ x86_64-softmmu/qemu-system-x86_64 -qmp stdio
>> {"QMP": {"version": {"qemu": {"micro": 50, "minor": 9, "major": 2},
>> "package": " (v2.9.0-632-g4a52d43-dirty)"}, "capabilities": []}}
>> {'execute': 'qmp_capabilities'}
>> {"return": {}}
>> {'execute': 'blockdev-add',
>> 'arguments': {'node-name': 'backing-node', 'driver': 'qcow2',
>> 'file': {'driver': 'file', 'filename':
>> 'backing.qcow2'}}}
>> {"return": {}}
>> {'execute': 'block-dirty-bitmap-add',
>> 'arguments': {'node': 'backing-node', 'name': 'foo',
>> 'persistent': true, 'autoload': true}}
>> {"return": {}}
>> {'execute': 'blockdev-del', 'arguments': {'node-name': 'backing-node'}}
>> {"return": {}}
>> {'execute': 'blockdev-add',
>> 'arguments': {'node-name': 'top-node', 'driver': 'qcow2',
>> 'file': {'driver': 'file', 'filename': 'top.qcow2'}}}
>> {"return": {}}
>> {'execute': 'human-monitor-command',
>> 'arguments': {'command-line': 'qemu-io top-node "write 0 64k"'}}
>> wrote 65536/65536 bytes at offset 0
>> 64 KiB, 1 ops; 0.0079 sec (7.852 MiB/sec and 125.6281 ops/sec)
>> {"return": ""}
>> {'execute': 'block-commit',
>> 'arguments': {'device': 'top-node', 'job-id': 'commit-job'}}
>> {"return": {}}
>> qemu-system-x86_64: block/dirty-bitmap.c:571: bdrv_set_dirty: Assertion
>> `!bdrv_dirty_bitmap_readonly(bitmap)' failed.
>> [1] 10872 abort (core dumped) x86_64-softmmu/qemu-system-x86_64 -qmp
>> stdio
>>
>> So there needs to be something else than just assertions to make sure
>> that read-only bitmaps are never written to.
>>
>> Max
>>
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-30 7:31 ` Vladimir Sementsov-Ogievskiy
@ 2017-05-30 7:47 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-30 7:47 UTC (permalink / raw)
To: Max Reitz, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
30.05.2017 10:31, Vladimir Sementsov-Ogievskiy wrote:
> 30.05.2017 09:50, Vladimir Sementsov-Ogievskiy wrote:
>> Thank you for this scenario. Hmm.
>>
>> So, as I need guarantee that image and bitmap are unchanged,
>> bdrv_set_dirty should return error and fail the whole write. Ok?
>
> No, bad idea. As bdrv_set_dirty is called after write completed.
will just check for readonly bitmaps in all callers of set/reset/clear.
They are few.
>
>>
>> 29.05.2017 21:35, Max Reitz wrote:
>>> On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
>>>> It will be needed in following commits for persistent bitmaps.
>>>> If bitmap is loaded from read-only storage (and we can't mark it
>>>> "in use" in this storage) corresponding BdrvDirtyBitmap should be
>>>> read-only.
>>>>
>>>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>>> ---
>>>> block/dirty-bitmap.c | 16 ++++++++++++++++
>>>> include/block/dirty-bitmap.h | 3 +++
>>>> 2 files changed, 19 insertions(+)
>>> Revisiting this again after the whole series: So you never really make
>>> sure that the read-only bitmaps are not written to (except for these
>>> assertions). The idea is that you only set it for read-only BDS and
>>> read-only BDS are never written to. But that assumption is not true,
>>> generally, and can be broken e.g. using a commit job:
>>>
>>> $ ./qemu-img create -f qcow2 backing.qcow2 64M
>>> Formatting 'backing.qcow2', fmt=qcow2 size=67108864 encryption=off
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> $ ./qemu-img create -f qcow2 -b backing.qcow2 top.qcow2
>>> Formatting 'top.qcow2', fmt=qcow2 size=67108864
>>> backing_file=backing.qcow2 encryption=off cluster_size=65536
>>> lazy_refcounts=off refcount_bits=16
>>> $ x86_64-softmmu/qemu-system-x86_64 -qmp stdio
>>> {"QMP": {"version": {"qemu": {"micro": 50, "minor": 9, "major": 2},
>>> "package": " (v2.9.0-632-g4a52d43-dirty)"}, "capabilities": []}}
>>> {'execute': 'qmp_capabilities'}
>>> {"return": {}}
>>> {'execute': 'blockdev-add',
>>> 'arguments': {'node-name': 'backing-node', 'driver': 'qcow2',
>>> 'file': {'driver': 'file', 'filename':
>>> 'backing.qcow2'}}}
>>> {"return": {}}
>>> {'execute': 'block-dirty-bitmap-add',
>>> 'arguments': {'node': 'backing-node', 'name': 'foo',
>>> 'persistent': true, 'autoload': true}}
>>> {"return": {}}
>>> {'execute': 'blockdev-del', 'arguments': {'node-name': 'backing-node'}}
>>> {"return": {}}
>>> {'execute': 'blockdev-add',
>>> 'arguments': {'node-name': 'top-node', 'driver': 'qcow2',
>>> 'file': {'driver': 'file', 'filename': 'top.qcow2'}}}
>>> {"return": {}}
>>> {'execute': 'human-monitor-command',
>>> 'arguments': {'command-line': 'qemu-io top-node "write 0 64k"'}}
>>> wrote 65536/65536 bytes at offset 0
>>> 64 KiB, 1 ops; 0.0079 sec (7.852 MiB/sec and 125.6281 ops/sec)
>>> {"return": ""}
>>> {'execute': 'block-commit',
>>> 'arguments': {'device': 'top-node', 'job-id': 'commit-job'}}
>>> {"return": {}}
>>> qemu-system-x86_64: block/dirty-bitmap.c:571: bdrv_set_dirty: Assertion
>>> `!bdrv_dirty_bitmap_readonly(bitmap)' failed.
>>> [1] 10872 abort (core dumped) x86_64-softmmu/qemu-system-x86_64 -qmp
>>> stdio
>>>
>>> So there needs to be something else than just assertions to make sure
>>> that read-only bitmaps are never written to.
>>>
>>> Max
>>>
>>
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 25/25] block: release persistent bitmaps on inactivate
2017-05-30 6:30 ` Vladimir Sementsov-Ogievskiy
@ 2017-05-31 13:37 ` Max Reitz
0 siblings, 0 replies; 56+ messages in thread
From: Max Reitz @ 2017-05-31 13:37 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
[-- Attachment #1: Type: text/plain, Size: 1706 bytes --]
On 2017-05-30 08:30, Vladimir Sementsov-Ogievskiy wrote:
> 29.05.2017 20:54, Max Reitz wrote:
>> On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
>>> We should release them here to reload on invalidate cache.
>>>
>>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>> ---
>>> block.c | 4 ++++
>>> block/dirty-bitmap.c | 29 +++++++++++++++++++++++------
>>> include/block/dirty-bitmap.h | 1 +
>>> 3 files changed, 28 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/block.c b/block.c
>>> index 795d36bb64..14896c65fa 100644
>>> --- a/block.c
>>> +++ b/block.c
>>> @@ -4001,6 +4001,10 @@ static int
>>> bdrv_inactivate_recurse(BlockDriverState *bs,
>>> if (setting_flag) {
>>> bs->open_flags |= BDRV_O_INACTIVE;
>>> }
>>> +
>>> + /* At this point persistent bitmaps should be stored by format
>>> driver */
>> s/by format driver/by the format driver/
>>
>>> + bdrv_release_persistent_dirty_bitmaps(bs);
>> Also, as far as I can see, this doesn't store the bitmaps but just
>> releases them (without storing them). I'm not sure whether that is
>> right, but it definitely contradicts the comment above.
>
> I mean they should be _already_ stored.. They actually are stored in
> qcow2_inactivate.
Ah, right, that makes more sense.
However, to be more clear the comment should then read "By this point"
instead of "At this point"; maybe even "By this point the persistent
bitmaps should have been stored by the format driver already", or a
totally different "The format driver's bdrv_inactivate() implementation
should have stored all persistent bitmaps".
Max
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 498 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-30 6:50 ` Vladimir Sementsov-Ogievskiy
2017-05-30 7:31 ` Vladimir Sementsov-Ogievskiy
@ 2017-05-31 13:43 ` Max Reitz
2017-05-31 14:29 ` Vladimir Sementsov-Ogievskiy
2017-06-01 22:29 ` John Snow
1 sibling, 2 replies; 56+ messages in thread
From: Max Reitz @ 2017-05-31 13:43 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
[-- Attachment #1: Type: text/plain, Size: 4465 bytes --]
On 2017-05-30 08:50, Vladimir Sementsov-Ogievskiy wrote:
> Thank you for this scenario. Hmm.
>
> So, as I need guarantee that image and bitmap are unchanged,
> bdrv_set_dirty should return error and fail the whole write. Ok?
I don't know. That would mean that you couldn't commit to an image that
has a persistent auto-loading bitmap, which doesn't seem very nice to me.
I'm not quite sure what to do myself. So first I'd definitely want the
commit operation to succeed. That means we'd have to automatically make
the bitmap non-readonly once we write to it. The "readonly" flag would
then be an "unchanged" flag, rather, to signify that the bitmap has not
been changed since it was loaded, which means that it does not need to
be written back to the image file.
Now the issue remains that if you modify a persistent bitmap that is
stored in an image file that is opened RO when it's closed, you won't be
able to write the modifications back.
So in addition, I guess we'd need to "flush" all persistent bitmaps
(that is, write all modifications back to the file and set the
"unchanged" flag (you could also call it "dirty" and then mean the
opposite) for each bitmap) not only when the image is closed or
invalidated, but also when it is reopened read-only.
(block-commit reopens the backing BDS R/W, then writes to them, thus
modifying the dirty bitmaps, and finally reopens the BDS as read-only;
before that happens, we will have to flush the modified bitmap data.)
Max
> 29.05.2017 21:35, Max Reitz wrote:
>> On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
>>> It will be needed in following commits for persistent bitmaps.
>>> If bitmap is loaded from read-only storage (and we can't mark it
>>> "in use" in this storage) corresponding BdrvDirtyBitmap should be
>>> read-only.
>>>
>>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>> ---
>>> block/dirty-bitmap.c | 16 ++++++++++++++++
>>> include/block/dirty-bitmap.h | 3 +++
>>> 2 files changed, 19 insertions(+)
>> Revisiting this again after the whole series: So you never really make
>> sure that the read-only bitmaps are not written to (except for these
>> assertions). The idea is that you only set it for read-only BDS and
>> read-only BDS are never written to. But that assumption is not true,
>> generally, and can be broken e.g. using a commit job:
>>
>> $ ./qemu-img create -f qcow2 backing.qcow2 64M
>> Formatting 'backing.qcow2', fmt=qcow2 size=67108864 encryption=off
>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> $ ./qemu-img create -f qcow2 -b backing.qcow2 top.qcow2
>> Formatting 'top.qcow2', fmt=qcow2 size=67108864
>> backing_file=backing.qcow2 encryption=off cluster_size=65536
>> lazy_refcounts=off refcount_bits=16
>> $ x86_64-softmmu/qemu-system-x86_64 -qmp stdio
>> {"QMP": {"version": {"qemu": {"micro": 50, "minor": 9, "major": 2},
>> "package": " (v2.9.0-632-g4a52d43-dirty)"}, "capabilities": []}}
>> {'execute': 'qmp_capabilities'}
>> {"return": {}}
>> {'execute': 'blockdev-add',
>> 'arguments': {'node-name': 'backing-node', 'driver': 'qcow2',
>> 'file': {'driver': 'file', 'filename': 'backing.qcow2'}}}
>> {"return": {}}
>> {'execute': 'block-dirty-bitmap-add',
>> 'arguments': {'node': 'backing-node', 'name': 'foo',
>> 'persistent': true, 'autoload': true}}
>> {"return": {}}
>> {'execute': 'blockdev-del', 'arguments': {'node-name': 'backing-node'}}
>> {"return": {}}
>> {'execute': 'blockdev-add',
>> 'arguments': {'node-name': 'top-node', 'driver': 'qcow2',
>> 'file': {'driver': 'file', 'filename': 'top.qcow2'}}}
>> {"return": {}}
>> {'execute': 'human-monitor-command',
>> 'arguments': {'command-line': 'qemu-io top-node "write 0 64k"'}}
>> wrote 65536/65536 bytes at offset 0
>> 64 KiB, 1 ops; 0.0079 sec (7.852 MiB/sec and 125.6281 ops/sec)
>> {"return": ""}
>> {'execute': 'block-commit',
>> 'arguments': {'device': 'top-node', 'job-id': 'commit-job'}}
>> {"return": {}}
>> qemu-system-x86_64: block/dirty-bitmap.c:571: bdrv_set_dirty: Assertion
>> `!bdrv_dirty_bitmap_readonly(bitmap)' failed.
>> [1] 10872 abort (core dumped) x86_64-softmmu/qemu-system-x86_64 -qmp
>> stdio
>>
>> So there needs to be something else than just assertions to make sure
>> that read-only bitmaps are never written to.
>>
>> Max
>>
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 498 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-31 13:43 ` Max Reitz
@ 2017-05-31 14:29 ` Vladimir Sementsov-Ogievskiy
2017-05-31 14:44 ` Max Reitz
2017-06-01 22:29 ` John Snow
1 sibling, 1 reply; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-31 14:29 UTC (permalink / raw)
To: Max Reitz, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
31.05.2017 16:43, Max Reitz wrote:
> On 2017-05-30 08:50, Vladimir Sementsov-Ogievskiy wrote:
>> Thank you for this scenario. Hmm.
>>
>> So, as I need guarantee that image and bitmap are unchanged,
>> bdrv_set_dirty should return error and fail the whole write. Ok?
> I don't know. That would mean that you couldn't commit to an image that
> has a persistent auto-loading bitmap, which doesn't seem very nice to me.
>
> I'm not quite sure what to do myself. So first I'd definitely want the
> commit operation to succeed. That means we'd have to automatically make
> the bitmap non-readonly once we write to it. The "readonly" flag would
> then be an "unchanged" flag, rather, to signify that the bitmap has not
> been changed since it was loaded, which means that it does not need to
> be written back to the image file.
>
> Now the issue remains that if you modify a persistent bitmap that is
> stored in an image file that is opened RO when it's closed, you won't be
> able to write the modifications back.
>
> So in addition, I guess we'd need to "flush" all persistent bitmaps
> (that is, write all modifications back to the file and set the
> "unchanged" flag (you could also call it "dirty" and then mean the
> opposite) for each bitmap) not only when the image is closed or
> invalidated, but also when it is reopened read-only.
>
> (block-commit reopens the backing BDS R/W, then writes to them, thus
> modifying the dirty bitmaps, and finally reopens the BDS as read-only;
> before that happens, we will have to flush the modified bitmap data.)
Ok, understand.
We need to consider also setting in_use flag in the image. We _must not_
write to image with dirty bitmap,
if in_use flag of this dirty bitmap is not set, as in case of something
fail we will have image with wrong bitmap with
unset in_use flag (which looks ok).
I see two ways to handle it:
variant 1:
1. readonly field stays as is (see v19, with normal errors, not only
asserts)
2. immediately after reopening r/w we do "reopening bitmaps r/w", i.e.
set in_use in the image and set BdrvDirtyBitmap.readonly = false
3. in reopen_prepare, if reopening r-o do "reopening bitmaps r-o", i.e.
save them into the image and set BdrvDirtyBitmap.readonly = true
variant 2:
1. instead of 'readonly' add 'dirty' field, set dirty to 0 for all
bitmaps on create
2. before write/discard check this field in all related bitmaps, and if
dirty=0 (and persistent=1), write IN_USE flag into the image first, set
dirty=1, and only then do write. (if writing IN_USE=1 failed, fail the
whole write)
3. in reopen_prepare, if reopening r-o do "reopening bitmaps r-o", i.e.
save them into the image and set BdrvDirtyBitmap.dirty = 0
>
> Max
>
>> 29.05.2017 21:35, Max Reitz wrote:
>>> On 2017-05-03 14:25, Vladimir Sementsov-Ogievskiy wrote:
>>>> It will be needed in following commits for persistent bitmaps.
>>>> If bitmap is loaded from read-only storage (and we can't mark it
>>>> "in use" in this storage) corresponding BdrvDirtyBitmap should be
>>>> read-only.
>>>>
>>>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>>> ---
>>>> block/dirty-bitmap.c | 16 ++++++++++++++++
>>>> include/block/dirty-bitmap.h | 3 +++
>>>> 2 files changed, 19 insertions(+)
>>> Revisiting this again after the whole series: So you never really make
>>> sure that the read-only bitmaps are not written to (except for these
>>> assertions). The idea is that you only set it for read-only BDS and
>>> read-only BDS are never written to. But that assumption is not true,
>>> generally, and can be broken e.g. using a commit job:
>>>
>>> $ ./qemu-img create -f qcow2 backing.qcow2 64M
>>> Formatting 'backing.qcow2', fmt=qcow2 size=67108864 encryption=off
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> $ ./qemu-img create -f qcow2 -b backing.qcow2 top.qcow2
>>> Formatting 'top.qcow2', fmt=qcow2 size=67108864
>>> backing_file=backing.qcow2 encryption=off cluster_size=65536
>>> lazy_refcounts=off refcount_bits=16
>>> $ x86_64-softmmu/qemu-system-x86_64 -qmp stdio
>>> {"QMP": {"version": {"qemu": {"micro": 50, "minor": 9, "major": 2},
>>> "package": " (v2.9.0-632-g4a52d43-dirty)"}, "capabilities": []}}
>>> {'execute': 'qmp_capabilities'}
>>> {"return": {}}
>>> {'execute': 'blockdev-add',
>>> 'arguments': {'node-name': 'backing-node', 'driver': 'qcow2',
>>> 'file': {'driver': 'file', 'filename': 'backing.qcow2'}}}
>>> {"return": {}}
>>> {'execute': 'block-dirty-bitmap-add',
>>> 'arguments': {'node': 'backing-node', 'name': 'foo',
>>> 'persistent': true, 'autoload': true}}
>>> {"return": {}}
>>> {'execute': 'blockdev-del', 'arguments': {'node-name': 'backing-node'}}
>>> {"return": {}}
>>> {'execute': 'blockdev-add',
>>> 'arguments': {'node-name': 'top-node', 'driver': 'qcow2',
>>> 'file': {'driver': 'file', 'filename': 'top.qcow2'}}}
>>> {"return": {}}
>>> {'execute': 'human-monitor-command',
>>> 'arguments': {'command-line': 'qemu-io top-node "write 0 64k"'}}
>>> wrote 65536/65536 bytes at offset 0
>>> 64 KiB, 1 ops; 0.0079 sec (7.852 MiB/sec and 125.6281 ops/sec)
>>> {"return": ""}
>>> {'execute': 'block-commit',
>>> 'arguments': {'device': 'top-node', 'job-id': 'commit-job'}}
>>> {"return": {}}
>>> qemu-system-x86_64: block/dirty-bitmap.c:571: bdrv_set_dirty: Assertion
>>> `!bdrv_dirty_bitmap_readonly(bitmap)' failed.
>>> [1] 10872 abort (core dumped) x86_64-softmmu/qemu-system-x86_64 -qmp
>>> stdio
>>>
>>> So there needs to be something else than just assertions to make sure
>>> that read-only bitmaps are never written to.
>>>
>>> Max
>>>
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-31 14:29 ` Vladimir Sementsov-Ogievskiy
@ 2017-05-31 14:44 ` Max Reitz
2017-05-31 15:05 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 56+ messages in thread
From: Max Reitz @ 2017-05-31 14:44 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
[-- Attachment #1: Type: text/plain, Size: 3272 bytes --]
On 2017-05-31 16:29, Vladimir Sementsov-Ogievskiy wrote:
> 31.05.2017 16:43, Max Reitz wrote:
>> On 2017-05-30 08:50, Vladimir Sementsov-Ogievskiy wrote:
>>> Thank you for this scenario. Hmm.
>>>
>>> So, as I need guarantee that image and bitmap are unchanged,
>>> bdrv_set_dirty should return error and fail the whole write. Ok?
>> I don't know. That would mean that you couldn't commit to an image that
>> has a persistent auto-loading bitmap, which doesn't seem very nice to me.
>>
>> I'm not quite sure what to do myself. So first I'd definitely want the
>> commit operation to succeed. That means we'd have to automatically make
>> the bitmap non-readonly once we write to it. The "readonly" flag would
>> then be an "unchanged" flag, rather, to signify that the bitmap has not
>> been changed since it was loaded, which means that it does not need to
>> be written back to the image file.
>>
>> Now the issue remains that if you modify a persistent bitmap that is
>> stored in an image file that is opened RO when it's closed, you won't be
>> able to write the modifications back.
>>
>> So in addition, I guess we'd need to "flush" all persistent bitmaps
>> (that is, write all modifications back to the file and set the
>> "unchanged" flag (you could also call it "dirty" and then mean the
>> opposite) for each bitmap) not only when the image is closed or
>> invalidated, but also when it is reopened read-only.
>>
>> (block-commit reopens the backing BDS R/W, then writes to them, thus
>> modifying the dirty bitmaps, and finally reopens the BDS as read-only;
>> before that happens, we will have to flush the modified bitmap data.)
>
> Ok, understand.
>
> We need to consider also setting in_use flag in the image. We _must not_
> write to image with dirty bitmap,
> if in_use flag of this dirty bitmap is not set, as in case of something
> fail we will have image with wrong bitmap with
> unset in_use flag (which looks ok).
Right.
> I see two ways to handle it:
>
> variant 1:
> 1. readonly field stays as is (see v19, with normal errors, not only
> asserts)
> 2. immediately after reopening r/w we do "reopening bitmaps r/w", i.e.
> set in_use in the image and set BdrvDirtyBitmap.readonly = false
> 3. in reopen_prepare, if reopening r-o do "reopening bitmaps r-o", i.e.
> save them into the image and set BdrvDirtyBitmap.readonly = true
Sounds good, yes.
> variant 2:
> 1. instead of 'readonly' add 'dirty' field, set dirty to 0 for all
> bitmaps on create
> 2. before write/discard check this field in all related bitmaps, and if
> dirty=0 (and persistent=1), write IN_USE flag into the image first, set
> dirty=1, and only then do write. (if writing IN_USE=1 failed, fail the
> whole write)
> 3. in reopen_prepare, if reopening r-o do "reopening bitmaps r-o", i.e.
> save them into the image and set BdrvDirtyBitmap.dirty = 0
Works, too.
I think the second variant would the more "efficient" way (because you
only have to flush out dirty dirty bitmaps), but the first one would be
simpler and has the great advantage of not requiring a write to the
image file when you just want to set a bit in the in-memory dirty
bitmap. So I'd personally go for the first variant.
Max
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 498 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-31 14:44 ` Max Reitz
@ 2017-05-31 15:05 ` Vladimir Sementsov-Ogievskiy
2017-05-31 15:53 ` Max Reitz
0 siblings, 1 reply; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-31 15:05 UTC (permalink / raw)
To: Max Reitz, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
31.05.2017 17:44, Max Reitz wrote:
> On 2017-05-31 16:29, Vladimir Sementsov-Ogievskiy wrote:
>> 31.05.2017 16:43, Max Reitz wrote:
>>> On 2017-05-30 08:50, Vladimir Sementsov-Ogievskiy wrote:
>>>> Thank you for this scenario. Hmm.
>>>>
>>>> So, as I need guarantee that image and bitmap are unchanged,
>>>> bdrv_set_dirty should return error and fail the whole write. Ok?
>>> I don't know. That would mean that you couldn't commit to an image that
>>> has a persistent auto-loading bitmap, which doesn't seem very nice to me.
>>>
>>> I'm not quite sure what to do myself. So first I'd definitely want the
>>> commit operation to succeed. That means we'd have to automatically make
>>> the bitmap non-readonly once we write to it. The "readonly" flag would
>>> then be an "unchanged" flag, rather, to signify that the bitmap has not
>>> been changed since it was loaded, which means that it does not need to
>>> be written back to the image file.
>>>
>>> Now the issue remains that if you modify a persistent bitmap that is
>>> stored in an image file that is opened RO when it's closed, you won't be
>>> able to write the modifications back.
>>>
>>> So in addition, I guess we'd need to "flush" all persistent bitmaps
>>> (that is, write all modifications back to the file and set the
>>> "unchanged" flag (you could also call it "dirty" and then mean the
>>> opposite) for each bitmap) not only when the image is closed or
>>> invalidated, but also when it is reopened read-only.
>>>
>>> (block-commit reopens the backing BDS R/W, then writes to them, thus
>>> modifying the dirty bitmaps, and finally reopens the BDS as read-only;
>>> before that happens, we will have to flush the modified bitmap data.)
>> Ok, understand.
>>
>> We need to consider also setting in_use flag in the image. We _must not_
>> write to image with dirty bitmap,
>> if in_use flag of this dirty bitmap is not set, as in case of something
>> fail we will have image with wrong bitmap with
>> unset in_use flag (which looks ok).
> Right.
>
>> I see two ways to handle it:
>>
>> variant 1:
>> 1. readonly field stays as is (see v19, with normal errors, not only
>> asserts)
>> 2. immediately after reopening r/w we do "reopening bitmaps r/w", i.e.
>> set in_use in the image and set BdrvDirtyBitmap.readonly = false
>> 3. in reopen_prepare, if reopening r-o do "reopening bitmaps r-o", i.e.
>> save them into the image and set BdrvDirtyBitmap.readonly = true
> Sounds good, yes.
>
>> variant 2:
>> 1. instead of 'readonly' add 'dirty' field, set dirty to 0 for all
>> bitmaps on create
>> 2. before write/discard check this field in all related bitmaps, and if
>> dirty=0 (and persistent=1), write IN_USE flag into the image first, set
>> dirty=1, and only then do write. (if writing IN_USE=1 failed, fail the
>> whole write)
>> 3. in reopen_prepare, if reopening r-o do "reopening bitmaps r-o", i.e.
>> save them into the image and set BdrvDirtyBitmap.dirty = 0
> Works, too.
>
> I think the second variant would the more "efficient" way (because you
> only have to flush out dirty dirty bitmaps), but the first one would be
> simpler and has the great advantage of not requiring a write to the
> image file when you just want to set a bit in the in-memory dirty
> bitmap. So I'd personally go for the first variant.
Hmm, why not requiring? Both 1 and 2 do write in_use=1, but (1) do this
on open/reopen, and (2) before the first write to the image.
"set a bit in the in-memory..." - are you saying about not-persistent
dirty bitmaps? In this case, of course, nothing should be written into
the image, just set dirty=1.
>
> Max
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-31 15:05 ` Vladimir Sementsov-Ogievskiy
@ 2017-05-31 15:53 ` Max Reitz
2017-06-01 22:35 ` John Snow
0 siblings, 1 reply; 56+ messages in thread
From: Max Reitz @ 2017-05-31 15:53 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, armbru, eblake, jsnow, famz, den, stefanha, pbonzini
[-- Attachment #1: Type: text/plain, Size: 4209 bytes --]
On 2017-05-31 17:05, Vladimir Sementsov-Ogievskiy wrote:
> 31.05.2017 17:44, Max Reitz wrote:
>> On 2017-05-31 16:29, Vladimir Sementsov-Ogievskiy wrote:
>>> 31.05.2017 16:43, Max Reitz wrote:
>>>> On 2017-05-30 08:50, Vladimir Sementsov-Ogievskiy wrote:
>>>>> Thank you for this scenario. Hmm.
>>>>>
>>>>> So, as I need guarantee that image and bitmap are unchanged,
>>>>> bdrv_set_dirty should return error and fail the whole write. Ok?
>>>> I don't know. That would mean that you couldn't commit to an image that
>>>> has a persistent auto-loading bitmap, which doesn't seem very nice
>>>> to me.
>>>>
>>>> I'm not quite sure what to do myself. So first I'd definitely want the
>>>> commit operation to succeed. That means we'd have to automatically make
>>>> the bitmap non-readonly once we write to it. The "readonly" flag would
>>>> then be an "unchanged" flag, rather, to signify that the bitmap has not
>>>> been changed since it was loaded, which means that it does not need to
>>>> be written back to the image file.
>>>>
>>>> Now the issue remains that if you modify a persistent bitmap that is
>>>> stored in an image file that is opened RO when it's closed, you
>>>> won't be
>>>> able to write the modifications back.
>>>>
>>>> So in addition, I guess we'd need to "flush" all persistent bitmaps
>>>> (that is, write all modifications back to the file and set the
>>>> "unchanged" flag (you could also call it "dirty" and then mean the
>>>> opposite) for each bitmap) not only when the image is closed or
>>>> invalidated, but also when it is reopened read-only.
>>>>
>>>> (block-commit reopens the backing BDS R/W, then writes to them, thus
>>>> modifying the dirty bitmaps, and finally reopens the BDS as read-only;
>>>> before that happens, we will have to flush the modified bitmap data.)
>>> Ok, understand.
>>>
>>> We need to consider also setting in_use flag in the image. We _must not_
>>> write to image with dirty bitmap,
>>> if in_use flag of this dirty bitmap is not set, as in case of something
>>> fail we will have image with wrong bitmap with
>>> unset in_use flag (which looks ok).
>> Right.
>>
>>> I see two ways to handle it:
>>>
>>> variant 1:
>>> 1. readonly field stays as is (see v19, with normal errors, not only
>>> asserts)
>>> 2. immediately after reopening r/w we do "reopening bitmaps r/w", i.e.
>>> set in_use in the image and set BdrvDirtyBitmap.readonly = false
>>> 3. in reopen_prepare, if reopening r-o do "reopening bitmaps r-o", i.e.
>>> save them into the image and set BdrvDirtyBitmap.readonly = true
>> Sounds good, yes.
>>
>>> variant 2:
>>> 1. instead of 'readonly' add 'dirty' field, set dirty to 0 for all
>>> bitmaps on create
>>> 2. before write/discard check this field in all related bitmaps, and if
>>> dirty=0 (and persistent=1), write IN_USE flag into the image first, set
>>> dirty=1, and only then do write. (if writing IN_USE=1 failed, fail the
>>> whole write)
>>> 3. in reopen_prepare, if reopening r-o do "reopening bitmaps r-o", i.e.
>>> save them into the image and set BdrvDirtyBitmap.dirty = 0
>> Works, too.
>>
>> I think the second variant would the more "efficient" way (because you
>> only have to flush out dirty dirty bitmaps), but the first one would be
>> simpler and has the great advantage of not requiring a write to the
>> image file when you just want to set a bit in the in-memory dirty
>> bitmap. So I'd personally go for the first variant.
>
> Hmm, why not requiring? Both 1 and 2 do write in_use=1, but (1) do this
> on open/reopen, and (2) before the first write to the image.
Oh, I didn't read the "before write/discard". Yes, if you check it
before writing, then you won't have to set the flag through
bdrv_set_dirty().
> "set a bit in the in-memory..." - are you saying about not-persistent
> dirty bitmaps? In this case, of course, nothing should be written into
> the image, just set dirty=1.
No, I did mean persistent bitmaps, but bdrv_set_dirty() just sets the
bit in main memory, of course. It only gets written to the image later
(on reopen/close/invalidate).
Well, your choice. I think both will work. :-)
Max
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 498 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-19 23:02 ` John Snow
2017-05-25 9:34 ` Vladimir Sementsov-Ogievskiy
@ 2017-05-31 22:58 ` John Snow
2017-06-01 7:23 ` Sementsov-Ogievskiy Vladimir
1 sibling, 1 reply; 56+ messages in thread
From: John Snow @ 2017-05-31 22:58 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, famz, armbru, mreitz, stefanha, pbonzini, den
On 05/19/2017 07:02 PM, John Snow wrote:
>
>
> On 05/03/2017 08:25 AM, Vladimir Sementsov-Ogievskiy wrote:
>> It will be needed in following commits for persistent bitmaps.
>> If bitmap is loaded from read-only storage (and we can't mark it
>> "in use" in this storage) corresponding BdrvDirtyBitmap should be
>> read-only.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>> block/dirty-bitmap.c | 16 ++++++++++++++++
>> include/block/dirty-bitmap.h | 3 +++
>> 2 files changed, 19 insertions(+)
>>
>> diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
>> index 90af37287f..ab6a95cf41 100644
>> --- a/block/dirty-bitmap.c
>> +++ b/block/dirty-bitmap.c
>> @@ -44,6 +44,8 @@ struct BdrvDirtyBitmap {
>> int64_t size; /* Size of the bitmap (Number of sectors) */
>> bool disabled; /* Bitmap is read-only */
>> int active_iterators; /* How many iterators are active */
>> + bool readonly; /* Bitmap is read-only and may be changed only
>> + by deserialize* functions */
>> QLIST_ENTRY(BdrvDirtyBitmap) list;
>> };
>>
>> @@ -436,6 +438,7 @@ void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
>> int64_t cur_sector, int64_t nr_sectors)
>> {
>> assert(bdrv_dirty_bitmap_enabled(bitmap));
>> + assert(!bdrv_dirty_bitmap_readonly(bitmap));
>
> Doesn't this change the nature of the assertion?
>
> We're going from
> return !(bitmap->disabled || bitmap->successor);
> to
> !bitmap->readonly
>
> That makes me a little nervous to ACK this patch, because the
> correctness depends on how readonly is updated and manipulated in the
> future, which makes it more prone to error.
>
I must have been *REALLY* tired when I sent this; I had thought you were
replacing an assertion instead of just adding one.
I'm sorry about that.
>> hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
>> }
>>
>> @@ -443,12 +446,14 @@ void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
>> int64_t cur_sector, int64_t nr_sectors)
>> {
>> assert(bdrv_dirty_bitmap_enabled(bitmap));
>> + assert(!bdrv_dirty_bitmap_readonly(bitmap));
>> hbitmap_reset(bitmap->bitmap, cur_sector, nr_sectors);
>> }
>>
>> void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap **out)
>> {
>> assert(bdrv_dirty_bitmap_enabled(bitmap));
>> + assert(!bdrv_dirty_bitmap_readonly(bitmap));
>> if (!out) {
>> hbitmap_reset_all(bitmap->bitmap);
>> } else {
>> @@ -519,6 +524,7 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
>> if (!bdrv_dirty_bitmap_enabled(bitmap)) {
>> continue;
>> }
>> + assert(!bdrv_dirty_bitmap_readonly(bitmap));
>> hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
>> }
>> }
>> @@ -540,3 +546,13 @@ int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap)
>> {
>> return hbitmap_count(bitmap->meta);
>> }
>> +
>> +bool bdrv_dirty_bitmap_readonly(const BdrvDirtyBitmap *bitmap)
>> +{
>> + return bitmap->readonly;
>> +}
>> +
>> +void bdrv_dirty_bitmap_set_readonly(BdrvDirtyBitmap *bitmap)
>> +{
>> + bitmap->readonly = true;
>> +}
>> diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
>> index 1e17729ac2..0aab5841f5 100644
>> --- a/include/block/dirty-bitmap.h
>> +++ b/include/block/dirty-bitmap.h
>> @@ -75,4 +75,7 @@ void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap,
>> bool finish);
>> void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
>>
>> +bool bdrv_dirty_bitmap_readonly(const BdrvDirtyBitmap *bitmap);
>> +void bdrv_dirty_bitmap_set_readonly(BdrvDirtyBitmap *bitmap);
>> +
>> #endif
>>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-31 22:58 ` John Snow
@ 2017-06-01 7:23 ` Sementsov-Ogievskiy Vladimir
0 siblings, 0 replies; 56+ messages in thread
From: Sementsov-Ogievskiy Vladimir @ 2017-06-01 7:23 UTC (permalink / raw)
To: John Snow, qemu-block, qemu-devel
Cc: kwolf, famz, armbru, mreitz, stefanha, pbonzini, den
On 01.06.2017 01:58, John Snow wrote:
>
> On 05/19/2017 07:02 PM, John Snow wrote:
>>
>> On 05/03/2017 08:25 AM, Vladimir Sementsov-Ogievskiy wrote:
>>> It will be needed in following commits for persistent bitmaps.
>>> If bitmap is loaded from read-only storage (and we can't mark it
>>> "in use" in this storage) corresponding BdrvDirtyBitmap should be
>>> read-only.
>>>
>>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>> ---
>>> block/dirty-bitmap.c | 16 ++++++++++++++++
>>> include/block/dirty-bitmap.h | 3 +++
>>> 2 files changed, 19 insertions(+)
>>>
>>> diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
>>> index 90af37287f..ab6a95cf41 100644
>>> --- a/block/dirty-bitmap.c
>>> +++ b/block/dirty-bitmap.c
>>> @@ -44,6 +44,8 @@ struct BdrvDirtyBitmap {
>>> int64_t size; /* Size of the bitmap (Number of sectors) */
>>> bool disabled; /* Bitmap is read-only */
>>> int active_iterators; /* How many iterators are active */
>>> + bool readonly; /* Bitmap is read-only and may be changed only
>>> + by deserialize* functions */
>>> QLIST_ENTRY(BdrvDirtyBitmap) list;
>>> };
>>>
>>> @@ -436,6 +438,7 @@ void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
>>> int64_t cur_sector, int64_t nr_sectors)
>>> {
>>> assert(bdrv_dirty_bitmap_enabled(bitmap));
>>> + assert(!bdrv_dirty_bitmap_readonly(bitmap));
>> Doesn't this change the nature of the assertion?
>>
>> We're going from
>> return !(bitmap->disabled || bitmap->successor);
>> to
>> !bitmap->readonly
>>
>> That makes me a little nervous to ACK this patch, because the
>> correctness depends on how readonly is updated and manipulated in the
>> future, which makes it more prone to error.
>>
> I must have been *REALLY* tired when I sent this; I had thought you were
> replacing an assertion instead of just adding one.
>
> I'm sorry about that.
I thought so, don't worry)
>
>>> hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
>>> }
>>>
>>> @@ -443,12 +446,14 @@ void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
>>> int64_t cur_sector, int64_t nr_sectors)
>>> {
>>> assert(bdrv_dirty_bitmap_enabled(bitmap));
>>> + assert(!bdrv_dirty_bitmap_readonly(bitmap));
>>> hbitmap_reset(bitmap->bitmap, cur_sector, nr_sectors);
>>> }
>>>
>>> void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap **out)
>>> {
>>> assert(bdrv_dirty_bitmap_enabled(bitmap));
>>> + assert(!bdrv_dirty_bitmap_readonly(bitmap));
>>> if (!out) {
>>> hbitmap_reset_all(bitmap->bitmap);
>>> } else {
>>> @@ -519,6 +524,7 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
>>> if (!bdrv_dirty_bitmap_enabled(bitmap)) {
>>> continue;
>>> }
>>> + assert(!bdrv_dirty_bitmap_readonly(bitmap));
>>> hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
>>> }
>>> }
>>> @@ -540,3 +546,13 @@ int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap)
>>> {
>>> return hbitmap_count(bitmap->meta);
>>> }
>>> +
>>> +bool bdrv_dirty_bitmap_readonly(const BdrvDirtyBitmap *bitmap)
>>> +{
>>> + return bitmap->readonly;
>>> +}
>>> +
>>> +void bdrv_dirty_bitmap_set_readonly(BdrvDirtyBitmap *bitmap)
>>> +{
>>> + bitmap->readonly = true;
>>> +}
>>> diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
>>> index 1e17729ac2..0aab5841f5 100644
>>> --- a/include/block/dirty-bitmap.h
>>> +++ b/include/block/dirty-bitmap.h
>>> @@ -75,4 +75,7 @@ void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap,
>>> bool finish);
>>> void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
>>>
>>> +bool bdrv_dirty_bitmap_readonly(const BdrvDirtyBitmap *bitmap);
>>> +void bdrv_dirty_bitmap_set_readonly(BdrvDirtyBitmap *bitmap);
>>> +
>>> #endif
>>>
--
Best regards,
Vladimir.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-31 13:43 ` Max Reitz
2017-05-31 14:29 ` Vladimir Sementsov-Ogievskiy
@ 2017-06-01 22:29 ` John Snow
2017-06-07 12:40 ` Max Reitz
1 sibling, 1 reply; 56+ messages in thread
From: John Snow @ 2017-06-01 22:29 UTC (permalink / raw)
To: Max Reitz, Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, famz, armbru, stefanha, pbonzini, den
On 05/31/2017 09:43 AM, Max Reitz wrote:
> On 2017-05-30 08:50, Vladimir Sementsov-Ogievskiy wrote:
>> Thank you for this scenario. Hmm.
>>
>> So, as I need guarantee that image and bitmap are unchanged,
>> bdrv_set_dirty should return error and fail the whole write. Ok?
>
> I don't know. That would mean that you couldn't commit to an image that
> has a persistent auto-loading bitmap, which doesn't seem very nice to me.
>
> I'm not quite sure what to do myself. So first I'd definitely want the
> commit operation to succeed. That means we'd have to automatically make
> the bitmap non-readonly once we write to it. The "readonly" flag would
> then be an "unchanged" flag, rather, to signify that the bitmap has not
> been changed since it was loaded, which means that it does not need to
> be written back to the image file.
>
> Now the issue remains that if you modify a persistent bitmap that is
> stored in an image file that is opened RO when it's closed, you won't be
> able to write the modifications back.
> > So in addition, I guess we'd need to "flush" all persistent bitmaps
> (that is, write all modifications back to the file and set the
> "unchanged" flag (you could also call it "dirty" and then mean the
> opposite) for each bitmap) not only when the image is closed or
> invalidated, but also when it is reopened read-only.
>
Makes sense.
> (block-commit reopens the backing BDS R/W, then writes to them, thus
> modifying the dirty bitmaps, and finally reopens the BDS as read-only;
> before that happens, we will have to flush the modified bitmap data.)
>
OK, so it would perhaps be enough to toggle the RO flag on/off when
nodes get reopened. When they get reopened RO, we'd need to flush at
that point.
(Right?)
Of course, a changed flag makes this a little moot as it is probably
more flexible; but there is something slightly attractive about the more
rigid form.
(Hmm, for the purposes of periodic flushing, we may want a changed flag
anyway...)
> Max
>
Thanks for the scenario and the explainer.
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-05-31 15:53 ` Max Reitz
@ 2017-06-01 22:35 ` John Snow
0 siblings, 0 replies; 56+ messages in thread
From: John Snow @ 2017-06-01 22:35 UTC (permalink / raw)
To: Max Reitz, Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, famz, armbru, stefanha, pbonzini, den
On 05/31/2017 11:53 AM, Max Reitz wrote:
> On 2017-05-31 17:05, Vladimir Sementsov-Ogievskiy wrote:
>> 31.05.2017 17:44, Max Reitz wrote:
>>> On 2017-05-31 16:29, Vladimir Sementsov-Ogievskiy wrote:
>>>> 31.05.2017 16:43, Max Reitz wrote:
>>>>> On 2017-05-30 08:50, Vladimir Sementsov-Ogievskiy wrote:
>>>>>> Thank you for this scenario. Hmm.
>>>>>>
>>>>>> So, as I need guarantee that image and bitmap are unchanged,
>>>>>> bdrv_set_dirty should return error and fail the whole write. Ok?
>>>>> I don't know. That would mean that you couldn't commit to an image that
>>>>> has a persistent auto-loading bitmap, which doesn't seem very nice
>>>>> to me.
>>>>>
>>>>> I'm not quite sure what to do myself. So first I'd definitely want the
>>>>> commit operation to succeed. That means we'd have to automatically make
>>>>> the bitmap non-readonly once we write to it. The "readonly" flag would
>>>>> then be an "unchanged" flag, rather, to signify that the bitmap has not
>>>>> been changed since it was loaded, which means that it does not need to
>>>>> be written back to the image file.
>>>>>
>>>>> Now the issue remains that if you modify a persistent bitmap that is
>>>>> stored in an image file that is opened RO when it's closed, you
>>>>> won't be
>>>>> able to write the modifications back.
>>>>>
>>>>> So in addition, I guess we'd need to "flush" all persistent bitmaps
>>>>> (that is, write all modifications back to the file and set the
>>>>> "unchanged" flag (you could also call it "dirty" and then mean the
>>>>> opposite) for each bitmap) not only when the image is closed or
>>>>> invalidated, but also when it is reopened read-only.
>>>>>
>>>>> (block-commit reopens the backing BDS R/W, then writes to them, thus
>>>>> modifying the dirty bitmaps, and finally reopens the BDS as read-only;
>>>>> before that happens, we will have to flush the modified bitmap data.)
>>>> Ok, understand.
>>>>
>>>> We need to consider also setting in_use flag in the image. We _must not_
>>>> write to image with dirty bitmap,
>>>> if in_use flag of this dirty bitmap is not set, as in case of something
>>>> fail we will have image with wrong bitmap with
>>>> unset in_use flag (which looks ok).
>>> Right.
>>>
>>>> I see two ways to handle it:
>>>>
>>>> variant 1:
>>>> 1. readonly field stays as is (see v19, with normal errors, not only
>>>> asserts)
>>>> 2. immediately after reopening r/w we do "reopening bitmaps r/w", i.e.
>>>> set in_use in the image and set BdrvDirtyBitmap.readonly = false
>>>> 3. in reopen_prepare, if reopening r-o do "reopening bitmaps r-o", i.e.
>>>> save them into the image and set BdrvDirtyBitmap.readonly = true
>>> Sounds good, yes.
>>>
>>>> variant 2:
>>>> 1. instead of 'readonly' add 'dirty' field, set dirty to 0 for all
>>>> bitmaps on create
>>>> 2. before write/discard check this field in all related bitmaps, and if
>>>> dirty=0 (and persistent=1), write IN_USE flag into the image first, set
>>>> dirty=1, and only then do write. (if writing IN_USE=1 failed, fail the
>>>> whole write)
>>>> 3. in reopen_prepare, if reopening r-o do "reopening bitmaps r-o", i.e.
>>>> save them into the image and set BdrvDirtyBitmap.dirty = 0
>>> Works, too.
>>>
>>> I think the second variant would the more "efficient" way (because you
>>> only have to flush out dirty dirty bitmaps), but the first one would be
>>> simpler and has the great advantage of not requiring a write to the
>>> image file when you just want to set a bit in the in-memory dirty
>>> bitmap. So I'd personally go for the first variant.
>>
>> Hmm, why not requiring? Both 1 and 2 do write in_use=1, but (1) do this
>> on open/reopen, and (2) before the first write to the image.
>
> Oh, I didn't read the "before write/discard". Yes, if you check it
> before writing, then you won't have to set the flag through
> bdrv_set_dirty().
>
>> "set a bit in the in-memory..." - are you saying about not-persistent
>> dirty bitmaps? In this case, of course, nothing should be written into
>> the image, just set dirty=1.
>
> No, I did mean persistent bitmaps, but bdrv_set_dirty() just sets the
> bit in main memory, of course. It only gets written to the image later
> (on reopen/close/invalidate).
>
There may be some benefit to setting in_use immediately as soon as we
admit that we are willing to tolerate writes to the bitmap. It's a
performance hit, but it may help on-disk consistency.
Or maybe that's a fool's errand? This is a design question we've largely
ignored so far, but it's something that will need investigating sooner
or later.
> Well, your choice. I think both will work. :-)
>
> Max
>
^ permalink raw reply [flat|nested] 56+ messages in thread
* Re: [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap
2017-06-01 22:29 ` John Snow
@ 2017-06-07 12:40 ` Max Reitz
0 siblings, 0 replies; 56+ messages in thread
From: Max Reitz @ 2017-06-07 12:40 UTC (permalink / raw)
To: John Snow, Vladimir Sementsov-Ogievskiy, qemu-block, qemu-devel
Cc: kwolf, famz, armbru, stefanha, pbonzini, den
[-- Attachment #1: Type: text/plain, Size: 2460 bytes --]
On 2017-06-02 00:29, John Snow wrote:
>
>
> On 05/31/2017 09:43 AM, Max Reitz wrote:
>> On 2017-05-30 08:50, Vladimir Sementsov-Ogievskiy wrote:
>>> Thank you for this scenario. Hmm.
>>>
>>> So, as I need guarantee that image and bitmap are unchanged,
>>> bdrv_set_dirty should return error and fail the whole write. Ok?
>>
>> I don't know. That would mean that you couldn't commit to an image that
>> has a persistent auto-loading bitmap, which doesn't seem very nice to me.
>>
>> I'm not quite sure what to do myself. So first I'd definitely want the
>> commit operation to succeed. That means we'd have to automatically make
>> the bitmap non-readonly once we write to it. The "readonly" flag would
>> then be an "unchanged" flag, rather, to signify that the bitmap has not
>> been changed since it was loaded, which means that it does not need to
>> be written back to the image file.
>>
>> Now the issue remains that if you modify a persistent bitmap that is
>> stored in an image file that is opened RO when it's closed, you won't be
>> able to write the modifications back.
>>> So in addition, I guess we'd need to "flush" all persistent bitmaps
>> (that is, write all modifications back to the file and set the
>> "unchanged" flag (you could also call it "dirty" and then mean the
>> opposite) for each bitmap) not only when the image is closed or
>> invalidated, but also when it is reopened read-only.
>>
>
> Makes sense.
>
>> (block-commit reopens the backing BDS R/W, then writes to them, thus
>> modifying the dirty bitmaps, and finally reopens the BDS as read-only;
>> before that happens, we will have to flush the modified bitmap data.)
>>
>
> OK, so it would perhaps be enough to toggle the RO flag on/off when
> nodes get reopened. When they get reopened RO, we'd need to flush at
> that point.
>
> (Right?)
Right.
> Of course, a changed flag makes this a little moot as it is probably
> more flexible; but there is something slightly attractive about the more
> rigid form.
>
> (Hmm, for the purposes of periodic flushing, we may want a changed flag
> anyway...)
I don't mind either way. I like a dirty flag because this is a common
concept (we basically cache the persistent bitmaps in RAM, so it's
natural for them to have a dirty/clean state); but I also like keeping
the read-only flag because it's probably simpler to implement (and makes
just as much sense).
Max
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 498 bytes --]
^ permalink raw reply [flat|nested] 56+ messages in thread
* [Qemu-devel] [PATCH 08/25] qcow2: add bitmaps extension
2017-05-30 8:16 [Qemu-devel] [PATCH v19 " Vladimir Sementsov-Ogievskiy
@ 2017-05-30 8:17 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 56+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2017-05-30 8:17 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: kwolf, mreitz, armbru, eblake, jsnow, famz, den, stefanha,
vsementsov, pbonzini
Add bitmap extension as specified in docs/specs/qcow2.txt.
For now, just mirror extension header into Qcow2 state and check
constraints. Also, calculate refcounts for qcow2 bitmaps, to not break
qemu-img check.
For now, disable image resize if it has bitmaps. It will be fixed later.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
block/Makefile.objs | 2 +-
block/qcow2-bitmap.c | 439 +++++++++++++++++++++++++++++++++++++++++++++++++
block/qcow2-refcount.c | 6 +
block/qcow2.c | 124 +++++++++++++-
block/qcow2.h | 27 +++
5 files changed, 592 insertions(+), 6 deletions(-)
create mode 100644 block/qcow2-bitmap.c
diff --git a/block/Makefile.objs b/block/Makefile.objs
index ea955302c8..9efc6c49ea 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -1,5 +1,5 @@
block-obj-y += raw-format.o qcow.o vdi.o vmdk.o cloop.o bochs.o vpc.o vvfat.o dmg.o
-block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o
+block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o qcow2-bitmap.o
block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o
block-obj-y += qed-check.o
block-obj-y += vhdx.o vhdx-endian.o vhdx-log.o
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
new file mode 100644
index 0000000000..b8e472b3e8
--- /dev/null
+++ b/block/qcow2-bitmap.c
@@ -0,0 +1,439 @@
+/*
+ * Bitmaps for the QCOW version 2 format
+ *
+ * Copyright (c) 2014-2017 Vladimir Sementsov-Ogievskiy
+ *
+ * This file is derived from qcow2-snapshot.c, original copyright:
+ * Copyright (c) 2004-2006 Fabrice Bellard
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+
+#include "block/block_int.h"
+#include "block/qcow2.h"
+
+/* NOTICE: BME here means Bitmaps Extension and used as a namespace for
+ * _internal_ constants. Please do not use this _internal_ abbreviation for
+ * other needs and/or outside of this file. */
+
+/* Bitmap directory entry constraints */
+#define BME_MAX_TABLE_SIZE 0x8000000
+#define BME_MAX_PHYS_SIZE 0x20000000 /* restrict BdrvDirtyBitmap size in RAM */
+#define BME_MAX_GRANULARITY_BITS 31
+#define BME_MIN_GRANULARITY_BITS 9
+#define BME_MAX_NAME_SIZE 1023
+
+/* Bitmap directory entry flags */
+#define BME_RESERVED_FLAGS 0xfffffffcU
+
+/* bits [1, 8] U [56, 63] are reserved */
+#define BME_TABLE_ENTRY_RESERVED_MASK 0xff000000000001feULL
+#define BME_TABLE_ENTRY_OFFSET_MASK 0x00fffffffffffe00ULL
+#define BME_TABLE_ENTRY_FLAG_ALL_ONES (1ULL << 0)
+
+typedef struct QEMU_PACKED Qcow2BitmapDirEntry {
+ /* header is 8 byte aligned */
+ uint64_t bitmap_table_offset;
+
+ uint32_t bitmap_table_size;
+ uint32_t flags;
+
+ uint8_t type;
+ uint8_t granularity_bits;
+ uint16_t name_size;
+ uint32_t extra_data_size;
+ /* extra data follows */
+ /* name follows */
+} Qcow2BitmapDirEntry;
+
+typedef struct Qcow2BitmapTable {
+ uint64_t offset;
+ uint32_t size; /* number of 64bit entries */
+ QSIMPLEQ_ENTRY(Qcow2BitmapTable) entry;
+} Qcow2BitmapTable;
+
+typedef struct Qcow2Bitmap {
+ Qcow2BitmapTable table;
+ uint32_t flags;
+ uint8_t granularity_bits;
+ char *name;
+
+ QSIMPLEQ_ENTRY(Qcow2Bitmap) entry;
+} Qcow2Bitmap;
+typedef QSIMPLEQ_HEAD(Qcow2BitmapList, Qcow2Bitmap) Qcow2BitmapList;
+
+typedef enum BitmapType {
+ BT_DIRTY_TRACKING_BITMAP = 1
+} BitmapType;
+
+static int check_table_entry(uint64_t entry, int cluster_size)
+{
+ uint64_t offset;
+
+ if (entry & BME_TABLE_ENTRY_RESERVED_MASK) {
+ return -EINVAL;
+ }
+
+ offset = entry & BME_TABLE_ENTRY_OFFSET_MASK;
+ if (offset != 0) {
+ /* if offset specified, bit 0 is reserved */
+ if (entry & BME_TABLE_ENTRY_FLAG_ALL_ONES) {
+ return -EINVAL;
+ }
+
+ if (offset % cluster_size != 0) {
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+static int bitmap_table_load(BlockDriverState *bs, Qcow2BitmapTable *tb,
+ uint64_t **bitmap_table)
+{
+ int ret;
+ BDRVQcow2State *s = bs->opaque;
+ uint32_t i;
+ uint64_t *table;
+
+ assert(tb->size != 0);
+ table = g_try_new(uint64_t, tb->size);
+ if (table == NULL) {
+ return -ENOMEM;
+ }
+
+ assert(tb->size <= BME_MAX_TABLE_SIZE);
+ ret = bdrv_pread(bs->file, tb->offset,
+ table, tb->size * sizeof(uint64_t));
+ if (ret < 0) {
+ goto fail;
+ }
+
+ for (i = 0; i < tb->size; ++i) {
+ be64_to_cpus(&table[i]);
+ ret = check_table_entry(table[i], s->cluster_size);
+ if (ret < 0) {
+ goto fail;
+ }
+ }
+
+ *bitmap_table = table;
+ return 0;
+
+fail:
+ g_free(table);
+
+ return ret;
+}
+
+/*
+ * Bitmap List
+ */
+
+/*
+ * Bitmap List private functions
+ * Only Bitmap List knows about bitmap directory structure in Qcow2.
+ */
+
+static inline void bitmap_dir_entry_to_cpu(Qcow2BitmapDirEntry *entry)
+{
+ be64_to_cpus(&entry->bitmap_table_offset);
+ be32_to_cpus(&entry->bitmap_table_size);
+ be32_to_cpus(&entry->flags);
+ be16_to_cpus(&entry->name_size);
+ be32_to_cpus(&entry->extra_data_size);
+}
+
+static inline int calc_dir_entry_size(size_t name_size, size_t extra_data_size)
+{
+ return align_offset(sizeof(Qcow2BitmapDirEntry) +
+ name_size + extra_data_size, 8);
+}
+
+static inline int dir_entry_size(Qcow2BitmapDirEntry *entry)
+{
+ return calc_dir_entry_size(entry->name_size, entry->extra_data_size);
+}
+
+static inline const char *dir_entry_name_field(Qcow2BitmapDirEntry *entry)
+{
+ return (const char *)(entry + 1) + entry->extra_data_size;
+}
+
+static inline char *dir_entry_copy_name(Qcow2BitmapDirEntry *entry)
+{
+ const char *name_field = dir_entry_name_field(entry);
+ return g_strndup(name_field, entry->name_size);
+}
+
+static inline Qcow2BitmapDirEntry *next_dir_entry(Qcow2BitmapDirEntry *entry)
+{
+ return (Qcow2BitmapDirEntry *)((uint8_t *)entry + dir_entry_size(entry));
+}
+
+static int check_dir_entry(BlockDriverState *bs, Qcow2BitmapDirEntry *entry)
+{
+ BDRVQcow2State *s = bs->opaque;
+ uint64_t phys_bitmap_bytes;
+ int64_t len;
+
+ bool fail = (entry->bitmap_table_size == 0) ||
+ (entry->bitmap_table_offset == 0) ||
+ (entry->bitmap_table_offset % s->cluster_size) ||
+ (entry->bitmap_table_size > BME_MAX_TABLE_SIZE) ||
+ (entry->granularity_bits > BME_MAX_GRANULARITY_BITS) ||
+ (entry->granularity_bits < BME_MIN_GRANULARITY_BITS) ||
+ (entry->flags & BME_RESERVED_FLAGS) ||
+ (entry->name_size > BME_MAX_NAME_SIZE) ||
+ (entry->type != BT_DIRTY_TRACKING_BITMAP);
+
+ if (fail) {
+ return -EINVAL;
+ }
+
+ phys_bitmap_bytes = (uint64_t)entry->bitmap_table_size * s->cluster_size;
+ len = bdrv_getlength(bs);
+
+ if (len < 0) {
+ return len;
+ }
+
+ fail = (phys_bitmap_bytes > BME_MAX_PHYS_SIZE) ||
+ (len > ((phys_bitmap_bytes * 8) << entry->granularity_bits));
+
+ return fail ? -EINVAL : 0;
+}
+
+/*
+ * Bitmap List public functions
+ */
+
+static void bitmap_free(Qcow2Bitmap *bm)
+{
+ g_free(bm->name);
+ g_free(bm);
+}
+
+static void bitmap_list_free(Qcow2BitmapList *bm_list)
+{
+ Qcow2Bitmap *bm;
+
+ if (bm_list == NULL) {
+ return;
+ }
+
+ while ((bm = QSIMPLEQ_FIRST(bm_list)) != NULL) {
+ QSIMPLEQ_REMOVE_HEAD(bm_list, entry);
+ bitmap_free(bm);
+ }
+
+ g_free(bm_list);
+}
+
+static Qcow2BitmapList *bitmap_list_new(void)
+{
+ Qcow2BitmapList *bm_list = g_new(Qcow2BitmapList, 1);
+ QSIMPLEQ_INIT(bm_list);
+
+ return bm_list;
+}
+
+/* bitmap_list_load
+ * Get bitmap list from qcow2 image. Actually reads bitmap directory,
+ * checks it and convert to bitmap list.
+ */
+static Qcow2BitmapList *bitmap_list_load(BlockDriverState *bs, uint64_t offset,
+ uint64_t size, Error **errp)
+{
+ int ret;
+ BDRVQcow2State *s = bs->opaque;
+ uint8_t *dir, *dir_end;
+ Qcow2BitmapDirEntry *e;
+ uint32_t nb_dir_entries = 0;
+ Qcow2BitmapList *bm_list = NULL;
+
+ if (size == 0) {
+ error_setg(errp, "Requested bitmap directory size is zero");
+ return NULL;
+ }
+
+ if (size > QCOW2_MAX_BITMAP_DIRECTORY_SIZE) {
+ error_setg(errp, "Requested bitmap directory size is too big");
+ return NULL;
+ }
+
+ dir = g_try_malloc(size);
+ if (dir == NULL) {
+ error_setg(errp, "Failed to allocate space for bitmap directory");
+ return NULL;
+ }
+ dir_end = dir + size;
+
+ ret = bdrv_pread(bs->file, offset, dir, size);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "Failed to read bitmap directory");
+ goto fail;
+ }
+
+ bm_list = bitmap_list_new();
+ for (e = (Qcow2BitmapDirEntry *)dir;
+ e < (Qcow2BitmapDirEntry *)dir_end;
+ e = next_dir_entry(e))
+ {
+ Qcow2Bitmap *bm;
+
+ if ((uint8_t *)(e + 1) > dir_end) {
+ goto broken_dir;
+ }
+
+ if (++nb_dir_entries > s->nb_bitmaps) {
+ error_setg(errp, "More bitmaps found than specified in header"
+ " extension");
+ goto fail;
+ }
+ bitmap_dir_entry_to_cpu(e);
+
+ if ((uint8_t *)next_dir_entry(e) > dir_end) {
+ goto broken_dir;
+ }
+
+ if (e->extra_data_size != 0) {
+ error_setg(errp, "Bitmap extra data is not supported");
+ goto fail;
+ }
+
+ ret = check_dir_entry(bs, e);
+ if (ret < 0) {
+ error_setg(errp, "Bitmap '%.*s' doesn't satisfy the constraints",
+ e->name_size, dir_entry_name_field(e));
+ goto fail;
+ }
+
+ bm = g_new(Qcow2Bitmap, 1);
+ bm->table.offset = e->bitmap_table_offset;
+ bm->table.size = e->bitmap_table_size;
+ bm->flags = e->flags;
+ bm->granularity_bits = e->granularity_bits;
+ bm->name = dir_entry_copy_name(e);
+ QSIMPLEQ_INSERT_TAIL(bm_list, bm, entry);
+ }
+
+ if (nb_dir_entries != s->nb_bitmaps) {
+ error_setg(errp, "Less bitmaps found than specified in header"
+ " extension");
+ goto fail;
+ }
+
+ if ((uint8_t *)e != dir_end) {
+ goto broken_dir;
+ }
+
+ g_free(dir);
+ return bm_list;
+
+broken_dir:
+ ret = -EINVAL;
+ error_setg(errp, "Broken bitmap directory");
+
+fail:
+ g_free(dir);
+ bitmap_list_free(bm_list);
+
+ return NULL;
+}
+
+int qcow2_check_bitmaps_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
+ void **refcount_table,
+ int64_t *refcount_table_size)
+{
+ int ret;
+ BDRVQcow2State *s = bs->opaque;
+ Qcow2BitmapList *bm_list;
+ Qcow2Bitmap *bm;
+
+ if (s->nb_bitmaps == 0) {
+ return 0;
+ }
+
+ ret = qcow2_inc_refcounts_imrt(bs, res, refcount_table, refcount_table_size,
+ s->bitmap_directory_offset,
+ s->bitmap_directory_size);
+ if (ret < 0) {
+ return ret;
+ }
+
+ bm_list = bitmap_list_load(bs, s->bitmap_directory_offset,
+ s->bitmap_directory_size, NULL);
+ if (bm_list == NULL) {
+ res->corruptions++;
+ return -EINVAL;
+ }
+
+ QSIMPLEQ_FOREACH(bm, bm_list, entry) {
+ uint64_t *bitmap_table = NULL;
+ int i;
+
+ ret = qcow2_inc_refcounts_imrt(bs, res,
+ refcount_table, refcount_table_size,
+ bm->table.offset,
+ bm->table.size * sizeof(uint64_t));
+ if (ret < 0) {
+ goto out;
+ }
+
+ ret = bitmap_table_load(bs, &bm->table, &bitmap_table);
+ if (ret < 0) {
+ res->corruptions++;
+ goto out;
+ }
+
+ for (i = 0; i < bm->table.size; ++i) {
+ uint64_t entry = bitmap_table[i];
+ uint64_t offset = entry & BME_TABLE_ENTRY_OFFSET_MASK;
+
+ if (check_table_entry(entry, s->cluster_size) < 0) {
+ res->corruptions++;
+ continue;
+ }
+
+ if (offset == 0) {
+ continue;
+ }
+
+ ret = qcow2_inc_refcounts_imrt(bs, res,
+ refcount_table, refcount_table_size,
+ offset, s->cluster_size);
+ if (ret < 0) {
+ g_free(bitmap_table);
+ goto out;
+ }
+ }
+
+ g_free(bitmap_table);
+ }
+
+out:
+ bitmap_list_free(bm_list);
+
+ return ret;
+}
diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c
index d7066c875b..bdb4cb3a6d 100644
--- a/block/qcow2-refcount.c
+++ b/block/qcow2-refcount.c
@@ -1859,6 +1859,12 @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
return ret;
}
+ /* bitmaps */
+ ret = qcow2_check_bitmaps_refcounts(bs, res, refcount_table, nb_clusters);
+ if (ret < 0) {
+ return ret;
+ }
+
return check_refblocks(bs, res, fix, rebuild, refcount_table, nb_clusters);
}
diff --git a/block/qcow2.c b/block/qcow2.c
index a8d61f0981..28a16789af 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -63,6 +63,7 @@ typedef struct {
#define QCOW2_EXT_MAGIC_END 0
#define QCOW2_EXT_MAGIC_BACKING_FORMAT 0xE2792ACA
#define QCOW2_EXT_MAGIC_FEATURE_TABLE 0x6803f857
+#define QCOW2_EXT_MAGIC_BITMAPS 0x23852875
static int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename)
{
@@ -86,12 +87,17 @@ static int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename)
*/
static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset,
uint64_t end_offset, void **p_feature_table,
- Error **errp)
+ bool *need_update_header, Error **errp)
{
BDRVQcow2State *s = bs->opaque;
QCowExtension ext;
uint64_t offset;
int ret;
+ Qcow2BitmapHeaderExt bitmaps_ext;
+
+ if (need_update_header != NULL) {
+ *need_update_header = false;
+ }
#ifdef DEBUG_EXT
printf("qcow2_read_extensions: start=%ld end=%ld\n", start_offset, end_offset);
@@ -162,6 +168,85 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset,
}
break;
+ case QCOW2_EXT_MAGIC_BITMAPS:
+ if (ext.len != sizeof(bitmaps_ext)) {
+ error_setg_errno(errp, -ret, "bitmaps_ext: "
+ "Invalid extension length");
+ return -EINVAL;
+ }
+
+ if (!(s->autoclear_features & QCOW2_AUTOCLEAR_BITMAPS)) {
+ error_report("WARNING: a program lacking bitmap support "
+ "modified this file, so all bitmaps are now "
+ "considered inconsistent. Some clusters may be "
+ "leaked, run 'qemu-img check -r' on the image "
+ "file to fix.");
+ if (need_update_header != NULL) {
+ /* Updating is needed to drop invalid bitmap extension. */
+ *need_update_header = true;
+ }
+ break;
+ }
+
+ ret = bdrv_pread(bs->file, offset, &bitmaps_ext, ext.len);
+ if (ret < 0) {
+ error_setg_errno(errp, -ret, "bitmaps_ext: "
+ "Could not read ext header");
+ return ret;
+ }
+
+ if (bitmaps_ext.reserved32 != 0) {
+ error_setg_errno(errp, -ret, "bitmaps_ext: "
+ "Reserved field is not zero");
+ return -EINVAL;
+ }
+
+ be32_to_cpus(&bitmaps_ext.nb_bitmaps);
+ be64_to_cpus(&bitmaps_ext.bitmap_directory_size);
+ be64_to_cpus(&bitmaps_ext.bitmap_directory_offset);
+
+ if (bitmaps_ext.nb_bitmaps > QCOW2_MAX_BITMAPS) {
+ error_setg(errp,
+ "bitmaps_ext: Image has %" PRIu32 " bitmaps, "
+ "exceeding the QEMU supported maximum of %d",
+ bitmaps_ext.nb_bitmaps, QCOW2_MAX_BITMAPS);
+ return -EINVAL;
+ }
+
+ if (bitmaps_ext.nb_bitmaps == 0) {
+ error_setg(errp, "found bitmaps extension with zero bitmaps");
+ return -EINVAL;
+ }
+
+ if (bitmaps_ext.bitmap_directory_offset & (s->cluster_size - 1)) {
+ error_setg(errp, "bitmaps_ext: "
+ "invalid bitmap directory offset");
+ return -EINVAL;
+ }
+
+ if (bitmaps_ext.bitmap_directory_size >
+ QCOW2_MAX_BITMAP_DIRECTORY_SIZE) {
+ error_setg(errp, "bitmaps_ext: "
+ "bitmap directory size (%" PRIu64 ") exceeds "
+ "the maximum supported size (%d)",
+ bitmaps_ext.bitmap_directory_size,
+ QCOW2_MAX_BITMAP_DIRECTORY_SIZE);
+ return -EINVAL;
+ }
+
+ s->nb_bitmaps = bitmaps_ext.nb_bitmaps;
+ s->bitmap_directory_offset =
+ bitmaps_ext.bitmap_directory_offset;
+ s->bitmap_directory_size =
+ bitmaps_ext.bitmap_directory_size;
+
+#ifdef DEBUG_EXT
+ printf("Qcow2: Got bitmaps extension: "
+ "offset=%" PRIu64 " nb_bitmaps=%" PRIu32 "\n",
+ s->bitmap_directory_offset, s->nb_bitmaps);
+#endif
+ break;
+
default:
/* unknown magic - save it in case we need to rewrite the header */
{
@@ -824,6 +909,7 @@ static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags,
Error *local_err = NULL;
uint64_t ext_end;
uint64_t l1_vm_state_index;
+ bool update_header = false;
ret = bdrv_pread(bs->file, 0, &header, sizeof(header));
if (ret < 0) {
@@ -929,7 +1015,7 @@ static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags,
if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
void *feature_table = NULL;
qcow2_read_extensions(bs, header.header_length, ext_end,
- &feature_table, NULL);
+ &feature_table, NULL, NULL);
report_unsupported_feature(errp, feature_table,
s->incompatible_features &
~QCOW2_INCOMPAT_MASK);
@@ -1116,7 +1202,7 @@ static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags,
/* read qcow2 extensions */
if (qcow2_read_extensions(bs, header.header_length, ext_end, NULL,
- &local_err)) {
+ &update_header, &local_err)) {
error_propagate(errp, local_err);
ret = -EINVAL;
goto fail;
@@ -1152,8 +1238,10 @@ static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags,
}
/* Clear unknown autoclear feature bits */
- if (!bs->read_only && !(flags & BDRV_O_INACTIVE) && s->autoclear_features) {
- s->autoclear_features = 0;
+ update_header |= s->autoclear_features & ~QCOW2_AUTOCLEAR_MASK;
+
+ if (update_header && !bs->read_only && !(flags & BDRV_O_INACTIVE)) {
+ s->autoclear_features &= QCOW2_AUTOCLEAR_MASK;
ret = qcow2_update_header(bs);
if (ret < 0) {
error_setg_errno(errp, -ret, "Could not update qcow2 header");
@@ -1966,6 +2054,25 @@ int qcow2_update_header(BlockDriverState *bs)
buflen -= ret;
}
+ /* Bitmap extension */
+ if (s->nb_bitmaps > 0) {
+ Qcow2BitmapHeaderExt bitmaps_header = {
+ .nb_bitmaps = cpu_to_be32(s->nb_bitmaps),
+ .bitmap_directory_size =
+ cpu_to_be64(s->bitmap_directory_size),
+ .bitmap_directory_offset =
+ cpu_to_be64(s->bitmap_directory_offset)
+ };
+ ret = header_ext_add(buf, QCOW2_EXT_MAGIC_BITMAPS,
+ &bitmaps_header, sizeof(bitmaps_header),
+ buflen);
+ if (ret < 0) {
+ goto fail;
+ }
+ buf += ret;
+ buflen -= ret;
+ }
+
/* Keep unknown header extensions */
QLIST_FOREACH(uext, &s->unknown_header_ext, next) {
ret = header_ext_add(buf, uext->magic, uext->data, uext->len, buflen);
@@ -2558,6 +2665,13 @@ static int qcow2_truncate(BlockDriverState *bs, int64_t offset, Error **errp)
return -ENOTSUP;
}
+ /* cannot proceed if image has bitmaps */
+ if (s->nb_bitmaps) {
+ /* TODO: resize bitmaps in the image */
+ error_setg(errp, "Can't resize an image which has bitmaps");
+ return -ENOTSUP;
+ }
+
/* shrinking is currently not supported */
if (offset < bs->total_sectors * 512) {
error_setg(errp, "qcow2 doesn't support shrinking images yet");
diff --git a/block/qcow2.h b/block/qcow2.h
index 41f8160629..f4a1f4cd78 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -52,6 +52,10 @@
* space for snapshot names and IDs */
#define QCOW_MAX_SNAPSHOTS_SIZE (1024 * QCOW_MAX_SNAPSHOTS)
+/* Bitmap header extension constraints */
+#define QCOW2_MAX_BITMAPS 65535
+#define QCOW2_MAX_BITMAP_DIRECTORY_SIZE (1024 * QCOW2_MAX_BITMAPS)
+
/* indicate that the refcount of the referenced cluster is exactly one. */
#define QCOW_OFLAG_COPIED (1ULL << 63)
/* indicate that the cluster is compressed (they never have the copied flag) */
@@ -195,6 +199,14 @@ enum {
QCOW2_COMPAT_FEAT_MASK = QCOW2_COMPAT_LAZY_REFCOUNTS,
};
+/* Autoclear feature bits */
+enum {
+ QCOW2_AUTOCLEAR_BITMAPS_BITNR = 0,
+ QCOW2_AUTOCLEAR_BITMAPS = 1 << QCOW2_AUTOCLEAR_BITMAPS_BITNR,
+
+ QCOW2_AUTOCLEAR_MASK = QCOW2_AUTOCLEAR_BITMAPS,
+};
+
enum qcow2_discard_type {
QCOW2_DISCARD_NEVER = 0,
QCOW2_DISCARD_ALWAYS,
@@ -222,6 +234,13 @@ typedef uint64_t Qcow2GetRefcountFunc(const void *refcount_array,
typedef void Qcow2SetRefcountFunc(void *refcount_array,
uint64_t index, uint64_t value);
+typedef struct Qcow2BitmapHeaderExt {
+ uint32_t nb_bitmaps;
+ uint32_t reserved32;
+ uint64_t bitmap_directory_size;
+ uint64_t bitmap_directory_offset;
+} QEMU_PACKED Qcow2BitmapHeaderExt;
+
typedef struct BDRVQcow2State {
int cluster_bits;
int cluster_size;
@@ -264,6 +283,10 @@ typedef struct BDRVQcow2State {
unsigned int nb_snapshots;
QCowSnapshot *snapshots;
+ uint32_t nb_bitmaps;
+ uint64_t bitmap_directory_size;
+ uint64_t bitmap_directory_offset;
+
int flags;
int qcow_version;
bool use_lazy_refcounts;
@@ -602,4 +625,8 @@ int qcow2_cache_get_empty(BlockDriverState *bs, Qcow2Cache *c, uint64_t offset,
void **table);
void qcow2_cache_put(BlockDriverState *bs, Qcow2Cache *c, void **table);
+/* qcow2-bitmap.c functions */
+int qcow2_check_bitmaps_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
+ void **refcount_table,
+ int64_t *refcount_table_size);
#endif
--
2.11.1
^ permalink raw reply related [flat|nested] 56+ messages in thread
end of thread, other threads:[~2017-06-07 12:40 UTC | newest]
Thread overview: 56+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-05-03 12:25 [Qemu-devel] [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 01/25] specs/qcow2: fix bitmap granularity qemu-specific note Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 02/25] specs/qcow2: do not use wording 'bitmap header' Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 03/25] hbitmap: improve dirty iter Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 04/25] tests: add hbitmap iter test Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 05/25] block: fix bdrv_dirty_bitmap_granularity signature Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 06/25] block/dirty-bitmap: add deserialize_ones func Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 07/25] qcow2-refcount: rename inc_refcounts() and make it public Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 08/25] qcow2: add bitmaps extension Vladimir Sementsov-Ogievskiy
2017-05-29 15:34 ` Max Reitz
2017-05-03 12:25 ` [Qemu-devel] [PATCH 09/25] block/dirty-bitmap: add readonly field to BdrvDirtyBitmap Vladimir Sementsov-Ogievskiy
2017-05-19 23:02 ` John Snow
2017-05-25 9:34 ` Vladimir Sementsov-Ogievskiy
2017-05-31 22:58 ` John Snow
2017-06-01 7:23 ` Sementsov-Ogievskiy Vladimir
2017-05-29 15:49 ` Max Reitz
2017-05-29 18:35 ` Max Reitz
2017-05-30 6:50 ` Vladimir Sementsov-Ogievskiy
2017-05-30 7:31 ` Vladimir Sementsov-Ogievskiy
2017-05-30 7:47 ` Vladimir Sementsov-Ogievskiy
2017-05-31 13:43 ` Max Reitz
2017-05-31 14:29 ` Vladimir Sementsov-Ogievskiy
2017-05-31 14:44 ` Max Reitz
2017-05-31 15:05 ` Vladimir Sementsov-Ogievskiy
2017-05-31 15:53 ` Max Reitz
2017-06-01 22:35 ` John Snow
2017-06-01 22:29 ` John Snow
2017-06-07 12:40 ` Max Reitz
2017-05-03 12:25 ` [Qemu-devel] [PATCH 10/25] qcow2: autoloading dirty bitmaps Vladimir Sementsov-Ogievskiy
2017-05-29 16:17 ` Max Reitz
2017-05-03 12:25 ` [Qemu-devel] [PATCH 11/25] block/dirty-bitmap: add autoload field to BdrvDirtyBitmap Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 12/25] block: bdrv_close: release bitmaps after drv->bdrv_close Vladimir Sementsov-Ogievskiy
2017-05-29 16:34 ` Max Reitz
2017-05-03 12:25 ` [Qemu-devel] [PATCH 13/25] block: introduce persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
2017-05-29 16:49 ` Max Reitz
2017-05-03 12:25 ` [Qemu-devel] [PATCH 14/25] block/dirty-bitmap: add bdrv_dirty_bitmap_next() Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 15/25] qcow2: add persistent dirty bitmaps support Vladimir Sementsov-Ogievskiy
2017-05-29 17:07 ` Max Reitz
2017-05-03 12:25 ` [Qemu-devel] [PATCH 16/25] block: add bdrv_can_store_new_dirty_bitmap Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 17/25] qcow2: add .bdrv_can_store_new_dirty_bitmap Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 18/25] qmp: add persistent flag to block-dirty-bitmap-add Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 19/25] qmp: add autoload parameter " Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 20/25] qmp: add x-debug-block-dirty-bitmap-sha256 Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 21/25] iotests: test qcow2 persistent dirty bitmap Vladimir Sementsov-Ogievskiy
2017-05-29 17:31 ` Max Reitz
2017-05-03 12:25 ` [Qemu-devel] [PATCH 22/25] block/dirty-bitmap: add bdrv_remove_persistent_dirty_bitmap Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 23/25] qcow2: add .bdrv_remove_persistent_dirty_bitmap Vladimir Sementsov-Ogievskiy
2017-05-03 12:25 ` [Qemu-devel] [PATCH 24/25] qmp: block-dirty-bitmap-remove: remove persistent Vladimir Sementsov-Ogievskiy
2017-05-29 17:38 ` Max Reitz
2017-05-03 12:25 ` [Qemu-devel] [PATCH 25/25] block: release persistent bitmaps on inactivate Vladimir Sementsov-Ogievskiy
2017-05-29 17:54 ` Max Reitz
2017-05-30 6:30 ` Vladimir Sementsov-Ogievskiy
2017-05-31 13:37 ` Max Reitz
2017-05-15 9:51 ` [Qemu-devel] ping Re: [PATCH v18 00/25] qcow2: persistent dirty bitmaps Vladimir Sementsov-Ogievskiy
2017-05-27 9:05 ` [Qemu-devel] ping2 " Vladimir Sementsov-Ogievskiy
2017-05-30 8:16 [Qemu-devel] [PATCH v19 " Vladimir Sementsov-Ogievskiy
2017-05-30 8:17 ` [Qemu-devel] [PATCH 08/25] qcow2: add bitmaps extension Vladimir Sementsov-Ogievskiy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.