* [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based
@ 2017-08-30 21:05 Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 01/18] block: Make bdrv_img_create() size selection easier to read Eric Blake
` (18 more replies)
0 siblings, 19 replies; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, vsementsov, qemu-block
There are patches floating around to add NBD_CMD_BLOCK_STATUS,
but NBD wants to report status on byte granularity (even if the
reporting will probably be naturally aligned to sectors or even
much higher levels). I've therefore started the task of
converting our block status code to report at a byte granularity
rather than sectors.
Now that 2.11 is open, I'm rebasing/reposting the remaining patches.
The overall conversion currently looks like:
part 1: bdrv_is_allocated (merged in 2.10, commit 51b0a488)
part 2: dirty-bitmap (this series, v5 was here [1])
part 3: bdrv_get_block_status (v3 is posted [2] and is mostly reviewed, but
needs a rebase)
part 4: .bdrv_co_block_status (v2 is posted [3], but needs a rebase)
Available as a tag at:
git fetch git://repo.or.cz/qemu/ericb.git nbd-byte-dirty-v6
[1] https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg03512.html
[2] https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg03853.html
[3] https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg04370.html
Diff from v5:
- add another patch (more for ease of bookkeeping, as it was previously
posted independently)
- drop bug fixes that were hoisted into 2.10 (v5 1/18, plus 14/18)
001/18:[down] 'block: Make bdrv_img_create() size selection easier to read'
002/18:[----] [--] 'hbitmap: Rename serialization_granularity to serialization_align'
003/18:[----] [--] 'qcow2: Ensure bitmap serialization is aligned'
004/18:[----] [--] 'dirty-bitmap: Drop unused functions'
005/18:[----] [--] 'dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes'
006/18:[----] [--] 'dirty-bitmap: Change bdrv_dirty_bitmap_*serialize*() to take bytes'
007/18:[----] [--] 'qcow2: Switch sectors_covered_by_bitmap_cluster() to byte-based'
008/18:[----] [--] 'dirty-bitmap: Set iterator start by offset, not sector'
009/18:[----] [--] 'dirty-bitmap: Change bdrv_dirty_iter_next() to report byte offset'
010/18:[----] [--] 'dirty-bitmap: Change bdrv_get_dirty_count() to report bytes'
011/18:[----] [--] 'dirty-bitmap: Change bdrv_get_dirty_locked() to take bytes'
012/18:[----] [--] 'dirty-bitmap: Change bdrv_[re]set_dirty_bitmap() to use bytes'
013/18:[----] [--] 'mirror: Switch mirror_dirty_init() to byte-based iteration'
014/18:[0004] [FC] 'qcow2: Switch qcow2_measure() to byte-based iteration'
015/18:[----] [--] 'qcow2: Switch load_bitmap_data() to byte-based iteration'
016/18:[----] [--] 'qcow2: Switch store_bitmap_data() to byte-based iteration'
017/18:[----] [--] 'dirty-bitmap: Switch bdrv_set_dirty() to bytes'
018/18:[----] [--] 'dirty-bitmap: Convert internal hbitmap size/granularity'
Eric Blake (18):
block: Make bdrv_img_create() size selection easier to read
hbitmap: Rename serialization_granularity to serialization_align
qcow2: Ensure bitmap serialization is aligned
dirty-bitmap: Drop unused functions
dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes
dirty-bitmap: Change bdrv_dirty_bitmap_*serialize*() to take bytes
qcow2: Switch sectors_covered_by_bitmap_cluster() to byte-based
dirty-bitmap: Set iterator start by offset, not sector
dirty-bitmap: Change bdrv_dirty_iter_next() to report byte offset
dirty-bitmap: Change bdrv_get_dirty_count() to report bytes
dirty-bitmap: Change bdrv_get_dirty_locked() to take bytes
dirty-bitmap: Change bdrv_[re]set_dirty_bitmap() to use bytes
mirror: Switch mirror_dirty_init() to byte-based iteration
qcow2: Switch qcow2_measure() to byte-based iteration
qcow2: Switch load_bitmap_data() to byte-based iteration
qcow2: Switch store_bitmap_data() to byte-based iteration
dirty-bitmap: Switch bdrv_set_dirty() to bytes
dirty-bitmap: Convert internal hbitmap size/granularity
include/block/block_int.h | 2 +-
include/block/dirty-bitmap.h | 41 +++++---------
include/qemu/hbitmap.h | 8 +--
block/io.c | 6 +-
block.c | 2 +-
block/backup.c | 7 +--
block/dirty-bitmap.c | 130 ++++++++++++++-----------------------------
block/mirror.c | 76 +++++++++++--------------
block/qcow2-bitmap.c | 57 +++++++++----------
block/qcow2.c | 22 ++++----
migration/block.c | 12 ++--
tests/test-hbitmap.c | 10 ++--
util/hbitmap.c | 8 +--
13 files changed, 154 insertions(+), 227 deletions(-)
--
2.13.5
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 01/18] block: Make bdrv_img_create() size selection easier to read
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 02/18] hbitmap: Rename serialization_granularity to serialization_align Eric Blake
` (17 subsequent siblings)
18 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, vsementsov, qemu-block, Max Reitz
All callers of bdrv_img_create() pass in a size, or -1 to read the
size from the backing file. We then set that size as the QemuOpt
default, which means we will reuse that default rather than the
final parameter to qemu_opt_get_size() several lines later. But
it is rather confusing to read subsequent checks of 'size == -1'
when it looks (without seeing the full context) like size defaults
to 0; it also doesn't help that a size of 0 is valid (for some
formats).
Rework the logic to make things more legible.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
v6: Combine into a series rather than being a standalone patch (more for
ease of tracking than for being on topic)
---
block.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/block.c b/block.c
index 3308814bba..3f84d141a7 100644
--- a/block.c
+++ b/block.c
@@ -4392,7 +4392,7 @@ void bdrv_img_create(const char *filename, const char *fmt,
/* The size for the image must always be specified, unless we have a backing
* file and we have not been forbidden from opening it. */
- size = qemu_opt_get_size(opts, BLOCK_OPT_SIZE, 0);
+ size = qemu_opt_get_size(opts, BLOCK_OPT_SIZE, img_size);
if (backing_file && !(flags & BDRV_O_NO_BACKING)) {
BlockDriverState *bs;
char *full_backing = g_new0(char, PATH_MAX);
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 02/18] hbitmap: Rename serialization_granularity to serialization_align
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 01/18] block: Make bdrv_img_create() size selection easier to read Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 03/18] qcow2: Ensure bitmap serialization is aligned Eric Blake
` (16 subsequent siblings)
18 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, vsementsov, qemu-block, Fam Zheng, Max Reitz
The only client of hbitmap_serialization_granularity() is dirty-bitmap's
bdrv_dirty_bitmap_serialization_align(). Keeping the two names consistent
is worthwhile, and the shorter name is more representative of what the
function returns (the required alignment to be used for start/count of
other serialization functions, where violating the alignment causes
assertion failures).
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
v5: no change
v4: new patch
---
include/qemu/hbitmap.h | 8 ++++----
block/dirty-bitmap.c | 2 +-
tests/test-hbitmap.c | 10 +++++-----
util/hbitmap.c | 8 ++++----
4 files changed, 14 insertions(+), 14 deletions(-)
diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
index d3a74a21fc..81e78043d1 100644
--- a/include/qemu/hbitmap.h
+++ b/include/qemu/hbitmap.h
@@ -159,16 +159,16 @@ bool hbitmap_get(const HBitmap *hb, uint64_t item);
bool hbitmap_is_serializable(const HBitmap *hb);
/**
- * hbitmap_serialization_granularity:
+ * hbitmap_serialization_align:
* @hb: HBitmap to operate on.
*
- * Granularity of serialization chunks, used by other serialization functions.
- * For every chunk:
+ * Required alignment of serialization chunks, used by other serialization
+ * functions. For every chunk:
* 1. Chunk start should be aligned to this granularity.
* 2. Chunk size should be aligned too, except for last chunk (for which
* start + count == hb->size)
*/
-uint64_t hbitmap_serialization_granularity(const HBitmap *hb);
+uint64_t hbitmap_serialization_align(const HBitmap *hb);
/**
* hbitmap_serialization_size:
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 30462d4f9a..0490ca3aff 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -617,7 +617,7 @@ uint64_t bdrv_dirty_bitmap_serialization_size(const BdrvDirtyBitmap *bitmap,
uint64_t bdrv_dirty_bitmap_serialization_align(const BdrvDirtyBitmap *bitmap)
{
- return hbitmap_serialization_granularity(bitmap->bitmap);
+ return hbitmap_serialization_align(bitmap->bitmap);
}
void bdrv_dirty_bitmap_serialize_part(const BdrvDirtyBitmap *bitmap,
diff --git a/tests/test-hbitmap.c b/tests/test-hbitmap.c
index 1acb353889..af41642346 100644
--- a/tests/test-hbitmap.c
+++ b/tests/test-hbitmap.c
@@ -738,15 +738,15 @@ static void test_hbitmap_meta_one(TestHBitmapData *data, const void *unused)
}
}
-static void test_hbitmap_serialize_granularity(TestHBitmapData *data,
- const void *unused)
+static void test_hbitmap_serialize_align(TestHBitmapData *data,
+ const void *unused)
{
int r;
hbitmap_test_init(data, L3 * 2, 3);
g_assert(hbitmap_is_serializable(data->hb));
- r = hbitmap_serialization_granularity(data->hb);
+ r = hbitmap_serialization_align(data->hb);
g_assert_cmpint(r, ==, 64 << 3);
}
@@ -974,8 +974,8 @@ int main(int argc, char **argv)
hbitmap_test_add("/hbitmap/meta/word", test_hbitmap_meta_word);
hbitmap_test_add("/hbitmap/meta/sector", test_hbitmap_meta_sector);
- hbitmap_test_add("/hbitmap/serialize/granularity",
- test_hbitmap_serialize_granularity);
+ hbitmap_test_add("/hbitmap/serialize/align",
+ test_hbitmap_serialize_align);
hbitmap_test_add("/hbitmap/serialize/basic",
test_hbitmap_serialize_basic);
hbitmap_test_add("/hbitmap/serialize/part",
diff --git a/util/hbitmap.c b/util/hbitmap.c
index 21535cc90b..2f9d0fdbd0 100644
--- a/util/hbitmap.c
+++ b/util/hbitmap.c
@@ -413,14 +413,14 @@ bool hbitmap_is_serializable(const HBitmap *hb)
{
/* Every serialized chunk must be aligned to 64 bits so that endianness
* requirements can be fulfilled on both 64 bit and 32 bit hosts.
- * We have hbitmap_serialization_granularity() which converts this
+ * We have hbitmap_serialization_align() which converts this
* alignment requirement from bitmap bits to items covered (e.g. sectors).
* That value is:
* 64 << hb->granularity
* Since this value must not exceed UINT64_MAX, hb->granularity must be
* less than 58 (== 64 - 6, where 6 is ld(64), i.e. 1 << 6 == 64).
*
- * In order for hbitmap_serialization_granularity() to always return a
+ * In order for hbitmap_serialization_align() to always return a
* meaningful value, bitmaps that are to be serialized must have a
* granularity of less than 58. */
@@ -437,7 +437,7 @@ bool hbitmap_get(const HBitmap *hb, uint64_t item)
return (hb->levels[HBITMAP_LEVELS - 1][pos >> BITS_PER_LEVEL] & bit) != 0;
}
-uint64_t hbitmap_serialization_granularity(const HBitmap *hb)
+uint64_t hbitmap_serialization_align(const HBitmap *hb)
{
assert(hbitmap_is_serializable(hb));
@@ -454,7 +454,7 @@ static void serialization_chunk(const HBitmap *hb,
unsigned long **first_el, uint64_t *el_count)
{
uint64_t last = start + count - 1;
- uint64_t gran = hbitmap_serialization_granularity(hb);
+ uint64_t gran = hbitmap_serialization_align(hb);
assert((start & (gran - 1)) == 0);
assert((last >> hb->granularity) < hb->size);
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 03/18] qcow2: Ensure bitmap serialization is aligned
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 01/18] block: Make bdrv_img_create() size selection easier to read Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 02/18] hbitmap: Rename serialization_granularity to serialization_align Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 04/18] dirty-bitmap: Drop unused functions Eric Blake
` (15 subsequent siblings)
18 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, vsementsov, qemu-block, Max Reitz
When subdividing a bitmap serialization, the code in hbitmap.c
enforces that start/count parameters are aligned (except that
count can end early at end-of-bitmap). We exposed this required
alignment through bdrv_dirty_bitmap_serialization_align(), but
forgot to actually check that we comply with it.
Fortunately, qcow2 is never dividing bitmap serialization smaller
than one cluster (which is a minimum of 512 bytes); so we are
always compliant with the serialization alignment (which insists
that we partition at least 64 bits per chunk) because we are doing
at least 4k bits per chunk.
Still, it's safer to add an assertion (for the unlikely case that
we'd ever support a cluster smaller than 512 bytes, or if the
hbitmap implementation changes what it considers to be aligned),
rather than leaving bdrv_dirty_bitmap_serialization_align()
without a caller.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
v5: no change
v4: new patch
---
block/qcow2-bitmap.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index e8d3bdbd6e..b3ee4c794a 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -274,10 +274,13 @@ static int free_bitmap_clusters(BlockDriverState *bs, Qcow2BitmapTable *tb)
static uint64_t sectors_covered_by_bitmap_cluster(const BDRVQcow2State *s,
const BdrvDirtyBitmap *bitmap)
{
- uint32_t sector_granularity =
+ uint64_t sector_granularity =
bdrv_dirty_bitmap_granularity(bitmap) >> BDRV_SECTOR_BITS;
+ uint64_t sbc = sector_granularity * (s->cluster_size << 3);
- return (uint64_t)sector_granularity * (s->cluster_size << 3);
+ assert(QEMU_IS_ALIGNED(sbc,
+ bdrv_dirty_bitmap_serialization_align(bitmap)));
+ return sbc;
}
/* load_bitmap_data
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 04/18] dirty-bitmap: Drop unused functions
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
` (2 preceding siblings ...)
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 03/18] qcow2: Ensure bitmap serialization is aligned Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 05/18] dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes Eric Blake
` (14 subsequent siblings)
18 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, vsementsov, qemu-block, Fam Zheng, Max Reitz
We had several functions that no one is currently using, and which
use sector-based interfaces. I'm trying to convert towards byte-based
interfaces, so it's easier to just drop the unused functions:
bdrv_dirty_bitmap_get_meta
bdrv_dirty_bitmap_get_meta_locked
bdrv_dirty_bitmap_reset_meta
bdrv_dirty_bitmap_meta_granularity
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
v5: no change
v4: rebase to Vladimir's persistent bitmaps (bdrv_dirty_bitmap_size now
in use), dropped R-b
v3: rebase to upstream changes (bdrv_dirty_bitmap_get_meta_locked was
added in b64bd51e with no clients), kept R-b
v2: tweak commit message based on review, no code change
---
include/block/dirty-bitmap.h | 10 ----------
block/dirty-bitmap.c | 44 --------------------------------------------
2 files changed, 54 deletions(-)
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index a79a58d2c3..8fd842eac9 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -34,7 +34,6 @@ void bdrv_enable_dirty_bitmap(BdrvDirtyBitmap *bitmap);
BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs);
uint32_t bdrv_get_default_bitmap_granularity(BlockDriverState *bs);
uint32_t bdrv_dirty_bitmap_granularity(const BdrvDirtyBitmap *bitmap);
-uint32_t bdrv_dirty_bitmap_meta_granularity(BdrvDirtyBitmap *bitmap);
bool bdrv_dirty_bitmap_enabled(BdrvDirtyBitmap *bitmap);
bool bdrv_dirty_bitmap_frozen(BdrvDirtyBitmap *bitmap);
const char *bdrv_dirty_bitmap_name(const BdrvDirtyBitmap *bitmap);
@@ -44,15 +43,6 @@ void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
int64_t cur_sector, int64_t nr_sectors);
void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
int64_t cur_sector, int64_t nr_sectors);
-int bdrv_dirty_bitmap_get_meta(BlockDriverState *bs,
- BdrvDirtyBitmap *bitmap, int64_t sector,
- int nb_sectors);
-int bdrv_dirty_bitmap_get_meta_locked(BlockDriverState *bs,
- BdrvDirtyBitmap *bitmap, int64_t sector,
- int nb_sectors);
-void bdrv_dirty_bitmap_reset_meta(BlockDriverState *bs,
- BdrvDirtyBitmap *bitmap, int64_t sector,
- int nb_sectors);
BdrvDirtyBitmapIter *bdrv_dirty_meta_iter_new(BdrvDirtyBitmap *bitmap);
BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap,
uint64_t first_sector);
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 0490ca3aff..42a55e4a4b 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -173,45 +173,6 @@ void bdrv_release_meta_dirty_bitmap(BdrvDirtyBitmap *bitmap)
qemu_mutex_unlock(bitmap->mutex);
}
-int bdrv_dirty_bitmap_get_meta_locked(BlockDriverState *bs,
- BdrvDirtyBitmap *bitmap, int64_t sector,
- int nb_sectors)
-{
- uint64_t i;
- int sectors_per_bit = 1 << hbitmap_granularity(bitmap->meta);
-
- /* To optimize: we can make hbitmap to internally check the range in a
- * coarse level, or at least do it word by word. */
- for (i = sector; i < sector + nb_sectors; i += sectors_per_bit) {
- if (hbitmap_get(bitmap->meta, i)) {
- return true;
- }
- }
- return false;
-}
-
-int bdrv_dirty_bitmap_get_meta(BlockDriverState *bs,
- BdrvDirtyBitmap *bitmap, int64_t sector,
- int nb_sectors)
-{
- bool dirty;
-
- qemu_mutex_lock(bitmap->mutex);
- dirty = bdrv_dirty_bitmap_get_meta_locked(bs, bitmap, sector, nb_sectors);
- qemu_mutex_unlock(bitmap->mutex);
-
- return dirty;
-}
-
-void bdrv_dirty_bitmap_reset_meta(BlockDriverState *bs,
- BdrvDirtyBitmap *bitmap, int64_t sector,
- int nb_sectors)
-{
- qemu_mutex_lock(bitmap->mutex);
- hbitmap_reset(bitmap->meta, sector, nb_sectors);
- qemu_mutex_unlock(bitmap->mutex);
-}
-
int64_t bdrv_dirty_bitmap_size(const BdrvDirtyBitmap *bitmap)
{
return bitmap->size;
@@ -511,11 +472,6 @@ uint32_t bdrv_dirty_bitmap_granularity(const BdrvDirtyBitmap *bitmap)
return BDRV_SECTOR_SIZE << hbitmap_granularity(bitmap->bitmap);
}
-uint32_t bdrv_dirty_bitmap_meta_granularity(BdrvDirtyBitmap *bitmap)
-{
- return BDRV_SECTOR_SIZE << hbitmap_granularity(bitmap->meta);
-}
-
BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap,
uint64_t first_sector)
{
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 05/18] dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
` (3 preceding siblings ...)
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 04/18] dirty-bitmap: Drop unused functions Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-09-08 12:22 ` Kevin Wolf
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 06/18] dirty-bitmap: Change bdrv_dirty_bitmap_*serialize*() to take bytes Eric Blake
` (13 subsequent siblings)
18 siblings, 1 reply; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, vsementsov, qemu-block, Fam Zheng, Max Reitz
We are still using an internal hbitmap that tracks a size in sectors,
with the granularity scaled down accordingly, because it lets us
use a shortcut for our iterators which are currently sector-based.
But there's no reason we can't track the dirty bitmap size in bytes,
since it is (mostly) an internal-only variable (remember, the size
is how many bytes are covered by the bitmap, not how many bytes the
bitmap occupies). Furthermore, we're already reporting bytes for
bdrv_dirty_bitmap_granularity(); mixing bytes and sectors in our
return values is a recipe for confusion. A later cleanup will
convert dirty bitmap internals to be entirely byte-based,
eliminating the intermediate sector rounding added here; and
technically, since bdrv_getlength() already rounds up to sectors,
our use of DIV_ROUND_UP is more for theoretical completeness than
for any actual rounding.
The only external caller in qcow2-bitmap.c is temporarily more verbose
(because it is still using sector-based math), but will later be
switched to track progress by bytes instead of sectors.
Use is_power_of_2() while at it, instead of open-coding that, and
add an assertion where bdrv_getlength() should not fail.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
v6: no change
v5: fix bdrv_dirty_bitmap_truncate [John], drop R-b
v4: retitle from "Track size in bytes", rebase to persistent bitmaps,
round up when converting bytes to sectors
v3: no change
v2: tweak commit message, no code change
---
block/dirty-bitmap.c | 26 +++++++++++++++-----------
block/qcow2-bitmap.c | 14 ++++++++------
2 files changed, 23 insertions(+), 17 deletions(-)
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 42a55e4a4b..e65ec4f7ec 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -1,7 +1,7 @@
/*
* Block Dirty Bitmap
*
- * Copyright (c) 2016 Red Hat. Inc
+ * Copyright (c) 2016-2017 Red Hat. Inc
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
@@ -42,7 +42,7 @@ struct BdrvDirtyBitmap {
HBitmap *meta; /* Meta dirty bitmap */
BdrvDirtyBitmap *successor; /* Anonymous child; implies frozen status */
char *name; /* Optional non-empty unique ID */
- int64_t size; /* Size of the bitmap (Number of sectors) */
+ int64_t size; /* Size of the bitmap, in bytes */
bool disabled; /* Bitmap is disabled. It ignores all writes to
the device */
int active_iterators; /* How many iterators are active */
@@ -115,17 +115,14 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
{
int64_t bitmap_size;
BdrvDirtyBitmap *bitmap;
- uint32_t sector_granularity;
- assert((granularity & (granularity - 1)) == 0);
+ assert(is_power_of_2(granularity) && granularity >= BDRV_SECTOR_SIZE);
if (name && bdrv_find_dirty_bitmap(bs, name)) {
error_setg(errp, "Bitmap already exists: %s", name);
return NULL;
}
- sector_granularity = granularity >> BDRV_SECTOR_BITS;
- assert(sector_granularity);
- bitmap_size = bdrv_nb_sectors(bs);
+ bitmap_size = bdrv_getlength(bs);
if (bitmap_size < 0) {
error_setg_errno(errp, -bitmap_size, "could not get length of device");
errno = -bitmap_size;
@@ -133,7 +130,12 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
}
bitmap = g_new0(BdrvDirtyBitmap, 1);
bitmap->mutex = &bs->dirty_bitmap_mutex;
- bitmap->bitmap = hbitmap_alloc(bitmap_size, ctz32(sector_granularity));
+ /*
+ * TODO - let hbitmap track full granularity. For now, it is tracking
+ * only sector granularity, as a shortcut for our iterators.
+ */
+ bitmap->bitmap = hbitmap_alloc(DIV_ROUND_UP(bitmap_size, BDRV_SECTOR_SIZE),
+ ctz32(granularity) - BDRV_SECTOR_BITS);
bitmap->size = bitmap_size;
bitmap->name = g_strdup(name);
bitmap->disabled = false;
@@ -305,13 +307,14 @@ BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
void bdrv_dirty_bitmap_truncate(BlockDriverState *bs)
{
BdrvDirtyBitmap *bitmap;
- uint64_t size = bdrv_nb_sectors(bs);
+ int64_t size = bdrv_getlength(bs);
+ assert(size >= 0);
bdrv_dirty_bitmaps_lock(bs);
QLIST_FOREACH(bitmap, &bs->dirty_bitmaps, list) {
assert(!bdrv_dirty_bitmap_frozen(bitmap));
assert(!bitmap->active_iterators);
- hbitmap_truncate(bitmap->bitmap, size);
+ hbitmap_truncate(bitmap->bitmap, DIV_ROUND_UP(size, BDRV_SECTOR_SIZE));
bitmap->size = size;
}
bdrv_dirty_bitmaps_unlock(bs);
@@ -549,7 +552,8 @@ void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap **out)
hbitmap_reset_all(bitmap->bitmap);
} else {
HBitmap *backup = bitmap->bitmap;
- bitmap->bitmap = hbitmap_alloc(bitmap->size,
+ bitmap->bitmap = hbitmap_alloc(DIV_ROUND_UP(bitmap->size,
+ BDRV_SECTOR_SIZE),
hbitmap_granularity(backup));
*out = backup;
}
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index b3ee4c794a..65122e9ae1 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -295,10 +295,11 @@ static int load_bitmap_data(BlockDriverState *bs,
BDRVQcow2State *s = bs->opaque;
uint64_t sector, sbc;
uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap);
+ uint64_t bm_sectors = DIV_ROUND_UP(bm_size, BDRV_SECTOR_SIZE);
uint8_t *buf = NULL;
uint64_t i, tab_size =
size_to_clusters(s,
- bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_size));
+ bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_sectors));
if (tab_size != bitmap_table_size || tab_size > BME_MAX_TABLE_SIZE) {
return -EINVAL;
@@ -307,7 +308,7 @@ static int load_bitmap_data(BlockDriverState *bs,
buf = g_malloc(s->cluster_size);
sbc = sectors_covered_by_bitmap_cluster(s, bitmap);
for (i = 0, sector = 0; i < tab_size; ++i, sector += sbc) {
- uint64_t count = MIN(bm_size - sector, sbc);
+ uint64_t count = MIN(bm_sectors - sector, sbc);
uint64_t entry = bitmap_table[i];
uint64_t offset = entry & BME_TABLE_ENTRY_OFFSET_MASK;
@@ -1077,13 +1078,14 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
int64_t sector;
uint64_t sbc;
uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap);
+ uint64_t bm_sectors = DIV_ROUND_UP(bm_size, BDRV_SECTOR_SIZE);
const char *bm_name = bdrv_dirty_bitmap_name(bitmap);
uint8_t *buf = NULL;
BdrvDirtyBitmapIter *dbi;
uint64_t *tb;
uint64_t tb_size =
size_to_clusters(s,
- bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_size));
+ bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_sectors));
if (tb_size > BME_MAX_TABLE_SIZE ||
tb_size * s->cluster_size > BME_MAX_PHYS_SIZE)
@@ -1101,7 +1103,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
dbi = bdrv_dirty_iter_new(bitmap, 0);
buf = g_malloc(s->cluster_size);
sbc = sectors_covered_by_bitmap_cluster(s, bitmap);
- assert(DIV_ROUND_UP(bm_size, sbc) == tb_size);
+ assert(DIV_ROUND_UP(bm_sectors, sbc) == tb_size);
while ((sector = bdrv_dirty_iter_next(dbi)) != -1) {
uint64_t cluster = sector / sbc;
@@ -1109,7 +1111,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
int64_t off;
sector = cluster * sbc;
- end = MIN(bm_size, sector + sbc);
+ end = MIN(bm_sectors, sector + sbc);
write_size =
bdrv_dirty_bitmap_serialization_size(bitmap, sector, end - sector);
assert(write_size <= s->cluster_size);
@@ -1141,7 +1143,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
goto fail;
}
- if (end >= bm_size) {
+ if (end >= bm_sectors) {
break;
}
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 06/18] dirty-bitmap: Change bdrv_dirty_bitmap_*serialize*() to take bytes
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
` (4 preceding siblings ...)
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 05/18] dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 07/18] qcow2: Switch sectors_covered_by_bitmap_cluster() to byte-based Eric Blake
` (12 subsequent siblings)
18 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, vsementsov, qemu-block, Fam Zheng, Max Reitz
Right now, the dirty-bitmap code exposes the fact that we use
a scale of sector granularity in the underlying hbitmap to anything
that wants to serialize a dirty bitmap. It's nicer to uniformly
expose bytes as our dirty-bitmap interface, matching the previous
change to bitmap size. The only caller to serialization is currently
qcow2-cluster.c, which becomes a bit more verbose because it is still
tracking sectors for other reasons, but a later patch will fix that
to more uniformly use byte offsets everywhere. Likewise, within
dirty-bitmap, we have to add more assertions that we are not
truncating incorrectly, which can go away once the internal hbitmap
is byte-based rather than sector-based.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
v5: no change
v4: new patch
---
include/block/dirty-bitmap.h | 14 +++++++-------
block/dirty-bitmap.c | 37 ++++++++++++++++++++++++-------------
block/qcow2-bitmap.c | 22 ++++++++++++++--------
3 files changed, 45 insertions(+), 28 deletions(-)
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 8fd842eac9..f4ccd3f33f 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -49,19 +49,19 @@ BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap,
void bdrv_dirty_iter_free(BdrvDirtyBitmapIter *iter);
uint64_t bdrv_dirty_bitmap_serialization_size(const BdrvDirtyBitmap *bitmap,
- uint64_t start, uint64_t count);
+ uint64_t offset, uint64_t bytes);
uint64_t bdrv_dirty_bitmap_serialization_align(const BdrvDirtyBitmap *bitmap);
void bdrv_dirty_bitmap_serialize_part(const BdrvDirtyBitmap *bitmap,
- uint8_t *buf, uint64_t start,
- uint64_t count);
+ uint8_t *buf, uint64_t offset,
+ uint64_t bytes);
void bdrv_dirty_bitmap_deserialize_part(BdrvDirtyBitmap *bitmap,
- uint8_t *buf, uint64_t start,
- uint64_t count, bool finish);
+ uint8_t *buf, uint64_t offset,
+ uint64_t bytes, bool finish);
void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap,
- uint64_t start, uint64_t count,
+ uint64_t offset, uint64_t bytes,
bool finish);
void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap,
- uint64_t start, uint64_t count,
+ uint64_t offset, uint64_t bytes,
bool finish);
void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index e65ec4f7ec..034effc8cd 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -570,42 +570,53 @@ void bdrv_undo_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap *in)
}
uint64_t bdrv_dirty_bitmap_serialization_size(const BdrvDirtyBitmap *bitmap,
- uint64_t start, uint64_t count)
+ uint64_t offset, uint64_t bytes)
{
- return hbitmap_serialization_size(bitmap->bitmap, start, count);
+ assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE));
+ return hbitmap_serialization_size(bitmap->bitmap,
+ offset >> BDRV_SECTOR_BITS,
+ bytes >> BDRV_SECTOR_BITS);
}
uint64_t bdrv_dirty_bitmap_serialization_align(const BdrvDirtyBitmap *bitmap)
{
- return hbitmap_serialization_align(bitmap->bitmap);
+ return hbitmap_serialization_align(bitmap->bitmap) * BDRV_SECTOR_SIZE;
}
void bdrv_dirty_bitmap_serialize_part(const BdrvDirtyBitmap *bitmap,
- uint8_t *buf, uint64_t start,
- uint64_t count)
+ uint8_t *buf, uint64_t offset,
+ uint64_t bytes)
{
- hbitmap_serialize_part(bitmap->bitmap, buf, start, count);
+ assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE));
+ hbitmap_serialize_part(bitmap->bitmap, buf, offset >> BDRV_SECTOR_BITS,
+ bytes >> BDRV_SECTOR_BITS);
}
void bdrv_dirty_bitmap_deserialize_part(BdrvDirtyBitmap *bitmap,
- uint8_t *buf, uint64_t start,
- uint64_t count, bool finish)
+ uint8_t *buf, uint64_t offset,
+ uint64_t bytes, bool finish)
{
- hbitmap_deserialize_part(bitmap->bitmap, buf, start, count, finish);
+ assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE));
+ hbitmap_deserialize_part(bitmap->bitmap, buf, offset >> BDRV_SECTOR_BITS,
+ bytes >> BDRV_SECTOR_BITS, finish);
}
void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap,
- uint64_t start, uint64_t count,
+ uint64_t offset, uint64_t bytes,
bool finish)
{
- hbitmap_deserialize_zeroes(bitmap->bitmap, start, count, finish);
+ assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE));
+ hbitmap_deserialize_zeroes(bitmap->bitmap, offset >> BDRV_SECTOR_BITS,
+ bytes >> BDRV_SECTOR_BITS, finish);
}
void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap,
- uint64_t start, uint64_t count,
+ uint64_t offset, uint64_t bytes,
bool finish)
{
- hbitmap_deserialize_ones(bitmap->bitmap, start, count, finish);
+ assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE));
+ hbitmap_deserialize_ones(bitmap->bitmap, offset >> BDRV_SECTOR_BITS,
+ bytes >> BDRV_SECTOR_BITS, finish);
}
void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap)
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index 65122e9ae1..92098bfa49 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -278,7 +278,7 @@ static uint64_t sectors_covered_by_bitmap_cluster(const BDRVQcow2State *s,
bdrv_dirty_bitmap_granularity(bitmap) >> BDRV_SECTOR_BITS;
uint64_t sbc = sector_granularity * (s->cluster_size << 3);
- assert(QEMU_IS_ALIGNED(sbc,
+ assert(QEMU_IS_ALIGNED(sbc << BDRV_SECTOR_BITS,
bdrv_dirty_bitmap_serialization_align(bitmap)));
return sbc;
}
@@ -299,7 +299,7 @@ static int load_bitmap_data(BlockDriverState *bs,
uint8_t *buf = NULL;
uint64_t i, tab_size =
size_to_clusters(s,
- bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_sectors));
+ bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_size));
if (tab_size != bitmap_table_size || tab_size > BME_MAX_TABLE_SIZE) {
return -EINVAL;
@@ -316,7 +316,9 @@ static int load_bitmap_data(BlockDriverState *bs,
if (offset == 0) {
if (entry & BME_TABLE_ENTRY_FLAG_ALL_ONES) {
- bdrv_dirty_bitmap_deserialize_ones(bitmap, sector, count,
+ bdrv_dirty_bitmap_deserialize_ones(bitmap,
+ sector * BDRV_SECTOR_SIZE,
+ count * BDRV_SECTOR_SIZE,
false);
} else {
/* No need to deserialize zeros because the dirty bitmap is
@@ -327,7 +329,9 @@ static int load_bitmap_data(BlockDriverState *bs,
if (ret < 0) {
goto finish;
}
- bdrv_dirty_bitmap_deserialize_part(bitmap, buf, sector, count,
+ bdrv_dirty_bitmap_deserialize_part(bitmap, buf,
+ sector * BDRV_SECTOR_SIZE,
+ count * BDRV_SECTOR_SIZE,
false);
}
}
@@ -1085,7 +1089,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
uint64_t *tb;
uint64_t tb_size =
size_to_clusters(s,
- bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_sectors));
+ bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_size));
if (tb_size > BME_MAX_TABLE_SIZE ||
tb_size * s->cluster_size > BME_MAX_PHYS_SIZE)
@@ -1112,8 +1116,8 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
sector = cluster * sbc;
end = MIN(bm_sectors, sector + sbc);
- write_size =
- bdrv_dirty_bitmap_serialization_size(bitmap, sector, end - sector);
+ write_size = bdrv_dirty_bitmap_serialization_size(bitmap,
+ sector * BDRV_SECTOR_SIZE, (end - sector) * BDRV_SECTOR_SIZE);
assert(write_size <= s->cluster_size);
off = qcow2_alloc_clusters(bs, s->cluster_size);
@@ -1125,7 +1129,9 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
}
tb[cluster] = off;
- bdrv_dirty_bitmap_serialize_part(bitmap, buf, sector, end - sector);
+ bdrv_dirty_bitmap_serialize_part(bitmap, buf,
+ sector * BDRV_SECTOR_SIZE,
+ (end - sector) * BDRV_SECTOR_SIZE);
if (write_size < s->cluster_size) {
memset(buf + write_size, 0, s->cluster_size - write_size);
}
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 07/18] qcow2: Switch sectors_covered_by_bitmap_cluster() to byte-based
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
` (5 preceding siblings ...)
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 06/18] dirty-bitmap: Change bdrv_dirty_bitmap_*serialize*() to take bytes Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 08/18] dirty-bitmap: Set iterator start by offset, not sector Eric Blake
` (11 subsequent siblings)
18 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, vsementsov, qemu-block, Max Reitz
We are gradually converting to byte-based interfaces, as they are
easier to reason about than sector-based. Change the qcow2 bitmap
helper function sectors_covered_by_bitmap_cluster(), renaming it
to bytes_covered_by_bitmap_cluster() in the process.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
v5: no change
v4: new patch
---
block/qcow2-bitmap.c | 28 ++++++++++++++--------------
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index 92098bfa49..4475273d8c 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -269,18 +269,16 @@ static int free_bitmap_clusters(BlockDriverState *bs, Qcow2BitmapTable *tb)
return 0;
}
-/* This function returns the number of disk sectors covered by a single qcow2
- * cluster of bitmap data. */
-static uint64_t sectors_covered_by_bitmap_cluster(const BDRVQcow2State *s,
- const BdrvDirtyBitmap *bitmap)
+/* Return the disk size covered by a single qcow2 cluster of bitmap data. */
+static uint64_t bytes_covered_by_bitmap_cluster(const BDRVQcow2State *s,
+ const BdrvDirtyBitmap *bitmap)
{
- uint64_t sector_granularity =
- bdrv_dirty_bitmap_granularity(bitmap) >> BDRV_SECTOR_BITS;
- uint64_t sbc = sector_granularity * (s->cluster_size << 3);
+ uint64_t granularity = bdrv_dirty_bitmap_granularity(bitmap);
+ uint64_t limit = granularity * (s->cluster_size << 3);
- assert(QEMU_IS_ALIGNED(sbc << BDRV_SECTOR_BITS,
+ assert(QEMU_IS_ALIGNED(limit,
bdrv_dirty_bitmap_serialization_align(bitmap)));
- return sbc;
+ return limit;
}
/* load_bitmap_data
@@ -293,7 +291,7 @@ static int load_bitmap_data(BlockDriverState *bs,
{
int ret = 0;
BDRVQcow2State *s = bs->opaque;
- uint64_t sector, sbc;
+ uint64_t sector, limit, sbc;
uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap);
uint64_t bm_sectors = DIV_ROUND_UP(bm_size, BDRV_SECTOR_SIZE);
uint8_t *buf = NULL;
@@ -306,7 +304,8 @@ static int load_bitmap_data(BlockDriverState *bs,
}
buf = g_malloc(s->cluster_size);
- sbc = sectors_covered_by_bitmap_cluster(s, bitmap);
+ limit = bytes_covered_by_bitmap_cluster(s, bitmap);
+ sbc = limit >> BDRV_SECTOR_BITS;
for (i = 0, sector = 0; i < tab_size; ++i, sector += sbc) {
uint64_t count = MIN(bm_sectors - sector, sbc);
uint64_t entry = bitmap_table[i];
@@ -1080,7 +1079,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
int ret;
BDRVQcow2State *s = bs->opaque;
int64_t sector;
- uint64_t sbc;
+ uint64_t limit, sbc;
uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap);
uint64_t bm_sectors = DIV_ROUND_UP(bm_size, BDRV_SECTOR_SIZE);
const char *bm_name = bdrv_dirty_bitmap_name(bitmap);
@@ -1106,8 +1105,9 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
dbi = bdrv_dirty_iter_new(bitmap, 0);
buf = g_malloc(s->cluster_size);
- sbc = sectors_covered_by_bitmap_cluster(s, bitmap);
- assert(DIV_ROUND_UP(bm_sectors, sbc) == tb_size);
+ limit = bytes_covered_by_bitmap_cluster(s, bitmap);
+ sbc = limit >> BDRV_SECTOR_BITS;
+ assert(DIV_ROUND_UP(bm_size, limit) == tb_size);
while ((sector = bdrv_dirty_iter_next(dbi)) != -1) {
uint64_t cluster = sector / sbc;
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 08/18] dirty-bitmap: Set iterator start by offset, not sector
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
` (6 preceding siblings ...)
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 07/18] qcow2: Switch sectors_covered_by_bitmap_cluster() to byte-based Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 09/18] dirty-bitmap: Change bdrv_dirty_iter_next() to report byte offset Eric Blake
` (10 subsequent siblings)
18 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, jsnow, vsementsov, qemu-block, Jeff Cody, Max Reitz, Fam Zheng
All callers to bdrv_dirty_iter_new() passed 0 for their initial
starting point, drop that parameter.
Most callers to bdrv_set_dirty_iter() were scaling a byte offset to
a sector number; the exception qcow2-bitmap will be converted later
to use byte rather than sector iteration. Move the scaling to occur
internally to dirty bitmap code instead, so that callers now pass
in bytes.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
v5: no change
v4: rebase to persistent bitmaps
v3: no change
v2: no change
---
include/block/dirty-bitmap.h | 5 ++---
block/backup.c | 5 ++---
block/dirty-bitmap.c | 9 ++++-----
block/mirror.c | 4 ++--
block/qcow2-bitmap.c | 4 ++--
5 files changed, 12 insertions(+), 15 deletions(-)
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index f4ccd3f33f..ece28e1edc 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -44,8 +44,7 @@ void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
int64_t cur_sector, int64_t nr_sectors);
BdrvDirtyBitmapIter *bdrv_dirty_meta_iter_new(BdrvDirtyBitmap *bitmap);
-BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap,
- uint64_t first_sector);
+BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap);
void bdrv_dirty_iter_free(BdrvDirtyBitmapIter *iter);
uint64_t bdrv_dirty_bitmap_serialization_size(const BdrvDirtyBitmap *bitmap,
@@ -80,7 +79,7 @@ void bdrv_set_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
void bdrv_reset_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
int64_t cur_sector, int64_t nr_sectors);
int64_t bdrv_dirty_iter_next(BdrvDirtyBitmapIter *iter);
-void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *hbi, int64_t sector_num);
+void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *hbi, int64_t offset);
int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap);
int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap);
void bdrv_dirty_bitmap_truncate(BlockDriverState *bs);
diff --git a/block/backup.c b/block/backup.c
index 504a089150..1a6ec7d079 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -372,7 +372,7 @@ static int coroutine_fn backup_run_incremental(BackupBlockJob *job)
granularity = bdrv_dirty_bitmap_granularity(job->sync_bitmap);
clusters_per_iter = MAX((granularity / job->cluster_size), 1);
- dbi = bdrv_dirty_iter_new(job->sync_bitmap, 0);
+ dbi = bdrv_dirty_iter_new(job->sync_bitmap);
/* Find the next dirty sector(s) */
while ((offset = bdrv_dirty_iter_next(dbi) * BDRV_SECTOR_SIZE) >= 0) {
@@ -403,8 +403,7 @@ static int coroutine_fn backup_run_incremental(BackupBlockJob *job)
/* If the bitmap granularity is smaller than the backup granularity,
* we need to advance the iterator pointer to the next cluster. */
if (granularity < job->cluster_size) {
- bdrv_set_dirty_iter(dbi,
- cluster * job->cluster_size / BDRV_SECTOR_SIZE);
+ bdrv_set_dirty_iter(dbi, cluster * job->cluster_size);
}
last_cluster = cluster - 1;
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 034effc8cd..c091f0c7ba 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -475,11 +475,10 @@ uint32_t bdrv_dirty_bitmap_granularity(const BdrvDirtyBitmap *bitmap)
return BDRV_SECTOR_SIZE << hbitmap_granularity(bitmap->bitmap);
}
-BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap,
- uint64_t first_sector)
+BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap)
{
BdrvDirtyBitmapIter *iter = g_new(BdrvDirtyBitmapIter, 1);
- hbitmap_iter_init(&iter->hbi, bitmap->bitmap, first_sector);
+ hbitmap_iter_init(&iter->hbi, bitmap->bitmap, 0);
iter->bitmap = bitmap;
bitmap->active_iterators++;
return iter;
@@ -647,9 +646,9 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
/**
* Advance a BdrvDirtyBitmapIter to an arbitrary offset.
*/
-void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *iter, int64_t sector_num)
+void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *iter, int64_t offset)
{
- hbitmap_iter_init(&iter->hbi, iter->hbi.hb, sector_num);
+ hbitmap_iter_init(&iter->hbi, iter->hbi.hb, offset >> BDRV_SECTOR_BITS);
}
int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap)
diff --git a/block/mirror.c b/block/mirror.c
index 429751b9fe..51824750ac 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -373,7 +373,7 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
next_dirty = bdrv_dirty_iter_next(s->dbi) * BDRV_SECTOR_SIZE;
if (next_dirty > next_offset || next_dirty < 0) {
/* The bitmap iterator's cache is stale, refresh it */
- bdrv_set_dirty_iter(s->dbi, next_offset >> BDRV_SECTOR_BITS);
+ bdrv_set_dirty_iter(s->dbi, next_offset);
next_dirty = bdrv_dirty_iter_next(s->dbi) * BDRV_SECTOR_SIZE;
}
assert(next_dirty == next_offset);
@@ -796,7 +796,7 @@ static void coroutine_fn mirror_run(void *opaque)
}
assert(!s->dbi);
- s->dbi = bdrv_dirty_iter_new(s->dirty_bitmap, 0);
+ s->dbi = bdrv_dirty_iter_new(s->dirty_bitmap);
for (;;) {
uint64_t delay_ns = 0;
int64_t cnt, delta;
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index 4475273d8c..44329fc74f 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -1103,7 +1103,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
return NULL;
}
- dbi = bdrv_dirty_iter_new(bitmap, 0);
+ dbi = bdrv_dirty_iter_new(bitmap);
buf = g_malloc(s->cluster_size);
limit = bytes_covered_by_bitmap_cluster(s, bitmap);
sbc = limit >> BDRV_SECTOR_BITS;
@@ -1153,7 +1153,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
break;
}
- bdrv_set_dirty_iter(dbi, end);
+ bdrv_set_dirty_iter(dbi, end * BDRV_SECTOR_SIZE);
}
*bitmap_table_size = tb_size;
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 09/18] dirty-bitmap: Change bdrv_dirty_iter_next() to report byte offset
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
` (7 preceding siblings ...)
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 08/18] dirty-bitmap: Set iterator start by offset, not sector Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 10/18] dirty-bitmap: Change bdrv_get_dirty_count() to report bytes Eric Blake
` (9 subsequent siblings)
18 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, jsnow, vsementsov, qemu-block, Jeff Cody, Max Reitz, Fam Zheng
Thanks to recent cleanups, most callers were scaling a return value
of sectors into bytes (the exception, in qcow2-bitmap, will be
converted to byte-based iteration later). Update the interface to
do the scaling internally instead.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-By: John Snow <jsnow@redhat.com>
---
v5: no change
v4: rebase to persistent bitmap
v3: no change
v2: no change
---
block/backup.c | 2 +-
block/dirty-bitmap.c | 2 +-
block/mirror.c | 8 ++++----
block/qcow2-bitmap.c | 2 +-
4 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/block/backup.c b/block/backup.c
index 1a6ec7d079..f53cde90d6 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -375,7 +375,7 @@ static int coroutine_fn backup_run_incremental(BackupBlockJob *job)
dbi = bdrv_dirty_iter_new(job->sync_bitmap);
/* Find the next dirty sector(s) */
- while ((offset = bdrv_dirty_iter_next(dbi) * BDRV_SECTOR_SIZE) >= 0) {
+ while ((offset = bdrv_dirty_iter_next(dbi)) >= 0) {
cluster = offset / job->cluster_size;
/* Fake progress updates for any clusters we skipped */
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index c091f0c7ba..20f230867d 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -505,7 +505,7 @@ void bdrv_dirty_iter_free(BdrvDirtyBitmapIter *iter)
int64_t bdrv_dirty_iter_next(BdrvDirtyBitmapIter *iter)
{
- return hbitmap_iter_next(&iter->hbi);
+ return hbitmap_iter_next(&iter->hbi) * BDRV_SECTOR_SIZE;
}
/* Called within bdrv_dirty_bitmap_lock..unlock */
diff --git a/block/mirror.c b/block/mirror.c
index 51824750ac..af13f5d658 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -336,10 +336,10 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
int max_io_bytes = MAX(s->buf_size / MAX_IN_FLIGHT, MAX_IO_BYTES);
bdrv_dirty_bitmap_lock(s->dirty_bitmap);
- offset = bdrv_dirty_iter_next(s->dbi) * BDRV_SECTOR_SIZE;
+ offset = bdrv_dirty_iter_next(s->dbi);
if (offset < 0) {
bdrv_set_dirty_iter(s->dbi, 0);
- offset = bdrv_dirty_iter_next(s->dbi) * BDRV_SECTOR_SIZE;
+ offset = bdrv_dirty_iter_next(s->dbi);
trace_mirror_restart_iter(s, bdrv_get_dirty_count(s->dirty_bitmap) *
BDRV_SECTOR_SIZE);
assert(offset >= 0);
@@ -370,11 +370,11 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
break;
}
- next_dirty = bdrv_dirty_iter_next(s->dbi) * BDRV_SECTOR_SIZE;
+ next_dirty = bdrv_dirty_iter_next(s->dbi);
if (next_dirty > next_offset || next_dirty < 0) {
/* The bitmap iterator's cache is stale, refresh it */
bdrv_set_dirty_iter(s->dbi, next_offset);
- next_dirty = bdrv_dirty_iter_next(s->dbi) * BDRV_SECTOR_SIZE;
+ next_dirty = bdrv_dirty_iter_next(s->dbi);
}
assert(next_dirty == next_offset);
nb_chunks++;
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index 44329fc74f..c7c60dfca2 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -1109,7 +1109,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
sbc = limit >> BDRV_SECTOR_BITS;
assert(DIV_ROUND_UP(bm_size, limit) == tb_size);
- while ((sector = bdrv_dirty_iter_next(dbi)) != -1) {
+ while ((sector = bdrv_dirty_iter_next(dbi) >> BDRV_SECTOR_BITS) != -1) {
uint64_t cluster = sector / sbc;
uint64_t end, write_size;
int64_t off;
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 10/18] dirty-bitmap: Change bdrv_get_dirty_count() to report bytes
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
` (8 preceding siblings ...)
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 09/18] dirty-bitmap: Change bdrv_dirty_iter_next() to report byte offset Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-09-08 12:51 ` Kevin Wolf
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 11/18] dirty-bitmap: Change bdrv_get_dirty_locked() to take bytes Eric Blake
` (8 subsequent siblings)
18 siblings, 1 reply; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, jsnow, vsementsov, qemu-block, Fam Zheng, Max Reitz,
Jeff Cody, Stefan Hajnoczi, Juan Quintela,
Dr. David Alan Gilbert
Thanks to recent cleanups, all callers were scaling a return value
of sectors into bytes; do the scaling internally instead.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
---
v4: no change
v3: no change, add R-b
v2: no change
---
block/dirty-bitmap.c | 4 ++--
block/mirror.c | 13 +++++--------
migration/block.c | 2 +-
3 files changed, 8 insertions(+), 11 deletions(-)
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 20f230867d..f983d99def 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -425,7 +425,7 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) {
BlockDirtyInfo *info = g_new0(BlockDirtyInfo, 1);
BlockDirtyInfoList *entry = g_new0(BlockDirtyInfoList, 1);
- info->count = bdrv_get_dirty_count(bm) << BDRV_SECTOR_BITS;
+ info->count = bdrv_get_dirty_count(bm);
info->granularity = bdrv_dirty_bitmap_granularity(bm);
info->has_name = !!bm->name;
info->name = g_strdup(bm->name);
@@ -653,7 +653,7 @@ void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *iter, int64_t offset)
int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap)
{
- return hbitmap_count(bitmap->bitmap);
+ return hbitmap_count(bitmap->bitmap) << BDRV_SECTOR_BITS;
}
int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap)
diff --git a/block/mirror.c b/block/mirror.c
index af13f5d658..cc47e21814 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -811,11 +811,10 @@ static void coroutine_fn mirror_run(void *opaque)
cnt = bdrv_get_dirty_count(s->dirty_bitmap);
/* s->common.offset contains the number of bytes already processed so
- * far, cnt is the number of dirty sectors remaining and
+ * far, cnt is the number of dirty bytes remaining and
* s->bytes_in_flight is the number of bytes currently being
* processed; together those are the current total operation length */
- s->common.len = s->common.offset + s->bytes_in_flight +
- cnt * BDRV_SECTOR_SIZE;
+ s->common.len = s->common.offset + s->bytes_in_flight + cnt;
/* Note that even when no rate limit is applied we need to yield
* periodically with no pending I/O so that bdrv_drain_all() returns.
@@ -827,8 +826,7 @@ static void coroutine_fn mirror_run(void *opaque)
s->common.iostatus == BLOCK_DEVICE_IO_STATUS_OK) {
if (s->in_flight >= MAX_IN_FLIGHT || s->buf_free_count == 0 ||
(cnt == 0 && s->in_flight > 0)) {
- trace_mirror_yield(s, cnt * BDRV_SECTOR_SIZE,
- s->buf_free_count, s->in_flight);
+ trace_mirror_yield(s, cnt, s->buf_free_count, s->in_flight);
mirror_wait_for_io(s);
continue;
} else if (cnt != 0) {
@@ -869,7 +867,7 @@ static void coroutine_fn mirror_run(void *opaque)
* whether to switch to target check one last time if I/O has
* come in the meanwhile, and if not flush the data to disk.
*/
- trace_mirror_before_drain(s, cnt * BDRV_SECTOR_SIZE);
+ trace_mirror_before_drain(s, cnt);
bdrv_drained_begin(bs);
cnt = bdrv_get_dirty_count(s->dirty_bitmap);
@@ -888,8 +886,7 @@ static void coroutine_fn mirror_run(void *opaque)
}
ret = 0;
- trace_mirror_before_sleep(s, cnt * BDRV_SECTOR_SIZE,
- s->synced, delay_ns);
+ trace_mirror_before_sleep(s, cnt, s->synced, delay_ns);
if (!s->synced) {
block_job_sleep_ns(&s->common, QEMU_CLOCK_REALTIME, delay_ns);
if (block_job_is_cancelled(&s->common)) {
diff --git a/migration/block.c b/migration/block.c
index 9171f60028..a3512945da 100644
--- a/migration/block.c
+++ b/migration/block.c
@@ -667,7 +667,7 @@ static int64_t get_remaining_dirty(void)
aio_context_release(blk_get_aio_context(bmds->blk));
}
- return dirty << BDRV_SECTOR_BITS;
+ return dirty;
}
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 11/18] dirty-bitmap: Change bdrv_get_dirty_locked() to take bytes
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
` (9 preceding siblings ...)
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 10/18] dirty-bitmap: Change bdrv_get_dirty_count() to report bytes Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 12/18] dirty-bitmap: Change bdrv_[re]set_dirty_bitmap() to use bytes Eric Blake
` (7 subsequent siblings)
18 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, jsnow, vsementsov, qemu-block, Fam Zheng, Max Reitz,
Jeff Cody, Stefan Hajnoczi, Juan Quintela,
Dr. David Alan Gilbert
Half the callers were already scaling bytes to sectors; the other
half can eventually be simplified to use byte iteration. Both
callers were already using the result as a bool, so make that
explicit. Making the change also makes it easier for a future
dirty-bitmap patch to offload scaling over to the internal hbitmap.
Remember, asking whether a byte is dirty is effectively asking
whether the entire granularity containing the byte is dirty, since
we only track dirtiness by granularity.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
---
v4: only context change
v3: rebase to _locked rename was straightforward enough that R-b kept
v2: tweak commit message, no code change
---
include/block/dirty-bitmap.h | 4 ++--
block/dirty-bitmap.c | 8 ++++----
block/mirror.c | 3 +--
migration/block.c | 3 ++-
4 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index ece28e1edc..1dddcd320b 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -72,8 +72,8 @@ void bdrv_dirty_bitmap_set_persistance(BdrvDirtyBitmap *bitmap,
/* Functions that require manual locking. */
void bdrv_dirty_bitmap_lock(BdrvDirtyBitmap *bitmap);
void bdrv_dirty_bitmap_unlock(BdrvDirtyBitmap *bitmap);
-int bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
- int64_t sector);
+bool bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
+ int64_t offset);
void bdrv_set_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
int64_t cur_sector, int64_t nr_sectors);
void bdrv_reset_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index f983d99def..8b3c0221c6 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -440,13 +440,13 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
}
/* Called within bdrv_dirty_bitmap_lock..unlock */
-int bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
- int64_t sector)
+bool bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
+ int64_t offset)
{
if (bitmap) {
- return hbitmap_get(bitmap->bitmap, sector);
+ return hbitmap_get(bitmap->bitmap, offset >> BDRV_SECTOR_BITS);
} else {
- return 0;
+ return false;
}
}
diff --git a/block/mirror.c b/block/mirror.c
index cc47e21814..42888ebd04 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -362,8 +362,7 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
int64_t next_offset = offset + nb_chunks * s->granularity;
int64_t next_chunk = next_offset / s->granularity;
if (next_offset >= s->bdev_length ||
- !bdrv_get_dirty_locked(source, s->dirty_bitmap,
- next_offset >> BDRV_SECTOR_BITS)) {
+ !bdrv_get_dirty_locked(source, s->dirty_bitmap, next_offset)) {
break;
}
if (test_bit(next_chunk, s->in_flight_bitmap)) {
diff --git a/migration/block.c b/migration/block.c
index a3512945da..b618869661 100644
--- a/migration/block.c
+++ b/migration/block.c
@@ -530,7 +530,8 @@ static int mig_save_device_dirty(QEMUFile *f, BlkMigDevState *bmds,
blk_mig_unlock();
}
bdrv_dirty_bitmap_lock(bmds->dirty_bitmap);
- if (bdrv_get_dirty_locked(bs, bmds->dirty_bitmap, sector)) {
+ if (bdrv_get_dirty_locked(bs, bmds->dirty_bitmap,
+ sector * BDRV_SECTOR_SIZE)) {
if (total_sectors - sector < BDRV_SECTORS_PER_DIRTY_CHUNK) {
nr_sectors = total_sectors - sector;
} else {
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 12/18] dirty-bitmap: Change bdrv_[re]set_dirty_bitmap() to use bytes
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
` (10 preceding siblings ...)
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 11/18] dirty-bitmap: Change bdrv_get_dirty_locked() to take bytes Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 13/18] mirror: Switch mirror_dirty_init() to byte-based iteration Eric Blake
` (6 subsequent siblings)
18 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, jsnow, vsementsov, qemu-block, Fam Zheng, Max Reitz,
Jeff Cody, Stefan Hajnoczi, Juan Quintela,
Dr. David Alan Gilbert
Some of the callers were already scaling bytes to sectors; others
can be easily converted to pass byte offsets, all in our shift
towards a consistent byte interface everywhere. Making the change
will also make it easier to write the hold-out callers to use byte
rather than sectors for their iterations; it also makes it easier
for a future dirty-bitmap patch to offload scaling over to the
internal hbitmap. Although all callers happen to pass
sector-aligned values, make the internal scaling robust to any
sub-sector requests.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
v5: only context change
v4: only context change, due to rebasing to persistent bitmaps
v3: rebase to addition of _locked interfaces; complex enough that I
dropped R-b
v2: no change
---
include/block/dirty-bitmap.h | 8 ++++----
block/dirty-bitmap.c | 22 ++++++++++++++--------
block/mirror.c | 16 ++++++++--------
migration/block.c | 7 +++++--
4 files changed, 31 insertions(+), 22 deletions(-)
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 1dddcd320b..0341a605d7 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -40,9 +40,9 @@ const char *bdrv_dirty_bitmap_name(const BdrvDirtyBitmap *bitmap);
int64_t bdrv_dirty_bitmap_size(const BdrvDirtyBitmap *bitmap);
DirtyBitmapStatus bdrv_dirty_bitmap_status(BdrvDirtyBitmap *bitmap);
void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
- int64_t cur_sector, int64_t nr_sectors);
+ int64_t offset, int64_t bytes);
void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
- int64_t cur_sector, int64_t nr_sectors);
+ int64_t offset, int64_t bytes);
BdrvDirtyBitmapIter *bdrv_dirty_meta_iter_new(BdrvDirtyBitmap *bitmap);
BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap);
void bdrv_dirty_iter_free(BdrvDirtyBitmapIter *iter);
@@ -75,9 +75,9 @@ void bdrv_dirty_bitmap_unlock(BdrvDirtyBitmap *bitmap);
bool bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
int64_t offset);
void bdrv_set_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
- int64_t cur_sector, int64_t nr_sectors);
+ int64_t offset, int64_t bytes);
void bdrv_reset_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
- int64_t cur_sector, int64_t nr_sectors);
+ int64_t offset, int64_t bytes);
int64_t bdrv_dirty_iter_next(BdrvDirtyBitmapIter *iter);
void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *hbi, int64_t offset);
int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap);
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 8b3c0221c6..9821225523 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -510,35 +510,41 @@ int64_t bdrv_dirty_iter_next(BdrvDirtyBitmapIter *iter)
/* Called within bdrv_dirty_bitmap_lock..unlock */
void bdrv_set_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
- int64_t cur_sector, int64_t nr_sectors)
+ int64_t offset, int64_t bytes)
{
+ int64_t end_sector = DIV_ROUND_UP(offset + bytes, BDRV_SECTOR_SIZE);
+
assert(bdrv_dirty_bitmap_enabled(bitmap));
assert(!bdrv_dirty_bitmap_readonly(bitmap));
- hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
+ hbitmap_set(bitmap->bitmap, offset >> BDRV_SECTOR_BITS,
+ end_sector - (offset >> BDRV_SECTOR_BITS));
}
void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
- int64_t cur_sector, int64_t nr_sectors)
+ int64_t offset, int64_t bytes)
{
bdrv_dirty_bitmap_lock(bitmap);
- bdrv_set_dirty_bitmap_locked(bitmap, cur_sector, nr_sectors);
+ bdrv_set_dirty_bitmap_locked(bitmap, offset, bytes);
bdrv_dirty_bitmap_unlock(bitmap);
}
/* Called within bdrv_dirty_bitmap_lock..unlock */
void bdrv_reset_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
- int64_t cur_sector, int64_t nr_sectors)
+ int64_t offset, int64_t bytes)
{
+ int64_t end_sector = DIV_ROUND_UP(offset + bytes, BDRV_SECTOR_SIZE);
+
assert(bdrv_dirty_bitmap_enabled(bitmap));
assert(!bdrv_dirty_bitmap_readonly(bitmap));
- hbitmap_reset(bitmap->bitmap, cur_sector, nr_sectors);
+ hbitmap_reset(bitmap->bitmap, offset >> BDRV_SECTOR_BITS,
+ end_sector - (offset >> BDRV_SECTOR_BITS));
}
void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
- int64_t cur_sector, int64_t nr_sectors)
+ int64_t offset, int64_t bytes)
{
bdrv_dirty_bitmap_lock(bitmap);
- bdrv_reset_dirty_bitmap_locked(bitmap, cur_sector, nr_sectors);
+ bdrv_reset_dirty_bitmap_locked(bitmap, offset, bytes);
bdrv_dirty_bitmap_unlock(bitmap);
}
diff --git a/block/mirror.c b/block/mirror.c
index 42888ebd04..18172a56a3 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -141,8 +141,7 @@ static void mirror_write_complete(void *opaque, int ret)
if (ret < 0) {
BlockErrorAction action;
- bdrv_set_dirty_bitmap(s->dirty_bitmap, op->offset >> BDRV_SECTOR_BITS,
- op->bytes >> BDRV_SECTOR_BITS);
+ bdrv_set_dirty_bitmap(s->dirty_bitmap, op->offset, op->bytes);
action = mirror_error_action(s, false, -ret);
if (action == BLOCK_ERROR_ACTION_REPORT && s->ret >= 0) {
s->ret = ret;
@@ -161,8 +160,7 @@ static void mirror_read_complete(void *opaque, int ret)
if (ret < 0) {
BlockErrorAction action;
- bdrv_set_dirty_bitmap(s->dirty_bitmap, op->offset >> BDRV_SECTOR_BITS,
- op->bytes >> BDRV_SECTOR_BITS);
+ bdrv_set_dirty_bitmap(s->dirty_bitmap, op->offset, op->bytes);
action = mirror_error_action(s, true, -ret);
if (action == BLOCK_ERROR_ACTION_REPORT && s->ret >= 0) {
s->ret = ret;
@@ -383,8 +381,8 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
* calling bdrv_get_block_status_above could yield - if some blocks are
* marked dirty in this window, we need to know.
*/
- bdrv_reset_dirty_bitmap_locked(s->dirty_bitmap, offset >> BDRV_SECTOR_BITS,
- nb_chunks * sectors_per_chunk);
+ bdrv_reset_dirty_bitmap_locked(s->dirty_bitmap, offset,
+ nb_chunks * s->granularity);
bdrv_dirty_bitmap_unlock(s->dirty_bitmap);
bitmap_set(s->in_flight_bitmap, offset / s->granularity, nb_chunks);
@@ -626,7 +624,7 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
if (base == NULL && !bdrv_has_zero_init(target_bs)) {
if (!bdrv_can_write_zeroes_with_unmap(target_bs)) {
- bdrv_set_dirty_bitmap(s->dirty_bitmap, 0, end);
+ bdrv_set_dirty_bitmap(s->dirty_bitmap, 0, s->bdev_length);
return 0;
}
@@ -682,7 +680,9 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
n = count >> BDRV_SECTOR_BITS;
assert(n > 0);
if (ret == 1) {
- bdrv_set_dirty_bitmap(s->dirty_bitmap, sector_num, n);
+ bdrv_set_dirty_bitmap(s->dirty_bitmap,
+ sector_num * BDRV_SECTOR_SIZE,
+ n * BDRV_SECTOR_SIZE);
}
sector_num += n;
}
diff --git a/migration/block.c b/migration/block.c
index b618869661..725f1974c3 100644
--- a/migration/block.c
+++ b/migration/block.c
@@ -329,7 +329,8 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds)
blk->aiocb = blk_aio_preadv(bb, cur_sector * BDRV_SECTOR_SIZE, &blk->qiov,
0, blk_mig_read_cb, blk);
- bdrv_reset_dirty_bitmap(bmds->dirty_bitmap, cur_sector, nr_sectors);
+ bdrv_reset_dirty_bitmap(bmds->dirty_bitmap, cur_sector * BDRV_SECTOR_SIZE,
+ nr_sectors * BDRV_SECTOR_SIZE);
aio_context_release(blk_get_aio_context(bmds->blk));
qemu_mutex_unlock_iothread();
@@ -537,7 +538,9 @@ static int mig_save_device_dirty(QEMUFile *f, BlkMigDevState *bmds,
} else {
nr_sectors = BDRV_SECTORS_PER_DIRTY_CHUNK;
}
- bdrv_reset_dirty_bitmap_locked(bmds->dirty_bitmap, sector, nr_sectors);
+ bdrv_reset_dirty_bitmap_locked(bmds->dirty_bitmap,
+ sector * BDRV_SECTOR_SIZE,
+ nr_sectors * BDRV_SECTOR_SIZE);
bdrv_dirty_bitmap_unlock(bmds->dirty_bitmap);
blk = g_new(BlkMigBlock, 1);
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 13/18] mirror: Switch mirror_dirty_init() to byte-based iteration
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
` (11 preceding siblings ...)
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 12/18] dirty-bitmap: Change bdrv_[re]set_dirty_bitmap() to use bytes Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 14/18] qcow2: Switch qcow2_measure() " Eric Blake
` (5 subsequent siblings)
18 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, vsementsov, qemu-block, Jeff Cody, Max Reitz
Now that we have adjusted the majority of the calls this function
makes to be byte-based, it is easier to read the code if it makes
passes over the image using bytes rather than sectors.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
v6: no change
v5: rebase to earlier changes
v2-v4: no change
---
block/mirror.c | 38 ++++++++++++++------------------------
1 file changed, 14 insertions(+), 24 deletions(-)
diff --git a/block/mirror.c b/block/mirror.c
index 18172a56a3..87d9857475 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -613,15 +613,13 @@ static void mirror_throttle(MirrorBlockJob *s)
static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
{
- int64_t sector_num, end;
+ int64_t offset;
BlockDriverState *base = s->base;
BlockDriverState *bs = s->source;
BlockDriverState *target_bs = blk_bs(s->target);
- int ret, n;
+ int ret;
int64_t count;
- end = s->bdev_length / BDRV_SECTOR_SIZE;
-
if (base == NULL && !bdrv_has_zero_init(target_bs)) {
if (!bdrv_can_write_zeroes_with_unmap(target_bs)) {
bdrv_set_dirty_bitmap(s->dirty_bitmap, 0, s->bdev_length);
@@ -629,9 +627,9 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
}
s->initial_zeroing_ongoing = true;
- for (sector_num = 0; sector_num < end; ) {
- int nb_sectors = MIN(end - sector_num,
- QEMU_ALIGN_DOWN(INT_MAX, s->granularity) >> BDRV_SECTOR_BITS);
+ for (offset = 0; offset < s->bdev_length; ) {
+ int bytes = MIN(s->bdev_length - offset,
+ QEMU_ALIGN_DOWN(INT_MAX, s->granularity));
mirror_throttle(s);
@@ -647,9 +645,8 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
continue;
}
- mirror_do_zero_or_discard(s, sector_num * BDRV_SECTOR_SIZE,
- nb_sectors * BDRV_SECTOR_SIZE, false);
- sector_num += nb_sectors;
+ mirror_do_zero_or_discard(s, offset, bytes, false);
+ offset += bytes;
}
mirror_wait_for_all_io(s);
@@ -657,10 +654,10 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
}
/* First part, loop on the sectors and initialize the dirty bitmap. */
- for (sector_num = 0; sector_num < end; ) {
+ for (offset = 0; offset < s->bdev_length; ) {
/* Just to make sure we are not exceeding int limit. */
- int nb_sectors = MIN(INT_MAX >> BDRV_SECTOR_BITS,
- end - sector_num);
+ int bytes = MIN(s->bdev_length - offset,
+ QEMU_ALIGN_DOWN(INT_MAX, s->granularity));
mirror_throttle(s);
@@ -668,23 +665,16 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
return 0;
}
- ret = bdrv_is_allocated_above(bs, base, sector_num * BDRV_SECTOR_SIZE,
- nb_sectors * BDRV_SECTOR_SIZE, &count);
+ ret = bdrv_is_allocated_above(bs, base, offset, bytes, &count);
if (ret < 0) {
return ret;
}
- /* TODO: Relax this once bdrv_is_allocated_above and dirty
- * bitmaps no longer require sector alignment. */
- assert(QEMU_IS_ALIGNED(count, BDRV_SECTOR_SIZE));
- n = count >> BDRV_SECTOR_BITS;
- assert(n > 0);
+ assert(count);
if (ret == 1) {
- bdrv_set_dirty_bitmap(s->dirty_bitmap,
- sector_num * BDRV_SECTOR_SIZE,
- n * BDRV_SECTOR_SIZE);
+ bdrv_set_dirty_bitmap(s->dirty_bitmap, offset, count);
}
- sector_num += n;
+ offset += count;
}
return 0;
}
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 14/18] qcow2: Switch qcow2_measure() to byte-based iteration
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
` (12 preceding siblings ...)
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 13/18] mirror: Switch mirror_dirty_init() to byte-based iteration Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-09-08 13:15 ` Kevin Wolf
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 15/18] qcow2: Switch load_bitmap_data() " Eric Blake
` (4 subsequent siblings)
18 siblings, 1 reply; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, vsementsov, qemu-block, Max Reitz
This is new code, but it is easier to read if it makes passes over
the image using bytes rather than sectors (and will get easier in
the future when bdrv_get_block_status is converted to byte-based).
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
v6: separate bug fix to earlier patch
v5: new patch
---
block/qcow2.c | 22 ++++++++++------------
1 file changed, 10 insertions(+), 12 deletions(-)
diff --git a/block/qcow2.c b/block/qcow2.c
index 40ba26c111..57e3c5e7d5 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -3666,20 +3666,19 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
*/
required = virtual_size;
} else {
- int cluster_sectors = cluster_size / BDRV_SECTOR_SIZE;
- int64_t sector_num;
+ int64_t offset;
int pnum = 0;
- for (sector_num = 0;
- sector_num < ssize / BDRV_SECTOR_SIZE;
- sector_num += pnum) {
- int nb_sectors = MIN(ssize / BDRV_SECTOR_SIZE - sector_num,
- BDRV_REQUEST_MAX_SECTORS);
+ for (offset = 0; offset < ssize;
+ offset += pnum * BDRV_SECTOR_SIZE) {
+ int nb_sectors = MIN(ssize - offset,
+ INT_MAX) / BDRV_SECTOR_SIZE;
BlockDriverState *file;
int64_t ret;
ret = bdrv_get_block_status_above(in_bs, NULL,
- sector_num, nb_sectors,
+ offset >> BDRV_SECTOR_BITS,
+ nb_sectors,
&pnum, &file);
if (ret < 0) {
error_setg_errno(&local_err, -ret,
@@ -3692,12 +3691,11 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
} else if ((ret & (BDRV_BLOCK_DATA | BDRV_BLOCK_ALLOCATED)) ==
(BDRV_BLOCK_DATA | BDRV_BLOCK_ALLOCATED)) {
/* Extend pnum to end of cluster for next iteration */
- pnum = ROUND_UP(sector_num + pnum, cluster_sectors) -
- sector_num;
+ pnum = (ROUND_UP(offset + pnum * BDRV_SECTOR_SIZE,
+ cluster_size) - offset) >> BDRV_SECTOR_BITS;
/* Count clusters we've seen */
- required += (sector_num % cluster_sectors + pnum) *
- BDRV_SECTOR_SIZE;
+ required += offset % cluster_size + pnum * BDRV_SECTOR_SIZE;
}
}
}
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 15/18] qcow2: Switch load_bitmap_data() to byte-based iteration
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
` (13 preceding siblings ...)
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 14/18] qcow2: Switch qcow2_measure() " Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 16/18] qcow2: Switch store_bitmap_data() " Eric Blake
` (3 subsequent siblings)
18 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, vsementsov, qemu-block, Max Reitz
Now that we have adjusted the majority of the calls this function
makes to be byte-based, it is easier to read the code if it makes
passes over the image using bytes rather than sectors.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
v5: no change
v4: new patch
---
block/qcow2-bitmap.c | 22 ++++++++--------------
1 file changed, 8 insertions(+), 14 deletions(-)
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index c7c60dfca2..b807298484 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -291,9 +291,8 @@ static int load_bitmap_data(BlockDriverState *bs,
{
int ret = 0;
BDRVQcow2State *s = bs->opaque;
- uint64_t sector, limit, sbc;
+ uint64_t offset, limit;
uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap);
- uint64_t bm_sectors = DIV_ROUND_UP(bm_size, BDRV_SECTOR_SIZE);
uint8_t *buf = NULL;
uint64_t i, tab_size =
size_to_clusters(s,
@@ -305,32 +304,27 @@ static int load_bitmap_data(BlockDriverState *bs,
buf = g_malloc(s->cluster_size);
limit = bytes_covered_by_bitmap_cluster(s, bitmap);
- sbc = limit >> BDRV_SECTOR_BITS;
- for (i = 0, sector = 0; i < tab_size; ++i, sector += sbc) {
- uint64_t count = MIN(bm_sectors - sector, sbc);
+ for (i = 0, offset = 0; i < tab_size; ++i, offset += limit) {
+ uint64_t count = MIN(bm_size - offset, limit);
uint64_t entry = bitmap_table[i];
- uint64_t offset = entry & BME_TABLE_ENTRY_OFFSET_MASK;
+ uint64_t data_offset = entry & BME_TABLE_ENTRY_OFFSET_MASK;
assert(check_table_entry(entry, s->cluster_size) == 0);
- if (offset == 0) {
+ if (data_offset == 0) {
if (entry & BME_TABLE_ENTRY_FLAG_ALL_ONES) {
- bdrv_dirty_bitmap_deserialize_ones(bitmap,
- sector * BDRV_SECTOR_SIZE,
- count * BDRV_SECTOR_SIZE,
+ bdrv_dirty_bitmap_deserialize_ones(bitmap, offset, count,
false);
} else {
/* No need to deserialize zeros because the dirty bitmap is
* already cleared */
}
} else {
- ret = bdrv_pread(bs->file, offset, buf, s->cluster_size);
+ ret = bdrv_pread(bs->file, data_offset, buf, s->cluster_size);
if (ret < 0) {
goto finish;
}
- bdrv_dirty_bitmap_deserialize_part(bitmap, buf,
- sector * BDRV_SECTOR_SIZE,
- count * BDRV_SECTOR_SIZE,
+ bdrv_dirty_bitmap_deserialize_part(bitmap, buf, offset, count,
false);
}
}
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 16/18] qcow2: Switch store_bitmap_data() to byte-based iteration
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
` (14 preceding siblings ...)
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 15/18] qcow2: Switch load_bitmap_data() " Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-09-08 13:27 ` Kevin Wolf
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 17/18] dirty-bitmap: Switch bdrv_set_dirty() to bytes Eric Blake
` (2 subsequent siblings)
18 siblings, 1 reply; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, vsementsov, qemu-block, Max Reitz
Now that we have adjusted the majority of the calls this function
makes to be byte-based, it is easier to read the code if it makes
passes over the image using bytes rather than sectors.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
v5: no change
v4: new patch
---
block/qcow2-bitmap.c | 26 +++++++++++---------------
1 file changed, 11 insertions(+), 15 deletions(-)
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index b807298484..63d845e35f 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -1072,10 +1072,9 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
{
int ret;
BDRVQcow2State *s = bs->opaque;
- int64_t sector;
- uint64_t limit, sbc;
+ int64_t offset;
+ uint64_t limit;
uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap);
- uint64_t bm_sectors = DIV_ROUND_UP(bm_size, BDRV_SECTOR_SIZE);
const char *bm_name = bdrv_dirty_bitmap_name(bitmap);
uint8_t *buf = NULL;
BdrvDirtyBitmapIter *dbi;
@@ -1100,18 +1099,17 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
dbi = bdrv_dirty_iter_new(bitmap);
buf = g_malloc(s->cluster_size);
limit = bytes_covered_by_bitmap_cluster(s, bitmap);
- sbc = limit >> BDRV_SECTOR_BITS;
assert(DIV_ROUND_UP(bm_size, limit) == tb_size);
- while ((sector = bdrv_dirty_iter_next(dbi) >> BDRV_SECTOR_BITS) != -1) {
- uint64_t cluster = sector / sbc;
+ while ((offset = bdrv_dirty_iter_next(dbi)) != -1) {
+ uint64_t cluster = offset / limit;
uint64_t end, write_size;
int64_t off;
- sector = cluster * sbc;
- end = MIN(bm_sectors, sector + sbc);
- write_size = bdrv_dirty_bitmap_serialization_size(bitmap,
- sector * BDRV_SECTOR_SIZE, (end - sector) * BDRV_SECTOR_SIZE);
+ offset = cluster * limit;
+ end = MIN(bm_size, offset + limit);
+ write_size = bdrv_dirty_bitmap_serialization_size(bitmap, offset,
+ end - offset);
assert(write_size <= s->cluster_size);
off = qcow2_alloc_clusters(bs, s->cluster_size);
@@ -1123,9 +1121,7 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
}
tb[cluster] = off;
- bdrv_dirty_bitmap_serialize_part(bitmap, buf,
- sector * BDRV_SECTOR_SIZE,
- (end - sector) * BDRV_SECTOR_SIZE);
+ bdrv_dirty_bitmap_serialize_part(bitmap, buf, offset, end - offset);
if (write_size < s->cluster_size) {
memset(buf + write_size, 0, s->cluster_size - write_size);
}
@@ -1143,11 +1139,11 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
goto fail;
}
- if (end >= bm_sectors) {
+ if (end >= bm_size) {
break;
}
- bdrv_set_dirty_iter(dbi, end * BDRV_SECTOR_SIZE);
+ bdrv_set_dirty_iter(dbi, end);
}
*bitmap_table_size = tb_size;
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 17/18] dirty-bitmap: Switch bdrv_set_dirty() to bytes
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
` (15 preceding siblings ...)
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 16/18] qcow2: Switch store_bitmap_data() " Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 18/18] dirty-bitmap: Convert internal hbitmap size/granularity Eric Blake
2017-08-30 22:43 ` [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based John Snow
18 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel
Cc: kwolf, jsnow, vsementsov, qemu-block, Fam Zheng, Max Reitz,
Stefan Hajnoczi
Both callers already had bytes available, but were scaling to
sectors. Move the scaling to internal code. In the case of
bdrv_aligned_pwritev(), we are now passing the exact offset
rather than a rounded sector-aligned value, but that's okay
as long as dirty bitmap widens start/bytes to granularity
boundaries.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
v4: only context changes
v3: rebase to lock context changes, R-b kept
v2: no change
---
include/block/block_int.h | 2 +-
block/io.c | 6 ++----
block/dirty-bitmap.c | 7 ++++---
3 files changed, 7 insertions(+), 8 deletions(-)
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 7571c0aaaf..4d01dc3fa6 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -969,7 +969,7 @@ void blk_dev_eject_request(BlockBackend *blk, bool force);
bool blk_dev_is_tray_open(BlockBackend *blk);
bool blk_dev_is_medium_locked(BlockBackend *blk);
-void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector, int64_t nr_sect);
+void bdrv_set_dirty(BlockDriverState *bs, int64_t offset, int64_t bytes);
bool bdrv_requests_pending(BlockDriverState *bs);
void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap **out);
diff --git a/block/io.c b/block/io.c
index 26003814eb..926beebc8f 100644
--- a/block/io.c
+++ b/block/io.c
@@ -1334,7 +1334,6 @@ static int coroutine_fn bdrv_aligned_pwritev(BdrvChild *child,
bool waited;
int ret;
- int64_t start_sector = offset >> BDRV_SECTOR_BITS;
int64_t end_sector = DIV_ROUND_UP(offset + bytes, BDRV_SECTOR_SIZE);
uint64_t bytes_remaining = bytes;
int max_transfer;
@@ -1409,7 +1408,7 @@ static int coroutine_fn bdrv_aligned_pwritev(BdrvChild *child,
bdrv_debug_event(bs, BLKDBG_PWRITEV_DONE);
atomic_inc(&bs->write_gen);
- bdrv_set_dirty(bs, start_sector, end_sector - start_sector);
+ bdrv_set_dirty(bs, offset, bytes);
stat64_max(&bs->wr_highest_offset, offset + bytes);
@@ -2412,8 +2411,7 @@ int coroutine_fn bdrv_co_pdiscard(BlockDriverState *bs, int64_t offset,
ret = 0;
out:
atomic_inc(&bs->write_gen);
- bdrv_set_dirty(bs, req.offset >> BDRV_SECTOR_BITS,
- req.bytes >> BDRV_SECTOR_BITS);
+ bdrv_set_dirty(bs, req.offset, req.bytes);
tracked_request_end(&req);
bdrv_dec_in_flight(bs);
return ret;
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 9821225523..b54eed46e4 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -629,10 +629,10 @@ void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap)
hbitmap_deserialize_finish(bitmap->bitmap);
}
-void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
- int64_t nr_sectors)
+void bdrv_set_dirty(BlockDriverState *bs, int64_t offset, int64_t bytes)
{
BdrvDirtyBitmap *bitmap;
+ int64_t end_sector = DIV_ROUND_UP(offset + bytes, BDRV_SECTOR_SIZE);
if (QLIST_EMPTY(&bs->dirty_bitmaps)) {
return;
@@ -644,7 +644,8 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
continue;
}
assert(!bdrv_dirty_bitmap_readonly(bitmap));
- hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
+ hbitmap_set(bitmap->bitmap, offset >> BDRV_SECTOR_BITS,
+ end_sector - (offset >> BDRV_SECTOR_BITS));
}
bdrv_dirty_bitmaps_unlock(bs);
}
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH v6 18/18] dirty-bitmap: Convert internal hbitmap size/granularity
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
` (16 preceding siblings ...)
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 17/18] dirty-bitmap: Switch bdrv_set_dirty() to bytes Eric Blake
@ 2017-08-30 21:05 ` Eric Blake
2017-08-30 22:43 ` [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based John Snow
18 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-08-30 21:05 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, jsnow, vsementsov, qemu-block, Fam Zheng, Max Reitz
Now that all callers are using byte-based interfaces, there's no
reason for our internal hbitmap to remain with sector-based
granularity. It also simplifies our internal scaling, since we
already know that hbitmap widens requests out to granularity
boundaries.
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
---
v6: no change
v5: fix bdrv_dirty_bitmap_truncate [John]
v4: rebase to earlier changes, include serialization, R-b dropped
v3: no change
v2: no change
---
block/dirty-bitmap.c | 61 ++++++++++++++++------------------------------------
1 file changed, 18 insertions(+), 43 deletions(-)
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index b54eed46e4..0b349f0b5a 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -38,7 +38,7 @@
*/
struct BdrvDirtyBitmap {
QemuMutex *mutex;
- HBitmap *bitmap; /* Dirty sector bitmap implementation */
+ HBitmap *bitmap; /* Dirty bitmap implementation */
HBitmap *meta; /* Meta dirty bitmap */
BdrvDirtyBitmap *successor; /* Anonymous child; implies frozen status */
char *name; /* Optional non-empty unique ID */
@@ -130,12 +130,7 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
}
bitmap = g_new0(BdrvDirtyBitmap, 1);
bitmap->mutex = &bs->dirty_bitmap_mutex;
- /*
- * TODO - let hbitmap track full granularity. For now, it is tracking
- * only sector granularity, as a shortcut for our iterators.
- */
- bitmap->bitmap = hbitmap_alloc(DIV_ROUND_UP(bitmap_size, BDRV_SECTOR_SIZE),
- ctz32(granularity) - BDRV_SECTOR_BITS);
+ bitmap->bitmap = hbitmap_alloc(bitmap_size, ctz32(granularity));
bitmap->size = bitmap_size;
bitmap->name = g_strdup(name);
bitmap->disabled = false;
@@ -314,7 +309,7 @@ void bdrv_dirty_bitmap_truncate(BlockDriverState *bs)
QLIST_FOREACH(bitmap, &bs->dirty_bitmaps, list) {
assert(!bdrv_dirty_bitmap_frozen(bitmap));
assert(!bitmap->active_iterators);
- hbitmap_truncate(bitmap->bitmap, DIV_ROUND_UP(size, BDRV_SECTOR_SIZE));
+ hbitmap_truncate(bitmap->bitmap, size);
bitmap->size = size;
}
bdrv_dirty_bitmaps_unlock(bs);
@@ -444,7 +439,7 @@ bool bdrv_get_dirty_locked(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
int64_t offset)
{
if (bitmap) {
- return hbitmap_get(bitmap->bitmap, offset >> BDRV_SECTOR_BITS);
+ return hbitmap_get(bitmap->bitmap, offset);
} else {
return false;
}
@@ -472,7 +467,7 @@ uint32_t bdrv_get_default_bitmap_granularity(BlockDriverState *bs)
uint32_t bdrv_dirty_bitmap_granularity(const BdrvDirtyBitmap *bitmap)
{
- return BDRV_SECTOR_SIZE << hbitmap_granularity(bitmap->bitmap);
+ return 1U << hbitmap_granularity(bitmap->bitmap);
}
BdrvDirtyBitmapIter *bdrv_dirty_iter_new(BdrvDirtyBitmap *bitmap)
@@ -505,19 +500,16 @@ void bdrv_dirty_iter_free(BdrvDirtyBitmapIter *iter)
int64_t bdrv_dirty_iter_next(BdrvDirtyBitmapIter *iter)
{
- return hbitmap_iter_next(&iter->hbi) * BDRV_SECTOR_SIZE;
+ return hbitmap_iter_next(&iter->hbi);
}
/* Called within bdrv_dirty_bitmap_lock..unlock */
void bdrv_set_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
int64_t offset, int64_t bytes)
{
- int64_t end_sector = DIV_ROUND_UP(offset + bytes, BDRV_SECTOR_SIZE);
-
assert(bdrv_dirty_bitmap_enabled(bitmap));
assert(!bdrv_dirty_bitmap_readonly(bitmap));
- hbitmap_set(bitmap->bitmap, offset >> BDRV_SECTOR_BITS,
- end_sector - (offset >> BDRV_SECTOR_BITS));
+ hbitmap_set(bitmap->bitmap, offset, bytes);
}
void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
@@ -532,12 +524,9 @@ void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
void bdrv_reset_dirty_bitmap_locked(BdrvDirtyBitmap *bitmap,
int64_t offset, int64_t bytes)
{
- int64_t end_sector = DIV_ROUND_UP(offset + bytes, BDRV_SECTOR_SIZE);
-
assert(bdrv_dirty_bitmap_enabled(bitmap));
assert(!bdrv_dirty_bitmap_readonly(bitmap));
- hbitmap_reset(bitmap->bitmap, offset >> BDRV_SECTOR_BITS,
- end_sector - (offset >> BDRV_SECTOR_BITS));
+ hbitmap_reset(bitmap->bitmap, offset, bytes);
}
void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
@@ -557,8 +546,7 @@ void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap **out)
hbitmap_reset_all(bitmap->bitmap);
} else {
HBitmap *backup = bitmap->bitmap;
- bitmap->bitmap = hbitmap_alloc(DIV_ROUND_UP(bitmap->size,
- BDRV_SECTOR_SIZE),
+ bitmap->bitmap = hbitmap_alloc(bitmap->size,
hbitmap_granularity(backup));
*out = backup;
}
@@ -577,51 +565,40 @@ void bdrv_undo_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap *in)
uint64_t bdrv_dirty_bitmap_serialization_size(const BdrvDirtyBitmap *bitmap,
uint64_t offset, uint64_t bytes)
{
- assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE));
- return hbitmap_serialization_size(bitmap->bitmap,
- offset >> BDRV_SECTOR_BITS,
- bytes >> BDRV_SECTOR_BITS);
+ return hbitmap_serialization_size(bitmap->bitmap, offset, bytes);
}
uint64_t bdrv_dirty_bitmap_serialization_align(const BdrvDirtyBitmap *bitmap)
{
- return hbitmap_serialization_align(bitmap->bitmap) * BDRV_SECTOR_SIZE;
+ return hbitmap_serialization_align(bitmap->bitmap);
}
void bdrv_dirty_bitmap_serialize_part(const BdrvDirtyBitmap *bitmap,
uint8_t *buf, uint64_t offset,
uint64_t bytes)
{
- assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE));
- hbitmap_serialize_part(bitmap->bitmap, buf, offset >> BDRV_SECTOR_BITS,
- bytes >> BDRV_SECTOR_BITS);
+ hbitmap_serialize_part(bitmap->bitmap, buf, offset, bytes);
}
void bdrv_dirty_bitmap_deserialize_part(BdrvDirtyBitmap *bitmap,
uint8_t *buf, uint64_t offset,
uint64_t bytes, bool finish)
{
- assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE));
- hbitmap_deserialize_part(bitmap->bitmap, buf, offset >> BDRV_SECTOR_BITS,
- bytes >> BDRV_SECTOR_BITS, finish);
+ hbitmap_deserialize_part(bitmap->bitmap, buf, offset, bytes, finish);
}
void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap,
uint64_t offset, uint64_t bytes,
bool finish)
{
- assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE));
- hbitmap_deserialize_zeroes(bitmap->bitmap, offset >> BDRV_SECTOR_BITS,
- bytes >> BDRV_SECTOR_BITS, finish);
+ hbitmap_deserialize_zeroes(bitmap->bitmap, offset, bytes, finish);
}
void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap,
uint64_t offset, uint64_t bytes,
bool finish)
{
- assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE));
- hbitmap_deserialize_ones(bitmap->bitmap, offset >> BDRV_SECTOR_BITS,
- bytes >> BDRV_SECTOR_BITS, finish);
+ hbitmap_deserialize_ones(bitmap->bitmap, offset, bytes, finish);
}
void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap)
@@ -632,7 +609,6 @@ void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap)
void bdrv_set_dirty(BlockDriverState *bs, int64_t offset, int64_t bytes)
{
BdrvDirtyBitmap *bitmap;
- int64_t end_sector = DIV_ROUND_UP(offset + bytes, BDRV_SECTOR_SIZE);
if (QLIST_EMPTY(&bs->dirty_bitmaps)) {
return;
@@ -644,8 +620,7 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t offset, int64_t bytes)
continue;
}
assert(!bdrv_dirty_bitmap_readonly(bitmap));
- hbitmap_set(bitmap->bitmap, offset >> BDRV_SECTOR_BITS,
- end_sector - (offset >> BDRV_SECTOR_BITS));
+ hbitmap_set(bitmap->bitmap, offset, bytes);
}
bdrv_dirty_bitmaps_unlock(bs);
}
@@ -655,12 +630,12 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t offset, int64_t bytes)
*/
void bdrv_set_dirty_iter(BdrvDirtyBitmapIter *iter, int64_t offset)
{
- hbitmap_iter_init(&iter->hbi, iter->hbi.hb, offset >> BDRV_SECTOR_BITS);
+ hbitmap_iter_init(&iter->hbi, iter->hbi.hb, offset);
}
int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap)
{
- return hbitmap_count(bitmap->bitmap) << BDRV_SECTOR_BITS;
+ return hbitmap_count(bitmap->bitmap);
}
int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap)
--
2.13.5
^ permalink raw reply related [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
` (17 preceding siblings ...)
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 18/18] dirty-bitmap: Convert internal hbitmap size/granularity Eric Blake
@ 2017-08-30 22:43 ` John Snow
18 siblings, 0 replies; 30+ messages in thread
From: John Snow @ 2017-08-30 22:43 UTC (permalink / raw)
To: Eric Blake, qemu-devel; +Cc: kwolf, vsementsov, qemu-block
On 08/30/2017 05:05 PM, Eric Blake wrote:
> There are patches floating around to add NBD_CMD_BLOCK_STATUS,
> but NBD wants to report status on byte granularity (even if the
> reporting will probably be naturally aligned to sectors or even
> much higher levels). I've therefore started the task of
> converting our block status code to report at a byte granularity
> rather than sectors.
>
> Now that 2.11 is open, I'm rebasing/reposting the remaining patches.
>
> The overall conversion currently looks like:
> part 1: bdrv_is_allocated (merged in 2.10, commit 51b0a488)
> part 2: dirty-bitmap (this series, v5 was here [1])
> part 3: bdrv_get_block_status (v3 is posted [2] and is mostly reviewed, but
> needs a rebase)
> part 4: .bdrv_co_block_status (v2 is posted [3], but needs a rebase)
>
> Available as a tag at:
> git fetch git://repo.or.cz/qemu/ericb.git nbd-byte-dirty-v6
>
> [1] https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg03512.html
> [2] https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg03853.html
> [3] https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg04370.html
>
> Diff from v5:
> - add another patch (more for ease of bookkeeping, as it was previously
> posted independently)
> - drop bug fixes that were hoisted into 2.10 (v5 1/18, plus 14/18)
>
> 001/18:[down] 'block: Make bdrv_img_create() size selection easier to read'
> 002/18:[----] [--] 'hbitmap: Rename serialization_granularity to serialization_align'
> 003/18:[----] [--] 'qcow2: Ensure bitmap serialization is aligned'
> 004/18:[----] [--] 'dirty-bitmap: Drop unused functions'
> 005/18:[----] [--] 'dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes'
> 006/18:[----] [--] 'dirty-bitmap: Change bdrv_dirty_bitmap_*serialize*() to take bytes'
> 007/18:[----] [--] 'qcow2: Switch sectors_covered_by_bitmap_cluster() to byte-based'
> 008/18:[----] [--] 'dirty-bitmap: Set iterator start by offset, not sector'
> 009/18:[----] [--] 'dirty-bitmap: Change bdrv_dirty_iter_next() to report byte offset'
> 010/18:[----] [--] 'dirty-bitmap: Change bdrv_get_dirty_count() to report bytes'
> 011/18:[----] [--] 'dirty-bitmap: Change bdrv_get_dirty_locked() to take bytes'
> 012/18:[----] [--] 'dirty-bitmap: Change bdrv_[re]set_dirty_bitmap() to use bytes'
> 013/18:[----] [--] 'mirror: Switch mirror_dirty_init() to byte-based iteration'
> 014/18:[0004] [FC] 'qcow2: Switch qcow2_measure() to byte-based iteration'
> 015/18:[----] [--] 'qcow2: Switch load_bitmap_data() to byte-based iteration'
> 016/18:[----] [--] 'qcow2: Switch store_bitmap_data() to byte-based iteration'
> 017/18:[----] [--] 'dirty-bitmap: Switch bdrv_set_dirty() to bytes'
> 018/18:[----] [--] 'dirty-bitmap: Convert internal hbitmap size/granularity'
>
> Eric Blake (18):
> block: Make bdrv_img_create() size selection easier to read
> hbitmap: Rename serialization_granularity to serialization_align
> qcow2: Ensure bitmap serialization is aligned
> dirty-bitmap: Drop unused functions
> dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes
> dirty-bitmap: Change bdrv_dirty_bitmap_*serialize*() to take bytes
> qcow2: Switch sectors_covered_by_bitmap_cluster() to byte-based
> dirty-bitmap: Set iterator start by offset, not sector
> dirty-bitmap: Change bdrv_dirty_iter_next() to report byte offset
> dirty-bitmap: Change bdrv_get_dirty_count() to report bytes
> dirty-bitmap: Change bdrv_get_dirty_locked() to take bytes
> dirty-bitmap: Change bdrv_[re]set_dirty_bitmap() to use bytes
> mirror: Switch mirror_dirty_init() to byte-based iteration
> qcow2: Switch qcow2_measure() to byte-based iteration
> qcow2: Switch load_bitmap_data() to byte-based iteration
> qcow2: Switch store_bitmap_data() to byte-based iteration
> dirty-bitmap: Switch bdrv_set_dirty() to bytes
> dirty-bitmap: Convert internal hbitmap size/granularity
>
> include/block/block_int.h | 2 +-
> include/block/dirty-bitmap.h | 41 +++++---------
> include/qemu/hbitmap.h | 8 +--
> block/io.c | 6 +-
> block.c | 2 +-
> block/backup.c | 7 +--
> block/dirty-bitmap.c | 130 ++++++++++++++-----------------------------
> block/mirror.c | 76 +++++++++++--------------
> block/qcow2-bitmap.c | 57 +++++++++----------
> block/qcow2.c | 22 ++++----
> migration/block.c | 12 ++--
> tests/test-hbitmap.c | 10 ++--
> util/hbitmap.c | 8 +--
> 13 files changed, 154 insertions(+), 227 deletions(-)
>
Should this go through the bitmap tree, or since it's touching qcow2,
I'll let Kevin/Max/Stefan stage it?
--js
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH v6 05/18] dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 05/18] dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes Eric Blake
@ 2017-09-08 12:22 ` Kevin Wolf
2017-09-08 14:04 ` Eric Blake
0 siblings, 1 reply; 30+ messages in thread
From: Kevin Wolf @ 2017-09-08 12:22 UTC (permalink / raw)
To: Eric Blake
Cc: qemu-devel, jsnow, vsementsov, qemu-block, Fam Zheng, Max Reitz
Am 30.08.2017 um 23:05 hat Eric Blake geschrieben:
> We are still using an internal hbitmap that tracks a size in sectors,
> with the granularity scaled down accordingly, because it lets us
> use a shortcut for our iterators which are currently sector-based.
> But there's no reason we can't track the dirty bitmap size in bytes,
> since it is (mostly) an internal-only variable (remember, the size
> is how many bytes are covered by the bitmap, not how many bytes the
> bitmap occupies). Furthermore, we're already reporting bytes for
> bdrv_dirty_bitmap_granularity(); mixing bytes and sectors in our
> return values is a recipe for confusion. A later cleanup will
> convert dirty bitmap internals to be entirely byte-based,
> eliminating the intermediate sector rounding added here; and
> technically, since bdrv_getlength() already rounds up to sectors,
> our use of DIV_ROUND_UP is more for theoretical completeness than
> for any actual rounding.
>
> The only external caller in qcow2-bitmap.c is temporarily more verbose
> (because it is still using sector-based math), but will later be
> switched to track progress by bytes instead of sectors.
>
> Use is_power_of_2() while at it, instead of open-coding that, and
> add an assertion where bdrv_getlength() should not fail.
>
> Signed-off-by: Eric Blake <eblake@redhat.com>
> Reviewed-by: John Snow <jsnow@redhat.com>
I think I would have preferred to change the unit of
BdrvDirtyBitmap.size in one patch and the unit of the return value of
bdrv_dirty_bitmap_size() in another one to keep review a bit easier.
> block/dirty-bitmap.c | 26 +++++++++++++++-----------
> block/qcow2-bitmap.c | 14 ++++++++------
> 2 files changed, 23 insertions(+), 17 deletions(-)
>
> diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
> index 42a55e4a4b..e65ec4f7ec 100644
> --- a/block/dirty-bitmap.c
> +++ b/block/dirty-bitmap.c
> @@ -1,7 +1,7 @@
> /*
> * Block Dirty Bitmap
> *
> - * Copyright (c) 2016 Red Hat. Inc
> + * Copyright (c) 2016-2017 Red Hat. Inc
> *
> * Permission is hereby granted, free of charge, to any person obtaining a copy
> * of this software and associated documentation files (the "Software"), to deal
> @@ -42,7 +42,7 @@ struct BdrvDirtyBitmap {
> HBitmap *meta; /* Meta dirty bitmap */
> BdrvDirtyBitmap *successor; /* Anonymous child; implies frozen status */
> char *name; /* Optional non-empty unique ID */
> - int64_t size; /* Size of the bitmap (Number of sectors) */
> + int64_t size; /* Size of the bitmap, in bytes */
> bool disabled; /* Bitmap is disabled. It ignores all writes to
> the device */
> int active_iterators; /* How many iterators are active */
> @@ -115,17 +115,14 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
> {
> int64_t bitmap_size;
> BdrvDirtyBitmap *bitmap;
> - uint32_t sector_granularity;
>
> - assert((granularity & (granularity - 1)) == 0);
> + assert(is_power_of_2(granularity) && granularity >= BDRV_SECTOR_SIZE);
>
> if (name && bdrv_find_dirty_bitmap(bs, name)) {
> error_setg(errp, "Bitmap already exists: %s", name);
> return NULL;
> }
> - sector_granularity = granularity >> BDRV_SECTOR_BITS;
> - assert(sector_granularity);
> - bitmap_size = bdrv_nb_sectors(bs);
> + bitmap_size = bdrv_getlength(bs);
> if (bitmap_size < 0) {
> error_setg_errno(errp, -bitmap_size, "could not get length of device");
> errno = -bitmap_size;
> @@ -133,7 +130,12 @@ BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
> }
> bitmap = g_new0(BdrvDirtyBitmap, 1);
> bitmap->mutex = &bs->dirty_bitmap_mutex;
> - bitmap->bitmap = hbitmap_alloc(bitmap_size, ctz32(sector_granularity));
> + /*
> + * TODO - let hbitmap track full granularity. For now, it is tracking
> + * only sector granularity, as a shortcut for our iterators.
> + */
> + bitmap->bitmap = hbitmap_alloc(DIV_ROUND_UP(bitmap_size, BDRV_SECTOR_SIZE),
> + ctz32(granularity) - BDRV_SECTOR_BITS);
> bitmap->size = bitmap_size;
> bitmap->name = g_strdup(name);
> bitmap->disabled = false;
> @@ -305,13 +307,14 @@ BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
> void bdrv_dirty_bitmap_truncate(BlockDriverState *bs)
> {
> BdrvDirtyBitmap *bitmap;
> - uint64_t size = bdrv_nb_sectors(bs);
> + int64_t size = bdrv_getlength(bs);
>
> + assert(size >= 0);
How can you assert that there will never be an error? Even if it's
correct (I don't know whether you can have dirty bitmaps on devices that
don't use the cached value), this needs at least a comment.
The rest looks okay.
Kevin
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH v6 10/18] dirty-bitmap: Change bdrv_get_dirty_count() to report bytes
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 10/18] dirty-bitmap: Change bdrv_get_dirty_count() to report bytes Eric Blake
@ 2017-09-08 12:51 ` Kevin Wolf
2017-09-08 14:05 ` Eric Blake
0 siblings, 1 reply; 30+ messages in thread
From: Kevin Wolf @ 2017-09-08 12:51 UTC (permalink / raw)
To: Eric Blake
Cc: qemu-devel, jsnow, vsementsov, qemu-block, Fam Zheng, Max Reitz,
Jeff Cody, Stefan Hajnoczi, Juan Quintela,
Dr. David Alan Gilbert
Am 30.08.2017 um 23:05 hat Eric Blake geschrieben:
> Thanks to recent cleanups, all callers were scaling a return value
> of sectors into bytes; do the scaling internally instead.
>
> Signed-off-by: Eric Blake <eblake@redhat.com>
> Reviewed-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Juan Quintela <quintela@redhat.com>
> diff --git a/block/mirror.c b/block/mirror.c
> index af13f5d658..cc47e21814 100644
> --- a/block/mirror.c
> +++ b/block/mirror.c
> @@ -811,11 +811,10 @@ static void coroutine_fn mirror_run(void *opaque)
There is one more place before this one that needs to be converted,
mirror_iteration() at line 343:
trace_mirror_restart_iter(s, bdrv_get_dirty_count(s->dirty_bitmap) *
BDRV_SECTOR_SIZE);
Kevin
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH v6 14/18] qcow2: Switch qcow2_measure() to byte-based iteration
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 14/18] qcow2: Switch qcow2_measure() " Eric Blake
@ 2017-09-08 13:15 ` Kevin Wolf
2017-09-08 13:43 ` Eric Blake
0 siblings, 1 reply; 30+ messages in thread
From: Kevin Wolf @ 2017-09-08 13:15 UTC (permalink / raw)
To: Eric Blake; +Cc: qemu-devel, jsnow, vsementsov, qemu-block, Max Reitz
Am 30.08.2017 um 23:05 hat Eric Blake geschrieben:
> This is new code, but it is easier to read if it makes passes over
> the image using bytes rather than sectors (and will get easier in
> the future when bdrv_get_block_status is converted to byte-based).
>
> Signed-off-by: Eric Blake <eblake@redhat.com>
> Reviewed-by: John Snow <jsnow@redhat.com>
>
> ---
> v6: separate bug fix to earlier patch
> v5: new patch
> ---
> block/qcow2.c | 22 ++++++++++------------
> 1 file changed, 10 insertions(+), 12 deletions(-)
>
> diff --git a/block/qcow2.c b/block/qcow2.c
> index 40ba26c111..57e3c5e7d5 100644
> --- a/block/qcow2.c
> +++ b/block/qcow2.c
> @@ -3666,20 +3666,19 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
> */
> required = virtual_size;
> } else {
> - int cluster_sectors = cluster_size / BDRV_SECTOR_SIZE;
> - int64_t sector_num;
> + int64_t offset;
> int pnum = 0;
>
> - for (sector_num = 0;
> - sector_num < ssize / BDRV_SECTOR_SIZE;
> - sector_num += pnum) {
> - int nb_sectors = MIN(ssize / BDRV_SECTOR_SIZE - sector_num,
> - BDRV_REQUEST_MAX_SECTORS);
> + for (offset = 0; offset < ssize;
> + offset += pnum * BDRV_SECTOR_SIZE) {
> + int nb_sectors = MIN(ssize - offset,
> + INT_MAX) / BDRV_SECTOR_SIZE;
Shouldn't this be BDRV_REQUEST_MAX_BYTES? (Which is close to INT_MAX,
but rounded down to sector alignment.)
Kevin
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH v6 16/18] qcow2: Switch store_bitmap_data() to byte-based iteration
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 16/18] qcow2: Switch store_bitmap_data() " Eric Blake
@ 2017-09-08 13:27 ` Kevin Wolf
2017-09-08 14:09 ` Eric Blake
0 siblings, 1 reply; 30+ messages in thread
From: Kevin Wolf @ 2017-09-08 13:27 UTC (permalink / raw)
To: Eric Blake; +Cc: qemu-devel, jsnow, vsementsov, qemu-block, Max Reitz
Am 30.08.2017 um 23:05 hat Eric Blake geschrieben:
> Now that we have adjusted the majority of the calls this function
> makes to be byte-based, it is easier to read the code if it makes
> passes over the image using bytes rather than sectors.
>
> Signed-off-by: Eric Blake <eblake@redhat.com>
> Reviewed-by: John Snow <jsnow@redhat.com>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>
> ---
> v5: no change
> v4: new patch
> ---
> block/qcow2-bitmap.c | 26 +++++++++++---------------
> 1 file changed, 11 insertions(+), 15 deletions(-)
>
> diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
> index b807298484..63d845e35f 100644
> --- a/block/qcow2-bitmap.c
> +++ b/block/qcow2-bitmap.c
> @@ -1072,10 +1072,9 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
> {
> int ret;
> BDRVQcow2State *s = bs->opaque;
> - int64_t sector;
> - uint64_t limit, sbc;
> + int64_t offset;
> + uint64_t limit;
> uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap);
> - uint64_t bm_sectors = DIV_ROUND_UP(bm_size, BDRV_SECTOR_SIZE);
> const char *bm_name = bdrv_dirty_bitmap_name(bitmap);
> uint8_t *buf = NULL;
> BdrvDirtyBitmapIter *dbi;
> @@ -1100,18 +1099,17 @@ static uint64_t *store_bitmap_data(BlockDriverState *bs,
> dbi = bdrv_dirty_iter_new(bitmap);
> buf = g_malloc(s->cluster_size);
> limit = bytes_covered_by_bitmap_cluster(s, bitmap);
> - sbc = limit >> BDRV_SECTOR_BITS;
> assert(DIV_ROUND_UP(bm_size, limit) == tb_size);
>
> - while ((sector = bdrv_dirty_iter_next(dbi) >> BDRV_SECTOR_BITS) != -1) {
> - uint64_t cluster = sector / sbc;
> + while ((offset = bdrv_dirty_iter_next(dbi)) != -1) {
Don't you have to multiply both sides of the equation? This would be
offset != -512, which points out that the previous patch to convert
bdrv_dirty_iter_next() to byte-based gave it a really awkward interface.
> + uint64_t cluster = offset / limit;
> uint64_t end, write_size;
> int64_t off;
>
> - sector = cluster * sbc;
> - end = MIN(bm_sectors, sector + sbc);
> - write_size = bdrv_dirty_bitmap_serialization_size(bitmap,
> - sector * BDRV_SECTOR_SIZE, (end - sector) * BDRV_SECTOR_SIZE);
> + offset = cluster * limit;
You just had cluster = offset / limit, so in other words, align down
offset? If so, this is how it should be written.
Kevin
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH v6 14/18] qcow2: Switch qcow2_measure() to byte-based iteration
2017-09-08 13:15 ` Kevin Wolf
@ 2017-09-08 13:43 ` Eric Blake
0 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-09-08 13:43 UTC (permalink / raw)
To: Kevin Wolf; +Cc: qemu-devel, jsnow, vsementsov, qemu-block, Max Reitz
[-- Attachment #1: Type: text/plain, Size: 1322 bytes --]
On 09/08/2017 08:15 AM, Kevin Wolf wrote:
> Am 30.08.2017 um 23:05 hat Eric Blake geschrieben:
>> This is new code, but it is easier to read if it makes passes over
>> the image using bytes rather than sectors (and will get easier in
>> the future when bdrv_get_block_status is converted to byte-based).
>>
>> Signed-off-by: Eric Blake <eblake@redhat.com>
>> Reviewed-by: John Snow <jsnow@redhat.com>
>>
>> - int nb_sectors = MIN(ssize / BDRV_SECTOR_SIZE - sector_num,
>> - BDRV_REQUEST_MAX_SECTORS);
>> + for (offset = 0; offset < ssize;
>> + offset += pnum * BDRV_SECTOR_SIZE) {
>> + int nb_sectors = MIN(ssize - offset,
>> + INT_MAX) / BDRV_SECTOR_SIZE;
>
> Shouldn't this be BDRV_REQUEST_MAX_BYTES? (Which is close to INT_MAX,
> but rounded down to sector alignment.)
The division rounds down to sector alignment after the MIN(), for the
same result in nb_sectors either way. But you are correct that an
absolutely literal translation of the pre-patch version would feed
BDRV_REQUEST_MAX_BYTES instead of INT_MAX into the MIN().
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH v6 05/18] dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes
2017-09-08 12:22 ` Kevin Wolf
@ 2017-09-08 14:04 ` Eric Blake
2017-09-12 19:35 ` Eric Blake
0 siblings, 1 reply; 30+ messages in thread
From: Eric Blake @ 2017-09-08 14:04 UTC (permalink / raw)
To: Kevin Wolf
Cc: qemu-devel, jsnow, vsementsov, qemu-block, Fam Zheng, Max Reitz
[-- Attachment #1: Type: text/plain, Size: 2787 bytes --]
On 09/08/2017 07:22 AM, Kevin Wolf wrote:
> Am 30.08.2017 um 23:05 hat Eric Blake geschrieben:
>> We are still using an internal hbitmap that tracks a size in sectors,
>> with the granularity scaled down accordingly, because it lets us
>> use a shortcut for our iterators which are currently sector-based.
>> But there's no reason we can't track the dirty bitmap size in bytes,
>> since it is (mostly) an internal-only variable (remember, the size
>> is how many bytes are covered by the bitmap, not how many bytes the
>> bitmap occupies). Furthermore, we're already reporting bytes for
>> bdrv_dirty_bitmap_granularity(); mixing bytes and sectors in our
>> return values is a recipe for confusion. A later cleanup will
>> convert dirty bitmap internals to be entirely byte-based,
>> eliminating the intermediate sector rounding added here; and
>> technically, since bdrv_getlength() already rounds up to sectors,
>> our use of DIV_ROUND_UP is more for theoretical completeness than
>> for any actual rounding.
>>
>> The only external caller in qcow2-bitmap.c is temporarily more verbose
>> (because it is still using sector-based math), but will later be
>> switched to track progress by bytes instead of sectors.
>>
>> Use is_power_of_2() while at it, instead of open-coding that, and
>> add an assertion where bdrv_getlength() should not fail.
>>
>> Signed-off-by: Eric Blake <eblake@redhat.com>
>> Reviewed-by: John Snow <jsnow@redhat.com>
>
> I think I would have preferred to change the unit of
> BdrvDirtyBitmap.size in one patch and the unit of the return value of
> bdrv_dirty_bitmap_size() in another one to keep review a bit easier.
I can split on respin, if there's still enough reason for a respin.
>> @@ -305,13 +307,14 @@ BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
>> void bdrv_dirty_bitmap_truncate(BlockDriverState *bs)
>> {
>> BdrvDirtyBitmap *bitmap;
>> - uint64_t size = bdrv_nb_sectors(bs);
>> + int64_t size = bdrv_getlength(bs);
>>
>> + assert(size >= 0);
>
> How can you assert that there will never be an error? Even if it's
> correct (I don't know whether you can have dirty bitmaps on devices that
> don't use the cached value), this needs at least a comment.
The old code wasn't checking for errors; if an error occurs, we have no
way to report it. So I indeed need to audit whether all callers have a
cached length at this point in time (it can't fail), or else change
bdrv_dirty_bitmap_truncate() to be able to fail (pass failure along) and
update all callers. This may indeed be reason for a respin, depending
on what I find.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH v6 10/18] dirty-bitmap: Change bdrv_get_dirty_count() to report bytes
2017-09-08 12:51 ` Kevin Wolf
@ 2017-09-08 14:05 ` Eric Blake
0 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-09-08 14:05 UTC (permalink / raw)
To: Kevin Wolf
Cc: qemu-devel, jsnow, vsementsov, qemu-block, Fam Zheng, Max Reitz,
Jeff Cody, Stefan Hajnoczi, Juan Quintela,
Dr. David Alan Gilbert
[-- Attachment #1: Type: text/plain, Size: 1083 bytes --]
On 09/08/2017 07:51 AM, Kevin Wolf wrote:
> Am 30.08.2017 um 23:05 hat Eric Blake geschrieben:
>> Thanks to recent cleanups, all callers were scaling a return value
>> of sectors into bytes; do the scaling internally instead.
>>
>> Signed-off-by: Eric Blake <eblake@redhat.com>
>> Reviewed-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Juan Quintela <quintela@redhat.com>
>
>> diff --git a/block/mirror.c b/block/mirror.c
>> index af13f5d658..cc47e21814 100644
>> --- a/block/mirror.c
>> +++ b/block/mirror.c
>> @@ -811,11 +811,10 @@ static void coroutine_fn mirror_run(void *opaque)
>
> There is one more place before this one that needs to be converted,
> mirror_iteration() at line 343:
>
> trace_mirror_restart_iter(s, bdrv_get_dirty_count(s->dirty_bitmap) *
> BDRV_SECTOR_SIZE);
Good catch. (I guess I'm a victim of sitting on a patch for several
months, across several rebases...)
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH v6 16/18] qcow2: Switch store_bitmap_data() to byte-based iteration
2017-09-08 13:27 ` Kevin Wolf
@ 2017-09-08 14:09 ` Eric Blake
2017-09-08 14:16 ` Kevin Wolf
0 siblings, 1 reply; 30+ messages in thread
From: Eric Blake @ 2017-09-08 14:09 UTC (permalink / raw)
To: Kevin Wolf; +Cc: qemu-devel, jsnow, vsementsov, qemu-block, Max Reitz
[-- Attachment #1: Type: text/plain, Size: 1762 bytes --]
On 09/08/2017 08:27 AM, Kevin Wolf wrote:
> Am 30.08.2017 um 23:05 hat Eric Blake geschrieben:
>> Now that we have adjusted the majority of the calls this function
>> makes to be byte-based, it is easier to read the code if it makes
>> passes over the image using bytes rather than sectors.
>>
>> Signed-off-by: Eric Blake <eblake@redhat.com>
>> Reviewed-by: John Snow <jsnow@redhat.com>
>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>
>>
>> - while ((sector = bdrv_dirty_iter_next(dbi) >> BDRV_SECTOR_BITS) != -1) {
>> - uint64_t cluster = sector / sbc;
>> + while ((offset = bdrv_dirty_iter_next(dbi)) != -1) {
>
> Don't you have to multiply both sides of the equation? This would be
> offset != -512, which points out that the previous patch to convert
> bdrv_dirty_iter_next() to byte-based gave it a really awkward interface.
I think what I really need to do is change '!= -1' to '< 0', as that's
much easier to reason about when scaling is present.
>
>> + uint64_t cluster = offset / limit;
>> uint64_t end, write_size;
>> int64_t off;
>>
>> - sector = cluster * sbc;
>> - end = MIN(bm_sectors, sector + sbc);
>> - write_size = bdrv_dirty_bitmap_serialization_size(bitmap,
>> - sector * BDRV_SECTOR_SIZE, (end - sector) * BDRV_SECTOR_SIZE);
>> + offset = cluster * limit;
>
> You just had cluster = offset / limit, so in other words, align down
> offset? If so, this is how it should be written.
Thanks for the close reviews; looks like I have enough things to do a
respin.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH v6 16/18] qcow2: Switch store_bitmap_data() to byte-based iteration
2017-09-08 14:09 ` Eric Blake
@ 2017-09-08 14:16 ` Kevin Wolf
0 siblings, 0 replies; 30+ messages in thread
From: Kevin Wolf @ 2017-09-08 14:16 UTC (permalink / raw)
To: Eric Blake; +Cc: qemu-devel, jsnow, vsementsov, qemu-block, Max Reitz
[-- Attachment #1: Type: text/plain, Size: 1424 bytes --]
Am 08.09.2017 um 16:09 hat Eric Blake geschrieben:
> On 09/08/2017 08:27 AM, Kevin Wolf wrote:
> > Am 30.08.2017 um 23:05 hat Eric Blake geschrieben:
> >> Now that we have adjusted the majority of the calls this function
> >> makes to be byte-based, it is easier to read the code if it makes
> >> passes over the image using bytes rather than sectors.
> >>
> >> Signed-off-by: Eric Blake <eblake@redhat.com>
> >> Reviewed-by: John Snow <jsnow@redhat.com>
> >> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> >>
>
> >>
> >> - while ((sector = bdrv_dirty_iter_next(dbi) >> BDRV_SECTOR_BITS) != -1) {
> >> - uint64_t cluster = sector / sbc;
> >> + while ((offset = bdrv_dirty_iter_next(dbi)) != -1) {
> >
> > Don't you have to multiply both sides of the equation? This would be
> > offset != -512, which points out that the previous patch to convert
> > bdrv_dirty_iter_next() to byte-based gave it a really awkward interface.
>
> I think what I really need to do is change '!= -1' to '< 0', as that's
> much easier to reason about when scaling is present.
Hm, I think I would prefer a special case for -1 in bdrv_dirty_iter_next()
so that it returns -1 after the last entry. Even if you check for < 0,
-512 is still an odd return value to signal the end.
Though I think after the final patch, we're back to -1 anyway, so it's
not that important.
Kevin
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH v6 05/18] dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes
2017-09-08 14:04 ` Eric Blake
@ 2017-09-12 19:35 ` Eric Blake
0 siblings, 0 replies; 30+ messages in thread
From: Eric Blake @ 2017-09-12 19:35 UTC (permalink / raw)
To: Kevin Wolf
Cc: vsementsov, Fam Zheng, qemu-block, qemu-devel, Max Reitz, jsnow
[-- Attachment #1: Type: text/plain, Size: 1631 bytes --]
On 09/08/2017 09:04 AM, Eric Blake wrote:
>>> void bdrv_dirty_bitmap_truncate(BlockDriverState *bs)
>>> {
>>> BdrvDirtyBitmap *bitmap;
>>> - uint64_t size = bdrv_nb_sectors(bs);
>>> + int64_t size = bdrv_getlength(bs);
>>>
>>> + assert(size >= 0);
>>
>> How can you assert that there will never be an error? Even if it's
>> correct (I don't know whether you can have dirty bitmaps on devices that
>> don't use the cached value), this needs at least a comment.
>
> The old code wasn't checking for errors; if an error occurs, we have no
> way to report it. So I indeed need to audit whether all callers have a
> cached length at this point in time (it can't fail), or else change
> bdrv_dirty_bitmap_truncate() to be able to fail (pass failure along) and
> update all callers. This may indeed be reason for a respin, depending
> on what I find.
Verdict - it can indeed fail; bdrv_truncate() was blindly calling the
dirty bitmap resize even after a failed refresh_total_sectors(), which
could then resize the dirty bitmap to -1. At least bdrv_truncate()
itself still failed, but it's cleaner to fail up front rather than get
internal state even more botched in the meantime, so fixing that will be
a separate patch in v7. Sadly, the failure is probably more
theoretical, and I did not quickly see an easy way to write an iotests
to expose it (which has been the case with a lot of our recent
bdrv_nb_sectors/bdrv_getlength failure cleanup patches).
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]
^ permalink raw reply [flat|nested] 30+ messages in thread
end of thread, other threads:[~2017-09-12 19:35 UTC | newest]
Thread overview: 30+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-08-30 21:05 [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 01/18] block: Make bdrv_img_create() size selection easier to read Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 02/18] hbitmap: Rename serialization_granularity to serialization_align Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 03/18] qcow2: Ensure bitmap serialization is aligned Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 04/18] dirty-bitmap: Drop unused functions Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 05/18] dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes Eric Blake
2017-09-08 12:22 ` Kevin Wolf
2017-09-08 14:04 ` Eric Blake
2017-09-12 19:35 ` Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 06/18] dirty-bitmap: Change bdrv_dirty_bitmap_*serialize*() to take bytes Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 07/18] qcow2: Switch sectors_covered_by_bitmap_cluster() to byte-based Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 08/18] dirty-bitmap: Set iterator start by offset, not sector Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 09/18] dirty-bitmap: Change bdrv_dirty_iter_next() to report byte offset Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 10/18] dirty-bitmap: Change bdrv_get_dirty_count() to report bytes Eric Blake
2017-09-08 12:51 ` Kevin Wolf
2017-09-08 14:05 ` Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 11/18] dirty-bitmap: Change bdrv_get_dirty_locked() to take bytes Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 12/18] dirty-bitmap: Change bdrv_[re]set_dirty_bitmap() to use bytes Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 13/18] mirror: Switch mirror_dirty_init() to byte-based iteration Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 14/18] qcow2: Switch qcow2_measure() " Eric Blake
2017-09-08 13:15 ` Kevin Wolf
2017-09-08 13:43 ` Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 15/18] qcow2: Switch load_bitmap_data() " Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 16/18] qcow2: Switch store_bitmap_data() " Eric Blake
2017-09-08 13:27 ` Kevin Wolf
2017-09-08 14:09 ` Eric Blake
2017-09-08 14:16 ` Kevin Wolf
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 17/18] dirty-bitmap: Switch bdrv_set_dirty() to bytes Eric Blake
2017-08-30 21:05 ` [Qemu-devel] [PATCH v6 18/18] dirty-bitmap: Convert internal hbitmap size/granularity Eric Blake
2017-08-30 22:43 ` [Qemu-devel] [PATCH v6 00/18] make dirty-bitmap byte-based John Snow
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.