* [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions
@ 2015-08-05 17:32 Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 01/17] " Max Reitz
` (16 more replies)
0 siblings, 17 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:32 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
This is a continuation of previous versions of this series. v2's cover
letter was the most elaborate, which you can find here (includes
benchmarks):
http://lists.nongnu.org/archive/html/qemu-devel/2014-11/msg03430.html
See patch 1 for an explanation of why this series exists and what it
does. Patch 1 is basically the core of this series, the rest just
employs the functions introduced there.
In a later patch, we may want to change the meaning of the "constant"
overlap checking option to mean the same as "cached", which is
everything except for inactive L2 tables. This series does make
checking for overlaps with inactive L2 tables at runtime just as cheap
as everything else (constant time plus caching), but using these checks
means qemu has to read all the snapshot L1 tables when opening a qcow2
file. This does not take long, of course, but it does result in a bit of
overhead so I did not want to enable it by default.
--- v4 part (RFC is still relevant) ---
The new thing about v4 is that there now is a way to limit the total
combined size of all data structures used for the new overlap prevention
algorithm. Bad news: The limit is disabled by default. This is mainly
because for image creation, it is pretty hard to find a sane default
(it depends on the desired image size; but the image is not created with
that size, but rather with size 0 and then truncated; inferring the
default from the image size seems to make sense, but we cannot do this
for images that are going to be truncated).
RFC: An alternative would be to infer the limit like so:
MAX(1M, guest_image_size / 1M). It seems sane to me and I guess it
should work even for images with very, very large guest sizes. But I
think it will break if you're trying to preallocate like a 1 PB image
(which is not really a concern right now, but let's see about it in
ten years)...
On the other hand, the worst that happens is that the event is generated
and the overlap checks won't be performed for some parts of the image
during its creation. Doesn't sound too bad to me. What do you think,
dears reviewers?
v6:
- Rebased on master
- Patch 1: Fixed check_metadata_overlap() function if there is no
metadata list (because overlap checks are switched off)
git-backport-diff against v5:
Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
001/17:[0004] [FC] 'qcow2: Add new overlap check functions'
002/17:[----] [--] 'qcow2: Pull up overlap check option evaluation'
003/17:[----] [--] 'qcow2: Create metadata list'
004/17:[----] [--] 'qcow2/overlaps: Protect image header'
005/17:[----] [--] 'qcow2/overlaps: Protect refcount table'
006/17:[----] [--] 'qcow2/overlaps: Protect refcount blocks'
007/17:[----] [--] 'qcow2/overlaps: Protect active L1 table'
008/17:[----] [--] 'qcow2/overlaps: Protect active L2 tables'
009/17:[----] [--] 'qcow2/overlaps: Protect snapshot table'
010/17:[----] [--] 'qcow2/overlaps: Protect inactive L1 tables'
011/17:[----] [--] 'qcow2/overlaps: Protect inactive L2 tables'
012/17:[----] [--] 'qcow2: Use new metadata overlap check function'
013/17:[----] [--] 'qcow2/overlaps: Add "memory limit reached" event'
014/17:[----] [--] 'qcow2/overlaps: Add memory usage limit'
015/17:[----] [--] 'qcow2: Add overlap structure memory size options'
016/17:[----] [--] 'qapi: Expose new qcow2 overlap check options'
017/17:[----] [--] 'iotests: Test qcow2's overlap check memory limit'
Max Reitz (17):
qcow2: Add new overlap check functions
qcow2: Pull up overlap check option evaluation
qcow2: Create metadata list
qcow2/overlaps: Protect image header
qcow2/overlaps: Protect refcount table
qcow2/overlaps: Protect refcount blocks
qcow2/overlaps: Protect active L1 table
qcow2/overlaps: Protect active L2 tables
qcow2/overlaps: Protect snapshot table
qcow2/overlaps: Protect inactive L1 tables
qcow2/overlaps: Protect inactive L2 tables
qcow2: Use new metadata overlap check function
qcow2/overlaps: Add "memory limit reached" event
qcow2/overlaps: Add memory usage limit
qcow2: Add overlap structure memory size options
qapi: Expose new qcow2 overlap check options
iotests: Test qcow2's overlap check memory limit
block/Makefile.objs | 3 +-
block/qcow2-cluster.c | 13 +
block/qcow2-overlap.c | 614 +++++++++++++++++++++++++++++++++++++++++++++
block/qcow2-refcount.c | 202 ++++++---------
block/qcow2-snapshot.c | 105 +++++++-
block/qcow2.c | 162 ++++++++----
block/qcow2.h | 15 ++
docs/qmp/qmp-events.txt | 27 ++
qapi/block-core.json | 31 +++
qapi/event.json | 27 ++
tests/qemu-iotests/060 | 222 ++++++++++++++++
tests/qemu-iotests/060.out | 47 ++++
tests/qemu-iotests/group | 2 +-
13 files changed, 1297 insertions(+), 173 deletions(-)
create mode 100644 block/qcow2-overlap.c
--
2.4.6
^ permalink raw reply [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 01/17] qcow2: Add new overlap check functions
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
@ 2015-08-05 17:32 ` Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 02/17] qcow2: Pull up overlap check option evaluation Max Reitz
` (15 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:32 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
The existing qcow2 metadata overlap detection function used existing
structures to determine the location of the image metadata, from plain
fields such as l1_table_offset and l1_size in the BDRVQcowState, over
image structures in memory such as the L1 table for the L2 tables'
positions, or it even read the required data directly from disk for
every requested check, such as the snapshot L1 tables for the inactive
L2 tables' positions.
These new functions instead keep a dedicated structure for keeping track
of the metadata positions in memory. It consists of two parts: First,
there is one structure which is basically a list of all metadata
structures. Each entry has a bitmask of types (because some metadata
structures may actually overlap, such as active and inactive L2 tables),
a number of clusters occupied and the offset from the previous entry in
clusters. This structure requires relatively little memory, but checking
a certain point may take relatively long. Each entry is called a
"fragment".
Therefore, there is another representation which is a bitmap, or rather
a bytemap, of metadata types. The previously described list is split
into multiple windows with each describing a constant number of clusters
(WINDOW_SIZE). If the list is to be queried or changed, the respective
window is selected in constant time and the bitmap is generated from the
fragments belonging to the window. This bitmap can then be queried in
constant time and easily be changed.
Because the bitmap representation generally requires more memory, it is
only used as a cache. Whenever a window is removed from the cache, the
fragment list will be rebuilt from the bitmap if the latter has been
modified; except for if the fragment list would actually be bigger than
the bitmap, in which case the bitmap is kept but no longer being
considered part of the cache. Therefore, the fragment list is only used
as the background representation to save memory, whereas the bitmap is
used whenever possible.
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
block/Makefile.objs | 3 +-
block/qcow2-overlap.c | 451 ++++++++++++++++++++++++++++++++++++++++++++++++++
block/qcow2.h | 13 ++
3 files changed, 466 insertions(+), 1 deletion(-)
create mode 100644 block/qcow2-overlap.c
diff --git a/block/Makefile.objs b/block/Makefile.objs
index 58ef2ef..b974db4 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -1,5 +1,6 @@
block-obj-y += raw_bsd.o qcow.o vdi.o vmdk.o cloop.o bochs.o vpc.o vvfat.o
-block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o
+block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o
+block-obj-y += qcow2-cache.o qcow2-overlap.o
block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o
block-obj-y += qed-check.o
block-obj-$(CONFIG_VHDX) += vhdx.o vhdx-endian.o vhdx-log.o
diff --git a/block/qcow2-overlap.c b/block/qcow2-overlap.c
new file mode 100644
index 0000000..54d5eb0
--- /dev/null
+++ b/block/qcow2-overlap.c
@@ -0,0 +1,451 @@
+/*
+ * QCOW2 runtime metadata overlap detection
+ *
+ * Copyright (c) 2015 Max Reitz <mreitz@redhat.com>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include "block/block_int.h"
+#include "qemu-common.h"
+#include "qemu/range.h"
+#include "qcow2.h"
+
+/* Number of clusters which are covered by each metadata window;
+ * note that this may not exceed 2^16 as long as
+ * Qcow2MetadataFragment::relative_start is a uint16_t */
+#define WINDOW_SIZE 4096
+
+/* Describes a fragment of or a whole metadata range; does not necessarily
+ * describe the whole range because it needs to be split on window boundaries */
+typedef struct Qcow2MetadataFragment {
+ /* Bitmask of QCow2MetadataOverlap values */
+ uint8_t types;
+ uint8_t nb_clusters_minus_one;
+ /* Number of clusters between the start of the window and this range */
+ uint16_t relative_start;
+} QEMU_PACKED Qcow2MetadataFragment;
+
+typedef struct Qcow2MetadataWindow {
+ /* This should normally not be NULL. However, it is possible that this list
+ * would require more space than the bitmap, in which case this must be NULL
+ * as long as @bitmap is not NULL.
+ * Note that therefore, the size of this array in bytes may never exceed
+ * WINDOW_SIZE. If that condition would arise during generation of this
+ * array (from the bitmap), this field must be set to NULL and @bitmap must
+ * continue to point to a valid bitmap. */
+ Qcow2MetadataFragment *fragments;
+ int nb_fragments, fragments_array_size;
+
+ /* If not NULL, this is an expanded version of the "RLE" version given by
+ * the fragments array; there are WINDOW_SIZE entries */
+ uint8_t *bitmap;
+ bool bitmap_modified;
+
+ /* Time of last access */
+ unsigned age;
+} Qcow2MetadataWindow;
+
+struct Qcow2MetadataList {
+ Qcow2MetadataWindow *windows;
+ uint64_t nb_windows;
+
+ unsigned current_age;
+
+ /* Index into the windows array */
+ int *cached_windows;
+ size_t nb_cached_windows;
+};
+
+/**
+ * Destroys the cached window bitmap. If it has been modified, the fragment list
+ * will be rebuilt accordingly.
+ */
+static void destroy_window_bitmap(Qcow2MetadataList *mdl,
+ Qcow2MetadataWindow *window)
+{
+ Qcow2MetadataFragment *new_fragments;
+
+ if (!window->bitmap) {
+ return;
+ }
+
+ if (window->bitmap_modified) {
+ int bitmap_i, fragment_i = 0;
+ QCow2MetadataOverlap current_types = 0;
+ int current_nb_clusters = 0;
+
+ /* Rebuild the fragment list; the case bitmap_i == WINDOW_SIZE is for
+ * entering the last fragment at the bitmap end */
+
+ for (bitmap_i = 0; bitmap_i <= WINDOW_SIZE; bitmap_i++) {
+ /* Qcow2MetadataFragment::nb_clusters_minus_one is a uint8_t, so
+ * current_nb_clusters may not exceed 256 */
+ if (bitmap_i < WINDOW_SIZE &&
+ current_types == window->bitmap[bitmap_i] &&
+ current_nb_clusters < 256)
+ {
+ current_nb_clusters++;
+ } else {
+ if (current_types && current_nb_clusters) {
+ if (fragment_i >= window->fragments_array_size) {
+ window->fragments_array_size =
+ 3 * window->fragments_array_size / 2 + 1;
+ if (sizeof(Qcow2MetadataFragment)
+ * window->fragments_array_size >= WINDOW_SIZE)
+ {
+ /* There is no reason to build this fragment list
+ * over keeping the bitmap. Abort. */
+ goto fail;
+ }
+
+ new_fragments =
+ g_try_renew(Qcow2MetadataFragment,
+ window->fragments,
+ window->fragments_array_size);
+ if (!new_fragments) {
+ goto fail;
+ }
+ window->fragments = new_fragments;
+ }
+
+ assert(fragment_i < window->fragments_array_size);
+ window->fragments[fragment_i++] = (Qcow2MetadataFragment){
+ .types = current_types,
+ .nb_clusters_minus_one = current_nb_clusters - 1,
+ .relative_start = bitmap_i - current_nb_clusters,
+ };
+ }
+
+ current_nb_clusters = 1;
+ if (bitmap_i < WINDOW_SIZE) {
+ current_types = window->bitmap[bitmap_i];
+ }
+ }
+ }
+
+ window->nb_fragments = fragment_i;
+ }
+
+ /* Shrink window->fragments if it is possible and makes sense */
+ if (window->nb_fragments < window->fragments_array_size / 2) {
+ new_fragments = g_try_renew(Qcow2MetadataFragment, window->fragments,
+ window->nb_fragments);
+ if (new_fragments) {
+ window->fragments = new_fragments;
+ window->fragments_array_size = window->nb_fragments;
+ }
+ }
+
+ g_free(window->bitmap);
+ window->bitmap = NULL;
+
+ return;
+
+fail:
+ g_free(window->fragments);
+ window->fragments = NULL;
+ window->nb_fragments = 0;
+ window->fragments_array_size = 0;
+}
+
+/**
+ * Creates a bitmap from the fragment list.
+ */
+static void build_window_bitmap(Qcow2MetadataList *mdl,
+ Qcow2MetadataWindow *window)
+{
+ int cache_i, oldest_cache_i = -1, i;
+ unsigned oldest_cache_age = 0;
+
+ for (cache_i = 0; cache_i < mdl->nb_cached_windows; cache_i++) {
+ unsigned age;
+
+ if (mdl->cached_windows[cache_i] < 0) {
+ break;
+ }
+
+ age = mdl->current_age - mdl->windows[mdl->cached_windows[cache_i]].age;
+ if (age > oldest_cache_age) {
+ oldest_cache_age = age;
+ oldest_cache_i = cache_i;
+ }
+ }
+
+ if (cache_i >= mdl->nb_cached_windows) {
+ destroy_window_bitmap(mdl,
+ &mdl->windows[mdl->cached_windows[oldest_cache_i]]);
+ cache_i = oldest_cache_i;
+ }
+
+ assert(cache_i >= 0);
+ mdl->cached_windows[cache_i] = window - mdl->windows;
+
+ window->age = mdl->current_age++;
+
+ /* Maybe there already is a bitmap because it was more space-efficient than
+ * the range list representation */
+ if (window->bitmap) {
+ return;
+ }
+
+ window->bitmap = g_new0(uint8_t, WINDOW_SIZE);
+
+ for (i = 0; i < window->nb_fragments; i++) {
+ Qcow2MetadataFragment *fragment = &window->fragments[i];
+
+ memset(&window->bitmap[fragment->relative_start], fragment->types,
+ fragment->nb_clusters_minus_one + 1);
+ }
+
+ window->bitmap_modified = false;
+}
+
+/**
+ * Enters a new range into the metadata list.
+ */
+void qcow2_metadata_list_enter(BlockDriverState *bs, uint64_t offset,
+ int nb_clusters, QCow2MetadataOverlap types)
+{
+ BDRVQcowState *s = bs->opaque;
+ uint64_t start_cluster = offset >> s->cluster_bits;
+ uint64_t end_cluster = start_cluster + nb_clusters;
+ uint64_t current_cluster = start_cluster;
+
+ types &= s->overlap_check;
+ if (!types) {
+ return;
+ }
+
+ if (offset_into_cluster(s, offset)) {
+ /* Do not enter apparently broken metadata ranges */
+ return;
+ }
+
+ while (current_cluster < end_cluster) {
+ int bitmap_i;
+ int bitmap_i_start = current_cluster % WINDOW_SIZE;
+ int bitmap_i_end = MIN(WINDOW_SIZE,
+ end_cluster - current_cluster + bitmap_i_start);
+ uint64_t window_i = current_cluster / WINDOW_SIZE;
+ Qcow2MetadataWindow *window;
+
+ if (window_i >= s->metadata_list->nb_windows) {
+ /* This should not be happening too often, so it is fine to resize
+ * the array to exactly the required size */
+ Qcow2MetadataWindow *new_windows;
+
+ new_windows = g_try_renew(Qcow2MetadataWindow,
+ s->metadata_list->windows,
+ window_i + 1);
+ if (!new_windows) {
+ return;
+ }
+
+ memset(new_windows + s->metadata_list->nb_windows, 0,
+ (window_i + 1 - s->metadata_list->nb_windows) *
+ sizeof(Qcow2MetadataWindow));
+
+ s->metadata_list->windows = new_windows;
+ s->metadata_list->nb_windows = window_i + 1;
+ }
+
+ window = &s->metadata_list->windows[window_i];
+ if (!window->bitmap) {
+ build_window_bitmap(s->metadata_list, window);
+ }
+
+ for (bitmap_i = bitmap_i_start; bitmap_i < bitmap_i_end; bitmap_i++) {
+ window->bitmap[bitmap_i] |= types;
+ }
+
+ window->age = s->metadata_list->current_age++;
+ window->bitmap_modified = true;
+
+ /* Go to the next window */
+ current_cluster += WINDOW_SIZE - bitmap_i_start;
+ }
+}
+
+/**
+ * Removes a range of the given types from the metadata list.
+ */
+void qcow2_metadata_list_remove(BlockDriverState *bs, uint64_t offset,
+ int nb_clusters, QCow2MetadataOverlap types)
+{
+ BDRVQcowState *s = bs->opaque;
+ uint64_t start_cluster = offset >> s->cluster_bits;
+ uint64_t end_cluster = start_cluster + nb_clusters;
+ uint64_t current_cluster = start_cluster;
+
+ types &= s->overlap_check;
+ if (!types) {
+ return;
+ }
+
+ if (offset_into_cluster(s, offset)) {
+ /* Try to remove even broken metadata ranges */
+ end_cluster++;
+ }
+
+ while (current_cluster < end_cluster) {
+ int bitmap_i;
+ int bitmap_i_start = current_cluster % WINDOW_SIZE;
+ int bitmap_i_end = MIN(WINDOW_SIZE,
+ end_cluster - current_cluster + bitmap_i_start);
+ uint64_t window_i = current_cluster / WINDOW_SIZE;
+ Qcow2MetadataWindow *window;
+
+ /* If the list is too small, there is no metadata structure here;
+ * because window_i will only grow, we can abort here */
+ if (window_i >= s->metadata_list->nb_windows) {
+ return;
+ }
+
+ window = &s->metadata_list->windows[window_i];
+ if (!window->bitmap) {
+ build_window_bitmap(s->metadata_list, window);
+ }
+
+ for (bitmap_i = bitmap_i_start; bitmap_i < bitmap_i_end; bitmap_i++) {
+ window->bitmap[bitmap_i] &= ~types;
+ }
+
+ window->age = s->metadata_list->current_age++;
+ window->bitmap_modified = true;
+
+ /* Go to the next window */
+ current_cluster += WINDOW_SIZE - bitmap_i_start;
+ }
+}
+
+static int single_check_metadata_overlap(Qcow2MetadataList *mdl, int ign,
+ uint64_t cluster)
+{
+ uint64_t window_i = cluster / WINDOW_SIZE;
+ int bitmap_i = cluster % WINDOW_SIZE;
+ Qcow2MetadataWindow *window;
+
+ if (window_i >= mdl->nb_windows) {
+ return 0;
+ }
+ window = &mdl->windows[window_i];
+
+ if (!window->bitmap) {
+ build_window_bitmap(mdl, window);
+ }
+
+ window->age = mdl->current_age++;
+
+ return window->bitmap[bitmap_i] & ~ign;
+}
+
+/* This will replace qcow2_check_metadata_overlap() */
+static int check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset,
+ int64_t size) __attribute__((used));
+
+static int check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset,
+ int64_t size)
+{
+ BDRVQcowState *s = bs->opaque;
+ uint64_t start_cluster = offset >> s->cluster_bits;
+ uint64_t end_cluster = DIV_ROUND_UP(offset + size, s->cluster_size);
+ uint64_t current_cluster;
+ int ret = 0;
+
+ if (!s->metadata_list) {
+ return 0;
+ }
+
+ for (current_cluster = start_cluster; current_cluster < end_cluster;
+ current_cluster++)
+ {
+ ret |= single_check_metadata_overlap(s->metadata_list, ign,
+ current_cluster);
+ }
+
+ return ret;
+}
+
+int qcow2_create_empty_metadata_list(BlockDriverState *bs, size_t cache_size,
+ Error **errp)
+{
+ BDRVQcowState *s = bs->opaque;
+ int ret;
+ size_t cache_entries, i;
+
+ s->metadata_list = g_new0(Qcow2MetadataList, 1);
+ s->metadata_list->nb_windows =
+ DIV_ROUND_UP(bdrv_nb_sectors(bs->file),
+ (uint64_t)s->cluster_sectors * WINDOW_SIZE);
+ s->metadata_list->windows = g_try_new0(Qcow2MetadataWindow,
+ s->metadata_list->nb_windows);
+ if (s->metadata_list->nb_windows && !s->metadata_list->windows) {
+ error_setg(errp, "Could not allocate metadata overlap check windows");
+ ret = -ENOMEM;
+ goto fail;
+ }
+
+ /* Do not count the size of s->metadata_list->cached_windows, because the
+ * size per element is negligible when compared to WINDOW_SIZE, and also
+ * because the user is more likely to specify multiples of WINDOW_SIZE than
+ * WINDOW_SIZE + sizeof(*s->metadata_list->cached_windows). */
+ cache_entries = cache_size / WINDOW_SIZE;
+ if (!cache_entries) {
+ cache_entries = 1;
+ }
+
+ s->metadata_list->nb_cached_windows = cache_entries;
+ s->metadata_list->cached_windows = g_try_new(int, cache_entries);
+ if (!s->metadata_list->cached_windows) {
+ error_setg(errp, "Could not allocate metadata overlap cache pointers");
+ ret = -ENOMEM;
+ goto fail;
+ }
+ for (i = 0; i < s->metadata_list->nb_cached_windows; i++) {
+ s->metadata_list->cached_windows[i] = -1;
+ }
+
+ return 0;
+
+fail:
+ qcow2_metadata_list_destroy(bs);
+ return ret;
+}
+
+void qcow2_metadata_list_destroy(BlockDriverState *bs)
+{
+ BDRVQcowState *s = bs->opaque;
+ uint64_t i;
+
+ if (s->metadata_list && s->metadata_list->windows) {
+ for (i = 0; i < s->metadata_list->nb_windows; i++) {
+ Qcow2MetadataWindow *window = &s->metadata_list->windows[i];
+
+ g_free(window->bitmap);
+ g_free(window->fragments);
+ }
+
+ g_free(s->metadata_list->cached_windows);
+ g_free(s->metadata_list->windows);
+ }
+
+ g_free(s->metadata_list);
+ s->metadata_list = NULL;
+}
diff --git a/block/qcow2.h b/block/qcow2.h
index 72e1328..e48dcdc 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -162,6 +162,9 @@ typedef struct QCowSnapshot {
struct Qcow2Cache;
typedef struct Qcow2Cache Qcow2Cache;
+struct Qcow2MetadataList;
+typedef struct Qcow2MetadataList Qcow2MetadataList;
+
typedef struct Qcow2UnknownHeaderExtension {
uint32_t magic;
uint32_t len;
@@ -272,6 +275,7 @@ typedef struct BDRVQcowState {
bool discard_passthrough[QCOW2_DISCARD_MAX];
+ Qcow2MetadataList *metadata_list;
int overlap_check; /* bitmask of Qcow2MetadataOverlap values */
bool signaled_corruption;
@@ -589,4 +593,13 @@ int qcow2_cache_get_empty(BlockDriverState *bs, Qcow2Cache *c, uint64_t offset,
void **table);
void qcow2_cache_put(BlockDriverState *bs, Qcow2Cache *c, void **table);
+/* qcow2-overlap.c functions */
+int qcow2_create_empty_metadata_list(BlockDriverState *bs, size_t cache_size,
+ Error **errp);
+void qcow2_metadata_list_destroy(BlockDriverState *bs);
+void qcow2_metadata_list_enter(BlockDriverState *bs, uint64_t offset,
+ int nb_clusters, QCow2MetadataOverlap type);
+void qcow2_metadata_list_remove(BlockDriverState *bs, uint64_t offset,
+ int nb_clusters, QCow2MetadataOverlap type);
+
#endif
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 02/17] qcow2: Pull up overlap check option evaluation
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 01/17] " Max Reitz
@ 2015-08-05 17:32 ` Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 03/17] qcow2: Create metadata list Max Reitz
` (14 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:32 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
Pull up the absorption of the qcow2-relevant command line options and
the evaluation of the overlap check options in qcow2_open().
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
block/qcow2.c | 96 +++++++++++++++++++++++++++++------------------------------
1 file changed, 48 insertions(+), 48 deletions(-)
diff --git a/block/qcow2.c b/block/qcow2.c
index 76c331b..97b624a 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -708,6 +708,54 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,
bs->encrypted = 1;
}
+ opts = qemu_opts_create(&qcow2_runtime_opts, NULL, 0, &error_abort);
+ qemu_opts_absorb_qdict(opts, options, &local_err);
+ if (local_err) {
+ error_propagate(errp, local_err);
+ ret = -EINVAL;
+ goto fail;
+ }
+
+ opt_overlap_check = qemu_opt_get(opts, QCOW2_OPT_OVERLAP);
+ opt_overlap_check_template = qemu_opt_get(opts, QCOW2_OPT_OVERLAP_TEMPLATE);
+ if (opt_overlap_check_template && opt_overlap_check &&
+ strcmp(opt_overlap_check_template, opt_overlap_check))
+ {
+ error_setg(errp, "Conflicting values for qcow2 options '"
+ QCOW2_OPT_OVERLAP "' ('%s') and '" QCOW2_OPT_OVERLAP_TEMPLATE
+ "' ('%s')", opt_overlap_check, opt_overlap_check_template);
+ ret = -EINVAL;
+ goto fail;
+ }
+ if (!opt_overlap_check) {
+ opt_overlap_check = opt_overlap_check_template ?: "cached";
+ }
+
+ if (!strcmp(opt_overlap_check, "none")) {
+ overlap_check_template = 0;
+ } else if (!strcmp(opt_overlap_check, "constant")) {
+ overlap_check_template = QCOW2_OL_CONSTANT;
+ } else if (!strcmp(opt_overlap_check, "cached")) {
+ overlap_check_template = QCOW2_OL_CACHED;
+ } else if (!strcmp(opt_overlap_check, "all")) {
+ overlap_check_template = QCOW2_OL_ALL;
+ } else {
+ error_setg(errp, "Unsupported value '%s' for qcow2 option "
+ "'overlap-check'. Allowed are either of the following: "
+ "none, constant, cached, all", opt_overlap_check);
+ ret = -EINVAL;
+ goto fail;
+ }
+
+ s->overlap_check = 0;
+ for (i = 0; i < QCOW2_OL_MAX_BITNR; i++) {
+ /* overlap-check defines a template bitmask, but every flag may be
+ * overwritten through the associated boolean option */
+ s->overlap_check |=
+ qemu_opt_get_bool(opts, overlap_bool_option_names[i],
+ overlap_check_template & (1 << i)) << i;
+ }
+
s->l2_bits = s->cluster_bits - 3; /* L2 is always one cluster */
s->l2_size = 1 << s->l2_bits;
/* 2^(s->refcount_order - 3) is the refcount width in bytes */
@@ -803,14 +851,6 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,
}
/* get L2 table/refcount block cache size from command line options */
- opts = qemu_opts_create(&qcow2_runtime_opts, NULL, 0, &error_abort);
- qemu_opts_absorb_qdict(opts, options, &local_err);
- if (local_err) {
- error_propagate(errp, local_err);
- ret = -EINVAL;
- goto fail;
- }
-
read_cache_sizes(bs, opts, &l2_cache_size, &refcount_cache_size,
&local_err);
if (local_err) {
@@ -946,46 +986,6 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,
s->discard_passthrough[QCOW2_DISCARD_OTHER] =
qemu_opt_get_bool(opts, QCOW2_OPT_DISCARD_OTHER, false);
- opt_overlap_check = qemu_opt_get(opts, QCOW2_OPT_OVERLAP);
- opt_overlap_check_template = qemu_opt_get(opts, QCOW2_OPT_OVERLAP_TEMPLATE);
- if (opt_overlap_check_template && opt_overlap_check &&
- strcmp(opt_overlap_check_template, opt_overlap_check))
- {
- error_setg(errp, "Conflicting values for qcow2 options '"
- QCOW2_OPT_OVERLAP "' ('%s') and '" QCOW2_OPT_OVERLAP_TEMPLATE
- "' ('%s')", opt_overlap_check, opt_overlap_check_template);
- ret = -EINVAL;
- goto fail;
- }
- if (!opt_overlap_check) {
- opt_overlap_check = opt_overlap_check_template ?: "cached";
- }
-
- if (!strcmp(opt_overlap_check, "none")) {
- overlap_check_template = 0;
- } else if (!strcmp(opt_overlap_check, "constant")) {
- overlap_check_template = QCOW2_OL_CONSTANT;
- } else if (!strcmp(opt_overlap_check, "cached")) {
- overlap_check_template = QCOW2_OL_CACHED;
- } else if (!strcmp(opt_overlap_check, "all")) {
- overlap_check_template = QCOW2_OL_ALL;
- } else {
- error_setg(errp, "Unsupported value '%s' for qcow2 option "
- "'overlap-check'. Allowed are either of the following: "
- "none, constant, cached, all", opt_overlap_check);
- ret = -EINVAL;
- goto fail;
- }
-
- s->overlap_check = 0;
- for (i = 0; i < QCOW2_OL_MAX_BITNR; i++) {
- /* overlap-check defines a template bitmask, but every flag may be
- * overwritten through the associated boolean option */
- s->overlap_check |=
- qemu_opt_get_bool(opts, overlap_bool_option_names[i],
- overlap_check_template & (1 << i)) << i;
- }
-
qemu_opts_del(opts);
opts = NULL;
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 03/17] qcow2: Create metadata list
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 01/17] " Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 02/17] qcow2: Pull up overlap check option evaluation Max Reitz
@ 2015-08-05 17:32 ` Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 04/17] qcow2/overlaps: Protect image header Max Reitz
` (13 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:32 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
Create and destroy the metadata list on creation and destruction of a
qcow2 BDS, respectively. Skip creation if no overlap checks should be
performed.
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
block/qcow2.c | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/block/qcow2.c b/block/qcow2.c
index 97b624a..d30d008 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -766,6 +766,14 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,
s->csize_mask = (1 << (s->cluster_bits - 8)) - 1;
s->cluster_offset_mask = (1LL << s->csize_shift) - 1;
+ if (s->overlap_check) {
+ /* TODO: Let the user override this default */
+ ret = qcow2_create_empty_metadata_list(bs, 65536, errp);
+ if (ret < 0) {
+ goto fail;
+ }
+ }
+
s->refcount_table_offset = header.refcount_table_offset;
s->refcount_table_size =
header.refcount_table_clusters << (s->cluster_bits - 3);
@@ -1021,6 +1029,7 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,
}
g_free(s->cluster_cache);
qemu_vfree(s->cluster_data);
+ qcow2_metadata_list_destroy(bs);
return ret;
}
@@ -1487,6 +1496,8 @@ static void qcow2_close(BlockDriverState *bs)
qemu_vfree(s->cluster_data);
qcow2_refcount_close(bs);
qcow2_free_snapshots(bs);
+
+ qcow2_metadata_list_destroy(bs);
}
static void qcow2_invalidate_cache(BlockDriverState *bs, Error **errp)
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 04/17] qcow2/overlaps: Protect image header
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
` (2 preceding siblings ...)
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 03/17] qcow2: Create metadata list Max Reitz
@ 2015-08-05 17:32 ` Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 05/17] qcow2/overlaps: Protect refcount table Max Reitz
` (12 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:32 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
Enter the image header into the metadata list to protect it against
accidental modifications.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
block/qcow2.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/block/qcow2.c b/block/qcow2.c
index d30d008..9815325 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -774,6 +774,8 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,
}
}
+ qcow2_metadata_list_enter(bs, 0, 1, QCOW2_OL_MAIN_HEADER);
+
s->refcount_table_offset = header.refcount_table_offset;
s->refcount_table_size =
header.refcount_table_clusters << (s->cluster_bits - 3);
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 05/17] qcow2/overlaps: Protect refcount table
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
` (3 preceding siblings ...)
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 04/17] qcow2/overlaps: Protect image header Max Reitz
@ 2015-08-05 17:32 ` Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 06/17] qcow2/overlaps: Protect refcount blocks Max Reitz
` (11 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:32 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
Keep track of the refcount table in the metadata list to protect it
against accidental modifications.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
block/qcow2-refcount.c | 18 ++++++++++++++++++
block/qcow2.c | 4 ++++
2 files changed, 22 insertions(+)
diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c
index b0ee42d..8e302bd 100644
--- a/block/qcow2-refcount.c
+++ b/block/qcow2-refcount.c
@@ -579,6 +579,14 @@ static int alloc_refcount_block(BlockDriverState *bs,
s->refcount_table_size = table_size;
s->refcount_table_offset = table_offset;
+ qcow2_metadata_list_remove(bs, old_table_offset,
+ size_to_clusters(s, old_table_size *
+ sizeof(uint64_t)),
+ QCOW2_OL_REFCOUNT_TABLE);
+
+ qcow2_metadata_list_enter(bs, table_offset, table_clusters,
+ QCOW2_OL_REFCOUNT_TABLE);
+
/* Free old table. */
qcow2_free_clusters(bs, old_table_offset, old_table_size * sizeof(uint64_t),
QCOW2_DISCARD_OTHER);
@@ -2151,6 +2159,16 @@ write_refblocks:
goto fail;
}
+ qcow2_metadata_list_remove(bs, s->refcount_table_offset,
+ size_to_clusters(s, s->refcount_table_size
+ * sizeof(uint64_t)),
+ QCOW2_OL_REFCOUNT_TABLE);
+
+ qcow2_metadata_list_enter(bs, reftable_offset,
+ size_to_clusters(s, reftable_size *
+ sizeof(uint64_t)),
+ QCOW2_OL_REFCOUNT_TABLE);
+
for (refblock_index = 0; refblock_index < reftable_size; refblock_index++) {
be64_to_cpus(&on_disk_reftable[refblock_index]);
}
diff --git a/block/qcow2.c b/block/qcow2.c
index 9815325..e34cd7c 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -792,6 +792,10 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,
error_setg(errp, "Invalid reference count table offset");
goto fail;
}
+ qcow2_metadata_list_enter(bs, s->refcount_table_offset,
+ size_to_clusters(s, s->refcount_table_size *
+ sizeof(uint64_t)),
+ QCOW2_OL_REFCOUNT_TABLE);
/* Snapshot table offset/length */
if (header.nb_snapshots > QCOW_MAX_SNAPSHOTS) {
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 06/17] qcow2/overlaps: Protect refcount blocks
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
` (4 preceding siblings ...)
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 05/17] qcow2/overlaps: Protect refcount table Max Reitz
@ 2015-08-05 17:32 ` Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 07/17] qcow2/overlaps: Protect active L1 table Max Reitz
` (10 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:32 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
Keep track of the refcount blocks in the metadata list to protect them
against accidental modifications.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
block/qcow2-refcount.c | 38 +++++++++++++++++++++++++++++++++++++-
1 file changed, 37 insertions(+), 1 deletion(-)
diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c
index 8e302bd..13799c8 100644
--- a/block/qcow2-refcount.c
+++ b/block/qcow2-refcount.c
@@ -106,8 +106,16 @@ int qcow2_refcount_init(BlockDriverState *bs)
if (ret < 0) {
goto fail;
}
- for(i = 0; i < s->refcount_table_size; i++)
+ for (i = 0; i < s->refcount_table_size; i++) {
+ uint64_t refblock_offset;
+
be64_to_cpus(&s->refcount_table[i]);
+ refblock_offset = s->refcount_table[i] & REFT_OFFSET_MASK;
+ if (refblock_offset) {
+ qcow2_metadata_list_enter(bs, refblock_offset, 1,
+ QCOW2_OL_REFCOUNT_BLOCK);
+ }
+ }
}
return 0;
fail:
@@ -439,6 +447,7 @@ static int alloc_refcount_block(BlockDriverState *bs,
}
s->refcount_table[refcount_table_index] = new_block;
+ qcow2_metadata_list_enter(bs, new_block, 1, QCOW2_OL_REFCOUNT_BLOCK);
/* The new refcount block may be where the caller intended to put its
* data, so let it restart the search. */
@@ -509,6 +518,7 @@ static int alloc_refcount_block(BlockDriverState *bs,
uint64_t table_offset = meta_offset + blocks_clusters * s->cluster_size;
uint64_t *new_table = g_try_new0(uint64_t, table_size);
void *new_blocks = g_try_malloc0(blocks_clusters * s->cluster_size);
+ uint64_t block_index;
assert(table_size > 0 && blocks_clusters > 0);
if (new_table == NULL || new_blocks == NULL) {
@@ -587,6 +597,12 @@ static int alloc_refcount_block(BlockDriverState *bs,
qcow2_metadata_list_enter(bs, table_offset, table_clusters,
QCOW2_OL_REFCOUNT_TABLE);
+ for (block_index = 0; block_index < blocks_clusters; block_index++) {
+ qcow2_metadata_list_enter(bs, meta_offset +
+ (block_index << s->cluster_bits), 1,
+ QCOW2_OL_REFCOUNT_BLOCK);
+ }
+
/* Free old table. */
qcow2_free_clusters(bs, old_table_offset, old_table_size * sizeof(uint64_t),
QCOW2_DISCARD_OTHER);
@@ -2164,14 +2180,34 @@ write_refblocks:
* sizeof(uint64_t)),
QCOW2_OL_REFCOUNT_TABLE);
+ for (refblock_index = 0; refblock_index < s->refcount_table_size;
+ refblock_index++)
+ {
+ uint64_t refblock_offset = s->refcount_table[refblock_index] &
+ REFT_OFFSET_MASK;
+ if (refblock_offset) {
+ qcow2_metadata_list_remove(bs, refblock_offset, 1,
+ QCOW2_OL_REFCOUNT_BLOCK);
+ }
+ }
+
qcow2_metadata_list_enter(bs, reftable_offset,
size_to_clusters(s, reftable_size *
sizeof(uint64_t)),
QCOW2_OL_REFCOUNT_TABLE);
for (refblock_index = 0; refblock_index < reftable_size; refblock_index++) {
+ uint64_t refblock_offset;
+
be64_to_cpus(&on_disk_reftable[refblock_index]);
+
+ refblock_offset = on_disk_reftable[refblock_index] & REFT_OFFSET_MASK;
+ if (refblock_offset) {
+ qcow2_metadata_list_enter(bs, refblock_offset, 1,
+ QCOW2_OL_REFCOUNT_BLOCK);
+ }
}
+
s->refcount_table = on_disk_reftable;
s->refcount_table_offset = reftable_offset;
s->refcount_table_size = reftable_size;
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 07/17] qcow2/overlaps: Protect active L1 table
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
` (5 preceding siblings ...)
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 06/17] qcow2/overlaps: Protect refcount blocks Max Reitz
@ 2015-08-05 17:32 ` Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 08/17] qcow2/overlaps: Protect active L2 tables Max Reitz
` (9 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:32 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
Keep track of the active L1 table in the metadata list to protect it
against accidental modifications.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
block/qcow2-cluster.c | 11 +++++++++++
block/qcow2-snapshot.c | 10 ++++++++++
block/qcow2.c | 4 ++++
3 files changed, 25 insertions(+)
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index b43f186..b95f6fe 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -125,6 +125,17 @@ int qcow2_grow_l1_table(BlockDriverState *bs, uint64_t min_size,
s->l1_table = new_l1_table;
old_l1_size = s->l1_size;
s->l1_size = new_l1_size;
+
+ qcow2_metadata_list_remove(bs, old_l1_table_offset,
+ size_to_clusters(s, old_l1_size *
+ sizeof(uint64_t)),
+ QCOW2_OL_ACTIVE_L1);
+
+ qcow2_metadata_list_enter(bs, s->l1_table_offset,
+ size_to_clusters(s, s->l1_size *
+ sizeof(uint64_t)),
+ QCOW2_OL_ACTIVE_L1);
+
qcow2_free_clusters(bs, old_l1_table_offset, old_l1_size * sizeof(uint64_t),
QCOW2_DISCARD_OTHER);
return 0;
diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c
index b6f58c1..05e814d 100644
--- a/block/qcow2-snapshot.c
+++ b/block/qcow2-snapshot.c
@@ -719,6 +719,11 @@ int qcow2_snapshot_load_tmp(BlockDriverState *bs,
return ret;
}
+ qcow2_metadata_list_remove(bs, s->l1_table_offset,
+ size_to_clusters(s, s->l1_size *
+ sizeof(uint64_t)),
+ QCOW2_OL_ACTIVE_L1);
+
/* Switch the L1 table */
qemu_vfree(s->l1_table);
@@ -730,5 +735,10 @@ int qcow2_snapshot_load_tmp(BlockDriverState *bs,
be64_to_cpus(&s->l1_table[i]);
}
+ qcow2_metadata_list_enter(bs, s->l1_table_offset,
+ size_to_clusters(s, s->l1_size *
+ sizeof(uint64_t)),
+ QCOW2_OL_ACTIVE_L1);
+
return 0;
}
diff --git a/block/qcow2.c b/block/qcow2.c
index e34cd7c..c186452 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -844,6 +844,10 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,
}
s->l1_table_offset = header.l1_table_offset;
+ qcow2_metadata_list_enter(bs, s->l1_table_offset,
+ size_to_clusters(s, s->l1_size *
+ sizeof(uint64_t)),
+ QCOW2_OL_ACTIVE_L1);
if (s->l1_size > 0) {
s->l1_table = qemu_try_blockalign(bs->file,
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 08/17] qcow2/overlaps: Protect active L2 tables
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
` (6 preceding siblings ...)
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 07/17] qcow2/overlaps: Protect active L1 table Max Reitz
@ 2015-08-05 17:32 ` Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 09/17] qcow2/overlaps: Protect snapshot table Max Reitz
` (8 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:32 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
Keep track of the active L2 tables in the metadata list to protect them
against accidental modifications.
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
block/qcow2-cluster.c | 2 ++
block/qcow2-refcount.c | 6 ++++++
block/qcow2-snapshot.c | 21 +++++++++++++++++++++
block/qcow2.c | 8 +++++++-
4 files changed, 36 insertions(+), 1 deletion(-)
diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index b95f6fe..ec42602 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -285,6 +285,8 @@ static int l2_allocate(BlockDriverState *bs, int l1_index, uint64_t **table)
goto fail;
}
+ qcow2_metadata_list_enter(bs, l2_offset, 1, QCOW2_OL_ACTIVE_L2);
+
*table = l2_table;
trace_qcow2_l2_allocate_done(bs, l1_index, 0);
return 0;
diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c
index 13799c8..76dd2bc 100644
--- a/block/qcow2-refcount.c
+++ b/block/qcow2-refcount.c
@@ -1209,6 +1209,12 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,
if (ret < 0) {
goto fail;
}
+ if (addend < 0) {
+ if (!l1_allocated) {
+ qcow2_metadata_list_remove(bs, l2_offset, 1,
+ QCOW2_OL_ACTIVE_L2);
+ }
+ }
}
ret = qcow2_get_refcount(bs, l2_offset >> s->cluster_bits,
&refcount);
diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c
index 05e814d..7add805f 100644
--- a/block/qcow2-snapshot.c
+++ b/block/qcow2-snapshot.c
@@ -560,6 +560,13 @@ int qcow2_snapshot_goto(BlockDriverState *bs, const char *snapshot_id)
g_free(sn_l1_table);
sn_l1_table = NULL;
+ for (i = 0; i < s->l1_size; i++) {
+ uint64_t l2_offset = s->l1_table[i] & L1E_OFFSET_MASK;
+ if (l2_offset) {
+ qcow2_metadata_list_enter(bs, l2_offset, 1, QCOW2_OL_ACTIVE_L2);
+ }
+ }
+
/*
* Update QCOW_OFLAG_COPIED in the active L1 table (it may have changed
* when we decreased the refcount of the old snapshot.
@@ -724,6 +731,13 @@ int qcow2_snapshot_load_tmp(BlockDriverState *bs,
sizeof(uint64_t)),
QCOW2_OL_ACTIVE_L1);
+ for (i = 0; i < s->l1_size; i++) {
+ uint64_t l2_offset = s->l1_table[i] & L1E_OFFSET_MASK;
+ if (l2_offset) {
+ qcow2_metadata_list_remove(bs, l2_offset, 1, QCOW2_OL_ACTIVE_L2);
+ }
+ }
+
/* Switch the L1 table */
qemu_vfree(s->l1_table);
@@ -740,5 +754,12 @@ int qcow2_snapshot_load_tmp(BlockDriverState *bs,
sizeof(uint64_t)),
QCOW2_OL_ACTIVE_L1);
+ for (i = 0; i < s->l1_size; i++) {
+ uint64_t l2_offset = s->l1_table[i] & L1E_OFFSET_MASK;
+ if (l2_offset) {
+ qcow2_metadata_list_enter(bs, l2_offset, 1, QCOW2_OL_ACTIVE_L2);
+ }
+ }
+
return 0;
}
diff --git a/block/qcow2.c b/block/qcow2.c
index c186452..b7d0f33 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -863,8 +863,14 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,
error_setg_errno(errp, -ret, "Could not read L1 table");
goto fail;
}
- for(i = 0;i < s->l1_size; i++) {
+ for (i = 0; i < s->l1_size; i++) {
+ uint64_t l2_offset;
+
be64_to_cpus(&s->l1_table[i]);
+ l2_offset = s->l1_table[i] & L1E_OFFSET_MASK;
+ if (l2_offset) {
+ qcow2_metadata_list_enter(bs, l2_offset, 1, QCOW2_OL_ACTIVE_L2);
+ }
}
}
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 09/17] qcow2/overlaps: Protect snapshot table
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
` (7 preceding siblings ...)
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 08/17] qcow2/overlaps: Protect active L2 tables Max Reitz
@ 2015-08-05 17:32 ` Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 10/17] qcow2/overlaps: Protect inactive L1 tables Max Reitz
` (7 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:32 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
Keep track of the snapshot table in the metadata list to protect it
against accidental modifications.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
block/qcow2-snapshot.c | 10 ++++++++++
block/qcow2.c | 6 ++++++
2 files changed, 16 insertions(+)
diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c
index 7add805f..6d1ae00 100644
--- a/block/qcow2-snapshot.c
+++ b/block/qcow2-snapshot.c
@@ -263,8 +263,18 @@ static int qcow2_write_snapshots(BlockDriverState *bs)
/* free the old snapshot table */
qcow2_free_clusters(bs, s->snapshots_offset, s->snapshots_size,
QCOW2_DISCARD_SNAPSHOT);
+
+ qcow2_metadata_list_remove(bs, s->snapshots_offset,
+ size_to_clusters(s, s->snapshots_size),
+ QCOW2_OL_SNAPSHOT_TABLE);
+
s->snapshots_offset = snapshots_offset;
s->snapshots_size = snapshots_size;
+
+ qcow2_metadata_list_enter(bs, s->snapshots_offset,
+ size_to_clusters(s, s->snapshots_size),
+ QCOW2_OL_SNAPSHOT_TABLE);
+
return 0;
fail:
diff --git a/block/qcow2.c b/block/qcow2.c
index b7d0f33..0e7b646 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -811,6 +811,12 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,
error_setg(errp, "Invalid snapshot table offset");
goto fail;
}
+ if (header.nb_snapshots) {
+ qcow2_metadata_list_enter(bs, header.snapshots_offset,
+ size_to_clusters(s, header.nb_snapshots *
+ sizeof(QCowSnapshotHeader)),
+ QCOW2_OL_SNAPSHOT_TABLE);
+ }
/* read the level 1 table */
if (header.l1_size > QCOW_MAX_L1_SIZE / sizeof(uint64_t)) {
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 10/17] qcow2/overlaps: Protect inactive L1 tables
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
` (8 preceding siblings ...)
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 09/17] qcow2/overlaps: Protect snapshot table Max Reitz
@ 2015-08-05 17:32 ` Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 11/17] qcow2/overlaps: Protect inactive L2 tables Max Reitz
` (6 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:32 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
Keep track of the inactive L1 tables in the metadata list to protect
them against accidental modifications.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
block/qcow2-snapshot.c | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c
index 6d1ae00..95afd87 100644
--- a/block/qcow2-snapshot.c
+++ b/block/qcow2-snapshot.c
@@ -122,6 +122,21 @@ int qcow2_read_snapshots(BlockDriverState *bs)
ret = -EFBIG;
goto fail;
}
+
+ if (!(s->overlap_check & QCOW2_OL_INACTIVE_L1)) {
+ continue;
+ }
+
+ if (sn->l1_size > INT_MAX / sizeof(uint64_t)) {
+ /* Do not fail opening the image because a snapshot is broken which
+ * might not be used anyway */
+ continue;
+ }
+
+ qcow2_metadata_list_enter(bs, sn->l1_table_offset,
+ size_to_clusters(s, sn->l1_size *
+ sizeof(uint64_t)),
+ QCOW2_OL_INACTIVE_L1);
}
assert(offset - s->snapshots_offset <= INT_MAX);
@@ -415,6 +430,11 @@ int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info)
g_free(l1_table);
l1_table = NULL;
+ qcow2_metadata_list_enter(bs, sn->l1_table_offset,
+ size_to_clusters(s, sn->l1_size *
+ sizeof(uint64_t)),
+ QCOW2_OL_INACTIVE_L1);
+
/*
* Increase the refcounts of all clusters and make sure everything is
* stable on disk before updating the snapshot table to contain a pointer
@@ -635,6 +655,11 @@ int qcow2_snapshot_delete(BlockDriverState *bs,
g_free(sn.id_str);
g_free(sn.name);
+ qcow2_metadata_list_remove(bs, sn.l1_table_offset,
+ size_to_clusters(s, sn.l1_size *
+ sizeof(uint64_t)),
+ QCOW2_OL_INACTIVE_L1);
+
/*
* Now decrease the refcounts of clusters referenced by the snapshot and
* free the L1 table.
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 11/17] qcow2/overlaps: Protect inactive L2 tables
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
` (9 preceding siblings ...)
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 10/17] qcow2/overlaps: Protect inactive L1 tables Max Reitz
@ 2015-08-05 17:32 ` Max Reitz
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 12/17] qcow2: Use new metadata overlap check function Max Reitz
` (5 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:32 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
Keep track of the inactive L2 tables in the metadata list to protect
them against accidental modifications.
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
block/qcow2-refcount.c | 20 ++++++++++++++++++++
block/qcow2-snapshot.c | 43 ++++++++++++++++++++++++++++++++++++++++---
2 files changed, 60 insertions(+), 3 deletions(-)
diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c
index 76dd2bc..2cdf535 100644
--- a/block/qcow2-refcount.c
+++ b/block/qcow2-refcount.c
@@ -1211,8 +1211,28 @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs,
}
if (addend < 0) {
if (!l1_allocated) {
+ /* This is easy */
qcow2_metadata_list_remove(bs, l2_offset, 1,
QCOW2_OL_ACTIVE_L2);
+ } else {
+ /* If refcount == 0, this is, too. If refcount > 1, we
+ * know that there must be some other inactive L2
+ * reference; and for refcount == 1, if this is an
+ * active L2 table, this was the last inactive L2
+ * reference. */
+ bool remove;
+ if (refcount == 0) {
+ remove = true;
+ } else if (refcount == 1) {
+ remove = qcow2_check_metadata_overlap(bs,
+ ~QCOW2_OL_ACTIVE_L2, l2_offset,s->cluster_size);
+ } else {
+ remove = false;
+ }
+ if (remove) {
+ qcow2_metadata_list_remove(bs, l2_offset, 1,
+ QCOW2_OL_INACTIVE_L2);
+ }
}
}
}
diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c
index 95afd87..e781bf2 100644
--- a/block/qcow2-snapshot.c
+++ b/block/qcow2-snapshot.c
@@ -47,9 +47,10 @@ int qcow2_read_snapshots(BlockDriverState *bs)
QCowSnapshotHeader h;
QCowSnapshotExtraData extra;
QCowSnapshot *sn;
- int i, id_str_size, name_size;
+ int i, j, id_str_size, name_size;
int64_t offset;
uint32_t extra_data_size;
+ uint64_t *l1_table;
int ret;
if (!s->nb_snapshots) {
@@ -123,11 +124,12 @@ int qcow2_read_snapshots(BlockDriverState *bs)
goto fail;
}
- if (!(s->overlap_check & QCOW2_OL_INACTIVE_L1)) {
+ if (!(s->overlap_check & (QCOW2_OL_INACTIVE_L1 | QCOW2_OL_INACTIVE_L2)))
+ {
continue;
}
- if (sn->l1_size > INT_MAX / sizeof(uint64_t)) {
+ if (sn->l1_size > QCOW_MAX_L1_SIZE) {
/* Do not fail opening the image because a snapshot is broken which
* might not be used anyway */
continue;
@@ -137,6 +139,34 @@ int qcow2_read_snapshots(BlockDriverState *bs)
size_to_clusters(s, sn->l1_size *
sizeof(uint64_t)),
QCOW2_OL_INACTIVE_L1);
+
+ if (!(s->overlap_check & QCOW2_OL_INACTIVE_L2)) {
+ continue;
+ }
+
+ l1_table = qemu_try_blockalign(bs->file,
+ sn->l1_size * sizeof(uint64_t));
+ if (!l1_table) {
+ /* Do not fail opening the image just because a snapshot's L2 tables
+ * cannot be covered by the overlap checks */
+ continue;
+ }
+
+ ret = bdrv_pread(bs->file, sn->l1_table_offset, l1_table,
+ sn->l1_size * sizeof(uint64_t));
+ if (ret < 0) {
+ qemu_vfree(l1_table);
+ continue;
+ }
+ for (j = 0; j < sn->l1_size; j++) {
+ uint64_t l2_offset = be64_to_cpu(l1_table[j]) & L1E_OFFSET_MASK;
+ if (l2_offset) {
+ qcow2_metadata_list_enter(bs, l2_offset, 1,
+ QCOW2_OL_INACTIVE_L2);
+ }
+ }
+
+ qemu_vfree(l1_table);
}
assert(offset - s->snapshots_offset <= INT_MAX);
@@ -435,6 +465,13 @@ int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info)
sizeof(uint64_t)),
QCOW2_OL_INACTIVE_L1);
+ for (i = 0; i < s->l1_size; i++) {
+ uint64_t l2_offset = s->l1_table[i] & L1E_OFFSET_MASK;
+ if (l2_offset) {
+ qcow2_metadata_list_enter(bs, l2_offset, 1, QCOW2_OL_INACTIVE_L2);
+ }
+ }
+
/*
* Increase the refcounts of all clusters and make sure everything is
* stable on disk before updating the snapshot table to contain a pointer
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 12/17] qcow2: Use new metadata overlap check function
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
` (10 preceding siblings ...)
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 11/17] qcow2/overlaps: Protect inactive L2 tables Max Reitz
@ 2015-08-05 17:33 ` Max Reitz
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 13/17] qcow2/overlaps: Add "memory limit reached" event Max Reitz
` (4 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:33 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
Make the static new overlap check function global and drop the old
function.
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
block/qcow2-overlap.c | 8 +---
block/qcow2-refcount.c | 120 -------------------------------------------------
2 files changed, 2 insertions(+), 126 deletions(-)
diff --git a/block/qcow2-overlap.c b/block/qcow2-overlap.c
index 54d5eb0..c0a3d6d 100644
--- a/block/qcow2-overlap.c
+++ b/block/qcow2-overlap.c
@@ -356,12 +356,8 @@ static int single_check_metadata_overlap(Qcow2MetadataList *mdl, int ign,
return window->bitmap[bitmap_i] & ~ign;
}
-/* This will replace qcow2_check_metadata_overlap() */
-static int check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset,
- int64_t size) __attribute__((used));
-
-static int check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset,
- int64_t size)
+int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset,
+ int64_t size)
{
BDRVQcowState *s = bs->opaque;
uint64_t start_cluster = offset >> s->cluster_bits;
diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c
index 2cdf535..940f424 100644
--- a/block/qcow2-refcount.c
+++ b/block/qcow2-refcount.c
@@ -2371,126 +2371,6 @@ fail:
return ret;
}
-#define overlaps_with(ofs, sz) \
- ranges_overlap(offset, size, ofs, sz)
-
-/*
- * Checks if the given offset into the image file is actually free to use by
- * looking for overlaps with important metadata sections (L1/L2 tables etc.),
- * i.e. a sanity check without relying on the refcount tables.
- *
- * The ign parameter specifies what checks not to perform (being a bitmask of
- * QCow2MetadataOverlap values), i.e., what sections to ignore.
- *
- * Returns:
- * - 0 if writing to this offset will not affect the mentioned metadata
- * - a positive QCow2MetadataOverlap value indicating one overlapping section
- * - a negative value (-errno) indicating an error while performing a check,
- * e.g. when bdrv_read failed on QCOW2_OL_INACTIVE_L2
- */
-int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset,
- int64_t size)
-{
- BDRVQcowState *s = bs->opaque;
- int chk = s->overlap_check & ~ign;
- int i, j;
-
- if (!size) {
- return 0;
- }
-
- if (chk & QCOW2_OL_MAIN_HEADER) {
- if (offset < s->cluster_size) {
- return QCOW2_OL_MAIN_HEADER;
- }
- }
-
- /* align range to test to cluster boundaries */
- size = align_offset(offset_into_cluster(s, offset) + size, s->cluster_size);
- offset = start_of_cluster(s, offset);
-
- if ((chk & QCOW2_OL_ACTIVE_L1) && s->l1_size) {
- if (overlaps_with(s->l1_table_offset, s->l1_size * sizeof(uint64_t))) {
- return QCOW2_OL_ACTIVE_L1;
- }
- }
-
- if ((chk & QCOW2_OL_REFCOUNT_TABLE) && s->refcount_table_size) {
- if (overlaps_with(s->refcount_table_offset,
- s->refcount_table_size * sizeof(uint64_t))) {
- return QCOW2_OL_REFCOUNT_TABLE;
- }
- }
-
- if ((chk & QCOW2_OL_SNAPSHOT_TABLE) && s->snapshots_size) {
- if (overlaps_with(s->snapshots_offset, s->snapshots_size)) {
- return QCOW2_OL_SNAPSHOT_TABLE;
- }
- }
-
- if ((chk & QCOW2_OL_INACTIVE_L1) && s->snapshots) {
- for (i = 0; i < s->nb_snapshots; i++) {
- if (s->snapshots[i].l1_size &&
- overlaps_with(s->snapshots[i].l1_table_offset,
- s->snapshots[i].l1_size * sizeof(uint64_t))) {
- return QCOW2_OL_INACTIVE_L1;
- }
- }
- }
-
- if ((chk & QCOW2_OL_ACTIVE_L2) && s->l1_table) {
- for (i = 0; i < s->l1_size; i++) {
- if ((s->l1_table[i] & L1E_OFFSET_MASK) &&
- overlaps_with(s->l1_table[i] & L1E_OFFSET_MASK,
- s->cluster_size)) {
- return QCOW2_OL_ACTIVE_L2;
- }
- }
- }
-
- if ((chk & QCOW2_OL_REFCOUNT_BLOCK) && s->refcount_table) {
- for (i = 0; i < s->refcount_table_size; i++) {
- if ((s->refcount_table[i] & REFT_OFFSET_MASK) &&
- overlaps_with(s->refcount_table[i] & REFT_OFFSET_MASK,
- s->cluster_size)) {
- return QCOW2_OL_REFCOUNT_BLOCK;
- }
- }
- }
-
- if ((chk & QCOW2_OL_INACTIVE_L2) && s->snapshots) {
- for (i = 0; i < s->nb_snapshots; i++) {
- uint64_t l1_ofs = s->snapshots[i].l1_table_offset;
- uint32_t l1_sz = s->snapshots[i].l1_size;
- uint64_t l1_sz2 = l1_sz * sizeof(uint64_t);
- uint64_t *l1 = g_try_malloc(l1_sz2);
- int ret;
-
- if (l1_sz2 && l1 == NULL) {
- return -ENOMEM;
- }
-
- ret = bdrv_pread(bs->file, l1_ofs, l1, l1_sz2);
- if (ret < 0) {
- g_free(l1);
- return ret;
- }
-
- for (j = 0; j < l1_sz; j++) {
- uint64_t l2_ofs = be64_to_cpu(l1[j]) & L1E_OFFSET_MASK;
- if (l2_ofs && overlaps_with(l2_ofs, s->cluster_size)) {
- g_free(l1);
- return QCOW2_OL_INACTIVE_L2;
- }
- }
-
- g_free(l1);
- }
- }
-
- return 0;
-}
-
static const char *metadata_ol_names[] = {
[QCOW2_OL_MAIN_HEADER_BITNR] = "qcow2_header",
[QCOW2_OL_ACTIVE_L1_BITNR] = "active L1 table",
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 13/17] qcow2/overlaps: Add "memory limit reached" event
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
` (11 preceding siblings ...)
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 12/17] qcow2: Use new metadata overlap check function Max Reitz
@ 2015-08-05 17:33 ` Max Reitz
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 14/17] qcow2/overlaps: Add memory usage limit Max Reitz
` (3 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:33 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
Later, a mechanism to set a limit on how much memory may be used for the
overlap prevention structures will be introduced. If that limit is about
to be exceeded, a QMP event should be emitted. This very event is
specified by this patch.
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
docs/qmp/qmp-events.txt | 27 +++++++++++++++++++++++++++
qapi/event.json | 27 +++++++++++++++++++++++++++
2 files changed, 54 insertions(+)
diff --git a/docs/qmp/qmp-events.txt b/docs/qmp/qmp-events.txt
index d92cc48..00c70d1 100644
--- a/docs/qmp/qmp-events.txt
+++ b/docs/qmp/qmp-events.txt
@@ -279,6 +279,33 @@ Example:
{ "event": "POWERDOWN",
"timestamp": { "seconds": 1267040730, "microseconds": 682951 } }
+QCOW2_OVERLAP_CHECK_MEMORY_LIMIT_REACHED
+----------------------------------------
+
+Emitted by the qcow2 block driver if the preset size limit for the in-memory
+structures for metadata overlap prevention has been reached and an allocation
+of further resources has been denied. This means that it cannot be guaranteed
+that overlap checks will be performed for the specified range; if no range is
+given, no checks can be expected to be performed whatsoever.
+
+Note that this event does not guarantee that no check will be performed in the
+given range (or on the whole image, if no range is given), but only signals the
+possibility that this might be the case. The ability to check the full range or
+a part of it may be restored at any point in time.
+
+Data:
+- "reference": Device name if set; node name otherwise. (json-string)
+- "start": Offset of the range of clusters (possibly) no longer being
+ checked for writes overlapping with existing metadata.
+ (json-int, optional)
+- "length": Length of that range in bytes. (json-int, optional)
+
+Example:
+
+{ "event": "QCOW2_OVERLAP_CHECK_MEMORY_LIMIT_REACHED",
+ "data": { "reference": "virtio0", "start": 805306368, "length": 268435456 },
+ "timestamp": { "seconds": 1429331400, "microseconds": 519454 } }
+
QUORUM_FAILURE
--------------
diff --git a/qapi/event.json b/qapi/event.json
index f0cef01..996cd3d 100644
--- a/qapi/event.json
+++ b/qapi/event.json
@@ -356,3 +356,30 @@
##
{ 'event': 'MEM_UNPLUG_ERROR',
'data': { 'device': 'str', 'msg': 'str' } }
+
+##
+# @QCOW2_OVERLAP_CHECK_MEMORY_LIMIT_REACHED
+#
+# Emitted by the qcow2 block driver if the preset size limit for the in-memory
+# structures for metadata overlap prevention has been reached and an allocation
+# of further resources has been denied. This means that no overlap checks will
+# be performed for the specified range.
+#
+# Note that this event does not guarantee that no check will be performed in the
+# given range (or on the whole image, if no range is given), but only signals
+# the possibility that this might be the case. The ability to check the full
+# range or a part of it may be restored at any point in time.
+#
+# @reference: device name if set; node name otherwise
+#
+# @start: #optional offset of the range of clusters (possibly) no longer
+# being checked for writes overlapping with existing metadata
+#
+# @length: #optional length of that range in bytes
+#
+# Since: 2.5
+##
+{ 'event': 'QCOW2_OVERLAP_CHECK_MEMORY_LIMIT_REACHED',
+ 'data': { 'reference': 'str',
+ '*start': 'int',
+ '*length': 'int' } }
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 14/17] qcow2/overlaps: Add memory usage limit
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
` (12 preceding siblings ...)
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 13/17] qcow2/overlaps: Add "memory limit reached" event Max Reitz
@ 2015-08-05 17:33 ` Max Reitz
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 15/17] qcow2: Add overlap structure memory size options Max Reitz
` (2 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:33 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
This adds an adjustable limit for the total memory usage of the overlap
prevention structures.
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
block/qcow2-overlap.c | 183 +++++++++++++++++++++++++++++++++++++++++++++++---
block/qcow2.c | 2 +-
block/qcow2.h | 2 +-
3 files changed, 177 insertions(+), 10 deletions(-)
diff --git a/block/qcow2-overlap.c b/block/qcow2-overlap.c
index c0a3d6d..a2ba5c7 100644
--- a/block/qcow2-overlap.c
+++ b/block/qcow2-overlap.c
@@ -23,9 +23,11 @@
*/
#include "block/block_int.h"
+#include "sysemu/block-backend.h"
#include "qemu-common.h"
#include "qemu/range.h"
#include "qcow2.h"
+#include "qapi-event.h"
/* Number of clusters which are covered by each metadata window;
* note that this may not exceed 2^16 as long as
@@ -68,12 +70,50 @@ struct Qcow2MetadataList {
unsigned current_age;
+ size_t mem_usage, max_mem_usage;
+
/* Index into the windows array */
int *cached_windows;
size_t nb_cached_windows;
};
/**
+ * Logs newly allocated memory in the @mdl->mem_usage field. Returns true iff
+ * @mdl->mem_usage + @size * @nmemb <= @mdl->max_mem_usage, that is, if
+ * allocating that memory is still within limits. In that case, @mdl->mem_usage
+ * is increased by @size * @nmemb; otherwise, it is left unchanged.
+ */
+static bool increase_mdl_mem_usage(Qcow2MetadataList *mdl, size_t size,
+ size_t nmemb)
+{
+ if (nmemb && SIZE_MAX / nmemb < size) {
+ /* Overflow */
+ return false;
+ }
+
+ size *= nmemb;
+ if (mdl->mem_usage + size < mdl->mem_usage ||
+ mdl->mem_usage + size > mdl->max_mem_usage)
+ {
+ return false;
+ }
+
+ mdl->mem_usage += size;
+ return true;
+}
+
+static void signal_memory_excess(BlockDriverState *bs, uint64_t start_cluster,
+ uint64_t nb_clusters)
+{
+ BDRVQcowState *s = bs->opaque;
+ const char *ref = bs->blk ? blk_name(bs->blk) : bs->node_name;
+
+ qapi_event_send_qcow2_overlap_check_memory_limit_reached(ref, true,
+ start_cluster * s->cluster_size, true, nb_clusters * s->cluster_size,
+ &error_abort);
+}
+
+/**
* Destroys the cached window bitmap. If it has been modified, the fragment list
* will be rebuilt accordingly.
*/
@@ -81,11 +121,16 @@ static void destroy_window_bitmap(Qcow2MetadataList *mdl,
Qcow2MetadataWindow *window)
{
Qcow2MetadataFragment *new_fragments;
+ bool increase_ret;
if (!window->bitmap) {
return;
}
+ /* Treat the bitmap as freed already; while not really correct, this gives
+ * us space for the fragment list */
+ mdl->mem_usage -= WINDOW_SIZE;
+
if (window->bitmap_modified) {
int bitmap_i, fragment_i = 0;
QCow2MetadataOverlap current_types = 0;
@@ -105,6 +150,9 @@ static void destroy_window_bitmap(Qcow2MetadataList *mdl,
} else {
if (current_types && current_nb_clusters) {
if (fragment_i >= window->fragments_array_size) {
+ mdl->mem_usage -= sizeof(Qcow2MetadataFragment)
+ * window->fragments_array_size;
+
window->fragments_array_size =
3 * window->fragments_array_size / 2 + 1;
if (sizeof(Qcow2MetadataFragment)
@@ -115,6 +163,16 @@ static void destroy_window_bitmap(Qcow2MetadataList *mdl,
goto fail;
}
+ increase_ret = increase_mdl_mem_usage(mdl,
+ sizeof(Qcow2MetadataFragment),
+ window->fragments_array_size);
+ /* This can never be false: Because mem_usage was
+ * reduced by WINDOW_SIZE at the beginning of this
+ * function and the size of the fragment list is certain
+ * to be less than WINDOW_SIZE, there must be enough
+ * memory available. */
+ assert(increase_ret);
+
new_fragments =
g_try_renew(Qcow2MetadataFragment,
window->fragments,
@@ -148,6 +206,7 @@ static void destroy_window_bitmap(Qcow2MetadataList *mdl,
new_fragments = g_try_renew(Qcow2MetadataFragment, window->fragments,
window->nb_fragments);
if (new_fragments) {
+ mdl->mem_usage -= window->fragments_array_size - window->nb_fragments;
window->fragments = new_fragments;
window->fragments_array_size = window->nb_fragments;
}
@@ -163,12 +222,50 @@ fail:
window->fragments = NULL;
window->nb_fragments = 0;
window->fragments_array_size = 0;
+
+ /* window->bitmap is not destroyed, so we need to account for it (because
+ * we "borrowed" the space before) */
+ increase_ret = increase_mdl_mem_usage(mdl, sizeof(uint8_t), WINDOW_SIZE);
+ /* If we land here, no additional memory was allocated (only
+ * window->fragments may grow in this function, but at this point it has
+ * been freed), so we must be able to reclaim the space */
+ assert(increase_ret);
+}
+
+/**
+ * Destroys the oldest bitmap.
+ * Returns the index in mdl->cached_windows[] of the window whose bitmap was
+ * destroyed, or -1 if the cache is empty.
+ */
+static int destroy_oldest_bitmap(Qcow2MetadataList *mdl)
+{
+ int cache_i, oldest_cache_i = -1;
+ unsigned oldest_cache_age = 0;
+
+ for (cache_i = 0; cache_i < mdl->nb_cached_windows; cache_i++) {
+ if (mdl->cached_windows[cache_i] >= 0) {
+ unsigned age = mdl->current_age
+ - mdl->windows[mdl->cached_windows[cache_i]].age;
+ if (age > oldest_cache_age) {
+ oldest_cache_age = age;
+ oldest_cache_i = cache_i;
+ }
+ }
+ }
+
+ if (oldest_cache_i < 0) {
+ return -1;
+ }
+
+ destroy_window_bitmap(mdl,
+ &mdl->windows[mdl->cached_windows[oldest_cache_i]]);
+ return oldest_cache_i;
}
/**
* Creates a bitmap from the fragment list.
*/
-static void build_window_bitmap(Qcow2MetadataList *mdl,
+static bool build_window_bitmap(Qcow2MetadataList *mdl,
Qcow2MetadataWindow *window)
{
int cache_i, oldest_cache_i = -1, i;
@@ -202,9 +299,21 @@ static void build_window_bitmap(Qcow2MetadataList *mdl,
/* Maybe there already is a bitmap because it was more space-efficient than
* the range list representation */
if (window->bitmap) {
- return;
+ return true;
}
+ while (!increase_mdl_mem_usage(mdl, sizeof(uint8_t), WINDOW_SIZE)) {
+ /* It is better to try to decrease memory usage until this bitmap can
+ * fit than just to give up. Otherwise, we may end up at a point where
+ * the area covered by the cache remains unchanged forever. */
+ mdl->cached_windows[cache_i] = -1;
+ cache_i = destroy_oldest_bitmap(mdl);
+ if (cache_i < 0) {
+ window->bitmap = NULL;
+ return false;
+ }
+ mdl->cached_windows[cache_i] = window - mdl->windows;
+ }
window->bitmap = g_new0(uint8_t, WINDOW_SIZE);
for (i = 0; i < window->nb_fragments; i++) {
@@ -215,6 +324,8 @@ static void build_window_bitmap(Qcow2MetadataList *mdl,
}
window->bitmap_modified = false;
+
+ return true;
}
/**
@@ -251,10 +362,24 @@ void qcow2_metadata_list_enter(BlockDriverState *bs, uint64_t offset,
* the array to exactly the required size */
Qcow2MetadataWindow *new_windows;
+ if (!increase_mdl_mem_usage(s->metadata_list,
+ sizeof(Qcow2MetadataWindow),
+ window_i + 1
+ - s->metadata_list->nb_windows))
+ {
+ /* This will happen for any cluster from here until end_cluster,
+ * so we can just abort immediately */
+ signal_memory_excess(bs, window_i * WINDOW_SIZE,
+ end_cluster - current_cluster);
+ return;
+ }
+
new_windows = g_try_renew(Qcow2MetadataWindow,
s->metadata_list->windows,
window_i + 1);
if (!new_windows) {
+ signal_memory_excess(bs, window_i * WINDOW_SIZE,
+ end_cluster - current_cluster);
return;
}
@@ -268,7 +393,10 @@ void qcow2_metadata_list_enter(BlockDriverState *bs, uint64_t offset,
window = &s->metadata_list->windows[window_i];
if (!window->bitmap) {
- build_window_bitmap(s->metadata_list, window);
+ if (!build_window_bitmap(s->metadata_list, window)) {
+ signal_memory_excess(bs, window_i * WINDOW_SIZE, WINDOW_SIZE);
+ goto next_window;
+ }
}
for (bitmap_i = bitmap_i_start; bitmap_i < bitmap_i_end; bitmap_i++) {
@@ -278,6 +406,7 @@ void qcow2_metadata_list_enter(BlockDriverState *bs, uint64_t offset,
window->age = s->metadata_list->current_age++;
window->bitmap_modified = true;
+next_window:
/* Go to the next window */
current_cluster += WINDOW_SIZE - bitmap_i_start;
}
@@ -320,7 +449,17 @@ void qcow2_metadata_list_remove(BlockDriverState *bs, uint64_t offset,
window = &s->metadata_list->windows[window_i];
if (!window->bitmap) {
- build_window_bitmap(s->metadata_list, window);
+ if (!build_window_bitmap(s->metadata_list, window)) {
+ signal_memory_excess(bs, window_i * WINDOW_SIZE, WINDOW_SIZE);
+ /* We *must* drop the given metadata types from the list, no
+ * matter what; therefore, the best thing we can do is empty
+ * the fragment list */
+ g_free(window->fragments);
+ window->fragments = NULL;
+ window->nb_fragments = 0;
+ window->fragments_array_size = 0;
+ goto next_window;
+ }
}
for (bitmap_i = bitmap_i_start; bitmap_i < bitmap_i_end; bitmap_i++) {
@@ -330,12 +469,14 @@ void qcow2_metadata_list_remove(BlockDriverState *bs, uint64_t offset,
window->age = s->metadata_list->current_age++;
window->bitmap_modified = true;
+next_window:
/* Go to the next window */
current_cluster += WINDOW_SIZE - bitmap_i_start;
}
}
-static int single_check_metadata_overlap(Qcow2MetadataList *mdl, int ign,
+static int single_check_metadata_overlap(BlockDriverState *bs,
+ Qcow2MetadataList *mdl, int ign,
uint64_t cluster)
{
uint64_t window_i = cluster / WINDOW_SIZE;
@@ -348,7 +489,10 @@ static int single_check_metadata_overlap(Qcow2MetadataList *mdl, int ign,
window = &mdl->windows[window_i];
if (!window->bitmap) {
- build_window_bitmap(mdl, window);
+ if (!build_window_bitmap(mdl, window)) {
+ signal_memory_excess(bs, window_i * WINDOW_SIZE, WINDOW_SIZE);
+ return 0;
+ }
}
window->age = mdl->current_age++;
@@ -372,7 +516,7 @@ int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset,
for (current_cluster = start_cluster; current_cluster < end_cluster;
current_cluster++)
{
- ret |= single_check_metadata_overlap(s->metadata_list, ign,
+ ret |= single_check_metadata_overlap(bs, s->metadata_list, ign,
current_cluster);
}
@@ -380,16 +524,33 @@ int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset,
}
int qcow2_create_empty_metadata_list(BlockDriverState *bs, size_t cache_size,
- Error **errp)
+ size_t max_total_mem_size, Error **errp)
{
BDRVQcowState *s = bs->opaque;
int ret;
size_t cache_entries, i;
+ if (max_total_mem_size < sizeof(Qcow2MetadataList)) {
+ error_setg(errp, "Cannot allocate metadata list");
+ ret = -ENOMEM;
+ goto fail;
+ }
+
s->metadata_list = g_new0(Qcow2MetadataList, 1);
+ s->metadata_list->max_mem_usage = max_total_mem_size;
+ s->metadata_list->mem_usage = sizeof(Qcow2MetadataList);
+
s->metadata_list->nb_windows =
DIV_ROUND_UP(bdrv_nb_sectors(bs->file),
(uint64_t)s->cluster_sectors * WINDOW_SIZE);
+ if (!increase_mdl_mem_usage(s->metadata_list, sizeof(Qcow2MetadataWindow),
+ s->metadata_list->nb_windows))
+ {
+ error_setg(errp, "Cannot allocate metadata overlap check windows");
+ ret = -ENOMEM;
+ goto fail;
+ }
+
s->metadata_list->windows = g_try_new0(Qcow2MetadataWindow,
s->metadata_list->nb_windows);
if (s->metadata_list->nb_windows && !s->metadata_list->windows) {
@@ -408,6 +569,12 @@ int qcow2_create_empty_metadata_list(BlockDriverState *bs, size_t cache_size,
}
s->metadata_list->nb_cached_windows = cache_entries;
+ if (!increase_mdl_mem_usage(s->metadata_list, sizeof(int), cache_entries)) {
+ error_setg(errp, "Cannot allocate metadata overlap cache pointers");
+ ret = -ENOMEM;
+ goto fail;
+ }
+
s->metadata_list->cached_windows = g_try_new(int, cache_entries);
if (!s->metadata_list->cached_windows) {
error_setg(errp, "Could not allocate metadata overlap cache pointers");
diff --git a/block/qcow2.c b/block/qcow2.c
index 0e7b646..d1257b1 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -768,7 +768,7 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,
if (s->overlap_check) {
/* TODO: Let the user override this default */
- ret = qcow2_create_empty_metadata_list(bs, 65536, errp);
+ ret = qcow2_create_empty_metadata_list(bs, 65536, SIZE_MAX, errp);
if (ret < 0) {
goto fail;
}
diff --git a/block/qcow2.h b/block/qcow2.h
index e48dcdc..7f4059e 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -595,7 +595,7 @@ void qcow2_cache_put(BlockDriverState *bs, Qcow2Cache *c, void **table);
/* qcow2-overlap.c functions */
int qcow2_create_empty_metadata_list(BlockDriverState *bs, size_t cache_size,
- Error **errp);
+ size_t max_total_mem_size, Error **errp);
void qcow2_metadata_list_destroy(BlockDriverState *bs);
void qcow2_metadata_list_enter(BlockDriverState *bs, uint64_t offset,
int nb_clusters, QCow2MetadataOverlap type);
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 15/17] qcow2: Add overlap structure memory size options
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
` (13 preceding siblings ...)
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 14/17] qcow2/overlaps: Add memory usage limit Max Reitz
@ 2015-08-05 17:33 ` Max Reitz
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 16/17] qapi: Expose new qcow2 overlap check options Max Reitz
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 17/17] iotests: Test qcow2's overlap check memory limit Max Reitz
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:33 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
Add runtime options to qcow2 to control the size of the structures used
for metadata overlap prevention (one option to control the size of the
bitmap cache, another one to control the total memory size limit).
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
block/qcow2.c | 35 +++++++++++++++++++++++++++++++++--
block/qcow2.h | 2 ++
2 files changed, 35 insertions(+), 2 deletions(-)
diff --git a/block/qcow2.c b/block/qcow2.c
index d1257b1..d3f2e71 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -452,6 +452,18 @@ static QemuOptsList qcow2_runtime_opts = {
.help = "Check for unintended writes into an inactive L2 table",
},
{
+ .name = QCOW2_OPT_OVERLAP_BITMAP_SIZE,
+ .type = QEMU_OPT_SIZE,
+ .help = "Size for the bitmap cache used by the metadata overlap "
+ "check",
+ },
+ {
+ .name = QCOW2_OPT_OVERLAP_TOTAL_SIZE_LIMIT,
+ .type = QEMU_OPT_SIZE,
+ .help = "Maximum total memory size to be used for the metadata "
+ "overlap check structures",
+ },
+ {
.name = QCOW2_OPT_CACHE_SIZE,
.type = QEMU_OPT_SIZE,
.help = "Maximum combined metadata (L2 tables and refcount blocks) "
@@ -767,8 +779,27 @@ static int qcow2_open(BlockDriverState *bs, QDict *options, int flags,
s->cluster_offset_mask = (1LL << s->csize_shift) - 1;
if (s->overlap_check) {
- /* TODO: Let the user override this default */
- ret = qcow2_create_empty_metadata_list(bs, 65536, SIZE_MAX, errp);
+ uint64_t cache_size, total_size;
+
+ cache_size = qemu_opt_get_size(opts, QCOW2_OPT_OVERLAP_BITMAP_SIZE,
+ 65536);
+ if (cache_size > SIZE_MAX) {
+ error_setg(errp, "qcow2 option '" QCOW2_OPT_OVERLAP_BITMAP_SIZE
+ "' expects an argument less or equal to %zu", SIZE_MAX);
+ ret = -EINVAL;
+ goto fail;
+ }
+
+ total_size = qemu_opt_get_size(opts, QCOW2_OPT_OVERLAP_TOTAL_SIZE_LIMIT,
+ SIZE_MAX);
+ if (total_size > SIZE_MAX) {
+ error_setg(errp, "qcow2 option '" QCOW2_OPT_OVERLAP_TOTAL_SIZE_LIMIT
+ "' expects an argument less or equal to %zu", SIZE_MAX);
+ ret = -EINVAL;
+ goto fail;
+ }
+
+ ret = qcow2_create_empty_metadata_list(bs, cache_size, total_size, errp);
if (ret < 0) {
goto fail;
}
diff --git a/block/qcow2.h b/block/qcow2.h
index 7f4059e..400fe52 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -93,6 +93,8 @@
#define QCOW2_OPT_OVERLAP_SNAPSHOT_TABLE "overlap-check.snapshot-table"
#define QCOW2_OPT_OVERLAP_INACTIVE_L1 "overlap-check.inactive-l1"
#define QCOW2_OPT_OVERLAP_INACTIVE_L2 "overlap-check.inactive-l2"
+#define QCOW2_OPT_OVERLAP_BITMAP_SIZE "overlap-structures.bitmap-size"
+#define QCOW2_OPT_OVERLAP_TOTAL_SIZE_LIMIT "overlap-structures.total-size-limit"
#define QCOW2_OPT_CACHE_SIZE "cache-size"
#define QCOW2_OPT_L2_CACHE_SIZE "l2-cache-size"
#define QCOW2_OPT_REFCOUNT_CACHE_SIZE "refcount-cache-size"
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 16/17] qapi: Expose new qcow2 overlap check options
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
` (14 preceding siblings ...)
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 15/17] qcow2: Add overlap structure memory size options Max Reitz
@ 2015-08-05 17:33 ` Max Reitz
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 17/17] iotests: Test qcow2's overlap check memory limit Max Reitz
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:33 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
Expose the two new options for controlling the memory usage of the
overlap check implementation via QAPI.
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
qapi/block-core.json | 31 +++++++++++++++++++++++++++++++
1 file changed, 31 insertions(+)
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 7b2efb8..0eb9c69 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1562,6 +1562,33 @@
'mode': 'Qcow2OverlapCheckMode' } }
##
+# @Qcow2OverlapStructures
+#
+# Contains options for controlling the behavior of the metadata overlap
+# prevention structures.
+#
+# These structures contain the position, length, and type of the metadata
+# structures; one contains all of them in compressed form, the other one takes
+# up more space, but has faster access times. Therefore, the latter acts as a
+# cache of the former.
+#
+# @cache-size: #optional size (in bytes) of the cache, defaults to 64 kB
+#
+# @total-size-limit: #optional maximum total size (in bytes) of all the metadata
+# overlap prevention data structures combined; if this limit
+# is exceeded, a QCOW2_OVERLAP_CHECK_MEMORY_LIMIT_REACHED
+# event will be emitted and some parts of the image may no
+# longer be protected against erroneous overwriting of
+# metadata by (meta)data of a different kind. Defaults to
+# SIZE_MAX.
+#
+# Since: 2.5
+##
+{ 'struct': 'Qcow2OverlapStructures',
+ 'data': { '*bitmap-size': 'int',
+ '*total-size-limit': 'int' } }
+
+##
# @BlockdevOptionsQcow2
#
# Driver specific block device options for qcow2.
@@ -1583,6 +1610,9 @@
# @overlap-check: #optional which overlap checks to perform for writes
# to the image, defaults to 'cached' (since 2.2)
#
+# @overlap-structures: #optional options for controlling the behavior of the
+# metadata overlap prevention structures (since 2.5)
+#
# @cache-size: #optional the maximum total size of the L2 table and
# refcount block caches in bytes (since 2.2)
#
@@ -1601,6 +1631,7 @@
'*pass-discard-snapshot': 'bool',
'*pass-discard-other': 'bool',
'*overlap-check': 'Qcow2OverlapChecks',
+ '*overlap-structures': 'Qcow2OverlapStructures',
'*cache-size': 'int',
'*l2-cache-size': 'int',
'*refcount-cache-size': 'int' } }
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [Qemu-devel] [PATCH v6 17/17] iotests: Test qcow2's overlap check memory limit
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
` (15 preceding siblings ...)
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 16/17] qapi: Expose new qcow2 overlap check options Max Reitz
@ 2015-08-05 17:33 ` Max Reitz
16 siblings, 0 replies; 18+ messages in thread
From: Max Reitz @ 2015-08-05 17:33 UTC (permalink / raw)
To: qemu-block; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi, Max Reitz
This patch adds some test cases for the memory limit concerning the
in-memory structures used to detect and prevent accidental metadata
overlaps.
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
tests/qemu-iotests/060 | 222 +++++++++++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/060.out | 47 ++++++++++
tests/qemu-iotests/group | 2 +-
3 files changed, 270 insertions(+), 1 deletion(-)
diff --git a/tests/qemu-iotests/060 b/tests/qemu-iotests/060
index c81319c..b0bc86a 100755
--- a/tests/qemu-iotests/060
+++ b/tests/qemu-iotests/060
@@ -37,6 +37,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
# get standard environment, filters and checks
. ./common.rc
. ./common.filter
+. ./common.qemu
# This tests qocw2-specific low-level functionality
_supported_fmt qcow2
@@ -243,6 +244,227 @@ poke_file "$TEST_IMG" "$(($l2_offset+8))" "\x80\x00\x00\x00\x00\x06\x2a\x00"
# Should emit two error messages
$QEMU_IO -c "discard 0 64k" -c "read 64k 64k" "$TEST_IMG" | _filter_qemu_io
+echo
+echo "=== Testing memory limit ==="
+echo
+
+_make_test_img 64M
+# Use blockdev-add instead of adding the drive at startup, so that events which
+# are generated when the image is opened will be sent over QMP
+_launch_qemu
+_send_qemu_cmd $QEMU_HANDLE "{ 'execute': 'qmp_capabilities' }" 'return'
+
+echo
+echo '--- Zero limit ---'
+echo
+
+# This should fail (because allocating the metadata structure list itself fails)
+_send_qemu_cmd $QEMU_HANDLE \
+ "{ 'execute': 'blockdev-add'," \
+ " 'arguments': {" \
+ " 'options': {" \
+ " 'id': 'drv0'," \
+ " 'driver': 'qcow2'," \
+ " 'overlap-structures': {" \
+ " 'total-size-limit': 0" \
+ " }," \
+ " 'file': {" \
+ " 'driver': 'file'," \
+ " 'filename': '$TEST_IMG'" \
+ " } } } }" \
+ 'error'
+
+echo
+echo '--- Zero limit with overlap checks disabled ---'
+echo
+
+# This should work (because overlap checks are disabled)
+_send_qemu_cmd $QEMU_HANDLE \
+ "{ 'execute': 'blockdev-add'," \
+ " 'arguments': {" \
+ " 'options': {" \
+ " 'id': 'drv1'," \
+ " 'driver': 'qcow2'," \
+ " 'overlap-check': 'none'," \
+ " 'overlap-structures': {" \
+ " 'total-size-limit': 0" \
+ " }," \
+ " 'file': {" \
+ " 'driver': 'file'," \
+ " 'filename': '$TEST_IMG'" \
+ " } } } }" \
+ 'return'
+
+echo
+echo '--- Limit too small for entering metadata ---'
+echo
+
+# For the initial metadata structures, the following data is required:
+# - Qcow2MetadataList (32 B on x86, 56 B on x64)
+# - nb_windows * Qcow2MetadataWindow (nb_windows is
+# ceil(x / (64k * WINDOW_SIZE)) = 1 (WINDOW_SIZE = 4096; x is the size of the
+# underlying file, which is probably below 1 MB)); therefore, this is 24 B on
+# x86, and 32 B on x64)
+# - cache_entries * int (cache_entries is cache_size / WINDOW_SIZE =
+# 65536 / 4096 = 16, therefore this is 64 B on both x86 and x64)
+# In total, we need 120 B on x86 and 152 B on x64 (Linux/gcc).
+#
+# Creating a single bytemap takes WINDOW_SIZE (4096) B. Therefore, setting the
+# limit to 256 B will definitely suffice for the structures required to create
+# the metadata structures, but trying to enter even a single metadata structure
+# will fail.
+# The "start" field of the event(s) generated will be 0; the "length" field will
+# be WINDOW_SIZE * cluster_size = 4096 * 64k = 256M.
+_send_qemu_cmd $QEMU_HANDLE \
+ "{ 'execute': 'blockdev-add'," \
+ " 'arguments': {" \
+ " 'options': {" \
+ " 'id': 'drv2'," \
+ " 'driver': 'qcow2'," \
+ " 'overlap-structures': {" \
+ " 'bitmap-size': 65536," \
+ " 'total-size-limit': 256" \
+ " }," \
+ " 'file': {" \
+ " 'driver': 'file'," \
+ " 'filename': '$TEST_IMG'" \
+ " } } } }" \
+ 'return'
+
+_cleanup_qemu
+
+echo
+echo '--- Limit too small for both bitmap and fragment list ---'
+echo
+
+# Now test the case of trying to convert the bitmap back to the fragment list,
+# but where memory does not suffice to hold both. Because the bitmap is released
+# after the fragment list has been created, it should work anyway, because the
+# implementation can consider the bitmap freed before it is actually freed.
+# In order for this to work on both x86 and x64, the fragment list needs to take
+# up more than 32 B (152 - 120) of memory. Every entry has 4 B, so we need 9
+# entries. One is the image header, then there is the L1 table, the refcount
+# table, a refcount block, and then there are more than 1000 free clusters; a
+# maximum of 256 clusters can be represented by a single fragment, so these are
+# another 4 entries. In total, there are thus 8 entries so far, so we need
+# another one, which we can do by writing data and thus creating an L2 table.
+$QEMU_IO -c 'write 0 64k' "$TEST_IMG" | _filter_qemu_io
+
+# Also, we need to point to an offset beyond WINDOW_SIZE * cluster_size (256M),
+# so that we can make qcow2 evict the first window.
+poke_file "$TEST_IMG" "$(($l2_offset+8))" "\x80\x00\x00\x00\x10\x00\x00\x00"
+
+_launch_qemu
+_send_qemu_cmd $QEMU_HANDLE "{ 'execute': 'qmp_capabilities' }" 'return'
+
+# For the planned cache eviction to work, bitmap-size must be WINDOW_SIZE;
+# therefore, the initial metadata structure size shrinks by 15 * sizeof(int),
+# that is, 60 B on both x86 and x64.
+# The total-size-limit should thus be 92 + 4096 = 4188
+_send_qemu_cmd $QEMU_HANDLE \
+ "{ 'execute': 'blockdev-add'," \
+ " 'arguments': {" \
+ " 'options': {" \
+ " 'id': 'drv0'," \
+ " 'driver': 'qcow2'," \
+ " 'overlap-structures': {" \
+ " 'bitmap-size': 4096," \
+ " 'total-size-limit': 4188" \
+ " }," \
+ " 'file': {" \
+ " 'driver': 'file'," \
+ " 'filename': '$TEST_IMG'" \
+ " } } } }" \
+ 'return'
+
+# Writing something to the second data cluster, thus making qemu load the second
+# metadata window, evicting the first one
+_send_qemu_cmd $QEMU_HANDLE \
+ "{ 'execute': 'human-monitor-command'," \
+ " 'arguments': { 'command-line': 'qemu-io drv0 \"write 64k 64k\"' } }" \
+ 'return'
+
+_cleanup_qemu
+
+echo
+echo '--- Fragment list is larger than bitmap ---'
+echo
+
+# Finally, test that bitmaps are not converted to fragment lists if the fragment
+# list actually is longer than the bitmap.
+CLUSTER_SIZE=512 _make_test_img 64M
+
+# Every L2 table describes 512 / 8 = 64 clusters, that is, a range of 32k.
+# Therefore, writing clusters 32k apart results in a alternating pattern of
+# L2 table and data cluster. If we write 1024 clusters in this way, the fragment
+# list for the first window would therefore have to contain at least 2048
+# fragments, which is longer (8 kB) than the bitmap is (4 kB fixed). Therefore,
+# the fragment list should not be generated on cache eviction.
+(for i in $(seq 0 1023); do
+ echo "write $(($i*32))k 512"
+done) | $QEMU_IO -t unsafe "$TEST_IMG" &> /dev/null
+
+l2_offset_512=$((0x4600))
+
+# Now point the first data cluster to WINDOW_SIZE * 512 = 2M
+# (the image length is about 2048 * 512 = 1M at this point)
+poke_file "$TEST_IMG" "$(($l2_offset_512))" "\x80\x00\x00\x00\x00\x20\x00\x00"
+# (the first cluster is now leaked, but who cares)
+
+# "Allocate" that cluster
+$QEMU_IO -c 'write 0 512' "$TEST_IMG" | _filter_qemu_io
+
+# And point the second data cluster to the first L2 table so we can test whether
+# the overlap check still works
+poke_file "$TEST_IMG" "$(($l2_offset_512+8))" "\x80\x00\x00\x00\x00\x00\x46\x00"
+
+cp "$TEST_IMG" /tmp
+
+_launch_qemu
+_send_qemu_cmd $QEMU_HANDLE "{ 'execute': 'qmp_capabilities' }" 'return'
+
+# Memory which we will allow to be allocated:
+# - Qcow2MetadataList (32 B on x86, 56 B on x64)
+# - nb_windows * Qcow2MetadataWindow (nb_windows is 2, because we made sure
+# that we would have an image with two windows; therefore, this is 48 B on
+# x86, and 64 B on x64)
+# - cache_entries * int (cache_entries is cache_size / WINDOW_SIZE =
+# 4096 / 4096 = 1, therefore this is 4 B on both x86 and x64)
+# - One bitmap (WINDOW_SIZE = 4096 B)
+# In total, that is 4220 B at most. If the bytemap would be converted to the
+# according fragment list, it would require 8 kB, which is definitely more.
+_send_qemu_cmd $QEMU_HANDLE \
+ "{ 'execute': 'blockdev-add'," \
+ " 'arguments': {" \
+ " 'options': {" \
+ " 'id': 'drv0'," \
+ " 'driver': 'qcow2'," \
+ " 'overlap-structures': {" \
+ " 'bitmap-size': 4096," \
+ " 'total-size-limit': 4220" \
+ " }," \
+ " 'file': {" \
+ " 'driver': 'file'," \
+ " 'filename': '$TEST_IMG'" \
+ " } } } }" \
+ 'return'
+
+# Force cache eviction by accessing a cluster in the second window
+# (This will result in a "memory limit reached" event because there is no space
+# for the second bitmap)
+_send_qemu_cmd $QEMU_HANDLE \
+ "{ 'execute': 'human-monitor-command'," \
+ " 'arguments': { 'command-line': 'qemu-io drv0 \"write 0 512\"' } }" \
+ 'return'
+
+# Trigger the overlap prevention
+_send_qemu_cmd $QEMU_HANDLE \
+ "{ 'execute': 'human-monitor-command'," \
+ " 'arguments': { 'command-line': 'qemu-io drv0 \"write 512 512\"' } }" \
+ 'return'
+
+_cleanup_qemu
+
# success, all done
echo "*** done"
rm -f $seq.full
diff --git a/tests/qemu-iotests/060.out b/tests/qemu-iotests/060.out
index 7511189..f471c2d 100644
--- a/tests/qemu-iotests/060.out
+++ b/tests/qemu-iotests/060.out
@@ -180,4 +180,51 @@ qcow2: Marking image as corrupt: Data cluster offset 0x62a00 unaligned (L2 offse
discard 65536/65536 bytes at offset 0
64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
read failed: Input/output error
+
+=== Testing memory limit ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+{"return": {}}
+
+--- Zero limit ---
+
+{"error": {"class": "GenericError", "desc": "Cannot allocate metadata list"}}
+
+--- Zero limit with overlap checks disabled ---
+
+{"return": {}}
+
+--- Limit too small for entering metadata ---
+
+{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "QCOW2_OVERLAP_CHECK_MEMORY_LIMIT_REACHED", "data": {"length": 268435456, "start": 0, "reference": "drv2"}}
+{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "QCOW2_OVERLAP_CHECK_MEMORY_LIMIT_REACHED", "data": {"length": 268435456, "start": 0, "reference": "drv2"}}
+{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "QCOW2_OVERLAP_CHECK_MEMORY_LIMIT_REACHED", "data": {"length": 268435456, "start": 0, "reference": "drv2"}}
+{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "QCOW2_OVERLAP_CHECK_MEMORY_LIMIT_REACHED", "data": {"length": 268435456, "start": 0, "reference": "drv2"}}
+{"return": {}}
+
+--- Limit too small for both bitmap and fragment list ---
+
+wrote 65536/65536 bytes at offset 0
+64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+{"return": {}}
+{"return": {}}
+wrote 65536/65536 bytes at offset 65536
+64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+{"return": ""}
+
+--- Fragment list is larger than bitmap ---
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+wrote 512/512 bytes at offset 0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+{"return": {}}
+{"return": {}}
+{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "QCOW2_OVERLAP_CHECK_MEMORY_LIMIT_REACHED", "data": {"length": 2097152, "start": 2097152, "reference": "drv0"}}
+wrote 512/512 bytes at offset 0
+512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+{"return": ""}
+qcow2: Marking image as corrupt: Preventing invalid write on metadata (overlaps with active L2 table); further corruption events will be suppressed
+{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_IMAGE_CORRUPTED", "data": {"device": "drv0", "msg": "Preventing invalid write on metadata (overlaps with active L2 table)", "offset": 17920, "fatal": true, "size": 512}}
+write failed: Input/output error
+{"return": ""}
*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index c430b6c..dc80359 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -66,7 +66,7 @@
057 rw auto
058 rw auto quick
059 rw auto quick
-060 rw auto quick
+060 rw auto
061 rw auto
062 rw auto quick
063 rw auto quick
--
2.4.6
^ permalink raw reply related [flat|nested] 18+ messages in thread
end of thread, other threads:[~2015-08-05 17:34 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-08-05 17:32 [Qemu-devel] [PATCH v6 00/17] qcow2: Add new overlap check functions Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 01/17] " Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 02/17] qcow2: Pull up overlap check option evaluation Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 03/17] qcow2: Create metadata list Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 04/17] qcow2/overlaps: Protect image header Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 05/17] qcow2/overlaps: Protect refcount table Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 06/17] qcow2/overlaps: Protect refcount blocks Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 07/17] qcow2/overlaps: Protect active L1 table Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 08/17] qcow2/overlaps: Protect active L2 tables Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 09/17] qcow2/overlaps: Protect snapshot table Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 10/17] qcow2/overlaps: Protect inactive L1 tables Max Reitz
2015-08-05 17:32 ` [Qemu-devel] [PATCH v6 11/17] qcow2/overlaps: Protect inactive L2 tables Max Reitz
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 12/17] qcow2: Use new metadata overlap check function Max Reitz
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 13/17] qcow2/overlaps: Add "memory limit reached" event Max Reitz
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 14/17] qcow2/overlaps: Add memory usage limit Max Reitz
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 15/17] qcow2: Add overlap structure memory size options Max Reitz
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 16/17] qapi: Expose new qcow2 overlap check options Max Reitz
2015-08-05 17:33 ` [Qemu-devel] [PATCH v6 17/17] iotests: Test qcow2's overlap check memory limit Max Reitz
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.