* [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format
@ 2012-07-31 16:51 Dong Xu Wang
2012-07-31 16:51 ` [Qemu-devel] [PATCH 2/6 v11 v11] block: make some functions public Dong Xu Wang
` (6 more replies)
0 siblings, 7 replies; 30+ messages in thread
From: Dong Xu Wang @ 2012-07-31 16:51 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, Dong Xu Wang
Introduce a new file format:add-cow. The usage can be found at this patch.
Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
---
Now add-cow is still using QEMUOptionParameter, not QemuOpts, I will send a
seperate patch series to convert.
docs/specs/add-cow.txt | 128 ++++++++++++++++++++++++++++++++++++++++++++++++
1 files changed, 128 insertions(+), 0 deletions(-)
create mode 100644 docs/specs/add-cow.txt
diff --git a/docs/specs/add-cow.txt b/docs/specs/add-cow.txt
new file mode 100644
index 0000000..4793a3e
--- /dev/null
+++ b/docs/specs/add-cow.txt
@@ -0,0 +1,128 @@
+== General ==
+
+Raw file format does not support backing file and copy on write feature.
+The add-cow image format makes it possible to use backing files with raw
+image by keeping a separate .add-cow metadata file. Once all sectors
+have been written into the raw image it is safe to discard the .add-cow
+and backing files, then we can use the raw image directly.
+
+While using add-cow, procedures may like this:
+(ubuntu.img is a disk image which has been installed OS.)
+ 1) Create a raw image with the same size of ubuntu.img
+ qemu-img create -f raw test.raw 8G
+ 2) Create an add-cow image which will store dirty bitmap
+ qemu-img create -f add-cow test.add-cow \
+ -o backing_file=ubuntu.img,image_file=test.raw
+ 3) Run qemu with add-cow image
+ qemu -drive if=virtio,file=test.add-cow
+
+test.raw may be larger than ubuntu.img, in that case, the size of test.add-cow
+will be calculated by the size of ubuntu.img, test.raw will be used from the
+1st byte, the rest part can be used for other purpose.
+
+=Specification=
+
+The file format looks like this:
+
+ +---------------+-------------+-----------------+
+ | Header | Reserved | COW bitmap |
+ +---------------+-------------+-----------------+
+
+All numbers in add-cow are stored in Little Endian byte order.
+
+== Header ==
+
+The Header is included in the first bytes:
+(#define HEADER_SIZE (4096 * header_pages_size))
+ Byte 0 - 7: magic
+ add-cow magic string ("ADD_COW\xff").
+
+ 8 - 11: version
+ Version number (only valid value is 1 now).
+
+ 12 - 15: backing file name offset
+ Offset in the add-cow file at which the backing file
+ name is stored (NB: The string is not null terminated).
+ If backing file name does NOT exist, this field will be
+ 0. Must be between 80 and [HEADER_SIZE - 2](a file name
+ must be at least 1 byte).
+
+ 16 - 19: backing file name size
+ Length of the backing file name in bytes. It will be 0
+ if the backing file name offset is 0. If backing file
+ name offset is non-zero, then it must be non-zero. Must
+ be less than [HEADER_SIZE - 80] to fit in the reserved
+ part of the header.
+
+ 20 - 23: image file name offset
+ Offset in the add-cow file at which the image file name
+ is stored (NB: The string is not null terminated). It
+ must be between 80 and [HEADER_SIZE - 2].
+
+ 24 - 27: image file name size
+ Length of the image file name in bytes.
+ Must be less than [HEADER_SIZE - 80] to fit in the reserved
+ part of the header.
+
+ 28 - 35: features
+ Currently only 3 feature bit is used:
+ Feature bits:
+ The image uses a backing file:
+ * ADD_COW_F_BACKING_FILE = 0x01.
+ The image uses a image file:
+ * ADD_COW_F_IMAGE_FILE = 0x02.
+ All bits in bitmap have been set to 1, add-cow wrapper
+ can be discarded.
+ * ADD_COW_F_All_ALLOCATED = 0x04.
+
+ 36 - 43: optional features
+ Not used now. Researved for future use.
+
+ 44 - 47: header pages size
+ The header field is variable-sized. This field indicates
+ how many pages(4k) will be used to store add-cow header.
+ In add-cow v1, it is fixed to 1, so the header size will
+ be 4k * 1 = 4096 bytes.
+
+Image file name and backing file name must NOT be the same, we prevent this
+while creating add-cow files.
+
+Image file and backing file are interpreted relative to the qcow2 file, not
+to the current working directory of the process that opened the qcow2 file.
+
+== Reserved ==
+
+ Byte 48 - 63: backing file format
+ format of backing file. It will be filled with 0 if
+ backing file name offset is 0. If backing file name
+ offset is none-zero, it must be non-zero.
+
+ 64 - 79: image file format
+ format of image file. It must be non-zero.
+
+ 80 - [HEADER_SIZE - 1]:
+ It is used to make sure COW bitmap field starts at the
+ HEADER_SIZE byte, backing file name and image file name
+ will be stored here.
+
+== COW bitmap ==
+
+The "COW bitmap" field starts at the 4096th byte, stores a bitmap related to
+backing file and image file. The bitmap will track whether the sector in
+backing file is dirty or not.
+
+Each bit in the bitmap indicates one cluster's status. One cluster includes 128
+sectors, then each bit indicates 512 * 128 = 64k bytes. the size of bitmap is
+calculated according to virtual size of backing file. In each byte, bit 0 to 7
+will track the 1st to 7th cluster in sequence, bit orders in one byte look like:
+ +----+----+----+----+----+----+----+----+
+ | b7 | b6 | b5 | b4 | b3 | b2 | b1 | b0 |
+ +----+----+----+----+----+----+----+----+
+
+If the bit is 0, indicates the sector has not been allocated in image file, data
+should be loaded from backing file while reading; if the bit is 1, indicates the
+related sector has been dirty, should be loaded from image file while reading.
+Writing to a sector causes the corresponding bit to be set to 1.
+
+If raw image is not an even multiple of cluster bytes, bits that correspond to
+bytes beyond the raw file size in add-cow will be 0.
--
1.7.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH 2/6 v11 v11] block: make some functions public
2012-07-31 16:51 [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format Dong Xu Wang
@ 2012-07-31 16:51 ` Dong Xu Wang
2012-08-01 13:53 ` Eric Blake
2012-08-01 14:01 ` Stefan Hajnoczi
2012-07-31 16:51 ` [Qemu-devel] [PATCH 3/6] add-cow file format Dong Xu Wang
` (5 subsequent siblings)
6 siblings, 2 replies; 30+ messages in thread
From: Dong Xu Wang @ 2012-07-31 16:51 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, Dong Xu Wang
In add-cow file format, we will use path_has_protocol and we will read
a NUL-terminated string from image , qed_read_string has done the samething,
so make the two functions public, then we will reuse them directly.
While creating images files, if no size is specified, will use size of backing
file. If no backing file is specified, we will use the size of image file.
Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
---
block.c | 37 +++++++++++++++++++++++++++++++++----
block.h | 3 +++
block/qed.c | 29 +----------------------------
3 files changed, 37 insertions(+), 32 deletions(-)
diff --git a/block.c b/block.c
index b38940b..3b4d27f 100644
--- a/block.c
+++ b/block.c
@@ -196,7 +196,7 @@ static void bdrv_io_limits_intercept(BlockDriverState *bs,
}
/* check if the path starts with "<protocol>:" */
-static int path_has_protocol(const char *path)
+int path_has_protocol(const char *path)
{
const char *p;
@@ -213,6 +213,21 @@ static int path_has_protocol(const char *path)
return *p == ':';
}
+int bdrv_read_string(BlockDriverState *file, uint64_t offset, size_t n,
+ char *buf, size_t buflen)
+{
+ int ret;
+ if (n >= buflen) {
+ return -EINVAL;
+ }
+ ret = bdrv_pread(file, offset, buf, n);
+ if (ret < 0) {
+ return ret;
+ }
+ buf[n] = '\0';
+ return 0;
+}
+
int path_is_absolute(const char *path)
{
#ifdef _WIN32
@@ -3884,7 +3899,7 @@ int bdrv_img_create(const char *filename, const char *fmt,
char *options, uint64_t img_size, int flags)
{
QEMUOptionParameter *param = NULL, *create_options = NULL;
- QEMUOptionParameter *backing_fmt, *backing_file, *size;
+ QEMUOptionParameter *backing_fmt, *backing_file, *size, *image_file;
BlockDriverState *bs = NULL;
BlockDriver *drv, *proto_drv;
BlockDriver *backing_drv = NULL;
@@ -3965,9 +3980,11 @@ int bdrv_img_create(const char *filename, const char *fmt,
}
}
- // The size for the image must always be specified, with one exception:
- // If we are using a backing file, we can obtain the size from there
+ /* The size for the image must always be specified, with one exception:
+ If we are using a backing file, we can obtain the size from there,
+ but if not, and we are using an image file, we will obtain the size from it.*/
size = get_option_parameter(param, BLOCK_OPT_SIZE);
+ image_file = get_option_parameter(param, BLOCK_OPT_IMAGE_FILE);
if (size && size->value.n == -1) {
if (backing_file && backing_file->value.s) {
uint64_t size;
@@ -3990,6 +4007,18 @@ int bdrv_img_create(const char *filename, const char *fmt,
snprintf(buf, sizeof(buf), "%" PRId64, size);
set_option_parameter(param, BLOCK_OPT_SIZE, buf);
+ } else if (image_file && image_file->value.s) {
+ uint64_t size;
+ char buf[32];
+ bs = bdrv_new("");
+ ret = bdrv_file_open(&bs, image_file->value.s, BDRV_O_RDWR);
+ if (ret < 0) {
+ error_report("Could not open '%s'", image_file->value.s);
+ goto out;
+ }
+ size = bdrv_getlength(bs);
+ snprintf(buf, sizeof(buf), "%" PRId64, size);
+ set_option_parameter(param, BLOCK_OPT_SIZE, buf);
} else {
error_report("Image creation needs a size parameter");
ret = -EINVAL;
diff --git a/block.h b/block.h
index c89590d..b523076 100644
--- a/block.h
+++ b/block.h
@@ -152,6 +152,8 @@ int bdrv_pwrite(BlockDriverState *bs, int64_t offset,
const void *buf, int count);
int bdrv_pwrite_sync(BlockDriverState *bs, int64_t offset,
const void *buf, int count);
+int bdrv_read_string(BlockDriverState *file, uint64_t offset, size_t n,
+ char *buf, size_t buflen);
int coroutine_fn bdrv_co_readv(BlockDriverState *bs, int64_t sector_num,
int nb_sectors, QEMUIOVector *qiov);
int coroutine_fn bdrv_co_copy_on_readv(BlockDriverState *bs,
@@ -306,6 +308,7 @@ char *bdrv_snapshot_dump(char *buf, int buf_size, QEMUSnapshotInfo *sn);
char *get_human_readable_size(char *buf, int buf_size, int64_t size);
int path_is_absolute(const char *path);
+int path_has_protocol(const char *path);
void path_combine(char *dest, int dest_size,
const char *base_path,
const char *filename);
diff --git a/block/qed.c b/block/qed.c
index 5f3eefa..311c589 100644
--- a/block/qed.c
+++ b/block/qed.c
@@ -217,33 +217,6 @@ static bool qed_is_image_size_valid(uint64_t image_size, uint32_t cluster_size,
}
/**
- * Read a string of known length from the image file
- *
- * @file: Image file
- * @offset: File offset to start of string, in bytes
- * @n: String length in bytes
- * @buf: Destination buffer
- * @buflen: Destination buffer length in bytes
- * @ret: 0 on success, -errno on failure
- *
- * The string is NUL-terminated.
- */
-static int qed_read_string(BlockDriverState *file, uint64_t offset, size_t n,
- char *buf, size_t buflen)
-{
- int ret;
- if (n >= buflen) {
- return -EINVAL;
- }
- ret = bdrv_pread(file, offset, buf, n);
- if (ret < 0) {
- return ret;
- }
- buf[n] = '\0';
- return 0;
-}
-
-/**
* Allocate new clusters
*
* @s: QED state
@@ -437,7 +410,7 @@ static int bdrv_qed_open(BlockDriverState *bs, int flags)
return -EINVAL;
}
- ret = qed_read_string(bs->file, s->header.backing_filename_offset,
+ ret = bdrv_read_string(bs->file, s->header.backing_filename_offset,
s->header.backing_filename_size, bs->backing_file,
sizeof(bs->backing_file));
if (ret < 0) {
--
1.7.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH 3/6] add-cow file format
2012-07-31 16:51 [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format Dong Xu Wang
2012-07-31 16:51 ` [Qemu-devel] [PATCH 2/6 v11 v11] block: make some functions public Dong Xu Wang
@ 2012-07-31 16:51 ` Dong Xu Wang
2012-08-01 13:57 ` Eric Blake
2012-08-01 15:31 ` Stefan Hajnoczi
2012-07-31 16:51 ` [Qemu-devel] [PATCH 4/6 v11] add-cow: support snapshot_blkde Dong Xu Wang
` (4 subsequent siblings)
6 siblings, 2 replies; 30+ messages in thread
From: Dong Xu Wang @ 2012-07-31 16:51 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, Dong Xu Wang
This is the implementation code for add-cow file format. Because image_file
might be very huge, then we can't read entire bitmap into memory, we must use
a cache. Since qcow-cache.c has implemted cache code, we can create our cache
code based on it.
Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
---
block/Makefile.objs | 1 +
block/add-cow-cache.c | 206 +++++++++++++++++
block/add-cow.c | 599 +++++++++++++++++++++++++++++++++++++++++++++++++
block/add-cow.h | 101 +++++++++
block_int.h | 2 +
5 files changed, 909 insertions(+), 0 deletions(-)
create mode 100644 block/add-cow-cache.c
create mode 100644 block/add-cow.c
create mode 100644 block/add-cow.h
diff --git a/block/Makefile.objs b/block/Makefile.objs
index b5754d3..357a3b1 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -2,6 +2,7 @@ block-obj-y += raw.o cow.o qcow.o vdi.o vmdk.o cloop.o dmg.o bochs.o vpc.o vvfat
block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o
block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o
block-obj-y += qed-check.o
+block-obj-y += add-cow.o add-cow-cache.o
block-obj-y += parallels.o nbd.o blkdebug.o sheepdog.o blkverify.o
block-obj-y += stream.o
block-obj-$(CONFIG_WIN32) += raw-win32.o
diff --git a/block/add-cow-cache.c b/block/add-cow-cache.c
new file mode 100644
index 0000000..5d5e8a4
--- /dev/null
+++ b/block/add-cow-cache.c
@@ -0,0 +1,206 @@
+/*
+ * Cache For QEMU ADD-COW Disk Format
+ *
+ * Copyright IBM, Corp. 2012
+ *
+ * Authors:
+ * Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
+ *
+ * This file is based on qcow2-cache.c, see its copyrights below:
+ *
+ * L2/refcount table cache for the QCOW2 format
+ *
+ * Copyright (c) 2010 Kevin Wolf <kwolf@redhat.com>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include "block_int.h"
+#include "qemu-common.h"
+#include "add-cow.h"
+
+AddCowCache *add_cow_cache_create(BlockDriverState *bs, int num_tables)
+{
+ AddCowCache *c;
+ int i;
+
+ c = g_malloc0(sizeof(*c));
+ c->size = num_tables;
+ c->entries = g_malloc0(sizeof(*c->entries) * num_tables);
+ c->entry_size = ADD_COW_CACHE_ENTRY_SIZE;
+
+ for (i = 0; i < c->size; i++) {
+ c->entries[i].table = qemu_blockalign(bs, c->entry_size);
+ c->entries[i].offset = -1;
+ }
+
+ return c;
+}
+
+int add_cow_cache_destroy(BlockDriverState *bs, AddCowCache *c)
+{
+ int i;
+
+ for (i = 0; i < c->size; i++) {
+ qemu_vfree(c->entries[i].table);
+ }
+
+ g_free(c->entries);
+ g_free(c);
+
+ return 0;
+}
+
+static int add_cow_cache_entry_flush(BlockDriverState *bs,
+ AddCowCache *c,
+ int i)
+{
+ BDRVAddCowState *s = bs->opaque;
+ int ret = 0;
+
+ if (!c->entries[i].dirty || -1 == c->entries[i].offset) {
+ return ret;
+ }
+
+ ret = bdrv_pwrite(bs->file,
+ ADD_COW_PAGE_SIZE * s->header.header_pages_size + c->entries[i].offset,
+ c->entries[i].table,
+ MIN(c->entry_size, s->bitmap_size - c->entries[i].offset));
+ if (ret < 0) {
+ return ret;
+ }
+
+ c->entries[i].dirty = false;
+
+ return 0;
+}
+
+int add_cow_cache_flush(BlockDriverState *bs, AddCowCache *c)
+{
+ BDRVAddCowState *s = bs->opaque;
+ int result = 0;
+ int ret;
+ int i;
+
+ ret = bdrv_flush(s->image_hd);
+ if (ret < 0) {
+ return result;
+ }
+
+ for (i = 0; i < c->size; i++) {
+ ret = add_cow_cache_entry_flush(bs, c, i);
+ if (ret < 0 && result != -ENOSPC) {
+ result = ret;
+ }
+ }
+
+ if (result == 0) {
+ ret = bdrv_flush(bs->file);
+ if (ret < 0) {
+ result = ret;
+ }
+ }
+
+ return result;
+}
+
+static int add_cow_cache_find_entry_to_replace(AddCowCache *c)
+{
+ int i;
+ int min_count = INT_MAX;
+ int min_index = -1;
+
+
+ for (i = 0; i < c->size; i++) {
+ if (c->entries[i].cache_hits < min_count) {
+ min_index = i;
+ min_count = c->entries[i].cache_hits;
+ }
+
+ c->entries[i].cache_hits /= 2;
+ }
+
+ if (min_index == -1) {
+ abort();
+ }
+ return min_index;
+}
+
+static int add_cow_cache_do_get(BlockDriverState *bs, AddCowCache *c,
+ uint64_t offset, void **table)
+{
+ int i, ret;
+ BDRVAddCowState *s = bs->opaque;
+
+ for (i = 0; i < c->size; i++) {
+ if (c->entries[i].offset == offset) {
+ goto found;
+ }
+ }
+
+ i = add_cow_cache_find_entry_to_replace(c);
+ if (i < 0) {
+ return i;
+ }
+
+ ret = add_cow_cache_entry_flush(bs, c, i);
+ if (ret < 0) {
+ return ret;
+ }
+ c->entries[i].offset = -1;
+ ret = bdrv_pread(bs->file,
+ s->header.header_pages_size * ADD_COW_PAGE_SIZE + offset,
+ c->entries[i].table,
+ c->entry_size);
+ if (ret < 0) {
+ return ret;
+ }
+
+ c->entries[i].cache_hits = 32;
+ c->entries[i].offset = offset;
+
+found:
+ c->entries[i].cache_hits++;
+ *table = c->entries[i].table;
+
+ return 0;
+}
+
+int add_cow_cache_get(BlockDriverState *bs, AddCowCache *c, uint64_t sector_num,
+ void **table)
+{
+ /* each byte in bitmap indicates 8 * SECTORS_PER_CLUSTER clusters */
+ uint64_t offset = offset_in_bitmap(sector_num) & (~(c->entry_size - 1));
+ return add_cow_cache_do_get(bs, c, offset, table);
+}
+
+void add_cow_cache_entry_mark_dirty(AddCowCache *c, void *table)
+{
+ int i;
+
+ for (i = 0; i < c->size; i++) {
+ if (c->entries[i].table == table) {
+ goto found;
+ }
+ }
+ abort();
+
+found:
+ c->entries[i].dirty = true;
+}
diff --git a/block/add-cow.c b/block/add-cow.c
new file mode 100644
index 0000000..5f6b708
--- /dev/null
+++ b/block/add-cow.c
@@ -0,0 +1,599 @@
+/*
+ * QEMU ADD-COW Disk Format
+ *
+ * Copyright IBM, Corp. 2012
+ *
+ * Authors:
+ * Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "block_int.h"
+#include "module.h"
+#include "add-cow.h"
+
+static void add_cow_header_le_to_cpu(const AddCowHeader *le, AddCowHeader *cpu)
+{
+ cpu->magic = le64_to_cpu(le->magic);
+ cpu->version = le32_to_cpu(le->version);
+
+ cpu->backing_filename_offset = le32_to_cpu(le->backing_filename_offset);
+ cpu->backing_filename_size = le32_to_cpu(le->backing_filename_size);
+
+ cpu->image_filename_offset = le32_to_cpu(le->image_filename_offset);
+ cpu->image_filename_size = le32_to_cpu(le->image_filename_size);
+
+ cpu->features = le64_to_cpu(le->features);
+ cpu->optional_features = le64_to_cpu(le->optional_features);
+ cpu->header_pages_size = le32_to_cpu(le->header_pages_size);
+}
+
+static void add_cow_header_cpu_to_le(const AddCowHeader *cpu, AddCowHeader *le)
+{
+ le->magic = cpu_to_le64(cpu->magic);
+ le->version = cpu_to_le32(cpu->version);
+
+ le->backing_filename_offset = cpu_to_le32(cpu->backing_filename_offset);
+ le->backing_filename_size = cpu_to_le32(cpu->backing_filename_size);
+
+ le->image_filename_offset = cpu_to_le32(cpu->image_filename_offset);
+ le->image_filename_size = cpu_to_le32(cpu->image_filename_size);
+
+ le->features = cpu_to_le64(cpu->features);
+ le->optional_features = cpu_to_le64(cpu->optional_features);
+ le->header_pages_size = cpu_to_le32(cpu->header_pages_size);
+}
+
+static int add_cow_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+ const AddCowHeader *header = (const AddCowHeader *)buf;
+
+ if (le64_to_cpu(header->magic) == ADD_COW_MAGIC &&
+ le32_to_cpu(header->version) == ADD_COW_VERSION) {
+ return 100;
+ } else {
+ return 0;
+ }
+}
+
+static int add_cow_create(const char *filename, QEMUOptionParameter *options)
+{
+ AddCowHeader header = {
+ .magic = ADD_COW_MAGIC,
+ .version = ADD_COW_VERSION,
+ .features = 0,
+ .optional_features = 0,
+ .header_pages_size = ADD_COW_DEFAULT_PAGE_SIZE,
+ };
+ AddCowHeader le_header;
+ int64_t backing_len = 0;
+ const char *backing_filename = NULL;
+ const char *backing_fmt = NULL;
+ const char *image_filename = NULL;
+ const char *image_format = NULL;
+ BlockDriverState *bs, *image_bs = NULL, *backing_bs = NULL;
+ BlockDriver *drv = bdrv_find_format("add-cow");
+ int ret;
+
+ while (options && options->name) {
+ if (!strcmp(options->name, BLOCK_OPT_SIZE)) {
+ backing_len = options->value.n;
+ } else if (!strcmp(options->name, BLOCK_OPT_BACKING_FILE)) {
+ backing_filename = options->value.s;
+ } else if (!strcmp(options->name, BLOCK_OPT_BACKING_FMT)) {
+ backing_fmt = options->value.s;
+ } else if (!strcmp(options->name, BLOCK_OPT_IMAGE_FILE)) {
+ image_filename = options->value.s;
+ } else if (!strcmp(options->name, BLOCK_OPT_IMAGE_FORMAT)) {
+ image_format = options->value.s;
+ }
+ options++;
+ }
+
+ if (backing_filename) {
+ header.features |= ADD_COW_F_BACKING_FILE;
+ header.backing_filename_offset = sizeof(header) + 16 * 2;
+ header.backing_filename_size = strlen(backing_filename);
+
+ if (!backing_fmt) {
+ backing_bs = bdrv_new("image");
+ ret = bdrv_open(backing_bs, backing_filename, BDRV_O_RDWR
+ | BDRV_O_CACHE_WB, NULL);
+ if (ret < 0) {
+ return ret;
+ }
+ backing_fmt = bdrv_get_format_name(backing_bs);
+ bdrv_delete(backing_bs);
+ }
+ } else {
+ header.features |= ADD_COW_F_All_ALLOCATED;
+ }
+
+ if (image_filename) {
+ header.features |= ADD_COW_F_IMAGE_FILE;
+ header.image_filename_offset =
+ sizeof(header) + 16 * 2 + header.backing_filename_size;
+ header.image_filename_size = strlen(image_filename);
+ } else {
+ error_report("image_file should be given.");
+ return -EINVAL;
+ }
+
+ if (backing_filename && !strcmp(backing_filename, image_filename)) {
+ error_report("Error: Trying to create an image with the "
+ "same backing file name as the image file name");
+ return -EINVAL;
+ }
+
+ if (!strcmp(filename, image_filename)) {
+ error_report("Error: Trying to create an image with the "
+ "same filename as the image file name");
+ return -EINVAL;
+ }
+
+ if (header.image_filename_offset + header.image_filename_size
+ > ADD_COW_PAGE_SIZE * ADD_COW_DEFAULT_PAGE_SIZE) {
+ error_report("image_file name or backing_file name too long.");
+ return -ENOSPC;
+ }
+
+ ret = bdrv_file_open(&image_bs, image_filename, BDRV_O_RDWR);
+ if (ret < 0) {
+ return ret;
+ }
+ bdrv_delete(image_bs);
+
+ ret = bdrv_create_file(filename, NULL);
+ if (ret < 0) {
+ return ret;
+ }
+
+ ret = bdrv_file_open(&bs, filename, BDRV_O_RDWR);
+ if (ret < 0) {
+ return ret;
+ }
+ add_cow_header_cpu_to_le(&header, &le_header);
+ ret = bdrv_pwrite(bs, 0, &le_header, sizeof(le_header));
+ if (ret < 0) {
+ bdrv_delete(bs);
+ return ret;
+ }
+
+ ret = bdrv_pwrite(bs, sizeof(le_header), backing_fmt ? backing_fmt : "",
+ backing_fmt ? sizeof(backing_fmt) : 0);
+ if (ret < 0) {
+ bdrv_delete(bs);
+ return ret;
+ }
+
+ ret = bdrv_pwrite(bs, sizeof(le_header) + 16,
+ image_format ? image_format : "raw",
+ image_format ? sizeof(image_format) : sizeof("raw"));
+ if (ret < 0) {
+ bdrv_delete(bs);
+ return ret;
+ }
+
+ if (backing_filename) {
+ ret = bdrv_pwrite(bs,
+ header.backing_filename_offset,
+ backing_filename,
+ header.backing_filename_size);
+ if (ret < 0) {
+ bdrv_delete(bs);
+ return ret;
+ }
+ }
+
+ ret = bdrv_pwrite(bs,
+ header.image_filename_offset,
+ image_filename,
+ header.image_filename_size);
+ if (ret < 0) {
+ bdrv_delete(bs);
+ return ret;
+ }
+
+ ret = bdrv_open(bs, filename, BDRV_O_RDWR | BDRV_O_NO_FLUSH, drv);
+ if (ret < 0) {
+ bdrv_delete(bs);
+ return ret;
+ }
+
+ ret = bdrv_truncate(bs, backing_len);
+ bdrv_delete(bs);
+ return ret;
+}
+
+static int add_cow_open(BlockDriverState *bs, int flags)
+{
+ char image_filename[ADD_COW_FILE_LEN];
+ char tmp_name[ADD_COW_FILE_LEN];
+ BlockDriver *image_drv = NULL;
+ BlockDriverState *backing_bs = NULL;
+ int ret;
+ int sector_per_byte;
+ BDRVAddCowState *s = bs->opaque;
+ AddCowHeader le_header;
+
+ ret = bdrv_pread(bs->file, 0, &le_header, sizeof(le_header));
+ if (ret != sizeof(s->header)) {
+ goto fail;
+ }
+
+ add_cow_header_le_to_cpu(&le_header, &s->header);
+
+ if (le64_to_cpu(s->header.magic) != ADD_COW_MAGIC) {
+ ret = -EINVAL;
+ goto fail;
+ }
+
+ if (s->header.version != ADD_COW_VERSION) {
+ char version[64];
+ snprintf(version, sizeof(version), "ADD-COW version %d",
+ s->header.version);
+ qerror_report(QERR_UNKNOWN_BLOCK_FORMAT_FEATURE,
+ bs->device_name, "add-cow", version);
+ ret = -ENOTSUP;
+ goto fail;
+ }
+
+ if (s->header.features & ~ADD_COW_FEATURE_MASK) {
+ char buf[64];
+ snprintf(buf, sizeof(buf), "%" PRIx64,
+ s->header.features & ~ADD_COW_FEATURE_MASK);
+ qerror_report(QERR_UNKNOWN_BLOCK_FORMAT_FEATURE,
+ bs->device_name, "add-cow", buf);
+ return -ENOTSUP;
+ }
+ if ((s->header.features & ADD_COW_F_IMAGE_FILE) == 0) {
+ ret = -ENOTSUP;
+ goto fail;
+ }
+
+ if ((s->header.features & ADD_COW_F_All_ALLOCATED) == 0) {
+ ret = bdrv_read_string(bs->file, sizeof(s->header),
+ 15, s->backing_file_format,
+ sizeof(s->backing_file_format));
+ if (ret < 0) {
+ goto fail;
+ }
+ }
+
+ ret = bdrv_read_string(bs->file, sizeof(s->header) + 16,
+ 15, s->image_file_format,
+ sizeof(s->image_file_format));
+ if (ret < 0) {
+ goto fail;
+ }
+
+ if ((s->header.features & ADD_COW_F_All_ALLOCATED) == 0) {
+ ret = bdrv_read_string(bs->file, s->header.backing_filename_offset,
+ s->header.backing_filename_size, bs->backing_file,
+ sizeof(bs->backing_file));
+ if (ret < 0) {
+ goto fail;
+ }
+ }
+
+ ret = bdrv_read_string(bs->file, s->header.image_filename_offset,
+ s->header.image_filename_size, tmp_name,
+ sizeof(tmp_name));
+ if (ret < 0) {
+ goto fail;
+ }
+
+ s->image_hd = bdrv_new("");
+ if (path_has_protocol(image_filename)) {
+ pstrcpy(image_filename, sizeof(image_filename), tmp_name);
+ } else {
+ path_combine(image_filename, sizeof(image_filename),
+ bs->filename, tmp_name);
+ }
+
+ ret = bdrv_open(s->image_hd, image_filename, flags, image_drv);
+ if (ret < 0) {
+ bdrv_delete(s->image_hd);
+ goto fail;
+ }
+
+ if (s->header.features & ADD_COW_F_All_ALLOCATED) {
+ bs->total_sectors = bdrv_getlength(s->image_hd) >> 9;
+ } else {
+ backing_bs = bdrv_new("image");
+ ret = bdrv_open(backing_bs, bs->backing_file, BDRV_O_RDWR
+ | BDRV_O_CACHE_WB, NULL);
+ if (ret < 0) {
+ return ret;
+ }
+ bs->total_sectors = bdrv_getlength(backing_bs) >> 9;
+ bdrv_delete(backing_bs);
+ }
+
+ s->cluster_size = ADD_COW_CLUSTER_SIZE;
+ sector_per_byte = SECTORS_PER_CLUSTER * 8;
+ s->bitmap_size =
+ (bs->total_sectors + sector_per_byte - 1) / sector_per_byte;
+ s->bitmap_cache =
+ add_cow_cache_create(bs, ADD_COW_CACHE_SIZE);
+
+ qemu_co_mutex_init(&s->lock);
+ return 0;
+fail:
+ if (s->bitmap_cache) {
+ add_cow_cache_destroy(bs, s->bitmap_cache);
+ }
+ return ret;
+}
+
+static void add_cow_close(BlockDriverState *bs)
+{
+ BDRVAddCowState *s = bs->opaque;
+ add_cow_cache_destroy(bs, s->bitmap_cache);
+ bdrv_delete(s->image_hd);
+}
+
+static bool is_allocated(BlockDriverState *bs, int64_t sector_num)
+{
+ BDRVAddCowState *s = bs->opaque;
+ int64_t cluster_num = sector_num / SECTORS_PER_CLUSTER;
+ uint8_t *table = NULL;
+ int ret = add_cow_cache_get(bs, s->bitmap_cache,
+ sector_num, (void **)&table);
+
+ if (ret < 0) {
+ return ret;
+ }
+ return table[cluster_num / 8 % ADD_COW_CACHE_ENTRY_SIZE]
+ & (1 << (cluster_num % 8));
+}
+
+static coroutine_fn int add_cow_is_allocated(BlockDriverState *bs,
+ int64_t sector_num, int nb_sectors, int *num_same)
+{
+ BDRVAddCowState *s = bs->opaque;
+ int changed;
+
+ if (nb_sectors == 0) {
+ *num_same = 0;
+ return 0;
+ }
+
+ if (s->header.features & ADD_COW_F_All_ALLOCATED) {
+ *num_same = nb_sectors - 1;
+ return 1;
+ }
+ changed = is_allocated(bs, sector_num);
+
+ for (*num_same = 1; *num_same < nb_sectors; (*num_same)++) {
+ if (is_allocated(bs, sector_num + *num_same) != changed) {
+ break;
+ }
+ }
+ return changed;
+}
+
+static coroutine_fn int add_cow_co_readv(BlockDriverState *bs,
+ int64_t sector_num, int remaining_sectors, QEMUIOVector *qiov)
+{
+ BDRVAddCowState *s = bs->opaque;
+ int cur_nr_sectors;
+ uint64_t bytes_done = 0;
+ QEMUIOVector hd_qiov;
+ int n, ret = 0;
+
+ qemu_iovec_init(&hd_qiov, qiov->niov);
+ qemu_co_mutex_lock(&s->lock);
+ while (remaining_sectors != 0) {
+ cur_nr_sectors = remaining_sectors;
+ if (add_cow_is_allocated(bs, sector_num, cur_nr_sectors, &n)) {
+ cur_nr_sectors = n;
+ qemu_iovec_reset(&hd_qiov);
+ qemu_iovec_concat(&hd_qiov, qiov, bytes_done,
+ cur_nr_sectors * BDRV_SECTOR_SIZE);
+ qemu_co_mutex_unlock(&s->lock);
+ ret = bdrv_co_readv(s->image_hd, sector_num, n, &hd_qiov);
+ qemu_co_mutex_lock(&s->lock);
+ if (ret < 0) {
+ goto fail;
+ }
+ } else {
+ cur_nr_sectors = n;
+ if (bs->backing_hd) {
+ qemu_iovec_reset(&hd_qiov);
+ qemu_iovec_concat(&hd_qiov, qiov, bytes_done,
+ cur_nr_sectors * BDRV_SECTOR_SIZE);
+ qemu_co_mutex_unlock(&s->lock);
+ ret = bdrv_co_readv(bs->backing_hd, sector_num,
+ n, &hd_qiov);
+ qemu_co_mutex_lock(&s->lock);
+ if (ret < 0) {
+ goto fail;
+ }
+ } else {
+ qemu_iovec_memset(&hd_qiov, 0, 0,
+ BDRV_SECTOR_SIZE * cur_nr_sectors);
+ }
+ }
+ remaining_sectors -= cur_nr_sectors;
+ sector_num += cur_nr_sectors;
+ bytes_done += cur_nr_sectors * BDRV_SECTOR_SIZE;
+ }
+fail:
+ qemu_co_mutex_unlock(&s->lock);
+ qemu_iovec_destroy(&hd_qiov);
+ return ret;
+}
+
+static int coroutine_fn copy_sectors(BlockDriverState *bs,
+ int n_start, int n_end)
+{
+ BDRVAddCowState *s = bs->opaque;
+ QEMUIOVector qiov;
+ struct iovec iov;
+ int n, ret;
+
+ n = n_end - n_start;
+ if (n <= 0) {
+ return 0;
+ }
+
+ iov.iov_len = n * BDRV_SECTOR_SIZE;
+ iov.iov_base = qemu_blockalign(bs, iov.iov_len);
+
+ qemu_iovec_init_external(&qiov, &iov, 1);
+
+ ret = bdrv_co_readv(bs->backing_hd, n_start, n, &qiov);
+ if (ret < 0) {
+ goto out;
+ }
+ ret = bdrv_co_writev(s->image_hd, n_start, n, &qiov);
+ if (ret < 0) {
+ goto out;
+ }
+
+ ret = 0;
+out:
+ qemu_vfree(iov.iov_base);
+ return ret;
+}
+
+static coroutine_fn int add_cow_co_writev(BlockDriverState *bs,
+ int64_t sector_num, int remaining_sectors, QEMUIOVector *qiov)
+{
+ BDRVAddCowState *s = bs->opaque;
+ int ret = 0, i;
+ QEMUIOVector hd_qiov;
+ uint8_t *table;
+
+ qemu_co_mutex_lock(&s->lock);
+ qemu_iovec_init(&hd_qiov, qiov->niov);
+ ret = bdrv_co_writev(s->image_hd,
+ sector_num,
+ remaining_sectors, qiov);
+
+ if (ret < 0) {
+ goto fail;
+ }
+ if ((s->header.features & ADD_COW_F_All_ALLOCATED) == 0) {
+ /* Copy content of unmodified sectors */
+ if (!is_cluster_head(sector_num) && !is_allocated(bs, sector_num)) {
+ ret = copy_sectors(bs, sector_num & ~(SECTORS_PER_CLUSTER - 1),
+ sector_num);
+ if (ret < 0) {
+ goto fail;
+ }
+ }
+
+ if (!is_cluster_tail(sector_num + remaining_sectors - 1)
+ && !is_allocated(bs, sector_num + remaining_sectors - 1)) {
+ ret = copy_sectors(bs, sector_num + remaining_sectors,
+ ((sector_num + remaining_sectors) | (SECTORS_PER_CLUSTER - 1)) + 1);
+ if (ret < 0) {
+ goto fail;
+ }
+ }
+
+ for (i = sector_num / SECTORS_PER_CLUSTER;
+ i <= (sector_num + remaining_sectors - 1) / SECTORS_PER_CLUSTER;
+ i++) {
+ ret = add_cow_cache_get(bs, s->bitmap_cache,
+ i * SECTORS_PER_CLUSTER, (void **)&table);
+ if (ret < 0) {
+ goto fail;
+ }
+ if ((table[i / 8] & (1 << (i % 8))) == 0) {
+ table[i / 8] |= (1 << (i % 8));
+ add_cow_cache_entry_mark_dirty(s->bitmap_cache, table);
+ }
+ }
+ }
+ ret = 0;
+fail:
+ qemu_co_mutex_unlock(&s->lock);
+ qemu_iovec_destroy(&hd_qiov);
+ return ret;
+}
+
+static int bdrv_add_cow_truncate(BlockDriverState *bs, int64_t size)
+{
+ BDRVAddCowState *s = bs->opaque;
+ int sector_per_byte = SECTORS_PER_CLUSTER * 8;
+ int ret;
+ uint32_t bitmap_pos = s->header.header_pages_size * ADD_COW_PAGE_SIZE;
+ int64_t bitmap_size =
+ (size / BDRV_SECTOR_SIZE + sector_per_byte - 1) / sector_per_byte;
+
+ ret = bdrv_truncate(bs->file, bitmap_pos + bitmap_size);
+ if (ret < 0) {
+ return ret;
+ }
+ return 0;
+}
+
+static coroutine_fn int add_cow_co_flush(BlockDriverState *bs)
+{
+ BDRVAddCowState *s = bs->opaque;
+ int ret;
+
+ qemu_co_mutex_lock(&s->lock);
+ ret = add_cow_cache_flush(bs, s->bitmap_cache);
+ qemu_co_mutex_unlock(&s->lock);
+ return ret;
+}
+
+static QEMUOptionParameter add_cow_create_options[] = {
+ {
+ .name = BLOCK_OPT_SIZE,
+ .type = OPT_SIZE,
+ .help = "Virtual disk size"
+ },
+ {
+ .name = BLOCK_OPT_BACKING_FILE,
+ .type = OPT_STRING,
+ .help = "File name of a base image"
+ },
+ {
+ .name = BLOCK_OPT_BACKING_FMT,
+ .type = OPT_STRING,
+ .help = "Image format of the base image"
+ },
+ {
+ .name = BLOCK_OPT_IMAGE_FILE,
+ .type = OPT_STRING,
+ .help = "File name of a image file"
+ },
+ {
+ .name = BLOCK_OPT_IMAGE_FORMAT,
+ .type = OPT_STRING,
+ .help = "Image format of the image file"
+ },
+ { NULL }
+};
+
+static BlockDriver bdrv_add_cow = {
+ .format_name = "add-cow",
+ .instance_size = sizeof(BDRVAddCowState),
+ .bdrv_probe = add_cow_probe,
+ .bdrv_open = add_cow_open,
+ .bdrv_close = add_cow_close,
+ .bdrv_create = add_cow_create,
+ .bdrv_co_readv = add_cow_co_readv,
+ .bdrv_co_writev = add_cow_co_writev,
+ .bdrv_truncate = bdrv_add_cow_truncate,
+ .bdrv_co_is_allocated = add_cow_is_allocated,
+
+ .create_options = add_cow_create_options,
+ .bdrv_co_flush_to_os = add_cow_co_flush,
+};
+
+static void bdrv_add_cow_init(void)
+{
+ bdrv_register(&bdrv_add_cow);
+}
+
+block_init(bdrv_add_cow_init);
diff --git a/block/add-cow.h b/block/add-cow.h
new file mode 100644
index 0000000..738249c
--- /dev/null
+++ b/block/add-cow.h
@@ -0,0 +1,101 @@
+/*
+ * QEMU ADD-COW Disk Format
+ *
+ * Copyright IBM, Corp. 2012
+ *
+ * Authors:
+ * Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#ifndef BLOCK_ADD_COW_H
+#define BLOCK_ADD_COW_H
+
+enum {
+ ADD_COW_F_BACKING_FILE = 0x01,
+ ADD_COW_F_IMAGE_FILE = 0x02,
+ ADD_COW_F_All_ALLOCATED = 0X04,
+ ADD_COW_FEATURE_MASK = ADD_COW_F_BACKING_FILE | \
+ ADD_COW_F_IMAGE_FILE | \
+ ADD_COW_F_All_ALLOCATED,
+
+ ADD_COW_MAGIC = (((uint64_t)'A' << 56) | ((uint64_t)'D' << 48) | \
+ ((uint64_t)'D' << 40) | ((uint64_t)'_' << 32) | \
+ ((uint64_t)'C' << 24) | ((uint64_t)'O' << 16) | \
+ ((uint64_t)'W' << 8) | 0xFF),
+ ADD_COW_VERSION = 1,
+ ADD_COW_FILE_LEN = 1024,
+ ADD_COW_CACHE_SIZE = 16,
+ ADD_COW_CACHE_ENTRY_SIZE = 65536,
+ ADD_COW_CLUSTER_SIZE = 65536,
+ SECTORS_PER_CLUSTER = (ADD_COW_CLUSTER_SIZE / BDRV_SECTOR_SIZE),
+ ADD_COW_PAGE_SIZE = 4096,
+ ADD_COW_DEFAULT_PAGE_SIZE = 1
+};
+
+typedef struct AddCowHeader {
+ uint64_t magic;
+ uint32_t version;
+
+ uint32_t backing_filename_offset;
+ uint32_t backing_filename_size;
+
+ uint32_t image_filename_offset;
+ uint32_t image_filename_size;
+
+ uint64_t features;
+ uint64_t optional_features;
+ uint32_t header_pages_size;
+} QEMU_PACKED AddCowHeader;
+
+typedef struct AddCowCachedTable {
+ void *table;
+ int64_t offset;
+ bool dirty;
+ int cache_hits;
+} AddCowCachedTable;
+
+typedef struct AddCowCache {
+ AddCowCachedTable *entries;
+ int entry_size;
+ int size;
+} AddCowCache;
+
+typedef struct BDRVAddCowState {
+ BlockDriverState *image_hd;
+ CoMutex lock;
+ int cluster_size;
+ AddCowCache *bitmap_cache;
+ uint64_t bitmap_size;
+ AddCowHeader header;
+ char backing_file_format[16];
+ char image_file_format[16];
+} BDRVAddCowState;
+
+/* Convert sector_num to offset in bitmap */
+static inline int64_t offset_in_bitmap(int64_t sector_num)
+{
+ int64_t cluster_num = sector_num / SECTORS_PER_CLUSTER;
+ return cluster_num / 8;
+}
+
+static inline bool is_cluster_head(int64_t sector_num)
+{
+ return sector_num % SECTORS_PER_CLUSTER == 0;
+}
+
+static inline bool is_cluster_tail(int64_t sector_num)
+{
+ return (sector_num + 1) % SECTORS_PER_CLUSTER == 0;
+}
+
+AddCowCache *add_cow_cache_create(BlockDriverState *bs, int num_tables);
+int add_cow_cache_destroy(BlockDriverState *bs, AddCowCache *c);
+void add_cow_cache_entry_mark_dirty(AddCowCache *c, void *table);
+int add_cow_cache_get(BlockDriverState *bs, AddCowCache *c, uint64_t offset,
+ void **table);
+int add_cow_cache_flush(BlockDriverState *bs, AddCowCache *c);
+#endif
diff --git a/block_int.h b/block_int.h
index d72317f..8f8bb2b 100644
--- a/block_int.h
+++ b/block_int.h
@@ -51,6 +51,8 @@
#define BLOCK_OPT_PREALLOC "preallocation"
#define BLOCK_OPT_SUBFMT "subformat"
#define BLOCK_OPT_COMPAT_LEVEL "compat"
+#define BLOCK_OPT_IMAGE_FILE "image_file"
+#define BLOCK_OPT_IMAGE_FORMAT "image_format"
typedef struct BdrvTrackedRequest BdrvTrackedRequest;
--
1.7.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH 4/6 v11] add-cow: support snapshot_blkde
2012-07-31 16:51 [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format Dong Xu Wang
2012-07-31 16:51 ` [Qemu-devel] [PATCH 2/6 v11 v11] block: make some functions public Dong Xu Wang
2012-07-31 16:51 ` [Qemu-devel] [PATCH 3/6] add-cow file format Dong Xu Wang
@ 2012-07-31 16:51 ` Dong Xu Wang
2012-08-01 15:37 ` Stefan Hajnoczi
2012-07-31 16:51 ` [Qemu-devel] [PATCH 5/6 v11] add-cow: hmp and qmp interface Dong Xu Wang
` (3 subsequent siblings)
6 siblings, 1 reply; 30+ messages in thread
From: Dong Xu Wang @ 2012-07-31 16:51 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, Dong Xu Wang
add-cow will let raw file support snapshot_blkdev indirectly.
Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
---
blockdev.c | 45 +++++++++++++++++++++++++++++++++++++--------
docs/live-block-ops.txt | 11 ++++++++++-
2 files changed, 47 insertions(+), 9 deletions(-)
diff --git a/blockdev.c b/blockdev.c
index 3d75015..a1e9268 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -19,7 +19,7 @@
#include "qmp-commands.h"
#include "trace.h"
#include "arch_init.h"
-
+#include "block/add-cow.h"
static QTAILQ_HEAD(drivelist, DriveInfo) drives = QTAILQ_HEAD_INITIALIZER(drives);
static const char *const if_name[IF_COUNT] = {
@@ -665,6 +665,8 @@ static void blockdev_do_action(int kind, void *data, Error **errp)
void qmp_blockdev_snapshot_sync(const char *device, const char *snapshot_file,
bool has_format, const char *format,
bool has_mode, enum NewImageMode mode,
+ bool has_image_file, const char *image_filename,
+ bool has_image_format, const char *image_format,
Error **errp)
{
BlockdevSnapshot snapshot = {
@@ -674,6 +676,10 @@ void qmp_blockdev_snapshot_sync(const char *device, const char *snapshot_file,
.format = (char *) format,
.has_mode = has_mode,
.mode = mode,
+ .has_image_file = has_image_file,
+ .image_file = (char *) image_filename,
+ .has_image_format = has_image_format,
+ .image_format = (char *) image_format,
};
blockdev_do_action(BLOCKDEV_ACTION_KIND_BLOCKDEV_SNAPSHOT_SYNC, &snapshot,
errp);
@@ -776,15 +782,30 @@ void qmp_transaction(BlockdevActionList *dev_list, Error **errp)
/* create new image w/backing file */
if (mode != NEW_IMAGE_MODE_EXISTING) {
- ret = bdrv_img_create(new_image_file, format,
- states->old_bs->filename,
- states->old_bs->drv->format_name,
- NULL, -1, flags);
- if (ret) {
- error_set(errp, QERR_OPEN_FILE_FAILED, new_image_file);
- goto delete_and_fail;
+ char option[1024];
+ uint64_t size;
+
+ bdrv_get_geometry(states->old_bs, &size);
+ size *= BDRV_SECTOR_SIZE;
+ if (dev_info->blockdev_snapshot_sync->image_file) {
+ sprintf(option, "image_file=%s,image_format=%s",
+ dev_info->blockdev_snapshot_sync->image_file,
+ dev_info->blockdev_snapshot_sync->has_image_format ? dev_info->blockdev_snapshot_sync->image_format : "raw");
+ ret = bdrv_img_create(new_image_file, format,
+ states->old_bs->filename,
+ states->old_bs->drv->format_name,
+ option, -1, flags);
+ } else {
+ ret = bdrv_img_create(new_image_file, format,
+ states->old_bs->filename,
+ states->old_bs->drv->format_name,
+ NULL, -1, flags);
}
}
+ if (ret) {
+ error_set(errp, QERR_OPEN_FILE_FAILED, new_image_file);
+ goto delete_and_fail;
+ }
/* We will manually add the backing_hd field to the bs later */
states->new_bs = bdrv_new("");
@@ -1083,6 +1104,14 @@ static void block_stream_cb(void *opaque, int ret)
}
qobject_decref(obj);
+ if (strcmp(bs->drv->format_name, "add-cow") == 0) {
+ BDRVAddCowState *s = bs->opaque;
+ char *format = s->image_file_format;
+ BlockDriver *drv = bdrv_find_format(format);
+ assert(drv);
+ bdrv_drain_all();
+ bdrv_swap(s->image_hd, bs);
+ }
drive_put_ref_bh_schedule(drive_get_by_blockdev(bs));
}
diff --git a/docs/live-block-ops.txt b/docs/live-block-ops.txt
index a257087..a53bab3 100644
--- a/docs/live-block-ops.txt
+++ b/docs/live-block-ops.txt
@@ -2,7 +2,8 @@ LIVE BLOCK OPERATIONS
=====================
High level description of live block operations. Note these are not
-supported for use with the raw format at the moment.
+supported for use with the raw format at the moment, but we can use
+add-cow as metadata to suport raw format.
Snapshot live merge
===================
@@ -56,3 +57,11 @@ into that image. Example:
(qemu) block_stream ide0-hd0
+
+Raw is not supported, but we can use add-cow in the 1st step:
+
+(qemu) snapshot_blkdev ide0-hd0 /new-path/disk.img add-cow source.img raw
+
+source.img is a raw file, and should be pre-created. disk.img will be created
+using soure.img as image file. After block_stream process finished, it will
+reassociate the drive with source.img automatically.
--
1.7.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH 5/6 v11] add-cow: hmp and qmp interface
2012-07-31 16:51 [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format Dong Xu Wang
` (2 preceding siblings ...)
2012-07-31 16:51 ` [Qemu-devel] [PATCH 4/6 v11] add-cow: support snapshot_blkde Dong Xu Wang
@ 2012-07-31 16:51 ` Dong Xu Wang
2012-07-31 16:51 ` [Qemu-devel] [PATCH 6/6 v11] add-cow: support qemu-iotests Dong Xu Wang
` (2 subsequent siblings)
6 siblings, 0 replies; 30+ messages in thread
From: Dong Xu Wang @ 2012-07-31 16:51 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, Dong Xu Wang
add HMP and QMP support for add-cow format. Because add-cow does not work
like other image formats, it will use image-file, so we need add add-cow
support while doing snapshot.
Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
---
hmp-commands.hx | 8 ++++++--
hmp.c | 5 ++++-
qapi-schema.json | 5 +++--
qmp-commands.hx | 2 +-
4 files changed, 14 insertions(+), 6 deletions(-)
diff --git a/hmp-commands.hx b/hmp-commands.hx
index eea8b32..bd14690 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -908,14 +908,18 @@ ETEXI
{
.name = "snapshot_blkdev",
- .args_type = "reuse:-n,device:B,snapshot-file:s?,format:s?",
- .params = "[-n] device [new-image-file] [format]",
+ .args_type = "reuse:-n,device:B,snapshot-file:s?,format:s?,image-file:s?,image-format:s?",
+ .params = "[-n] device [new-image-file] [format] [image-file] [image-format]",
.help = "initiates a live snapshot\n\t\t\t"
"of device. If a new image file is specified, the\n\t\t\t"
"new image file will become the new root image.\n\t\t\t"
"If format is specified, the snapshot file will\n\t\t\t"
"be created in that format. Otherwise the\n\t\t\t"
"snapshot will be internal! (currently unsupported).\n\t\t\t"
+ "If image file is specified, the snapshot file will\n\t\t\t"
+ "be created using image_file option.\n\t\t\t"
+ "If image-format is specified, the image file will\n\t\t\t"
+ "be created in that format. Otherwise will use raw format.\n\t\t\t"
"The default format is qcow2. The -n flag requests QEMU\n\t\t\t"
"to reuse the image found in new-image-file, instead of\n\t\t\t"
"recreating it from scratch.",
diff --git a/hmp.c b/hmp.c
index 6b72a64..4c7026c 100644
--- a/hmp.c
+++ b/hmp.c
@@ -701,6 +701,8 @@ void hmp_snapshot_blkdev(Monitor *mon, const QDict *qdict)
const char *filename = qdict_get_try_str(qdict, "snapshot-file");
const char *format = qdict_get_try_str(qdict, "format");
int reuse = qdict_get_try_bool(qdict, "reuse", 0);
+ const char *image_filename = qdict_get_try_str(qdict, "image-file");
+ const char *image_format = qdict_get_try_str(qdict, "image-format");
enum NewImageMode mode;
Error *errp = NULL;
@@ -714,7 +716,8 @@ void hmp_snapshot_blkdev(Monitor *mon, const QDict *qdict)
mode = reuse ? NEW_IMAGE_MODE_EXISTING : NEW_IMAGE_MODE_ABSOLUTE_PATHS;
qmp_blockdev_snapshot_sync(device, filename, !!format, format,
- true, mode, &errp);
+ true, mode, !!image_filename, image_filename,
+ !!image_format, image_format, &errp);
hmp_handle_error(mon, &errp);
}
diff --git a/qapi-schema.json b/qapi-schema.json
index bc55ed2..c90fe0d 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -1204,7 +1204,7 @@
##
{ 'type': 'BlockdevSnapshot',
'data': { 'device': 'str', 'snapshot-file': 'str', '*format': 'str',
- '*mode': 'NewImageMode' } }
+ '*mode': 'NewImageMode', '*image_file': 'str', '*image_format': 'str' } }
##
# @BlockdevAction
@@ -1269,7 +1269,8 @@
##
{ 'command': 'blockdev-snapshot-sync',
'data': { 'device': 'str', 'snapshot-file': 'str', '*format': 'str',
- '*mode': 'NewImageMode'} }
+ '*mode': 'NewImageMode', '*image-file': 'str',
+ '*image-format': 'str'} }
##
# @human-monitor-command:
diff --git a/qmp-commands.hx b/qmp-commands.hx
index e3cf3c5..fc776a5 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -789,7 +789,7 @@ EQMP
{
.name = "blockdev-snapshot-sync",
- .args_type = "device:B,snapshot-file:s,format:s?,mode:s?",
+ .args_type = "device:B,snapshot-file:s,format:s?,mode:s?,image-file:s?,image-format:s?",
.mhandler.cmd_new = qmp_marshal_input_blockdev_snapshot_sync,
},
--
1.7.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH 6/6 v11] add-cow: support qemu-iotests
2012-07-31 16:51 [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format Dong Xu Wang
` (3 preceding siblings ...)
2012-07-31 16:51 ` [Qemu-devel] [PATCH 5/6 v11] add-cow: hmp and qmp interface Dong Xu Wang
@ 2012-07-31 16:51 ` Dong Xu Wang
2012-08-01 13:51 ` [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format Eric Blake
2012-08-01 13:55 ` Stefan Hajnoczi
6 siblings, 0 replies; 30+ messages in thread
From: Dong Xu Wang @ 2012-07-31 16:51 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, Dong Xu Wang
add qemu-iotests support for add-cow file format.
Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
---
tests/qemu-iotests/017 | 2 +-
tests/qemu-iotests/020 | 2 +-
tests/qemu-iotests/check | 4 ++--
tests/qemu-iotests/common | 6 ++++++
tests/qemu-iotests/common.rc | 19 +++++++++++++++++++
5 files changed, 29 insertions(+), 4 deletions(-)
diff --git a/tests/qemu-iotests/017 b/tests/qemu-iotests/017
index 66951eb..d31432f 100755
--- a/tests/qemu-iotests/017
+++ b/tests/qemu-iotests/017
@@ -40,7 +40,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
. ./common.pattern
# Any format supporting backing files
-_supported_fmt qcow qcow2 vmdk qed
+_supported_fmt qcow qcow2 vmdk qed add-cow
_supported_proto generic
_supported_os Linux
diff --git a/tests/qemu-iotests/020 b/tests/qemu-iotests/020
index 2fb0ff8..3dbb495 100755
--- a/tests/qemu-iotests/020
+++ b/tests/qemu-iotests/020
@@ -42,7 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
. ./common.pattern
# Any format supporting backing files
-_supported_fmt qcow qcow2 vmdk qed
+_supported_fmt qcow qcow2 vmdk qed add-cow
_supported_proto generic
_supported_os Linux
diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index 432732c..122267b 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -243,7 +243,7 @@ do
echo " - no qualified output"
err=true
else
- if diff -w $seq.out $tmp.out >/dev/null 2>&1
+ if diff -w -I "^Formatting" $seq.out $tmp.out >/dev/null 2>&1
then
echo ""
if $err
@@ -255,7 +255,7 @@ do
else
echo " - output mismatch (see $seq.out.bad)"
mv $tmp.out $seq.out.bad
- $diff -w $seq.out $seq.out.bad
+ $diff -w -I "^Formatting" $seq.out $seq.out.bad
err=true
fi
fi
diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
index 1f6fdf5..1c81b09 100644
--- a/tests/qemu-iotests/common
+++ b/tests/qemu-iotests/common
@@ -128,6 +128,7 @@ common options
check options
-raw test raw (default)
-cow test cow
+ -add-cow test add-cow
-qcow test qcow
-qcow2 test qcow2
-qed test qed
@@ -163,6 +164,11 @@ testlist options
xpand=false
;;
+ -add-cow)
+ IMGFMT=add-cow
+ xpand=false
+ ;;
+
-qcow)
IMGFMT=qcow
xpand=false
diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index 5e3a524..b2bdc6c 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -97,6 +97,18 @@ _make_test_img()
fi
if [ \( "$IMGFMT" = "qcow2" -o "$IMGFMT" = "qed" \) -a -n "$CLUSTER_SIZE" ]; then
optstr=$(_optstr_add "$optstr" "cluster_size=$CLUSTER_SIZE")
+ elif [ "$IMGFMT" = "add-cow" ]; then
+ local BACKING="$TEST_IMG"".qcow2"
+ local IMG="$TEST_IMG"".raw"
+ if [ "$1" = "-b" ]; then
+ IMG="$IMG"".b"
+ $QEMU_IMG create -f raw $IMG $image_size>/dev/null
+ extra_img_options="-o image_file=$IMG $extra_img_options"
+ else
+ $QEMU_IMG create -f raw $IMG $image_size>/dev/null
+ $QEMU_IMG create -f qcow2 $BACKING $image_size>/dev/null
+ extra_img_options="-o backing_file=$BACKING,image_file=$IMG"
+ fi
fi
if [ -n "$optstr" ]; then
@@ -124,6 +136,13 @@ _cleanup_test_img()
rm -f $TEST_DIR/t.$IMGFMT
rm -f $TEST_DIR/t.$IMGFMT.orig
rm -f $TEST_DIR/t.$IMGFMT.base
+ if [ "$IMGFMT" = "add-cow" ]; then
+ rm -f $TEST_DIR/t.$IMGFMT.qcow2
+ rm -f $TEST_DIR/t.$IMGFMT.raw
+ rm -f $TEST_DIR/t.$IMGFMT.raw.b
+ rm -f $TEST_DIR/t.$IMGFMT.ct.qcow2
+ rm -f $TEST_DIR/t.$IMGFMT.ct.raw
+ fi
;;
rbd)
--
1.7.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format
2012-07-31 16:51 [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format Dong Xu Wang
` (4 preceding siblings ...)
2012-07-31 16:51 ` [Qemu-devel] [PATCH 6/6 v11] add-cow: support qemu-iotests Dong Xu Wang
@ 2012-08-01 13:51 ` Eric Blake
2012-08-02 7:03 ` Dong Xu Wang
2012-08-01 13:55 ` Stefan Hajnoczi
6 siblings, 1 reply; 30+ messages in thread
From: Eric Blake @ 2012-08-01 13:51 UTC (permalink / raw)
To: Dong Xu Wang; +Cc: kwolf, qemu-devel
[-- Attachment #1: Type: text/plain, Size: 7721 bytes --]
On 07/31/2012 10:51 AM, Dong Xu Wang wrote:
> Introduce a new file format:add-cow. The usage can be found at this patch.
>
> Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
> ---
> Now add-cow is still using QEMUOptionParameter, not QemuOpts, I will send a
> seperate patch series to convert.
s/seperate/separate/
>
> docs/specs/add-cow.txt | 128 ++++++++++++++++++++++++++++++++++++++++++++++++
> 1 files changed, 128 insertions(+), 0 deletions(-)
> create mode 100644 docs/specs/add-cow.txt
>
> diff --git a/docs/specs/add-cow.txt b/docs/specs/add-cow.txt
> new file mode 100644
> index 0000000..4793a3e
> --- /dev/null
> +++ b/docs/specs/add-cow.txt
> @@ -0,0 +1,128 @@
> +== General ==
> +
> +Raw file format does not support backing file and copy on write feature.
grammar:
The raw file format does not support backing files or the copy-on-write
feature.
> +The add-cow image format makes it possible to use backing files with raw
> +image by keeping a separate .add-cow metadata file. Once all sectors
> +have been written into the raw image it is safe to discard the .add-cow
> +and backing files, then we can use the raw image directly.
I'm still not sure how this series fits in with the recent discussions
on adding drive-mirror with the capability of creating a raw file mirror
of a qcow2 snapshot.
> +
> +While using add-cow, procedures may like this:
grammar:
An example usage of add-cow would look like:
> +(ubuntu.img is a disk image which has been installed OS.)
> + 1) Create a raw image with the same size of ubuntu.img
> + qemu-img create -f raw test.raw 8G
> + 2) Create an add-cow image which will store dirty bitmap
> + qemu-img create -f add-cow test.add-cow \
> + -o backing_file=ubuntu.img,image_file=test.raw
> + 3) Run qemu with add-cow image
> + qemu -drive if=virtio,file=test.add-cow
> +
> +test.raw may be larger than ubuntu.img, in that case, the size of test.add-cow
> +will be calculated by the size of ubuntu.img, test.raw will be used from the
> +1st byte, the rest part can be used for other purpose.
Grammar was off here, but I'm not sure what you meant to suggest a
replacement. Maybe:
the size of test.add-cow will be calculated from the size of ubuntu.img,
and extra space at the tail of test.raw can be used for other purposes.
> +(#define HEADER_SIZE (4096 * header_pages_size))
> + Byte 0 - 7: magic
> + add-cow magic string ("ADD_COW\xff").
> +
> + 8 - 11: version
> + Version number (only valid value is 1 now).
> +
> + 12 - 15: backing file name offset
> + Offset in the add-cow file at which the backing file
> + name is stored (NB: The string is not null terminated).
Nit: this should be nul-terminated (NUL is the one-byte all-0 character
in single byte encodings that ends a string, and NULL is the four- or
eight-byte value, typically all-0, for a pointer to nowhere).
> +
> + 28 - 35: features
> + Currently only 3 feature bit is used:
> + Feature bits:
> + The image uses a backing file:
> + * ADD_COW_F_BACKING_FILE = 0x01.
Isn't this bit redundant with the earlier field at byte 12 stating
whether a backing file is present?
> + The image uses a image file:
> + * ADD_COW_F_IMAGE_FILE = 0x02.
The field at byte 20 implies that an image file name is mandatory,
meaning this bit is always 1 and therefore pointless.
> + All bits in bitmap have been set to 1, add-cow wrapper
> + can be discarded.
> + * ADD_COW_F_All_ALLOCATED = 0x04.
> +
> + 36 - 43: optional features
> + Not used now. Researved for future use.
s/Researved/Reserved/, mention that it must be set to 0
> +
> + 44 - 47: header pages size
> + The header field is variable-sized. This field indicates
> + how many pages(4k) will be used to store add-cow header.
> + In add-cow v1, it is fixed to 1, so the header size will
> + be 4k * 1 = 4096 bytes.
> +
> +Image file name and backing file name must NOT be the same, we prevent this
> +while creating add-cow files.
> +
> +Image file and backing file are interpreted relative to the qcow2 file, not
> +to the current working directory of the process that opened the qcow2 file.
Either indent this description to match the field it is describing, or
sink it down until after you have called out the header layout.
> +
> +== Reserved ==
> +
> + Byte 48 - 63: backing file format
> + format of backing file. It will be filled with 0 if
> + backing file name offset is 0. If backing file name
> + offset is none-zero, it must be non-zero.
s/none-zero/non-zero/
Why are you defining these byte offsets in the Reserved section? This
text should occur earlier in the mandatory header format. Is this field
free-form ASCII? Must the field be NUL-terminated? For that matter, I
think you can just delete the ==Reserved== header, as you are calling
out every possible offset.
> +
> + 64 - 79: image file format
> + format of image file. It must be non-zero.
Same question about whether this field is ASCII, and must be NUL-terminated.
> +
> + 80 - [HEADER_SIZE - 1]:
> + It is used to make sure COW bitmap field starts at the
> + HEADER_SIZE byte, backing file name and image file name
> + will be stored here.
Is it required that bytes not pointed to by backing file and image names
must have any particular value?
> +
> +== COW bitmap ==
> +
> +The "COW bitmap" field starts at the 4096th byte, stores a bitmap related to
> +backing file and image file. The bitmap will track whether the sector in
> +backing file is dirty or not.
> +
> +Each bit in the bitmap indicates one cluster's status. One cluster includes 128
> +sectors, then each bit indicates 512 * 128 = 64k bytes. the size of bitmap is
> +calculated according to virtual size of backing file. In each byte, bit 0 to 7
> +will track the 1st to 7th cluster in sequence, bit orders in one byte look like:
1st to 7th is only 7 clusters. You mean either '0th to 7th' or '1st to
8th'. Or just simplify to:
Within each byte, the least significant bit covers the first cluster.
> + +----+----+----+----+----+----+----+----+
> + | b7 | b6 | b5 | b4 | b3 | b2 | b1 | b0 |
> + +----+----+----+----+----+----+----+----+
> +
> +If the bit is 0, indicates the sector has not been allocated in image file, data
> +should be loaded from backing file while reading; if the bit is 1, indicates the
> +related sector has been dirty, should be loaded from image file while reading.
> +Writing to a sector causes the corresponding bit to be set to 1.
> +
> +If raw image is not an even multiple of cluster bytes, bits that correspond to
> +bytes beyond the raw file size in add-cow will be 0.
Will this affect the use of the ADD_COW_F_ALL_ALLOCATED feature bit in
the header?
--
Eric Blake eblake@redhat.com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 620 bytes --]
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 2/6 v11 v11] block: make some functions public
2012-07-31 16:51 ` [Qemu-devel] [PATCH 2/6 v11 v11] block: make some functions public Dong Xu Wang
@ 2012-08-01 13:53 ` Eric Blake
2012-08-02 7:10 ` Dong Xu Wang
2012-08-01 14:01 ` Stefan Hajnoczi
1 sibling, 1 reply; 30+ messages in thread
From: Eric Blake @ 2012-08-01 13:53 UTC (permalink / raw)
To: Dong Xu Wang; +Cc: kwolf, qemu-devel
[-- Attachment #1: Type: text/plain, Size: 1244 bytes --]
On 07/31/2012 10:51 AM, Dong Xu Wang wrote:
> In add-cow file format, we will use path_has_protocol and we will read
> a NUL-terminated string from image , qed_read_string has done the samething,
s/image ,/image,/
s/samething/same thing/
> so make the two functions public, then we will reuse them directly.
>
> While creating images files, if no size is specified, will use size of backing
> file. If no backing file is specified, we will use the size of image file.
>
> Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
> ---
> block.c | 37 +++++++++++++++++++++++++++++++++----
> block.h | 3 +++
> block/qed.c | 29 +----------------------------
> 3 files changed, 37 insertions(+), 32 deletions(-)
>
> + /* The size for the image must always be specified, with one exception:
> + If we are using a backing file, we can obtain the size from there,
> + but if not, and we are using an image file, we will obtain the size from it.*/
grammar:
The size for the image must be known, either by direct specification, or
by reading it from a backing file or image file.
--
Eric Blake eblake@redhat.com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 620 bytes --]
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format
2012-07-31 16:51 [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format Dong Xu Wang
` (5 preceding siblings ...)
2012-08-01 13:51 ` [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format Eric Blake
@ 2012-08-01 13:55 ` Stefan Hajnoczi
2012-08-02 7:09 ` Dong Xu Wang
6 siblings, 1 reply; 30+ messages in thread
From: Stefan Hajnoczi @ 2012-08-01 13:55 UTC (permalink / raw)
To: Dong Xu Wang; +Cc: kwolf, qemu-devel
On Tue, Jul 31, 2012 at 5:51 PM, Dong Xu Wang
<wdongxu@linux.vnet.ibm.com> wrote:
> Introduce a new file format:add-cow. The usage can be found at this patch.
>
> Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
> ---
> Now add-cow is still using QEMUOptionParameter, not QemuOpts, I will send a
> seperate patch series to convert.
I suggest including a cover letter in future patch series:
git format-patch --cover-letter --numbered -o my-series/ master..
Or to do it in a single command with git-send-email(1) use the --compose option.
The cover letter makes it easy for reviewers to add their Reviewed-by:
to the entire series by replying only once. It's also a good place to
include a changelog that tells reviewers what you changed from the
last published version.
> docs/specs/add-cow.txt | 128 ++++++++++++++++++++++++++++++++++++++++++++++++
> 1 files changed, 128 insertions(+), 0 deletions(-)
> create mode 100644 docs/specs/add-cow.txt
>
> diff --git a/docs/specs/add-cow.txt b/docs/specs/add-cow.txt
> new file mode 100644
> index 0000000..4793a3e
> --- /dev/null
> +++ b/docs/specs/add-cow.txt
> @@ -0,0 +1,128 @@
> +== General ==
> +
> +Raw file format does not support backing file and copy on write feature.
> +The add-cow image format makes it possible to use backing files with raw
> +image by keeping a separate .add-cow metadata file. Once all sectors
> +have been written into the raw image it is safe to discard the .add-cow
> +and backing files, then we can use the raw image directly.
> +
> +While using add-cow, procedures may like this:
> +(ubuntu.img is a disk image which has been installed OS.)
> + 1) Create a raw image with the same size of ubuntu.img
> + qemu-img create -f raw test.raw 8G
> + 2) Create an add-cow image which will store dirty bitmap
> + qemu-img create -f add-cow test.add-cow \
> + -o backing_file=ubuntu.img,image_file=test.raw
> + 3) Run qemu with add-cow image
> + qemu -drive if=virtio,file=test.add-cow
> +
> +test.raw may be larger than ubuntu.img, in that case, the size of test.add-cow
> +will be calculated by the size of ubuntu.img, test.raw will be used from the
> +1st byte, the rest part can be used for other purpose.
This is not how backing files normally work. With qcow2 or qed a
smaller backing file just means that the guest reads zeroes from the
areas beyond the end of the backing file. Is there a special reason
why you want to implement the behavior you described in the spec?
Otherwise I suggest implementing the same behavior as qcow2/qed.
> +
> +=Specification=
> +
> +The file format looks like this:
> +
> + +---------------+-------------+-----------------+
> + | Header | Reserved | COW bitmap |
> + +---------------+-------------+-----------------+
> +
> +All numbers in add-cow are stored in Little Endian byte order.
> +
> +== Header ==
> +
> +The Header is included in the first bytes:
> +(#define HEADER_SIZE (4096 * header_pages_size))
> + Byte 0 - 7: magic
> + add-cow magic string ("ADD_COW\xff").
> +
> + 8 - 11: version
> + Version number (only valid value is 1 now).
> +
> + 12 - 15: backing file name offset
> + Offset in the add-cow file at which the backing file
> + name is stored (NB: The string is not null terminated).
> + If backing file name does NOT exist, this field will be
> + 0. Must be between 80 and [HEADER_SIZE - 2](a file name
> + must be at least 1 byte).
> +
> + 16 - 19: backing file name size
> + Length of the backing file name in bytes. It will be 0
> + if the backing file name offset is 0. If backing file
> + name offset is non-zero, then it must be non-zero. Must
> + be less than [HEADER_SIZE - 80] to fit in the reserved
> + part of the header.
> +
> + 20 - 23: image file name offset
> + Offset in the add-cow file at which the image file name
> + is stored (NB: The string is not null terminated). It
> + must be between 80 and [HEADER_SIZE - 2].
> +
> + 24 - 27: image file name size
> + Length of the image file name in bytes.
> + Must be less than [HEADER_SIZE - 80] to fit in the reserved
> + part of the header.
> +
> + 28 - 35: features
> + Currently only 3 feature bit is used:
> + Feature bits:
> + The image uses a backing file:
> + * ADD_COW_F_BACKING_FILE = 0x01.
> + The image uses a image file:
> + * ADD_COW_F_IMAGE_FILE = 0x02.
> + All bits in bitmap have been set to 1, add-cow wrapper
> + can be discarded.
> + * ADD_COW_F_All_ALLOCATED = 0x04.
> +
> + 36 - 43: optional features
> + Not used now. Researved for future use.
s/Researved/Reserved/
> +
> + 44 - 47: header pages size
> + The header field is variable-sized. This field indicates
> + how many pages(4k) will be used to store add-cow header.
> + In add-cow v1, it is fixed to 1, so the header size will
> + be 4k * 1 = 4096 bytes.
> +
> +Image file name and backing file name must NOT be the same, we prevent this
> +while creating add-cow files.
> +
> +Image file and backing file are interpreted relative to the qcow2 file, not
> +to the current working directory of the process that opened the qcow2 file.
> +
> +== Reserved ==
> +
> + Byte 48 - 63: backing file format
> + format of backing file. It will be filled with 0 if
> + backing file name offset is 0. If backing file name
> + offset is none-zero, it must be non-zero.
s/none-zero/non-zero/
> +
> + 64 - 79: image file format
> + format of image file. It must be non-zero.
> +
> + 80 - [HEADER_SIZE - 1]:
> + It is used to make sure COW bitmap field starts at the
> + HEADER_SIZE byte, backing file name and image file name
> + will be stored here.
> +
> +== COW bitmap ==
> +
> +The "COW bitmap" field starts at the 4096th byte, stores a bitmap related to
I would say it starts at offset HEADER_SIZE. 4096th byte == offset
0x0fff, 4097th byte == offset 0x1000.
Stefan
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 3/6] add-cow file format
2012-07-31 16:51 ` [Qemu-devel] [PATCH 3/6] add-cow file format Dong Xu Wang
@ 2012-08-01 13:57 ` Eric Blake
2012-08-01 14:14 ` Stefan Hajnoczi
2012-08-02 7:20 ` Dong Xu Wang
2012-08-01 15:31 ` Stefan Hajnoczi
1 sibling, 2 replies; 30+ messages in thread
From: Eric Blake @ 2012-08-01 13:57 UTC (permalink / raw)
To: Dong Xu Wang; +Cc: kwolf, qemu-devel
[-- Attachment #1: Type: text/plain, Size: 1078 bytes --]
On 07/31/2012 10:51 AM, Dong Xu Wang wrote:
> This is the implementation code for add-cow file format. Because image_file
> might be very huge, then we can't read entire bitmap into memory, we must use
> a cache. Since qcow-cache.c has implemted cache code, we can create our cache
s/implemted/implemented/
> code based on it.
Just wondering if Paolo's HBitmap code for drive-mirror might be a more
efficient way to implement your caching.
>
> Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
> ---
> block/Makefile.objs | 1 +
> block/add-cow-cache.c | 206 +++++++++++++++++
> block/add-cow.c | 599 +++++++++++++++++++++++++++++++++++++++++++++++++
> block/add-cow.h | 101 +++++++++
> block_int.h | 2 +
Rather than adding a new implementation for code duplication, can you
refactor the existing implementation to be reusable, and update
qcow-cache.c to call into the common refactored code?
--
Eric Blake eblake@redhat.com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 620 bytes --]
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 2/6 v11 v11] block: make some functions public
2012-07-31 16:51 ` [Qemu-devel] [PATCH 2/6 v11 v11] block: make some functions public Dong Xu Wang
2012-08-01 13:53 ` Eric Blake
@ 2012-08-01 14:01 ` Stefan Hajnoczi
2012-08-02 7:11 ` Dong Xu Wang
1 sibling, 1 reply; 30+ messages in thread
From: Stefan Hajnoczi @ 2012-08-01 14:01 UTC (permalink / raw)
To: Dong Xu Wang; +Cc: kwolf, qemu-devel
On Tue, Jul 31, 2012 at 5:51 PM, Dong Xu Wang
<wdongxu@linux.vnet.ibm.com> wrote:
> diff --git a/block.h b/block.h
> index c89590d..b523076 100644
> --- a/block.h
> +++ b/block.h
> @@ -152,6 +152,8 @@ int bdrv_pwrite(BlockDriverState *bs, int64_t offset,
> const void *buf, int count);
> int bdrv_pwrite_sync(BlockDriverState *bs, int64_t offset,
> const void *buf, int count);
> +int bdrv_read_string(BlockDriverState *file, uint64_t offset, size_t n,
> + char *buf, size_t buflen);
I suggest renaming the "file" argument to "bs" like all the other
block.h functions.
> /**
> - * Read a string of known length from the image file
> - *
> - * @file: Image file
> - * @offset: File offset to start of string, in bytes
> - * @n: String length in bytes
> - * @buf: Destination buffer
> - * @buflen: Destination buffer length in bytes
> - * @ret: 0 on success, -errno on failure
> - *
> - * The string is NUL-terminated.
> - */
Please keep the doc comment.
Stefan
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 3/6] add-cow file format
2012-08-01 13:57 ` Eric Blake
@ 2012-08-01 14:14 ` Stefan Hajnoczi
2012-08-01 14:21 ` Kevin Wolf
2012-08-02 7:20 ` Dong Xu Wang
1 sibling, 1 reply; 30+ messages in thread
From: Stefan Hajnoczi @ 2012-08-01 14:14 UTC (permalink / raw)
To: Eric Blake; +Cc: kwolf, Dong Xu Wang, qemu-devel
On Wed, Aug 1, 2012 at 2:57 PM, Eric Blake <eblake@redhat.com> wrote:
> On 07/31/2012 10:51 AM, Dong Xu Wang wrote:
>> This is the implementation code for add-cow file format. Because image_file
>> might be very huge, then we can't read entire bitmap into memory, we must use
>> a cache. Since qcow-cache.c has implemted cache code, we can create our cache
>
> s/implemted/implemented/
>
>> code based on it.
>
> Just wondering if Paolo's HBitmap code for drive-mirror might be a more
> efficient way to implement your caching.
>
>>
>> Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
>> ---
>> block/Makefile.objs | 1 +
>> block/add-cow-cache.c | 206 +++++++++++++++++
>> block/add-cow.c | 599 +++++++++++++++++++++++++++++++++++++++++++++++++
>> block/add-cow.h | 101 +++++++++
>> block_int.h | 2 +
>
> Rather than adding a new implementation for code duplication, can you
> refactor the existing implementation to be reusable, and update
> qcow-cache.c to call into the common refactored code?
I looked at this and block/qcow2-cache.c seems fairly clean and
generic. The only things that need to be parameterized are the
s->l2_table_cache/s->refcount_block_cache and s->cluster_size.
These can be fixed by adding fields to Qcow2Cache:
size_t table_size; /* cached table size in bytes */
BlkDebugEvent flush_event;
BlkDebugEvent load_event;
Or maybe a table type enum (L2/refcount blocks/bitmap blocks).
If Kevin agrees with extracting this from qcow2 (i.e. we can't add
more qcow2-specific behavior easily in the future), then this looks
doable.
Stefan
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 3/6] add-cow file format
2012-08-01 14:14 ` Stefan Hajnoczi
@ 2012-08-01 14:21 ` Kevin Wolf
0 siblings, 0 replies; 30+ messages in thread
From: Kevin Wolf @ 2012-08-01 14:21 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: Dong Xu Wang, Eric Blake, qemu-devel
Am 01.08.2012 16:14, schrieb Stefan Hajnoczi:
> On Wed, Aug 1, 2012 at 2:57 PM, Eric Blake <eblake@redhat.com> wrote:
>> On 07/31/2012 10:51 AM, Dong Xu Wang wrote:
>>> This is the implementation code for add-cow file format. Because image_file
>>> might be very huge, then we can't read entire bitmap into memory, we must use
>>> a cache. Since qcow-cache.c has implemted cache code, we can create our cache
>>
>> s/implemted/implemented/
>>
>>> code based on it.
>>
>> Just wondering if Paolo's HBitmap code for drive-mirror might be a more
>> efficient way to implement your caching.
>>
>>>
>>> Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
>>> ---
>>> block/Makefile.objs | 1 +
>>> block/add-cow-cache.c | 206 +++++++++++++++++
>>> block/add-cow.c | 599 +++++++++++++++++++++++++++++++++++++++++++++++++
>>> block/add-cow.h | 101 +++++++++
>>> block_int.h | 2 +
>>
>> Rather than adding a new implementation for code duplication, can you
>> refactor the existing implementation to be reusable, and update
>> qcow-cache.c to call into the common refactored code?
>
> I looked at this and block/qcow2-cache.c seems fairly clean and
> generic. The only things that need to be parameterized are the
> s->l2_table_cache/s->refcount_block_cache and s->cluster_size.
>
> These can be fixed by adding fields to Qcow2Cache:
> size_t table_size; /* cached table size in bytes */
> BlkDebugEvent flush_event;
> BlkDebugEvent load_event;
>
> Or maybe a table type enum (L2/refcount blocks/bitmap blocks).
>
> If Kevin agrees with extracting this from qcow2 (i.e. we can't add
> more qcow2-specific behavior easily in the future), then this looks
> doable.
I don't think we'd want to add qcow2 specific stuff there, and I've been
meaning to generalise it for quite a while, so feel free to do it.
One thing to note is that it currently depends on qcow2's s->lock being
taken while it's called. Eventually we'll want finer grained locking in
the cache itself and I believe this will also require some changes in
the interface, but for now add-cow can just do the same qcow2 and take
its global CoMutex.
Kevin
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 3/6] add-cow file format
2012-07-31 16:51 ` [Qemu-devel] [PATCH 3/6] add-cow file format Dong Xu Wang
2012-08-01 13:57 ` Eric Blake
@ 2012-08-01 15:31 ` Stefan Hajnoczi
2012-08-02 7:20 ` Dong Xu Wang
1 sibling, 1 reply; 30+ messages in thread
From: Stefan Hajnoczi @ 2012-08-01 15:31 UTC (permalink / raw)
To: Dong Xu Wang; +Cc: kwolf, qemu-devel
On Tue, Jul 31, 2012 at 5:51 PM, Dong Xu Wang
<wdongxu@linux.vnet.ibm.com> wrote:
> + if (backing_filename) {
> + header.features |= ADD_COW_F_BACKING_FILE;
> + header.backing_filename_offset = sizeof(header) + 16 * 2;
It's not obvious what 16 * 2 is.
> + ret = bdrv_pwrite(bs, sizeof(le_header) + 16,
> + image_format ? image_format : "raw",
> + image_format ? sizeof(image_format) : sizeof("raw"));
sizeof(image_format)? I think this should be strlen(image_format).
Stefan
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 4/6 v11] add-cow: support snapshot_blkde
2012-07-31 16:51 ` [Qemu-devel] [PATCH 4/6 v11] add-cow: support snapshot_blkde Dong Xu Wang
@ 2012-08-01 15:37 ` Stefan Hajnoczi
2012-08-02 7:28 ` Dong Xu Wang
0 siblings, 1 reply; 30+ messages in thread
From: Stefan Hajnoczi @ 2012-08-01 15:37 UTC (permalink / raw)
To: Dong Xu Wang; +Cc: kwolf, qemu-devel
On Tue, Jul 31, 2012 at 5:51 PM, Dong Xu Wang
<wdongxu@linux.vnet.ibm.com> wrote:
> add-cow will let raw file support snapshot_blkdev indirectly.
>
> Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
> ---
> blockdev.c | 45 +++++++++++++++++++++++++++++++++++++--------
> docs/live-block-ops.txt | 11 ++++++++++-
> 2 files changed, 47 insertions(+), 9 deletions(-)
This patch would need to update qapi-schema.json and qmp-commands.hx
to make use of the new arguments.
I don't think QEMU compiles cleanly after this patch. It's important
to keep the build clean after every patch so that git-bisect(1) can be
used (if you hit a broken build during a bisect it makes things
harder).
However, I think the QMP/HMP changes should not be included in this
series unless you are sure the interface is stable. When you convert
QEMUOptionsParameter will transaction or snapshot-blkdev-sync change?
If so, please only send the add-cow image format and leave the QMP/HMP
changes until you have the QemuOpts solution.
Stefan
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format
2012-08-01 13:51 ` [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format Eric Blake
@ 2012-08-02 7:03 ` Dong Xu Wang
0 siblings, 0 replies; 30+ messages in thread
From: Dong Xu Wang @ 2012-08-02 7:03 UTC (permalink / raw)
To: Eric Blake; +Cc: kwolf, qemu-devel
On Wed, Aug 1, 2012 at 9:51 PM, Eric Blake <eblake@redhat.com> wrote:
> On 07/31/2012 10:51 AM, Dong Xu Wang wrote:
>> Introduce a new file format:add-cow. The usage can be found at this patch.
>>
>> Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
>> ---
>> Now add-cow is still using QEMUOptionParameter, not QemuOpts, I will send a
>> seperate patch series to convert.
>
> s/seperate/separate/
>
Yep.
>>
>> docs/specs/add-cow.txt | 128 ++++++++++++++++++++++++++++++++++++++++++++++++
>> 1 files changed, 128 insertions(+), 0 deletions(-)
>> create mode 100644 docs/specs/add-cow.txt
>>
>> diff --git a/docs/specs/add-cow.txt b/docs/specs/add-cow.txt
>> new file mode 100644
>> index 0000000..4793a3e
>> --- /dev/null
>> +++ b/docs/specs/add-cow.txt
>> @@ -0,0 +1,128 @@
>> +== General ==
>> +
>> +Raw file format does not support backing file and copy on write feature.
>
> grammar:
> The raw file format does not support backing files or the copy-on-write
> feature.
Yep.
>
>> +The add-cow image format makes it possible to use backing files with raw
>> +image by keeping a separate .add-cow metadata file. Once all sectors
>> +have been written into the raw image it is safe to discard the .add-cow
>> +and backing files, then we can use the raw image directly.
>
> I'm still not sure how this series fits in with the recent discussions
> on adding drive-mirror with the capability of creating a raw file mirror
> of a qcow2 snapshot.
>
>
>> +
>> +While using add-cow, procedures may like this:
>
> grammar:
> An example usage of add-cow would look like:
>
Yep.
>> +(ubuntu.img is a disk image which has been installed OS.)
>> + 1) Create a raw image with the same size of ubuntu.img
>> + qemu-img create -f raw test.raw 8G
>> + 2) Create an add-cow image which will store dirty bitmap
>> + qemu-img create -f add-cow test.add-cow \
>> + -o backing_file=ubuntu.img,image_file=test.raw
>> + 3) Run qemu with add-cow image
>> + qemu -drive if=virtio,file=test.add-cow
>> +
>> +test.raw may be larger than ubuntu.img, in that case, the size of test.add-cow
>> +will be calculated by the size of ubuntu.img, test.raw will be used from the
>> +1st byte, the rest part can be used for other purpose.
>
> Grammar was off here, but I'm not sure what you meant to suggest a
> replacement. Maybe:
>
> the size of test.add-cow will be calculated from the size of ubuntu.img,
> and extra space at the tail of test.raw can be used for other purposes.
Yes, this is what I mean.
>
>> +(#define HEADER_SIZE (4096 * header_pages_size))
>> + Byte 0 - 7: magic
>> + add-cow magic string ("ADD_COW\xff").
>> +
>> + 8 - 11: version
>> + Version number (only valid value is 1 now).
>> +
>> + 12 - 15: backing file name offset
>> + Offset in the add-cow file at which the backing file
>> + name is stored (NB: The string is not null terminated).
>
> Nit: this should be nul-terminated (NUL is the one-byte all-0 character
> in single byte encodings that ends a string, and NULL is the four- or
> eight-byte value, typically all-0, for a pointer to nowhere).
>
>
>> +
>> + 28 - 35: features
>> + Currently only 3 feature bit is used:
>> + Feature bits:
>> + The image uses a backing file:
>> + * ADD_COW_F_BACKING_FILE = 0x01.
>
> Isn't this bit redundant with the earlier field at byte 12 stating
> whether a backing file is present?
>
Yes, it is redundant. I just want to make if the add-cow image is
using backing_file more clear.
>> + The image uses a image file:
>> + * ADD_COW_F_IMAGE_FILE = 0x02.
>
> The field at byte 20 implies that an image file name is mandatory,
> meaning this bit is always 1 and therefore pointless.
>
>> + All bits in bitmap have been set to 1, add-cow wrapper
>> + can be discarded.
>> + * ADD_COW_F_All_ALLOCATED = 0x04.
>> +
>> + 36 - 43: optional features
>> + Not used now. Researved for future use.
>
> s/Researved/Reserved/, mention that it must be set to 0
Okay.
>
>> +
>> + 44 - 47: header pages size
>> + The header field is variable-sized. This field indicates
>> + how many pages(4k) will be used to store add-cow header.
>> + In add-cow v1, it is fixed to 1, so the header size will
>> + be 4k * 1 = 4096 bytes.
>> +
>> +Image file name and backing file name must NOT be the same, we prevent this
>> +while creating add-cow files.
>> +
>> +Image file and backing file are interpreted relative to the qcow2 file, not
>> +to the current working directory of the process that opened the qcow2 file.
>
> Either indent this description to match the field it is describing, or
> sink it down until after you have called out the header layout.
Okay.
>
>> +
>> +== Reserved ==
>> +
>> + Byte 48 - 63: backing file format
>> + format of backing file. It will be filled with 0 if
>> + backing file name offset is 0. If backing file name
>> + offset is none-zero, it must be non-zero.
>
> s/none-zero/non-zero/
Okay.
>
> Why are you defining these byte offsets in the Reserved section? This
> text should occur earlier in the mandatory header format. Is this field
> free-form ASCII? Must the field be NUL-terminated? For that matter, I
> think you can just delete the ==Reserved== header, as you are calling
> out every possible offset.
>
Okay. I will describle more clearly in v12.
>> +
>> + 64 - 79: image file format
>> + format of image file. It must be non-zero.
>
> Same question about whether this field is ASCII, and must be NUL-terminated.
>
Okay.
>> +
>> + 80 - [HEADER_SIZE - 1]:
>> + It is used to make sure COW bitmap field starts at the
>> + HEADER_SIZE byte, backing file name and image file name
>> + will be stored here.
>
> Is it required that bytes not pointed to by backing file and image names
> must have any particular value?
It must be 0, I will correct it later.
>
>> +
>> +== COW bitmap ==
>> +
>> +The "COW bitmap" field starts at the 4096th byte, stores a bitmap related to
>> +backing file and image file. The bitmap will track whether the sector in
>> +backing file is dirty or not.
>> +
>> +Each bit in the bitmap indicates one cluster's status. One cluster includes 128
>> +sectors, then each bit indicates 512 * 128 = 64k bytes. the size of bitmap is
>> +calculated according to virtual size of backing file. In each byte, bit 0 to 7
>> +will track the 1st to 7th cluster in sequence, bit orders in one byte look like:
>
> 1st to 7th is only 7 clusters. You mean either '0th to 7th' or '1st to
> 8th'. Or just simplify to:
>
> Within each byte, the least significant bit covers the first cluster.
Okay.
>
>> + +----+----+----+----+----+----+----+----+
>> + | b7 | b6 | b5 | b4 | b3 | b2 | b1 | b0 |
>> + +----+----+----+----+----+----+----+----+
>> +
>> +If the bit is 0, indicates the sector has not been allocated in image file, data
>> +should be loaded from backing file while reading; if the bit is 1, indicates the
>> +related sector has been dirty, should be loaded from image file while reading.
>> +Writing to a sector causes the corresponding bit to be set to 1.
>> +
>> +If raw image is not an even multiple of cluster bytes, bits that correspond to
>> +bytes beyond the raw file size in add-cow will be 0.
>
> Will this affect the use of the ADD_COW_F_ALL_ALLOCATED feature bit in
> the header?
No, beyond the raw file size, the related bits in bitmap will make no sense.
I mean: ADD_COW_F_ALL_ALLOCATED will be set when:
1) If add-cow is created only use image_file option, but no
backing_file option.
In this case, bitmap will be useless.
2) when all sectors have been allocated in image_file, we will not use
image_file.
but in this case, I think it is hard to determine whether all sectors
have been allocated
or not, scanning all bitmap will take some time, maybe we can do this
when we call
"qemu-io check"? Or before opening add-cow files, we do a check?
Any comments?
Thank you, Eric,
>
> --
> Eric Blake eblake@redhat.com +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format
2012-08-01 13:55 ` Stefan Hajnoczi
@ 2012-08-02 7:09 ` Dong Xu Wang
2012-08-02 10:44 ` Stefan Hajnoczi
0 siblings, 1 reply; 30+ messages in thread
From: Dong Xu Wang @ 2012-08-02 7:09 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: kwolf, qemu-devel
On Wed, Aug 1, 2012 at 9:55 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Tue, Jul 31, 2012 at 5:51 PM, Dong Xu Wang
> <wdongxu@linux.vnet.ibm.com> wrote:
>> Introduce a new file format:add-cow. The usage can be found at this patch.
>>
>> Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
>> ---
>> Now add-cow is still using QEMUOptionParameter, not QemuOpts, I will send a
>> seperate patch series to convert.
>
> I suggest including a cover letter in future patch series:
>
> git format-patch --cover-letter --numbered -o my-series/ master..
>
> Or to do it in a single command with git-send-email(1) use the --compose option.
>
> The cover letter makes it easy for reviewers to add their Reviewed-by:
> to the entire series by replying only once. It's also a good place to
> include a changelog that tells reviewers what you changed from the
> last published version.
Okay.
>
>> docs/specs/add-cow.txt | 128 ++++++++++++++++++++++++++++++++++++++++++++++++
>> 1 files changed, 128 insertions(+), 0 deletions(-)
>> create mode 100644 docs/specs/add-cow.txt
>>
>> diff --git a/docs/specs/add-cow.txt b/docs/specs/add-cow.txt
>> new file mode 100644
>> index 0000000..4793a3e
>> --- /dev/null
>> +++ b/docs/specs/add-cow.txt
>> @@ -0,0 +1,128 @@
>> +== General ==
>> +
>> +Raw file format does not support backing file and copy on write feature.
>> +The add-cow image format makes it possible to use backing files with raw
>> +image by keeping a separate .add-cow metadata file. Once all sectors
>> +have been written into the raw image it is safe to discard the .add-cow
>> +and backing files, then we can use the raw image directly.
>> +
>> +While using add-cow, procedures may like this:
>> +(ubuntu.img is a disk image which has been installed OS.)
>> + 1) Create a raw image with the same size of ubuntu.img
>> + qemu-img create -f raw test.raw 8G
>> + 2) Create an add-cow image which will store dirty bitmap
>> + qemu-img create -f add-cow test.add-cow \
>> + -o backing_file=ubuntu.img,image_file=test.raw
>> + 3) Run qemu with add-cow image
>> + qemu -drive if=virtio,file=test.add-cow
>> +
>> +test.raw may be larger than ubuntu.img, in that case, the size of test.add-cow
>> +will be calculated by the size of ubuntu.img, test.raw will be used from the
>> +1st byte, the rest part can be used for other purpose.
>
> This is not how backing files normally work. With qcow2 or qed a
> smaller backing file just means that the guest reads zeroes from the
> areas beyond the end of the backing file. Is there a special reason
> why you want to implement the behavior you described in the spec?
> Otherwise I suggest implementing the same behavior as qcow2/qed.
The size of add-cow will be caclulated by the size of backing_file, if
backing_file can be smaller, how can I get the size of add-cow while
being created?
Do you mean like following steps?
1) qemu-img create -f qcow2 source.qcow2 *8G*
2) qemu-img create -f t.add-cow -o backing_file=source.qcow2,image_file=t *10G*
And then reading un-allocated bytes from add-cow after 8G will be 0?
>
>> +
>> +=Specification=
>> +
>> +The file format looks like this:
>> +
>> + +---------------+-------------+-----------------+
>> + | Header | Reserved | COW bitmap |
>> + +---------------+-------------+-----------------+
>> +
>> +All numbers in add-cow are stored in Little Endian byte order.
>> +
>> +== Header ==
>> +
>> +The Header is included in the first bytes:
>> +(#define HEADER_SIZE (4096 * header_pages_size))
>> + Byte 0 - 7: magic
>> + add-cow magic string ("ADD_COW\xff").
>> +
>> + 8 - 11: version
>> + Version number (only valid value is 1 now).
>> +
>> + 12 - 15: backing file name offset
>> + Offset in the add-cow file at which the backing file
>> + name is stored (NB: The string is not null terminated).
>> + If backing file name does NOT exist, this field will be
>> + 0. Must be between 80 and [HEADER_SIZE - 2](a file name
>> + must be at least 1 byte).
>> +
>> + 16 - 19: backing file name size
>> + Length of the backing file name in bytes. It will be 0
>> + if the backing file name offset is 0. If backing file
>> + name offset is non-zero, then it must be non-zero. Must
>> + be less than [HEADER_SIZE - 80] to fit in the reserved
>> + part of the header.
>> +
>> + 20 - 23: image file name offset
>> + Offset in the add-cow file at which the image file name
>> + is stored (NB: The string is not null terminated). It
>> + must be between 80 and [HEADER_SIZE - 2].
>> +
>> + 24 - 27: image file name size
>> + Length of the image file name in bytes.
>> + Must be less than [HEADER_SIZE - 80] to fit in the reserved
>> + part of the header.
>> +
>> + 28 - 35: features
>> + Currently only 3 feature bit is used:
>> + Feature bits:
>> + The image uses a backing file:
>> + * ADD_COW_F_BACKING_FILE = 0x01.
>> + The image uses a image file:
>> + * ADD_COW_F_IMAGE_FILE = 0x02.
>> + All bits in bitmap have been set to 1, add-cow wrapper
>> + can be discarded.
>> + * ADD_COW_F_All_ALLOCATED = 0x04.
>> +
>> + 36 - 43: optional features
>> + Not used now. Researved for future use.
>
> s/Researved/Reserved/
Okay.
>
>> +
>> + 44 - 47: header pages size
>> + The header field is variable-sized. This field indicates
>> + how many pages(4k) will be used to store add-cow header.
>> + In add-cow v1, it is fixed to 1, so the header size will
>> + be 4k * 1 = 4096 bytes.
>> +
>> +Image file name and backing file name must NOT be the same, we prevent this
>> +while creating add-cow files.
>> +
>> +Image file and backing file are interpreted relative to the qcow2 file, not
>> +to the current working directory of the process that opened the qcow2 file.
>> +
>> +== Reserved ==
>> +
>> + Byte 48 - 63: backing file format
>> + format of backing file. It will be filled with 0 if
>> + backing file name offset is 0. If backing file name
>> + offset is none-zero, it must be non-zero.
>
> s/none-zero/non-zero/
Okay.
>
>> +
>> + 64 - 79: image file format
>> + format of image file. It must be non-zero.
>> +
>> + 80 - [HEADER_SIZE - 1]:
>> + It is used to make sure COW bitmap field starts at the
>> + HEADER_SIZE byte, backing file name and image file name
>> + will be stored here.
>> +
>> +== COW bitmap ==
>> +
>> +The "COW bitmap" field starts at the 4096th byte, stores a bitmap related to
>
> I would say it starts at offset HEADER_SIZE. 4096th byte == offset
> 0x0fff, 4097th byte == offset 0x1000.
Okay.
>
> Stefan
>
Thank you Stefan.
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 2/6 v11 v11] block: make some functions public
2012-08-01 13:53 ` Eric Blake
@ 2012-08-02 7:10 ` Dong Xu Wang
0 siblings, 0 replies; 30+ messages in thread
From: Dong Xu Wang @ 2012-08-02 7:10 UTC (permalink / raw)
To: Eric Blake; +Cc: kwolf, qemu-devel
On Wed, Aug 1, 2012 at 9:53 PM, Eric Blake <eblake@redhat.com> wrote:
> On 07/31/2012 10:51 AM, Dong Xu Wang wrote:
>> In add-cow file format, we will use path_has_protocol and we will read
>> a NUL-terminated string from image , qed_read_string has done the samething,
>
> s/image ,/image,/
> s/samething/same thing/
>
>> so make the two functions public, then we will reuse them directly.
>>
>> While creating images files, if no size is specified, will use size of backing
>> file. If no backing file is specified, we will use the size of image file.
>>
>> Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
>> ---
>> block.c | 37 +++++++++++++++++++++++++++++++++----
>> block.h | 3 +++
>> block/qed.c | 29 +----------------------------
>> 3 files changed, 37 insertions(+), 32 deletions(-)
>>
>> + /* The size for the image must always be specified, with one exception:
>> + If we are using a backing file, we can obtain the size from there,
>> + but if not, and we are using an image file, we will obtain the size from it.*/
>
> grammar:
> The size for the image must be known, either by direct specification, or
> by reading it from a backing file or image file.
>
> --
> Eric Blake eblake@redhat.com +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
>
Nod, Thank you, Eric
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 2/6 v11 v11] block: make some functions public
2012-08-01 14:01 ` Stefan Hajnoczi
@ 2012-08-02 7:11 ` Dong Xu Wang
0 siblings, 0 replies; 30+ messages in thread
From: Dong Xu Wang @ 2012-08-02 7:11 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: kwolf, qemu-devel
On Wed, Aug 1, 2012 at 10:01 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Tue, Jul 31, 2012 at 5:51 PM, Dong Xu Wang
> <wdongxu@linux.vnet.ibm.com> wrote:
>> diff --git a/block.h b/block.h
>> index c89590d..b523076 100644
>> --- a/block.h
>> +++ b/block.h
>> @@ -152,6 +152,8 @@ int bdrv_pwrite(BlockDriverState *bs, int64_t offset,
>> const void *buf, int count);
>> int bdrv_pwrite_sync(BlockDriverState *bs, int64_t offset,
>> const void *buf, int count);
>> +int bdrv_read_string(BlockDriverState *file, uint64_t offset, size_t n,
>> + char *buf, size_t buflen);
>
> I suggest renaming the "file" argument to "bs" like all the other
> block.h functions.
>
Okay.
>> /**
>> - * Read a string of known length from the image file
>> - *
>> - * @file: Image file
>> - * @offset: File offset to start of string, in bytes
>> - * @n: String length in bytes
>> - * @buf: Destination buffer
>> - * @buflen: Destination buffer length in bytes
>> - * @ret: 0 on success, -errno on failure
>> - *
>> - * The string is NUL-terminated.
>> - */
>
> Please keep the doc comment.
>
> Stefan
Okay. thank you.
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 3/6] add-cow file format
2012-08-01 13:57 ` Eric Blake
2012-08-01 14:14 ` Stefan Hajnoczi
@ 2012-08-02 7:20 ` Dong Xu Wang
1 sibling, 0 replies; 30+ messages in thread
From: Dong Xu Wang @ 2012-08-02 7:20 UTC (permalink / raw)
To: Eric Blake; +Cc: kwolf, qemu-devel
On Wed, Aug 1, 2012 at 9:57 PM, Eric Blake <eblake@redhat.com> wrote:
> On 07/31/2012 10:51 AM, Dong Xu Wang wrote:
>> This is the implementation code for add-cow file format. Because image_file
>> might be very huge, then we can't read entire bitmap into memory, we must use
>> a cache. Since qcow-cache.c has implemted cache code, we can create our cache
>
> s/implemted/implemented/
>
Okay.
>> code based on it.
>
> Just wondering if Paolo's HBitmap code for drive-mirror might be a more
> efficient way to implement your caching.
>
>>
>> Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
>> ---
>> block/Makefile.objs | 1 +
>> block/add-cow-cache.c | 206 +++++++++++++++++
>> block/add-cow.c | 599 +++++++++++++++++++++++++++++++++++++++++++++++++
>> block/add-cow.h | 101 +++++++++
>> block_int.h | 2 +
>
> Rather than adding a new implementation for code duplication, can you
> refactor the existing implementation to be reusable, and update
> qcow-cache.c to call into the common refactored code?
>
Okay. thanks.
> --
> Eric Blake eblake@redhat.com +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 3/6] add-cow file format
2012-08-01 15:31 ` Stefan Hajnoczi
@ 2012-08-02 7:20 ` Dong Xu Wang
0 siblings, 0 replies; 30+ messages in thread
From: Dong Xu Wang @ 2012-08-02 7:20 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: kwolf, qemu-devel
On Wed, Aug 1, 2012 at 11:31 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Tue, Jul 31, 2012 at 5:51 PM, Dong Xu Wang
> <wdongxu@linux.vnet.ibm.com> wrote:
>> + if (backing_filename) {
>> + header.features |= ADD_COW_F_BACKING_FILE;
>> + header.backing_filename_offset = sizeof(header) + 16 * 2;
>
> It's not obvious what 16 * 2 is.
Okay we give meaningful constant later.
>
>> + ret = bdrv_pwrite(bs, sizeof(le_header) + 16,
>> + image_format ? image_format : "raw",
>> + image_format ? sizeof(image_format) : sizeof("raw"));
>
> sizeof(image_format)? I think this should be strlen(image_format).
Okay. thanks.
>
> Stefan
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 4/6 v11] add-cow: support snapshot_blkde
2012-08-01 15:37 ` Stefan Hajnoczi
@ 2012-08-02 7:28 ` Dong Xu Wang
2012-08-02 10:37 ` Stefan Hajnoczi
0 siblings, 1 reply; 30+ messages in thread
From: Dong Xu Wang @ 2012-08-02 7:28 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: kwolf, qemu-devel
On Wed, Aug 1, 2012 at 11:37 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Tue, Jul 31, 2012 at 5:51 PM, Dong Xu Wang
> <wdongxu@linux.vnet.ibm.com> wrote:
>> add-cow will let raw file support snapshot_blkdev indirectly.
>>
>> Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
>> ---
>> blockdev.c | 45 +++++++++++++++++++++++++++++++++++++--------
>> docs/live-block-ops.txt | 11 ++++++++++-
>> 2 files changed, 47 insertions(+), 9 deletions(-)
>
> This patch would need to update qapi-schema.json and qmp-commands.hx
> to make use of the new arguments.
>
> I don't think QEMU compiles cleanly after this patch. It's important
> to keep the build clean after every patch so that git-bisect(1) can be
> used (if you hit a broken build during a bisect it makes things
> harder).
>
> However, I think the QMP/HMP changes should not be included in this
> series unless you are sure the interface is stable. When you convert
> QEMUOptionsParameter will transaction or snapshot-blkdev-sync change?
> If so, please only send the add-cow image format and leave the QMP/HMP
> changes until you have the QemuOpts solution.
Ah, yes, sorry, the patch must be compiled with 5/6, HMP now works like:
[-n] device [new-image-file] [format] [image-file] [image-format]
snapshot_blockdev /dev/ide-id0 1.add-cow add-cow t.raw raw
Also qmp's args_type:
.args_type = "device:B,snapshot-file:s,format:s?,mode:s?,image-file:s?,image-format:s?",
Before converting QEMUOptionsParameter, can the code be like this?
Thanks.
>
> Stefan
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 4/6 v11] add-cow: support snapshot_blkde
2012-08-02 7:28 ` Dong Xu Wang
@ 2012-08-02 10:37 ` Stefan Hajnoczi
0 siblings, 0 replies; 30+ messages in thread
From: Stefan Hajnoczi @ 2012-08-02 10:37 UTC (permalink / raw)
To: Dong Xu Wang; +Cc: kwolf, qemu-devel
On Thu, Aug 2, 2012 at 8:28 AM, Dong Xu Wang <wdongxu@linux.vnet.ibm.com> wrote:
> On Wed, Aug 1, 2012 at 11:37 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Tue, Jul 31, 2012 at 5:51 PM, Dong Xu Wang
>> <wdongxu@linux.vnet.ibm.com> wrote:
>>> add-cow will let raw file support snapshot_blkdev indirectly.
>>>
>>> Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
>>> ---
>>> blockdev.c | 45 +++++++++++++++++++++++++++++++++++++--------
>>> docs/live-block-ops.txt | 11 ++++++++++-
>>> 2 files changed, 47 insertions(+), 9 deletions(-)
>>
>> This patch would need to update qapi-schema.json and qmp-commands.hx
>> to make use of the new arguments.
>>
>> I don't think QEMU compiles cleanly after this patch. It's important
>> to keep the build clean after every patch so that git-bisect(1) can be
>> used (if you hit a broken build during a bisect it makes things
>> harder).
>>
>> However, I think the QMP/HMP changes should not be included in this
>> series unless you are sure the interface is stable. When you convert
>> QEMUOptionsParameter will transaction or snapshot-blkdev-sync change?
>> If so, please only send the add-cow image format and leave the QMP/HMP
>> changes until you have the QemuOpts solution.
>
> Ah, yes, sorry, the patch must be compiled with 5/6, HMP now works like:
> [-n] device [new-image-file] [format] [image-file] [image-format]
>
> snapshot_blockdev /dev/ide-id0 1.add-cow add-cow t.raw raw
>
> Also qmp's args_type:
> .args_type = "device:B,snapshot-file:s,format:s?,mode:s?,image-file:s?,image-format:s?",
>
> Before converting QEMUOptionsParameter, can the code be like this?
I think it's cleanest to submit just the image format without
snapshot_blockdev interface changes. It's definitely not a good idea
to temporarily change the QMP/HMP interface if the
QEMUOptionsParameter conversion will change it again.
Stefan
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format
2012-08-02 7:09 ` Dong Xu Wang
@ 2012-08-02 10:44 ` Stefan Hajnoczi
2012-08-03 5:56 ` Dong Xu Wang
0 siblings, 1 reply; 30+ messages in thread
From: Stefan Hajnoczi @ 2012-08-02 10:44 UTC (permalink / raw)
To: Dong Xu Wang; +Cc: kwolf, qemu-devel
On Thu, Aug 2, 2012 at 8:09 AM, Dong Xu Wang <wdongxu@linux.vnet.ibm.com> wrote:
> On Wed, Aug 1, 2012 at 9:55 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Tue, Jul 31, 2012 at 5:51 PM, Dong Xu Wang
>> <wdongxu@linux.vnet.ibm.com> wrote:
>>> +test.raw may be larger than ubuntu.img, in that case, the size of test.add-cow
>>> +will be calculated by the size of ubuntu.img, test.raw will be used from the
>>> +1st byte, the rest part can be used for other purpose.
>>
>> This is not how backing files normally work. With qcow2 or qed a
>> smaller backing file just means that the guest reads zeroes from the
>> areas beyond the end of the backing file. Is there a special reason
>> why you want to implement the behavior you described in the spec?
>> Otherwise I suggest implementing the same behavior as qcow2/qed.
>
> The size of add-cow will be caclulated by the size of backing_file, if
> backing_file can be smaller, how can I get the size of add-cow while
> being created?
>
> Do you mean like following steps?
> 1) qemu-img create -f qcow2 source.qcow2 *8G*
> 2) qemu-img create -f t.add-cow -o backing_file=source.qcow2,image_file=t *10G*
> And then reading un-allocated bytes from add-cow after 8G will be 0?
Yes. You could also get the virtual disk size from the size of the
image_file during creation.
Stefan
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format
2012-08-02 10:44 ` Stefan Hajnoczi
@ 2012-08-03 5:56 ` Dong Xu Wang
2012-08-03 8:26 ` Stefan Hajnoczi
0 siblings, 1 reply; 30+ messages in thread
From: Dong Xu Wang @ 2012-08-03 5:56 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: kwolf, qemu-devel
On Thu, Aug 2, 2012 at 6:44 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Thu, Aug 2, 2012 at 8:09 AM, Dong Xu Wang <wdongxu@linux.vnet.ibm.com> wrote:
>> On Wed, Aug 1, 2012 at 9:55 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>> On Tue, Jul 31, 2012 at 5:51 PM, Dong Xu Wang
>>> <wdongxu@linux.vnet.ibm.com> wrote:
>>>> +test.raw may be larger than ubuntu.img, in that case, the size of test.add-cow
>>>> +will be calculated by the size of ubuntu.img, test.raw will be used from the
>>>> +1st byte, the rest part can be used for other purpose.
>>>
>>> This is not how backing files normally work. With qcow2 or qed a
>>> smaller backing file just means that the guest reads zeroes from the
>>> areas beyond the end of the backing file. Is there a special reason
>>> why you want to implement the behavior you described in the spec?
>>> Otherwise I suggest implementing the same behavior as qcow2/qed.
>>
>> The size of add-cow will be caclulated by the size of backing_file, if
>> backing_file can be smaller, how can I get the size of add-cow while
>> being created?
>>
>> Do you mean like following steps?
>> 1) qemu-img create -f qcow2 source.qcow2 *8G*
>> 2) qemu-img create -f t.add-cow -o backing_file=source.qcow2,image_file=t *10G*
>> And then reading un-allocated bytes from add-cow after 8G will be 0?
>
> Yes. You could also get the virtual disk size from the size of the
> image_file during creation.
Eric said I shoud make sure we also support a raw file larger than
the backing file. so
add-cow will have the same virtual size with image_file, and their
virtual size can be larger
than backing_file's.
Am I right?
Thanks.
> Stefan
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format
2012-08-03 5:56 ` Dong Xu Wang
@ 2012-08-03 8:26 ` Stefan Hajnoczi
2012-08-06 2:05 ` Dong Xu Wang
0 siblings, 1 reply; 30+ messages in thread
From: Stefan Hajnoczi @ 2012-08-03 8:26 UTC (permalink / raw)
To: Dong Xu Wang; +Cc: kwolf, qemu-devel
On Fri, Aug 3, 2012 at 6:56 AM, Dong Xu Wang <wdongxu@linux.vnet.ibm.com> wrote:
> On Thu, Aug 2, 2012 at 6:44 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Thu, Aug 2, 2012 at 8:09 AM, Dong Xu Wang <wdongxu@linux.vnet.ibm.com> wrote:
>>> On Wed, Aug 1, 2012 at 9:55 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>>> On Tue, Jul 31, 2012 at 5:51 PM, Dong Xu Wang
>>>> <wdongxu@linux.vnet.ibm.com> wrote:
>>>>> +test.raw may be larger than ubuntu.img, in that case, the size of test.add-cow
>>>>> +will be calculated by the size of ubuntu.img, test.raw will be used from the
>>>>> +1st byte, the rest part can be used for other purpose.
>>>>
>>>> This is not how backing files normally work. With qcow2 or qed a
>>>> smaller backing file just means that the guest reads zeroes from the
>>>> areas beyond the end of the backing file. Is there a special reason
>>>> why you want to implement the behavior you described in the spec?
>>>> Otherwise I suggest implementing the same behavior as qcow2/qed.
>>>
>>> The size of add-cow will be caclulated by the size of backing_file, if
>>> backing_file can be smaller, how can I get the size of add-cow while
>>> being created?
>>>
>>> Do you mean like following steps?
>>> 1) qemu-img create -f qcow2 source.qcow2 *8G*
>>> 2) qemu-img create -f t.add-cow -o backing_file=source.qcow2,image_file=t *10G*
>>> And then reading un-allocated bytes from add-cow after 8G will be 0?
>>
>> Yes. You could also get the virtual disk size from the size of the
>> image_file during creation.
>
> Eric said I shoud make sure we also support a raw file larger than
> the backing file. so
> add-cow will have the same virtual size with image_file, and their
> virtual size can be larger
> than backing_file's.
Yes. Please update the spec because it says the opposite:
"test.raw may be larger than ubuntu.img, in that case, the size of
test.add-cow will be calculated by the size of ubuntu.img"
It should use the size of test.raw.
Stefan
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format
2012-08-03 8:26 ` Stefan Hajnoczi
@ 2012-08-06 2:05 ` Dong Xu Wang
0 siblings, 0 replies; 30+ messages in thread
From: Dong Xu Wang @ 2012-08-06 2:05 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: kwolf, qemu-devel
On Fri, Aug 3, 2012 at 4:26 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Fri, Aug 3, 2012 at 6:56 AM, Dong Xu Wang <wdongxu@linux.vnet.ibm.com> wrote:
>> On Thu, Aug 2, 2012 at 6:44 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>> On Thu, Aug 2, 2012 at 8:09 AM, Dong Xu Wang <wdongxu@linux.vnet.ibm.com> wrote:
>>>> On Wed, Aug 1, 2012 at 9:55 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>>>> On Tue, Jul 31, 2012 at 5:51 PM, Dong Xu Wang
>>>>> <wdongxu@linux.vnet.ibm.com> wrote:
>>>>>> +test.raw may be larger than ubuntu.img, in that case, the size of test.add-cow
>>>>>> +will be calculated by the size of ubuntu.img, test.raw will be used from the
>>>>>> +1st byte, the rest part can be used for other purpose.
>>>>>
>>>>> This is not how backing files normally work. With qcow2 or qed a
>>>>> smaller backing file just means that the guest reads zeroes from the
>>>>> areas beyond the end of the backing file. Is there a special reason
>>>>> why you want to implement the behavior you described in the spec?
>>>>> Otherwise I suggest implementing the same behavior as qcow2/qed.
>>>>
>>>> The size of add-cow will be caclulated by the size of backing_file, if
>>>> backing_file can be smaller, how can I get the size of add-cow while
>>>> being created?
>>>>
>>>> Do you mean like following steps?
>>>> 1) qemu-img create -f qcow2 source.qcow2 *8G*
>>>> 2) qemu-img create -f t.add-cow -o backing_file=source.qcow2,image_file=t *10G*
>>>> And then reading un-allocated bytes from add-cow after 8G will be 0?
>>>
>>> Yes. You could also get the virtual disk size from the size of the
>>> image_file during creation.
>>
>> Eric said I shoud make sure we also support a raw file larger than
>> the backing file. so
>> add-cow will have the same virtual size with image_file, and their
>> virtual size can be larger
>> than backing_file's.
>
> Yes. Please update the spec because it says the opposite:
>
> "test.raw may be larger than ubuntu.img, in that case, the size of
> test.add-cow will be calculated by the size of ubuntu.img"
Okay. Thanks.
>
> It should use the size of test.raw.
>
> Stefan
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 3/6] add-cow file format
2012-06-14 11:13 ` Paolo Bonzini
@ 2012-06-18 2:08 ` Dong Xu Wang
0 siblings, 0 replies; 30+ messages in thread
From: Dong Xu Wang @ 2012-06-18 2:08 UTC (permalink / raw)
To: Paolo Bonzini; +Cc: kwolf, qemu-devel
On Thu, Jun 14, 2012 at 7:13 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> I just took a quick look at the flush code.
>
> Il 13/06/2012 16:36, Dong Xu Wang ha scritto:
>>
>> +bool add_cow_cache_set_writethrough(BlockDriverState *bs, AddCowCache *c,
>> + bool enable)
>> +{
>> + bool old = c->writethrough;
>> +
>> + if (!old && enable) {
>> + add_cow_cache_flush(bs, c);
>> + }
>> +
>> + c->writethrough = enable;
>> + return old;
>> +}
>
> Writethrough mode should not be needed anymore in 1.2 if the new
> implementation of writethrough is added.
>
> And anyway...
>
>> + if (changed) {
>> + ret = add_cow_cache_flush(bs, s->bitmap_cache);
>> + }
>
> ... here you're treating the cache essentially as writethrough. Is this
> flush necessary?
>
>> + qemu_co_mutex_unlock(&s->lock);
>> + qemu_iovec_destroy(&hd_qiov);
>> + return ret;
>> +}
>> +
>> +static int bdrv_add_cow_truncate(BlockDriverState *bs, int64_t size)
>> +{
>> + BDRVAddCowState *s = bs->opaque;
>> + int sector_per_byte = SECTORS_PER_CLUSTER * 8;
>> + int ret;
>> + int64_t bitmap_size =
>> + (size / BDRV_SECTOR_SIZE + sector_per_byte - 1) / sector_per_byte;
>> +
>> + ret = bdrv_truncate(bs->file,
>> + ADD_COW_BITMAP_POS + bitmap_size);
>> + if (ret < 0) {
>> + return ret;
>> + }
>> + bdrv_truncate(s->image_hd, size);
>> + return 0;
>> +}
>> +
>> +static coroutine_fn int add_cow_co_flush(BlockDriverState *bs)
>> +{
>> + BDRVAddCowState *s = bs->opaque;
>> + int ret;
>> +
>> + qemu_co_mutex_lock(&s->lock);
>> + ret = add_cow_cache_flush(bs, s->bitmap_cache);
>> + qemu_co_mutex_unlock(&s->lock);
>> + if (ret < 0) {
>> + return ret;
>> + }
>> +
>> + return bdrv_co_flush(bs->file);
>
> Flushing bs->file here is not needed anymore...
>
>> +}
>> +
>> +static QEMUOptionParameter add_cow_create_options[] = {
>> + {
>> + .name = BLOCK_OPT_SIZE,
>> + .type = OPT_SIZE,
>> + .help = "Virtual disk size"
>> + },
>> + {
>> + .name = BLOCK_OPT_BACKING_FILE,
>> + .type = OPT_STRING,
>> + .help = "File name of a base image"
>> + },
>> + {
>> + .name = BLOCK_OPT_IMAGE_FILE,
>> + .type = OPT_STRING,
>> + .help = "File name of a image file"
>> + },
>> + {
>> + .name = BLOCK_OPT_BACKING_FMT,
>> + .type = OPT_STRING,
>> + .help = "Image format of the base image"
>> + },
>> + { NULL }
>> +};
>> +
>> +static BlockDriver bdrv_add_cow = {
>> + .format_name = "add-cow",
>> + .instance_size = sizeof(BDRVAddCowState),
>> + .bdrv_probe = add_cow_probe,
>> + .bdrv_open = add_cow_open,
>> + .bdrv_close = add_cow_close,
>> + .bdrv_create = add_cow_create,
>> + .bdrv_co_readv = add_cow_co_readv,
>> + .bdrv_co_writev = add_cow_co_writev,
>> + .bdrv_truncate = bdrv_add_cow_truncate,
>> + .bdrv_co_is_allocated = add_cow_is_allocated,
>> +
>> + .create_options = add_cow_create_options,
>> + .bdrv_co_flush_to_disk = add_cow_co_flush,
>
> ... and this should be bdrv_co_flush_to_os.
>
> Paolo
>
Okay, thanks for your review, Paolo. :)
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [Qemu-devel] [PATCH 3/6] add-cow file format
2012-06-13 14:36 ` [Qemu-devel] [PATCH 3/6] " Dong Xu Wang
@ 2012-06-14 11:13 ` Paolo Bonzini
2012-06-18 2:08 ` Dong Xu Wang
0 siblings, 1 reply; 30+ messages in thread
From: Paolo Bonzini @ 2012-06-14 11:13 UTC (permalink / raw)
To: Dong Xu Wang; +Cc: kwolf, qemu-devel
I just took a quick look at the flush code.
Il 13/06/2012 16:36, Dong Xu Wang ha scritto:
>
> +bool add_cow_cache_set_writethrough(BlockDriverState *bs, AddCowCache *c,
> + bool enable)
> +{
> + bool old = c->writethrough;
> +
> + if (!old && enable) {
> + add_cow_cache_flush(bs, c);
> + }
> +
> + c->writethrough = enable;
> + return old;
> +}
Writethrough mode should not be needed anymore in 1.2 if the new
implementation of writethrough is added.
And anyway...
> + if (changed) {
> + ret = add_cow_cache_flush(bs, s->bitmap_cache);
> + }
... here you're treating the cache essentially as writethrough. Is this
flush necessary?
> + qemu_co_mutex_unlock(&s->lock);
> + qemu_iovec_destroy(&hd_qiov);
> + return ret;
> +}
> +
> +static int bdrv_add_cow_truncate(BlockDriverState *bs, int64_t size)
> +{
> + BDRVAddCowState *s = bs->opaque;
> + int sector_per_byte = SECTORS_PER_CLUSTER * 8;
> + int ret;
> + int64_t bitmap_size =
> + (size / BDRV_SECTOR_SIZE + sector_per_byte - 1) / sector_per_byte;
> +
> + ret = bdrv_truncate(bs->file,
> + ADD_COW_BITMAP_POS + bitmap_size);
> + if (ret < 0) {
> + return ret;
> + }
> + bdrv_truncate(s->image_hd, size);
> + return 0;
> +}
> +
> +static coroutine_fn int add_cow_co_flush(BlockDriverState *bs)
> +{
> + BDRVAddCowState *s = bs->opaque;
> + int ret;
> +
> + qemu_co_mutex_lock(&s->lock);
> + ret = add_cow_cache_flush(bs, s->bitmap_cache);
> + qemu_co_mutex_unlock(&s->lock);
> + if (ret < 0) {
> + return ret;
> + }
> +
> + return bdrv_co_flush(bs->file);
Flushing bs->file here is not needed anymore...
> +}
> +
> +static QEMUOptionParameter add_cow_create_options[] = {
> + {
> + .name = BLOCK_OPT_SIZE,
> + .type = OPT_SIZE,
> + .help = "Virtual disk size"
> + },
> + {
> + .name = BLOCK_OPT_BACKING_FILE,
> + .type = OPT_STRING,
> + .help = "File name of a base image"
> + },
> + {
> + .name = BLOCK_OPT_IMAGE_FILE,
> + .type = OPT_STRING,
> + .help = "File name of a image file"
> + },
> + {
> + .name = BLOCK_OPT_BACKING_FMT,
> + .type = OPT_STRING,
> + .help = "Image format of the base image"
> + },
> + { NULL }
> +};
> +
> +static BlockDriver bdrv_add_cow = {
> + .format_name = "add-cow",
> + .instance_size = sizeof(BDRVAddCowState),
> + .bdrv_probe = add_cow_probe,
> + .bdrv_open = add_cow_open,
> + .bdrv_close = add_cow_close,
> + .bdrv_create = add_cow_create,
> + .bdrv_co_readv = add_cow_co_readv,
> + .bdrv_co_writev = add_cow_co_writev,
> + .bdrv_truncate = bdrv_add_cow_truncate,
> + .bdrv_co_is_allocated = add_cow_is_allocated,
> +
> + .create_options = add_cow_create_options,
> + .bdrv_co_flush_to_disk = add_cow_co_flush,
... and this should be bdrv_co_flush_to_os.
Paolo
^ permalink raw reply [flat|nested] 30+ messages in thread
* [Qemu-devel] [PATCH 3/6] add-cow file format
2012-06-13 14:36 [Qemu-devel] [PATCH 1/6 v10] " Dong Xu Wang
@ 2012-06-13 14:36 ` Dong Xu Wang
2012-06-14 11:13 ` Paolo Bonzini
0 siblings, 1 reply; 30+ messages in thread
From: Dong Xu Wang @ 2012-06-13 14:36 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, Dong Xu Wang
This is the implementation code for add-cow file format. Because image_file
might be very huge, then we can't read entire bitmap into memory, we must use
a cache. Since qcow-cache.c has implemted cache code, we can create our cache
code based on it.
Signed-off-by: Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
---
block/Makefile.objs | 1 +
block/add-cow-cache.c | 218 ++++++++++++++++++++++
block/add-cow.c | 488 +++++++++++++++++++++++++++++++++++++++++++++++++
block/add-cow.h | 95 ++++++++++
block_int.h | 1 +
5 files changed, 803 insertions(+), 0 deletions(-)
create mode 100644 block/add-cow-cache.c
create mode 100644 block/add-cow.c
create mode 100644 block/add-cow.h
diff --git a/block/Makefile.objs b/block/Makefile.objs
index b5754d3..357a3b1 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -2,6 +2,7 @@ block-obj-y += raw.o cow.o qcow.o vdi.o vmdk.o cloop.o dmg.o bochs.o vpc.o vvfat
block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o qcow2-cache.o
block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o
block-obj-y += qed-check.o
+block-obj-y += add-cow.o add-cow-cache.o
block-obj-y += parallels.o nbd.o blkdebug.o sheepdog.o blkverify.o
block-obj-y += stream.o
block-obj-$(CONFIG_WIN32) += raw-win32.o
diff --git a/block/add-cow-cache.c b/block/add-cow-cache.c
new file mode 100644
index 0000000..b4b1546
--- /dev/null
+++ b/block/add-cow-cache.c
@@ -0,0 +1,218 @@
+/*
+ * Cache For QEMU ADD-COW Disk Format
+ *
+ * Copyright IBM, Corp. 2012
+ *
+ * Authors:
+ * Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
+ *
+ * This file is based on qcow2-cache.c, see its copyrights below:
+ *
+ * L2/refcount table cache for the QCOW2 format
+ *
+ * Copyright (c) 2010 Kevin Wolf <kwolf@redhat.com>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include "block_int.h"
+#include "qemu-common.h"
+#include "add-cow.h"
+
+AddCowCache *add_cow_cache_create(BlockDriverState *bs, int num_tables,
+ bool writethrough)
+{
+ AddCowCache *c;
+ int i;
+
+ c = g_malloc0(sizeof(*c));
+ c->size = num_tables;
+ c->entries = g_malloc0(sizeof(*c->entries) * num_tables);
+ c->writethrough = writethrough;
+ c->entry_size = ADD_COW_CACHE_ENTRY_SIZE;
+
+ for (i = 0; i < c->size; i++) {
+ c->entries[i].table = qemu_blockalign(bs, c->entry_size);
+ c->entries[i].offset = -1;
+ }
+
+ return c;
+}
+
+int add_cow_cache_destroy(BlockDriverState *bs, AddCowCache *c)
+{
+ int i;
+
+ for (i = 0; i < c->size; i++) {
+ qemu_vfree(c->entries[i].table);
+ }
+
+ g_free(c->entries);
+ g_free(c);
+
+ return 0;
+}
+
+static int add_cow_cache_entry_flush(BlockDriverState *bs,
+ AddCowCache *c,
+ int i)
+{
+ BDRVAddCowState *s = bs->opaque;
+ int ret = 0;
+
+ if (!c->entries[i].dirty || -1 == c->entries[i].offset) {
+ return ret;
+ }
+
+ ret = bdrv_pwrite(bs->file,
+ ADD_COW_BITMAP_POS + c->entries[i].offset,
+ c->entries[i].table,
+ MIN(c->entry_size, s->bitmap_size - c->entries[i].offset));
+ if (ret < 0) {
+ return ret;
+ }
+
+ c->entries[i].dirty = false;
+
+ return 0;
+}
+
+int add_cow_cache_flush(BlockDriverState *bs, AddCowCache *c)
+{
+ BDRVAddCowState *s = bs->opaque;
+ int result = 0;
+ int ret;
+ int i;
+
+ ret = bdrv_flush(s->image_hd);
+ if (ret < 0) {
+ return result;
+ }
+
+ for (i = 0; i < c->size; i++) {
+ ret = add_cow_cache_entry_flush(bs, c, i);
+ if (ret < 0 && result != -ENOSPC) {
+ result = ret;
+ }
+ }
+
+ if (result == 0) {
+ ret = bdrv_flush(bs->file);
+ if (ret < 0) {
+ result = ret;
+ }
+ }
+
+ return result;
+}
+
+static int add_cow_cache_find_entry_to_replace(AddCowCache *c)
+{
+ int i;
+ int min_count = INT_MAX;
+ int min_index = -1;
+
+
+ for (i = 0; i < c->size; i++) {
+ if (c->entries[i].cache_hits < min_count) {
+ min_index = i;
+ min_count = c->entries[i].cache_hits;
+ }
+
+ c->entries[i].cache_hits /= 2;
+ }
+
+ if (min_index == -1) {
+ abort();
+ }
+ return min_index;
+}
+
+static int add_cow_cache_do_get(BlockDriverState *bs, AddCowCache *c,
+ uint64_t offset, void **table)
+{
+ int i, ret;
+
+ for (i = 0; i < c->size; i++) {
+ if (c->entries[i].offset == offset) {
+ goto found;
+ }
+ }
+
+ i = add_cow_cache_find_entry_to_replace(c);
+ if (i < 0) {
+ return i;
+ }
+
+ ret = add_cow_cache_entry_flush(bs, c, i);
+ if (ret < 0) {
+ return ret;
+ }
+ c->entries[i].offset = -1;
+ ret = bdrv_pread(bs->file, ADD_COW_BITMAP_POS + offset,
+ c->entries[i].table, c->entry_size);
+ if (ret < 0) {
+ return ret;
+ }
+
+ c->entries[i].cache_hits = 32;
+ c->entries[i].offset = offset;
+
+found:
+ c->entries[i].cache_hits++;
+ *table = c->entries[i].table;
+
+ return 0;
+}
+
+int add_cow_cache_get(BlockDriverState *bs, AddCowCache *c, uint64_t sector_num,
+ void **table)
+{
+ /* each byte in bitmap indicates 8 * SECTORS_PER_CLUSTER clusters */
+ uint64_t offset = offset_in_bitmap(sector_num) & (~(c->entry_size - 1));
+ return add_cow_cache_do_get(bs, c, offset, table);
+}
+
+void add_cow_cache_entry_mark_dirty(AddCowCache *c, void *table)
+{
+ int i;
+
+ for (i = 0; i < c->size; i++) {
+ if (c->entries[i].table == table) {
+ goto found;
+ }
+ }
+ abort();
+
+found:
+ c->entries[i].dirty = true;
+}
+
+bool add_cow_cache_set_writethrough(BlockDriverState *bs, AddCowCache *c,
+ bool enable)
+{
+ bool old = c->writethrough;
+
+ if (!old && enable) {
+ add_cow_cache_flush(bs, c);
+ }
+
+ c->writethrough = enable;
+ return old;
+}
diff --git a/block/add-cow.c b/block/add-cow.c
new file mode 100644
index 0000000..f8bc299
--- /dev/null
+++ b/block/add-cow.c
@@ -0,0 +1,488 @@
+/*
+ * QEMU ADD-COW Disk Format
+ *
+ * Copyright IBM, Corp. 2012
+ *
+ * Authors:
+ * Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "block_int.h"
+#include "module.h"
+#include "add-cow.h"
+
+static int add_cow_probe(const uint8_t *buf, int buf_size, const char *filename)
+{
+ const AddCowHeader *header = (const AddCowHeader *)buf;
+
+ if (le64_to_cpu(header->magic) == ADD_COW_MAGIC &&
+ le32_to_cpu(header->version) == ADD_COW_VERSION) {
+ return 100;
+ } else {
+ return 0;
+ }
+}
+
+static int add_cow_create(const char *filename, QEMUOptionParameter *options)
+{
+ AddCowHeader header;
+ int64_t image_len = 0;
+ const char *backing_filename = NULL;
+ const char *backing_fmt = NULL;
+ const char *image_filename = NULL;
+ int ret;
+ BlockDriverState *bs, *image_bs = NULL;
+ BlockDriver *drv = bdrv_find_format("add-cow");
+
+ while (options && options->name) {
+ if (!strcmp(options->name, BLOCK_OPT_SIZE)) {
+ image_len = options->value.n;
+ } else if (!strcmp(options->name, BLOCK_OPT_BACKING_FILE)) {
+ backing_filename = options->value.s;
+ } else if (!strcmp(options->name, BLOCK_OPT_BACKING_FMT)) {
+ backing_fmt = options->value.s;
+ } else if (!strcmp(options->name, BLOCK_OPT_IMAGE_FILE)) {
+ image_filename = options->value.s;
+ }
+ options++;
+ }
+ memset(&header, 0, sizeof(header));
+ header.magic = cpu_to_le64(ADD_COW_MAGIC);
+ header.version = cpu_to_le32(ADD_COW_VERSION);
+ header.features = cpu_to_le64(ADD_COW_F_BACKING_FILE);
+
+ if (backing_filename) {
+ header.features |= ADD_COW_F_BACKING_FILE;
+ header.backing_filename_offset = sizeof(header);
+ header.backing_filename_size = strlen(backing_filename);
+
+ if (backing_fmt && strcmp(backing_fmt, "raw") == 0) {
+ header.features |= ADD_COW_F_BACKING_FORMAT_NO_PROBE;
+ pstrcpy(bs->backing_format, sizeof(bs->backing_format), "raw");
+ }
+ } else {
+ error_report("backing_file should be given.");
+ return -EINVAL;
+ }
+
+ if (image_filename) {
+ header.image_filename_offset =
+ sizeof(header) + header.backing_filename_size;
+ header.image_filename_size = strlen(image_filename);
+ } else {
+ error_report("image_file should be given.");
+ return -EINVAL;
+ }
+
+ if (header.image_filename_offset + header.image_filename_size > ADD_COW_BITMAP_POS) {
+ error_report("image_file name or backing_file name too long.");
+ return -ENOSPC;
+ }
+
+ ret = bdrv_file_open(&image_bs, image_filename, BDRV_O_RDWR
+ | BDRV_O_CACHE_WB);
+ if (ret < 0) {
+ return ret;
+ }
+ bdrv_delete(image_bs);
+
+ ret = bdrv_create_file(filename, NULL);
+ if (ret < 0) {
+ return ret;
+ }
+
+ ret = bdrv_file_open(&bs, filename, BDRV_O_RDWR);
+ if (ret < 0) {
+ return ret;
+ }
+
+ ret = bdrv_pwrite(bs, 0, &header, sizeof(header));
+ if (ret < 0) {
+ bdrv_delete(bs);
+ return ret;
+ }
+
+ ret = bdrv_pwrite(bs,
+ header.backing_filename_offset,
+ backing_filename,
+ header.backing_filename_size);
+ if (ret < 0) {
+ bdrv_delete(bs);
+ return ret;
+ }
+
+ ret = bdrv_pwrite(bs,
+ header.image_filename_offset,
+ image_filename,
+ header.image_filename_size);
+ if (ret < 0) {
+ bdrv_delete(bs);
+ return ret;
+ }
+
+ ret = bdrv_open(bs, filename, BDRV_O_RDWR | BDRV_O_NO_FLUSH, drv);
+ if (ret < 0) {
+ bdrv_delete(bs);
+ return ret;
+ }
+
+ ret = bdrv_truncate(bs, image_len);
+ bdrv_delete(bs);
+ return ret;
+}
+
+static int add_cow_open(BlockDriverState *bs, int flags)
+{
+ char image_filename[ADD_COW_FILE_LEN];
+ char tmp_name[ADD_COW_FILE_LEN];
+ BlockDriver *image_drv = NULL;
+ int ret;
+ bool writethrough;
+ int sector_per_byte;
+ BDRVAddCowState *s = bs->opaque;
+ uint64_t features;
+
+ ret = bdrv_pread(bs->file, 0, &s->header, sizeof(s->header));
+ if (ret != sizeof(s->header)) {
+ goto fail;
+ }
+
+ if (le64_to_cpu(s->header.magic) != ADD_COW_MAGIC) {
+ ret = -EINVAL;
+ goto fail;
+ }
+ if (le32_to_cpu(s->header.version) != ADD_COW_VERSION) {
+ char version[64];
+ snprintf(version, sizeof(version), "ADD-COW version %d",
+ le32_to_cpu(s->header.version));
+ qerror_report(QERR_UNKNOWN_BLOCK_FORMAT_FEATURE,
+ bs->device_name, "add-cow", version);
+ ret = -ENOTSUP;
+ goto fail;
+ }
+ features = le64_to_cpu(s->header.features);
+ if (features & ~ADD_COW_FEATURE_MASK) {
+ char buf[64];
+ snprintf(buf, sizeof(buf), "%" PRIx64,
+ s->header.features & ~ADD_COW_FEATURE_MASK);
+ qerror_report(QERR_UNKNOWN_BLOCK_FORMAT_FEATURE,
+ bs->device_name, "add-cow", buf);
+ return -ENOTSUP;
+ }
+ if ((features & ADD_COW_F_BACKING_FILE) == 0) {
+ /* Now, only with backing_file is supported */
+ ret = -ENOTSUP;
+ goto fail;
+ }
+ ret = bdrv_read_string(bs->file, s->header.backing_filename_offset,
+ s->header.backing_filename_size, bs->backing_file,
+ sizeof(bs->backing_file));
+ if (ret < 0) {
+ goto fail;
+ }
+ ret = bdrv_read_string(bs->file, s->header.image_filename_offset,
+ s->header.image_filename_size, tmp_name,
+ sizeof(tmp_name));
+ if (ret < 0) {
+ goto fail;
+ }
+ s->image_hd = bdrv_new("");
+ if (path_has_protocol(image_filename)) {
+ pstrcpy(image_filename, sizeof(image_filename), tmp_name);
+ } else {
+ path_combine(image_filename, sizeof(image_filename),
+ bs->filename, tmp_name);
+ }
+ ret = bdrv_open(s->image_hd, image_filename, flags, image_drv);
+ if (ret < 0) {
+ bdrv_delete(s->image_hd);
+ goto fail;
+ }
+ bs->total_sectors = bdrv_getlength(s->image_hd) >> 9;
+ s->cluster_size = ADD_COW_CLUSTER_SIZE;
+ sector_per_byte = SECTORS_PER_CLUSTER * 8;
+ s->bitmap_size =
+ (bs->total_sectors + sector_per_byte - 1) / sector_per_byte;
+ writethrough = ((flags & BDRV_O_CACHE_WB) == 0);
+ s->bitmap_cache =
+ add_cow_cache_create(bs, ADD_COW_CACHE_SIZE, writethrough);
+
+ qemu_co_mutex_init(&s->lock);
+ return 0;
+fail:
+ if (s->bitmap_cache) {
+ add_cow_cache_destroy(bs, s->bitmap_cache);
+ }
+ return ret;
+}
+
+static void add_cow_close(BlockDriverState *bs)
+{
+ BDRVAddCowState *s = bs->opaque;
+ add_cow_cache_destroy(bs, s->bitmap_cache);
+ bdrv_delete(s->image_hd);
+}
+
+static bool is_allocated(BlockDriverState *bs, int64_t sector_num)
+{
+ BDRVAddCowState *s = bs->opaque;
+ int64_t cluster_num = sector_num / SECTORS_PER_CLUSTER;
+ uint8_t *table = NULL;
+ int ret = add_cow_cache_get(bs, s->bitmap_cache,
+ sector_num, (void **)&table);
+
+ if (ret < 0) {
+ return ret;
+ }
+ return table[cluster_num / 8 % ADD_COW_CACHE_ENTRY_SIZE]
+ & (1 << (cluster_num % 8));
+}
+
+static coroutine_fn int add_cow_is_allocated(BlockDriverState *bs,
+ int64_t sector_num, int nb_sectors, int *num_same)
+{
+ int changed;
+
+ if (nb_sectors == 0) {
+ *num_same = 0;
+ return 0;
+ }
+ changed = is_allocated(bs, sector_num);
+
+ for (*num_same = 1; *num_same < nb_sectors; (*num_same)++) {
+ if (is_allocated(bs, sector_num + *num_same) != changed) {
+ break;
+ }
+ }
+ return changed;
+}
+
+static coroutine_fn int add_cow_co_readv(BlockDriverState *bs,
+ int64_t sector_num, int remaining_sectors, QEMUIOVector *qiov)
+{
+ BDRVAddCowState *s = bs->opaque;
+ int cur_nr_sectors;
+ uint64_t bytes_done = 0;
+ QEMUIOVector hd_qiov;
+ int n, ret = 0;
+
+ qemu_iovec_init(&hd_qiov, qiov->niov);
+ qemu_co_mutex_lock(&s->lock);
+ while (remaining_sectors != 0) {
+ cur_nr_sectors = remaining_sectors;
+ if (add_cow_is_allocated(bs, sector_num, cur_nr_sectors, &n)) {
+ cur_nr_sectors = n;
+ qemu_iovec_reset(&hd_qiov);
+ qemu_iovec_copy(&hd_qiov, qiov, bytes_done,
+ cur_nr_sectors * BDRV_SECTOR_SIZE);
+ qemu_co_mutex_unlock(&s->lock);
+ ret = bdrv_co_readv(s->image_hd, sector_num, n, &hd_qiov);
+ qemu_co_mutex_lock(&s->lock);
+ if (ret < 0) {
+ goto fail;
+ }
+ } else {
+ cur_nr_sectors = n;
+ if (bs->backing_hd) {
+ qemu_iovec_reset(&hd_qiov);
+ qemu_iovec_copy(&hd_qiov, qiov, bytes_done,
+ cur_nr_sectors * BDRV_SECTOR_SIZE);
+ qemu_co_mutex_unlock(&s->lock);
+ ret = bdrv_co_readv(bs->backing_hd, sector_num,
+ n, &hd_qiov);
+ qemu_co_mutex_lock(&s->lock);
+ if (ret < 0) {
+ goto fail;
+ }
+ } else {
+ qemu_iovec_memset(&hd_qiov, 0,
+ BDRV_SECTOR_SIZE * cur_nr_sectors);
+ }
+ }
+ remaining_sectors -= cur_nr_sectors;
+ sector_num += cur_nr_sectors;
+ bytes_done += cur_nr_sectors * BDRV_SECTOR_SIZE;
+ }
+fail:
+ qemu_co_mutex_unlock(&s->lock);
+ qemu_iovec_destroy(&hd_qiov);
+ return ret;
+}
+
+static int coroutine_fn copy_sectors(BlockDriverState *bs,
+ int n_start, int n_end)
+{
+ BDRVAddCowState *s = bs->opaque;
+ QEMUIOVector qiov;
+ struct iovec iov;
+ int n, ret;
+
+ n = n_end - n_start;
+ if (n <= 0) {
+ return 0;
+ }
+
+ iov.iov_len = n * BDRV_SECTOR_SIZE;
+ iov.iov_base = qemu_blockalign(bs, iov.iov_len);
+
+ qemu_iovec_init_external(&qiov, &iov, 1);
+
+ ret = bdrv_co_readv(bs->backing_hd, n_start, n, &qiov);
+ if (ret < 0) {
+ goto out;
+ }
+ ret = bdrv_co_writev(s->image_hd, n_start, n, &qiov);
+ if (ret < 0) {
+ goto out;
+ }
+
+ ret = 0;
+out:
+ qemu_vfree(iov.iov_base);
+ return ret;
+}
+
+static coroutine_fn int add_cow_co_writev(BlockDriverState *bs,
+ int64_t sector_num, int remaining_sectors, QEMUIOVector *qiov)
+{
+ BDRVAddCowState *s = bs->opaque;
+ int ret = 0, i;
+ QEMUIOVector hd_qiov;
+ uint8_t *table;
+ bool changed = false;
+
+ qemu_co_mutex_lock(&s->lock);
+ qemu_iovec_init(&hd_qiov, qiov->niov);
+ ret = bdrv_co_writev(s->image_hd,
+ sector_num,
+ remaining_sectors, qiov);
+
+ if (ret < 0) {
+ goto fail;
+ }
+ /* Copy content of unmodified sectors */
+ if (!is_cluster_head(sector_num) && !is_allocated(bs, sector_num)) {
+ ret = copy_sectors(bs, sector_num & ~(SECTORS_PER_CLUSTER - 1),
+ sector_num);
+ if (ret < 0) {
+ goto fail;
+ }
+ }
+
+ if (!is_cluster_tail(sector_num + remaining_sectors - 1)
+ && !is_allocated(bs, sector_num + remaining_sectors - 1)) {
+ ret = copy_sectors(bs, sector_num + remaining_sectors,
+ ((sector_num + remaining_sectors) | (SECTORS_PER_CLUSTER - 1)) + 1);
+ if (ret < 0) {
+ goto fail;
+ }
+ }
+
+ for (i = sector_num / SECTORS_PER_CLUSTER;
+ i <= (sector_num + remaining_sectors - 1) / SECTORS_PER_CLUSTER;
+ i++) {
+ ret = add_cow_cache_get(bs, s->bitmap_cache,
+ i * SECTORS_PER_CLUSTER, (void **)&table);
+ if (ret < 0) {
+ return ret;
+ }
+ if ((table[i / 8] & (1 << (i % 8))) == 0) {
+ table[i / 8] |= (1 << (i % 8));
+ changed = true;
+ add_cow_cache_entry_mark_dirty(s->bitmap_cache, table);
+ }
+
+ }
+ ret = 0;
+fail:
+ if (changed) {
+ ret = add_cow_cache_flush(bs, s->bitmap_cache);
+ }
+ qemu_co_mutex_unlock(&s->lock);
+ qemu_iovec_destroy(&hd_qiov);
+ return ret;
+}
+
+static int bdrv_add_cow_truncate(BlockDriverState *bs, int64_t size)
+{
+ BDRVAddCowState *s = bs->opaque;
+ int sector_per_byte = SECTORS_PER_CLUSTER * 8;
+ int ret;
+ int64_t bitmap_size =
+ (size / BDRV_SECTOR_SIZE + sector_per_byte - 1) / sector_per_byte;
+
+ ret = bdrv_truncate(bs->file,
+ ADD_COW_BITMAP_POS + bitmap_size);
+ if (ret < 0) {
+ return ret;
+ }
+ bdrv_truncate(s->image_hd, size);
+ return 0;
+}
+
+static coroutine_fn int add_cow_co_flush(BlockDriverState *bs)
+{
+ BDRVAddCowState *s = bs->opaque;
+ int ret;
+
+ qemu_co_mutex_lock(&s->lock);
+ ret = add_cow_cache_flush(bs, s->bitmap_cache);
+ qemu_co_mutex_unlock(&s->lock);
+ if (ret < 0) {
+ return ret;
+ }
+
+ return bdrv_co_flush(bs->file);
+}
+
+static QEMUOptionParameter add_cow_create_options[] = {
+ {
+ .name = BLOCK_OPT_SIZE,
+ .type = OPT_SIZE,
+ .help = "Virtual disk size"
+ },
+ {
+ .name = BLOCK_OPT_BACKING_FILE,
+ .type = OPT_STRING,
+ .help = "File name of a base image"
+ },
+ {
+ .name = BLOCK_OPT_IMAGE_FILE,
+ .type = OPT_STRING,
+ .help = "File name of a image file"
+ },
+ {
+ .name = BLOCK_OPT_BACKING_FMT,
+ .type = OPT_STRING,
+ .help = "Image format of the base image"
+ },
+ { NULL }
+};
+
+static BlockDriver bdrv_add_cow = {
+ .format_name = "add-cow",
+ .instance_size = sizeof(BDRVAddCowState),
+ .bdrv_probe = add_cow_probe,
+ .bdrv_open = add_cow_open,
+ .bdrv_close = add_cow_close,
+ .bdrv_create = add_cow_create,
+ .bdrv_co_readv = add_cow_co_readv,
+ .bdrv_co_writev = add_cow_co_writev,
+ .bdrv_truncate = bdrv_add_cow_truncate,
+ .bdrv_co_is_allocated = add_cow_is_allocated,
+
+ .create_options = add_cow_create_options,
+ .bdrv_co_flush_to_disk = add_cow_co_flush,
+};
+
+static void bdrv_add_cow_init(void)
+{
+ bdrv_register(&bdrv_add_cow);
+}
+
+block_init(bdrv_add_cow_init);
diff --git a/block/add-cow.h b/block/add-cow.h
new file mode 100644
index 0000000..f83f8e7
--- /dev/null
+++ b/block/add-cow.h
@@ -0,0 +1,95 @@
+/*
+ * QEMU ADD-COW Disk Format
+ *
+ * Copyright IBM, Corp. 2012
+ *
+ * Authors:
+ * Dong Xu Wang <wdongxu@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#ifndef BLOCK_ADD_COW_H
+#define BLOCK_ADD_COW_H
+
+enum {
+ ADD_COW_F_BACKING_FILE = 0x01,
+ ADD_COW_F_BACKING_FORMAT_NO_PROBE = 0x02,
+ ADD_COW_FEATURE_MASK = ADD_COW_F_BACKING_FILE |
+ ADD_COW_F_BACKING_FORMAT_NO_PROBE,
+
+ ADD_COW_MAGIC = (((uint64_t)'A' << 56) | ((uint64_t)'D' << 48) | \
+ ((uint64_t)'D' << 40) | ((uint64_t)'_' << 32) | \
+ ((uint64_t)'C' << 24) | ((uint64_t)'O' << 16) | \
+ ((uint64_t)'W' << 8) | 0xFF),
+ ADD_COW_VERSION = 1,
+ ADD_COW_FILE_LEN = 1024,
+ ADD_COW_CACHE_SIZE = 16,
+ ADD_COW_CACHE_ENTRY_SIZE = 65536,
+ ADD_COW_CLUSTER_SIZE = 65536,
+ SECTORS_PER_CLUSTER = (ADD_COW_CLUSTER_SIZE / BDRV_SECTOR_SIZE),
+ ADD_COW_BITMAP_POS = 4096
+};
+
+typedef struct AddCowHeader {
+ uint64_t magic;
+ uint32_t version;
+ uint32_t backing_filename_offset;
+ uint32_t backing_filename_size;
+ uint32_t image_filename_offset;
+ uint32_t image_filename_size;
+ uint64_t features;
+} QEMU_PACKED AddCowHeader;
+
+typedef struct AddCowCachedTable {
+ void *table;
+ int64_t offset;
+ bool dirty;
+ int cache_hits;
+} AddCowCachedTable;
+
+typedef struct AddCowCache {
+ AddCowCachedTable *entries;
+ int entry_size;
+ int size;
+ bool writethrough;
+} AddCowCache;
+
+typedef struct BDRVAddCowState {
+ BlockDriverState *image_hd;
+ CoMutex lock;
+ int cluster_size;
+ AddCowCache *bitmap_cache;
+ uint64_t bitmap_size;
+ AddCowHeader header;
+} BDRVAddCowState;
+
+/* Convert sector_num to offset in bitmap */
+static inline int64_t offset_in_bitmap(int64_t sector_num)
+{
+ int64_t cluster_num = sector_num / SECTORS_PER_CLUSTER;
+ return cluster_num / 8;
+}
+
+static inline bool is_cluster_head(int64_t sector_num)
+{
+ return sector_num % SECTORS_PER_CLUSTER == 0;
+}
+
+static inline bool is_cluster_tail(int64_t sector_num)
+{
+ return (sector_num + 1) % SECTORS_PER_CLUSTER == 0;
+}
+
+AddCowCache *add_cow_cache_create(BlockDriverState *bs, int num_tables,
+ bool writethrough);
+int add_cow_cache_destroy(BlockDriverState *bs, AddCowCache *c);
+void add_cow_cache_entry_mark_dirty(AddCowCache *c, void *table);
+int add_cow_cache_get(BlockDriverState *bs, AddCowCache *c, uint64_t offset,
+ void **table);
+int add_cow_cache_flush(BlockDriverState *bs, AddCowCache *c);
+bool add_cow_cache_set_writethrough(BlockDriverState *bs, AddCowCache *c,
+ bool enable);
+#endif
diff --git a/block_int.h b/block_int.h
index 3d4abc6..a0be58c 100644
--- a/block_int.h
+++ b/block_int.h
@@ -51,6 +51,7 @@
#define BLOCK_OPT_PREALLOC "preallocation"
#define BLOCK_OPT_SUBFMT "subformat"
#define BLOCK_OPT_COMPAT_LEVEL "compat"
+#define BLOCK_OPT_IMAGE_FILE "image_file"
typedef struct BdrvTrackedRequest BdrvTrackedRequest;
--
1.7.1
^ permalink raw reply related [flat|nested] 30+ messages in thread
end of thread, other threads:[~2012-08-06 2:06 UTC | newest]
Thread overview: 30+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-07-31 16:51 [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format Dong Xu Wang
2012-07-31 16:51 ` [Qemu-devel] [PATCH 2/6 v11 v11] block: make some functions public Dong Xu Wang
2012-08-01 13:53 ` Eric Blake
2012-08-02 7:10 ` Dong Xu Wang
2012-08-01 14:01 ` Stefan Hajnoczi
2012-08-02 7:11 ` Dong Xu Wang
2012-07-31 16:51 ` [Qemu-devel] [PATCH 3/6] add-cow file format Dong Xu Wang
2012-08-01 13:57 ` Eric Blake
2012-08-01 14:14 ` Stefan Hajnoczi
2012-08-01 14:21 ` Kevin Wolf
2012-08-02 7:20 ` Dong Xu Wang
2012-08-01 15:31 ` Stefan Hajnoczi
2012-08-02 7:20 ` Dong Xu Wang
2012-07-31 16:51 ` [Qemu-devel] [PATCH 4/6 v11] add-cow: support snapshot_blkde Dong Xu Wang
2012-08-01 15:37 ` Stefan Hajnoczi
2012-08-02 7:28 ` Dong Xu Wang
2012-08-02 10:37 ` Stefan Hajnoczi
2012-07-31 16:51 ` [Qemu-devel] [PATCH 5/6 v11] add-cow: hmp and qmp interface Dong Xu Wang
2012-07-31 16:51 ` [Qemu-devel] [PATCH 6/6 v11] add-cow: support qemu-iotests Dong Xu Wang
2012-08-01 13:51 ` [Qemu-devel] [PATCH 1/6 v11] docs: spec for add-cow file format Eric Blake
2012-08-02 7:03 ` Dong Xu Wang
2012-08-01 13:55 ` Stefan Hajnoczi
2012-08-02 7:09 ` Dong Xu Wang
2012-08-02 10:44 ` Stefan Hajnoczi
2012-08-03 5:56 ` Dong Xu Wang
2012-08-03 8:26 ` Stefan Hajnoczi
2012-08-06 2:05 ` Dong Xu Wang
-- strict thread matches above, loose matches on Subject: below --
2012-06-13 14:36 [Qemu-devel] [PATCH 1/6 v10] " Dong Xu Wang
2012-06-13 14:36 ` [Qemu-devel] [PATCH 3/6] " Dong Xu Wang
2012-06-14 11:13 ` Paolo Bonzini
2012-06-18 2:08 ` Dong Xu Wang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.