[Qemu-devel] [PATCH v4 00/14] qemu-img map: Allow driver to return file of the allocated block

[Qemu-devel] [PATCH v4 00/14] qemu-img map: Allow driver to return file of the allocated block

[Fam Zheng]

v4: Rebase and resend, adding Eric's and Stefan's reviewed-by.

    Fix one typo in patch 13.

    Drop previous patch 14 for a later rework because it is not a hard
    requirement, but it is pending on Eric's QAPI-to-JSON visitor series:

    https://lists.gnu.org/archive/html/qemu-devel/2015-12/msg03929.html

v3: Add Eric's rev-by in patches 6, 7, 13, 14.
    12: New, split out from the previous 13.
    12->13: Refactor "entry_mergable" from imp_map().
        Don't mess the merge conditions. [Paolo]
        Address Eric's comments:
        - Check has_foo before using foo.
        - Remove blank line between comments and definition in schema.
        - Use PRId64 instead of %ld.
        - Merge short lines.

v2: Add Eric's rev-by in patches 2, 4, 5.
    01: Refering -> referring in commit message. [Eric]
        Recurse to "file" for sensible "zero" flag. [Paolo]
    12: New. Make MapEntry a QAPI struct. [Paolo, Markus]

Original cover letter
---------------------

I stumbled upon this when looking at external bitmap formats.

Current "qemu-img map" command only displays filename if the data is allocated
in bs (bs->file) itself, or in the backing chain. Otherwise, it displays an
unfriendly error message:

    $ qemu-img create -f vmdk -o subformat=monolithicFlat /tmp/test.vmdk 1G

    $ qemu-img map /tmp/test.vmdk
    Offset          Length          Mapped to       File
    qemu-img: File contains external, encrypted or compressed clusters.

This can be improved. This series extends the .bdrv_co_get_block_status
callback, to let block driver return the BDS of file; then updates all driver
to implement it; and lastly, it changes qemu-img to use this information in
"map" command:


    $ qemu-img map /tmp/test.vmdk
    Offset          Length          Mapped to       File
    0               0x40000000      0               /tmp/test-flat.vmdk

    $ qemu-img map --output json /tmp/test.vmdk
    [{"length": 1073741824, "start": 0, "zero": false, "offset": 0, "depth": 0,
      "file": "/tmp/test-flat.vmdk", "data": true}
    ]

Fam Zheng (14):
  block: Add "file" output parameter to block status query functions
  qcow: Assign bs->file->bs to file in qcow_co_get_block_status
  qcow2: Assign bs->file->bs to file in qcow2_co_get_block_status
  raw: Assign bs to file in raw_co_get_block_status
  iscsi: Assign bs to file in iscsi_co_get_block_status
  parallels: Assign bs->file->bs to file in
    parallels_co_get_block_status
  qed: Assign bs->file->bs to file in bdrv_qed_co_get_block_status
  sheepdog: Assign bs to file in sd_co_get_block_status
  vdi: Assign bs->file->bs to file in vdi_co_get_block_status
  vpc: Assign bs->file->bs to file in vpc_co_get_block_status
  vmdk: Return extent's file in bdrv_get_block_status
  qemu-img: In "map", use the returned "file" from bdrv_get_block_status
  qemu-img: Make MapEntry a QAPI struct
  iotests: Add "qemu-img map" test for VMDK extents

 block/io.c                 | 42 ++++++++++++++++---------
 block/iscsi.c              |  9 ++++--
 block/mirror.c             |  3 +-
 block/parallels.c          |  3 +-
 block/qcow.c               |  3 +-
 block/qcow2.c              |  3 +-
 block/qed.c                |  6 +++-
 block/raw-posix.c          |  4 ++-
 block/raw_bsd.c            |  4 ++-
 block/sheepdog.c           |  5 ++-
 block/vdi.c                |  3 +-
 block/vmdk.c               | 13 ++++----
 block/vpc.c                |  4 ++-
 block/vvfat.c              |  2 +-
 include/block/block.h      |  6 ++--
 include/block/block_int.h  |  3 +-
 qapi/block-core.json       | 27 ++++++++++++++++
 qemu-img.c                 | 78 ++++++++++++++++++++++++++++------------------
 tests/qemu-iotests/059     | 10 ++++++
 tests/qemu-iotests/059.out | 38 ++++++++++++++++++++++
 20 files changed, 198 insertions(+), 68 deletions(-)

-- 
2.4.3

[Qemu-devel] [PATCH v4 01/14] block: Add "file" output parameter to block status query functions

[Fam Zheng]

The added parameter can be used to return the BDS pointer which the
valid offset is referring to. It's value should be ignored unless
BDRV_BLOCK_OFFSET_VALID in ret is set.

Until block drivers fill in the right value, let's clear it explicitly
right before calling .bdrv_get_block_status.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 block/io.c                | 42 ++++++++++++++++++++++++++++--------------
 block/iscsi.c             |  6 ++++--
 block/mirror.c            |  3 ++-
 block/parallels.c         |  2 +-
 block/qcow.c              |  2 +-
 block/qcow2.c             |  2 +-
 block/qed.c               |  3 ++-
 block/raw-posix.c         |  3 ++-
 block/raw_bsd.c           |  3 ++-
 block/sheepdog.c          |  2 +-
 block/vdi.c               |  2 +-
 block/vmdk.c              |  2 +-
 block/vpc.c               |  2 +-
 block/vvfat.c             |  2 +-
 include/block/block.h     |  6 ++++--
 include/block/block_int.h |  3 ++-
 qemu-img.c                |  7 +++++--
 17 files changed, 59 insertions(+), 33 deletions(-)

diff --git a/block/io.c b/block/io.c
index 63e3678..1ca4e61 100644
--- a/block/io.c
+++ b/block/io.c
@@ -663,6 +663,7 @@ int bdrv_write_zeroes(BlockDriverState *bs, int64_t sector_num,
 int bdrv_make_zero(BlockDriverState *bs, BdrvRequestFlags flags)
 {
     int64_t target_sectors, ret, nb_sectors, sector_num = 0;
+    BlockDriverState *file;
     int n;
 
     target_sectors = bdrv_nb_sectors(bs);
@@ -675,7 +676,7 @@ int bdrv_make_zero(BlockDriverState *bs, BdrvRequestFlags flags)
         if (nb_sectors <= 0) {
             return 0;
         }
-        ret = bdrv_get_block_status(bs, sector_num, nb_sectors, &n);
+        ret = bdrv_get_block_status(bs, sector_num, nb_sectors, &n, &file);
         if (ret < 0) {
             error_report("error getting block status at sector %" PRId64 ": %s",
                          sector_num, strerror(-ret));
@@ -1464,6 +1465,7 @@ int bdrv_flush_all(void)
 typedef struct BdrvCoGetBlockStatusData {
     BlockDriverState *bs;
     BlockDriverState *base;
+    BlockDriverState **file;
     int64_t sector_num;
     int nb_sectors;
     int *pnum;
@@ -1488,7 +1490,8 @@ typedef struct BdrvCoGetBlockStatusData {
  */
 static int64_t coroutine_fn bdrv_co_get_block_status(BlockDriverState *bs,
                                                      int64_t sector_num,
-                                                     int nb_sectors, int *pnum)
+                                                     int nb_sectors, int *pnum,
+                                                     BlockDriverState **file)
 {
     int64_t total_sectors;
     int64_t n;
@@ -1518,16 +1521,19 @@ static int64_t coroutine_fn bdrv_co_get_block_status(BlockDriverState *bs,
         return ret;
     }
 
-    ret = bs->drv->bdrv_co_get_block_status(bs, sector_num, nb_sectors, pnum);
+    *file = NULL;
+    ret = bs->drv->bdrv_co_get_block_status(bs, sector_num, nb_sectors, pnum,
+                                            file);
     if (ret < 0) {
         *pnum = 0;
+        *file = NULL;
         return ret;
     }
 
     if (ret & BDRV_BLOCK_RAW) {
         assert(ret & BDRV_BLOCK_OFFSET_VALID);
         return bdrv_get_block_status(bs->file->bs, ret >> BDRV_SECTOR_BITS,
-                                     *pnum, pnum);
+                                     *pnum, pnum, file);
     }
 
     if (ret & (BDRV_BLOCK_DATA | BDRV_BLOCK_ZERO)) {
@@ -1544,13 +1550,14 @@ static int64_t coroutine_fn bdrv_co_get_block_status(BlockDriverState *bs,
         }
     }
 
-    if (bs->file &&
+    if (*file && *file != bs &&
         (ret & BDRV_BLOCK_DATA) && !(ret & BDRV_BLOCK_ZERO) &&
         (ret & BDRV_BLOCK_OFFSET_VALID)) {
+        BlockDriverState *file2;
         int file_pnum;
 
-        ret2 = bdrv_co_get_block_status(bs->file->bs, ret >> BDRV_SECTOR_BITS,
-                                        *pnum, &file_pnum);
+        ret2 = bdrv_co_get_block_status(*file, ret >> BDRV_SECTOR_BITS,
+                                        *pnum, &file_pnum, &file2);
         if (ret2 >= 0) {
             /* Ignore errors.  This is just providing extra information, it
              * is useful but not necessary.
@@ -1575,14 +1582,15 @@ static int64_t coroutine_fn bdrv_co_get_block_status_above(BlockDriverState *bs,
         BlockDriverState *base,
         int64_t sector_num,
         int nb_sectors,
-        int *pnum)
+        int *pnum,
+        BlockDriverState **file)
 {
     BlockDriverState *p;
     int64_t ret = 0;
 
     assert(bs != base);
     for (p = bs; p != base; p = backing_bs(p)) {
-        ret = bdrv_co_get_block_status(p, sector_num, nb_sectors, pnum);
+        ret = bdrv_co_get_block_status(p, sector_num, nb_sectors, pnum, file);
         if (ret < 0 || ret & BDRV_BLOCK_ALLOCATED) {
             break;
         }
@@ -1601,7 +1609,8 @@ static void coroutine_fn bdrv_get_block_status_above_co_entry(void *opaque)
     data->ret = bdrv_co_get_block_status_above(data->bs, data->base,
                                                data->sector_num,
                                                data->nb_sectors,
-                                               data->pnum);
+                                               data->pnum,
+                                               data->file);
     data->done = true;
 }
 
@@ -1613,12 +1622,14 @@ static void coroutine_fn bdrv_get_block_status_above_co_entry(void *opaque)
 int64_t bdrv_get_block_status_above(BlockDriverState *bs,
                                     BlockDriverState *base,
                                     int64_t sector_num,
-                                    int nb_sectors, int *pnum)
+                                    int nb_sectors, int *pnum,
+                                    BlockDriverState **file)
 {
     Coroutine *co;
     BdrvCoGetBlockStatusData data = {
         .bs = bs,
         .base = base,
+        .file = file,
         .sector_num = sector_num,
         .nb_sectors = nb_sectors,
         .pnum = pnum,
@@ -1642,16 +1653,19 @@ int64_t bdrv_get_block_status_above(BlockDriverState *bs,
 
 int64_t bdrv_get_block_status(BlockDriverState *bs,
                               int64_t sector_num,
-                              int nb_sectors, int *pnum)
+                              int nb_sectors, int *pnum,
+                              BlockDriverState **file)
 {
     return bdrv_get_block_status_above(bs, backing_bs(bs),
-                                       sector_num, nb_sectors, pnum);
+                                       sector_num, nb_sectors, pnum, file);
 }
 
 int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t sector_num,
                                    int nb_sectors, int *pnum)
 {
-    int64_t ret = bdrv_get_block_status(bs, sector_num, nb_sectors, pnum);
+    BlockDriverState *file;
+    int64_t ret = bdrv_get_block_status(bs, sector_num, nb_sectors, pnum,
+                                        &file);
     if (ret < 0) {
         return ret;
     }
diff --git a/block/iscsi.c b/block/iscsi.c
index bd1f1bf..2d1e230 100644
--- a/block/iscsi.c
+++ b/block/iscsi.c
@@ -532,7 +532,8 @@ static bool iscsi_allocationmap_is_allocated(IscsiLun *iscsilun,
 
 static int64_t coroutine_fn iscsi_co_get_block_status(BlockDriverState *bs,
                                                   int64_t sector_num,
-                                                  int nb_sectors, int *pnum)
+                                                  int nb_sectors, int *pnum,
+                                                  BlockDriverState **file)
 {
     IscsiLun *iscsilun = bs->opaque;
     struct scsi_get_lba_status *lbas = NULL;
@@ -650,7 +651,8 @@ static int coroutine_fn iscsi_co_readv(BlockDriverState *bs,
         !iscsi_allocationmap_is_allocated(iscsilun, sector_num, nb_sectors)) {
         int64_t ret;
         int pnum;
-        ret = iscsi_co_get_block_status(bs, sector_num, INT_MAX, &pnum);
+        BlockDriverState *file;
+        ret = iscsi_co_get_block_status(bs, sector_num, INT_MAX, &pnum, &file);
         if (ret < 0) {
             return ret;
         }
diff --git a/block/mirror.c b/block/mirror.c
index f201f2b..e80a913 100644
--- a/block/mirror.c
+++ b/block/mirror.c
@@ -167,6 +167,7 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
     MirrorOp *op;
     int pnum;
     int64_t ret;
+    BlockDriverState *file;
 
     max_iov = MIN(source->bl.max_iov, s->target->bl.max_iov);
 
@@ -305,7 +306,7 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
     trace_mirror_one_iteration(s, sector_num, nb_sectors);
 
     ret = bdrv_get_block_status_above(source, NULL, sector_num,
-                                      nb_sectors, &pnum);
+                                      nb_sectors, &pnum, &file);
     if (ret < 0 || pnum < nb_sectors ||
             (ret & BDRV_BLOCK_DATA && !(ret & BDRV_BLOCK_ZERO))) {
         bdrv_aio_readv(source, sector_num, &op->qiov, nb_sectors,
diff --git a/block/parallels.c b/block/parallels.c
index e4a56a5..d83246b 100644
--- a/block/parallels.c
+++ b/block/parallels.c
@@ -260,7 +260,7 @@ static coroutine_fn int parallels_co_flush_to_os(BlockDriverState *bs)
 
 
 static int64_t coroutine_fn parallels_co_get_block_status(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors, int *pnum)
+        int64_t sector_num, int nb_sectors, int *pnum, BlockDriverState **file)
 {
     BDRVParallelsState *s = bs->opaque;
     int64_t offset;
diff --git a/block/qcow.c b/block/qcow.c
index 635085e..558f443 100644
--- a/block/qcow.c
+++ b/block/qcow.c
@@ -488,7 +488,7 @@ static uint64_t get_cluster_offset(BlockDriverState *bs,
 }
 
 static int64_t coroutine_fn qcow_co_get_block_status(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors, int *pnum)
+        int64_t sector_num, int nb_sectors, int *pnum, BlockDriverState **file)
 {
     BDRVQcowState *s = bs->opaque;
     int index_in_cluster, n;
diff --git a/block/qcow2.c b/block/qcow2.c
index 1789af4..7096a29 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1329,7 +1329,7 @@ static void qcow2_join_options(QDict *options, QDict *old_options)
 }
 
 static int64_t coroutine_fn qcow2_co_get_block_status(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors, int *pnum)
+        int64_t sector_num, int nb_sectors, int *pnum, BlockDriverState **file)
 {
     BDRVQcow2State *s = bs->opaque;
     uint64_t cluster_offset;
diff --git a/block/qed.c b/block/qed.c
index 9b88895..a6bbd8b 100644
--- a/block/qed.c
+++ b/block/qed.c
@@ -724,7 +724,8 @@ static void qed_is_allocated_cb(void *opaque, int ret, uint64_t offset, size_t l
 
 static int64_t coroutine_fn bdrv_qed_co_get_block_status(BlockDriverState *bs,
                                                  int64_t sector_num,
-                                                 int nb_sectors, int *pnum)
+                                                 int nb_sectors, int *pnum,
+                                                 BlockDriverState **file)
 {
     BDRVQEDState *s = bs->opaque;
     size_t len = (size_t)nb_sectors * BDRV_SECTOR_SIZE;
diff --git a/block/raw-posix.c b/block/raw-posix.c
index 076d070..6fc0b71 100644
--- a/block/raw-posix.c
+++ b/block/raw-posix.c
@@ -1826,7 +1826,8 @@ static int find_allocation(BlockDriverState *bs, off_t start,
  */
 static int64_t coroutine_fn raw_co_get_block_status(BlockDriverState *bs,
                                                     int64_t sector_num,
-                                                    int nb_sectors, int *pnum)
+                                                    int nb_sectors, int *pnum,
+                                                    BlockDriverState **file)
 {
     off_t start, data = 0, hole = 0;
     int64_t total_size;
diff --git a/block/raw_bsd.c b/block/raw_bsd.c
index 915d6fd..7d23c08 100644
--- a/block/raw_bsd.c
+++ b/block/raw_bsd.c
@@ -114,7 +114,8 @@ fail:
 
 static int64_t coroutine_fn raw_co_get_block_status(BlockDriverState *bs,
                                             int64_t sector_num,
-                                            int nb_sectors, int *pnum)
+                                            int nb_sectors, int *pnum,
+                                            BlockDriverState **file)
 {
     *pnum = nb_sectors;
     return BDRV_BLOCK_RAW | BDRV_BLOCK_OFFSET_VALID | BDRV_BLOCK_DATA |
diff --git a/block/sheepdog.c b/block/sheepdog.c
index d80e4ed..0f6789e 100644
--- a/block/sheepdog.c
+++ b/block/sheepdog.c
@@ -2709,7 +2709,7 @@ retry:
 
 static coroutine_fn int64_t
 sd_co_get_block_status(BlockDriverState *bs, int64_t sector_num, int nb_sectors,
-                       int *pnum)
+                       int *pnum, BlockDriverState **file)
 {
     BDRVSheepdogState *s = bs->opaque;
     SheepdogInode *inode = &s->inode;
diff --git a/block/vdi.c b/block/vdi.c
index 17f435f..2199fd3 100644
--- a/block/vdi.c
+++ b/block/vdi.c
@@ -526,7 +526,7 @@ static int vdi_reopen_prepare(BDRVReopenState *state,
 }
 
 static int64_t coroutine_fn vdi_co_get_block_status(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors, int *pnum)
+        int64_t sector_num, int nb_sectors, int *pnum, BlockDriverState **file)
 {
     /* TODO: Check for too large sector_num (in bdrv_is_allocated or here). */
     BDRVVdiState *s = (BDRVVdiState *)bs->opaque;
diff --git a/block/vmdk.c b/block/vmdk.c
index 6f819e4..f5a56fd 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -1248,7 +1248,7 @@ static inline uint64_t vmdk_find_index_in_cluster(VmdkExtent *extent,
 }
 
 static int64_t coroutine_fn vmdk_co_get_block_status(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors, int *pnum)
+        int64_t sector_num, int nb_sectors, int *pnum, BlockDriverState **file)
 {
     BDRVVmdkState *s = bs->opaque;
     int64_t index_in_cluster, n, ret;
diff --git a/block/vpc.c b/block/vpc.c
index 299d373..912f5d0 100644
--- a/block/vpc.c
+++ b/block/vpc.c
@@ -578,7 +578,7 @@ static coroutine_fn int vpc_co_write(BlockDriverState *bs, int64_t sector_num,
 }
 
 static int64_t coroutine_fn vpc_co_get_block_status(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors, int *pnum)
+        int64_t sector_num, int nb_sectors, int *pnum, BlockDriverState **file)
 {
     BDRVVPCState *s = bs->opaque;
     VHDFooter *footer = (VHDFooter*) s->footer_buf;
diff --git a/block/vvfat.c b/block/vvfat.c
index b184eca..b6b410a 100644
--- a/block/vvfat.c
+++ b/block/vvfat.c
@@ -2884,7 +2884,7 @@ static coroutine_fn int vvfat_co_write(BlockDriverState *bs, int64_t sector_num,
 }
 
 static int64_t coroutine_fn vvfat_co_get_block_status(BlockDriverState *bs,
-	int64_t sector_num, int nb_sectors, int* n)
+	int64_t sector_num, int nb_sectors, int *n, BlockDriverState **file)
 {
     BDRVVVFATState* s = bs->opaque;
     *n = s->sector_count - sector_num;
diff --git a/include/block/block.h b/include/block/block.h
index db8e096..70b4984 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -384,11 +384,13 @@ int bdrv_has_zero_init(BlockDriverState *bs);
 bool bdrv_unallocated_blocks_are_zero(BlockDriverState *bs);
 bool bdrv_can_write_zeroes_with_unmap(BlockDriverState *bs);
 int64_t bdrv_get_block_status(BlockDriverState *bs, int64_t sector_num,
-                              int nb_sectors, int *pnum);
+                              int nb_sectors, int *pnum,
+                              BlockDriverState **file);
 int64_t bdrv_get_block_status_above(BlockDriverState *bs,
                                     BlockDriverState *base,
                                     int64_t sector_num,
-                                    int nb_sectors, int *pnum);
+                                    int nb_sectors, int *pnum,
+                                    BlockDriverState **file);
 int bdrv_is_allocated(BlockDriverState *bs, int64_t sector_num, int nb_sectors,
                       int *pnum);
 int bdrv_is_allocated_above(BlockDriverState *top, BlockDriverState *base,
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 256609d..fcd32eb 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -166,7 +166,8 @@ struct BlockDriver {
     int coroutine_fn (*bdrv_co_discard)(BlockDriverState *bs,
         int64_t sector_num, int nb_sectors);
     int64_t coroutine_fn (*bdrv_co_get_block_status)(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors, int *pnum);
+        int64_t sector_num, int nb_sectors, int *pnum,
+        BlockDriverState **file);
 
     /*
      * Invalidate any cached meta-data.
diff --git a/qemu-img.c b/qemu-img.c
index 3d48b4f..ac4eee5 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -1259,9 +1259,10 @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
     n = MIN(s->total_sectors - sector_num, BDRV_REQUEST_MAX_SECTORS);
 
     if (s->sector_next_status <= sector_num) {
+        BlockDriverState *file;
         ret = bdrv_get_block_status(blk_bs(s->src[s->src_cur]),
                                     sector_num - s->src_cur_offset,
-                                    n, &n);
+                                    n, &n, &file);
         if (ret < 0) {
             return ret;
         }
@@ -2192,6 +2193,7 @@ static int get_block_status(BlockDriverState *bs, int64_t sector_num,
 {
     int64_t ret;
     int depth;
+    BlockDriverState *file;
 
     /* As an optimization, we could cache the current range of unallocated
      * clusters in each file of the chain, and avoid querying the same
@@ -2200,7 +2202,8 @@ static int get_block_status(BlockDriverState *bs, int64_t sector_num,
 
     depth = 0;
     for (;;) {
-        ret = bdrv_get_block_status(bs, sector_num, nb_sectors, &nb_sectors);
+        ret = bdrv_get_block_status(bs, sector_num, nb_sectors, &nb_sectors,
+                                    &file);
         if (ret < 0) {
             return ret;
         }
-- 
2.4.3

[Qemu-devel] [PATCH v4 02/14] qcow: Assign bs->file->bs to file in qcow_co_get_block_status

[Fam Zheng]

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 block/qcow.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/block/qcow.c b/block/qcow.c
index 558f443..b59383f 100644
--- a/block/qcow.c
+++ b/block/qcow.c
@@ -509,6 +509,7 @@ static int64_t coroutine_fn qcow_co_get_block_status(BlockDriverState *bs,
         return BDRV_BLOCK_DATA;
     }
     cluster_offset |= (index_in_cluster << BDRV_SECTOR_BITS);
+    *file = bs->file->bs;
     return BDRV_BLOCK_DATA | BDRV_BLOCK_OFFSET_VALID | cluster_offset;
 }
 
-- 
2.4.3

[Qemu-devel] [PATCH v4 03/14] qcow2: Assign bs->file->bs to file in qcow2_co_get_block_status

[Fam Zheng]

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 block/qcow2.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/block/qcow2.c b/block/qcow2.c
index 7096a29..da74eb7 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1348,6 +1348,7 @@ static int64_t coroutine_fn qcow2_co_get_block_status(BlockDriverState *bs,
         !s->cipher) {
         index_in_cluster = sector_num & (s->cluster_sectors - 1);
         cluster_offset |= (index_in_cluster << BDRV_SECTOR_BITS);
+        *file = bs->file->bs;
         status |= BDRV_BLOCK_OFFSET_VALID | cluster_offset;
     }
     if (ret == QCOW2_CLUSTER_ZERO) {
-- 
2.4.3

[Qemu-devel] [PATCH v4 04/14] raw: Assign bs to file in raw_co_get_block_status

[Fam Zheng]

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 block/raw-posix.c | 1 +
 block/raw_bsd.c   | 1 +
 2 files changed, 2 insertions(+)

diff --git a/block/raw-posix.c b/block/raw-posix.c
index 6fc0b71..344272f 100644
--- a/block/raw-posix.c
+++ b/block/raw-posix.c
@@ -1869,6 +1869,7 @@ static int64_t coroutine_fn raw_co_get_block_status(BlockDriverState *bs,
         *pnum = MIN(nb_sectors, (data - start) / BDRV_SECTOR_SIZE);
         ret = BDRV_BLOCK_ZERO;
     }
+    *file = bs;
     return ret | BDRV_BLOCK_OFFSET_VALID | start;
 }
 
diff --git a/block/raw_bsd.c b/block/raw_bsd.c
index 7d23c08..2d4c896 100644
--- a/block/raw_bsd.c
+++ b/block/raw_bsd.c
@@ -118,6 +118,7 @@ static int64_t coroutine_fn raw_co_get_block_status(BlockDriverState *bs,
                                             BlockDriverState **file)
 {
     *pnum = nb_sectors;
+    *file = bs;
     return BDRV_BLOCK_RAW | BDRV_BLOCK_OFFSET_VALID | BDRV_BLOCK_DATA |
            (sector_num << BDRV_SECTOR_BITS);
 }
-- 
2.4.3

[Qemu-devel] [PATCH v4 05/14] iscsi: Assign bs to file in iscsi_co_get_block_status

[Fam Zheng]

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 block/iscsi.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/block/iscsi.c b/block/iscsi.c
index 2d1e230..8c7f1b3 100644
--- a/block/iscsi.c
+++ b/block/iscsi.c
@@ -625,6 +625,9 @@ out:
     if (iTask.task != NULL) {
         scsi_free_scsi_task(iTask.task);
     }
+    if (ret > 0 && ret & BDRV_BLOCK_OFFSET_VALID) {
+        *file = bs;
+    }
     return ret;
 }
 
-- 
2.4.3

[Qemu-devel] [PATCH v4 06/14] parallels: Assign bs->file->bs to file in parallels_co_get_block_status

[Fam Zheng]

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 block/parallels.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/block/parallels.c b/block/parallels.c
index d83246b..129668b 100644
--- a/block/parallels.c
+++ b/block/parallels.c
@@ -273,6 +273,7 @@ static int64_t coroutine_fn parallels_co_get_block_status(BlockDriverState *bs,
         return 0;
     }
 
+    *file = bs->file->bs;
     return (offset << BDRV_SECTOR_BITS) |
         BDRV_BLOCK_DATA | BDRV_BLOCK_OFFSET_VALID;
 }
-- 
2.4.3

[Qemu-devel] [PATCH v4 07/14] qed: Assign bs->file->bs to file in bdrv_qed_co_get_block_status

[Fam Zheng]

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 block/qed.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/block/qed.c b/block/qed.c
index a6bbd8b..03af9c1 100644
--- a/block/qed.c
+++ b/block/qed.c
@@ -692,6 +692,7 @@ typedef struct {
     uint64_t pos;
     int64_t status;
     int *pnum;
+    BlockDriverState **file;
 } QEDIsAllocatedCB;
 
 static void qed_is_allocated_cb(void *opaque, int ret, uint64_t offset, size_t len)
@@ -703,6 +704,7 @@ static void qed_is_allocated_cb(void *opaque, int ret, uint64_t offset, size_t l
     case QED_CLUSTER_FOUND:
         offset |= qed_offset_into_cluster(s, cb->pos);
         cb->status = BDRV_BLOCK_DATA | BDRV_BLOCK_OFFSET_VALID | offset;
+        *cb->file = cb->bs->file->bs;
         break;
     case QED_CLUSTER_ZERO:
         cb->status = BDRV_BLOCK_ZERO;
@@ -734,6 +736,7 @@ static int64_t coroutine_fn bdrv_qed_co_get_block_status(BlockDriverState *bs,
         .pos = (uint64_t)sector_num * BDRV_SECTOR_SIZE,
         .status = BDRV_BLOCK_OFFSET_MASK,
         .pnum = pnum,
+        .file = file,
     };
     QEDRequest request = { .l2_table = NULL };
 
-- 
2.4.3

[Qemu-devel] [PATCH v4 08/14] sheepdog: Assign bs to file in sd_co_get_block_status

[Fam Zheng]

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 block/sheepdog.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/block/sheepdog.c b/block/sheepdog.c
index 0f6789e..d5e7ff8 100644
--- a/block/sheepdog.c
+++ b/block/sheepdog.c
@@ -2740,6 +2740,9 @@ sd_co_get_block_status(BlockDriverState *bs, int64_t sector_num, int nb_sectors,
     if (*pnum > nb_sectors) {
         *pnum = nb_sectors;
     }
+    if (ret > 0 && ret & BDRV_BLOCK_OFFSET_VALID) {
+        *file = bs;
+    }
     return ret;
 }
 
-- 
2.4.3

[Qemu-devel] [PATCH v4 09/14] vdi: Assign bs->file->bs to file in vdi_co_get_block_status

[Fam Zheng]

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 block/vdi.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/block/vdi.c b/block/vdi.c
index 2199fd3..6b1a57b 100644
--- a/block/vdi.c
+++ b/block/vdi.c
@@ -550,6 +550,7 @@ static int64_t coroutine_fn vdi_co_get_block_status(BlockDriverState *bs,
     offset = s->header.offset_data +
                               (uint64_t)bmap_entry * s->block_size +
                               sector_in_block * SECTOR_SIZE;
+    *file = bs->file->bs;
     return BDRV_BLOCK_DATA | BDRV_BLOCK_OFFSET_VALID | offset;
 }
 
-- 
2.4.3

[Qemu-devel] [PATCH v4 10/14] vpc: Assign bs->file->bs to file in vpc_co_get_block_status

[Fam Zheng]

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 block/vpc.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/block/vpc.c b/block/vpc.c
index 912f5d0..412ff41 100644
--- a/block/vpc.c
+++ b/block/vpc.c
@@ -588,6 +588,7 @@ static int64_t coroutine_fn vpc_co_get_block_status(BlockDriverState *bs,
 
     if (be32_to_cpu(footer->type) == VHD_FIXED) {
         *pnum = nb_sectors;
+        *file = bs->file->bs;
         return BDRV_BLOCK_RAW | BDRV_BLOCK_OFFSET_VALID | BDRV_BLOCK_DATA |
                (sector_num << BDRV_SECTOR_BITS);
     }
@@ -609,6 +610,7 @@ static int64_t coroutine_fn vpc_co_get_block_status(BlockDriverState *bs,
         /* *pnum can't be greater than one block for allocated
          * sectors since there is always a bitmap in between. */
         if (allocated) {
+            *file = bs->file->bs;
             return BDRV_BLOCK_DATA | BDRV_BLOCK_OFFSET_VALID | start;
         }
         if (nb_sectors == 0) {
-- 
2.4.3

[Qemu-devel] [PATCH v4 11/14] vmdk: Return extent's file in bdrv_get_block_status

[Fam Zheng]

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 block/vmdk.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c
index f5a56fd..b60a5af 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -1265,6 +1265,7 @@ static int64_t coroutine_fn vmdk_co_get_block_status(BlockDriverState *bs,
                              0, 0);
     qemu_co_mutex_unlock(&s->lock);
 
+    index_in_cluster = vmdk_find_index_in_cluster(extent, sector_num);
     switch (ret) {
     case VMDK_ERROR:
         ret = -EIO;
@@ -1276,15 +1277,13 @@ static int64_t coroutine_fn vmdk_co_get_block_status(BlockDriverState *bs,
         ret = BDRV_BLOCK_ZERO;
         break;
     case VMDK_OK:
-        ret = BDRV_BLOCK_DATA;
-        if (extent->file == bs->file && !extent->compressed) {
-            ret |= BDRV_BLOCK_OFFSET_VALID | offset;
-        }
-
+        ret = BDRV_BLOCK_DATA | BDRV_BLOCK_OFFSET_VALID;
+        ret |= (offset + (index_in_cluster << BDRV_SECTOR_BITS))
+                & BDRV_BLOCK_OFFSET_MASK;
+        *file = extent->file->bs;
         break;
     }
 
-    index_in_cluster = vmdk_find_index_in_cluster(extent, sector_num);
     n = extent->cluster_sectors - index_in_cluster;
     if (n > nb_sectors) {
         n = nb_sectors;
-- 
2.4.3

[Qemu-devel] [PATCH v4 12/14] qemu-img: In "map", use the returned "file" from bdrv_get_block_status

[Fam Zheng]

Now all drivers should return a correct "file", we can make use of it,
even with the recursion into backing chain above.

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 qemu-img.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/qemu-img.c b/qemu-img.c
index ac4eee5..a35edee 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -2225,7 +2225,7 @@ static int get_block_status(BlockDriverState *bs, int64_t sector_num,
     e->flags = ret & ~BDRV_BLOCK_OFFSET_MASK;
     e->offset = ret & BDRV_BLOCK_OFFSET_MASK;
     e->depth = depth;
-    e->bs = bs;
+    e->bs = file;
     return 0;
 }
 
-- 
2.4.3

[Qemu-devel] [PATCH v4 13/14] qemu-img: Make MapEntry a QAPI struct

[Fam Zheng]

The "flags" bit mask is expanded to two booleans, "data" and "zero";
"bs" is replaced with "filename" string.

Refactor the merge conditions in img_map() into entry_mergeable().

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 qapi/block-core.json | 27 ++++++++++++++++++++
 qemu-img.c           | 71 +++++++++++++++++++++++++++++++---------------------
 2 files changed, 69 insertions(+), 29 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index 1a5d9ce..90f7467 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -186,6 +186,33 @@
            '*fragmented-clusters': 'int', '*compressed-clusters': 'int' } }
 
 ##
+# @MapEntry:
+#
+# Mapping information from a virtual block range to a host file range
+#
+# @start: the start byte of the mapped virtual range
+#
+# @length: the number of bytes of the mapped virtual range
+#
+# @data: whether the mapped range has data
+#
+# @zero: whether the virtual blocks are zeroed
+#
+# @depth: the depth of the mapping
+#
+# @offset: #optional the offset in file that the virtual sectors are mapped to
+#
+# @filename: #optional filename that is referred to by @offset
+#
+# Since: 2.6
+#
+##
+{ 'struct': 'MapEntry',
+  'data': {'start': 'int', 'length': 'int', 'data': 'bool',
+           'zero': 'bool', 'depth': 'int', '*offset': 'int',
+           '*filename': 'str' } }
+
+##
 # @BlockdevCacheInfo
 #
 # Cache mode information for a block device
diff --git a/qemu-img.c b/qemu-img.c
index a35edee..78b755c 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -2136,47 +2136,37 @@ static int img_info(int argc, char **argv)
     return 0;
 }
 
-
-typedef struct MapEntry {
-    int flags;
-    int depth;
-    int64_t start;
-    int64_t length;
-    int64_t offset;
-    BlockDriverState *bs;
-} MapEntry;
-
 static void dump_map_entry(OutputFormat output_format, MapEntry *e,
                            MapEntry *next)
 {
     switch (output_format) {
     case OFORMAT_HUMAN:
-        if ((e->flags & BDRV_BLOCK_DATA) &&
-            !(e->flags & BDRV_BLOCK_OFFSET_VALID)) {
+        if (e->data && !e->has_offset) {
             error_report("File contains external, encrypted or compressed clusters.");
             exit(1);
         }
-        if ((e->flags & (BDRV_BLOCK_DATA|BDRV_BLOCK_ZERO)) == BDRV_BLOCK_DATA) {
+        if (e->data && !e->zero) {
             printf("%#-16"PRIx64"%#-16"PRIx64"%#-16"PRIx64"%s\n",
-                   e->start, e->length, e->offset, e->bs->filename);
+                   e->start, e->length,
+                   e->has_offset ? e->offset : 0,
+                   e->has_filename ? e->filename : "");
         }
         /* This format ignores the distinction between 0, ZERO and ZERO|DATA.
          * Modify the flags here to allow more coalescing.
          */
-        if (next &&
-            (next->flags & (BDRV_BLOCK_DATA|BDRV_BLOCK_ZERO)) != BDRV_BLOCK_DATA) {
-            next->flags &= ~BDRV_BLOCK_DATA;
-            next->flags |= BDRV_BLOCK_ZERO;
+        if (next && (!next->data || next->zero)) {
+            next->data = false;
+            next->zero = true;
         }
         break;
     case OFORMAT_JSON:
-        printf("%s{ \"start\": %"PRId64", \"length\": %"PRId64", \"depth\": %d,"
-               " \"zero\": %s, \"data\": %s",
+        printf("%s{ \"start\": %"PRId64", \"length\": %"PRId64","
+               " \"depth\": %"PRId64", \"zero\": %s, \"data\": %s",
                (e->start == 0 ? "[" : ",\n"),
                e->start, e->length, e->depth,
-               (e->flags & BDRV_BLOCK_ZERO) ? "true" : "false",
-               (e->flags & BDRV_BLOCK_DATA) ? "true" : "false");
-        if (e->flags & BDRV_BLOCK_OFFSET_VALID) {
+               e->zero ? "true" : "false",
+               e->data ? "true" : "false");
+        if (e->has_offset) {
             printf(", \"offset\": %"PRId64"", e->offset);
         }
         putchar('}');
@@ -2222,13 +2212,39 @@ static int get_block_status(BlockDriverState *bs, int64_t sector_num,
 
     e->start = sector_num * BDRV_SECTOR_SIZE;
     e->length = nb_sectors * BDRV_SECTOR_SIZE;
-    e->flags = ret & ~BDRV_BLOCK_OFFSET_MASK;
+    e->data = !!(ret & BDRV_BLOCK_DATA);
+    e->zero = !!(ret & BDRV_BLOCK_ZERO);
     e->offset = ret & BDRV_BLOCK_OFFSET_MASK;
+    e->has_offset = !!(ret & BDRV_BLOCK_OFFSET_VALID);
     e->depth = depth;
-    e->bs = file;
+    if (file && e->has_offset) {
+        e->has_filename = true;
+        e->filename = file->filename;
+    }
     return 0;
 }
 
+static inline bool entry_mergeable(const MapEntry *curr, const MapEntry *next)
+{
+    if (curr->length == 0) {
+        return false;
+    }
+    if (curr->zero != next->zero ||
+        curr->data != next->data ||
+        curr->depth != next->depth ||
+        curr->has_filename != next->has_filename ||
+        curr->has_offset != next->has_offset) {
+        return false;
+    }
+    if (curr->has_filename && strcmp(curr->filename, next->filename)) {
+        return false;
+    }
+    if (curr->has_offset && curr->offset + curr->length != next->offset) {
+        return false;
+    }
+    return true;
+}
+
 static int img_map(int argc, char **argv)
 {
     int c;
@@ -2310,10 +2326,7 @@ static int img_map(int argc, char **argv)
             goto out;
         }
 
-        if (curr.length != 0 && curr.flags == next.flags &&
-            curr.depth == next.depth &&
-            ((curr.flags & BDRV_BLOCK_OFFSET_VALID) == 0 ||
-             curr.offset + curr.length == next.offset)) {
+        if (entry_mergeable(&curr, &next)) {
             curr.length += next.length;
             continue;
         }
-- 
2.4.3

[Qemu-devel] [PATCH v4 14/14] iotests: Add "qemu-img map" test for VMDK extents

[Fam Zheng]

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/qemu-iotests/059     | 10 ++++++++++
 tests/qemu-iotests/059.out | 38 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+)

diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059
index 0ded0c3..261d8b0 100755
--- a/tests/qemu-iotests/059
+++ b/tests/qemu-iotests/059
@@ -133,6 +133,16 @@ $QEMU_IO -c "write -P 0xa 900G 512" "$TEST_IMG" | _filter_qemu_io
 $QEMU_IO -c "read -v 900G 1024" "$TEST_IMG" | _filter_qemu_io
 
 echo
+echo "=== Testing qemu-img map on extents ==="
+for fmt in twoGbMaxExtentSparse twoGbMaxExtentFlat; do
+    IMGOPTS="subformat=$fmt" _make_test_img 31G
+    $QEMU_IO -c "write 65024 1k" "$TEST_IMG" | _filter_qemu_io
+    $QEMU_IO -c "write 2147483136 1k" "$TEST_IMG" | _filter_qemu_io
+    $QEMU_IO -c "write 5G 1k" "$TEST_IMG" | _filter_qemu_io
+    $QEMU_IMG map "$TEST_IMG" | _filter_testdir
+done
+
+echo
 echo "=== Testing afl image with a very large capacity ==="
 _use_sample_img afl9.vmdk.bz2
 _img_info
diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out
index 00057fe..54eb530 100644
--- a/tests/qemu-iotests/059.out
+++ b/tests/qemu-iotests/059.out
@@ -2337,6 +2337,44 @@ e1000003f0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
 read 1024/1024 bytes at offset 966367641600
 1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
+=== Testing qemu-img map on extents ===
+Formatting 'TEST_DIR/iotest-version3.IMGFMT', fmt=IMGFMT size=33285996544 subformat=twoGbMaxExtentSparse
+wrote 1024/1024 bytes at offset 65024
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 1024/1024 bytes at offset 2147483136
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 1024/1024 bytes at offset 5368709120
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+Offset          Length          Mapped to       File
+0               0x20000         0x50000         TEST_DIR/iotest-version3-s001.vmdk
+0x7fff0000      0x10000         0x70000         TEST_DIR/iotest-version3-s001.vmdk
+0x80000000      0x10000         0x50000         TEST_DIR/iotest-version3-s002.vmdk
+0x140000000     0x10000         0x50000         TEST_DIR/iotest-version3-s003.vmdk
+Formatting 'TEST_DIR/iotest-version3.IMGFMT', fmt=IMGFMT size=33285996544 subformat=twoGbMaxExtentFlat
+wrote 1024/1024 bytes at offset 65024
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 1024/1024 bytes at offset 2147483136
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 1024/1024 bytes at offset 5368709120
+1 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+Offset          Length          Mapped to       File
+0               0x80000000      0               TEST_DIR/iotest-version3-f001.vmdk
+0x80000000      0x80000000      0               TEST_DIR/iotest-version3-f002.vmdk
+0x100000000     0x80000000      0               TEST_DIR/iotest-version3-f003.vmdk
+0x180000000     0x80000000      0               TEST_DIR/iotest-version3-f004.vmdk
+0x200000000     0x80000000      0               TEST_DIR/iotest-version3-f005.vmdk
+0x280000000     0x80000000      0               TEST_DIR/iotest-version3-f006.vmdk
+0x300000000     0x80000000      0               TEST_DIR/iotest-version3-f007.vmdk
+0x380000000     0x80000000      0               TEST_DIR/iotest-version3-f008.vmdk
+0x400000000     0x80000000      0               TEST_DIR/iotest-version3-f009.vmdk
+0x480000000     0x80000000      0               TEST_DIR/iotest-version3-f010.vmdk
+0x500000000     0x80000000      0               TEST_DIR/iotest-version3-f011.vmdk
+0x580000000     0x80000000      0               TEST_DIR/iotest-version3-f012.vmdk
+0x600000000     0x80000000      0               TEST_DIR/iotest-version3-f013.vmdk
+0x680000000     0x80000000      0               TEST_DIR/iotest-version3-f014.vmdk
+0x700000000     0x80000000      0               TEST_DIR/iotest-version3-f015.vmdk
+0x780000000     0x40000000      0               TEST_DIR/iotest-version3-f016.vmdk
+
 === Testing afl image with a very large capacity ===
 qemu-img: Can't get size of device 'image': File too large
 *** done
-- 
2.4.3

Re: [Qemu-devel] [PATCH v4 11/14] vmdk: Return extent's file in bdrv_get_block_status

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 1536 bytes --]

On 24.12.2015 06:50, Fam Zheng wrote:
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Fam Zheng <famz@redhat.com>
> ---
>  block/vmdk.c | 11 +++++------
>  1 file changed, 5 insertions(+), 6 deletions(-)
> 
> diff --git a/block/vmdk.c b/block/vmdk.c
> index f5a56fd..b60a5af 100644
> --- a/block/vmdk.c
> +++ b/block/vmdk.c
> @@ -1265,6 +1265,7 @@ static int64_t coroutine_fn vmdk_co_get_block_status(BlockDriverState *bs,
>                               0, 0);
>      qemu_co_mutex_unlock(&s->lock);
>  
> +    index_in_cluster = vmdk_find_index_in_cluster(extent, sector_num);
>      switch (ret) {
>      case VMDK_ERROR:
>          ret = -EIO;
> @@ -1276,15 +1277,13 @@ static int64_t coroutine_fn vmdk_co_get_block_status(BlockDriverState *bs,
>          ret = BDRV_BLOCK_ZERO;
>          break;
>      case VMDK_OK:
> -        ret = BDRV_BLOCK_DATA;
> -        if (extent->file == bs->file && !extent->compressed) {
> -            ret |= BDRV_BLOCK_OFFSET_VALID | offset;
> -        }
> -
> +        ret = BDRV_BLOCK_DATA | BDRV_BLOCK_OFFSET_VALID;
> +        ret |= (offset + (index_in_cluster << BDRV_SECTOR_BITS))
> +                & BDRV_BLOCK_OFFSET_MASK;
> +        *file = extent->file->bs;

What if the extent is compressed?

Max

>          break;
>      }
>  
> -    index_in_cluster = vmdk_find_index_in_cluster(extent, sector_num);
>      n = extent->cluster_sectors - index_in_cluster;
>      if (n > nb_sectors) {
>          n = nb_sectors;
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH v4 01/14] block: Add "file" output parameter to block status query functions

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 1639 bytes --]

On 24.12.2015 06:50, Fam Zheng wrote:
> The added parameter can be used to return the BDS pointer which the
> valid offset is referring to. It's value should be ignored unless

*Its

> BDRV_BLOCK_OFFSET_VALID in ret is set.
> 
> Until block drivers fill in the right value, let's clear it explicitly
> right before calling .bdrv_get_block_status.
> 
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Fam Zheng <famz@redhat.com>
> ---
>  block/io.c                | 42 ++++++++++++++++++++++++++++--------------
>  block/iscsi.c             |  6 ++++--
>  block/mirror.c            |  3 ++-
>  block/parallels.c         |  2 +-
>  block/qcow.c              |  2 +-
>  block/qcow2.c             |  2 +-
>  block/qed.c               |  3 ++-
>  block/raw-posix.c         |  3 ++-
>  block/raw_bsd.c           |  3 ++-
>  block/sheepdog.c          |  2 +-
>  block/vdi.c               |  2 +-
>  block/vmdk.c              |  2 +-
>  block/vpc.c               |  2 +-
>  block/vvfat.c             |  2 +-
>  include/block/block.h     |  6 ++++--
>  include/block/block_int.h |  3 ++-
>  qemu-img.c                |  7 +++++--
>  17 files changed, 59 insertions(+), 33 deletions(-)
> 

[...]

> diff --git a/include/block/block.h b/include/block/block.h
> index db8e096..70b4984 100644
> --- a/include/block/block.h
> +++ b/include/block/block.h

The comment explaining BDRV_BLOCK_OFFSET_VALID should be changed
accordingly (you could also say "fixed", because apparently it wasn't
always bs->file; sometimes it was bs itself (in case of raw-posix, iscsi
and sheepdog)).

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH v4 00/14] qemu-img map: Allow driver to return file of the allocated block

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 4564 bytes --]

On 24.12.2015 06:50, Fam Zheng wrote:
> v4: Rebase and resend, adding Eric's and Stefan's reviewed-by.
> 
>     Fix one typo in patch 13.
> 
>     Drop previous patch 14 for a later rework because it is not a hard
>     requirement, but it is pending on Eric's QAPI-to-JSON visitor series:
> 
>     https://lists.gnu.org/archive/html/qemu-devel/2015-12/msg03929.html
> 
> v3: Add Eric's rev-by in patches 6, 7, 13, 14.
>     12: New, split out from the previous 13.
>     12->13: Refactor "entry_mergable" from imp_map().
>         Don't mess the merge conditions. [Paolo]
>         Address Eric's comments:
>         - Check has_foo before using foo.
>         - Remove blank line between comments and definition in schema.
>         - Use PRId64 instead of %ld.
>         - Merge short lines.
> 
> v2: Add Eric's rev-by in patches 2, 4, 5.
>     01: Refering -> referring in commit message. [Eric]
>         Recurse to "file" for sensible "zero" flag. [Paolo]
>     12: New. Make MapEntry a QAPI struct. [Paolo, Markus]
> 
> Original cover letter
> ---------------------
> 
> I stumbled upon this when looking at external bitmap formats.
> 
> Current "qemu-img map" command only displays filename if the data is allocated
> in bs (bs->file) itself, or in the backing chain. Otherwise, it displays an
> unfriendly error message:
> 
>     $ qemu-img create -f vmdk -o subformat=monolithicFlat /tmp/test.vmdk 1G
> 
>     $ qemu-img map /tmp/test.vmdk
>     Offset          Length          Mapped to       File
>     qemu-img: File contains external, encrypted or compressed clusters.
> 
> This can be improved. This series extends the .bdrv_co_get_block_status
> callback, to let block driver return the BDS of file; then updates all driver
> to implement it; and lastly, it changes qemu-img to use this information in
> "map" command:
> 
> 
>     $ qemu-img map /tmp/test.vmdk
>     Offset          Length          Mapped to       File
>     0               0x40000000      0               /tmp/test-flat.vmdk
> 
>     $ qemu-img map --output json /tmp/test.vmdk
>     [{"length": 1073741824, "start": 0, "zero": false, "offset": 0, "depth": 0,
>       "file": "/tmp/test-flat.vmdk", "data": true}
>     ]
> 
> Fam Zheng (14):
>   block: Add "file" output parameter to block status query functions
>   qcow: Assign bs->file->bs to file in qcow_co_get_block_status
>   qcow2: Assign bs->file->bs to file in qcow2_co_get_block_status

Minor comment: I'd swap these two patches (2 and 3). Patch 1 breaks test
102, patch 3 fixes it again. It would be better to break it for as short
a time as possible.

Alternatively, in order not to break 102 at all, patch 1 would need to
leave the "if (bs->file &&" part of bdrv_co_get_block_status()
(@@ -1544,13 +1550,14 @@) as-is and change it only after the format
block drivers set *file.

Max

>   raw: Assign bs to file in raw_co_get_block_status
>   iscsi: Assign bs to file in iscsi_co_get_block_status
>   parallels: Assign bs->file->bs to file in
>     parallels_co_get_block_status
>   qed: Assign bs->file->bs to file in bdrv_qed_co_get_block_status
>   sheepdog: Assign bs to file in sd_co_get_block_status
>   vdi: Assign bs->file->bs to file in vdi_co_get_block_status
>   vpc: Assign bs->file->bs to file in vpc_co_get_block_status
>   vmdk: Return extent's file in bdrv_get_block_status
>   qemu-img: In "map", use the returned "file" from bdrv_get_block_status
>   qemu-img: Make MapEntry a QAPI struct
>   iotests: Add "qemu-img map" test for VMDK extents
> 
>  block/io.c                 | 42 ++++++++++++++++---------
>  block/iscsi.c              |  9 ++++--
>  block/mirror.c             |  3 +-
>  block/parallels.c          |  3 +-
>  block/qcow.c               |  3 +-
>  block/qcow2.c              |  3 +-
>  block/qed.c                |  6 +++-
>  block/raw-posix.c          |  4 ++-
>  block/raw_bsd.c            |  4 ++-
>  block/sheepdog.c           |  5 ++-
>  block/vdi.c                |  3 +-
>  block/vmdk.c               | 13 ++++----
>  block/vpc.c                |  4 ++-
>  block/vvfat.c              |  2 +-
>  include/block/block.h      |  6 ++--
>  include/block/block_int.h  |  3 +-
>  qapi/block-core.json       | 27 ++++++++++++++++
>  qemu-img.c                 | 78 ++++++++++++++++++++++++++++------------------
>  tests/qemu-iotests/059     | 10 ++++++
>  tests/qemu-iotests/059.out | 38 ++++++++++++++++++++++
>  20 files changed, 198 insertions(+), 68 deletions(-)
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH v4 01/14] block: Add "file" output parameter to block status query functions

[Fam Zheng]

On Mon, 01/04 22:00, Max Reitz wrote:
> On 24.12.2015 06:50, Fam Zheng wrote:
> > The added parameter can be used to return the BDS pointer which the
> > valid offset is referring to. It's value should be ignored unless
> 
> *Its
> 
> > BDRV_BLOCK_OFFSET_VALID in ret is set.
> > 
> > Until block drivers fill in the right value, let's clear it explicitly
> > right before calling .bdrv_get_block_status.
> > 
> > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> > Signed-off-by: Fam Zheng <famz@redhat.com>
> > ---
> >  block/io.c                | 42 ++++++++++++++++++++++++++++--------------
> >  block/iscsi.c             |  6 ++++--
> >  block/mirror.c            |  3 ++-
> >  block/parallels.c         |  2 +-
> >  block/qcow.c              |  2 +-
> >  block/qcow2.c             |  2 +-
> >  block/qed.c               |  3 ++-
> >  block/raw-posix.c         |  3 ++-
> >  block/raw_bsd.c           |  3 ++-
> >  block/sheepdog.c          |  2 +-
> >  block/vdi.c               |  2 +-
> >  block/vmdk.c              |  2 +-
> >  block/vpc.c               |  2 +-
> >  block/vvfat.c             |  2 +-
> >  include/block/block.h     |  6 ++++--
> >  include/block/block_int.h |  3 ++-
> >  qemu-img.c                |  7 +++++--
> >  17 files changed, 59 insertions(+), 33 deletions(-)
> > 
> 
> [...]
> 
> > diff --git a/include/block/block.h b/include/block/block.h
> > index db8e096..70b4984 100644
> > --- a/include/block/block.h
> > +++ b/include/block/block.h
> 
> The comment explaining BDRV_BLOCK_OFFSET_VALID should be changed
> accordingly (you could also say "fixed", because apparently it wasn't
> always bs->file; sometimes it was bs itself (in case of raw-posix, iscsi
> and sheepdog)).

Yes, good point! Will fix this, and the typo above.

Fam

Re: [Qemu-devel] [PATCH v4 11/14] vmdk: Return extent's file in bdrv_get_block_status

[Fam Zheng]

On Mon, 01/04 21:48, Max Reitz wrote:
> On 24.12.2015 06:50, Fam Zheng wrote:
> > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> > Signed-off-by: Fam Zheng <famz@redhat.com>
> > ---
> >  block/vmdk.c | 11 +++++------
> >  1 file changed, 5 insertions(+), 6 deletions(-)
> > 
> > diff --git a/block/vmdk.c b/block/vmdk.c
> > index f5a56fd..b60a5af 100644
> > --- a/block/vmdk.c
> > +++ b/block/vmdk.c
> > @@ -1265,6 +1265,7 @@ static int64_t coroutine_fn vmdk_co_get_block_status(BlockDriverState *bs,
> >                               0, 0);
> >      qemu_co_mutex_unlock(&s->lock);
> >  
> > +    index_in_cluster = vmdk_find_index_in_cluster(extent, sector_num);
> >      switch (ret) {
> >      case VMDK_ERROR:
> >          ret = -EIO;
> > @@ -1276,15 +1277,13 @@ static int64_t coroutine_fn vmdk_co_get_block_status(BlockDriverState *bs,
> >          ret = BDRV_BLOCK_ZERO;
> >          break;
> >      case VMDK_OK:
> > -        ret = BDRV_BLOCK_DATA;
> > -        if (extent->file == bs->file && !extent->compressed) {
> > -            ret |= BDRV_BLOCK_OFFSET_VALID | offset;
> > -        }
> > -
> > +        ret = BDRV_BLOCK_DATA | BDRV_BLOCK_OFFSET_VALID;
> > +        ret |= (offset + (index_in_cluster << BDRV_SECTOR_BITS))
> > +                & BDRV_BLOCK_OFFSET_MASK;
> > +        *file = extent->file->bs;
> 
> What if the extent is compressed?
> 

You're right, the offset shouldn't be set if compressed. Will fix.

Thanks!

Fam

Re: [Qemu-devel] [PATCH v4 00/14] qemu-img map: Allow driver to return file of the allocated block

[Fam Zheng]

On Mon, 01/04 22:02, Max Reitz wrote:
> On 24.12.2015 06:50, Fam Zheng wrote:
> > v4: Rebase and resend, adding Eric's and Stefan's reviewed-by.
> > 
> >     Fix one typo in patch 13.
> > 
> >     Drop previous patch 14 for a later rework because it is not a hard
> >     requirement, but it is pending on Eric's QAPI-to-JSON visitor series:
> > 
> >     https://lists.gnu.org/archive/html/qemu-devel/2015-12/msg03929.html
> > 
> > v3: Add Eric's rev-by in patches 6, 7, 13, 14.
> >     12: New, split out from the previous 13.
> >     12->13: Refactor "entry_mergable" from imp_map().
> >         Don't mess the merge conditions. [Paolo]
> >         Address Eric's comments:
> >         - Check has_foo before using foo.
> >         - Remove blank line between comments and definition in schema.
> >         - Use PRId64 instead of %ld.
> >         - Merge short lines.
> > 
> > v2: Add Eric's rev-by in patches 2, 4, 5.
> >     01: Refering -> referring in commit message. [Eric]
> >         Recurse to "file" for sensible "zero" flag. [Paolo]
> >     12: New. Make MapEntry a QAPI struct. [Paolo, Markus]
> > 
> > Original cover letter
> > ---------------------
> > 
> > I stumbled upon this when looking at external bitmap formats.
> > 
> > Current "qemu-img map" command only displays filename if the data is allocated
> > in bs (bs->file) itself, or in the backing chain. Otherwise, it displays an
> > unfriendly error message:
> > 
> >     $ qemu-img create -f vmdk -o subformat=monolithicFlat /tmp/test.vmdk 1G
> > 
> >     $ qemu-img map /tmp/test.vmdk
> >     Offset          Length          Mapped to       File
> >     qemu-img: File contains external, encrypted or compressed clusters.
> > 
> > This can be improved. This series extends the .bdrv_co_get_block_status
> > callback, to let block driver return the BDS of file; then updates all driver
> > to implement it; and lastly, it changes qemu-img to use this information in
> > "map" command:
> > 
> > 
> >     $ qemu-img map /tmp/test.vmdk
> >     Offset          Length          Mapped to       File
> >     0               0x40000000      0               /tmp/test-flat.vmdk
> > 
> >     $ qemu-img map --output json /tmp/test.vmdk
> >     [{"length": 1073741824, "start": 0, "zero": false, "offset": 0, "depth": 0,
> >       "file": "/tmp/test-flat.vmdk", "data": true}
> >     ]
> > 
> > Fam Zheng (14):
> >   block: Add "file" output parameter to block status query functions
> >   qcow: Assign bs->file->bs to file in qcow_co_get_block_status
> >   qcow2: Assign bs->file->bs to file in qcow2_co_get_block_status
> 
> Minor comment: I'd swap these two patches (2 and 3). Patch 1 breaks test
> 102, patch 3 fixes it again. It would be better to break it for as short
> a time as possible.
> 
> Alternatively, in order not to break 102 at all, patch 1 would need to
> leave the "if (bs->file &&" part of bdrv_co_get_block_status()
> (@@ -1544,13 +1550,14 @@) as-is and change it only after the format
> block drivers set *file.

Yes, this is better.

Fam