All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] qemu-img: Support bitmap --merge into backing image
@ 2020-09-09 12:33 Eric Blake
  2020-09-11  8:31 ` Max Reitz
  0 siblings, 1 reply; 4+ messages in thread
From: Eric Blake @ 2020-09-09 12:33 UTC (permalink / raw)
  To: qemu-devel; +Cc: Kevin Wolf, Eyal Shenitzky, qemu-block, Max Reitz

If you have the chain 'base.qcow2 <- top.qcow2' and want to merge a
bitmap from top into base, qemu-img was failing with:

qemu-img: Could not open 'top.qcow2': Could not open backing file: Failed to get shared "write" lock
Is another process using the image [base.qcow2]?

The easiest fix is to not open the entire backing chain of the source
image, so that we aren't worrying about competing BDS visiting the
backing image as both a read-only backing of the source and the
writeable destination.

Fixes: http://bugzilla.redhat.com/1877209
Reported-by: Eyal Shenitzky <eshenitz@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
 qemu-img.c                 |  3 +-
 tests/qemu-iotests/291     | 12 ++++++++
 tests/qemu-iotests/291.out | 56 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 70 insertions(+), 1 deletion(-)

diff --git a/qemu-img.c b/qemu-img.c
index eb2fc1f86243..b15098a2f9b3 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -4755,7 +4755,8 @@ static int img_bitmap(int argc, char **argv)
     }
     bs = blk_bs(blk);
     if (src_filename) {
-        src = img_open(false, src_filename, src_fmt, 0, false, false, false);
+        src = img_open(false, src_filename, src_fmt, BDRV_O_NO_BACKING,
+                       false, false, false);
         if (!src) {
             goto out;
         }
diff --git a/tests/qemu-iotests/291 b/tests/qemu-iotests/291
index 1e0bb76959bb..4f837b205655 100755
--- a/tests/qemu-iotests/291
+++ b/tests/qemu-iotests/291
@@ -91,6 +91,15 @@ $QEMU_IMG bitmap --remove --image-opts \
     driver=$IMGFMT,file.driver=file,file.filename="$TEST_IMG" tmp
 _img_info --format-specific

+echo
+echo "=== Merge from top layer into backing image ==="
+echo
+
+$QEMU_IMG rebase -u -F qcow2 -b "$TEST_IMG.base" "$TEST_IMG"
+$QEMU_IMG bitmap --add --merge b2 -b "$TEST_IMG" -F $IMGFMT \
+     -f $IMGFMT "$TEST_IMG.base" b3
+_img_info --format-specific --backing-chain
+
 echo
 echo "=== Check bitmap contents ==="
 echo
@@ -107,6 +116,9 @@ $QEMU_IMG map --output=json --image-opts \
 nbd_server_start_unix_socket -r -f qcow2 -B b2 "$TEST_IMG"
 $QEMU_IMG map --output=json --image-opts \
     "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b2" | _filter_qemu_img_map
+nbd_server_start_unix_socket -r -f qcow2 -B b3 "$TEST_IMG"
+$QEMU_IMG map --output=json --image-opts \
+    "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b3" | _filter_qemu_img_map

 # success, all done
 echo '*** done'
diff --git a/tests/qemu-iotests/291.out b/tests/qemu-iotests/291.out
index ee89a728855f..3990f7aacc7b 100644
--- a/tests/qemu-iotests/291.out
+++ b/tests/qemu-iotests/291.out
@@ -68,6 +68,59 @@ Format specific information:
     corrupt: false
     extended l2: false

+=== Merge from top layer into backing image ===
+
+image: TEST_DIR/t.IMGFMT
+file format: IMGFMT
+virtual size: 10 MiB (10485760 bytes)
+cluster_size: 65536
+backing file: TEST_DIR/t.IMGFMT.base
+backing file format: IMGFMT
+Format specific information:
+    compat: 1.1
+    compression type: zlib
+    lazy refcounts: false
+    bitmaps:
+        [0]:
+            flags:
+            name: b1
+            granularity: 524288
+        [1]:
+            flags:
+                [0]: auto
+            name: b2
+            granularity: 65536
+        [2]:
+            flags:
+            name: b0
+            granularity: 65536
+    refcount bits: 16
+    corrupt: false
+    extended l2: false
+
+image: TEST_DIR/t.IMGFMT.base
+file format: IMGFMT
+virtual size: 10 MiB (10485760 bytes)
+cluster_size: 65536
+Format specific information:
+    compat: 1.1
+    compression type: zlib
+    lazy refcounts: false
+    bitmaps:
+        [0]:
+            flags:
+                [0]: auto
+            name: b0
+            granularity: 65536
+        [1]:
+            flags:
+                [0]: auto
+            name: b3
+            granularity: 65536
+    refcount bits: 16
+    corrupt: false
+    extended l2: false
+
 === Check bitmap contents ===

 [{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
@@ -79,4 +132,7 @@ Format specific information:
 [{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
 { "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": false},
 { "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
+[{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": false},
+{ "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
 *** done
-- 
2.28.0



^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [PATCH] qemu-img: Support bitmap --merge into backing image
  2020-09-09 12:33 [PATCH] qemu-img: Support bitmap --merge into backing image Eric Blake
@ 2020-09-11  8:31 ` Max Reitz
  2020-09-11 15:17   ` Eric Blake
  0 siblings, 1 reply; 4+ messages in thread
From: Max Reitz @ 2020-09-11  8:31 UTC (permalink / raw)
  To: Eric Blake, qemu-devel; +Cc: Kevin Wolf, Eyal Shenitzky, qemu-block


[-- Attachment #1.1: Type: text/plain, Size: 2214 bytes --]

On 09.09.20 14:33, Eric Blake wrote:
> If you have the chain 'base.qcow2 <- top.qcow2' and want to merge a
> bitmap from top into base, qemu-img was failing with:
> 
> qemu-img: Could not open 'top.qcow2': Could not open backing file: Failed to get shared "write" lock
> Is another process using the image [base.qcow2]?
> 
> The easiest fix is to not open the entire backing chain of the source
> image, so that we aren't worrying about competing BDS visiting the
> backing image as both a read-only backing of the source and the
> writeable destination.
> 
> Fixes: http://bugzilla.redhat.com/1877209
> Reported-by: Eyal Shenitzky <eshenitz@redhat.com>
> Signed-off-by: Eric Blake <eblake@redhat.com>
> ---
>  qemu-img.c                 |  3 +-
>  tests/qemu-iotests/291     | 12 ++++++++
>  tests/qemu-iotests/291.out | 56 ++++++++++++++++++++++++++++++++++++++
>  3 files changed, 70 insertions(+), 1 deletion(-)
> 
> diff --git a/qemu-img.c b/qemu-img.c
> index eb2fc1f86243..b15098a2f9b3 100644
> --- a/qemu-img.c
> +++ b/qemu-img.c
> @@ -4755,7 +4755,8 @@ static int img_bitmap(int argc, char **argv)
>      }
>      bs = blk_bs(blk);
>      if (src_filename) {
> -        src = img_open(false, src_filename, src_fmt, 0, false, false, false);
> +        src = img_open(false, src_filename, src_fmt, BDRV_O_NO_BACKING,
> +                       false, false, false);

Why not do the same for the destination BB?

>          if (!src) {
>              goto out;
>          }
> diff --git a/tests/qemu-iotests/291 b/tests/qemu-iotests/291
> index 1e0bb76959bb..4f837b205655 100755
> --- a/tests/qemu-iotests/291
> +++ b/tests/qemu-iotests/291

[...]

> @@ -107,6 +116,9 @@ $QEMU_IMG map --output=json --image-opts \
>  nbd_server_start_unix_socket -r -f qcow2 -B b2 "$TEST_IMG"
>  $QEMU_IMG map --output=json --image-opts \
>      "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b2" | _filter_qemu_img_map
> +nbd_server_start_unix_socket -r -f qcow2 -B b3 "$TEST_IMG"

Why not look into $TEST_IMG.base to see specifically whether the bitmap
is there?

(I also am quite surprised that it’s even possible to export bitmaps
from backing nodes, but, well.)

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH] qemu-img: Support bitmap --merge into backing image
  2020-09-11  8:31 ` Max Reitz
@ 2020-09-11 15:17   ` Eric Blake
  2020-09-14  9:35     ` Max Reitz
  0 siblings, 1 reply; 4+ messages in thread
From: Eric Blake @ 2020-09-11 15:17 UTC (permalink / raw)
  To: Max Reitz, qemu-devel; +Cc: Kevin Wolf, Eyal Shenitzky, qemu-block

On 9/11/20 3:31 AM, Max Reitz wrote:
> On 09.09.20 14:33, Eric Blake wrote:
>> If you have the chain 'base.qcow2 <- top.qcow2' and want to merge a
>> bitmap from top into base, qemu-img was failing with:
>>
>> qemu-img: Could not open 'top.qcow2': Could not open backing file: Failed to get shared "write" lock
>> Is another process using the image [base.qcow2]?
>>
>> The easiest fix is to not open the entire backing chain of the source
>> image, so that we aren't worrying about competing BDS visiting the
>> backing image as both a read-only backing of the source and the
>> writeable destination.
>>
>> Fixes: http://bugzilla.redhat.com/1877209
>> Reported-by: Eyal Shenitzky <eshenitz@redhat.com>
>> Signed-off-by: Eric Blake <eblake@redhat.com>
>> ---
>>   qemu-img.c                 |  3 +-
>>   tests/qemu-iotests/291     | 12 ++++++++
>>   tests/qemu-iotests/291.out | 56 ++++++++++++++++++++++++++++++++++++++
>>   3 files changed, 70 insertions(+), 1 deletion(-)
>>
>> diff --git a/qemu-img.c b/qemu-img.c
>> index eb2fc1f86243..b15098a2f9b3 100644
>> --- a/qemu-img.c
>> +++ b/qemu-img.c
>> @@ -4755,7 +4755,8 @@ static int img_bitmap(int argc, char **argv)
>>       }
>>       bs = blk_bs(blk);
>>       if (src_filename) {
>> -        src = img_open(false, src_filename, src_fmt, 0, false, false, false);
>> +        src = img_open(false, src_filename, src_fmt, BDRV_O_NO_BACKING,
>> +                       false, false, false);
> 
> Why not do the same for the destination BB?

Yeah, that should work, too.

> 
>>           if (!src) {
>>               goto out;
>>           }
>> diff --git a/tests/qemu-iotests/291 b/tests/qemu-iotests/291
>> index 1e0bb76959bb..4f837b205655 100755
>> --- a/tests/qemu-iotests/291
>> +++ b/tests/qemu-iotests/291
> 
> [...]
> 
>> @@ -107,6 +116,9 @@ $QEMU_IMG map --output=json --image-opts \
>>   nbd_server_start_unix_socket -r -f qcow2 -B b2 "$TEST_IMG"
>>   $QEMU_IMG map --output=json --image-opts \
>>       "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b2" | _filter_qemu_img_map
>> +nbd_server_start_unix_socket -r -f qcow2 -B b3 "$TEST_IMG"
> 
> Why not look into $TEST_IMG.base to see specifically whether the bitmap
> is there?

We did just that, several lines earlier, with the qemu-img info 
--backing-chain.

> 
> (I also am quite surprised that it’s even possible to export bitmaps
> from backing nodes, but, well.)

I actually ought to call that out in the commit message.  It used to be 
that we were inconsistent on what we could see from the backing chain (a 
filter would make it so we can't), but as soon as your filter patches 
land, then we _do_ want to always be able to find a bitmap from the 
backing chain (incremental backup depends on that: we create an overlay 
disk to run the block-copy job as a filter, and _want_ to expose that 
overlay image with the bitmap it inherits from the original image).  So 
being able to export bitmaps from a backing node is normally a feature; 
and it is only in 'qemu-img bitmap' where we don't want accidental 
inheritance to get in the way from what we are actually merging.

I'll send a v2.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org



^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH] qemu-img: Support bitmap --merge into backing image
  2020-09-11 15:17   ` Eric Blake
@ 2020-09-14  9:35     ` Max Reitz
  0 siblings, 0 replies; 4+ messages in thread
From: Max Reitz @ 2020-09-14  9:35 UTC (permalink / raw)
  To: Eric Blake, qemu-devel; +Cc: Kevin Wolf, Eyal Shenitzky, qemu-block


[-- Attachment #1.1: Type: text/plain, Size: 3445 bytes --]

On 11.09.20 17:17, Eric Blake wrote:
> On 9/11/20 3:31 AM, Max Reitz wrote:
>> On 09.09.20 14:33, Eric Blake wrote:
>>> If you have the chain 'base.qcow2 <- top.qcow2' and want to merge a
>>> bitmap from top into base, qemu-img was failing with:
>>>
>>> qemu-img: Could not open 'top.qcow2': Could not open backing file:
>>> Failed to get shared "write" lock
>>> Is another process using the image [base.qcow2]?
>>>
>>> The easiest fix is to not open the entire backing chain of the source
>>> image, so that we aren't worrying about competing BDS visiting the
>>> backing image as both a read-only backing of the source and the
>>> writeable destination.
>>>
>>> Fixes: http://bugzilla.redhat.com/1877209
>>> Reported-by: Eyal Shenitzky <eshenitz@redhat.com>
>>> Signed-off-by: Eric Blake <eblake@redhat.com>
>>> ---
>>>   qemu-img.c                 |  3 +-
>>>   tests/qemu-iotests/291     | 12 ++++++++
>>>   tests/qemu-iotests/291.out | 56 ++++++++++++++++++++++++++++++++++++++
>>>   3 files changed, 70 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/qemu-img.c b/qemu-img.c
>>> index eb2fc1f86243..b15098a2f9b3 100644
>>> --- a/qemu-img.c
>>> +++ b/qemu-img.c
>>> @@ -4755,7 +4755,8 @@ static int img_bitmap(int argc, char **argv)
>>>       }
>>>       bs = blk_bs(blk);
>>>       if (src_filename) {
>>> -        src = img_open(false, src_filename, src_fmt, 0, false,
>>> false, false);
>>> +        src = img_open(false, src_filename, src_fmt, BDRV_O_NO_BACKING,
>>> +                       false, false, false);
>>
>> Why not do the same for the destination BB?
> 
> Yeah, that should work, too.
> 
>>
>>>           if (!src) {
>>>               goto out;
>>>           }
>>> diff --git a/tests/qemu-iotests/291 b/tests/qemu-iotests/291
>>> index 1e0bb76959bb..4f837b205655 100755
>>> --- a/tests/qemu-iotests/291
>>> +++ b/tests/qemu-iotests/291
>>
>> [...]
>>
>>> @@ -107,6 +116,9 @@ $QEMU_IMG map --output=json --image-opts \
>>>   nbd_server_start_unix_socket -r -f qcow2 -B b2 "$TEST_IMG"
>>>   $QEMU_IMG map --output=json --image-opts \
>>>       "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b2" | _filter_qemu_img_map
>>> +nbd_server_start_unix_socket -r -f qcow2 -B b3 "$TEST_IMG"
>>
>> Why not look into $TEST_IMG.base to see specifically whether the bitmap
>> is there?
> 
> We did just that, several lines earlier, with the qemu-img info
> --backing-chain.

OK, perfect.

>>
>> (I also am quite surprised that it’s even possible to export bitmaps
>> from backing nodes, but, well.)
> 
> I actually ought to call that out in the commit message.  It used to be
> that we were inconsistent on what we could see from the backing chain (a
> filter would make it so we can't), but as soon as your filter patches
> land, then we _do_ want to always be able to find a bitmap from the
> backing chain (incremental backup depends on that: we create an overlay
> disk to run the block-copy job as a filter, and _want_ to expose that
> overlay image with the bitmap it inherits from the original image).  So
> being able to export bitmaps from a backing node is normally a feature;
> and it is only in 'qemu-img bitmap' where we don't want accidental
> inheritance to get in the way from what we are actually merging.

Understood.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2020-09-14  9:36 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-09 12:33 [PATCH] qemu-img: Support bitmap --merge into backing image Eric Blake
2020-09-11  8:31 ` Max Reitz
2020-09-11 15:17   ` Eric Blake
2020-09-14  9:35     ` Max Reitz

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.