bitmaps -- copying allocation status into dirty bitmaps

bitmaps -- copying allocation status into dirty bitmaps

[John Snow]

Hi, in one of my infamously unreadable and long status emails, I
mentioned possibly wanting to copy allocation data into bitmaps as a way
to enable users to create (external) snapshots from outside of the
libvirt/qemu context.

(That is: to repair checkpoints in libvirt after a user extended the
backing chain themselves, you want to restore bitmap information for
that node. Conveniently, this information IS the allocation map, so we
can do this.)

It came up at KVM Forum that we probably do want this, because oVirt
likes the idea of being able to manipulate these chains from outside of
libvirt/qemu.

Denis suggested that instead of a new command, we can create a special
name -- maybe "#ALLOCATED" or something similar that can never be
allocated as a user-defined bitmap name -- as a special source for the
merge command.

You'd issue a merge from "#ALLOCATED" to "myBitmap0" to copy the current
allocation data into "myBitmap0", for instance.

Some thoughts:

- The only commands where this pseudo-bitmap makes sense is merge.
enable/disable/remove/clear/add don't make sense here.

- This pseudo bitmap might make sense for backup, but it's not needed;
you can just merge into an empty/enabled bitmap and then use that.

- Creating an allocation bitmap on-the-fly is probably not possible
directly in the merge command, because the disk status calls might take
too long...

Hm, actually, I'm not sure how to solve that one. Merge would need to
become a job (or an async QMP command?) or we'd need to keep an
allocation bitmap object around and in-sync. I don't really want to do
either, so maybe I'm missing an obvious/better solution.

Also, with regards to introspection, if we do create a special reserved
name like #ALLOCATED, we need to make sure that this is available and
obvious via the QAPI schema.

--js

Re: bitmaps -- copying allocation status into dirty bitmaps

[Denis V. Lunev]

On 11/1/19 4:42 PM, John Snow wrote:
> Hi, in one of my infamously unreadable and long status emails, I
> mentioned possibly wanting to copy allocation data into bitmaps as a way
> to enable users to create (external) snapshots from outside of the
> libvirt/qemu context.
>
> (That is: to repair checkpoints in libvirt after a user extended the
> backing chain themselves, you want to restore bitmap information for
> that node. Conveniently, this information IS the allocation map, so we
> can do this.)
>
> It came up at KVM Forum that we probably do want this, because oVirt
> likes the idea of being able to manipulate these chains from outside of
> libvirt/qemu.
>
> Denis suggested that instead of a new command, we can create a special
> name -- maybe "#ALLOCATED" or something similar that can never be
> allocated as a user-defined bitmap name -- as a special source for the
> merge command.
>
> You'd issue a merge from "#ALLOCATED" to "myBitmap0" to copy the current
> allocation data into "myBitmap0", for instance.
original problem was a little bit incorrect. After some thoughts I found
that this is NOT enough. We should also add zeroed clusters to the
bitmap to merge! They do cover some data clusters from the original
image.

Thus we should either provide "ALLOCATED" bitmap for other purposes,
and we should supply "CHANGED" which contains allocated AND
explicitly zeroed clusters.

Den
> Some thoughts:
>
> - The only commands where this pseudo-bitmap makes sense is merge.
> enable/disable/remove/clear/add don't make sense here.
>
> - This pseudo bitmap might make sense for backup, but it's not needed;
> you can just merge into an empty/enabled bitmap and then use that.
>
> - Creating an allocation bitmap on-the-fly is probably not possible
> directly in the merge command, because the disk status calls might take
> too long...
>
> Hm, actually, I'm not sure how to solve that one. Merge would need to
> become a job (or an async QMP command?) or we'd need to keep an
> allocation bitmap object around and in-sync. I don't really want to do
> either, so maybe I'm missing an obvious/better solution.
>
> Also, with regards to introspection, if we do create a special reserved
> name like #ALLOCATED, we need to make sure that this is available and
> obvious via the QAPI schema.
>
> --js
>

Re: bitmaps -- copying allocation status into dirty bitmaps

[Eric Blake]

On 11/1/19 4:49 PM, Denis V. Lunev wrote:
> On 11/1/19 4:42 PM, John Snow wrote:
>> Hi, in one of my infamously unreadable and long status emails, I
>> mentioned possibly wanting to copy allocation data into bitmaps as a way
>> to enable users to create (external) snapshots from outside of the
>> libvirt/qemu context.
>>
>> (That is: to repair checkpoints in libvirt after a user extended the
>> backing chain themselves, you want to restore bitmap information for
>> that node. Conveniently, this information IS the allocation map, so we
>> can do this.)
>>
>> It came up at KVM Forum that we probably do want this, because oVirt
>> likes the idea of being able to manipulate these chains from outside of
>> libvirt/qemu.
>>
>> Denis suggested that instead of a new command, we can create a special
>> name -- maybe "#ALLOCATED" or something similar that can never be
>> allocated as a user-defined bitmap name -- as a special source for the
>> merge command.
>>
>> You'd issue a merge from "#ALLOCATED" to "myBitmap0" to copy the current
>> allocation data into "myBitmap0", for instance.
> original problem was a little bit incorrect. After some thoughts I found
> that this is NOT enough. We should also add zeroed clusters to the
> bitmap to merge! They do cover some data clusters from the original
> image.
> 
> Thus we should either provide "ALLOCATED" bitmap for other purposes,
> and we should supply "CHANGED" which contains allocated AND
> explicitly zeroed clusters.

I'm also wondering if 'nbd-server-add' should add support to expose a 
'qemu:allocation:XXX' meta context, in addition to the existing 
'qemu:dirty-bitmap:XXX' contexts (would it just be a 0/1 bit for what is 
allocated in block layer XXX, or would it be an integer depth 0,1,2,... 
based on how deep in the chain a cluster is allocated, or?)

It may also be interesting to have a way to have 'nbd-server-add' force 
EIO errors to reads from an area not covered by a bitmap (whether that 
be the dirty bitmap or the #ALLOCATED bitmap), rather than falling back 
to a read from the backing chain, to ensure that an NBD client using 
such a backup image cannot read any more data than what the bitmap says 
is available.

> 
> Den
>> Some thoughts:
>>
>> - The only commands where this pseudo-bitmap makes sense is merge.
>> enable/disable/remove/clear/add don't make sense here.
>>
>> - This pseudo bitmap might make sense for backup, but it's not needed;
>> you can just merge into an empty/enabled bitmap and then use that.
>>
>> - Creating an allocation bitmap on-the-fly is probably not possible
>> directly in the merge command, because the disk status calls might take
>> too long...
>>
>> Hm, actually, I'm not sure how to solve that one. Merge would need to
>> become a job (or an async QMP command?) or we'd need to keep an
>> allocation bitmap object around and in-sync. I don't really want to do
>> either, so maybe I'm missing an obvious/better solution.
>>
>> Also, with regards to introspection, if we do create a special reserved
>> name like #ALLOCATED, we need to make sure that this is available and
>> obvious via the QAPI schema.

If nothing else, the recent addition of introspectible QMP features 
should make this possible.


-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: bitmaps -- copying allocation status into dirty bitmaps

[Vladimir Sementsov-Ogievskiy]

01.11.2019 18:49, Denis Lunev wrote:
> On 11/1/19 4:42 PM, John Snow wrote:
>> Hi, in one of my infamously unreadable and long status emails, I
>> mentioned possibly wanting to copy allocation data into bitmaps as a way
>> to enable users to create (external) snapshots from outside of the
>> libvirt/qemu context.
>>
>> (That is: to repair checkpoints in libvirt after a user extended the
>> backing chain themselves, you want to restore bitmap information for
>> that node. Conveniently, this information IS the allocation map, so we
>> can do this.)
>>
>> It came up at KVM Forum that we probably do want this, because oVirt
>> likes the idea of being able to manipulate these chains from outside of
>> libvirt/qemu.
>>
>> Denis suggested that instead of a new command, we can create a special
>> name -- maybe "#ALLOCATED" or something similar that can never be
>> allocated as a user-defined bitmap name -- as a special source for the
>> merge command.
>>
>> You'd issue a merge from "#ALLOCATED" to "myBitmap0" to copy the current
>> allocation data into "myBitmap0", for instance.
> original problem was a little bit incorrect. After some thoughts I found
> that this is NOT enough. We should also add zeroed clusters to the
> bitmap to merge! They do cover some data clusters from the original
> image.
> 
> Thus we should either provide "ALLOCATED" bitmap for other purposes,
> and we should supply "CHANGED" which contains allocated AND
> explicitly zeroed clusters.

Actually in terms of Qemu (bdrv_is_allocated) zeroed clusters (in qcow2)
are considered allocated.

BDRV_BLOCK_ALLOCATED is defined as

  * BDRV_BLOCK_ALLOCATED: the content of the block is determined by this
  *                       layer rather than any backing, set by block layer

- so, it's nothing about allocated regions on host disk, it's an internal
concept about is data (or zero) in this layer or in backing.

> 
> Den
>> Some thoughts:
>>
>> - The only commands where this pseudo-bitmap makes sense is merge.
>> enable/disable/remove/clear/add don't make sense here.
>>
>> - This pseudo bitmap might make sense for backup, but it's not needed;
>> you can just merge into an empty/enabled bitmap and then use that.
>>
>> - Creating an allocation bitmap on-the-fly is probably not possible
>> directly in the merge command, because the disk status calls might take
>> too long...
>>
>> Hm, actually, I'm not sure how to solve that one. Merge would need to
>> become a job (or an async QMP command?) or we'd need to keep an
>> allocation bitmap object around and in-sync. I don't really want to do
>> either, so maybe I'm missing an obvious/better solution.
>>
>> Also, with regards to introspection, if we do create a special reserved
>> name like #ALLOCATED, we need to make sure that this is available and
>> obvious via the QAPI schema.
>>
>> --js
>>
> 


-- 
Best regards,
Vladimir

Re: bitmaps -- copying allocation status into dirty bitmaps

[Vladimir Sementsov-Ogievskiy]

01.11.2019 18:42, John Snow wrote:
> Hi, in one of my infamously unreadable and long status emails, I
> mentioned possibly wanting to copy allocation data into bitmaps as a way
> to enable users to create (external) snapshots from outside of the
> libvirt/qemu context.
> 
> (That is: to repair checkpoints in libvirt after a user extended the
> backing chain themselves, you want to restore bitmap information for
> that node. Conveniently, this information IS the allocation map, so we
> can do this.)
> 
> It came up at KVM Forum that we probably do want this, because oVirt
> likes the idea of being able to manipulate these chains from outside of
> libvirt/qemu.
> 
> Denis suggested that instead of a new command, we can create a special
> name -- maybe "#ALLOCATED" or something similar that can never be
> allocated as a user-defined bitmap name -- as a special source for the
> merge command.
> 
> You'd issue a merge from "#ALLOCATED" to "myBitmap0" to copy the current
> allocation data into "myBitmap0", for instance.

I can suggest also the following:

We now have

   { 'alternate': 'BlockDirtyBitmapMergeSource',
     'data': { 'local': 'str',
               'external': 'BlockDirtyBitmap' } }

We may add separate qapy enum

BlockStatusBitmapType = {'allocated', ...}

and struct

BlockStatusBitmap = {'node': 'str', 'type': BlockStatusBitmapType}

and then extend BlockDirtyBitmapMergeSource like

   { 'alternate': 'BlockDirtyBitmapMergeSource',
     'data': { 'local': 'str',
               'external': 'BlockDirtyBitmap',
               'status': 'BlockStatusBitmap' } }

So, we can avoid special name "#ALLOCATED", which we'll have to define
in qcow2 spec and nbd spec.

> 
> Some thoughts:
> 
> - The only commands where this pseudo-bitmap makes sense is merge.
> enable/disable/remove/clear/add don't make sense here.
> 
> - This pseudo bitmap might make sense for backup, but it's not needed;
> you can just merge into an empty/enabled bitmap and then use that.
> 
> - Creating an allocation bitmap on-the-fly is probably not possible
> directly in the merge command, because the disk status calls might take
> too long...
> 
> Hm, actually, I'm not sure how to solve that one. Merge would need to
> become a job (or an async QMP command?) or we'd need to keep an
> allocation bitmap object around and in-sync. I don't really want to do
> either, so maybe I'm missing an obvious/better solution.
> 
> Also, with regards to introspection, if we do create a special reserved
> name like #ALLOCATED, we need to make sure that this is available and
> obvious via the QAPI schema.
> 
> --js
> 


-- 
Best regards,
Vladimir

Re: bitmaps -- copying allocation status into dirty bitmaps

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 2442 bytes --]

On 01.11.19 16:42, John Snow wrote:
> Hi, in one of my infamously unreadable and long status emails, I
> mentioned possibly wanting to copy allocation data into bitmaps as a way
> to enable users to create (external) snapshots from outside of the
> libvirt/qemu context.
> 
> (That is: to repair checkpoints in libvirt after a user extended the
> backing chain themselves, you want to restore bitmap information for
> that node. Conveniently, this information IS the allocation map, so we
> can do this.)
> 
> It came up at KVM Forum that we probably do want this, because oVirt
> likes the idea of being able to manipulate these chains from outside of
> libvirt/qemu.
> 
> Denis suggested that instead of a new command, we can create a special
> name -- maybe "#ALLOCATED" or something similar that can never be
> allocated as a user-defined bitmap name -- as a special source for the
> merge command.
> 
> You'd issue a merge from "#ALLOCATED" to "myBitmap0" to copy the current
> allocation data into "myBitmap0", for instance.

Sounds fun, but is there actually any use for this if the only purpose
is to work as a source for merge?

I mean, it would be interesting if it worked exactly like a perma-RO
pseudo-bitmap that whenever you try to get data from it performs a
block-status call.  But as you say, that would probably be too slow, and
it would take a lot of code modifications, so I wonder if there is
actually any purpose for this.

> Some thoughts:
> 
> - The only commands where this pseudo-bitmap makes sense is merge.
> enable/disable/remove/clear/add don't make sense here.
> 
> - This pseudo bitmap might make sense for backup, but it's not needed;
> you can just merge into an empty/enabled bitmap and then use that.
> 
> - Creating an allocation bitmap on-the-fly is probably not possible
> directly in the merge command, because the disk status calls might take
> too long...
> 
> Hm, actually, I'm not sure how to solve that one. Merge would need to
> become a job (or an async QMP command?) or we'd need to keep an
> allocation bitmap object around and in-sync. I don't really want to do
> either, so maybe I'm missing an obvious/better solution.

All of what you wrote in this mail makes me think it would make much
more sense to just add a “block-dirty-bitmap-create-from” job with an
enum of targets.  (One of which would be “allocated-blocks”.)

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: bitmaps -- copying allocation status into dirty bitmaps

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 2899 bytes --]

On 04.11.19 12:21, Max Reitz wrote:
> On 01.11.19 16:42, John Snow wrote:
>> Hi, in one of my infamously unreadable and long status emails, I
>> mentioned possibly wanting to copy allocation data into bitmaps as a way
>> to enable users to create (external) snapshots from outside of the
>> libvirt/qemu context.
>>
>> (That is: to repair checkpoints in libvirt after a user extended the
>> backing chain themselves, you want to restore bitmap information for
>> that node. Conveniently, this information IS the allocation map, so we
>> can do this.)
>>
>> It came up at KVM Forum that we probably do want this, because oVirt
>> likes the idea of being able to manipulate these chains from outside of
>> libvirt/qemu.
>>
>> Denis suggested that instead of a new command, we can create a special
>> name -- maybe "#ALLOCATED" or something similar that can never be
>> allocated as a user-defined bitmap name -- as a special source for the
>> merge command.
>>
>> You'd issue a merge from "#ALLOCATED" to "myBitmap0" to copy the current
>> allocation data into "myBitmap0", for instance.
> 
> Sounds fun, but is there actually any use for this if the only purpose
> is to work as a source for merge?
> 
> I mean, it would be interesting if it worked exactly like a perma-RO
> pseudo-bitmap that whenever you try to get data from it performs a
> block-status call.  But as you say, that would probably be too slow, and
> it would take a lot of code modifications, so I wonder if there is
> actually any purpose for this.
> 
>> Some thoughts:
>>
>> - The only commands where this pseudo-bitmap makes sense is merge.
>> enable/disable/remove/clear/add don't make sense here.
>>
>> - This pseudo bitmap might make sense for backup, but it's not needed;
>> you can just merge into an empty/enabled bitmap and then use that.
>>
>> - Creating an allocation bitmap on-the-fly is probably not possible
>> directly in the merge command, because the disk status calls might take
>> too long...
>>
>> Hm, actually, I'm not sure how to solve that one. Merge would need to
>> become a job (or an async QMP command?) or we'd need to keep an
>> allocation bitmap object around and in-sync. I don't really want to do
>> either, so maybe I'm missing an obvious/better solution.
> 
> All of what you wrote in this mail makes me think it would make much
> more sense to just add a “block-dirty-bitmap-create-from” job with an
> enum of targets.  (One of which would be “allocated-blocks”.)

I forgot to add that of course the advantage of a pseudo-bitmap would be
that it’s always up to date, but as you said, it would be slow to query
(and it might even yield, which isn’t what callers expect) and at least
for block allocation, it seems unnecessary to me (because writes will
keep the new bitmap created from allocated-blocks up-to-date).

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: bitmaps -- copying allocation status into dirty bitmaps

[John Snow]

On 11/4/19 6:27 AM, Max Reitz wrote:
> On 04.11.19 12:21, Max Reitz wrote:
>> On 01.11.19 16:42, John Snow wrote:
>>> Hi, in one of my infamously unreadable and long status emails, I
>>> mentioned possibly wanting to copy allocation data into bitmaps as a way
>>> to enable users to create (external) snapshots from outside of the
>>> libvirt/qemu context.
>>>
>>> (That is: to repair checkpoints in libvirt after a user extended the
>>> backing chain themselves, you want to restore bitmap information for
>>> that node. Conveniently, this information IS the allocation map, so we
>>> can do this.)
>>>
>>> It came up at KVM Forum that we probably do want this, because oVirt
>>> likes the idea of being able to manipulate these chains from outside of
>>> libvirt/qemu.
>>>
>>> Denis suggested that instead of a new command, we can create a special
>>> name -- maybe "#ALLOCATED" or something similar that can never be
>>> allocated as a user-defined bitmap name -- as a special source for the
>>> merge command.
>>>
>>> You'd issue a merge from "#ALLOCATED" to "myBitmap0" to copy the current
>>> allocation data into "myBitmap0", for instance.
>>
>> Sounds fun, but is there actually any use for this if the only purpose
>> is to work as a source for merge?
>>
>> I mean, it would be interesting if it worked exactly like a perma-RO
>> pseudo-bitmap that whenever you try to get data from it performs a
>> block-status call.  But as you say, that would probably be too slow, and
>> it would take a lot of code modifications, so I wonder if there is
>> actually any purpose for this.
>>
>>> Some thoughts:
>>>
>>> - The only commands where this pseudo-bitmap makes sense is merge.
>>> enable/disable/remove/clear/add don't make sense here.
>>>
>>> - This pseudo bitmap might make sense for backup, but it's not needed;
>>> you can just merge into an empty/enabled bitmap and then use that.
>>>
>>> - Creating an allocation bitmap on-the-fly is probably not possible
>>> directly in the merge command, because the disk status calls might take
>>> too long...
>>>
>>> Hm, actually, I'm not sure how to solve that one. Merge would need to
>>> become a job (or an async QMP command?) or we'd need to keep an
>>> allocation bitmap object around and in-sync. I don't really want to do
>>> either, so maybe I'm missing an obvious/better solution.
>>
>> All of what you wrote in this mail makes me think it would make much
>> more sense to just add a “block-dirty-bitmap-create-from” job with an
>> enum of targets.  (One of which would be “allocated-blocks”.)

Sounds good. (What are the other targets? Questions-for-later?)

> 
> I forgot to add that of course the advantage of a pseudo-bitmap would be
> that it’s always up to date, but as you said, it would be slow to query
> (and it might even yield, which isn’t what callers expect) and at least
> for block allocation, it seems unnecessary to me (because writes will
> keep the new bitmap created from allocated-blocks up-to-date).
> 
> Max
> 

Who knows what's happened in the month since I've been gone, but I think
I agree completely with your assessment.

In our meeting with Denis it seemed like it was the optimal thing to
make a pseudo-bitmap for merge so we didn't have to add a new command,
but I think it's clear that the async properties are going to prohibit
that nice solution and we will indeed need a job.

--js