[Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Qemu-devel] Persistent bitmaps for non-qcow2 formats

[John Snow]

Well, we knew we'd want this sooner or later. I've got some pings
downstream over whether or not we support persistent bitmaps for
non-qcow2 formats.

Currently: no, we don't.

We tried two different ideas for storage agnostic bitmap persistence:


(1) Using qcow2 as a storage container format (similar to VM save
states) for information not associated with that particular drive.

This didn't go over so well; we decided it was too messy logistically to
manage data in any meaningful sense in a file that didn't share any
semantic relationship to the qcow2 in question.

Well, "We" decided is more like "Kevin and Max" decided. They are right,
though.


(2) Using a new proto-wrapper format that Fam Zheng designed that serves
as a simple top-layer redirect for metadata that allows us to associate
raw bitmap data with an arbitrary backing image.

This also didn't go over so well, the desire for a "new format" was
pretty thin, even if it was only a pseudo-format.


We'd still like to be able to use bitmaps with non-qcow2 formats
however, and people are going to keep asking. So here's a third thought:


(3) Add either a new flag that turns qcow2's backing file into a full
R/W backing file, or add a new extension to qcow2 entirely (bypassing
the traditional backing file mechanism to avoid confusion for older
tooling) that adds a new read-write backing file field.

This RW backing file field will be used for all reads AND writes; the
qcow2 in question becomes a metadata container on top of the BDS chain.
We can re-use Vladimir's bitmap persistence extension to save bitmaps in
a qcow2 shell.

The qcow2 becomes effectively a metadata cache for a new (essentially)
filter node that handles features such as bitmaps. This could also be
used to provide allocation map data for RAW files and other goodies down
the road.

Hopefully this achieves our desire to not create new formats AND our
desire to concentrate features (and debugging, testing, etc) into qcow2,
while allowing users to "have bitmaps with raw files."

Of course, in this scenario, users now have two files: a qcow2 wrapper
and the actual raw file in question; but regardless of how we were going
to solve this, a raw file necessitates an external file of some sort,
else we give up the idea that it was a raw file.


Sound good?

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Vladimir Sementsov-Ogievskiy]

22.08.2017 22:07, John Snow wrote:
> Well, we knew we'd want this sooner or later. I've got some pings
> downstream over whether or not we support persistent bitmaps for
> non-qcow2 formats.
>
> Currently: no, we don't.
>
> We tried two different ideas for storage agnostic bitmap persistence:
>
>
> (1) Using qcow2 as a storage container format (similar to VM save
> states) for information not associated with that particular drive.
>
> This didn't go over so well; we decided it was too messy logistically to
> manage data in any meaningful sense in a file that didn't share any
> semantic relationship to the qcow2 in question.
>
> Well, "We" decided is more like "Kevin and Max" decided. They are right,
> though.
>
>
> (2) Using a new proto-wrapper format that Fam Zheng designed that serves
> as a simple top-layer redirect for metadata that allows us to associate
> raw bitmap data with an arbitrary backing image.
>
> This also didn't go over so well, the desire for a "new format" was
> pretty thin, even if it was only a pseudo-format.
>
>
> We'd still like to be able to use bitmaps with non-qcow2 formats
> however, and people are going to keep asking. So here's a third thought:
>
>
> (3) Add either a new flag that turns qcow2's backing file into a full
> R/W backing file, or add a new extension to qcow2 entirely (bypassing
> the traditional backing file mechanism to avoid confusion for older
> tooling) that adds a new read-write backing file field.
>
> This RW backing file field will be used for all reads AND writes; the
> qcow2 in question becomes a metadata container on top of the BDS chain.
> We can re-use Vladimir's bitmap persistence extension to save bitmaps in
> a qcow2 shell.
>
> The qcow2 becomes effectively a metadata cache for a new (essentially)
> filter node that handles features such as bitmaps. This could also be
> used to provide allocation map data for RAW files and other goodies down
> the road.
>
> Hopefully this achieves our desire to not create new formats AND our
> desire to concentrate features (and debugging, testing, etc) into qcow2,
> while allowing users to "have bitmaps with raw files."
>
> Of course, in this scenario, users now have two files: a qcow2 wrapper
> and the actual raw file in question; but regardless of how we were going
> to solve this, a raw file necessitates an external file of some sort,
> else we give up the idea that it was a raw file.
>
>
> Sound good?

Looks interesting. It is a clean reusing of qcow2-bitmaps without any 
modifications to them.

Should there be some problems with internal snapshots and other things?



-- 
Best regards,
Vladimir

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 1745 bytes --]

On 2017-08-22 21:07, John Snow wrote:

[...]

> (3) Add either a new flag that turns qcow2's backing file into a full
> R/W backing file, or add a new extension to qcow2 entirely (bypassing
> the traditional backing file mechanism to avoid confusion for older
> tooling) that adds a new read-write backing file field.
> 
> This RW backing file field will be used for all reads AND writes; the
> qcow2 in question becomes a metadata container on top of the BDS chain.
> We can re-use Vladimir's bitmap persistence extension to save bitmaps in
> a qcow2 shell.
> 
> The qcow2 becomes effectively a metadata cache for a new (essentially)
> filter node that handles features such as bitmaps. This could also be
> used to provide allocation map data for RAW files and other goodies down
> the road.
> 
> Hopefully this achieves our desire to not create new formats AND our
> desire to concentrate features (and debugging, testing, etc) into qcow2,
> while allowing users to "have bitmaps with raw files."
> 
> Of course, in this scenario, users now have two files: a qcow2 wrapper
> and the actual raw file in question; but regardless of how we were going
> to solve this, a raw file necessitates an external file of some sort,
> else we give up the idea that it was a raw file.
> 
> 
> Sound good?

While I don't quite like the idea of R/W backing files, I guess "don't
quite like" is still rather good.

I like how this idea would allow us to use the existing code without
putting just arbitrary binary data into the qcow2 file; the data still
relates to the virtual disk represented by the qcow2 image.

So design-wise, I don't think I have any complaints.

As for Kevin, he's on PTO, though. ;-)

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[John Snow]

On 08/23/2017 01:31 PM, Max Reitz wrote:
> On 2017-08-22 21:07, John Snow wrote:
> 
> [...]
> 
>> (3) Add either a new flag that turns qcow2's backing file into a full
>> R/W backing file, or add a new extension to qcow2 entirely (bypassing
>> the traditional backing file mechanism to avoid confusion for older
>> tooling) that adds a new read-write backing file field.
>>
>> This RW backing file field will be used for all reads AND writes; the
>> qcow2 in question becomes a metadata container on top of the BDS chain.
>> We can re-use Vladimir's bitmap persistence extension to save bitmaps in
>> a qcow2 shell.
>>
>> The qcow2 becomes effectively a metadata cache for a new (essentially)
>> filter node that handles features such as bitmaps. This could also be
>> used to provide allocation map data for RAW files and other goodies down
>> the road.
>>
>> Hopefully this achieves our desire to not create new formats AND our
>> desire to concentrate features (and debugging, testing, etc) into qcow2,
>> while allowing users to "have bitmaps with raw files."
>>
>> Of course, in this scenario, users now have two files: a qcow2 wrapper
>> and the actual raw file in question; but regardless of how we were going
>> to solve this, a raw file necessitates an external file of some sort,
>> else we give up the idea that it was a raw file.
>>
>>
>> Sound good?
> 
> While I don't quite like the idea of R/W backing files, I guess "don't
> quite like" is still rather good.
> 

Yeah, it's not necessarily my first pick, but it might be the least bad.
If you have any suggestions or alternatives for a way to accomplish the
same in a way that does not leave any, even faint, bad taste in your
mouth you are more than welcome to suggest it.

> I like how this idea would allow us to use the existing code without
> putting just arbitrary binary data into the qcow2 file; the data still
> relates to the virtual disk represented by the qcow2 image.
> 
> So design-wise, I don't think I have any complaints.
> 
> As for Kevin, he's on PTO, though. ;-)
> 
> Max
> 

Ah, right. Well... I'll just have to have something fun for him to look at.

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Vladimir Sementsov-Ogievskiy]

23.08.2017 11:59, Vladimir Sementsov-Ogievskiy wrote:
> 22.08.2017 22:07, John Snow wrote:
>> Well, we knew we'd want this sooner or later. I've got some pings
>> downstream over whether or not we support persistent bitmaps for
>> non-qcow2 formats.
>>
>> Currently: no, we don't.
>>
>> We tried two different ideas for storage agnostic bitmap persistence:
>>
>>
>> (1) Using qcow2 as a storage container format (similar to VM save
>> states) for information not associated with that particular drive.
>>
>> This didn't go over so well; we decided it was too messy logistically to
>> manage data in any meaningful sense in a file that didn't share any
>> semantic relationship to the qcow2 in question.
>>
>> Well, "We" decided is more like "Kevin and Max" decided. They are right,
>> though.
>>
>>
>> (2) Using a new proto-wrapper format that Fam Zheng designed that serves
>> as a simple top-layer redirect for metadata that allows us to associate
>> raw bitmap data with an arbitrary backing image.
>>
>> This also didn't go over so well, the desire for a "new format" was
>> pretty thin, even if it was only a pseudo-format.
>>
>>
>> We'd still like to be able to use bitmaps with non-qcow2 formats
>> however, and people are going to keep asking. So here's a third thought:
>>
>>
>> (3) Add either a new flag that turns qcow2's backing file into a full
>> R/W backing file, or add a new extension to qcow2 entirely (bypassing
>> the traditional backing file mechanism to avoid confusion for older
>> tooling) that adds a new read-write backing file field.
>>
>> This RW backing file field will be used for all reads AND writes; the
>> qcow2 in question becomes a metadata container on top of the BDS chain.
>> We can re-use Vladimir's bitmap persistence extension to save bitmaps in
>> a qcow2 shell.
>>
>> The qcow2 becomes effectively a metadata cache for a new (essentially)
>> filter node that handles features such as bitmaps. This could also be
>> used to provide allocation map data for RAW files and other goodies down
>> the road.
>>
>> Hopefully this achieves our desire to not create new formats AND our
>> desire to concentrate features (and debugging, testing, etc) into qcow2,
>> while allowing users to "have bitmaps with raw files."
>>
>> Of course, in this scenario, users now have two files: a qcow2 wrapper
>> and the actual raw file in question; but regardless of how we were going
>> to solve this, a raw file necessitates an external file of some sort,
>> else we give up the idea that it was a raw file.
>>
>>
>> Sound good?
>
> Looks interesting. It is a clean reusing of qcow2-bitmaps without any 
> modifications to them.
>
> Should there be some problems with internal snapshots and other things?
>
>
>
Hm. looks like that this backing file should not only receive all reads 
and writes, but almost everything ->bdrv_ handlers, except bitmap 
related of course. This doesn't seems simple to implement. Especially if 
imaging some not-raw feature-full format under this thin qcow2 layer.. 
Or we can restrict this RW backing file to be raw-only?


-- 
Best regards,
Vladimir

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Vladimir Sementsov-Ogievskiy]

23.08.2017 21:04, Vladimir Sementsov-Ogievskiy wrote:
> 23.08.2017 11:59, Vladimir Sementsov-Ogievskiy wrote:
>> 22.08.2017 22:07, John Snow wrote:
>>> Well, we knew we'd want this sooner or later. I've got some pings
>>> downstream over whether or not we support persistent bitmaps for
>>> non-qcow2 formats.
>>>
>>> Currently: no, we don't.
>>>
>>> We tried two different ideas for storage agnostic bitmap persistence:
>>>
>>>
>>> (1) Using qcow2 as a storage container format (similar to VM save
>>> states) for information not associated with that particular drive.
>>>
>>> This didn't go over so well; we decided it was too messy 
>>> logistically to
>>> manage data in any meaningful sense in a file that didn't share any
>>> semantic relationship to the qcow2 in question.
>>>
>>> Well, "We" decided is more like "Kevin and Max" decided. They are 
>>> right,
>>> though.
>>>
>>>
>>> (2) Using a new proto-wrapper format that Fam Zheng designed that 
>>> serves
>>> as a simple top-layer redirect for metadata that allows us to associate
>>> raw bitmap data with an arbitrary backing image.
>>>
>>> This also didn't go over so well, the desire for a "new format" was
>>> pretty thin, even if it was only a pseudo-format.
>>>
>>>
>>> We'd still like to be able to use bitmaps with non-qcow2 formats
>>> however, and people are going to keep asking. So here's a third 
>>> thought:
>>>
>>>
>>> (3) Add either a new flag that turns qcow2's backing file into a full
>>> R/W backing file, or add a new extension to qcow2 entirely (bypassing
>>> the traditional backing file mechanism to avoid confusion for older
>>> tooling) that adds a new read-write backing file field.
>>>
>>> This RW backing file field will be used for all reads AND writes; the
>>> qcow2 in question becomes a metadata container on top of the BDS chain.
>>> We can re-use Vladimir's bitmap persistence extension to save 
>>> bitmaps in
>>> a qcow2 shell.
>>>
>>> The qcow2 becomes effectively a metadata cache for a new (essentially)
>>> filter node that handles features such as bitmaps. This could also be
>>> used to provide allocation map data for RAW files and other goodies 
>>> down
>>> the road.
>>>
>>> Hopefully this achieves our desire to not create new formats AND our
>>> desire to concentrate features (and debugging, testing, etc) into 
>>> qcow2,
>>> while allowing users to "have bitmaps with raw files."
>>>
>>> Of course, in this scenario, users now have two files: a qcow2 wrapper
>>> and the actual raw file in question; but regardless of how we were 
>>> going
>>> to solve this, a raw file necessitates an external file of some sort,
>>> else we give up the idea that it was a raw file.
>>>
>>>
>>> Sound good?
>>
>> Looks interesting. It is a clean reusing of qcow2-bitmaps without any 
>> modifications to them.
>>
>> Should there be some problems with internal snapshots and other things?
>>
>>
>>
> Hm. looks like that this backing file should not only receive all 
> reads and writes, but almost everything ->bdrv_ handlers, except 
> bitmap related of course. This doesn't seems simple to implement. 
> Especially if imaging some not-raw feature-full format under this thin 
> qcow2 layer.. Or we can restrict this RW backing file to be raw-only?
>
>

So, anyway, I see only two differences (from the outside) between this 
approach and just a separate bitmap-only qcow2 without a data:

1. in RW-backing approach qcow2-bitmap file has a link to data file (as 
a backing). It looks good.

2. in RW-backing approach qcow2-bitmap file is a top of the virtual 
disk, in separate-file approach it is an option of the real data drive. 
In my opinion the second is more clean for users ("to add this feature 
you should use other file as your disk" vs "to add this feature you 
should add an option to your disk description")


I think (may be I'm not right) loading bitmaps from additional 
qcow2-only-bitmaps file is simpler to implement (it will be specified in 
command line in drive options, like bitmaps_file=/path/to/it and then 
attached directly to BlockDriverState). The only drawback of simple 
qcow2-bitmap file  is that it has not a link inside it to the data file 
(like shabacking). We can ignore it, or we can implement this link as a 
separate extension to qcow2.

-- 
Best regards,
Vladimir

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[John Snow]

Sorry in advance for :words: ...

On 08/23/2017 02:04 PM, Vladimir Sementsov-Ogievskiy wrote:
> 23.08.2017 11:59, Vladimir Sementsov-Ogievskiy wrote:
>> 22.08.2017 22:07, John Snow wrote:
>>> Well, we knew we'd want this sooner or later. I've got some pings
>>> downstream over whether or not we support persistent bitmaps for
>>> non-qcow2 formats.
>>>
>>> Currently: no, we don't.
>>>
>>> We tried two different ideas for storage agnostic bitmap persistence:
>>>
>>>
>>> (1) Using qcow2 as a storage container format (similar to VM save
>>> states) for information not associated with that particular drive.
>>>
>>> This didn't go over so well; we decided it was too messy logistically to
>>> manage data in any meaningful sense in a file that didn't share any
>>> semantic relationship to the qcow2 in question.
>>>
>>> Well, "We" decided is more like "Kevin and Max" decided. They are right,
>>> though.
>>>
>>>
>>> (2) Using a new proto-wrapper format that Fam Zheng designed that serves
>>> as a simple top-layer redirect for metadata that allows us to associate
>>> raw bitmap data with an arbitrary backing image.
>>>
>>> This also didn't go over so well, the desire for a "new format" was
>>> pretty thin, even if it was only a pseudo-format.
>>>
>>>
>>> We'd still like to be able to use bitmaps with non-qcow2 formats
>>> however, and people are going to keep asking. So here's a third thought:
>>>
>>>
>>> (3) Add either a new flag that turns qcow2's backing file into a full
>>> R/W backing file, or add a new extension to qcow2 entirely (bypassing
>>> the traditional backing file mechanism to avoid confusion for older
>>> tooling) that adds a new read-write backing file field.
>>>
>>> This RW backing file field will be used for all reads AND writes; the
>>> qcow2 in question becomes a metadata container on top of the BDS chain.
>>> We can re-use Vladimir's bitmap persistence extension to save bitmaps in
>>> a qcow2 shell.
>>>
>>> The qcow2 becomes effectively a metadata cache for a new (essentially)
>>> filter node that handles features such as bitmaps. This could also be
>>> used to provide allocation map data for RAW files and other goodies down
>>> the road.
>>>
>>> Hopefully this achieves our desire to not create new formats AND our
>>> desire to concentrate features (and debugging, testing, etc) into qcow2,
>>> while allowing users to "have bitmaps with raw files."
>>>
>>> Of course, in this scenario, users now have two files: a qcow2 wrapper
>>> and the actual raw file in question; but regardless of how we were going
>>> to solve this, a raw file necessitates an external file of some sort,
>>> else we give up the idea that it was a raw file.
>>>
>>>
>>> Sound good?
>>
>> Looks interesting. It is a clean reusing of qcow2-bitmaps without any
>> modifications to them.
>>
>> Should there be some problems with internal snapshots and other things?
>>
>>
>>
> Hm. looks like that this backing file should not only receive all reads
> and writes, but almost everything ->bdrv_ handlers, except bitmap
> related of course. This doesn't seems simple to implement. Especially if
> imaging some not-raw feature-full format under this thin qcow2 layer..
> Or we can restrict this RW backing file to be raw-only?
> 
> 

The idea would really be to support any arbitrary data store, so I
wouldn't want to restrict it to just raw.

You're right though, this might be a kind of messy approach.

[From your other mail:]

> 
> So, anyway, I see only two differences (from the outside) between this approach and just a separate bitmap-only qcow2 without a data:
> 

It's very delicately similar, yes.

> 1. in RW-backing approach qcow2-bitmap file has a link to data file (as a backing). It looks good.
> 

Right. The information necessary to establish a link between the bitmap
data and the data being described is fully contained within a file fully
specified by the QCOW2 spec.

> 2. in RW-backing approach qcow2-bitmap file is a top of the virtual disk, in separate-file approach it is an option of the real data drive. In my opinion the second is more clean for users ("to add this feature you should use other file as your disk" vs "to add this feature you should add an option to your disk description")

This puts us a little closer to the original idea that was rejected by
Kevin at the time. To recap:

"1": Use qcow2 as a container. This was rejected because we didn't want
qcow2 containing data with no semantic relationship to the qcow2
container or to each other. The way it sounds like you're proposing it,
though, it would be one-qcow2-with-bitmaps-per-drive, so the data would
at least stay strictly related, but it would be meaningless outside of
QEMU itself. I think this is something that Kevin wanted to avoid, but I
can't speak for him.

It's certainly not beyond the realm of management software to remember
to correlate a qcow2 metadata file alongside its actual data stores
whenever it needed to do so, but it does mean the introduction of a
feature that essentially requires the use of management software, which
sees resistance in the community at times.

In this model, you'd probably have the raw drive at the top, with the
qcow2-with-bitmaps as a child node with some kind of new named child
relationship. All IO stays at the root node, but the bitmap method
handlers would know to look for this special bitmap-child. It shouldn't
be too hard to implement.

> 
> 
> I think (may be I'm not right) loading bitmaps from additional qcow2-only-bitmaps file is simpler to implement (it will be specified in command line in drive options, like bitmaps_file=/path/to/it and then attached directly to BlockDriverState). The only drawback of simple qcow2-bitmap file  is that it has not a link inside it to the data file (like shabacking). We can ignore it, or we can implement this link as a separate extension to qcow2.

Yes, definitely easier to implement as you say.

The hard part is going to come in defining that semantic link. At this
point, the only difference between the approaches is whether or not we
allow the qcow2 to point to the implementation of the data;

(1) The qcow2 is referenced by name from the CLI as an option to the
other drive.

(2) The qcow2 is referenced by name on the CLI, and its backing file
field intuits the location of the implementation storage.


In (1), we avoid saving or specifying the relationship between these two
data stores in any way. This is certainly easy to do, and will save us
some headache on the CLI. As a downside, we now have random orphaned
files that aren't very interesting or useful on their own. The
likelihood for desync between metadata and data increases. The use of
management software is all but necessitated.

In (2) We have to now specify, with a dizzying long list of
possibilities, the location of the implementation data. qcow2 only has a
filename for backing files presently, but this is likely inadequate.
What if the data store isn't a locally kept file? What if it's a socket,
or a stream, or literally anything else?

We'd have to develop a new syntax for specifying these resources that
can be stored in a qcow2 file, or otherwise co-opt an existing syntax
in-use by QEMU. This syntax would likely be useful only to QEMU, which
would steer the qcow2 format in a direction not too useful by other
emulators, and qcow2 is an open format, so we may want to avoid this.





I feel like what will make the difference between heading down either
path would be helped along by answers to these questions:

- What type of data stores do we wish to support with bitmaps? Simple
file-based ones, or the full spectrum of all types? Only qcow2, and to
hell with people who ask for otherwise?

- How important is it that the qcow2 remains a fully independent file
capable of describing its own relationship to the data?

- Is it OK to allow robust, QEMU-specific data descriptors in a qcow2 file?


Where I sit:

- I'd like to support truly arbitrary protocols and formats. We can add
transient / non-persistent bitmaps to any of these today, so I'd like to
maintain that flexibility.

- I think it will be difficult or impossible to agree upon a
specification for the syntax to store the backing storage identifier in
the qcow2 itself, because we'd like to let third party tools make use of
qcow2 files.

- I don't think Kevin will like the idea of us using qcow2 as a
container that does not get treated as a first class citizen in the
backing chain, but it's the easiest option to implement and avoids a lot
of the syntax-of-backing-storage questions.

- There's a definite usability factor to having the qcow2 just specify
where the rest of the data sits like it does for normal backing files,
but with what I am proposing, this complicates the open() call
considerably, not to mention the syntactic issues addressed above.



I guess I've talked myself out of wanting a "RW backing file" because it
seems hard to implement, spec-wise ...

--js

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Vladimir Sementsov-Ogievskiy]

25.08.2017 03:55, John Snow wrote:
> Sorry in advance for :words: ...
>
> On 08/23/2017 02:04 PM, Vladimir Sementsov-Ogievskiy wrote:
>> 23.08.2017 11:59, Vladimir Sementsov-Ogievskiy wrote:
>>> 22.08.2017 22:07, John Snow wrote:
>>>> Well, we knew we'd want this sooner or later. I've got some pings
>>>> downstream over whether or not we support persistent bitmaps for
>>>> non-qcow2 formats.
>>>>
>>>> Currently: no, we don't.
>>>>
>>>> We tried two different ideas for storage agnostic bitmap persistence:
>>>>
>>>>
>>>> (1) Using qcow2 as a storage container format (similar to VM save
>>>> states) for information not associated with that particular drive.
>>>>
>>>> This didn't go over so well; we decided it was too messy logistically to
>>>> manage data in any meaningful sense in a file that didn't share any
>>>> semantic relationship to the qcow2 in question.
>>>>
>>>> Well, "We" decided is more like "Kevin and Max" decided. They are right,
>>>> though.
>>>>
>>>>
>>>> (2) Using a new proto-wrapper format that Fam Zheng designed that serves
>>>> as a simple top-layer redirect for metadata that allows us to associate
>>>> raw bitmap data with an arbitrary backing image.
>>>>
>>>> This also didn't go over so well, the desire for a "new format" was
>>>> pretty thin, even if it was only a pseudo-format.
>>>>
>>>>
>>>> We'd still like to be able to use bitmaps with non-qcow2 formats
>>>> however, and people are going to keep asking. So here's a third thought:
>>>>
>>>>
>>>> (3) Add either a new flag that turns qcow2's backing file into a full
>>>> R/W backing file, or add a new extension to qcow2 entirely (bypassing
>>>> the traditional backing file mechanism to avoid confusion for older
>>>> tooling) that adds a new read-write backing file field.
>>>>
>>>> This RW backing file field will be used for all reads AND writes; the
>>>> qcow2 in question becomes a metadata container on top of the BDS chain.
>>>> We can re-use Vladimir's bitmap persistence extension to save bitmaps in
>>>> a qcow2 shell.
>>>>
>>>> The qcow2 becomes effectively a metadata cache for a new (essentially)
>>>> filter node that handles features such as bitmaps. This could also be
>>>> used to provide allocation map data for RAW files and other goodies down
>>>> the road.
>>>>
>>>> Hopefully this achieves our desire to not create new formats AND our
>>>> desire to concentrate features (and debugging, testing, etc) into qcow2,
>>>> while allowing users to "have bitmaps with raw files."
>>>>
>>>> Of course, in this scenario, users now have two files: a qcow2 wrapper
>>>> and the actual raw file in question; but regardless of how we were going
>>>> to solve this, a raw file necessitates an external file of some sort,
>>>> else we give up the idea that it was a raw file.
>>>>
>>>>
>>>> Sound good?
>>> Looks interesting. It is a clean reusing of qcow2-bitmaps without any
>>> modifications to them.
>>>
>>> Should there be some problems with internal snapshots and other things?
>>>
>>>
>>>
>> Hm. looks like that this backing file should not only receive all reads
>> and writes, but almost everything ->bdrv_ handlers, except bitmap
>> related of course. This doesn't seems simple to implement. Especially if
>> imaging some not-raw feature-full format under this thin qcow2 layer..
>> Or we can restrict this RW backing file to be raw-only?
>>
>>
> The idea would really be to support any arbitrary data store, so I
> wouldn't want to restrict it to just raw.
>
> You're right though, this might be a kind of messy approach.
>
> [From your other mail:]
>
>> So, anyway, I see only two differences (from the outside) between this approach and just a separate bitmap-only qcow2 without a data:
>>
> It's very delicately similar, yes.
>
>> 1. in RW-backing approach qcow2-bitmap file has a link to data file (as a backing). It looks good.
>>
> Right. The information necessary to establish a link between the bitmap
> data and the data being described is fully contained within a file fully
> specified by the QCOW2 spec.
>
>> 2. in RW-backing approach qcow2-bitmap file is a top of the virtual disk, in separate-file approach it is an option of the real data drive. In my opinion the second is more clean for users ("to add this feature you should use other file as your disk" vs "to add this feature you should add an option to your disk description")
> This puts us a little closer to the original idea that was rejected by
> Kevin at the time. To recap:
>
> "1": Use qcow2 as a container. This was rejected because we didn't want
> qcow2 containing data with no semantic relationship to the qcow2
> container or to each other. The way it sounds like you're proposing it,
> though, it would be one-qcow2-with-bitmaps-per-drive, so the data would
> at least stay strictly related, but it would be meaningless outside of
> QEMU itself. I think this is something that Kevin wanted to avoid, but I
> can't speak for him.
>
> It's certainly not beyond the realm of management software to remember
> to correlate a qcow2 metadata file alongside its actual data stores
> whenever it needed to do so, but it does mean the introduction of a
> feature that essentially requires the use of management software, which
> sees resistance in the community at times.
>
> In this model, you'd probably have the raw drive at the top, with the
> qcow2-with-bitmaps as a child node with some kind of new named child
> relationship. All IO stays at the root node, but the bitmap method
> handlers would know to look for this special bitmap-child. It shouldn't
> be too hard to implement.
>
>>
>> I think (may be I'm not right) loading bitmaps from additional qcow2-only-bitmaps file is simpler to implement (it will be specified in command line in drive options, like bitmaps_file=/path/to/it and then attached directly to BlockDriverState). The only drawback of simple qcow2-bitmap file  is that it has not a link inside it to the data file (like shabacking). We can ignore it, or we can implement this link as a separate extension to qcow2.
> Yes, definitely easier to implement as you say.
>
> The hard part is going to come in defining that semantic link. At this
> point, the only difference between the approaches is whether or not we
> allow the qcow2 to point to the implementation of the data;
>
> (1) The qcow2 is referenced by name from the CLI as an option to the
> other drive.
>
> (2) The qcow2 is referenced by name on the CLI, and its backing file
> field intuits the location of the implementation storage.
>
>
> In (1), we avoid saving or specifying the relationship between these two
> data stores in any way. This is certainly easy to do, and will save us
> some headache on the CLI. As a downside, we now have random orphaned
> files that aren't very interesting or useful on their own. The
> likelihood for desync between metadata and data increases. The use of
> management software is all but necessitated.
>
> In (2) We have to now specify, with a dizzying long list of
> possibilities, the location of the implementation data. qcow2 only has a
> filename for backing files presently, but this is likely inadequate.
> What if the data store isn't a locally kept file? What if it's a socket,
> or a stream, or literally anything else?
>
> We'd have to develop a new syntax for specifying these resources that
> can be stored in a qcow2 file, or otherwise co-opt an existing syntax
> in-use by QEMU. This syntax would likely be useful only to QEMU, which
> would steer the qcow2 format in a direction not too useful by other
> emulators, and qcow2 is an open format, so we may want to avoid this.
>
>
>
>
>
> I feel like what will make the difference between heading down either
> path would be helped along by answers to these questions:
>
> - What type of data stores do we wish to support with bitmaps? Simple
> file-based ones, or the full spectrum of all types? Only qcow2, and to
> hell with people who ask for otherwise?
>
> - How important is it that the qcow2 remains a fully independent file
> capable of describing its own relationship to the data?
>
> - Is it OK to allow robust, QEMU-specific data descriptors in a qcow2 file?
>
>
> Where I sit:
>
> - I'd like to support truly arbitrary protocols and formats. We can add
> transient / non-persistent bitmaps to any of these today, so I'd like to
> maintain that flexibility.
>
> - I think it will be difficult or impossible to agree upon a
> specification for the syntax to store the backing storage identifier in
> the qcow2 itself, because we'd like to let third party tools make use of
> qcow2 files.
>
> - I don't think Kevin will like the idea of us using qcow2 as a
> container that does not get treated as a first class citizen in the
> backing chain, but it's the easiest option to implement and avoids a lot
> of the syntax-of-backing-storage questions.
>
> - There's a definite usability factor to having the qcow2 just specify
> where the rest of the data sits like it does for normal backing files,
> but with what I am proposing, this complicates the open() call
> considerably, not to mention the syntactic issues addressed above.

Is it difficult to add an additional file link - like backing, and just 
use it in qcow2 code (not on generic level) instead of bs->file for all 
data (not metadata) io? So that, l2 table entries will point to clusters 
in this separate file.

>
>
>
> I guess I've talked myself out of wanting a "RW backing file" because it
> seems hard to implement, spec-wise ...
>
> --js


-- 
Best regards,
Vladimir

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 13195 bytes --]

On 2017-08-25 02:55, John Snow wrote:
> Sorry in advance for :words: ...
> 
> On 08/23/2017 02:04 PM, Vladimir Sementsov-Ogievskiy wrote:
>> 23.08.2017 11:59, Vladimir Sementsov-Ogievskiy wrote:
>>> 22.08.2017 22:07, John Snow wrote:
>>>> Well, we knew we'd want this sooner or later. I've got some pings
>>>> downstream over whether or not we support persistent bitmaps for
>>>> non-qcow2 formats.
>>>>
>>>> Currently: no, we don't.
>>>>
>>>> We tried two different ideas for storage agnostic bitmap persistence:
>>>>
>>>>
>>>> (1) Using qcow2 as a storage container format (similar to VM save
>>>> states) for information not associated with that particular drive.
>>>>
>>>> This didn't go over so well; we decided it was too messy logistically to
>>>> manage data in any meaningful sense in a file that didn't share any
>>>> semantic relationship to the qcow2 in question.
>>>>
>>>> Well, "We" decided is more like "Kevin and Max" decided. They are right,
>>>> though.
>>>>
>>>>
>>>> (2) Using a new proto-wrapper format that Fam Zheng designed that serves
>>>> as a simple top-layer redirect for metadata that allows us to associate
>>>> raw bitmap data with an arbitrary backing image.
>>>>
>>>> This also didn't go over so well, the desire for a "new format" was
>>>> pretty thin, even if it was only a pseudo-format.
>>>>
>>>>
>>>> We'd still like to be able to use bitmaps with non-qcow2 formats
>>>> however, and people are going to keep asking. So here's a third thought:
>>>>
>>>>
>>>> (3) Add either a new flag that turns qcow2's backing file into a full
>>>> R/W backing file, or add a new extension to qcow2 entirely (bypassing
>>>> the traditional backing file mechanism to avoid confusion for older
>>>> tooling) that adds a new read-write backing file field.
>>>>
>>>> This RW backing file field will be used for all reads AND writes; the
>>>> qcow2 in question becomes a metadata container on top of the BDS chain.
>>>> We can re-use Vladimir's bitmap persistence extension to save bitmaps in
>>>> a qcow2 shell.
>>>>
>>>> The qcow2 becomes effectively a metadata cache for a new (essentially)
>>>> filter node that handles features such as bitmaps. This could also be
>>>> used to provide allocation map data for RAW files and other goodies down
>>>> the road.
>>>>
>>>> Hopefully this achieves our desire to not create new formats AND our
>>>> desire to concentrate features (and debugging, testing, etc) into qcow2,
>>>> while allowing users to "have bitmaps with raw files."
>>>>
>>>> Of course, in this scenario, users now have two files: a qcow2 wrapper
>>>> and the actual raw file in question; but regardless of how we were going
>>>> to solve this, a raw file necessitates an external file of some sort,
>>>> else we give up the idea that it was a raw file.
>>>>
>>>>
>>>> Sound good?
>>>
>>> Looks interesting. It is a clean reusing of qcow2-bitmaps without any
>>> modifications to them.
>>>
>>> Should there be some problems with internal snapshots and other things?

I'd suspect you get exactly the same problems when using internal
snapshots together with backing files.  Imagine a newly created overlay
file and taking a snapshot.  This should give you exactly the same
issue, doesn't it?

>>>
>>>
>>>
>> Hm. looks like that this backing file should not only receive all reads
>> and writes, but almost everything ->bdrv_ handlers, except bitmap
>> related of course.

How so?  Shouldn't it just work like a backing file, except it also
receives writes instead of just reads?

>>                    This doesn't seems simple to implement. Especially if
>> imaging some not-raw feature-full format under this thin qcow2 layer..
>> Or we can restrict this RW backing file to be raw-only?
>>
>>
> 
> The idea would really be to support any arbitrary data store, so I
> wouldn't want to restrict it to just raw.
> 
> You're right though, this might be a kind of messy approach.
> 
> [From your other mail:]
> 
>>
>> So, anyway, I see only two differences (from the outside) between this approach and just a separate bitmap-only qcow2 without a data:
>>
> 
> It's very delicately similar, yes.
> 
>> 1. in RW-backing approach qcow2-bitmap file has a link to data file (as a backing). It looks good.

And this is rather important to me.

> Right. The information necessary to establish a link between the bitmap
> data and the data being described is fully contained within a file fully
> specified by the QCOW2 spec.
> 
>> 2. in RW-backing approach qcow2-bitmap file is a top of the virtual disk, in separate-file approach it is an option of the real data drive. In my opinion the second is more clean for users ("to add this feature you should use other file as your disk" vs "to add this feature you should add an option to your disk description")

I'd argue it's rather: "You cannot use this feature unless your format
supports it.  The only format supporting persistent bitmaps currently is
qcow2.  To use persistent bitmaps with other formats you can attach them
as R/W backing files to an empty qcow2 file, though."

So the difference is that you are saying it's a feature that is added to
a non-qcow2 image whereas I'm saying it's a feature that only a qcow2
image can provide (currently).

> This puts us a little closer to the original idea that was rejected by
> Kevin at the time. To recap:
> 
> "1": Use qcow2 as a container. This was rejected because we didn't want
> qcow2 containing data with no semantic relationship to the qcow2
> container or to each other. The way it sounds like you're proposing it,
> though, it would be one-qcow2-with-bitmaps-per-drive, so the data would
> at least stay strictly related, but it would be meaningless outside of
> QEMU itself. I think this is something that Kevin wanted to avoid, but I
> can't speak for him.
> 
> It's certainly not beyond the realm of management software to remember
> to correlate a qcow2 metadata file alongside its actual data stores
> whenever it needed to do so, but it does mean the introduction of a
> feature that essentially requires the use of management software, which
> sees resistance in the community at times.
> 
> In this model, you'd probably have the raw drive at the top, with the
> qcow2-with-bitmaps as a child node with some kind of new named child
> relationship. All IO stays at the root node, but the bitmap method
> handlers would know to look for this special bitmap-child. It shouldn't
> be too hard to implement.

I'd still like to throw in how much I dislike this approach, and I can't
really think of a way to make it palatable to me.  Not even "just write
the file name of the image the bitmaps cover into the qcow2 file" sounds
good to me, because then it still is basically unrelated data.

The only approach that I might see myself liking is to indeed add a flag
or whatever to say a qcow2 backing image is supposed to be R/W; and then
(after somehow verifying that the qcow2 image itself is empty) just make
qemu interpret this as "load the backing file as the real disk and
attach the qcow2 image as a 'metadata' child" or whatever.  But I fear
this gets uglier and uglier because how qemu loads the files will then
depend on whether the overlay is empty or not, and this may be very
confusing.

>> I think (may be I'm not right) loading bitmaps from additional qcow2-only-bitmaps file is simpler to implement (it will be specified in command line in drive options, like bitmaps_file=/path/to/it and then attached directly to BlockDriverState). The only drawback of simple qcow2-bitmap file  is that it has not a link inside it to the data file (like backing). We can ignore it, or we can implement this link as a separate extension to qcow2.
> 
> Yes, definitely easier to implement as you say.
> 
> The hard part is going to come in defining that semantic link. At this
> point, the only difference between the approaches is whether or not we
> allow the qcow2 to point to the implementation of the data;
> 
> (1) The qcow2 is referenced by name from the CLI as an option to the
> other drive.
> 
> (2) The qcow2 is referenced by name on the CLI, and its backing file
> field intuits the location of the implementation storage.
> 
> 
> In (1), we avoid saving or specifying the relationship between these two
> data stores in any way. This is certainly easy to do, and will save us
> some headache on the CLI. As a downside, we now have random orphaned
> files that aren't very interesting or useful on their own. The
> likelihood for desync between metadata and data increases. The use of
> management software is all but necessitated.
> 
> In (2) We have to now specify, with a dizzying long list of
> possibilities, the location of the implementation data. qcow2 only has a
> filename for backing files presently, but this is likely inadequate.
> What if the data store isn't a locally kept file? What if it's a socket,
> or a stream, or literally anything else?

I don't see the difference.  In (1), your data image gets a "bitmap" or
"metadata" child.  In (2), your qcow2 image gets the usual "backing"
child, or maybe call it "passthrough" or whatever, if you want to make
the difference more explicit than just passing an option to the qcow2
image to pass writes to its backing file.

> We'd have to develop a new syntax for specifying these resources that
> can be stored in a qcow2 file,

It's called the json-pseudo-protocol and was developed exactly for this.

>                                or otherwise co-opt an existing syntax
> in-use by QEMU. This syntax would likely be useful only to QEMU, which
> would steer the qcow2 format in a direction not too useful by other
> emulators, and qcow2 is an open format, so we may want to avoid this.

Storing a file name in the backing link field that cannot be interpreted
by other programs is in my opinion still very much better than not
storing any information whatsoever, because in the former case other
programs can at least say "sorry, I have no idea what this means" (or
maybe they can indeed interpret it, who knows), whereas in the latter
they may not even know that the qcow2 image is incomplete.

Also note that we are making an effort to be able to generate plain file
names (such as URLs which should be usable by other programs) whenever
possible.

> I feel like what will make the difference between heading down either
> path would be helped along by answers to these questions:
> 
> - What type of data stores do we wish to support with bitmaps? Simple
> file-based ones, or the full spectrum of all types? Only qcow2, and to
> hell with people who ask for otherwise?

I don't really know how this question relates to the issue other than
"If we only want to support qcow2, the whole discussion is moot; since
this discussion exists, we apparently do want to support something other
than qcow2 at some point."

Well, OK.  The main argument against supporting anything but qcow2 is
"if you want features, use qcow2; and we are working on making qcow2 as
fast as possible."  I think that's a very good argument still.  At some
point I (and probably others, too) had the idea of making qcow2 files in
raw layout: Have the data as a blob, just like a raw file, padded by
metadata around it.  An autoclear flag would specify that the qcow2 file
is in this format, and if so, you could simply access it like a raw file
and should have exactly the same speed as a raw file.  Maybe that would
solve this whole issue, too?

And as for non-file based backing files, see above.

> - How important is it that the qcow2 remains a fully independent file
> capable of describing its own relationship to the data?

Technically not important.  To me, very important.

> - Is it OK to allow robust, QEMU-specific data descriptors in a qcow2 file?

Depends.  At least Red Hat's QA does use json file names, so there's
already that.

I think QEMU-specific is OK as long as it still makes sense, as long as
there is no other way, and as long as there is some kind of
documentation still.

Technically, you can always put QEMU-specific stuff into the qcow2
specification and thus make it generic.

> Where I sit:

[...]

> - I don't think Kevin will like the idea of us using qcow2 as a
> container that does not get treated as a first class citizen in the
> backing chain, but it's the easiest option to implement and avoids a lot
> of the syntax-of-backing-storage questions.

I'd like to throw in that I was not at all opposed to Fam's approach of
having an independent format (tar, wasn't it?) just for storing bitmaps.
 I know Kevin was, but I don't quite remember why.  Probably because it
was a real format, but a very strange one, and then implementing it
through the block layer was weird...?

OTOH, there is one issue I have with the R/W backing approach: Every
request to the raw file has to go through the qcow2 layer.  And since
you probably want to use raw because of the speed, this is not so nice.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Fam Zheng]

On Fri, 08/25 15:44, Max Reitz wrote:
> Well, OK.  The main argument against supporting anything but qcow2 is
> "if you want features, use qcow2; and we are working on making qcow2 as
> fast as possible."  I think that's a very good argument still.  At some
> point I (and probably others, too) had the idea of making qcow2 files in
> raw layout: 


Yes! I think this idea makes a whole lot of sense, too. Metadata tables can be
generated so old implementation can still use it.

Fam

> Have the data as a blob, just like a raw file, padded by
> metadata around it.  An autoclear flag would specify that the qcow2 file
> is in this format, and if so, you could simply access it like a raw file
> and should have exactly the same speed as a raw file.  Maybe that would
> solve this whole issue, too?

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[John Snow]

On 08/27/2017 10:57 PM, Fam Zheng wrote:
> On Fri, 08/25 15:44, Max Reitz wrote:
>> Well, OK.  The main argument against supporting anything but qcow2 is
>> "if you want features, use qcow2; and we are working on making qcow2 as
>> fast as possible."  I think that's a very good argument still.  At some
>> point I (and probably others, too) had the idea of making qcow2 files in
>> raw layout: 
> 
> 
> Yes! I think this idea makes a whole lot of sense, too. Metadata tables can be
> generated so old implementation can still use it.
> 
> Fam
> 
>> Have the data as a blob, just like a raw file, padded by
>> metadata around it.  An autoclear flag would specify that the qcow2 file
>> is in this format, and if so, you could simply access it like a raw file
>> and should have exactly the same speed as a raw file.  Maybe that would
>> solve this whole issue, too?

I wonder if this would be sufficient to alleviate the desire to use raw
files...

(Eh, well, realistically, someone's still always going to ask if they
can use various features with non-qcow2 files...)

Nir, Yaniv; any input?

(Context: We're debating how to add persistent bitmaps to raw files as I
was informed that RHV was 'asking about it.' Max is reminding me there
is a proposal for a style of QCOW2 that uses a raw layout for data,
mitigating or eliminating any performance hits related to the L2 cache.
What I am not aware of is why RHV would use raw files for any purpose.
Is it performance? Simplicity? Could RHV use a raw-layout qcow2?)

--js

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[John Snow]

On 08/25/2017 09:44 AM, Max Reitz wrote:
> On 2017-08-25 02:55, John Snow wrote:
>> Sorry in advance for :words: ...
>>
>> On 08/23/2017 02:04 PM, Vladimir Sementsov-Ogievskiy wrote:
>>> 23.08.2017 11:59, Vladimir Sementsov-Ogievskiy wrote:
>>>> 22.08.2017 22:07, John Snow wrote:

[snip]

>>>>
>>>> Should there be some problems with internal snapshots and other things?
> 
> I'd suspect you get exactly the same problems when using internal
> snapshots together with backing files.  Imagine a newly created overlay
> file and taking a snapshot.  This should give you exactly the same
> issue, doesn't it?
> 
>>>>
>>>>
>>>>
>>> Hm. looks like that this backing file should not only receive all reads
>>> and writes, but almost everything ->bdrv_ handlers, except bitmap
>>> related of course.
> 
> How so?  Shouldn't it just work like a backing file, except it also
> receives writes instead of just reads?
> 
>>>                    This doesn't seems simple to implement. Especially if
>>> imaging some not-raw feature-full format under this thin qcow2 layer..
>>> Or we can restrict this RW backing file to be raw-only?
>>>
>>>
>>
>> The idea would really be to support any arbitrary data store, so I
>> wouldn't want to restrict it to just raw.
>>
>> You're right though, this might be a kind of messy approach.
>>
>> [From your other mail:]
>>
>>>
>>> So, anyway, I see only two differences (from the outside) between this approach and just a separate bitmap-only qcow2 without a data:
>>>
>>
>> It's very delicately similar, yes.
>>
>>> 1. in RW-backing approach qcow2-bitmap file has a link to data file (as a backing). It looks good.
> 
> And this is rather important to me.
> 

Good to know. Some good solid opinions to work around. ;)

>> Right. The information necessary to establish a link between the bitmap
>> data and the data being described is fully contained within a file fully
>> specified by the QCOW2 spec.
>>
>>> 2. in RW-backing approach qcow2-bitmap file is a top of the virtual disk, in separate-file approach it is an option of the real data drive. In my opinion the second is more clean for users ("to add this feature you should use other file as your disk" vs "to add this feature you should add an option to your disk description")
> 
> I'd argue it's rather: "You cannot use this feature unless your format
> supports it.  The only format supporting persistent bitmaps currently is
> qcow2.  To use persistent bitmaps with other formats you can attach them
> as R/W backing files to an empty qcow2 file, though."
> 
> So the difference is that you are saying it's a feature that is added to
> a non-qcow2 image whereas I'm saying it's a feature that only a qcow2
> image can provide (currently).
> 
>> This puts us a little closer to the original idea that was rejected by
>> Kevin at the time. To recap:
>>
>> "1": Use qcow2 as a container. This was rejected because we didn't want
>> qcow2 containing data with no semantic relationship to the qcow2
>> container or to each other. The way it sounds like you're proposing it,
>> though, it would be one-qcow2-with-bitmaps-per-drive, so the data would
>> at least stay strictly related, but it would be meaningless outside of
>> QEMU itself. I think this is something that Kevin wanted to avoid, but I
>> can't speak for him.
>>
>> It's certainly not beyond the realm of management software to remember
>> to correlate a qcow2 metadata file alongside its actual data stores
>> whenever it needed to do so, but it does mean the introduction of a
>> feature that essentially requires the use of management software, which
>> sees resistance in the community at times.
>>
>> In this model, you'd probably have the raw drive at the top, with the
>> qcow2-with-bitmaps as a child node with some kind of new named child
>> relationship. All IO stays at the root node, but the bitmap method
>> handlers would know to look for this special bitmap-child. It shouldn't
>> be too hard to implement.
> 
> I'd still like to throw in how much I dislike this approach, and I can't
> really think of a way to make it palatable to me.  Not even "just write
> the file name of the image the bitmaps cover into the qcow2 file" sounds
> good to me, because then it still is basically unrelated data.
> 

Understood. It's something I'd like to avoid too, but I have some real
concerns about implementation of that semantic link.

> The only approach that I might see myself liking is to indeed add a flag
> or whatever to say a qcow2 backing image is supposed to be R/W; and then
> (after somehow verifying that the qcow2 image itself is empty) just make
> qemu interpret this as "load the backing file as the real disk and
> attach the qcow2 image as a 'metadata' child" or whatever.  But I fear
> this gets uglier and uglier because how qemu loads the files will then
> depend on whether the overlay is empty or not, and this may be very
> confusing.
> 

Right, it makes opening a little more convoluted than it normally is.

"Oh look, it's empty and it has a RW backing file. Please stand by as we
open the "RW backing file" and install it as our parent!"

It may actually not be so bad, but it does add a complexity...

>>> I think (may be I'm not right) loading bitmaps from additional qcow2-only-bitmaps file is simpler to implement (it will be specified in command line in drive options, like bitmaps_file=/path/to/it and then attached directly to BlockDriverState). The only drawback of simple qcow2-bitmap file  is that it has not a link inside it to the data file (like backing). We can ignore it, or we can implement this link as a separate extension to qcow2.
>>
>> Yes, definitely easier to implement as you say.
>>
>> The hard part is going to come in defining that semantic link. At this
>> point, the only difference between the approaches is whether or not we
>> allow the qcow2 to point to the implementation of the data;
>>
>> (1) The qcow2 is referenced by name from the CLI as an option to the
>> other drive.
>>
>> (2) The qcow2 is referenced by name on the CLI, and its backing file
>> field intuits the location of the implementation storage.
>>
>>
>> In (1), we avoid saving or specifying the relationship between these two
>> data stores in any way. This is certainly easy to do, and will save us
>> some headache on the CLI. As a downside, we now have random orphaned
>> files that aren't very interesting or useful on their own. The
>> likelihood for desync between metadata and data increases. The use of
>> management software is all but necessitated.
>>
>> In (2) We have to now specify, with a dizzying long list of
>> possibilities, the location of the implementation data. qcow2 only has a
>> filename for backing files presently, but this is likely inadequate.
>> What if the data store isn't a locally kept file? What if it's a socket,
>> or a stream, or literally anything else?
> 
> I don't see the difference.  In (1), your data image gets a "bitmap" or
> "metadata" child.  In (2), your qcow2 image gets the usual "backing"
> child, or maybe call it "passthrough" or whatever, if you want to make
> the difference more explicit than just passing an option to the qcow2
> image to pass writes to its backing file.
> 

The difference in the relationship in-memory is actually kind of
uninteresting.

The difference as I see it is primarily how we specify the relationship
between the qcow2 and the implementation storage; in (1) It's defined
on-disk, and in (2) It's defined via CLI only, so

(1) Incurs a cost of having to define the link syntax (possibly causing
a rather qemu-specific syntax), and
(2) Avoids that cost, but leaves the data on-disk unrelated, which you hate.

>> We'd have to develop a new syntax for specifying these resources that
>> can be stored in a qcow2 file,
> 
> It's called the json-pseudo-protocol and was developed exactly for this.
> 

That's what I was hinting at for "or otherwise co-opt an existing
syntax" but I was unaware that it was intended for "exactly" this.

Do we actually use it in any on-disk format, currently? qcow2 only lets
you specify simple filenames in the qcow2 metadata, right?

>>                                or otherwise co-opt an existing syntax
>> in-use by QEMU. This syntax would likely be useful only to QEMU, which
>> would steer the qcow2 format in a direction not too useful by other
>> emulators, and qcow2 is an open format, so we may want to avoid this.
> 
> Storing a file name in the backing link field that cannot be interpreted
> by other programs is in my opinion still very much better than not
> storing any information whatsoever, because in the former case other
> programs can at least say "sorry, I have no idea what this means" (or
> maybe they can indeed interpret it, who knows), whereas in the latter
> they may not even know that the qcow2 image is incomplete.
> 

I don't disagree personally, but I seem to recall that Kevin was adamant
that the qcow2 bitmap extension should remain useful and semantically
meaningful to third parties, so I try to keep that in mind. Maybe I
should let him chime in instead of try to "concern troll" my own
suggestions into the ground.

> Also note that we are making an effort to be able to generate plain file
> names (such as URLs which should be usable by other programs) whenever
> possible.
> 

Noted. Do we have a useful discriminator anywhere that allows us to
easily check if a filename/locator/URI/whatever is in an accepted
format, or if we still have QEMU-specific garbage?

We could always just disallow QEMU-specific protocol-talk from getting
written and allow this only for configurations that QEMU understands to
be universal...

>> I feel like what will make the difference between heading down either
>> path would be helped along by answers to these questions:
>>
>> - What type of data stores do we wish to support with bitmaps? Simple
>> file-based ones, or the full spectrum of all types? Only qcow2, and to
>> hell with people who ask for otherwise?
> 
> I don't really know how this question relates to the issue other than
> "If we only want to support qcow2, the whole discussion is moot; since
> this discussion exists, we apparently do want to support something other
> than qcow2 at some point."
> 

Yeah, I mean, if the answer is a strong no here we can avoid the rest of
the discussion. Worthwhile to know, right?

> Well, OK.  The main argument against supporting anything but qcow2 is
> "if you want features, use qcow2; and we are working on making qcow2 as
> fast as possible."  I think that's a very good argument still.  At some
> point I (and probably others, too) had the idea of making qcow2 files in
> raw layout: Have the data as a blob, just like a raw file, padded by
> metadata around it.  An autoclear flag would specify that the qcow2 file
> is in this format, and if so, you could simply access it like a raw file
> and should have exactly the same speed as a raw file.  Maybe that would
> solve this whole issue, too?
> 
> And as for non-file based backing files, see above.
> 
>> - How important is it that the qcow2 remains a fully independent file
>> capable of describing its own relationship to the data?
> 
> Technically not important.  To me, very important.
> 
>> - Is it OK to allow robust, QEMU-specific data descriptors in a qcow2 file?
> 
> Depends.  At least Red Hat's QA does use json file names, so there's
> already that.
> 
> I think QEMU-specific is OK as long as it still makes sense, as long as
> there is no other way, and as long as there is some kind of
> documentation still.
> 
> Technically, you can always put QEMU-specific stuff into the qcow2
> specification and thus make it generic.
> 
>> Where I sit:
> 
> [...]
> 
>> - I don't think Kevin will like the idea of us using qcow2 as a
>> container that does not get treated as a first class citizen in the
>> backing chain, but it's the easiest option to implement and avoids a lot
>> of the syntax-of-backing-storage questions.
> 
> I'd like to throw in that I was not at all opposed to Fam's approach of
> having an independent format (tar, wasn't it?) just for storing bitmaps.
>  I know Kevin was, but I don't quite remember why.  Probably because it
> was a real format, but a very strange one, and then implementing it
> through the block layer was weird...?
> 
> OTOH, there is one issue I have with the R/W backing approach: Every
> request to the raw file has to go through the qcow2 layer.  And since
> you probably want to use raw because of the speed, this is not so nice.
> 

We probably could fix that by changing the relationship so that it isn't
*really* a backing file, but that maybe creates other problems.

> Max
> 

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Yaniv Lavi (Dary)]

YANIV LAVI (YANIV DARY)

SENIOR TECHNICAL PRODUCT MANAGER

Red Hat Israel Ltd. <https://www.redhat.com/>

34 Jerusalem Road, Building A, 1st floor

Ra'anana, Israel 4350109

ylavi@redhat.com    T: +972-9-7692306/8272306     F: +972-9-7692223    IM: ylavi
 <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
@redhatnews <https://twitter.com/redhatnews>   Red Hat
<https://www.linkedin.com/company/red-hat>   Red Hat
<https://www.facebook.com/RedHatInc>


On Mon, Aug 28, 2017 at 9:11 PM, John Snow <jsnow@redhat.com> wrote:

>
>
> On 08/27/2017 10:57 PM, Fam Zheng wrote:
> > On Fri, 08/25 15:44, Max Reitz wrote:
> >> Well, OK.  The main argument against supporting anything but qcow2 is
> >> "if you want features, use qcow2; and we are working on making qcow2 as
> >> fast as possible."  I think that's a very good argument still.  At some
> >> point I (and probably others, too) had the idea of making qcow2 files in
> >> raw layout:
> >
> >
> > Yes! I think this idea makes a whole lot of sense, too. Metadata tables
> can be
> > generated so old implementation can still use it.
> >
> > Fam
> >
> >> Have the data as a blob, just like a raw file, padded by
> >> metadata around it.  An autoclear flag would specify that the qcow2 file
> >> is in this format, and if so, you could simply access it like a raw file
> >> and should have exactly the same speed as a raw file.  Maybe that would
> >> solve this whole issue, too?
>
> I wonder if this would be sufficient to alleviate the desire to use raw
> files...
>
> (Eh, well, realistically, someone's still always going to ask if they
> can use various features with non-qcow2 files...)
>
> Nir, Yaniv; any input?
>

We are using raw format for performance reasons.
As we have many customers that currently use this format, not support it
would be a blocker the use of the feature.
At a minimum we would require ability to convert raw to qcow2 raw-layout.

Please also consider that we are planning to go on the OSP route of LUN per
disk and would still want the tracking to work.
I makes sense that for that and raw format you will be able to save the
mapping to another file other than a qcow.


>
> (Context: We're debating how to add persistent bitmaps to raw files as I
> was informed that RHV was 'asking about it.' Max is reminding me there
> is a proposal for a style of QCOW2 that uses a raw layout for data,
> mitigating or eliminating any performance hits related to the L2 cache.
> What I am not aware of is why RHV would use raw files for any purpose.
> Is it performance? Simplicity? Could RHV use a raw-layout qcow2?)
>
> --js
>

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2226 bytes --]

On 08/28/2017 08:18 PM, John Snow wrote:
>>> We'd have to develop a new syntax for specifying these resources that
>>> can be stored in a qcow2 file,
>>
>> It's called the json-pseudo-protocol and was developed exactly for this.
>>
> 
> That's what I was hinting at for "or otherwise co-opt an existing
> syntax" but I was unaware that it was intended for "exactly" this.
> 
> Do we actually use it in any on-disk format, currently? qcow2 only lets
> you specify simple filenames in the qcow2 metadata, right?

You can specify json-pseudo backing names both on the command line AND
embedded in the qcow2 file itself (within the length limits imposed by
the qcow2 header size).  Yes, this means it is already possible to
create qcow2 files that can only be opened by qemu (or else teaching
your alternative program how to parse qemu's json-pseudo-protocol).

> 
>>>                                or otherwise co-opt an existing syntax
>>> in-use by QEMU. This syntax would likely be useful only to QEMU, which
>>> would steer the qcow2 format in a direction not too useful by other
>>> emulators, and qcow2 is an open format, so we may want to avoid this.
>>
>> Storing a file name in the backing link field that cannot be interpreted
>> by other programs is in my opinion still very much better than not
>> storing any information whatsoever, because in the former case other
>> programs can at least say "sorry, I have no idea what this means" (or
>> maybe they can indeed interpret it, who knows), whereas in the latter
>> they may not even know that the qcow2 image is incomplete.
>>
> 
> I don't disagree personally, but I seem to recall that Kevin was adamant
> that the qcow2 bitmap extension should remain useful and semantically
> meaningful to third parties, so I try to keep that in mind. Maybe I
> should let him chime in instead of try to "concern troll" my own
> suggestions into the ground.

We already have that situation today, but you are right to worry about
whether making it even more prevalent is something we can try to minimize.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[John Snow]

On 08/29/2017 10:30 AM, Eric Blake wrote:
> On 08/28/2017 08:18 PM, John Snow wrote:
>>>> We'd have to develop a new syntax for specifying these resources that
>>>> can be stored in a qcow2 file,
>>>
>>> It's called the json-pseudo-protocol and was developed exactly for this.
>>>
>>
>> That's what I was hinting at for "or otherwise co-opt an existing
>> syntax" but I was unaware that it was intended for "exactly" this.
>>
>> Do we actually use it in any on-disk format, currently? qcow2 only lets
>> you specify simple filenames in the qcow2 metadata, right?
> 
> You can specify json-pseudo backing names both on the command line AND
> embedded in the qcow2 file itself (within the length limits imposed by
> the qcow2 header size).  Yes, this means it is already possible to
> create qcow2 files that can only be opened by qemu (or else teaching
> your alternative program how to parse qemu's json-pseudo-protocol).
> 
>>
>>>>                                or otherwise co-opt an existing syntax
>>>> in-use by QEMU. This syntax would likely be useful only to QEMU, which
>>>> would steer the qcow2 format in a direction not too useful by other
>>>> emulators, and qcow2 is an open format, so we may want to avoid this.
>>>
>>> Storing a file name in the backing link field that cannot be interpreted
>>> by other programs is in my opinion still very much better than not
>>> storing any information whatsoever, because in the former case other
>>> programs can at least say "sorry, I have no idea what this means" (or
>>> maybe they can indeed interpret it, who knows), whereas in the latter
>>> they may not even know that the qcow2 image is incomplete.
>>>
>>
>> I don't disagree personally, but I seem to recall that Kevin was adamant
>> that the qcow2 bitmap extension should remain useful and semantically
>> meaningful to third parties, so I try to keep that in mind. Maybe I
>> should let him chime in instead of try to "concern troll" my own
>> suggestions into the ground.
> 
> We already have that situation today, but you are right to worry about
> whether making it even more prevalent is something we can try to minimize.
> 

Proposal distillate:

(1) Specify relationship on CLI, QCOW becomes a bitmap-child of any
arbitrary node.

Pros: Easy to implement
      Adds bitmap support to literally everything
Cons: Bitmap has no semantic link to data it describes
      Relationship must be re-specified every launch
      Max and Kevin are firmly NACK

(2) Raw file becomes a R/W backing file of the QCOW2, implemented as
either a bitmap-child or a more traditional backing file that can
additionally service writes

Pros: Easy to understand
      relationship between files exists outside of the QEMU process
      Adds bitmap support to just about everything that can be expressed
via JSON
Cons: All but necessitates QEMU-specific syntax in a qcow2 file
      Depending on implementation, possibly messy in bdrv_open
      Adding bitmaps to non-qcow2 files after open makes the launch CLI
invalid for future launches (Not any different to snapshots.)

(3) Add a raw-like mapping mode to QCOW2 instead, skipping the whole affair

Pros: Adds a nice, performant hybrid mode to qcow2
      Solves the problem of "bitmaps for raw" more or less
      Avoids bdrv_open() complications
      Avoids writing qemu-specific jargon in qcow2 files
Cons: Doesn't actually add arbitrary bitmaps to any file format
      Users are still gonna ask for bitmaps for other formats anyway



I think I like 2 or 3 -- or perhaps indeed two AND three. The qcow2-raw
mode sounds like something we ought to have anyway. I'll try to start an
RFC.

--js

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 3335 bytes --]

On 2017-08-29 11:26, Yaniv Lavi (Dary) wrote:
> 
> 
> YANIV LAVI (YANIV DARY)
> 
> SENIOR TECHNICAL PRODUCT MANAGER
> 
> Red Hat Israel Ltd. <https://www.redhat.com/>
> 
> 34 Jerusalem Road, Building A, 1st floor
> 
> Ra'anana, Israel 4350109
> 
> ylavi@redhat.com <mailto:ylavi@redhat.com>    T: +972-9-7692306
> <tel:+972-9-7692306>/8272306 <tel:8272306>     F: +972-9-7692223
> <tel:+972-9-7692223>    IM: ylavi
> 
> <https://red.ht/sig>	TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
> 
> @redhatnews <https://twitter.com/redhatnews>   Red Hat
> <https://www.linkedin.com/company/red-hat>   Red Hat
> <https://www.facebook.com/RedHatInc>
> 
> 
> On Mon, Aug 28, 2017 at 9:11 PM, John Snow <jsnow@redhat.com
> <mailto:jsnow@redhat.com>> wrote:
> 
> 
> 
>     On 08/27/2017 10:57 PM, Fam Zheng wrote:
>     > On Fri, 08/25 15:44, Max Reitz wrote:
>     >> Well, OK.  The main argument against supporting anything but qcow2 is
>     >> "if you want features, use qcow2; and we are working on making
>     qcow2 as
>     >> fast as possible."  I think that's a very good argument still. 
>     At some
>     >> point I (and probably others, too) had the idea of making qcow2
>     files in
>     >> raw layout:
>     >
>     >
>     > Yes! I think this idea makes a whole lot of sense, too. Metadata
>     tables can be
>     > generated so old implementation can still use it.
>     >
>     > Fam
>     >
>     >> Have the data as a blob, just like a raw file, padded by
>     >> metadata around it.  An autoclear flag would specify that the
>     qcow2 file
>     >> is in this format, and if so, you could simply access it like a
>     raw file
>     >> and should have exactly the same speed as a raw file.  Maybe that
>     would
>     >> solve this whole issue, too?
> 
>     I wonder if this would be sufficient to alleviate the desire to use raw
>     files...
> 
>     (Eh, well, realistically, someone's still always going to ask if they
>     can use various features with non-qcow2 files...)
> 
>     Nir, Yaniv; any input?
> 
> 
> We are using raw format for performance reasons.

Using raw layout for the data in a qcow2 file would give you exactly the
same performance as raw.

(Or better "should", but I can't think of a reason why it would not.)

Max

> As we have many customers that currently use this format, not support it
> would be a blocker the use of the feature.
> At a minimum we would require ability to convert raw to qcow2 raw-layout.
> 
> Please also consider that we are planning to go on the OSP route of LUN
> per disk and would still want the tracking to work.
> I makes sense that for that and raw format you will be able to save the
> mapping to another file other than a qcow.
>  
> 
> 
>     (Context: We're debating how to add persistent bitmaps to raw files as I
>     was informed that RHV was 'asking about it.' Max is reminding me there
>     is a proposal for a style of QCOW2 that uses a raw layout for data,
>     mitigating or eliminating any performance hits related to the L2 cache.
>     What I am not aware of is why RHV would use raw files for any purpose.
>     Is it performance? Simplicity? Could RHV use a raw-layout qcow2?)
> 
>     --js
> 
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 5664 bytes --]

On 2017-08-29 03:18, John Snow wrote:
> 
> 
> On 08/25/2017 09:44 AM, Max Reitz wrote:
>> On 2017-08-25 02:55, John Snow wrote:

[...]

>>> In (1), we avoid saving or specifying the relationship between these two
>>> data stores in any way. This is certainly easy to do, and will save us
>>> some headache on the CLI. As a downside, we now have random orphaned
>>> files that aren't very interesting or useful on their own. The
>>> likelihood for desync between metadata and data increases. The use of
>>> management software is all but necessitated.
>>>
>>> In (2) We have to now specify, with a dizzying long list of
>>> possibilities, the location of the implementation data. qcow2 only has a
>>> filename for backing files presently, but this is likely inadequate.
>>> What if the data store isn't a locally kept file? What if it's a socket,
>>> or a stream, or literally anything else?
>>
>> I don't see the difference.  In (1), your data image gets a "bitmap" or
>> "metadata" child.  In (2), your qcow2 image gets the usual "backing"
>> child, or maybe call it "passthrough" or whatever, if you want to make
>> the difference more explicit than just passing an option to the qcow2
>> image to pass writes to its backing file.
>>
> 
> The difference in the relationship in-memory is actually kind of
> uninteresting.
> 
> The difference as I see it is primarily how we specify the relationship
> between the qcow2 and the implementation storage; in (1) It's defined
> on-disk, and in (2) It's defined via CLI only, so
> 
> (1) Incurs a cost of having to define the link syntax (possibly causing
> a rather qemu-specific syntax), and
> (2) Avoids that cost, but leaves the data on-disk unrelated, which you hate.

What I meant was that in the end both have some parent-child
relationship in the BDS graph, so you can freely define both on the CLI.
So for (1), it usually is defined on-disk, yes, but you can also define
it on the command line -- just like it works for backing files.

>>> We'd have to develop a new syntax for specifying these resources that
>>> can be stored in a qcow2 file,
>>
>> It's called the json-pseudo-protocol and was developed exactly for this.
>>
> 
> That's what I was hinting at for "or otherwise co-opt an existing
> syntax" but I was unaware that it was intended for "exactly" this.
> 
> Do we actually use it in any on-disk format, currently? qcow2 only lets
> you specify simple filenames in the qcow2 metadata, right?

I don't think it's specified.  And I'm pretty sure that people would
agree that e.g. HTTP URLs are fine to specify in the backing link field.
 So I think that non-simple filenames are fine, too, but the question of
course is where to draw the line.

I'd argue that every time your start to allow filenames to have
protocols, you end up with some that not all programs can interpret;
like the json: protocol.

>>>                                or otherwise co-opt an existing syntax
>>> in-use by QEMU. This syntax would likely be useful only to QEMU, which
>>> would steer the qcow2 format in a direction not too useful by other
>>> emulators, and qcow2 is an open format, so we may want to avoid this.
>>
>> Storing a file name in the backing link field that cannot be interpreted
>> by other programs is in my opinion still very much better than not
>> storing any information whatsoever, because in the former case other
>> programs can at least say "sorry, I have no idea what this means" (or
>> maybe they can indeed interpret it, who knows), whereas in the latter
>> they may not even know that the qcow2 image is incomplete.
>>
> 
> I don't disagree personally, but I seem to recall that Kevin was adamant
> that the qcow2 bitmap extension should remain useful and semantically
> meaningful to third parties, so I try to keep that in mind. Maybe I
> should let him chime in instead of try to "concern troll" my own
> suggestions into the ground.

I seem to recall that I was adamant, too. :-)

All I'm saying is if you allow qcow2 to have writable backing files, it
doesn't make it any less meaningful to 3rd party programs than the
backing link field currently is.

>> Also note that we are making an effort to be able to generate plain file
>> names (such as URLs which should be usable by other programs) whenever
>> possible.
>>
> 
> Noted. Do we have a useful discriminator anywhere that allows us to
> easily check if a filename/locator/URI/whatever is in an accepted
> format, or if we still have QEMU-specific garbage?

Are NBD URLs qemu-specific? :-)

I think protocol URLs are just always icky.  If it's json: (when
bs->exact_filename is empty), you can be sure it's qemu-specific --
otherwise, it may be.

> We could always just disallow QEMU-specific protocol-talk from getting
> written and allow this only for configurations that QEMU understands to
> be universal...

I'd say it's too late for that, and also that json: was introduced
precisely to be able to get full BDS graph configuration into a backing
filename field.

[...]

>> OTOH, there is one issue I have with the R/W backing approach: Every
>> request to the raw file has to go through the qcow2 layer.  And since
>> you probably want to use raw because of the speed, this is not so nice.
>>
> 
> We probably could fix that by changing the relationship so that it isn't
> *really* a backing file, but that maybe creates other problems.

Yes.  This will be reflected in the BDS graph relationship, and it's
just weird that sometimes a qcow2 backing file would be loaded as its
parent.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 4039 bytes --]

On 2017-08-29 23:02, John Snow wrote:
> 
> 
> On 08/29/2017 10:30 AM, Eric Blake wrote:
>> On 08/28/2017 08:18 PM, John Snow wrote:
>>>>> We'd have to develop a new syntax for specifying these resources that
>>>>> can be stored in a qcow2 file,
>>>>
>>>> It's called the json-pseudo-protocol and was developed exactly for this.
>>>>
>>>
>>> That's what I was hinting at for "or otherwise co-opt an existing
>>> syntax" but I was unaware that it was intended for "exactly" this.
>>>
>>> Do we actually use it in any on-disk format, currently? qcow2 only lets
>>> you specify simple filenames in the qcow2 metadata, right?
>>
>> You can specify json-pseudo backing names both on the command line AND
>> embedded in the qcow2 file itself (within the length limits imposed by
>> the qcow2 header size).  Yes, this means it is already possible to
>> create qcow2 files that can only be opened by qemu (or else teaching
>> your alternative program how to parse qemu's json-pseudo-protocol).
>>
>>>
>>>>>                                or otherwise co-opt an existing syntax
>>>>> in-use by QEMU. This syntax would likely be useful only to QEMU, which
>>>>> would steer the qcow2 format in a direction not too useful by other
>>>>> emulators, and qcow2 is an open format, so we may want to avoid this.
>>>>
>>>> Storing a file name in the backing link field that cannot be interpreted
>>>> by other programs is in my opinion still very much better than not
>>>> storing any information whatsoever, because in the former case other
>>>> programs can at least say "sorry, I have no idea what this means" (or
>>>> maybe they can indeed interpret it, who knows), whereas in the latter
>>>> they may not even know that the qcow2 image is incomplete.
>>>>
>>>
>>> I don't disagree personally, but I seem to recall that Kevin was adamant
>>> that the qcow2 bitmap extension should remain useful and semantically
>>> meaningful to third parties, so I try to keep that in mind. Maybe I
>>> should let him chime in instead of try to "concern troll" my own
>>> suggestions into the ground.
>>
>> We already have that situation today, but you are right to worry about
>> whether making it even more prevalent is something we can try to minimize.
>>
> 
> Proposal distillate:
> 
> (1) Specify relationship on CLI, QCOW becomes a bitmap-child of any
> arbitrary node.
> 
> Pros: Easy to implement
>       Adds bitmap support to literally everything
> Cons: Bitmap has no semantic link to data it describes
>       Relationship must be re-specified every launch
>       Max and Kevin are firmly NACK
> 
> (2) Raw file becomes a R/W backing file of the QCOW2, implemented as
> either a bitmap-child or a more traditional backing file that can
> additionally service writes
> 
> Pros: Easy to understand
>       relationship between files exists outside of the QEMU process
>       Adds bitmap support to just about everything that can be expressed
> via JSON
> Cons: All but necessitates QEMU-specific syntax in a qcow2 file
>       Depending on implementation, possibly messy in bdrv_open
>       Adding bitmaps to non-qcow2 files after open makes the launch CLI
> invalid for future launches (Not any different to snapshots.)
> 
> (3) Add a raw-like mapping mode to QCOW2 instead, skipping the whole affair
> 
> Pros: Adds a nice, performant hybrid mode to qcow2
>       Solves the problem of "bitmaps for raw" more or less
>       Avoids bdrv_open() complications
>       Avoids writing qemu-specific jargon in qcow2 files
> Cons: Doesn't actually add arbitrary bitmaps to any file format
>       Users are still gonna ask for bitmaps for other formats anyway
> 
> 
> 
> I think I like 2 or 3 -- or perhaps indeed two AND three. The qcow2-raw
> mode sounds like something we ought to have anyway. I'll try to start an
> RFC.

I think the only main thing that's missing for implementing (3) is a way
to mark allocated clusters to be fall-through-to-backing-file.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Yaniv Lavi (Dary)]

YANIV LAVI (YANIV DARY)

SENIOR TECHNICAL PRODUCT MANAGER

Red Hat Israel Ltd. <https://www.redhat.com/>

34 Jerusalem Road, Building A, 1st floor

Ra'anana, Israel 4350109

ylavi@redhat.com    T: +972-9-7692306/8272306     F: +972-9-7692223    IM: ylavi
 <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
@redhatnews <https://twitter.com/redhatnews>   Red Hat
<https://www.linkedin.com/company/red-hat>   Red Hat
<https://www.facebook.com/RedHatInc>


On Wed, Aug 30, 2017 at 1:35 PM, Max Reitz <mreitz@redhat.com> wrote:

> On 2017-08-29 11:26, Yaniv Lavi (Dary) wrote:
> >
> >
> > YANIV LAVI (YANIV DARY)
> >
> > SENIOR TECHNICAL PRODUCT MANAGER
> >
> > Red Hat Israel Ltd. <https://www.redhat.com/>
> >
> > 34 Jerusalem Road, Building A, 1st floor
> >
> > Ra'anana, Israel 4350109
> >
> > ylavi@redhat.com <mailto:ylavi@redhat.com>    T: +972-9-7692306
> > <tel:+972-9-7692306>/8272306 <tel:8272306>     F: +972-9-7692223
> > <tel:+972-9-7692223>    IM: ylavi
> >
> > <https://red.ht/sig>  TRIED. TESTED. TRUSTED. <
> https://redhat.com/trusted>
> >
> > @redhatnews <https://twitter.com/redhatnews>   Red Hat
> > <https://www.linkedin.com/company/red-hat>   Red Hat
> > <https://www.facebook.com/RedHatInc>
> >
> >
> > On Mon, Aug 28, 2017 at 9:11 PM, John Snow <jsnow@redhat.com
> > <mailto:jsnow@redhat.com>> wrote:
> >
> >
> >
> >     On 08/27/2017 10:57 PM, Fam Zheng wrote:
> >     > On Fri, 08/25 15:44, Max Reitz wrote:
> >     >> Well, OK.  The main argument against supporting anything but
> qcow2 is
> >     >> "if you want features, use qcow2; and we are working on making
> >     qcow2 as
> >     >> fast as possible."  I think that's a very good argument still.
> >     At some
> >     >> point I (and probably others, too) had the idea of making qcow2
> >     files in
> >     >> raw layout:
> >     >
> >     >
> >     > Yes! I think this idea makes a whole lot of sense, too. Metadata
> >     tables can be
> >     > generated so old implementation can still use it.
> >     >
> >     > Fam
> >     >
> >     >> Have the data as a blob, just like a raw file, padded by
> >     >> metadata around it.  An autoclear flag would specify that the
> >     qcow2 file
> >     >> is in this format, and if so, you could simply access it like a
> >     raw file
> >     >> and should have exactly the same speed as a raw file.  Maybe that
> >     would
> >     >> solve this whole issue, too?
> >
> >     I wonder if this would be sufficient to alleviate the desire to use
> raw
> >     files...
> >
> >     (Eh, well, realistically, someone's still always going to ask if they
> >     can use various features with non-qcow2 files...)
> >
> >     Nir, Yaniv; any input?
> >
> >
> > We are using raw format for performance reasons.
>
> Using raw layout for the data in a qcow2 file would give you exactly the
> same performance as raw.
>

We had no reason to switch to anything else so far and I'm sure this option
was not available when we started supporting raw format.


>
> (Or better "should", but I can't think of a reason why it would not.)
>
> Max
>
> > As we have many customers that currently use this format, not support it
> > would be a blocker the use of the feature.
> > At a minimum we would require ability to convert raw to qcow2 raw-layout.
> >
> > Please also consider that we are planning to go on the OSP route of LUN
> > per disk and would still want the tracking to work.
> > I makes sense that for that and raw format you will be able to save the
> > mapping to another file other than a qcow.
> >
> >
> >
> >     (Context: We're debating how to add persistent bitmaps to raw files
> as I
> >     was informed that RHV was 'asking about it.' Max is reminding me
> there
> >     is a proposal for a style of QCOW2 that uses a raw layout for data,
> >     mitigating or eliminating any performance hits related to the L2
> cache.
> >     What I am not aware of is why RHV would use raw files for any
> purpose.
> >     Is it performance? Simplicity? Could RHV use a raw-layout qcow2?)
> >
> >     --js
> >
> >
>
>
>

Re: [Qemu-devel] [Qemu-block] Persistent bitmaps for non-qcow2 formats

[Stefan Hajnoczi]

On Tue, Aug 22, 2017 at 03:07:04PM -0400, John Snow wrote:
> (3) Add either a new flag that turns qcow2's backing file into a full
> R/W backing file, or add a new extension to qcow2 entirely (bypassing
> the traditional backing file mechanism to avoid confusion for older
> tooling) that adds a new read-write backing file field.
> 
> This RW backing file field will be used for all reads AND writes; the
> qcow2 in question becomes a metadata container on top of the BDS chain.
> We can re-use Vladimir's bitmap persistence extension to save bitmaps in
> a qcow2 shell.
> 
> The qcow2 becomes effectively a metadata cache for a new (essentially)
> filter node that handles features such as bitmaps. This could also be
> used to provide allocation map data for RAW files and other goodies down
> the road.
> 
> Hopefully this achieves our desire to not create new formats AND our
> desire to concentrate features (and debugging, testing, etc) into qcow2,
> while allowing users to "have bitmaps with raw files."
> 
> Of course, in this scenario, users now have two files: a qcow2 wrapper
> and the actual raw file in question; but regardless of how we were going
> to solve this, a raw file necessitates an external file of some sort,
> else we give up the idea that it was a raw file.

There is some complexity here for management tools:

If the underlying image is resized, who resizes the qcow2 and how do
they know to do it?

If QEMU's resize/truncate command it used, does first try to resize the
underlying image and then resize the qcow2?  This is probably the sanest
approach.

If the underlying image is moved to a new location, does the qcow2 file
need to be modified and who does that?

Management tools need to figure out how to represent manage this extra
qcow2 file.  The easiest solution is to punt it to the user and treat it
as part of a backing file chain.  If the management tool wants to
automatically manage the qcow2 so the user just specifies the underlying
image and enables the persistent bitmap checkbox, then it becomes more
complicated.

Stefan

Re: [Qemu-devel] [Qemu-block] Persistent bitmaps for non-qcow2 formats

[Daniel P. Berrange]

On Wed, Aug 30, 2017 at 02:36:11PM +0100, Stefan Hajnoczi wrote:
> On Tue, Aug 22, 2017 at 03:07:04PM -0400, John Snow wrote:
> > (3) Add either a new flag that turns qcow2's backing file into a full
> > R/W backing file, or add a new extension to qcow2 entirely (bypassing
> > the traditional backing file mechanism to avoid confusion for older
> > tooling) that adds a new read-write backing file field.
> > 
> > This RW backing file field will be used for all reads AND writes; the
> > qcow2 in question becomes a metadata container on top of the BDS chain.
> > We can re-use Vladimir's bitmap persistence extension to save bitmaps in
> > a qcow2 shell.
> > 
> > The qcow2 becomes effectively a metadata cache for a new (essentially)
> > filter node that handles features such as bitmaps. This could also be
> > used to provide allocation map data for RAW files and other goodies down
> > the road.
> > 
> > Hopefully this achieves our desire to not create new formats AND our
> > desire to concentrate features (and debugging, testing, etc) into qcow2,
> > while allowing users to "have bitmaps with raw files."
> > 
> > Of course, in this scenario, users now have two files: a qcow2 wrapper
> > and the actual raw file in question; but regardless of how we were going
> > to solve this, a raw file necessitates an external file of some sort,
> > else we give up the idea that it was a raw file.
> 
> There is some complexity here for management tools:
> 
> If the underlying image is resized, who resizes the qcow2 and how do
> they know to do it?
> 
> If QEMU's resize/truncate command it used, does first try to resize the
> underlying image and then resize the qcow2?  This is probably the sanest
> approach.
> 
> If the underlying image is moved to a new location, does the qcow2 file
> need to be modified and who does that?
> 
> Management tools need to figure out how to represent manage this extra
> qcow2 file.  The easiest solution is to punt it to the user and treat it
> as part of a backing file chain.  If the management tool wants to
> automatically manage the qcow2 so the user just specifies the underlying
> image and enables the persistent bitmap checkbox, then it becomes more
> complicated.

Indeed, I don't think it is practical to have libvirt / QEMU automagically
create a qcow2 overlay on disk. Something has to decide where this would
be stored. You might say just put it alongside the raw file, but it might
not be a local file at all, it could be a NBD, or RBD raw "file". So do
we create  local qcow2 file, or store a qcow2 file inside another RBD
volume to hold the persistent bitmap. This kind of decision needs to be
made by the mgmt app since only it knows about its storage mgmt model.
At this point you might as well just let the mgmt app take care of it
all and not try to do anything magical with qcow2 overlays in libvirt/QEMU

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[John Snow]

On 08/30/2017 08:58 AM, Yaniv Lavi (Dary) wrote:
> 
> 
> We had no reason to switch to anything else so far and I'm sure this
> option was not available when we started supporting raw format. 
>  
> 

Yeah, they don't exist yet...! I've looped you in to see if the proposal
being discussed would alleviate the need for bitmaps for "other formats"
if we can offer a "raw mode qcow2."

At the moment I am going to still try to add bitmaps to other formats
(through the use of a qcow2 wrapper) but it sounds like a raw-layout
qcow2 might provide some benefits too.

-js

Re: [Qemu-devel] [Qemu-block] Persistent bitmaps for non-qcow2 formats

[John Snow]

On 08/30/2017 09:45 AM, Daniel P. Berrange wrote:
> On Wed, Aug 30, 2017 at 02:36:11PM +0100, Stefan Hajnoczi wrote:
>> On Tue, Aug 22, 2017 at 03:07:04PM -0400, John Snow wrote:
>>> (3) Add either a new flag that turns qcow2's backing file into a full
>>> R/W backing file, or add a new extension to qcow2 entirely (bypassing
>>> the traditional backing file mechanism to avoid confusion for older
>>> tooling) that adds a new read-write backing file field.
>>>
>>> This RW backing file field will be used for all reads AND writes; the
>>> qcow2 in question becomes a metadata container on top of the BDS chain.
>>> We can re-use Vladimir's bitmap persistence extension to save bitmaps in
>>> a qcow2 shell.
>>>
>>> The qcow2 becomes effectively a metadata cache for a new (essentially)
>>> filter node that handles features such as bitmaps. This could also be
>>> used to provide allocation map data for RAW files and other goodies down
>>> the road.
>>>
>>> Hopefully this achieves our desire to not create new formats AND our
>>> desire to concentrate features (and debugging, testing, etc) into qcow2,
>>> while allowing users to "have bitmaps with raw files."
>>>
>>> Of course, in this scenario, users now have two files: a qcow2 wrapper
>>> and the actual raw file in question; but regardless of how we were going
>>> to solve this, a raw file necessitates an external file of some sort,
>>> else we give up the idea that it was a raw file.
>>
>> There is some complexity here for management tools:
>>
>> If the underlying image is resized, who resizes the qcow2 and how do
>> they know to do it?
>>
>> If QEMU's resize/truncate command it used, does first try to resize the
>> underlying image and then resize the qcow2?  This is probably the sanest
>> approach.
>>
>> If the underlying image is moved to a new location, does the qcow2 file
>> need to be modified and who does that?
>>
>> Management tools need to figure out how to represent manage this extra
>> qcow2 file.  The easiest solution is to punt it to the user and treat it
>> as part of a backing file chain.  If the management tool wants to
>> automatically manage the qcow2 so the user just specifies the underlying
>> image and enables the persistent bitmap checkbox, then it becomes more
>> complicated.
> 
> Indeed, I don't think it is practical to have libvirt / QEMU automagically
> create a qcow2 overlay on disk. Something has to decide where this would
> be stored. You might say just put it alongside the raw file, but it might
> not be a local file at all, it could be a NBD, or RBD raw "file". So do
> we create  local qcow2 file, or store a qcow2 file inside another RBD
> volume to hold the persistent bitmap. This kind of decision needs to be
> made by the mgmt app since only it knows about its storage mgmt model.

Oh, you mean to say mgmt app like VMM or something even above libvirt,
yes? Who currently makes the decision for where snapshot files and the
like goes, does libvirt not decide that, but the app using it?

> At this point you might as well just let the mgmt app take care of it
> all and not try to do anything magical with qcow2 overlays in libvirt/QEMU
> 

Might be the sanest, yes -- but say I don't offer an "automagic" way to
add a bitmap to a raw file to a running QEMU instance but rather offer a
way to qcow2ify an existing node:

We could create a wrapper:

qemu-img create -f qcow2 -o wrapper wrapper.qcow2

Then add a node:

blockdev_add driver=qcow2 node-name=foo file.driver=file
file.filename=wrapper.qcow2

(The size of the drive could perhaps remain as zero temporarily here)

Then some magic to make it the active target of whatever blockbackend
was using the old image, perhaps?:

blockdev_magic target=virtioblk0 node-name=foo

At this point, virtioblk0 is now backed by our new magic qcow2 node
which provides bitmap features for whatever kind of backing storage
virtioblk0 had, and nothing automagic happened -- we passed explicit
file references.

Any magic that we wish to provide can happen in libvirt or above.

--js

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Yaniv Lavi (Dary)]

YANIV LAVI (YANIV DARY)

SENIOR TECHNICAL PRODUCT MANAGER

Red Hat Israel Ltd. <https://www.redhat.com/>

34 Jerusalem Road, Building A, 1st floor

Ra'anana, Israel 4350109

ylavi@redhat.com    T: +972-9-7692306/8272306     F: +972-9-7692223    IM: ylavi
 <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
@redhatnews <https://twitter.com/redhatnews>   Red Hat
<https://www.linkedin.com/company/red-hat>   Red Hat
<https://www.facebook.com/RedHatInc>


On Thu, Aug 31, 2017 at 12:25 AM, John Snow <jsnow@redhat.com> wrote:

>
>
> On 08/30/2017 08:58 AM, Yaniv Lavi (Dary) wrote:
> >
> >
> > We had no reason to switch to anything else so far and I'm sure this
> > option was not available when we started supporting raw format.
> >
> >
>
> Yeah, they don't exist yet...! I've looped you in to see if the proposal
> being discussed would alleviate the need for bitmaps for "other formats"
> if we can offer a "raw mode qcow2."
>

As we our direction is to move a file system/LUN per disk I don't think
adding a new type disk image is the right priority.
The below option of separate file sound like the right direction.


>
> At the moment I am going to still try to add bitmaps to other formats
> (through the use of a qcow2 wrapper) but it sounds like a raw-layout
> qcow2 might provide some benefits too.
>
> -js
>

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Kevin Wolf]

Am 22.08.2017 um 21:07 hat John Snow geschrieben:
> Well, we knew we'd want this sooner or later. I've got some pings
> downstream over whether or not we support persistent bitmaps for
> non-qcow2 formats.
> 
> Currently: no, we don't.

I was going to write some things about r/w backing file, and I might
still do that later, but for now, let me take a step back:

What is the use case here?

Using a specific image format is not a value in and of itself; it should
be chosen depending on which features you need. What prevents those
downstream users from simply choosing qcow2 for their data, too?

Kevin

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Kevin Wolf]

Am 28.08.2017 um 04:57 hat Fam Zheng geschrieben:
> On Fri, 08/25 15:44, Max Reitz wrote:
> > Well, OK.  The main argument against supporting anything but qcow2 is
> > "if you want features, use qcow2; and we are working on making qcow2 as
> > fast as possible."  I think that's a very good argument still.  At some
> > point I (and probably others, too) had the idea of making qcow2 files in
> > raw layout: 
> 
> Yes! I think this idea makes a whole lot of sense, too. Metadata tables can be
> generated so old implementation can still use it.

Maybe a nice way to attack this would be "huge pages", i.e. have a L1
entry flag that tells "this points directly to a huge cluster instead of
an L2 table". Gives us 512 MB clusters with a 64k cluster size, or a
maximum of 512 GB clusters with a 2 MB cluster size.

Huge clusters would only be used by qemu-img create if the respective
option is given, and only with preallocation.

Kevin

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Kevin Wolf]

Am 23.08.2017 um 19:44 hat John Snow geschrieben:
> On 08/23/2017 01:31 PM, Max Reitz wrote:
> > On 2017-08-22 21:07, John Snow wrote:
> > 
> > [...]
> > 
> >> (3) Add either a new flag that turns qcow2's backing file into a full
> >> R/W backing file, or add a new extension to qcow2 entirely (bypassing
> >> the traditional backing file mechanism to avoid confusion for older
> >> tooling) that adds a new read-write backing file field.
> >>
> >> This RW backing file field will be used for all reads AND writes; the
> >> qcow2 in question becomes a metadata container on top of the BDS chain.
> >> We can re-use Vladimir's bitmap persistence extension to save bitmaps in
> >> a qcow2 shell.
> >>
> >> The qcow2 becomes effectively a metadata cache for a new (essentially)
> >> filter node that handles features such as bitmaps. This could also be
> >> used to provide allocation map data for RAW files and other goodies down
> >> the road.
> >>
> >> Hopefully this achieves our desire to not create new formats AND our
> >> desire to concentrate features (and debugging, testing, etc) into qcow2,
> >> while allowing users to "have bitmaps with raw files."
> >>
> >> Of course, in this scenario, users now have two files: a qcow2 wrapper
> >> and the actual raw file in question; but regardless of how we were going
> >> to solve this, a raw file necessitates an external file of some sort,
> >> else we give up the idea that it was a raw file.
> >>
> >>
> >> Sound good?
> > 
> > While I don't quite like the idea of R/W backing files, I guess "don't
> > quite like" is still rather good.
> 
> Yeah, it's not necessarily my first pick, but it might be the least bad.

As for backing files always being read-only, that ship has sailed long
ago. We really never had it because I think HMP 'commit' was introduced
at the same time as backing files, but at least it was still contained
in synchronous function protected by the BQL. But at latest since we
introduced live block jobs, it's not a reasonable assumption to make any
more.

At this point, I have no problems with adding more users of writable
backing files any more. We'll have to use op blockers to make it safe.

> If you have any suggestions or alternatives for a way to accomplish the
> same in a way that does not leave any, even faint, bad taste in your
> mouth you are more than welcome to suggest it.

Maybe one thing that I don't want to dismiss right away would be a
separate BlockDriver for passthrough qcow2 images so that we don't
clutter all functions with special cases. But I'm not compeltely
convinced of it either.

I think I once had a use case for clusters that are marked as
"read/write to backing file", but I can't seem to remember it now. If we
had such a per-cluster setting, then a separate BlockDriver wouldn't be
an option, obviously.

Kevin

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Fam Zheng]

On Tue, 09/05 15:01, Kevin Wolf wrote:
> Am 28.08.2017 um 04:57 hat Fam Zheng geschrieben:
> > On Fri, 08/25 15:44, Max Reitz wrote:
> > > Well, OK.  The main argument against supporting anything but qcow2 is
> > > "if you want features, use qcow2; and we are working on making qcow2 as
> > > fast as possible."  I think that's a very good argument still.  At some
> > > point I (and probably others, too) had the idea of making qcow2 files in
> > > raw layout: 
> > 
> > Yes! I think this idea makes a whole lot of sense, too. Metadata tables can be
> > generated so old implementation can still use it.
> 
> Maybe a nice way to attack this would be "huge pages", i.e. have a L1
> entry flag that tells "this points directly to a huge cluster instead of
> an L2 table". Gives us 512 MB clusters with a 64k cluster size, or a
> maximum of 512 GB clusters with a 2 MB cluster size.
> 
> Huge clusters would only be used by qemu-img create if the respective
> option is given, and only with preallocation.

Then this image is not usable on old QEMUs, right? So this is going to be an
incompatible_feature, whereas with the linear mapping L1/L2 tables approach, it
can be an autoclear_feature.

Fam

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Kevin Wolf]

Am 05.09.2017 um 15:18 hat Fam Zheng geschrieben:
> On Tue, 09/05 15:01, Kevin Wolf wrote:
> > Am 28.08.2017 um 04:57 hat Fam Zheng geschrieben:
> > > On Fri, 08/25 15:44, Max Reitz wrote:
> > > > Well, OK.  The main argument against supporting anything but qcow2 is
> > > > "if you want features, use qcow2; and we are working on making qcow2 as
> > > > fast as possible."  I think that's a very good argument still.  At some
> > > > point I (and probably others, too) had the idea of making qcow2 files in
> > > > raw layout: 
> > > 
> > > Yes! I think this idea makes a whole lot of sense, too. Metadata tables can be
> > > generated so old implementation can still use it.
> > 
> > Maybe a nice way to attack this would be "huge pages", i.e. have a L1
> > entry flag that tells "this points directly to a huge cluster instead of
> > an L2 table". Gives us 512 MB clusters with a 64k cluster size, or a
> > maximum of 512 GB clusters with a 2 MB cluster size.
> > 
> > Huge clusters would only be used by qemu-img create if the respective
> > option is given, and only with preallocation.
> 
> Then this image is not usable on old QEMUs, right? So this is going to be an
> incompatible_feature, whereas with the linear mapping L1/L2 tables approach, it
> can be an autoclear_feature.

Ah yes, that's true. Maybe that's important enough that just a single
global flag is better, even if it's less flexible.

Of course, this approach also means that we still need to write all of
the L2 tables and use disk space for them, even though newer versions
will never look at them.

Kevin

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Fam Zheng]

On Tue, 09/05 15:27, Kevin Wolf wrote:
> Am 05.09.2017 um 15:18 hat Fam Zheng geschrieben:
> > On Tue, 09/05 15:01, Kevin Wolf wrote:
> > > Am 28.08.2017 um 04:57 hat Fam Zheng geschrieben:
> > > > On Fri, 08/25 15:44, Max Reitz wrote:
> > > > > Well, OK.  The main argument against supporting anything but qcow2 is
> > > > > "if you want features, use qcow2; and we are working on making qcow2 as
> > > > > fast as possible."  I think that's a very good argument still.  At some
> > > > > point I (and probably others, too) had the idea of making qcow2 files in
> > > > > raw layout: 
> > > > 
> > > > Yes! I think this idea makes a whole lot of sense, too. Metadata tables can be
> > > > generated so old implementation can still use it.
> > > 
> > > Maybe a nice way to attack this would be "huge pages", i.e. have a L1
> > > entry flag that tells "this points directly to a huge cluster instead of
> > > an L2 table". Gives us 512 MB clusters with a 64k cluster size, or a
> > > maximum of 512 GB clusters with a 2 MB cluster size.
> > > 
> > > Huge clusters would only be used by qemu-img create if the respective
> > > option is given, and only with preallocation.
> > 
> > Then this image is not usable on old QEMUs, right? So this is going to be an
> > incompatible_feature, whereas with the linear mapping L1/L2 tables approach, it
> > can be an autoclear_feature.
> 
> Ah yes, that's true. Maybe that's important enough that just a single
> global flag is better, even if it's less flexible.
> 
> Of course, this approach also means that we still need to write all of
> the L2 tables and use disk space for them, even though newer versions
> will never look at them.

To speculate^Wmitigate, "qemu-img create" is free to pick a larger cluster size
when this feature is used?

Fam

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Kevin Wolf]

Am 05.09.2017 um 15:39 hat Fam Zheng geschrieben:
> On Tue, 09/05 15:27, Kevin Wolf wrote:
> > Am 05.09.2017 um 15:18 hat Fam Zheng geschrieben:
> > > On Tue, 09/05 15:01, Kevin Wolf wrote:
> > > > Am 28.08.2017 um 04:57 hat Fam Zheng geschrieben:
> > > > > On Fri, 08/25 15:44, Max Reitz wrote:
> > > > > > Well, OK.  The main argument against supporting anything but qcow2 is
> > > > > > "if you want features, use qcow2; and we are working on making qcow2 as
> > > > > > fast as possible."  I think that's a very good argument still.  At some
> > > > > > point I (and probably others, too) had the idea of making qcow2 files in
> > > > > > raw layout: 
> > > > > 
> > > > > Yes! I think this idea makes a whole lot of sense, too. Metadata tables can be
> > > > > generated so old implementation can still use it.
> > > > 
> > > > Maybe a nice way to attack this would be "huge pages", i.e. have a L1
> > > > entry flag that tells "this points directly to a huge cluster instead of
> > > > an L2 table". Gives us 512 MB clusters with a 64k cluster size, or a
> > > > maximum of 512 GB clusters with a 2 MB cluster size.
> > > > 
> > > > Huge clusters would only be used by qemu-img create if the respective
> > > > option is given, and only with preallocation.
> > > 
> > > Then this image is not usable on old QEMUs, right? So this is going to be an
> > > incompatible_feature, whereas with the linear mapping L1/L2 tables approach, it
> > > can be an autoclear_feature.
> > 
> > Ah yes, that's true. Maybe that's important enough that just a single
> > global flag is better, even if it's less flexible.
> > 
> > Of course, this approach also means that we still need to write all of
> > the L2 tables and use disk space for them, even though newer versions
> > will never look at them.
> 
> To speculate^Wmitigate, "qemu-img create" is free to pick a larger cluster size
> when this feature is used?

Seems reasonable, though the 2 MB limit for the cluster size still
applies.

Kevin

Re: [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[Stefan Hajnoczi]

On Tue, Sep 05, 2017 at 01:46:24PM +0200, Kevin Wolf wrote:
> Am 22.08.2017 um 21:07 hat John Snow geschrieben:
> > Well, we knew we'd want this sooner or later. I've got some pings
> > downstream over whether or not we support persistent bitmaps for
> > non-qcow2 formats.
> > 
> > Currently: no, we don't.
> 
> I was going to write some things about r/w backing file, and I might
> still do that later, but for now, let me take a step back:
> 
> What is the use case here?

A user has 100s or 1000s of disk images over iSCSI.  They would like to
use the new incremental backup feature.  Converting all existing disk
images to qcow2 is not an option because it would be too invasive.

Stefan