[Qemu-devel] blobstore disk format (was Re: Design of the blobstore)

[Qemu-devel] blobstore disk format (was Re: Design of the blobstore)

[Michael S. Tsirkin]

So the below is a proposal for a directory scheme
for storing (optionally multiple) nvram images,
along with any metadata.
Data is encoded using BER:
http://en.wikipedia.org/wiki/Basic_Encoding_Rules
Specifically, we mostly use the subsets.

We use a directory as a SET in a CER format.
This allows generating directory online without scanning
the entries beforehand.

The rest of the encoding uses a DER format.
This makes for fast parsing as entries are easy to skip.

Each entry is encoded in DER format.
Each entry is a SEQUENCE with two objects:
1. nvram
2. optional name - a UTF8String

Binary data is stored as OCTET-STRING values on disk.
Any RW metadata is stored as OCTET-STRING value as well.
Any RO metadata is stored in appropriate universal encoding,
by type.

On the context below, an attribute is either a IA5String or a SEQUENCE.
If IA5String, this is the attribute name, and it has no value.
If SEQUENCE, the first entry in the sequence is an
IA5String, it is the attribute name. The rest of the entries
represent the attribute value.

Mandatory/optional attributes: depends on type.
tpm will have realsize as RW mandatory attribute.

Each nvram is built as a SEQUENCE including 4 objects
1. type - an IA5String. downstreams can use other types such as
                     UUIDs instead to ensure no conflicts with upstream
2. SET of mandatory attributes
3. SET of optional attributes
4. data - a RW OCTET-STRING

It is envisioned that attributes won't be too large,
so they can easily be kept in memory.


-- 
MST

Re: [Qemu-devel] blobstore disk format (was Re: Design of the blobstore)

[Stefan Berger]

On 09/15/2011 08:28 AM, Michael S. Tsirkin wrote:
> So the below is a proposal for a directory scheme
> for storing (optionally multiple) nvram images,
> along with any metadata.
> Data is encoded using BER:
> http://en.wikipedia.org/wiki/Basic_Encoding_Rules
> Specifically, we mostly use the subsets.
>
Would it change anything if we were to think of the NVRAM image as 
another piece of metadata?

I am also wondering whether each device shouldn't just handle the 
metadata itself, so generate a blob from data structures containing all 
the metadata it needs, arranging attribute and value pairs itself (maybe 
using some convenience function for serialization/deserialization) and 
let the NVRAM layer not handle the metadata at all but only blobs, their 
maximum sizes, actual sizes encryption, integrity value (crc32 or sha1) 
and so on. What metadata should there be that really need to be handled 
on the NVRAM API and below level rather than on the device-specific code 
level?

> We use a directory as a SET in a CER format.
> This allows generating directory online without scanning
> the entries beforehand.
>
I guess it is the 'unknown' for me... but what is the advantage of using 
ASN1 for this rather than just writing out packed and 
endianess-normalized data structures (with revision value), having them 
crc32-protected to have some sanity checking in place?

     Stefan
> The rest of the encoding uses a DER format.
> This makes for fast parsing as entries are easy to skip.
>
> Each entry is encoded in DER format.
> Each entry is a SEQUENCE with two objects:
> 1. nvram
> 2. optional name - a UTF8String
>
> Binary data is stored as OCTET-STRING values on disk.
> Any RW metadata is stored as OCTET-STRING value as well.
> Any RO metadata is stored in appropriate universal encoding,
> by type.
>
> On the context below, an attribute is either a IA5String or a SEQUENCE.
> If IA5String, this is the attribute name, and it has no value.
> If SEQUENCE, the first entry in the sequence is an
> IA5String, it is the attribute name. The rest of the entries
> represent the attribute value.
>
> Mandatory/optional attributes: depends on type.
> tpm will have realsize as RW mandatory attribute.
>
> Each nvram is built as a SEQUENCE including 4 objects
> 1. type - an IA5String. downstreams can use other types such as
>                       UUIDs instead to ensure no conflicts with upstream
> 2. SET of mandatory attributes
> 3. SET of optional attributes
> 4. data - a RW OCTET-STRING
>
> It is envisioned that attributes won't be too large,
> so they can easily be kept in memory.
>
>

Re: [Qemu-devel] blobstore disk format (was Re: Design of the blobstore)

[Michael S. Tsirkin]

On Thu, Sep 15, 2011 at 10:33:13AM -0400, Stefan Berger wrote:
> On 09/15/2011 08:28 AM, Michael S. Tsirkin wrote:
> >So the below is a proposal for a directory scheme
> >for storing (optionally multiple) nvram images,
> >along with any metadata.
> >Data is encoded using BER:
> >http://en.wikipedia.org/wiki/Basic_Encoding_Rules
> >Specifically, we mostly use the subsets.
> >
> Would it change anything if we were to think of the NVRAM image as
> another piece of metadata?

Yes, we can do that, sure. I had the feeling that it will help to lay
out the image at the end, to make directory listing
more efficient - the rest of metadata is usually small,
image might be somewhat large.

> I am also wondering whether each device shouldn't just handle the
> metadata itself,

It could be that just means we will have custom code with
different bugs in each device.
Note that from experience with formats, the problem with
time becomes less trivial than it seems as we
need to provide forward and backward compatibility
guarantees.

> so generate a blob from data structures containing
> all the metadata it needs, arranging attribute and value pairs
> itself (maybe using some convenience function for
> serialization/deserialization) and let the NVRAM layer not handle
> the metadata at all but only blobs, their maximum sizes, actual
> sizes

Actual size seems to be a TPM specific thing.

> encryption, integrity value (crc32 or sha1) and so on. What
> metadata should there be that really need to be handled on the NVRAM
> API and below level rather than on the device-specific code level?

So checksum  (checksum value and type) 'and so on' are what I call
metadata :) Doing it at device level seems wrong.

> >We use a directory as a SET in a CER format.
> >This allows generating directory online without scanning
> >the entries beforehand.
> >
> I guess it is the 'unknown' for me... but what is the advantage of
> using ASN1 for this rather than just writing out packed and
> endianess-normalized data structures (with revision value),

If you want an example of where this 'custom formats are easy
so let us write one' leads to in the end,
look no further than live migration code.
It's a mess of hacks that does not even work across
upstream qemu versions, leave alone across
downstreams (different linux distros).

> having
> them crc32-protected to have some sanity checking in place?
> 
>     Stefan

I'm not sure why we want crc specifically in TPM.
If it is 'just because we can' then it probably
applies to other non-volatile storage?
Storage generally?

> >The rest of the encoding uses a DER format.
> >This makes for fast parsing as entries are easy to skip.
> >
> >Each entry is encoded in DER format.
> >Each entry is a SEQUENCE with two objects:
> >1. nvram
> >2. optional name - a UTF8String
> >
> >Binary data is stored as OCTET-STRING values on disk.
> >Any RW metadata is stored as OCTET-STRING value as well.
> >Any RO metadata is stored in appropriate universal encoding,
> >by type.
> >
> >On the context below, an attribute is either a IA5String or a SEQUENCE.
> >If IA5String, this is the attribute name, and it has no value.
> >If SEQUENCE, the first entry in the sequence is an
> >IA5String, it is the attribute name. The rest of the entries
> >represent the attribute value.
> >
> >Mandatory/optional attributes: depends on type.
> >tpm will have realsize as RW mandatory attribute.
> >
> >Each nvram is built as a SEQUENCE including 4 objects
> >1. type - an IA5String. downstreams can use other types such as
> >                      UUIDs instead to ensure no conflicts with upstream
> >2. SET of mandatory attributes
> >3. SET of optional attributes
> >4. data - a RW OCTET-STRING
> >
> >It is envisioned that attributes won't be too large,
> >so they can easily be kept in memory.
> >
> >

Re: [Qemu-devel] blobstore disk format (was Re: Design of the blobstore)

[Stefan Berger]

On 09/16/2011 10:44 AM, Michael S. Tsirkin wrote:
> On Thu, Sep 15, 2011 at 10:33:13AM -0400, Stefan Berger wrote:
>> On 09/15/2011 08:28 AM, Michael S. Tsirkin wrote:
>>> So the below is a proposal for a directory scheme
>>> for storing (optionally multiple) nvram images,
>>> along with any metadata.
>>> Data is encoded using BER:
>>> http://en.wikipedia.org/wiki/Basic_Encoding_Rules
>>> Specifically, we mostly use the subsets.
>>>
>> Would it change anything if we were to think of the NVRAM image as
>> another piece of metadata?
> Yes, we can do that, sure. I had the feeling that it will help to lay
> out the image at the end, to make directory listing
> more efficient - the rest of metadata is usually small,
> image might be somewhat large.
>
Why not let a convenience library handle the metadata on the device 
level, having it create the blob that the NVRAM layer ends up writing 
and parsing before the device uses it? Otherwise I should maybe rename 
the nvram to meatdata_store :-/
>> I am also wondering whether each device shouldn't just handle the
>> metadata itself,
> It could be that just means we will have custom code with
> different bugs in each device.
> Note that from experience with formats, the problem with
> time becomes less trivial than it seems as we
> need to provide forward and backward compatibility
> guarantees.
>
Is that guaranteed just by using ASN.1 ? Do we need to add a revision to 
the metadata? How do we handle metadata that was to change over time, 
i.e., new attribute/values being added into a finite store?
>> so generate a blob from data structures containing
>> all the metadata it needs, arranging attribute and value pairs
>> itself (maybe using some convenience function for
>> serialization/deserialization) and let the NVRAM layer not handle
>> the metadata at all but only blobs, their maximum sizes, actual
>> sizes
> Actual size seems to be a TPM specific thing.
>
Yes, it could also be metadata. One should probably always be allowed to 
write a shorter blob than registered, but not a longer one. If the 
device did that, maybe it should assume it needs to prepend a header to 
the actual blob indicating the actual size of the following blob so 
trailing garbage can be ignored.
>> encryption, integrity value (crc32 or sha1) and so on. What
>> metadata should there be that really need to be handled on the NVRAM
>> API and below level rather than on the device-specific code level?
> So checksum  (checksum value and type) 'and so on' are what I call
> metadata :) Doing it at device level seems wrong.
>
You mean doing it at the NVRAM level seems wrong. Of course, again 
something a device could write into a header prepended to the actual 
blob. Maybe every device that needs it should do that so that if we were 
to support encryption of blobs and the key for decryption was wrong one 
could detect it early without feeding badly decrypted / corrupted state 
into the device and see what happens.
>>> We use a directory as a SET in a CER format.
>>> This allows generating directory online without scanning
>>> the entries beforehand.
>>>
>> I guess it is the 'unknown' for me... but what is the advantage of
>> using ASN1 for this rather than just writing out packed and
>> endianess-normalized data structures (with revision value),
> If you want an example of where this 'custom formats are easy
> so let us write one' leads to in the end,
> look no further than live migration code.
> It's a mess of hacks that does not even work across
> upstream qemu versions, leave alone across
> downstreams (different linux distros).
>
So is ASN1 the answer or does one still need to add a revision tag to 
each blob putting in custom code for parsing the different revisions of 
data structures (I guess) that may be extended/changed over time?

    Stefan
>> having
>> them crc32-protected to have some sanity checking in place?
>>
>>      Stefan
> I'm not sure why we want crc specifically in TPM.
> If it is 'just because we can' then it probably
> applies to other non-volatile storage?
> Storage generally?
>
>>> The rest of the encoding uses a DER format.
>>> This makes for fast parsing as entries are easy to skip.
>>>
>>> Each entry is encoded in DER format.
>>> Each entry is a SEQUENCE with two objects:
>>> 1. nvram
>>> 2. optional name - a UTF8String
>>>
>>> Binary data is stored as OCTET-STRING values on disk.
>>> Any RW metadata is stored as OCTET-STRING value as well.
>>> Any RO metadata is stored in appropriate universal encoding,
>>> by type.
>>>
>>> On the context below, an attribute is either a IA5String or a SEQUENCE.
>>> If IA5String, this is the attribute name, and it has no value.
>>> If SEQUENCE, the first entry in the sequence is an
>>> IA5String, it is the attribute name. The rest of the entries
>>> represent the attribute value.
>>>
>>> Mandatory/optional attributes: depends on type.
>>> tpm will have realsize as RW mandatory attribute.
>>>
>>> Each nvram is built as a SEQUENCE including 4 objects
>>> 1. type - an IA5String. downstreams can use other types such as
>>>                       UUIDs instead to ensure no conflicts with upstream
>>> 2. SET of mandatory attributes
>>> 3. SET of optional attributes
>>> 4. data - a RW OCTET-STRING
>>>
>>> It is envisioned that attributes won't be too large,
>>> so they can easily be kept in memory.
>>>
>>>

Re: [Qemu-devel] blobstore disk format (was Re: Design of the blobstore)

[Michael S. Tsirkin]

On Fri, Sep 16, 2011 at 12:46:40PM -0400, Stefan Berger wrote:
> On 09/16/2011 10:44 AM, Michael S. Tsirkin wrote:
> >On Thu, Sep 15, 2011 at 10:33:13AM -0400, Stefan Berger wrote:
> >>On 09/15/2011 08:28 AM, Michael S. Tsirkin wrote:
> >>>So the below is a proposal for a directory scheme
> >>>for storing (optionally multiple) nvram images,
> >>>along with any metadata.
> >>>Data is encoded using BER:
> >>>http://en.wikipedia.org/wiki/Basic_Encoding_Rules
> >>>Specifically, we mostly use the subsets.
> >>>
> >>Would it change anything if we were to think of the NVRAM image as
> >>another piece of metadata?
> >Yes, we can do that, sure. I had the feeling that it will help to lay
> >out the image at the end, to make directory listing
> >more efficient - the rest of metadata is usually small,
> >image might be somewhat large.
> >
> Why not let a convenience library handle the metadata on the device
> level, having it create the blob that the NVRAM layer ends up
> writing and parsing before the device uses it? Otherwise I should
> maybe rename the nvram to meatdata_store :-/

Maybe we are talking about different things. All I agrue for
is using a common standard format for storing metadata,
instead of having each device roll its own.

> >>I am also wondering whether each device shouldn't just handle the
> >>metadata itself,
> >It could be that just means we will have custom code with
> >different bugs in each device.
> >Note that from experience with formats, the problem with
> >time becomes less trivial than it seems as we
> >need to provide forward and backward compatibility
> >guarantees.
> >
> Is that guaranteed just by using ASN.1 ?

At least for BER, yes. We can always skip an optional field
that we don't recognize without knowing anything about
its internal format.

> Do we need to add a
> revision to the metadata?

IMO, no. Instead we add optional attributes as long as we can
preserve backwards compatibility, and madatory attributes
if we can't.

> How do we handle metadata that was to
> change over time, i.e., new attribute/values being added into a
> finite store?

Add them as optional attributes.

> >>so generate a blob from data structures containing
> >>all the metadata it needs, arranging attribute and value pairs
> >>itself (maybe using some convenience function for
> >>serialization/deserialization) and let the NVRAM layer not handle
> >>the metadata at all but only blobs, their maximum sizes, actual
> >>sizes
> >Actual size seems to be a TPM specific thing.
> >
> Yes, it could also be metadata. One should probably always be
> allowed to write a shorter blob than registered, but not a longer
> one. If the device did that, maybe it should assume it needs to
> prepend a header to the actual blob indicating the actual size of
> the following blob so trailing garbage can be ignored.

And then when we need more info we get to deal with
versioning of that header.

> >>encryption, integrity value (crc32 or sha1) and so on. What
> >>metadata should there be that really need to be handled on the NVRAM
> >>API and below level rather than on the device-specific code level?
> >So checksum  (checksum value and type) 'and so on' are what I call
> >metadata :) Doing it at device level seems wrong.
> >
> You mean doing it at the NVRAM level seems wrong. Of course, again
> something a device could write into a header prepended to the actual
> blob. Maybe every device that needs it should do that so that if we
> were to support encryption of blobs and the key for decryption was
> wrong one could detect it early without feeding badly decrypted /
> corrupted state into the device and see what happens.

Do what? Checksum the data? Well, error detection is nice,
but it could be that people actually care about not losing
all of the data on nvram if qemu is killed.  I also wonder whether
invalidating all data because of a single bit corruption is a bug or a
feature.

> >>>We use a directory as a SET in a CER format.
> >>>This allows generating directory online without scanning
> >>>the entries beforehand.
> >>>
> >>I guess it is the 'unknown' for me... but what is the advantage of
> >>using ASN1 for this rather than just writing out packed and
> >>endianess-normalized data structures (with revision value),
> >If you want an example of where this 'custom formats are easy
> >so let us write one' leads to in the end,
> >look no further than live migration code.
> >It's a mess of hacks that does not even work across
> >upstream qemu versions, leave alone across
> >downstreams (different linux distros).
> >
> So is ASN1 the answer or does one still need to add a revision tag
> to each blob putting in custom code for parsing the different
> revisions of data structures (I guess) that may be extended/changed
> over time?
> 
>    Stefan

We don't need revisions. We can always parse a new structure
skipping optional attributes we don't recognize. In case we want to
break old qemu versions intentially, we can add
a mandatory attribute.

> >>having
> >>them crc32-protected to have some sanity checking in place?
> >>
> >>     Stefan
> >I'm not sure why we want crc specifically in TPM.
> >If it is 'just because we can' then it probably
> >applies to other non-volatile storage?
> >Storage generally?
> >
> >>>The rest of the encoding uses a DER format.
> >>>This makes for fast parsing as entries are easy to skip.
> >>>
> >>>Each entry is encoded in DER format.
> >>>Each entry is a SEQUENCE with two objects:
> >>>1. nvram
> >>>2. optional name - a UTF8String
> >>>
> >>>Binary data is stored as OCTET-STRING values on disk.
> >>>Any RW metadata is stored as OCTET-STRING value as well.
> >>>Any RO metadata is stored in appropriate universal encoding,
> >>>by type.
> >>>
> >>>On the context below, an attribute is either a IA5String or a SEQUENCE.
> >>>If IA5String, this is the attribute name, and it has no value.
> >>>If SEQUENCE, the first entry in the sequence is an
> >>>IA5String, it is the attribute name. The rest of the entries
> >>>represent the attribute value.
> >>>
> >>>Mandatory/optional attributes: depends on type.
> >>>tpm will have realsize as RW mandatory attribute.
> >>>
> >>>Each nvram is built as a SEQUENCE including 4 objects
> >>>1. type - an IA5String. downstreams can use other types such as
> >>>                      UUIDs instead to ensure no conflicts with upstream
> >>>2. SET of mandatory attributes
> >>>3. SET of optional attributes
> >>>4. data - a RW OCTET-STRING
> >>>
> >>>It is envisioned that attributes won't be too large,
> >>>so they can easily be kept in memory.
> >>>
> >>>

Re: [Qemu-devel] blobstore disk format (was Re: Design of the blobstore)

[Stefan Berger]

On 09/17/2011 03:28 PM, Michael S. Tsirkin wrote:
> On Fri, Sep 16, 2011 at 12:46:40PM -0400, Stefan Berger wrote:
>> On 09/16/2011 10:44 AM, Michael S. Tsirkin wrote:
>>> On Thu, Sep 15, 2011 at 10:33:13AM -0400, Stefan Berger wrote:
>>>> On 09/15/2011 08:28 AM, Michael S. Tsirkin wrote:
>>>>> So the below is a proposal for a directory scheme
>>>>> for storing (optionally multiple) nvram images,
>>>>> along with any metadata.
>>>>> Data is encoded using BER:
>>>>> http://en.wikipedia.org/wiki/Basic_Encoding_Rules
>>>>> Specifically, we mostly use the subsets.
>>>>>
>>>> Would it change anything if we were to think of the NVRAM image as
>>>> another piece of metadata?
>>> Yes, we can do that, sure. I had the feeling that it will help to lay
>>> out the image at the end, to make directory listing
>>> more efficient - the rest of metadata is usually small,
>>> image might be somewhat large.
>>>
>> Why not let a convenience library handle the metadata on the device
>> level, having it create the blob that the NVRAM layer ends up
>> writing and parsing before the device uses it? Otherwise I should
>> maybe rename the nvram to meatdata_store :-/
> Maybe we are talking about different things. All I agrue for
> is using a common standard format for storing metadata,
> instead of having each device roll its own.
That's fine. The TPM code inside libtpms serializes all internal data 
structures for later resumption. It doesn't use ASN.1 but in effect 
endianess-normalizes them and stores them in a packed format that can 
later be resumed along with a version tag prepended where need . Are you 
suggesting to change that ? I hope not...



>>>> I am also wondering whether each device shouldn't just handle the
>>>> metadata itself,
>>> It could be that just means we will have custom code with
>>> different bugs in each device.
>>> Note that from experience with formats, the problem with
>>> time becomes less trivial than it seems as we
>>> need to provide forward and backward compatibility
>>> guarantees.
>>>
>> Is that guaranteed just by using ASN.1 ?
> At least for BER, yes. We can always skip an optional field
> that we don't recognize without knowing anything about
> its internal format.

>> Do we need to add a
>> revision to the metadata?
> IMO, no. Instead we add optional attributes as long as we can
> preserve backwards compatibility, and madatory attributes
> if we can't.
>
Are devices doing this right now or are these future changes to devices' 
code?

>>>> encryption, integrity value (crc32 or sha1) and so on. What
>>>> metadata should there be that really need to be handled on the NVRAM
>>>> API and below level rather than on the device-specific code level?
>>> So checksum  (checksum value and type) 'and so on' are what I call
>>> metadata :) Doing it at device level seems wrong.
>>>
>> You mean doing it at the NVRAM level seems wrong. Of course, again
>> something a device could write into a header prepended to the actual
>> blob. Maybe every device that needs it should do that so that if we
>> were to support encryption of blobs and the key for decryption was
>> wrong one could detect it early without feeding badly decrypted /
>> corrupted state into the device and see what happens.
> Do what? Checksum the data? Well, error detection is nice,
> but it could be that people actually care about not losing
> all of the data on nvram if qemu is killed.  I also wonder whether
> invalidating all data because of a single bit corruption is a bug or a
> feature.
>
The checksuming I think makes sense if encryption is being added so 
decryption and testing for proper key material remains an NVRAM 
operation rather than a device operation.
>>>>> We use a directory as a SET in a CER format.
>>>>> This allows generating directory online without scanning
>>>>> the entries beforehand.
>>>>>
>>>> I guess it is the 'unknown' for me... but what is the advantage of
>>>> using ASN1 for this rather than just writing out packed and
>>>> endianess-normalized data structures (with revision value),
>>> If you want an example of where this 'custom formats are easy
>>> so let us write one' leads to in the end,
>>> look no further than live migration code.
>>> It's a mess of hacks that does not even work across
>>> upstream qemu versions, leave alone across
>>> downstreams (different linux distros).
>>>
>> So is ASN1 the answer or does one still need to add a revision tag
>> to each blob putting in custom code for parsing the different
>> revisions of data structures (I guess) that may be extended/changed
>> over time?
>>
>>     Stefan
> We don't need revisions. We can always parse a new structure
> skipping optional attributes we don't recognize. In case we want to
> break old qemu versions intentially, we can add
> a mandatory attribute.
So you said you had some code for the handling of ASN.1. Can sketch how 
the interaction of devices would work with mandatory and optional 
attributes along with an API? I'd prefer to NOT have the attributes and 
values be a part of the NVRAM API itself but let a (mandatory) library 
handle the serialization and deserialization of these metadata when a 
device wants to write or read state respectively. But maybe I just want 
to keep the NVRAM API 'too simple'.

    Stefan

>>>> having
>>>> them crc32-protected to have some sanity checking in place?
>>>>
>>>>      Stefan
>>> I'm not sure why we want crc specifically in TPM.
>>> If it is 'just because we can' then it probably
>>> applies to other non-volatile storage?
>>> Storage generally?
>>>
>>>>> The rest of the encoding uses a DER format.
>>>>> This makes for fast parsing as entries are easy to skip.
>>>>>
>>>>> Each entry is encoded in DER format.
>>>>> Each entry is a SEQUENCE with two objects:
>>>>> 1. nvram
>>>>> 2. optional name - a UTF8String
>>>>>
>>>>> Binary data is stored as OCTET-STRING values on disk.
>>>>> Any RW metadata is stored as OCTET-STRING value as well.
>>>>> Any RO metadata is stored in appropriate universal encoding,
>>>>> by type.
>>>>>
>>>>> On the context below, an attribute is either a IA5String or a SEQUENCE.
>>>>> If IA5String, this is the attribute name, and it has no value.
>>>>> If SEQUENCE, the first entry in the sequence is an
>>>>> IA5String, it is the attribute name. The rest of the entries
>>>>> represent the attribute value.
>>>>>
>>>>> Mandatory/optional attributes: depends on type.
>>>>> tpm will have realsize as RW mandatory attribute.
>>>>>
>>>>> Each nvram is built as a SEQUENCE including 4 objects
>>>>> 1. type - an IA5String. downstreams can use other types such as
>>>>>                       UUIDs instead to ensure no conflicts with upstream
>>>>> 2. SET of mandatory attributes
>>>>> 3. SET of optional attributes
>>>>> 4. data - a RW OCTET-STRING
>>>>>
>>>>> It is envisioned that attributes won't be too large,
>>>>> so they can easily be kept in memory.
>>>>>
>>>>>

Re: [Qemu-devel] blobstore disk format (was Re: Design of the blobstore)

[Michael S. Tsirkin]

On Mon, Sep 19, 2011 at 12:22:02PM -0400, Stefan Berger wrote:
> On 09/17/2011 03:28 PM, Michael S. Tsirkin wrote:
> >On Fri, Sep 16, 2011 at 12:46:40PM -0400, Stefan Berger wrote:
> >>On 09/16/2011 10:44 AM, Michael S. Tsirkin wrote:
> >>>On Thu, Sep 15, 2011 at 10:33:13AM -0400, Stefan Berger wrote:
> >>>>On 09/15/2011 08:28 AM, Michael S. Tsirkin wrote:
> >>>>>So the below is a proposal for a directory scheme
> >>>>>for storing (optionally multiple) nvram images,
> >>>>>along with any metadata.
> >>>>>Data is encoded using BER:
> >>>>>http://en.wikipedia.org/wiki/Basic_Encoding_Rules
> >>>>>Specifically, we mostly use the subsets.
> >>>>>
> >>>>Would it change anything if we were to think of the NVRAM image as
> >>>>another piece of metadata?
> >>>Yes, we can do that, sure. I had the feeling that it will help to lay
> >>>out the image at the end, to make directory listing
> >>>more efficient - the rest of metadata is usually small,
> >>>image might be somewhat large.
> >>>
> >>Why not let a convenience library handle the metadata on the device
> >>level, having it create the blob that the NVRAM layer ends up
> >>writing and parsing before the device uses it? Otherwise I should
> >>maybe rename the nvram to meatdata_store :-/
> >Maybe we are talking about different things. All I agrue for
> >is using a common standard format for storing metadata,
> >instead of having each device roll its own.
> That's fine. The TPM code inside libtpms serializes all internal
> data structures for later resumption. It doesn't use ASN.1 but in
> effect endianess-normalizes them and stores them in a packed format
> that can later be resumed along with a version tag prepended where
> need . Are you suggesting to change that ? I hope not...

I'm guessing that it makes sense to use whatever libtpms does
as is - if we change the format we'd have to parse it ourselves.
We can use it as a binary blob.
It's only when we add our own versioning/structure
on top of it, such as adding actual size, where
using a standard format makes sense IMO.

> >>>>I am also wondering whether each device shouldn't just handle the
> >>>>metadata itself,
> >>>It could be that just means we will have custom code with
> >>>different bugs in each device.
> >>>Note that from experience with formats, the problem with
> >>>time becomes less trivial than it seems as we
> >>>need to provide forward and backward compatibility
> >>>guarantees.
> >>>
> >>Is that guaranteed just by using ASN.1 ?
> >At least for BER, yes. We can always skip an optional field
> >that we don't recognize without knowing anything about
> >its internal format.
> 
> >>Do we need to add a
> >>revision to the metadata?
> >IMO, no. Instead we add optional attributes as long as we can
> >preserve backwards compatibility, and madatory attributes
> >if we can't.
> >
> Are devices doing this right now or are these future changes to
> devices' code?

The don't do right now.
AFAIK Anthony's working on changing devices to use a visitor
for migration, and I want to add a visitor backend
that will use BER format.

> >>>>encryption, integrity value (crc32 or sha1) and so on. What
> >>>>metadata should there be that really need to be handled on the NVRAM
> >>>>API and below level rather than on the device-specific code level?
> >>>So checksum  (checksum value and type) 'and so on' are what I call
> >>>metadata :) Doing it at device level seems wrong.
> >>>
> >>You mean doing it at the NVRAM level seems wrong. Of course, again
> >>something a device could write into a header prepended to the actual
> >>blob. Maybe every device that needs it should do that so that if we
> >>were to support encryption of blobs and the key for decryption was
> >>wrong one could detect it early without feeding badly decrypted /
> >>corrupted state into the device and see what happens.
> >Do what? Checksum the data? Well, error detection is nice,
> >but it could be that people actually care about not losing
> >all of the data on nvram if qemu is killed.  I also wonder whether
> >invalidating all data because of a single bit corruption is a bug or a
> >feature.
> >
> The checksuming I think makes sense if encryption is being added so
> decryption and testing for proper key material remains an NVRAM
> operation rather than a device operation.

Not sure how this addresses the question of what to do
on checksum failure.

> >>>>>We use a directory as a SET in a CER format.
> >>>>>This allows generating directory online without scanning
> >>>>>the entries beforehand.
> >>>>>
> >>>>I guess it is the 'unknown' for me... but what is the advantage of
> >>>>using ASN1 for this rather than just writing out packed and
> >>>>endianess-normalized data structures (with revision value),
> >>>If you want an example of where this 'custom formats are easy
> >>>so let us write one' leads to in the end,
> >>>look no further than live migration code.
> >>>It's a mess of hacks that does not even work across
> >>>upstream qemu versions, leave alone across
> >>>downstreams (different linux distros).
> >>>
> >>So is ASN1 the answer or does one still need to add a revision tag
> >>to each blob putting in custom code for parsing the different
> >>revisions of data structures (I guess) that may be extended/changed
> >>over time?
> >>
> >>    Stefan
> >We don't need revisions. We can always parse a new structure
> >skipping optional attributes we don't recognize. In case we want to
> >break old qemu versions intentially, we can add
> >a mandatory attribute.
> So you said you had some code for the handling of ASN.1. Can sketch
> how the interaction of devices would work with mandatory and
> optional attributes along with an API?

So for now what I'm happy with is a low level BER parser API.
NVRAM would implement a scheme on top of that
and bdrv. I'll pack up what I have and post ASAP.

> I'd prefer to NOT have the
> attributes and values be a part of the NVRAM API itself but let a
> (mandatory) library handle the serialization and deserialization of
> these metadata when a device wants to write or read state
> respectively. But maybe I just want to keep the NVRAM API 'too
> simple'.
> 
>    Stefan

I'm fine with that. Still tpm has some device specific
things like 'actual size' so the api will need to address that
in some generic way.

> >>>>having
> >>>>them crc32-protected to have some sanity checking in place?
> >>>>
> >>>>     Stefan
> >>>I'm not sure why we want crc specifically in TPM.
> >>>If it is 'just because we can' then it probably
> >>>applies to other non-volatile storage?
> >>>Storage generally?
> >>>
> >>>>>The rest of the encoding uses a DER format.
> >>>>>This makes for fast parsing as entries are easy to skip.
> >>>>>
> >>>>>Each entry is encoded in DER format.
> >>>>>Each entry is a SEQUENCE with two objects:
> >>>>>1. nvram
> >>>>>2. optional name - a UTF8String
> >>>>>
> >>>>>Binary data is stored as OCTET-STRING values on disk.
> >>>>>Any RW metadata is stored as OCTET-STRING value as well.
> >>>>>Any RO metadata is stored in appropriate universal encoding,
> >>>>>by type.
> >>>>>
> >>>>>On the context below, an attribute is either a IA5String or a SEQUENCE.
> >>>>>If IA5String, this is the attribute name, and it has no value.
> >>>>>If SEQUENCE, the first entry in the sequence is an
> >>>>>IA5String, it is the attribute name. The rest of the entries
> >>>>>represent the attribute value.
> >>>>>
> >>>>>Mandatory/optional attributes: depends on type.
> >>>>>tpm will have realsize as RW mandatory attribute.
> >>>>>
> >>>>>Each nvram is built as a SEQUENCE including 4 objects
> >>>>>1. type - an IA5String. downstreams can use other types such as
> >>>>>                      UUIDs instead to ensure no conflicts with upstream
> >>>>>2. SET of mandatory attributes
> >>>>>3. SET of optional attributes
> >>>>>4. data - a RW OCTET-STRING
> >>>>>
> >>>>>It is envisioned that attributes won't be too large,
> >>>>>so they can easily be kept in memory.
> >>>>>
> >>>>>

Re: [Qemu-devel] blobstore disk format (was Re: Design of the blobstore)

[Stefan Berger]

On 09/19/2011 03:04 PM, Michael S. Tsirkin wrote:
> On Mon, Sep 19, 2011 at 12:22:02PM -0400, Stefan Berger wrote:
>> On 09/17/2011 03:28 PM, Michael S. Tsirkin wrote:
>>> On Fri, Sep 16, 2011 at 12:46:40PM -0400, Stefan Berger wrote:
>>
>> The checksuming I think makes sense if encryption is being added so
>> decryption and testing for proper key material remains an NVRAM
>> operation rather than a device operation.
> Not sure how this addresses the question of what to do
> on checksum failure.
>
Checksum failure on an unencrypted blob would mean that the blob is 
corrupted. In case of encryption 'corrupted' would overlap with a 'badly 
decrypted' blob. In either way the startup of the device cannot happen. 
We could refuse the NVRAM key suggesting that likely this is the wrong 
key for decryption but corruption is also possible.
>>>>>>> We use a directory as a SET in a CER format.
>>>>>>> This allows generating directory online without scanning
>>>>>>> the entries beforehand.
>>>>>>>
>>>>>> I guess it is the 'unknown' for me... but what is the advantage of
>>>>>> using ASN1 for this rather than just writing out packed and
>>>>>> endianess-normalized data structures (with revision value),
>>>>> If you want an example of where this 'custom formats are easy
>>>>> so let us write one' leads to in the end,
>>>>> look no further than live migration code.
>>>>> It's a mess of hacks that does not even work across
>>>>> upstream qemu versions, leave alone across
>>>>> downstreams (different linux distros).
>>>>>
>>>> So is ASN1 the answer or does one still need to add a revision tag
>>>> to each blob putting in custom code for parsing the different
>>>> revisions of data structures (I guess) that may be extended/changed
>>>> over time?
>>>>
>>>>     Stefan
>>> We don't need revisions. We can always parse a new structure
>>> skipping optional attributes we don't recognize. In case we want to
>>> break old qemu versions intentially, we can add
>>> a mandatory attribute.
>> So you said you had some code for the handling of ASN.1. Can sketch
>> how the interaction of devices would work with mandatory and
>> optional attributes along with an API?
> So for now what I'm happy with is a low level BER parser API.
> NVRAM would implement a scheme on top of that
> and bdrv. I'll pack up what I have and post ASAP.
>
I had thought that registration of a blob could include the registration 
of something similar to the  VMStateDescription's fields array 
(VMStateField[]) that currently is not encoded as ASN.1 but seems to be 
heading that way. I am not sure what the time frame for this is but I 
guess recycling as much code as possible would be helpful. For the 
purpose of NVRAM we would have to be able to determine the size of these 
additional metadata, something that is not necessary for the case of VM 
migration/suspend. I guess using VIsitors it should be possible to 
implement this type of functionality. Also extending the metadata in the 
future with new fields will be a lot harder with NVRAM due to the 
limited space we will have reserved when the blob + metadata was written 
out the first time.

>> I'd prefer to NOT have the
>> attributes and values be a part of the NVRAM API itself but let a
>> (mandatory) library handle the serialization and deserialization of
>> these metadata when a device wants to write or read state
>> respectively. But maybe I just want to keep the NVRAM API 'too
>> simple'.
>>
>>     Stefan
> I'm fine with that. Still tpm has some device specific
> things like 'actual size' so the api will need to address that
> in some generic way.
Maybe the above mentioned registration of the VMStateField[] could help.

    Stefan

Re: [Qemu-devel] blobstore disk format (was Re: Design of the blobstore)

[Michael S. Tsirkin]

On Wed, Sep 21, 2011 at 09:44:37PM -0400, Stefan Berger wrote:
> On 09/19/2011 03:04 PM, Michael S. Tsirkin wrote:
> >On Mon, Sep 19, 2011 at 12:22:02PM -0400, Stefan Berger wrote:
> >>On 09/17/2011 03:28 PM, Michael S. Tsirkin wrote:
> >>>On Fri, Sep 16, 2011 at 12:46:40PM -0400, Stefan Berger wrote:
> >>
> >>The checksuming I think makes sense if encryption is being added so
> >>decryption and testing for proper key material remains an NVRAM
> >>operation rather than a device operation.
> >Not sure how this addresses the question of what to do
> >on checksum failure.
> >
> Checksum failure on an unencrypted blob would mean that the blob is
> corrupted. In case of encryption 'corrupted' would overlap with a
> 'badly decrypted' blob. In either way the startup of the device
> cannot happen.

With corruption - why not? A specific block being corrupted does not mean all
data is lost.

> We could refuse the NVRAM key suggesting that likely
> this is the wrong key for decryption but corruption is also
> possible.

I'm guessing that if we find a correct ber structure in the file, this
most likely means the key is correct.

-- 
MST

Re: [Qemu-devel] blobstore disk format (was Re: Design of the blobstore)

[Stefan Berger]

On 09/22/2011 02:37 AM, Michael S. Tsirkin wrote:
> On Wed, Sep 21, 2011 at 09:44:37PM -0400, Stefan Berger wrote:
>> On 09/19/2011 03:04 PM, Michael S. Tsirkin wrote:
>>> On Mon, Sep 19, 2011 at 12:22:02PM -0400, Stefan Berger wrote:
>>>> On 09/17/2011 03:28 PM, Michael S. Tsirkin wrote:
>>>>> On Fri, Sep 16, 2011 at 12:46:40PM -0400, Stefan Berger wrote:
>>>> The checksuming I think makes sense if encryption is being added so
>>>> decryption and testing for proper key material remains an NVRAM
>>>> operation rather than a device operation.
>>> Not sure how this addresses the question of what to do
>>> on checksum failure.
>>>
>> Checksum failure on an unencrypted blob would mean that the blob is
>> corrupted. In case of encryption 'corrupted' would overlap with a
>> 'badly decrypted' blob. In either way the startup of the device
>> cannot happen.
> With corruption - why not? A specific block being corrupted does not mean all
> data is lost.
>
Presumably if you feed bad data into a device it either has its own way 
of integrity checking the blob (we actually do this for the TPM) or it 
will blow up/show wrong behavior at some point - hopefully sooner rather 
than later. Though the detection of bad data *can* be an NVRAM operation 
rather than the operation of a device using the data stored in the NVRAM.
>> We could refuse the NVRAM key suggesting that likely
>> this is the wrong key for decryption but corruption is also
>> possible.
> I'm guessing that if we find a correct ber structure in the file, this
> most likely means the key is correct.
[I still would add at least a CRC32 (or maybe even a SHA1) for detection 
of corruption of the ASN.1 encoded blob without having to hunt the data 
through a ASN.1 decoder.]

If we now say that data should be encryptable even if QCoW2 wasn't used, 
then is a command line option

-nvram id=<driveid>,key=<hex key>,...

something we should support to make the key applicable to the whole NVRAM?

      Stefan

Re: [Qemu-devel] blobstore disk format (was Re: Design of the blobstore)

[Daniel P. Berrange]

On Wed, Sep 28, 2011 at 11:48:19AM -0400, Stefan Berger wrote:
> On 09/22/2011 02:37 AM, Michael S. Tsirkin wrote:
> >On Wed, Sep 21, 2011 at 09:44:37PM -0400, Stefan Berger wrote:
> >>On 09/19/2011 03:04 PM, Michael S. Tsirkin wrote:
> >>>On Mon, Sep 19, 2011 at 12:22:02PM -0400, Stefan Berger wrote:
> >>>>On 09/17/2011 03:28 PM, Michael S. Tsirkin wrote:
> >>>>>On Fri, Sep 16, 2011 at 12:46:40PM -0400, Stefan Berger wrote:
> >>>>The checksuming I think makes sense if encryption is being added so
> >>>>decryption and testing for proper key material remains an NVRAM
> >>>>operation rather than a device operation.
> >>>Not sure how this addresses the question of what to do
> >>>on checksum failure.
> >>>
> >>Checksum failure on an unencrypted blob would mean that the blob is
> >>corrupted. In case of encryption 'corrupted' would overlap with a
> >>'badly decrypted' blob. In either way the startup of the device
> >>cannot happen.
> >With corruption - why not? A specific block being corrupted does not mean all
> >data is lost.
> >
> Presumably if you feed bad data into a device it either has its own
> way of integrity checking the blob (we actually do this for the TPM)
> or it will blow up/show wrong behavior at some point - hopefully
> sooner rather than later. Though the detection of bad data *can* be
> an NVRAM operation rather than the operation of a device using the
> data stored in the NVRAM.
> >>We could refuse the NVRAM key suggesting that likely
> >>this is the wrong key for decryption but corruption is also
> >>possible.
> >I'm guessing that if we find a correct ber structure in the file, this
> >most likely means the key is correct.
> [I still would add at least a CRC32 (or maybe even a SHA1) for
> detection of corruption of the ASN.1 encoded blob without having to
> hunt the data through a ASN.1 decoder.]
> 
> If we now say that data should be encryptable even if QCoW2 wasn't
> used, then is a command line option
> 
> -nvram id=<driveid>,key=<hex key>,...
> 
> something we should support to make the key applicable to the whole NVRAM?

Try to avoid requiring secret keys to be set on the command line...
At least allow setting them via a monitor command

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] blobstore disk format (was Re: Design of the blobstore)

[Stefan Berger]

On 09/28/2011 11:50 AM, Daniel P. Berrange wrote:
> On Wed, Sep 28, 2011 at 11:48:19AM -0400, Stefan Berger wrote:
>> On 09/22/2011 02:37 AM, Michael S. Tsirkin wrote:
>>>
>>> I'm guessing that if we find a correct ber structure in the file, this
>>> most likely means the key is correct.
>> [I still would add at least a CRC32 (or maybe even a SHA1) for
>> detection of corruption of the ASN.1 encoded blob without having to
>> hunt the data through a ASN.1 decoder.]
>>
>> If we now say that data should be encryptable even if QCoW2 wasn't
>> used, then is a command line option
>>
>> -nvram id=<driveid>,key=<hex key>,...
>>
>> something we should support to make the key applicable to the whole NVRAM?
> Try to avoid requiring secret keys to be set on the command line...
> At least allow setting them via a monitor command
>
Hm, this brings me back to the previous problem of the ordering of 
things that ended up being problematic:

In the case of encrypted QCoW2 the monitor queries for the password when 
the use types 'c' for continue. That happens *after* device's 'init' 
function was called and also *after* their 'reset' handler was invoked, 
so not being able to decrypt encrypted state blobs and not being able to 
feed devices with their persistent state even until the 'reset' handler 
was invoked. The password comes quite late.

The monitor reacts to key typing, which in turn is handled in the 
main_loop() in vl.c.

So, the solution could be that each NVRAM client also registers its 
reset handler (along with the DeviceState pointer). Each NVRAM client 
would have to be written in such a way that it ignores previous failed 
attempts to read its state due to the key coming so late and once the 
NVRAM has the key it invokes the reset handlers again, which now trigger 
the reading of the state in the NVRAM that now can be decrypted.  Does 
this sound 'sane' or more like a 'hack' ?

    Stefan

> Daniel

Re: [Qemu-devel] blobstore disk format (was Re: Design of the blobstore)

[Michael S. Tsirkin]

On Wed, Sep 28, 2011 at 11:48:19AM -0400, Stefan Berger wrote:
> On 09/22/2011 02:37 AM, Michael S. Tsirkin wrote:
> >On Wed, Sep 21, 2011 at 09:44:37PM -0400, Stefan Berger wrote:
> >>On 09/19/2011 03:04 PM, Michael S. Tsirkin wrote:
> >>>On Mon, Sep 19, 2011 at 12:22:02PM -0400, Stefan Berger wrote:
> >>>>On 09/17/2011 03:28 PM, Michael S. Tsirkin wrote:
> >>>>>On Fri, Sep 16, 2011 at 12:46:40PM -0400, Stefan Berger wrote:
> >>>>The checksuming I think makes sense if encryption is being added so
> >>>>decryption and testing for proper key material remains an NVRAM
> >>>>operation rather than a device operation.
> >>>Not sure how this addresses the question of what to do
> >>>on checksum failure.
> >>>
> >>Checksum failure on an unencrypted blob would mean that the blob is
> >>corrupted. In case of encryption 'corrupted' would overlap with a
> >>'badly decrypted' blob. In either way the startup of the device
> >>cannot happen.
> >With corruption - why not? A specific block being corrupted does not mean all
> >data is lost.
> >
> Presumably if you feed bad data into a device it either has its own
> way of integrity checking the blob (we actually do this for the TPM)
> or it will blow up/show wrong behavior at some point - hopefully
> sooner rather than later. Though the detection of bad data *can* be
> an NVRAM operation rather than the operation of a device using the
> data stored in the NVRAM.
> >>We could refuse the NVRAM key suggesting that likely
> >>this is the wrong key for decryption but corruption is also
> >>possible.
> >I'm guessing that if we find a correct ber structure in the file, this
> >most likely means the key is correct.
> [I still would add at least a CRC32 (or maybe even a SHA1) for
> detection of corruption of the ASN.1 encoded blob without having to
> hunt the data through a ASN.1 decoder.]
> 
> If we now say that data should be encryptable even if QCoW2 wasn't
> used, then is a command line option
> 
> -nvram id=<driveid>,key=<hex key>,...
> 
> something we should support to make the key applicable to the whole NVRAM?
> 
>      Stefan
> 

Why? Is there a reason not to use qcow2 if one wants encryption?

One thing I don't understand is why crc is mixed with
encryption in this discussion. If what we want is integrity checks on the
data, it's an interesting feature that seems somewhat
unrelated to TPM.

If we want to use a hash for cryptography, such as verifying the key,
crc is probably not a good match, and one can imagine that it might in fact
weaken the security of the system.

-- 
MST