[Qemu-devel] GSoC: Improved image format compatibility

[Qemu-devel] GSoC: Improved image format compatibility

[Lyu Mitnick]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

Hello all,

I have used QEMU to assist me developing embedded system for 3 years. And I
want to contribute to QEMU and participate Google Summer of Code this year.
I have port binutils so I have some experienced with binary format. I
noticed that there is no VHD support in qemu-img. A Virtual Hard Disk(VHD)
is a virtual hard disk file format used in Microsoft virtual PC, Hyper-V,
virtual box, xVM and Complete PC Backup  of vista and windows 7. I want to
add support for VHD in GSoC. I am wondering whether this kind of idea is
valuable to QEMU?? Or any another topic related image format is valuable to
QEMU??

thanks

Mitnick

[-- Attachment #2: Type: text/html, Size: 1256 bytes --]

Re: [Qemu-devel] GSoC: Improved image format compatibility

[Stefan Weil]

Am 29.03.2011 23:55, schrieb Lyu Mitnick:
> Hello all,
>
> I have used QEMU to assist me developing embedded system for 3 years. And I
> want to contribute to QEMU and participate Google Summer of Code this year.
> I have port binutils so I have some experienced with binary format. I
> noticed that there is no VHD support in qemu-img. A Virtual Hard Disk(VHD)
> is a virtual hard disk file format used in Microsoft virtual PC, Hyper-V,
> virtual box, xVM and Complete PC Backup  of vista and windows 7. I want to
> add support for VHD in GSoC. I am wondering whether this kind of idea is
> valuable to QEMU?? Or any another topic related image format is valuable to
> QEMU??
>
> thanks
>
> Mitnick

Hello Mitnick,

QEMU (including qemu-img) already supports the format vpc.
The code in block/vpc.c also uses the name vhd, so I think
vpc == vhd.

Regards
Stefan

Re: [Qemu-devel] GSoC: Improved image format compatibility

[Stefan Hajnoczi]

On Wed, Mar 30, 2011 at 5:50 AM, Stefan Weil <weil@mail.berlios.de> wrote:
> Am 29.03.2011 23:55, schrieb Lyu Mitnick:
>>
>> Hello all,
>>
>> I have used QEMU to assist me developing embedded system for 3 years. And
>> I
>> want to contribute to QEMU and participate Google Summer of Code this
>> year.
>> I have port binutils so I have some experienced with binary format. I
>> noticed that there is no VHD support in qemu-img. A Virtual Hard Disk(VHD)
>> is a virtual hard disk file format used in Microsoft virtual PC, Hyper-V,
>> virtual box, xVM and Complete PC Backup  of vista and windows 7. I want to
>> add support for VHD in GSoC. I am wondering whether this kind of idea is
>> valuable to QEMU?? Or any another topic related image format is valuable
>> to
>> QEMU??
>>
>> thanks
>>
>> Mitnick
>
> Hello Mitnick,
>
> QEMU (including qemu-img) already supports the format vpc.
> The code in block/vpc.c also uses the name vhd, so I think
> vpc == vhd.

However, it is possible that vpc/vhd has new features that are not yet
supported in QEMU.  There is a listed project idea for updating image
formats on the GSoC wiki page:

http://wiki.qemu.org/Google_Summer_of_Code_2011#Improved_image_format_compatibility

You could assess QEMU's implementation in block/vpc.c and compare
against the latest public file format specification:

http://en.wikipedia.org/wiki/VHD_(file_format)

If you find there are new features or format changes that are not yet
supported in QEMU, this might make a good "Improved image format
compatibility" GSoC project.

Stefan

Re: [Qemu-devel] GSoC: Improved image format compatibility

[Lyu Mitnick]

[-- Attachment #1: Type: text/plain, Size: 2310 bytes --]

Hello Stefan,

I have spent a whole night to trace the code of block/vpc.c and compare
against the VHD specifications
(at
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=C2D03242-2FFB-48EF-A211-F0C44741109E&displaylang=en
).
It seems that there isn't Fixed hard disk image and Differencing hard disk
image support now. I am wondering whether what I has
learned from tracing code is true or false? I also found some possibility of
optimization in block/vpc.c. Should I contact mentor
assigned to this project in private or post the whole idea on the mailing
list??

thanks very much

Mitnick

2011/3/30 Stefan Hajnoczi <stefanha@gmail.com>

> On Wed, Mar 30, 2011 at 5:50 AM, Stefan Weil <weil@mail.berlios.de> wrote:
> > Am 29.03.2011 23:55, schrieb Lyu Mitnick:
> >>
> >> Hello all,
> >>
> >> I have used QEMU to assist me developing embedded system for 3 years.
> And
> >> I
> >> want to contribute to QEMU and participate Google Summer of Code this
> >> year.
> >> I have port binutils so I have some experienced with binary format. I
> >> noticed that there is no VHD support in qemu-img. A Virtual Hard
> Disk(VHD)
> >> is a virtual hard disk file format used in Microsoft virtual PC,
> Hyper-V,
> >> virtual box, xVM and Complete PC Backup  of vista and windows 7. I want
> to
> >> add support for VHD in GSoC. I am wondering whether this kind of idea is
> >> valuable to QEMU?? Or any another topic related image format is valuable
> >> to
> >> QEMU??
> >>
> >> thanks
> >>
> >> Mitnick
> >
> > Hello Mitnick,
> >
> > QEMU (including qemu-img) already supports the format vpc.
> > The code in block/vpc.c also uses the name vhd, so I think
> > vpc == vhd.
>
> However, it is possible that vpc/vhd has new features that are not yet
> supported in QEMU.  There is a listed project idea for updating image
> formats on the GSoC wiki page:
>
>
> http://wiki.qemu.org/Google_Summer_of_Code_2011#Improved_image_format_compatibility
>
> You could assess QEMU's implementation in block/vpc.c and compare
> against the latest public file format specification:
>
> http://en.wikipedia.org/wiki/VHD_(file_format)
>
> If you find there are new features or format changes that are not yet
> supported in QEMU, this might make a good "Improved image format
> compatibility" GSoC project.
>
> Stefan
>

[-- Attachment #2: Type: text/html, Size: 3389 bytes --]

Re: [Qemu-devel] GSoC: Improved image format compatibility

[Stefan Weil]

Am 30.03.2011 20:40, schrieb Lyu Mitnick:
> Hello Stefan,
>
> I have spent a whole night to trace the code of block/vpc.c and 
> compare against the VHD specifications
> (at 
> http://www.microsoft.com/downloads/en/details.aspx?FamilyID=C2D03242-2FFB-48EF-A211-F0C44741109E&displaylang=en 
> <http://www.microsoft.com/downloads/en/details.aspx?FamilyID=C2D03242-2FFB-48EF-A211-F0C44741109E&amp;displaylang=en>).
> It seems that there isn't Fixed hard disk image and Differencing hard 
> disk image support now. I am wondering whether what I has
> learned from tracing code is true or false? I also found some 
> possibility of optimization in block/vpc.c. Should I contact mentor
> assigned to this project in private or post the whole idea on the 
> mailing list??
>
> thanks very much
>
> Mitnick
>


Hello Mitnick,

asynchronous i/o is also missing in block/vpc.c.

All newer block drivers support asynchronous reads and writes,
only some old drivers don't.

So adding asynchronous i/o to at least some of the old drivers
would improve their usability.

Look for "aio" in block/*.c to get a starting point.

Regards
Stefan (W.)

Re: [Qemu-devel] GSoC: Improved image format compatibility

[Lyu Mitnick]

[-- Attachment #1: Type: text/plain, Size: 1964 bytes --]

Hello Stefan,

Let me summarize the ideas of "Improved image format compatibility"  now:

(1) add support of "Fixed hard disk image" into block/vpc.c
(2) add support of "Differencing hard disk image" into block/vpc.c
(3) add asynchronous i/o into block/vpc,c
(4) have some optimization of block/vpc.c

Would you mind to tell me the project containing four topics above is
suitable for GSoC of size and skill level??
And would anyone is interested in this project and being my mentor??

thanks very much

2011/3/31 Stefan Weil <weil@mail.berlios.de>

> Am 30.03.2011 20:40, schrieb Lyu Mitnick:
>
>> Hello Stefan,
>>
>> I have spent a whole night to trace the code of block/vpc.c and compare
>> against the VHD specifications
>> (at
>> http://www.microsoft.com/downloads/en/details.aspx?FamilyID=C2D03242-2FFB-48EF-A211-F0C44741109E&amp;displaylang=en<http://www.microsoft.com/downloads/en/details.aspx?FamilyID=C2D03242-2FFB-48EF-A211-F0C44741109E&displaylang=en><
>> http://www.microsoft.com/downloads/en/details.aspx?FamilyID=C2D03242-2FFB-48EF-A211-F0C44741109E&amp;displaylang=en<http://www.microsoft.com/downloads/en/details.aspx?FamilyID=C2D03242-2FFB-48EF-A211-F0C44741109E&displaylang=en>
>> >).
>>
>> It seems that there isn't Fixed hard disk image and Differencing hard disk
>> image support now. I am wondering whether what I has
>> learned from tracing code is true or false? I also found some possibility
>> of optimization in block/vpc.c. Should I contact mentor
>> assigned to this project in private or post the whole idea on the mailing
>> list??
>>
>> thanks very much
>>
>> Mitnick
>>
>>
>
> Hello Mitnick,
>
> asynchronous i/o is also missing in block/vpc.c.
>
> All newer block drivers support asynchronous reads and writes,
> only some old drivers don't.
>
> So adding asynchronous i/o to at least some of the old drivers
> would improve their usability.
>
> Look for "aio" in block/*.c to get a starting point.
>
> Regards
> Stefan (W.)
>
>

[-- Attachment #2: Type: text/html, Size: 4226 bytes --]

Re: [Qemu-devel] GSoC: Improved image format compatibility

[Lyu Mitnick]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

Hello Stefan

Hello Mitnick,
>
> asynchronous i/o is also missing in block/vpc.c.
>
> All newer block drivers support asynchronous reads and writes,
> only some old drivers don't.
>
> So adding asynchronous i/o to at least some of the old drivers
> would improve their usability.
>
> Look for "aio" in block/*.c to get a starting point.
>
> Regards
> Stefan (W.)
>
>
I look for "aio" in block/*.c as a starting point of asynchronous i/o.
And I
found that there isn't any "aio" keywords in block/vmdk.c. block/vmdk.c
is used as block driver for the VMDK format. Is it means that asynchronous
i/o is also missing in VMDK format?

thanks

Mitnick

[-- Attachment #2: Type: text/html, Size: 1001 bytes --]

Re: [Qemu-devel] GSoC: Improved image format compatibility

[Kevin Wolf]

Am 01.04.2011 17:15, schrieb Lyu Mitnick:
> Hello Stefan
> 
>     Hello Mitnick,
> 
>     asynchronous i/o is also missing in block/vpc.c.
> 
>     All newer block drivers support asynchronous reads and writes,
>     only some old drivers don't.
> 
>     So adding asynchronous i/o to at least some of the old drivers
>     would improve their usability.
> 
>     Look for "aio" in block/*.c to get a starting point.
> 
>     Regards
>     Stefan (W.)
> 
> 
> I look for "aio" in block/*.c as a starting point of asynchronous i/o.
> And I 
> found that there isn't any "aio" keywords in block/vmdk.c. block/vmdk.c
> is used as block driver for the VMDK format. Is it means that asynchronous 
> i/o is also missing in VMDK format?

Yes, the VMDK implementation is completely synchronous.

Kevin

Re: [Qemu-devel] GSoC: Improved image format compatibility

[Stefan Weil]

[-- Attachment #1: Type: text/plain, Size: 796 bytes --]

Am 01.04.2011 17:15, schrieb Lyu Mitnick:
> Hello Stefan
>
>     Hello Mitnick,
>
>     asynchronous i/o is also missing in block/vpc.c.
>
>     All newer block drivers support asynchronous reads and writes,
>     only some old drivers don't.
>
>     So adding asynchronous i/o to at least some of the old drivers
>     would improve their usability.
>
>     Look for "aio" in block/*.c to get a starting point.
>
>     Regards
>     Stefan (W.)
>
>
> I look for "aio" in block/*.c as a starting point of asynchronous i/o. 
> And I
> found that there isn't any "aio" keywords in block/vmdk.c. block/vmdk.c
> is used as block driver for the VMDK format. Is it means 
> that asynchronous
> i/o is also missing in VMDK format?
>
> thanks
>
> Mitnick
>

Yes, it's also missing for VMDK.

Stefan (W.)

[-- Attachment #2: Type: text/html, Size: 1474 bytes --]

Re: [Qemu-devel] GSoC: Improved image format compatibility

[Lyu Mitnick]

[-- Attachment #1: Type: text/plain, Size: 1179 bytes --]

Hello

I have some question about asynchronous i/o in QEMU block driver: Why a file
format with asynchronous i/o support(ex. qcow) doesn't need to register
bdrv_read/bdrv_write which is registered in vhd file format?? Would qcow
block driver
also support synchronous i/o??

thanks a lot

Mitnick

2011/4/2 Kevin Wolf <kwolf@redhat.com>

> Am 01.04.2011 17:15, schrieb Lyu Mitnick:
> > Hello Stefan
> >
> >     Hello Mitnick,
> >
> >     asynchronous i/o is also missing in block/vpc.c.
> >
> >     All newer block drivers support asynchronous reads and writes,
> >     only some old drivers don't.
> >
> >     So adding asynchronous i/o to at least some of the old drivers
> >     would improve their usability.
> >
> >     Look for "aio" in block/*.c to get a starting point.
> >
> >     Regards
> >     Stefan (W.)
> >
> >
> > I look for "aio" in block/*.c as a starting point of asynchronous i/o.
> > And I
> > found that there isn't any "aio" keywords in block/vmdk.c. block/vmdk.c
> > is used as block driver for the VMDK format. Is it means that
> asynchronous
> > i/o is also missing in VMDK format?
>
> Yes, the VMDK implementation is completely synchronous.
>
> Kevin
>

[-- Attachment #2: Type: text/html, Size: 1808 bytes --]

Re: [Qemu-devel] GSoC: Improved image format compatibility

[Stefan Hajnoczi]

On Fri, Apr 1, 2011 at 7:29 PM, Lyu Mitnick <mitnick.lyu@gmail.com> wrote:
> Hello
> I have some question about asynchronous i/o in QEMU block driver: Why a file
> format with asynchronous i/o support(ex. qcow) doesn't need to register
> bdrv_read/bdrv_write which is registered in vhd file format?? Would qcow
> block driver
> also support synchronous i/o??

Take a look at block.c.  There are synchronous I/O emulation (and
asynchronous I/O emulation) functions in case the block driver
provides only asynchronous (or synchronous, respectively) functions.

One thing to mention about synchronous and asynchronous interfaces is
that block drivers which are synchronous really don't work well for
running VMs.  They block execution until I/O completes (which can take
a long time) and therefore steal time from the guest (i.e. laggy, poor
responsiveness from the guest).  Therefore synchronous block drivers
are useful for qemu-img convert but not running VMs.

Stefan

Re: [Qemu-devel] GSoC: Improved image format compatibility

[Lyu Mitnick]

[-- Attachment #1: Type: text/plain, Size: 1529 bytes --]

Hello Stefan,

I have take a look at block.c. But I am a little confused about the meaning
of synchronous/asynchronous i/o. I know the two concept in a operating
system.  However I am not sure whether it is analogous in virtual machine.

2011/4/2 Stefan Hajnoczi <stefanha@gmail.com>

> On Fri, Apr 1, 2011 at 7:29 PM, Lyu Mitnick <mitnick.lyu@gmail.com> wrote:
> > Hello
> > I have some question about asynchronous i/o in QEMU block driver: Why a
> file
> > format with asynchronous i/o support(ex. qcow) doesn't need to register
> > bdrv_read/bdrv_write which is registered in vhd file format?? Would qcow
> > block driver
> > also support synchronous i/o??
>
> Take a look at block.c.  There are synchronous I/O emulation (and
> asynchronous I/O emulation) functions in case the block driver
> provides only asynchronous (or synchronous, respectively) functions.
>
> One thing to mention about synchronous and asynchronous interfaces is
> that block drivers which are synchronous really don't work well for
> running VMs.  They block execution until I/O completes (which can take
> a long time) and therefore steal time from the guest (i.e. laggy, poor
>

I am wondering the meaning of execution block. It is for host host
operating system or for guest operating system?? (ie, Could I issue an
asynchronous i/o in guest operating system when the vhd file system image
mounted)


> responsiveness from the guest).  Therefore synchronous block drivers
> are useful for qemu-img convert but not running VMs.
>
> Stefan


Mitnick

[-- Attachment #2: Type: text/html, Size: 2219 bytes --]

Re: [Qemu-devel] GSoC: Improved image format compatibility

[Stefan Hajnoczi]

On Sat, Apr 2, 2011 at 8:03 PM, Lyu Mitnick <mitnick.lyu@gmail.com> wrote:
> Hello Stefan,
> I have take a look at block.c. But I am a little confused about the meaning
> of synchronous/asynchronous i/o. I know the two concept in a operating
> system.  However I am not sure whether it is analogous in virtual machine.

You may want to read up on the QEMU threading model:
http://blog.vmsplice.net/2011/03/qemu-internals-overall-architecture-and.html

(Over)simplifying a bit, only one host thread can be executing QEMU
code at any given time because QEMU is not thread-safe internally.  If
the VM issues a disk read and QEMU uses a read(2) system call to
service it, then QEMU and the VM is blocked until the read(2) system
call completes.  Therefore it is better to use asynchronous I/O in
QEMU and resume the VM.  Once the read operation completes a callback
in QEMU is invoked and we can signal completion back to the VM.

Also, in terms of the storage controller interfaces that QEMU emulates
(IDE, SCSI, virtio-blk) disk operations are queued or handed to the
hardware.  The guest operating system expects to continue executing
while the disk operation is in progress.  On completion the storage
controller raises an interrupt and the operating system will complete
the I/O.  Again, this is an asynchronous (request/response) model
rather than a synchronous operation from the guest operating system
perspective.

Look at hw/virtio-blk.c:virtio_blk_handle_output() for an example of
how disk emulation works in QEMU.

For qemu-img it is more natural to use synchronous I/O because it is
just a sequential C program, not an event-driven program like QEMU.

Stefan

Re: [Qemu-devel] GSoC: Improved image format compatibility

[Lyu Mitnick]

[-- Attachment #1: Type: text/plain, Size: 2079 bytes --]

Hello Stefan,

2011/4/3 Stefan Hajnoczi <stefanha@gmail.com>

> On Sat, Apr 2, 2011 at 8:03 PM, Lyu Mitnick <mitnick.lyu@gmail.com> wrote:
> > Hello Stefan,
> > I have take a look at block.c. But I am a little confused about the
> meaning
> > of synchronous/asynchronous i/o. I know the two concept in a operating
> > system.  However I am not sure whether it is analogous in virtual
> machine.
>
> You may want to read up on the QEMU threading model:
>
> http://blog.vmsplice.net/2011/03/qemu-internals-overall-architecture-and.html
>
> (Over)simplifying a bit, only one host thread can be executing QEMU
> code at any given time because QEMU is not thread-safe internally.  If
> the VM issues a disk read and QEMU uses a read(2) system call to
> service it, then QEMU and the VM is blocked until the read(2) system
> call completes.  Therefore it is better to use asynchronous I/O in
> QEMU and resume the VM.  Once the read operation completes a callback
> in QEMU is invoked and we can signal completion back to the VM.
>
> Also, in terms of the storage controller interfaces that QEMU emulates
> (IDE, SCSI, virtio-blk) disk operations are queued or handed to the
> hardware.  The guest operating system expects to continue executing
> while the disk operation is in progress.  On completion the storage
> controller raises an interrupt and the operating system will complete
> the I/O.  Again, this is an asynchronous (request/response) model
> rather than a synchronous operation from the guest operating system
> perspective.
>
> Look at hw/virtio-blk.c:virtio_blk_handle_output() for an example of
> how disk emulation works in QEMU.
>
> For qemu-img it is more natural to use synchronous I/O because it is
> just a sequential C program, not an event-driven program like QEMU.
>
> Stefan
>

I am writing my proposal now. I met a problem about describing "Adding
Fixed hard disk image and Differencing hard disk image support
into VHD format. This is a format with three sub-format . Is
exist another format containing sub-format and I can take a
look??

Thanks

Mitnick

[-- Attachment #2: Type: text/html, Size: 3730 bytes --]