[Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Wenchao Xia]

Hi, Paolo and folks,
  qemu have good capabilities to access different virtual disks, I want
to expose its block layer API to let 3rd party program linked in, such
as management stack or block tools, to access images data directly.

Following is the objects:
  (1) API to write/read block device at offset.
  (2) Determine the image type,qcow2/qed/raw
  (3) Determine which blocks are allocated.
  (4) Determine backing file.

Following is my implementing plan draft:
  1 introduce libqblock.so in sub directory in qemu.
  2 write a nbd client in libqblock, similar to qemu nbd client. Then
use it to talk with nbd server, by default is qemu-nbd, to get access
to images. In this way, libqblock.so could be friendly LGPL licensed.
  3 still not got a good way to get additional info in (2)(3)(4),
currently in my head is patch qemu-nbd to add an additional nbd command,
"image-info", in which returns related info.

  What do you think about it?

-- 
Best Regards

Wenchao Xia
mail:xiawenc@linux.vnet.ibm.com
tel:86-010-82450803

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Paolo Bonzini]

Il 09/07/2012 10:54, Wenchao Xia ha scritto:
> Following is my implementing plan draft:
>   1 introduce libqblock.so in sub directory in qemu.
>   2 write a nbd client in libqblock, similar to qemu nbd client. Then
> use it to talk with nbd server, by default is qemu-nbd, to get access
> to images. In this way, libqblock.so could be friendly LGPL licensed.

Did you actually assess the license situation of the block layer?
block.c and large parts of block/* are under a BSD license, for example.
 If the library only has to support raw files, it might do so using
synchronous I/O only.  This would remove a large body of GPL-licensed code.

>   3 still not got a good way to get additional info in (2)(3)(4),
> currently in my head is patch qemu-nbd to add an additional nbd command,
> "image-info", in which returns related info.

On the Linux kernel mailing list I would have no qualms labeling such
command as "crap".  However, since the social standards on qemu-devel
are a bit higher, I'll ask instead: what information would the command
provide beyond the size?

Paolo

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Daniel P. Berrange]

On Mon, Jul 09, 2012 at 04:54:08PM +0800, Wenchao Xia wrote:
> Hi, Paolo and folks,
>   qemu have good capabilities to access different virtual disks, I want
> to expose its block layer API to let 3rd party program linked in, such
> as management stack or block tools, to access images data directly.
> 
> Following is the objects:
>   (1) API to write/read block device at offset.
>   (2) Determine the image type,qcow2/qed/raw
>   (3) Determine which blocks are allocated.
>   (4) Determine backing file.
> 
> Following is my implementing plan draft:
>   1 introduce libqblock.so in sub directory in qemu.
>   2 write a nbd client in libqblock, similar to qemu nbd client. Then
> use it to talk with nbd server, by default is qemu-nbd, to get access
> to images. In this way, libqblock.so could be friendly LGPL licensed.
>   3 still not got a good way to get additional info in (2)(3)(4),
> currently in my head is patch qemu-nbd to add an additional nbd command,
> "image-info", in which returns related info.
> 
>   What do you think about it?

For arbirary read/write access to disk images, I can see a little value
in having a standalone libnbd client API, that is able to just talk to
any NBD server. Arguably such a thing does not need to be part of the
QEMU source tree - eg see the recently written libiscsi.so client.

For getting the other metadata about the disk image you mention, another
possibility to is just make 'qemu-img info' return the data in a machine
parseable format, ie JSON & make a client API for extracting data from
this JSON document.

For a full-blown RPC API for doing arbitrary tasks related to block
devices, then many apps will tend towards libguestfs, since it provides
such a wide range of functionality for manipulating disk images.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Christoph Hellwig]

On Mon, Jul 09, 2012 at 04:54:08PM +0800, Wenchao Xia wrote:
> Hi, Paolo and folks,
>   qemu have good capabilities to access different virtual disks, I want
> to expose its block layer API to let 3rd party program linked in, such
> as management stack or block tools, to access images data directly.
> 
> Following is the objects:
>   (1) API to write/read block device at offset.
>   (2) Determine the image type,qcow2/qed/raw
>   (3) Determine which blocks are allocated.
>   (4) Determine backing file.

Sounds like you want a procedural interface for that.  At least for (1)
I have patches I'll submit soon to add qemu img read/write commands.

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Wenchao Xia]

于 2012-7-9 17:13, Paolo Bonzini 写道:
> Il 09/07/2012 10:54, Wenchao Xia ha scritto:
>> Following is my implementing plan draft:
>>    1 introduce libqblock.so in sub directory in qemu.
>>    2 write a nbd client in libqblock, similar to qemu nbd client. Then
>> use it to talk with nbd server, by default is qemu-nbd, to get access
>> to images. In this way, libqblock.so could be friendly LGPL licensed.
> 
> Did you actually assess the license situation of the block layer?
> block.c and large parts of block/* are under a BSD license, for example.
>   If the library only has to support raw files, it might do so using
> synchronous I/O only.  This would remove a large body of GPL-licensed code.
> 
  If the library was built as nbd-client communicating with nbd-server,
which then employ the BSO licensed code, could the library ignore the
server side's license problem? The reason using nbd-client approach are:
work around qemu block layer license issue and easy to implement, if
other tool found this labrary useful then considering about directly
employ the qemu block code.


>>    3 still not got a good way to get additional info in (2)(3)(4),
>> currently in my head is patch qemu-nbd to add an additional nbd command,
>> "image-info", in which returns related info.
> 
> On the Linux kernel mailing list I would have no qualms labeling such
> command as "crap".  However, since the social standards on qemu-devel
> are a bit higher, I'll ask instead: what information would the command
> provide beyond the size?
> 
  The API need to report the image format it is using, such as
"qcow2". And also API should report if a block at offset have been
allocated or it is a hole.

> Paolo
> 


-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Wenchao Xia]

于 2012-7-9 17:27, Daniel P. Berrange 写道:
> On Mon, Jul 09, 2012 at 04:54:08PM +0800, Wenchao Xia wrote:
>> Hi, Paolo and folks,
>>    qemu have good capabilities to access different virtual disks, I want
>> to expose its block layer API to let 3rd party program linked in, such
>> as management stack or block tools, to access images data directly.
>>
>> Following is the objects:
>>    (1) API to write/read block device at offset.
>>    (2) Determine the image type,qcow2/qed/raw
>>    (3) Determine which blocks are allocated.
>>    (4) Determine backing file.
>>
>> Following is my implementing plan draft:
>>    1 introduce libqblock.so in sub directory in qemu.
>>    2 write a nbd client in libqblock, similar to qemu nbd client. Then
>> use it to talk with nbd server, by default is qemu-nbd, to get access
>> to images. In this way, libqblock.so could be friendly LGPL licensed.
>>    3 still not got a good way to get additional info in (2)(3)(4),
>> currently in my head is patch qemu-nbd to add an additional nbd command,
>> "image-info", in which returns related info.
>>
>>    What do you think about it?
>
> For arbirary read/write access to disk images, I can see a little value
> in having a standalone libnbd client API, that is able to just talk to
> any NBD server. Arguably such a thing does not need to be part of the
> QEMU source tree - eg see the recently written libiscsi.so client.
>
> For getting the other metadata about the disk image you mention, another
> possibility to is just make 'qemu-img info' return the data in a machine
> parseable format, ie JSON&  make a client API for extracting data from
> this JSON document.
>
   Thank u for the idea. The .so is introduced to let program access the
image more directly, parsing string is not so fast and it depends on
another program's stdout output, I hope to get a faster way.

> For a full-blown RPC API for doing arbitrary tasks related to block
> devices, then many apps will tend towards libguestfs, since it provides
> such a wide range of functionality for manipulating disk images.
>
   I used libguestfs to make my image too, but the target of the .so is
a bit different: it expose block data in a lower level, expose
everything qemu main code(except block code) can see. Potential purpose
is as first step to make qemu block layer independent, then qemu and
other tool would be just employers of the library. But now it is out of
plan because license issue, and the library acts as client now lowering
the performance.


> Regards,
> Daniel


-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Wenchao Xia]

于 2012-7-9 22:36, Christoph Hellwig 写道:
> On Mon, Jul 09, 2012 at 04:54:08PM +0800, Wenchao Xia wrote:
>> Hi, Paolo and folks,
>>    qemu have good capabilities to access different virtual disks, I want
>> to expose its block layer API to let 3rd party program linked in, such
>> as management stack or block tools, to access images data directly.
>>
>> Following is the objects:
>>    (1) API to write/read block device at offset.
>>    (2) Determine the image type,qcow2/qed/raw
>>    (3) Determine which blocks are allocated.
>>    (4) Determine backing file.
>
> Sounds like you want a procedural interface for that.  At least for (1)
> I have patches I'll submit soon to add qemu img read/write commands.
   Yes, the purpose is introduce API interface about block data, the
operation was supposed to happen frequently, a linked-in library may be
better than process output string parsing in performance.

>
>


-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Paolo Bonzini]

Il 10/07/2012 07:04, Wenchao Xia ha scritto:
> 于 2012-7-9 17:13, Paolo Bonzini 写道:
>> Il 09/07/2012 10:54, Wenchao Xia ha scritto:
>>> Following is my implementing plan draft:
>>>    1 introduce libqblock.so in sub directory in qemu.
>>>    2 write a nbd client in libqblock, similar to qemu nbd client. Then
>>> use it to talk with nbd server, by default is qemu-nbd, to get access
>>> to images. In this way, libqblock.so could be friendly LGPL licensed.
>>
>> Did you actually assess the license situation of the block layer?
>> block.c and large parts of block/* are under a BSD license, for example.
>>   If the library only has to support raw files, it might do so using
>> synchronous I/O only.  This would remove a large body of GPL-licensed code.
>>
>   If the library was built as nbd-client communicating with nbd-server,
> which then employ the BSO licensed code, could the library ignore the
> server side's license problem?

Yes, but if your first worry is the legal problem you are doomed to
design an awful library.

> The reason using nbd-client approach are:
> work around qemu block layer license issue and easy to implement

"Working around the QEMU block layer license" is not a goal per se,
especially because you haven't a) assessed _what_ is the GPL code that
the library would use; b) told us why the library should not be under
the GPL.

Please design first according to the functionality you want to
implement, then think about the implementation.

> , if
> other tool found this labrary useful then considering about directly
> employ the qemu block code.

Again, I find this to be quite backwards.  Writing a replacement for the
QEMU block layer just for licensing reasons is going to be a waste of
resources, since 90% of it is already BSD-licensed.

Perhaps you can produce two variants of the library, one using GPLed
backends and one entirely under the BSD license.  That would be good.
However we cannot help much in finding the best way to reach your goals,
again because you haven't reported on what actually is the GPL code that
you're worried about and why.

>>>    3 still not got a good way to get additional info in (2)(3)(4),
>>> currently in my head is patch qemu-nbd to add an additional nbd command,
>>> "image-info", in which returns related info.
>>
>> On the Linux kernel mailing list I would have no qualms labeling such
>> command as "crap".  However, since the social standards on qemu-devel
>> are a bit higher, I'll ask instead: what information would the command
>> provide beyond the size?
>>
>   The API need to report the image format it is using, such as
> "qcow2". And also API should report if a block at offset have been
> allocated or it is a hole.

qemu-nbd is designed to always provide the image format as raw, so its
client has no business knowing whether the image is originally stored as
qcow2 or something else.

Paolo

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Paolo Bonzini]

Il 10/07/2012 07:37, Wenchao Xia ha scritto:
>>
>> For getting the other metadata about the disk image you mention, another
>> possibility to is just make 'qemu-img info' return the data in a machine
>> parseable format, ie JSON&  make a client API for extracting data from
>> this JSON document.
>>
>   Thank u for the idea. The .so is introduced to let program access the
> image more directly, parsing string is not so fast and it depends on
> another program's stdout output, I hope to get a faster way.

I doubt you actually have profiled it.

Paolo

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Stefan Hajnoczi]

On Tue, Jul 10, 2012 at 09:18:01AM +0200, Paolo Bonzini wrote:
> Il 10/07/2012 07:37, Wenchao Xia ha scritto:
> >>
> >> For getting the other metadata about the disk image you mention, another
> >> possibility to is just make 'qemu-img info' return the data in a machine
> >> parseable format, ie JSON&  make a client API for extracting data from
> >> this JSON document.
> >>
> >   Thank u for the idea. The .so is introduced to let program access the
> > image more directly, parsing string is not so fast and it depends on
> > another program's stdout output, I hope to get a faster way.
> 
> I doubt you actually have profiled it.

I think speed is not the issue, instead it's just providing an API that
external programs can use.  Management tools, backup software, custom
administration tools, etc.  It's convenient to have an API.

Stefan

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Stefan Hajnoczi]

On Mon, Jul 09, 2012 at 04:36:07PM +0200, Christoph Hellwig wrote:
> On Mon, Jul 09, 2012 at 04:54:08PM +0800, Wenchao Xia wrote:
> > Hi, Paolo and folks,
> >   qemu have good capabilities to access different virtual disks, I want
> > to expose its block layer API to let 3rd party program linked in, such
> > as management stack or block tools, to access images data directly.
> > 
> > Following is the objects:
> >   (1) API to write/read block device at offset.
> >   (2) Determine the image type,qcow2/qed/raw
> >   (3) Determine which blocks are allocated.
> >   (4) Determine backing file.
> 
> Sounds like you want a procedural interface for that.  At least for (1)
> I have patches I'll submit soon to add qemu img read/write commands.

How is that different from all the qemu-io commands?

Stefan

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Stefan Hajnoczi]

On Tue, Jul 10, 2012 at 09:17:05AM +0200, Paolo Bonzini wrote:
> Il 10/07/2012 07:04, Wenchao Xia ha scritto:
> > 于 2012-7-9 17:13, Paolo Bonzini 写道:
> >> Il 09/07/2012 10:54, Wenchao Xia ha scritto:
> >>> Following is my implementing plan draft:
> >>>    1 introduce libqblock.so in sub directory in qemu.
> >>>    2 write a nbd client in libqblock, similar to qemu nbd client. Then
> >>> use it to talk with nbd server, by default is qemu-nbd, to get access
> >>> to images. In this way, libqblock.so could be friendly LGPL licensed.
> >>
> >> Did you actually assess the license situation of the block layer?
> >> block.c and large parts of block/* are under a BSD license, for example.
> >>   If the library only has to support raw files, it might do so using
> >> synchronous I/O only.  This would remove a large body of GPL-licensed code.
> >>
> >   If the library was built as nbd-client communicating with nbd-server,
> > which then employ the BSO licensed code, could the library ignore the
> > server side's license problem?
> 
> Yes, but if your first worry is the legal problem you are doomed to
> design an awful library.
> 
> > The reason using nbd-client approach are:
> > work around qemu block layer license issue and easy to implement
> 
> "Working around the QEMU block layer license" is not a goal per se,
> especially because you haven't a) assessed _what_ is the GPL code that
> the library would use; b) told us why the library should not be under
> the GPL.
> 
> Please design first according to the functionality you want to
> implement, then think about the implementation.

Licensing is one headache but the real challenge is that the QEMU block
layer relies on the QEMU main loop and a bunch of other architecture.

Using NBD allows us to focus on the library API instead worrying about
untangling the block layer.  If the library becomes useful it may be
worth fully moving the block layer code over into the library.

I think it makes sense to hammer out the library first before going down
a long road of internal QEMU refactoring before we know whether the
library will take off.

Stefan

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Daniel P. Berrange]

On Fri, Jul 13, 2012 at 10:12:47AM +0100, Stefan Hajnoczi wrote:
> On Tue, Jul 10, 2012 at 09:18:01AM +0200, Paolo Bonzini wrote:
> > Il 10/07/2012 07:37, Wenchao Xia ha scritto:
> > >>
> > >> For getting the other metadata about the disk image you mention, another
> > >> possibility to is just make 'qemu-img info' return the data in a machine
> > >> parseable format, ie JSON&  make a client API for extracting data from
> > >> this JSON document.
> > >>
> > >   Thank u for the idea. The .so is introduced to let program access the
> > > image more directly, parsing string is not so fast and it depends on
> > > another program's stdout output, I hope to get a faster way.
> > 
> > I doubt you actually have profiled it.
> 
> I think speed is not the issue, instead it's just providing an API that
> external programs can use.  Management tools, backup software, custom
> administration tools, etc.  It's convenient to have an API.

Actually I think speed could well be quite relevant. In large deployments
it would not be surpising to see 1000's of images in a directory. If you
want to be able to query metadata about all of them at once, then being
able to open()+read(4k)+close() 1000 times is going to be dramatically
faster than doing fork()+execve(qemu-img) 1000 times.

NB, I still think qemu-img info should be able to return a JSON parsable
data format, regardless of what any block library does.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Christoph Hellwig]

On Fri, Jul 13, 2012 at 10:13:15AM +0100, Stefan Hajnoczi wrote:
> How is that different from all the qemu-io commands?

qemu-io has no modes to just dumb the output without additional
information / statistics or for the write case just take user input
instead of a pattern.  I actually tried to add raw arguments to
qemu-io, which still worked ou ok for reads but started to get
fairly ugly for the write.

What I use in production right now is a trivial qemu-cat tool that
just does the raw reads and writes, but I think adding it as a new
sub command to qemu-img instead of another tool seems a bit cleaner.

If you and Kevin or Anthony disagree and want the qemu-cat tool I can
submit a patch for that instead.

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Stefan Hajnoczi]

On Fri, Jul 13, 2012 at 11:27:55AM +0200, Christoph Hellwig wrote:
> On Fri, Jul 13, 2012 at 10:13:15AM +0100, Stefan Hajnoczi wrote:
> > How is that different from all the qemu-io commands?
> 
> qemu-io has no modes to just dumb the output without additional
> information / statistics or for the write case just take user input
> instead of a pattern.  I actually tried to add raw arguments to
> qemu-io, which still worked ou ok for reads but started to get
> fairly ugly for the write.
> 
> What I use in production right now is a trivial qemu-cat tool that
> just does the raw reads and writes, but I think adding it as a new
> sub command to qemu-img instead of another tool seems a bit cleaner.
> 
> If you and Kevin or Anthony disagree and want the qemu-cat tool I can
> submit a patch for that instead.

Okay, I see what you mean.  I have used the hex output mode (when you
use the verbose option) but it's not raw.

Sounds like you want a qemu-dd :).  I think adding that to qemu-img is
fine though since it's already the tool that users are familiar with for
image file manipulation and that gets shipped.

Stefan

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Stefan Hajnoczi]

On Fri, Jul 13, 2012 at 10:16:40AM +0100, Daniel P. Berrange wrote:
> On Fri, Jul 13, 2012 at 10:12:47AM +0100, Stefan Hajnoczi wrote:
> > On Tue, Jul 10, 2012 at 09:18:01AM +0200, Paolo Bonzini wrote:
> > > Il 10/07/2012 07:37, Wenchao Xia ha scritto:
> > > >>
> > > >> For getting the other metadata about the disk image you mention, another
> > > >> possibility to is just make 'qemu-img info' return the data in a machine
> > > >> parseable format, ie JSON&  make a client API for extracting data from
> > > >> this JSON document.
> > > >>
> > > >   Thank u for the idea. The .so is introduced to let program access the
> > > > image more directly, parsing string is not so fast and it depends on
> > > > another program's stdout output, I hope to get a faster way.
> > > 
> > > I doubt you actually have profiled it.
> > 
> > I think speed is not the issue, instead it's just providing an API that
> > external programs can use.  Management tools, backup software, custom
> > administration tools, etc.  It's convenient to have an API.
> 
> Actually I think speed could well be quite relevant. In large deployments
> it would not be surpising to see 1000's of images in a directory. If you
> want to be able to query metadata about all of them at once, then being
> able to open()+read(4k)+close() 1000 times is going to be dramatically
> faster than doing fork()+execve(qemu-img) 1000 times.

Yes, that's true.  For situations like querying a whole repository of
image files speed matters.

I still think that we should focus on the API first.  Whether the .so
forks qemu-nbd/qemu-img or includes the actual block layer code only
matters once we are satisfied that this library is useful and works.
And for applications that use qemu-img today there won't be a speed
decrease, at least.

Stefan

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Paolo Bonzini]

Il 13/07/2012 11:16, Stefan Hajnoczi ha scritto:
>> "Working around the QEMU block layer license" is not a goal per se,
>> especially because you haven't a) assessed _what_ is the GPL code that
>> the library would use; b) told us why the library should not be under
>> the GPL.
>>
>> Please design first according to the functionality you want to
>> implement, then think about the implementation.
> 
> Licensing is one headache but the real challenge is that the QEMU block
> layer relies on the QEMU main loop and a bunch of other architecture.

It doesn't really, not on Windows which has no AIO for example.  That's
why I suggested:

- assessing what code is GPL and what are the dependencies on it

- only including the formats in the library + a synchronous file
protocol.  No NBD, no libiscsi, no hdev, etc.

My suspicion is that with this strategy the only complicated dependency
is on coroutines.

Paolo

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Kevin Wolf]

Am 13.07.2012 11:43, schrieb Stefan Hajnoczi:
> On Fri, Jul 13, 2012 at 11:27:55AM +0200, Christoph Hellwig wrote:
>> On Fri, Jul 13, 2012 at 10:13:15AM +0100, Stefan Hajnoczi wrote:
>>> How is that different from all the qemu-io commands?
>>
>> qemu-io has no modes to just dumb the output without additional
>> information / statistics or for the write case just take user input
>> instead of a pattern.  I actually tried to add raw arguments to
>> qemu-io, which still worked ou ok for reads but started to get
>> fairly ugly for the write.
>>
>> What I use in production right now is a trivial qemu-cat tool that
>> just does the raw reads and writes, but I think adding it as a new
>> sub command to qemu-img instead of another tool seems a bit cleaner.
>>
>> If you and Kevin or Anthony disagree and want the qemu-cat tool I can
>> submit a patch for that instead.
> 
> Okay, I see what you mean.  I have used the hex output mode (when you
> use the verbose option) but it's not raw.
> 
> Sounds like you want a qemu-dd :).  I think adding that to qemu-img is
> fine though since it's already the tool that users are familiar with for
> image file manipulation and that gets shipped.

It still feels a bit more like qemu-io-style operations. Not sure what
your use case looks like exactly, but adding a qemu-io command that
reads data from a file and writes it at a given offset into the images
(or vice versa) should be easy. This would be more or less a qemu-dd.

If you need to get data from stdin or output it to stdout, then it might
not be the right solution.

Kevin

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Christoph Hellwig]

On Fri, Jul 13, 2012 at 12:42:41PM +0200, Kevin Wolf wrote:
> It still feels a bit more like qemu-io-style operations. Not sure what
> your use case looks like exactly, but adding a qemu-io command that
> reads data from a file and writes it at a given offset into the images
> (or vice versa) should be easy. This would be more or less a qemu-dd.

We already have that if you read/write the whole image, it's called
qemu-img convert.

> If you need to get data from stdin or output it to stdout, then it might
> not be the right solution.

That is the use case.  We could appromite it by writing a temp file and
using qemu-img convert, but that's not very efficient.

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Kevin Wolf]

Am 13.07.2012 12:55, schrieb Christoph Hellwig:
> On Fri, Jul 13, 2012 at 12:42:41PM +0200, Kevin Wolf wrote:
>> It still feels a bit more like qemu-io-style operations. Not sure what
>> your use case looks like exactly, but adding a qemu-io command that
>> reads data from a file and writes it at a given offset into the images
>> (or vice versa) should be easy. This would be more or less a qemu-dd.
> 
> We already have that if you read/write the whole image, it's called
> qemu-img convert.

Hm. Fair point, I guess. Though qemu-img tends to have commands that
deal with complete disks instead of just areas.

>> If you need to get data from stdin or output it to stdout, then it might
>> not be the right solution.
> 
> That is the use case.  We could appromite it by writing a temp file and
> using qemu-img convert, but that's not very efficient. 

Yeah, it's not what you want then.

So the question is whether to have it integrated in qemu-img or
standalone. I'm undecided: Having everything in one well-known tool has
its advantages. But then, a qemu-dd that feels like a real dd, just that
it opens image formats with the right driver instead of always using
raw, certainly sounds attractive, too.

Let's wait a bit for more opinions. If there aren't any - you write the
code, you decide.

Kevin

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Paolo Bonzini]

Il 13/07/2012 11:51, Paolo Bonzini ha scritto:
> Il 13/07/2012 11:16, Stefan Hajnoczi ha scritto:
>>> "Working around the QEMU block layer license" is not a goal per se,
>>> especially because you haven't a) assessed _what_ is the GPL code that
>>> the library would use; b) told us why the library should not be under
>>> the GPL.
>>>
>>> Please design first according to the functionality you want to
>>> implement, then think about the implementation.
>>
>> Licensing is one headache but the real challenge is that the QEMU block
>> layer relies on the QEMU main loop and a bunch of other architecture.
> 
> It doesn't really, not on Windows which has no AIO for example.  That's
> why I suggested:
> 
> - assessing what code is GPL and what are the dependencies on it

So I tried trimming down the list of files needed to compile
qemu tools, and here is a list:

Easy to relicense to LGPLv2+:
block/raw.c                     none (GPLv2+: Red Hat, IBM)
error.c                         LGPLv2 (Red Hat, IBM, Stefan Weil)
iov.c                           GPLv2 (Red Hat, SuSE/Hannes Reinecke, Michael Tokarev)
module.c                        GPLv2 (Red Hat, IBM, Blue Swirl)
qemu-error.c                    GPLv2+ (Red Hat, Blue Swirl, IBM)
trace/control.c                 GPLv2 (Lluis Vilanova)
trace/default.c                 GPLv2 (Lluis Vilanova)

(I added some people to Cc.  Lluis and Michael, can you also look at
http://wiki.qemu.org/Relicensing if you're willing to relicense
your past contributions from GPLv2 to GPLv2+?.  Blue Swirl said 
he'd accept any other GPLv2 or GPLv3 compatible license, which
should include LGPLv2+).

Harder to relicense to LGPLv2+:
block/vdi.c                     GPLv2+

"Good" license:
aes.c                           BSD
async.c                         BSD
block.c                         BSD
block/bochs.c                   BSD
block/cloop.c                   BSD
block/cow.c                     BSD
block/dmg.c                     BSD
block/parallels.c               BSD
block/qcow.c                    BSD
block/qcow2-cache.c             BSD
block/qcow2-cluster.c           BSD
block/qcow2-refcount.c          BSD
block/qcow2-snapshot.c          BSD
block/qcow2.c                   BSD
block/qed-check.c               BSD
block/qed-cluster.c             BSD
block/qed-gencb.c               BSD
block/qed-l2-cache.c            BSD
block/qed-table.c               BSD
block/qed.c                     BSD
block/vmdk.c                    BSD
block/vpc.c                     BSD
block/vvfat.c                   BSD
cutils.c                        BSD
osdep.c                         BSD
oslib-posix.c                   BSD
qemu-coroutine-io.c             BSD
qemu-coroutine-lock.c           BSD
qemu-option.c                   BSD
qemu-progress.c                 BSD
coroutine-ucontext.c            LGPLv2+
json-lexer.c                    LGPLv2+
json-parser.c                   LGPLv2+
json-streamer.c                 LGPLv2+
qbool.c                         LGPLv2+
qdict.c                         LGPLv2+
qemu-coroutine.c                LGPLv2+
qerror.c                        LGPLv2+
qfloat.c                        LGPLv2+
qint.c                          LGPLv2+
qjson.c                         LGPLv2+
qlist.c                         LGPLv2+
qstring.c                       LGPLv2+

Doesn't need to be included in a library:
qemu-tool.c                     GPLv2

Autogenerated:
trace.c

Remaining undefined symbols:
qemu_aio_flush
qemu_aio_wait
qemu_free_timer
qemu_new_timer
qemu_mod_timer
qemu_del_timer
qemu_get_clock_ns
vm_clock
+ those defined in qemu-tool.c

Paolo

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Michael Tokarev]

On 13.07.2012 15:33, Paolo Bonzini wrote:
[]
> So I tried trimming down the list of files needed to compile
> qemu tools, and here is a list:
> 
> Easy to relicense to LGPLv2+:
> block/raw.c                     none (GPLv2+: Red Hat, IBM)
> error.c                         LGPLv2 (Red Hat, IBM, Stefan Weil)
> iov.c                           GPLv2 (Red Hat, SuSE/Hannes Reinecke, Michael Tokarev)
> module.c                        GPLv2 (Red Hat, IBM, Blue Swirl)
> qemu-error.c                    GPLv2+ (Red Hat, Blue Swirl, IBM)
> trace/control.c                 GPLv2 (Lluis Vilanova)
> trace/default.c                 GPLv2 (Lluis Vilanova)
> 
> (I added some people to Cc.  Lluis and Michael, can you also look at
> http://wiki.qemu.org/Relicensing if you're willing to relicense
> your past contributions from GPLv2 to GPLv2+?.  Blue Swirl said 
> he'd accept any other GPLv2 or GPLv3 compatible license, which
> should include LGPLv2+).

I'm fine with relicensing any my contributions to qemu project under
any version of GPL or LGPL, BSD, or any other open license, as the
project see fit.

Thank you!


/mjt

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Blue Swirl]

On Fri, Jul 13, 2012 at 11:33 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 13/07/2012 11:51, Paolo Bonzini ha scritto:
>> Il 13/07/2012 11:16, Stefan Hajnoczi ha scritto:
>>>> "Working around the QEMU block layer license" is not a goal per se,
>>>> especially because you haven't a) assessed _what_ is the GPL code that
>>>> the library would use; b) told us why the library should not be under
>>>> the GPL.
>>>>
>>>> Please design first according to the functionality you want to
>>>> implement, then think about the implementation.
>>>
>>> Licensing is one headache but the real challenge is that the QEMU block
>>> layer relies on the QEMU main loop and a bunch of other architecture.
>>
>> It doesn't really, not on Windows which has no AIO for example.  That's
>> why I suggested:
>>
>> - assessing what code is GPL and what are the dependencies on it
>
> So I tried trimming down the list of files needed to compile
> qemu tools, and here is a list:
>
> Easy to relicense to LGPLv2+:
> block/raw.c                     none (GPLv2+: Red Hat, IBM)
> error.c                         LGPLv2 (Red Hat, IBM, Stefan Weil)
> iov.c                           GPLv2 (Red Hat, SuSE/Hannes Reinecke, Michael Tokarev)
> module.c                        GPLv2 (Red Hat, IBM, Blue Swirl)
> qemu-error.c                    GPLv2+ (Red Hat, Blue Swirl, IBM)
> trace/control.c                 GPLv2 (Lluis Vilanova)
> trace/default.c                 GPLv2 (Lluis Vilanova)
>
> (I added some people to Cc.  Lluis and Michael, can you also look at
> http://wiki.qemu.org/Relicensing if you're willing to relicense
> your past contributions from GPLv2 to GPLv2+?.  Blue Swirl said
> he'd accept any other GPLv2 or GPLv3 compatible license, which
> should include LGPLv2+).

I'm fine with LGPLv2+ too.

I have some reservations to supporting a block device library API/ABI
by external users, but if this is OK with block maintainer and we
allow some decent API/ABI changes in the future if needed, this should
be OK.

>
> Harder to relicense to LGPLv2+:
> block/vdi.c                     GPLv2+
>
> "Good" license:
> aes.c                           BSD
> async.c                         BSD
> block.c                         BSD
> block/bochs.c                   BSD
> block/cloop.c                   BSD
> block/cow.c                     BSD
> block/dmg.c                     BSD
> block/parallels.c               BSD
> block/qcow.c                    BSD
> block/qcow2-cache.c             BSD
> block/qcow2-cluster.c           BSD
> block/qcow2-refcount.c          BSD
> block/qcow2-snapshot.c          BSD
> block/qcow2.c                   BSD
> block/qed-check.c               BSD
> block/qed-cluster.c             BSD
> block/qed-gencb.c               BSD
> block/qed-l2-cache.c            BSD
> block/qed-table.c               BSD
> block/qed.c                     BSD
> block/vmdk.c                    BSD
> block/vpc.c                     BSD
> block/vvfat.c                   BSD
> cutils.c                        BSD
> osdep.c                         BSD
> oslib-posix.c                   BSD
> qemu-coroutine-io.c             BSD
> qemu-coroutine-lock.c           BSD
> qemu-option.c                   BSD
> qemu-progress.c                 BSD
> coroutine-ucontext.c            LGPLv2+
> json-lexer.c                    LGPLv2+
> json-parser.c                   LGPLv2+
> json-streamer.c                 LGPLv2+
> qbool.c                         LGPLv2+
> qdict.c                         LGPLv2+
> qemu-coroutine.c                LGPLv2+
> qerror.c                        LGPLv2+
> qfloat.c                        LGPLv2+
> qint.c                          LGPLv2+
> qjson.c                         LGPLv2+
> qlist.c                         LGPLv2+
> qstring.c                       LGPLv2+
>
> Doesn't need to be included in a library:
> qemu-tool.c                     GPLv2
>
> Autogenerated:
> trace.c
>
> Remaining undefined symbols:
> qemu_aio_flush
> qemu_aio_wait
> qemu_free_timer
> qemu_new_timer
> qemu_mod_timer
> qemu_del_timer
> qemu_get_clock_ns
> vm_clock
> + those defined in qemu-tool.c
>
> Paolo

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Stefan Weil]

Am 13.07.2012 13:33, schrieb Paolo Bonzini:
> Il 13/07/2012 11:51, Paolo Bonzini ha scritto:
>> Il 13/07/2012 11:16, Stefan Hajnoczi ha scritto:
>>>> "Working around the QEMU block layer license" is not a goal per se,
>>>> especially because you haven't a) assessed _what_ is the GPL code that
>>>> the library would use; b) told us why the library should not be under
>>>> the GPL.
>>>>
>>>> Please design first according to the functionality you want to
>>>> implement, then think about the implementation.
>>>
>>> Licensing is one headache but the real challenge is that the QEMU block
>>> layer relies on the QEMU main loop and a bunch of other architecture.
>>
>> It doesn't really, not on Windows which has no AIO for example.  That's
>> why I suggested:
>>
>> - assessing what code is GPL and what are the dependencies on it
>
> So I tried trimming down the list of files needed to compile
> qemu tools, and here is a list:
>
> Easy to relicense to LGPLv2+:
> block/raw.c                     none (GPLv2+: Red Hat, IBM)
> error.c                         LGPLv2 (Red Hat, IBM, Stefan Weil)

I only added an include statement and don't mind
changing the license for error.c to LGPLv2+.

>
> iov.c                           GPLv2 (Red Hat, SuSE/Hannes Reinecke, Michael Tokarev)
> module.c                        GPLv2 (Red Hat, IBM, Blue Swirl)
> qemu-error.c                    GPLv2+ (Red Hat, Blue Swirl, IBM)
> trace/control.c                 GPLv2 (Lluis Vilanova)
> trace/default.c                 GPLv2 (Lluis Vilanova)
>
> (I added some people to Cc.  Lluis and Michael, can you also look at
> http://wiki.qemu.org/Relicensing if you're willing to relicense
> your past contributions from GPLv2 to GPLv2+?.  Blue Swirl said
> he'd accept any other GPLv2 or GPLv3 compatible license, which
> should include LGPLv2+).
>
> Harder to relicense to LGPLv2+:
> block/vdi.c                     GPLv2+

Indeed, that one is harder. Most of the code is from me,
and I need a good reason why the license should be changed.

Of course the dynamic library can also be compiled without
VDI support.

Regards,
Stefan W.

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Lluís Vilanova]

Paolo Bonzini writes:

> Il 13/07/2012 11:51, Paolo Bonzini ha scritto:
>> Il 13/07/2012 11:16, Stefan Hajnoczi ha scritto:
>>>> "Working around the QEMU block layer license" is not a goal per se,
>>>> especially because you haven't a) assessed _what_ is the GPL code that
>>>> the library would use; b) told us why the library should not be under
>>>> the GPL.
>>>> 
>>>> Please design first according to the functionality you want to
>>>> implement, then think about the implementation.
>>> 
>>> Licensing is one headache but the real challenge is that the QEMU block
>>> layer relies on the QEMU main loop and a bunch of other architecture.
>> 
>> It doesn't really, not on Windows which has no AIO for example.  That's
>> why I suggested:
>> 
>> - assessing what code is GPL and what are the dependencies on it

> So I tried trimming down the list of files needed to compile
> qemu tools, and here is a list:

> Easy to relicense to LGPLv2+:
> block/raw.c                     none (GPLv2+: Red Hat, IBM)
> error.c                         LGPLv2 (Red Hat, IBM, Stefan Weil)
> iov.c                           GPLv2 (Red Hat, SuSE/Hannes Reinecke, Michael Tokarev)
> module.c                        GPLv2 (Red Hat, IBM, Blue Swirl)
> qemu-error.c                    GPLv2+ (Red Hat, Blue Swirl, IBM)
> trace/control.c                 GPLv2 (Lluis Vilanova)
> trace/default.c                 GPLv2 (Lluis Vilanova)

> (I added some people to Cc.  Lluis and Michael, can you also look at
> http://wiki.qemu.org/Relicensing if you're willing to relicense
> your past contributions from GPLv2 to GPLv2+?.  Blue Swirl said 
> he'd accept any other GPLv2 or GPLv3 compatible license, which
> should include LGPLv2+).

I have no problems relicensing to "GPLv2 or later" or "GPLv3 or later".


Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Wenchao Xia]

 > On Fri, Jul 13, 2012 at 10:12:47AM +0100, Stefan Hajnoczi wrote:
>> On Tue, Jul 10, 2012 at 09:18:01AM +0200, Paolo Bonzini wrote:
>>> Il 10/07/2012 07:37, Wenchao Xia ha scritto:
>>>>>
>>>>> For getting the other metadata about the disk image you mention, another
>>>>> possibility to is just make 'qemu-img info' return the data in a machine
>>>>> parseable format, ie JSON&  make a client API for extracting data from
>>>>> this JSON document.
>>>>>
>>>>    Thank u for the idea. The .so is introduced to let program access the
>>>> image more directly, parsing string is not so fast and it depends on
>>>> another program's stdout output, I hope to get a faster way.
>>>
>>> I doubt you actually have profiled it.
>>
>> I think speed is not the issue, instead it's just providing an API that
>> external programs can use.  Management tools, backup software, custom
>> administration tools, etc.  It's convenient to have an API.
>
> Actually I think speed could well be quite relevant. In large deployments
> it would not be surpising to see 1000's of images in a directory. If you
> want to be able to query metadata about all of them at once, then being
> able to open()+read(4k)+close() 1000 times is going to be dramatically
> faster than doing fork()+execve(qemu-img) 1000 times.
>
   I tried to avoid fork() and execv() in the library as much as
possible. Dealing with another program's string output will cost quite
some codes about its timeout or exception error, more "native" code
seems better to me.

> NB, I still think qemu-img info should be able to return a JSON parsable
> data format, regardless of what any block library does.
   An option to qemu-img to specify output format may be a good way.

>
> Daniel
>


-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Wenchao Xia]

于 2012-7-13 17:27, Christoph Hellwig 写道:
> On Fri, Jul 13, 2012 at 10:13:15AM +0100, Stefan Hajnoczi wrote:
>> How is that different from all the qemu-io commands?
>
> qemu-io has no modes to just dumb the output without additional
> information / statistics or for the write case just take user input
> instead of a pattern.  I actually tried to add raw arguments to
> qemu-io, which still worked ou ok for reads but started to get
> fairly ugly for the write.
>
> What I use in production right now is a trivial qemu-cat tool that
> just does the raw reads and writes, but I think adding it as a new
> sub command to qemu-img instead of another tool seems a bit cleaner.
>
> If you and Kevin or Anthony disagree and want the qemu-cat tool I can
> submit a patch for that instead.
>
   That seems one user case the library should support. The library is
supposed to do more to expose generic block library which qemu device
emulator have used, the ideal expect is that other program linked with
the library could operate image the same way as qemu core.



-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Wenchao Xia]

于 2012-7-13 19:33, Paolo Bonzini 写道:
> Il 13/07/2012 11:51, Paolo Bonzini ha scritto:
>> Il 13/07/2012 11:16, Stefan Hajnoczi ha scritto:
>>>> "Working around the QEMU block layer license" is not a goal per se,
>>>> especially because you haven't a) assessed _what_ is the GPL code that
>>>> the library would use; b) told us why the library should not be under
>>>> the GPL.
>>>>
>>>> Please design first according to the functionality you want to
>>>> implement, then think about the implementation.
>>>
>>> Licensing is one headache but the real challenge is that the QEMU block
>>> layer relies on the QEMU main loop and a bunch of other architecture.
>>
>> It doesn't really, not on Windows which has no AIO for example.  That's
>> why I suggested:
>>
>> - assessing what code is GPL and what are the dependencies on it
>
> So I tried trimming down the list of files needed to compile
> qemu tools, and here is a list:
>
   Really thanks for the investigation, I paid quite sometime to dig out
which license is compatible to LGPL, this have sorted it out.
   The coroutine and structure inside is quite a challenge. What about
provide the library first in nbd + sync access, and waiting for the
library employer response? If it is good to use, then replace implement
code to native qemu block layer code, change code's license, while keep
API unchanged.


> Easy to relicense to LGPLv2+:
> block/raw.c                     none (GPLv2+: Red Hat, IBM)
> error.c                         LGPLv2 (Red Hat, IBM, Stefan Weil)
> iov.c                           GPLv2 (Red Hat, SuSE/Hannes Reinecke, Michael Tokarev)
> module.c                        GPLv2 (Red Hat, IBM, Blue Swirl)
> qemu-error.c                    GPLv2+ (Red Hat, Blue Swirl, IBM)
> trace/control.c                 GPLv2 (Lluis Vilanova)
> trace/default.c                 GPLv2 (Lluis Vilanova)
>
> (I added some people to Cc.  Lluis and Michael, can you also look at
> http://wiki.qemu.org/Relicensing if you're willing to relicense
> your past contributions from GPLv2 to GPLv2+?.  Blue Swirl said
> he'd accept any other GPLv2 or GPLv3 compatible license, which
> should include LGPLv2+).
>
> Harder to relicense to LGPLv2+:
> block/vdi.c                     GPLv2+
>
> "Good" license:
> aes.c                           BSD
> async.c                         BSD
> block.c                         BSD
> block/bochs.c                   BSD
> block/cloop.c                   BSD
> block/cow.c                     BSD
> block/dmg.c                     BSD
> block/parallels.c               BSD
> block/qcow.c                    BSD
> block/qcow2-cache.c             BSD
> block/qcow2-cluster.c           BSD
> block/qcow2-refcount.c          BSD
> block/qcow2-snapshot.c          BSD
> block/qcow2.c                   BSD
> block/qed-check.c               BSD
> block/qed-cluster.c             BSD
> block/qed-gencb.c               BSD
> block/qed-l2-cache.c            BSD
> block/qed-table.c               BSD
> block/qed.c                     BSD
> block/vmdk.c                    BSD
> block/vpc.c                     BSD
> block/vvfat.c                   BSD
> cutils.c                        BSD
> osdep.c                         BSD
> oslib-posix.c                   BSD
> qemu-coroutine-io.c             BSD
> qemu-coroutine-lock.c           BSD
> qemu-option.c                   BSD
> qemu-progress.c                 BSD
> coroutine-ucontext.c            LGPLv2+
> json-lexer.c                    LGPLv2+
> json-parser.c                   LGPLv2+
> json-streamer.c                 LGPLv2+
> qbool.c                         LGPLv2+
> qdict.c                         LGPLv2+
> qemu-coroutine.c                LGPLv2+
> qerror.c                        LGPLv2+
> qfloat.c                        LGPLv2+
> qint.c                          LGPLv2+
> qjson.c                         LGPLv2+
> qlist.c                         LGPLv2+
> qstring.c                       LGPLv2+
>
> Doesn't need to be included in a library:
> qemu-tool.c                     GPLv2
>
> Autogenerated:
> trace.c
>
> Remaining undefined symbols:
> qemu_aio_flush
> qemu_aio_wait
> qemu_free_timer
> qemu_new_timer
> qemu_mod_timer
> qemu_del_timer
> qemu_get_clock_ns
> vm_clock
> + those defined in qemu-tool.c
>
> Paolo
>


-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Paolo Bonzini]

Il 16/07/2012 10:16, Wenchao Xia ha scritto:
>>
>   Really thanks for the investigation, I paid quite sometime to dig out
> which license is compatible to LGPL, this have sorted it out.
>   The coroutine and structure inside is quite a challenge.

Coroutines are really just a small complication in the program flow if
all you support is synchronous access to files (i.e. no HTTP etc.).
Their usage should be completely transparent.

> What about
> provide the library first in nbd + sync access, and waiting for the
> library employer response? If it is good to use, then replace implement
> code to native qemu block layer code, change code's license, while keep
> API unchanged.

You can start by proposing the API.

Paolo

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Stefan Hajnoczi]

On Sat, Jul 14, 2012 at 01:55:07AM +0300, Lluís Vilanova wrote:
> Paolo Bonzini writes:
> 
> > Il 13/07/2012 11:51, Paolo Bonzini ha scritto:
> >> Il 13/07/2012 11:16, Stefan Hajnoczi ha scritto:
> >>>> "Working around the QEMU block layer license" is not a goal per se,
> >>>> especially because you haven't a) assessed _what_ is the GPL code that
> >>>> the library would use; b) told us why the library should not be under
> >>>> the GPL.
> >>>> 
> >>>> Please design first according to the functionality you want to
> >>>> implement, then think about the implementation.
> >>> 
> >>> Licensing is one headache but the real challenge is that the QEMU block
> >>> layer relies on the QEMU main loop and a bunch of other architecture.
> >> 
> >> It doesn't really, not on Windows which has no AIO for example.  That's
> >> why I suggested:
> >> 
> >> - assessing what code is GPL and what are the dependencies on it
> 
> > So I tried trimming down the list of files needed to compile
> > qemu tools, and here is a list:
> 
> > Easy to relicense to LGPLv2+:
> > block/raw.c                     none (GPLv2+: Red Hat, IBM)
> > error.c                         LGPLv2 (Red Hat, IBM, Stefan Weil)
> > iov.c                           GPLv2 (Red Hat, SuSE/Hannes Reinecke, Michael Tokarev)
> > module.c                        GPLv2 (Red Hat, IBM, Blue Swirl)
> > qemu-error.c                    GPLv2+ (Red Hat, Blue Swirl, IBM)
> > trace/control.c                 GPLv2 (Lluis Vilanova)
> > trace/default.c                 GPLv2 (Lluis Vilanova)
> 
> > (I added some people to Cc.  Lluis and Michael, can you also look at
> > http://wiki.qemu.org/Relicensing if you're willing to relicense
> > your past contributions from GPLv2 to GPLv2+?.  Blue Swirl said 
> > he'd accept any other GPLv2 or GPLv3 compatible license, which
> > should include LGPLv2+).
> 
> I have no problems relicensing to "GPLv2 or later" or "GPLv3 or later".

What about LGPLv2+?  (Note the "L".)

Stefan

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Wenchao Xia]

   Hi, following is API draft, prototypes were taken from qemu/block.h,
and the API prefix is changed frpm bdrv to qbdrvs, to declare related
object is BlockDriverState, not BlockDriver. One issue here is it may
require include block_int.h, which is not LGPL2 licensed yet.
   API format is kept mostly the same with qemu generic block layer, to
make it easier for implement, and easier to make qemu migrate on it if
possible.


/* structure init and uninit */
BlockDriverState *qbdrvs_new(const char *device_name);
void qbdrvs_delete(BlockDriverState *bs);


/* file open and close */
int qbdrvs_open(BlockDriverState *bs, const char *filename, int flags,
               BlockDriver *drv);
void qbdrvs_close(BlockDriverState *bs);
int qbdrvs_img_create(const char *filename, const char *fmt,
                     const char *base_filename, const char *base_fmt,
                     char *options, uint64_t img_size, int flags);


/* sync access */
int qbdrvs_read(BlockDriverState *bs, int64_t sector_num,
               uint8_t *buf, int nb_sectors);
int qbdrvs_write(BlockDriverState *bs, int64_t sector_num,
                const uint8_t *buf, int nb_sectors);


/* info retrieve */
//sector, size and geometry info
int qbdrvs_get_info(BlockDriverState *bs, BlockDriverInfo *bdi);
int64_t qbdrvs_getlength(BlockDriverState *bs);
int64_t qbdrvs_get_allocated_file_size(BlockDriverState *bs);
void qbdrvs_get_geometry(BlockDriverState *bs, uint64_t *nb_sectors_ptr);
//image type
const char *qbdrvs_get_format_name(BlockDriverState *bs);
//backing file info
void qbdrvs_get_backing_filename(BlockDriverState *bs,
                                char *filename, int filename_size);
void qbdrvs_get_full_backing_filename(BlockDriverState *bs,
                                     char *dest, size_t sz);


/* advanced image content access */
int qbdrvs_is_allocated(BlockDriverState *bs, int64_t sector_num, int 
nb_sectors,
                       int *pnum);
int qbdrvs_discard(BlockDriverState *bs, int64_t sector_num, int 
nb_sectors);
int qbdrvs_has_zero_init(BlockDriverState *bs);

> Il 16/07/2012 10:16, Wenchao Xia ha scritto:
>>>
>>    Really thanks for the investigation, I paid quite sometime to dig out
>> which license is compatible to LGPL, this have sorted it out.
>>    The coroutine and structure inside is quite a challenge.
>
> Coroutines are really just a small complication in the program flow if
> all you support is synchronous access to files (i.e. no HTTP etc.).
> Their usage should be completely transparent.
>
>> What about
>> provide the library first in nbd + sync access, and waiting for the
>> library employer response? If it is good to use, then replace implement
>> code to native qemu block layer code, change code's license, while keep
>> API unchanged.
>
> You can start by proposing the API.
>
> Paolo
>


-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Paolo Bonzini]

Il 18/07/2012 10:51, Wenchao Xia ha scritto:
>   Hi, following is API draft, prototypes were taken from qemu/block.h,
> and the API prefix is changed frpm bdrv to qbdrvs, to declare related
> object is BlockDriverState, not BlockDriver. One issue here is it may
> require include block_int.h, which is not LGPL2 licensed yet.

block_int.h is BSD-licensed.  It's compatible.

>   API format is kept mostly the same with qemu generic block layer, to
> make it easier for implement, and easier to make qemu migrate on it if
> possible.

I'll note that img_create, and almost all retrieval functions cannot be
implemented on top of NBD.

> /* structure init and uninit */
> BlockDriverState *qbdrvs_new(const char *device_name);

device_name is not needed in the library.

> void qbdrvs_delete(BlockDriverState *bs);

Being able to reuse the same BDS with close/open is one of the worst
parts of the QEMU block layer.  It is only needed to implement eject in
QEMU.  Let's get things right, and only have open/close:

int qbdrvs_open(BlockDriverState **bs, const char *filename, int flags,
                const char *format_name);
void qbdrvs_close(BlockDriverState *bs);

> 
> 
> /* file open and close */
> int qbdrvs_open(BlockDriverState *bs, const char *filename, int flags,
>               BlockDriver *drv);

You didn't have an API to find the BlockDriver *.  Let's just pass a
format name, or NULL for probing (see above). This is consistent with
qbdrvs_img_create.

> void qbdrvs_close(BlockDriverState *bs);
> int qbdrvs_img_create(const char *filename, const char *fmt,
>                     const char *base_filename, const char *base_fmt,
>                     char *options, uint64_t img_size, int flags);
> 
> 
> /* sync access */
> int qbdrvs_read(BlockDriverState *bs, int64_t sector_num,
>               uint8_t *buf, int nb_sectors);
> int qbdrvs_write(BlockDriverState *bs, int64_t sector_num,
>                const uint8_t *buf, int nb_sectors);

I would like to have also a scatter gather API (qbdrvs_readv and
qbdrvs_writev) taking a "struct iovec *iov, int niov" instead of
"uint8_t *buf, int nb_sectors".

flush is missing.

> 
> /* info retrieve */
> //sector, size and geometry info
> int qbdrvs_get_info(BlockDriverState *bs, BlockDriverInfo *bdi);

One problem here is that BlockDriverInfo may grow in the future.  This
is a problem for the ABI of the library.  I don't have any particular
ideas here, except adding a separate API to the library for each member
of BlockDriverInfo.

> int64_t qbdrvs_getlength(BlockDriverState *bs);

qbdrvs_get_length.

> int64_t qbdrvs_get_allocated_file_size(BlockDriverState *bs);
> void qbdrvs_get_geometry(BlockDriverState *bs, uint64_t *nb_sectors_ptr);

Not needed, it's the same as bdrv_getlength.

> //image type
> const char *qbdrvs_get_format_name(BlockDriverState *bs);
> //backing file info
> void qbdrvs_get_backing_filename(BlockDriverState *bs,
>                                char *filename, int filename_size);
> void qbdrvs_get_full_backing_filename(BlockDriverState *bs,
>                                     char *dest, size_t sz);
> 
> 
> /* advanced image content access */
> int qbdrvs_is_allocated(BlockDriverState *bs, int64_t sector_num, int
> nb_sectors,
>                       int *pnum);
> int qbdrvs_discard(BlockDriverState *bs, int64_t sector_num, int
> nb_sectors);
> int qbdrvs_has_zero_init(BlockDriverState *bs);

Paolo

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Stefan Hajnoczi]

On Wed, Jul 18, 2012 at 9:51 AM, Wenchao Xia <xiawenc@linux.vnet.ibm.com> wrote:
> /* sync access */
> int qbdrvs_read(BlockDriverState *bs, int64_t sector_num,
>               uint8_t *buf, int nb_sectors);
> int qbdrvs_write(BlockDriverState *bs, int64_t sector_num,
>                const uint8_t *buf, int nb_sectors);

Whether to provide sync and/or async access is a key question.

Synchronous APIs are great for writing dedicated tools like dd, cp,
convert, etc.

Asynchronous APIs are essential for integrating image file I/O into
event-driven programs like libvirt.  Here, the ability to do other
things while image file I/O is in progress is a requirement.  It may
also be necessary to cancel or timeout if an operation is not making
progress or the user decides to stop it.

I think we need to provide both sync and async.  Libraries like
libssh2 and libcurl already do this so their APIs can be used as a
starting point for async I/O.

Stefan

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Paolo Bonzini]

> Whether to provide sync and/or async access is a key question.

Indeed.

> Synchronous APIs are great for writing dedicated tools like dd, cp,
> convert, etc.
> 
> Asynchronous APIs are essential for integrating image file I/O into
> event-driven programs like libvirt.  Here, the ability to do other
> things while image file I/O is in progress is a requirement.  It may
> also be necessary to cancel or timeout if an operation is not making
> progress or the user decides to stop it.
> 
> I think we need to provide both sync and async.  Libraries like
> libssh2 and libcurl already do this so their APIs can be used as a
> starting point for async I/O.

If we want to provide an asynchronous API, the easiest thing would
be to provide a GSource and that's it.  That would even make sense for
QEMU itself, in fact.

What I'm worried about, is how to support _both_ synchronous and
asynchronous access.  I'd like the library to be clean of things like
qemu_aio_wait() and qemu_aio_flush(), at least in the beginning.
That's why I think async can come later, once we actually get
applications needing it.  Right now, libvirt's requirements are
simple (e.g. probing the backing file chain) and would be synchronous
anyway.

Paolo

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Stefan Hajnoczi]

On Wed, Jul 18, 2012 at 06:42:24AM -0400, Paolo Bonzini wrote:
> > Synchronous APIs are great for writing dedicated tools like dd, cp,
> > convert, etc.
> > 
> > Asynchronous APIs are essential for integrating image file I/O into
> > event-driven programs like libvirt.  Here, the ability to do other
> > things while image file I/O is in progress is a requirement.  It may
> > also be necessary to cancel or timeout if an operation is not making
> > progress or the user decides to stop it.
> > 
> > I think we need to provide both sync and async.  Libraries like
> > libssh2 and libcurl already do this so their APIs can be used as a
> > starting point for async I/O.
> 
> If we want to provide an asynchronous API, the easiest thing would
> be to provide a GSource and that's it.  That would even make sense for
> QEMU itself, in fact.
> 
> What I'm worried about, is how to support _both_ synchronous and
> asynchronous access.  I'd like the library to be clean of things like
> qemu_aio_wait() and qemu_aio_flush(), at least in the beginning.
> That's why I think async can come later, once we actually get
> applications needing it.  Right now, libvirt's requirements are
> simple (e.g. probing the backing file chain) and would be synchronous
> anyway.

Yes, qemu_aio_wait() and qemu_aio_flush() are ugly.

Starting with sync makes sense, it's a convenient API to have even if we
add async later.

Stefan

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Andreas Färber]

Am 18.07.2012 10:51, schrieb Wenchao Xia:
>   Hi, following is API draft, prototypes were taken from qemu/block.h,
> and the API prefix is changed frpm bdrv to qbdrvs, to declare related
> object is BlockDriverState, not BlockDriver. [...]

So let the bikeshedding begin: ;)

What about qbds_ prefix rather than qbdrvs_? I find the proposed mixture
of acronym (q for QEMU, b for Block, s for State) and abbreviation (drv
as in Driver) a bit ugly.

Or just simply go for qblock - might be better memorable. :)

Just my 2¢ (on something I don't use much),

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Kevin Wolf]

Am 18.07.2012 15:51, schrieb Andreas Färber:
> Am 18.07.2012 10:51, schrieb Wenchao Xia:
>>   Hi, following is API draft, prototypes were taken from qemu/block.h,
>> and the API prefix is changed frpm bdrv to qbdrvs, to declare related
>> object is BlockDriverState, not BlockDriver. [...]

After the refactoring that Markus is working on it won't refer to a
BlockDriverState, but to a BlockBackend. (And changes like this make
quite clear why the internals of the current block layer are not
suitable as a public API. The API needs to be defined in a separate
layer than can abstract such changes away.)

> So let the bikeshedding begin: ;)
> 
> What about qbds_ prefix rather than qbdrvs_? I find the proposed mixture
> of acronym (q for QEMU, b for Block, s for State) and abbreviation (drv
> as in Driver) a bit ugly.
> 
> Or just simply go for qblock - might be better memorable. :)

Yes, something like qblk that isn't tied to internals sounds better.

Kevin

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Daniel P. Berrange]

On Wed, Jul 18, 2012 at 04:51:03PM +0800, Wenchao Xia wrote:
>   Hi, following is API draft, prototypes were taken from qemu/block.h,
> and the API prefix is changed frpm bdrv to qbdrvs, to declare related
> object is BlockDriverState, not BlockDriver. One issue here is it may
> require include block_int.h, which is not LGPL2 licensed yet.
>   API format is kept mostly the same with qemu generic block layer, to
> make it easier for implement, and easier to make qemu migrate on it if
> possible.


How is error reporting dealt with, and what is the intent around
thread safety of the APIs ?  I'd like to see a fully thread safe
API - multiple threads can use the same 'BlockDriverState *'
concurrently, and thread-local error reporting.

> 
> 
> /* structure init and uninit */
> BlockDriverState *qbdrvs_new(const char *device_name);
> void qbdrvs_delete(BlockDriverState *bs);
> 
> 
> /* file open and close */
> int qbdrvs_open(BlockDriverState *bs, const char *filename, int flags,
>               BlockDriver *drv);
> void qbdrvs_close(BlockDriverState *bs);
> int qbdrvs_img_create(const char *filename, const char *fmt,
>                     const char *base_filename, const char *base_fmt,
>                     char *options, uint64_t img_size, int flags);

s/img_create/create/

Can this return an actual BlockDriverState struct too.

> 
> 
> /* sync access */
> int qbdrvs_read(BlockDriverState *bs, int64_t sector_num,
>               uint8_t *buf, int nb_sectors);
> int qbdrvs_write(BlockDriverState *bs, int64_t sector_num,
>                const uint8_t *buf, int nb_sectors);
> 
> 
> /* info retrieve */
> //sector, size and geometry info
> int qbdrvs_get_info(BlockDriverState *bs, BlockDriverInfo *bdi);

What is in BlockDriverInfo and what is the intended ABI stability
policy for it ?

> int64_t qbdrvs_getlength(BlockDriverState *bs);
> int64_t qbdrvs_get_allocated_file_size(BlockDriverState *bs);
> void qbdrvs_get_geometry(BlockDriverState *bs, uint64_t *nb_sectors_ptr);

Could this data all just be part of BlockDriverInfo data ?

> //image type
> const char *qbdrvs_get_format_name(BlockDriverState *bs);
> //backing file info
> void qbdrvs_get_backing_filename(BlockDriverState *bs,
>                                char *filename, int filename_size);

You need to include the backing file format here too.

> void qbdrvs_get_full_backing_filename(BlockDriverState *bs,
>                                     char *dest, size_t sz);

Not sure I see why we need this in addition to the above ?


Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Paolo Bonzini]

Il 18/07/2012 15:58, Daniel P. Berrange ha scritto:
> How is error reporting dealt with

These APIs just return errno values.

> , and what is the intent around
> thread safety of the APIs ?  I'd like to see a fully thread safe
> API - multiple threads can use the same 'BlockDriverState *'
> concurrently, and thread-local error reporting.

This is a bit difficult to provide, since the QEMU block layer itself is
not thread-safe.

Another missing feature is passwords.

Paolo

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Daniel P. Berrange]

On Wed, Jul 18, 2012 at 04:02:15PM +0200, Paolo Bonzini wrote:
> Il 18/07/2012 15:58, Daniel P. Berrange ha scritto:
> > How is error reporting dealt with
> 
> These APIs just return errno values.

Which has led to somewhat unhelpful error reporting in the past. If we're
designing a library API it'd be nice to improve on this.

> 
> > , and what is the intent around
> > thread safety of the APIs ?  I'd like to see a fully thread safe
> > API - multiple threads can use the same 'BlockDriverState *'
> > concurrently, and thread-local error reporting.
> 
> This is a bit difficult to provide, since the QEMU block layer itself is
> not thread-safe.

Yep, I'd expect that this is something we'd need to fix when turning the
code into a library.  NB, I don't mean to say QEMU should protect against
an app doing stupid things like letting 2 threads write to the same area
of the file at once. That's upto the application. I simply mean that the
BlockDriverState shouldn't corrupt itself if 2 separate APIs are called
concurrently on the same instance.

> Another missing feature is passwords.

Oh yes.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Kevin Wolf]

Am 18.07.2012 16:12, schrieb Daniel P. Berrange:
> On Wed, Jul 18, 2012 at 04:02:15PM +0200, Paolo Bonzini wrote:
>> Il 18/07/2012 15:58, Daniel P. Berrange ha scritto:
>>> How is error reporting dealt with
>>
>> These APIs just return errno values.
> 
> Which has led to somewhat unhelpful error reporting in the past. If we're
> designing a library API it'd be nice to improve on this.

But in most cases, errno is what we get from the OS, so we can't do much
more than passing it on. Maybe we can do a bit better with bdrv_open(),
which is relatively likely to fail in qemu rather than in the kernel
because something's wrong with the content of the image.

>>> , and what is the intent around
>>> thread safety of the APIs ?  I'd like to see a fully thread safe
>>> API - multiple threads can use the same 'BlockDriverState *'
>>> concurrently, and thread-local error reporting.
>>
>> This is a bit difficult to provide, since the QEMU block layer itself is
>> not thread-safe.
> 
> Yep, I'd expect that this is something we'd need to fix when turning the
> code into a library.  NB, I don't mean to say QEMU should protect against
> an app doing stupid things like letting 2 threads write to the same area
> of the file at once. That's upto the application. I simply mean that the
> BlockDriverState shouldn't corrupt itself if 2 separate APIs are called
> concurrently on the same instance.

I think it makes sense to make the library thread-safe - and if it only
means taking the global mutex like we do in qemu. I think threading is a
good interface to allow clients to do AIO (even though possibly not the
only one we want to provide), and eventually it will match what qemu is
doing internally.

Kevin

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Kevin Wolf]

Am 18.07.2012 11:03, schrieb Paolo Bonzini:
> Let's get things right, and only have open/close:
> 
> int qbdrvs_open(BlockDriverState **bs, const char *filename, int flags,
>                 const char *format_name);
> void qbdrvs_close(BlockDriverState *bs);

What is flags?

Are we really happy with a function that can't provide the features that
-blockdev will give us?

>> int qbdrvs_img_create(const char *filename, const char *fmt,
>>                     const char *base_filename, const char *base_fmt,
>>                     char *options, uint64_t img_size, int flags);

This prototype is totally wrong. It's already not nice that the
signature of some block layer internal function looks like this.
Basically all of the options need to be replaced by something like a
single QDict.

>> /* sync access */
>> int qbdrvs_read(BlockDriverState *bs, int64_t sector_num,
>>               uint8_t *buf, int nb_sectors);
>> int qbdrvs_write(BlockDriverState *bs, int64_t sector_num,
>>                const uint8_t *buf, int nb_sectors);
> 
> I would like to have also a scatter gather API (qbdrvs_readv and
> qbdrvs_writev) taking a "struct iovec *iov, int niov" instead of
> "uint8_t *buf, int nb_sectors".
> 
> flush is missing.

Yes, both very important.

Kevin

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Daniel P. Berrange]

On Wed, Jul 18, 2012 at 02:58:46PM +0100, Daniel P. Berrange wrote:
> On Wed, Jul 18, 2012 at 04:51:03PM +0800, Wenchao Xia wrote:
> >   Hi, following is API draft, prototypes were taken from qemu/block.h,
> > and the API prefix is changed frpm bdrv to qbdrvs, to declare related
> > object is BlockDriverState, not BlockDriver. One issue here is it may
> > require include block_int.h, which is not LGPL2 licensed yet.
> >   API format is kept mostly the same with qemu generic block layer, to
> > make it easier for implement, and easier to make qemu migrate on it if
> > possible.
> 
> 
> How is error reporting dealt with, and what is the intent around
> thread safety of the APIs ?  I'd like to see a fully thread safe
> API - multiple threads can use the same 'BlockDriverState *'
> concurrently, and thread-local error reporting.

Oh, and will this library depend on glib, and will it have the
abort-on-oom behaviour QEMU has ? From a libvirt POV, we won't
use any library that aborts-on-oom.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Paolo Bonzini]

Il 18/07/2012 17:35, Daniel P. Berrange ha scritto:
> Oh, and will this library depend on glib

Yes, in all likelihood.

, and will it have the
> abort-on-oom behaviour QEMU has ? From a libvirt POV, we won't
> use any library that aborts-on-oom.

Out of curiosity, how do you test OOM?

Paolo

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Daniel P. Berrange]

On Thu, Jul 19, 2012 at 01:37:11PM +0200, Paolo Bonzini wrote:
> Il 18/07/2012 17:35, Daniel P. Berrange ha scritto:
> > Oh, and will this library depend on glib
> 
> Yes, in all likelihood.
> 
> , and will it have the
> > abort-on-oom behaviour QEMU has ? From a libvirt POV, we won't
> > use any library that aborts-on-oom.
> 
> Out of curiosity, how do you test OOM?

There's two aspects to it. First we forbid use of malloc/free/realloc
in favour of an alternative set of APIs designed such that we can get
compile time errors when people don't check the result for NULL.

  http://berrange.com/posts/2008/05/23/better-living-through-api-design-low-level-memory-allocation/

This has been very successful in ensuring code checks for OOM correctly.
Second there is a mode in our test suites where you can run under
simulated OOM - it runs once to count the total set of mallocs in the
test, and then re-run it 'n' times failing each malloc in turn and
verifying that we correctly report the OOM error condition.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Paolo Bonzini]

Il 20/07/2012 13:38, Daniel P. Berrange ha scritto:
> There's two aspects to it. First we forbid use of malloc/free/realloc
> in favour of an alternative set of APIs designed such that we can get
> compile time errors when people don't check the result for NULL.
> 
>   http://berrange.com/posts/2008/05/23/better-living-through-api-design-low-level-memory-allocation/
> 
> This has been very successful in ensuring code checks for OOM correctly.
> Second there is a mode in our test suites where you can run under
> simulated OOM - it runs once to count the total set of mallocs in the
> test, and then re-run it 'n' times failing each malloc in turn and
> verifying that we correctly report the OOM error condition.

Does it fail each malloc in turn, or does it fail each malloc "above X
bytes" after the Y-th?  Because the latter is what you actually get in
practice, and it seriously hampers your recovery capabilities.

Right now, we always use g_malloc and abort on failure, but we could
replace it with g_try_malloc in specific large allocation cases, where
large could even be not that large, for example around 64K.

Paolo

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Lluís Vilanova]

Stefan Hajnoczi writes:
[...]
>> > So I tried trimming down the list of files needed to compile
>> > qemu tools, and here is a list:
>> 
>> > Easy to relicense to LGPLv2+:
>> > block/raw.c                     none (GPLv2+: Red Hat, IBM)
>> > error.c                         LGPLv2 (Red Hat, IBM, Stefan Weil)
>> > iov.c                           GPLv2 (Red Hat, SuSE/Hannes Reinecke, Michael Tokarev)
>> > module.c                        GPLv2 (Red Hat, IBM, Blue Swirl)
>> > qemu-error.c                    GPLv2+ (Red Hat, Blue Swirl, IBM)
>> > trace/control.c                 GPLv2 (Lluis Vilanova)
>> > trace/default.c                 GPLv2 (Lluis Vilanova)
>> 
>> > (I added some people to Cc.  Lluis and Michael, can you also look at
>> > http://wiki.qemu.org/Relicensing if you're willing to relicense
>> > your past contributions from GPLv2 to GPLv2+?.  Blue Swirl said 
>> > he'd accept any other GPLv2 or GPLv3 compatible license, which
>> > should include LGPLv2+).
>> 
>> I have no problems relicensing to "GPLv2 or later" or "GPLv3 or later".

> What about LGPLv2+?  (Note the "L".)

I'd prefer to keep it non-lesser. Is it absolutely necessary?


PS: I suppose the "+" stands for "or later"

Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Paolo Bonzini]

Il 23/07/2012 13:55, Lluís Vilanova ha scritto:
>>> >> I have no problems relicensing to "GPLv2 or later" or "GPLv3 or later".
>> > What about LGPLv2+?  (Note the "L".)
> I'd prefer to keep it non-lesser. Is it absolutely necessary?

This is about making the block layer part of a library.  The block layer
is 90% under a BSD license, but the LGPLv2 is a good compromise.

Paolo

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Blue Swirl]

On Wed, Jul 18, 2012 at 8:51 AM, Wenchao Xia <xiawenc@linux.vnet.ibm.com> wrote:
>   Hi, following is API draft, prototypes were taken from qemu/block.h,
> and the API prefix is changed frpm bdrv to qbdrvs, to declare related
> object is BlockDriverState, not BlockDriver. One issue here is it may
> require include block_int.h, which is not LGPL2 licensed yet.
>   API format is kept mostly the same with qemu generic block layer, to
> make it easier for implement, and easier to make qemu migrate on it if
> possible.
>
>
> /* structure init and uninit */
> BlockDriverState *qbdrvs_new(const char *device_name);
> void qbdrvs_delete(BlockDriverState *bs);
>
>
> /* file open and close */
> int qbdrvs_open(BlockDriverState *bs, const char *filename, int flags,
>               BlockDriver *drv);

How are the errors passed?

Alternative version with file descriptor or struct FILE instead of
filename might become useful but can be added later.

> void qbdrvs_close(BlockDriverState *bs);
> int qbdrvs_img_create(const char *filename, const char *fmt,
>                     const char *base_filename, const char *base_fmt,
>                     char *options, uint64_t img_size, int flags);

'const char *options'

>
>
> /* sync access */
> int qbdrvs_read(BlockDriverState *bs, int64_t sector_num,
>               uint8_t *buf, int nb_sectors);
> int qbdrvs_write(BlockDriverState *bs, int64_t sector_num,
>                const uint8_t *buf, int nb_sectors);

Do we want to use sectors here? How about just raw byte offsets and
number of bytes? I'd leave any hardware details out and just provide
file semantics (open/read/write/close). Future QEMU refactorings could
make supporting HW info inconvenient. If HW details (geometry etc., VM
state) are needed, there should be a separate API.

Vectored I/O might be useful too.

>
>
> /* info retrieve */
> //sector, size and geometry info
> int qbdrvs_get_info(BlockDriverState *bs, BlockDriverInfo *bdi);

Currently BlockDriverInfo does not look very useful for outside users,
with the exception of dirty state. How about accessors instead:
bool qbdrvs_is_dirty(BlockDriverState *bs);

Also the format (QCOW) is needed so that the user can use backing file
functions:
const char *qbdrvs_get_format(BlockDriverState *bs);

> int64_t qbdrvs_getlength(BlockDriverState *bs);
> int64_t qbdrvs_get_allocated_file_size(BlockDriverState *bs);
> void qbdrvs_get_geometry(BlockDriverState *bs, uint64_t *nb_sectors_ptr);

Also this function shouldn't be needed if we don't give HW details
with this API.

> //image type
> const char *qbdrvs_get_format_name(BlockDriverState *bs);
> //backing file info
> void qbdrvs_get_backing_filename(BlockDriverState *bs,
>                                char *filename, int filename_size);
> void qbdrvs_get_full_backing_filename(BlockDriverState *bs,
>                                     char *dest, size_t sz);

These are specific to QCOW etc., so
bool qbdrvs_has_backing_files(BlockDriverState *bs)?

>
>
> /* advanced image content access */
> int qbdrvs_is_allocated(BlockDriverState *bs, int64_t sector_num, int
> nb_sectors,
>                       int *pnum);
> int qbdrvs_discard(BlockDriverState *bs, int64_t sector_num, int
> nb_sectors);

Again, some files do not have a concept of allocation.

> int qbdrvs_has_zero_init(BlockDriverState *bs);
>
>
>> Il 16/07/2012 10:16, Wenchao Xia ha scritto:
>>>>
>>>>
>>>    Really thanks for the investigation, I paid quite sometime to dig out
>>> which license is compatible to LGPL, this have sorted it out.
>>>    The coroutine and structure inside is quite a challenge.
>>
>>
>> Coroutines are really just a small complication in the program flow if
>> all you support is synchronous access to files (i.e. no HTTP etc.).
>> Their usage should be completely transparent.
>>
>>> What about
>>> provide the library first in nbd + sync access, and waiting for the
>>> library employer response? If it is good to use, then replace implement
>>> code to native qemu block layer code, change code's license, while keep
>>> API unchanged.
>>
>>
>> You can start by proposing the API.
>>
>> Paolo
>>
>
>
> --
> Best Regards
>
> Wenchao Xia
>
>
>

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Lluís Vilanova]

Paolo Bonzini writes:

> Il 23/07/2012 13:55, Lluís Vilanova ha scritto:
>>>> >> I have no problems relicensing to "GPLv2 or later" or "GPLv3 or later".
>>> > What about LGPLv2+?  (Note the "L".)
>> I'd prefer to keep it non-lesser. Is it absolutely necessary?

> This is about making the block layer part of a library.  The block layer
> is 90% under a BSD license, but the LGPLv2 is a good compromise.

I understand. I'm not much of an expert on licensing, but if it's necessary to
make the tracing control API LGPLv2+ or LGPLv3+ in order to provide a non-viric
API to the users of the block layer library, I have no problem with it.


Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: [Qemu-devel] [RFC] introduce a dynamic library to expose qemu block API

[Wenchao Xia]

于 2012-7-24 2:15, Blue Swirl 写道:
> On Wed, Jul 18, 2012 at 8:51 AM, Wenchao Xia <xiawenc@linux.vnet.ibm.com> wrote:
>>    Hi, following is API draft, prototypes were taken from qemu/block.h,
>> and the API prefix is changed frpm bdrv to qbdrvs, to declare related
>> object is BlockDriverState, not BlockDriver. One issue here is it may
>> require include block_int.h, which is not LGPL2 licensed yet.
>>    API format is kept mostly the same with qemu generic block layer, to
>> make it easier for implement, and easier to make qemu migrate on it if
>> possible.
>>
>>
>> /* structure init and uninit */
>> BlockDriverState *qbdrvs_new(const char *device_name);
>> void qbdrvs_delete(BlockDriverState *bs);
>>
>>
>> /* file open and close */
>> int qbdrvs_open(BlockDriverState *bs, const char *filename, int flags,
>>                BlockDriver *drv);
>
> How are the errors passed?
>
    Returning negative value should indicate errors. I plan to insert
a qblock-err.h to define all number there, and then a function of
const char *qberr2str(int err) to translate error.

> Alternative version with file descriptor or struct FILE instead of
> filename might become useful but can be added later.
>
   Agree.
>> void qbdrvs_close(BlockDriverState *bs);
>> int qbdrvs_img_create(const char *filename, const char *fmt,
>>                      const char *base_filename, const char *base_fmt,
>>                      char *options, uint64_t img_size, int flags);
>
> 'const char *options'
>
>>
>>
>> /* sync access */
>> int qbdrvs_read(BlockDriverState *bs, int64_t sector_num,
>>                uint8_t *buf, int nb_sectors);
>> int qbdrvs_write(BlockDriverState *bs, int64_t sector_num,
>>                 const uint8_t *buf, int nb_sectors);
>
> Do we want to use sectors here? How about just raw byte offsets and
> number of bytes? I'd leave any hardware details out and just provide
> file semantics (open/read/write/close). Future QEMU refactorings could
> make supporting HW info inconvenient. If HW details (geometry etc., VM
> state) are needed, there should be a separate API.
>
   not sure why original API used sector instead of  raw address,
if no special requirement for sector I think raw address is better.

> Vectored I/O might be useful too.
>
>>
>>
>> /* info retrieve */
>> //sector, size and geometry info
>> int qbdrvs_get_info(BlockDriverState *bs, BlockDriverInfo *bdi);
>
> Currently BlockDriverInfo does not look very useful for outside users,
> with the exception of dirty state. How about accessors instead:
> bool qbdrvs_is_dirty(BlockDriverState *bs);
>
> Also the format (QCOW) is needed so that the user can use backing file
> functions:
> const char *qbdrvs_get_format(BlockDriverState *bs);
>
   I hope to package the info into a structure, such as
#define QBLOCK_INFO_SIZE (128)
struct QblockInfo {
     int dirty;
     char *format;
     int reserved[RESERVE_SPACE];
};
assert sizeof(QblockInfo) == QBLOCK_INFO_SIZE in every version. This
will make caller's code more clean.

>> int64_t qbdrvs_getlength(BlockDriverState *bs);
>> int64_t qbdrvs_get_allocated_file_size(BlockDriverState *bs);
>> void qbdrvs_get_geometry(BlockDriverState *bs, uint64_t *nb_sectors_ptr);
>
> Also this function shouldn't be needed if we don't give HW details
> with this API.
>
     OK.

>> //image type
>> const char *qbdrvs_get_format_name(BlockDriverState *bs);
>> //backing file info
>> void qbdrvs_get_backing_filename(BlockDriverState *bs,
>>                                 char *filename, int filename_size);
>> void qbdrvs_get_full_backing_filename(BlockDriverState *bs,
>>                                      char *dest, size_t sz);
>
> These are specific to QCOW etc., so
> bool qbdrvs_has_backing_files(BlockDriverState *bs)?
>
   How about using return value to indicate it?

int qbdrvs_get_backing_filename(BlockDriverState *bs, char *filename, 
int filename_size);

>>
>>
>> /* advanced image content access */
>> int qbdrvs_is_allocated(BlockDriverState *bs, int64_t sector_num, int
>> nb_sectors,
>>                        int *pnum);
>> int qbdrvs_discard(BlockDriverState *bs, int64_t sector_num, int
>> nb_sectors);
>
> Again, some files do not have a concept of allocation.
>
   Maybe return value could indicate it.

>> int qbdrvs_has_zero_init(BlockDriverState *bs);
>>
>>
>>> Il 16/07/2012 10:16, Wenchao Xia ha scritto:
>>>>>
>>>>>
>>>>     Really thanks for the investigation, I paid quite sometime to dig out
>>>> which license is compatible to LGPL, this have sorted it out.
>>>>     The coroutine and structure inside is quite a challenge.
>>>
>>>
>>> Coroutines are really just a small complication in the program flow if
>>> all you support is synchronous access to files (i.e. no HTTP etc.).
>>> Their usage should be completely transparent.
>>>
>>>> What about
>>>> provide the library first in nbd + sync access, and waiting for the
>>>> library employer response? If it is good to use, then replace implement
>>>> code to native qemu block layer code, change code's license, while keep
>>>> API unchanged.
>>>
>>>
>>> You can start by proposing the API.
>>>
>>> Paolo
>>>
>>
>>
>> --
>> Best Regards
>>
>> Wenchao Xia
>>
>>
>>
>


-- 
Best Regards

Wenchao Xia