[Qemu-devel] QAPI conversion status and async commands support

[Qemu-devel] QAPI conversion status and async commands support

[Luiz Capitulino]

Hi there,

In the last few weeks we've had some proposals for new QMP commands that need
to be asynchronous. As we lack a standard asynchronous API today, each command
ends up adding its own way to execute in the background.

This multiplies the API complexity as each command has to be implemented and
learned by clients separately, with their own way of doing more or less the
same things.

The solution for this, envisioned for us for a long time now, is to introduce
an unified QMP API for asynchronous commands.

But before doing this we have to:

  1. Finish the commands conversion to the QAPI

     This is almost done, the only missing commands are: add_graphics_client,
     do_closefd, do_device_add, do_device_del, do_getfd, do_migrate,
     do_netdev_add, do_netdev_del, do_qmp_capabilities and do_screen_dump.

     Note that do_migrate has already been posted to the list, and I have
     the screendump more or less done. Also, Anthony has an old branch where most
     of the conversions are already done, they just need to be rebased & tested.

  2. Integrate the new QAPI server

     Implemented by Anthony, may have missing pieces.

  3. Implement async command support


I think the missing commands to be converted can be done in around one week,
but unfortunately I've been busy at other things and will need a few days to
resume this work. Then there's the new QAPI server & async support, which I'm
not sure how much time we'll need to integrate them, but we should have this
done for 1.1.

The main question is: what should we do for the already posted async commands?
Should we hold them until we finish this work?

Re: [Qemu-devel] QAPI conversion status and async commands support

[Paolo Bonzini]

Il 07/03/2012 17:36, Luiz Capitulino ha scritto:
> Hi there,
> 
> In the last few weeks we've had some proposals for new QMP commands that need
> to be asynchronous. As we lack a standard asynchronous API today, each command
> ends up adding its own way to execute in the background.
> 
> This multiplies the API complexity as each command has to be implemented and
> learned by clients separately, with their own way of doing more or less the
> same things.
> 
> The solution for this, envisioned for us for a long time now, is to introduce
> an unified QMP API for asynchronous commands.
> 
> But before doing this we have to:
> 
>   1. Finish the commands conversion to the QAPI
> 
>      This is almost done, the only missing commands are: add_graphics_client,
>      do_closefd, do_device_add, do_device_del, do_getfd, do_migrate,
>      do_netdev_add, do_netdev_del, do_qmp_capabilities and do_screen_dump.
> 
>      Note that do_migrate has already been posted to the list, and I have
>      the screendump more or less done. Also, Anthony has an old branch where most
>      of the conversions are already done, they just need to be rebased & tested.
> 
>   2. Integrate the new QAPI server
> 
>      Implemented by Anthony, may have missing pieces.
> 
>   3. Implement async command support
> 
> 
> I think the missing commands to be converted can be done in around one week,
> but unfortunately I've been busy at other things and will need a few days to
> resume this work. Then there's the new QAPI server & async support, which I'm
> not sure how much time we'll need to integrate them, but we should have this
> done for 1.1.
> 
> The main question is: what should we do for the already posted async commands?
> Should we hold them until we finish this work?

I think yes, and we could even have a list of features without which 1.1
should not ship.  QOM buses, drive mirroring and QAPI async command
support may be them.  Perhaps qtest too.

Paolo

Re: [Qemu-devel] QAPI conversion status and async commands support

[Anthony Liguori]

On 03/07/2012 11:29 AM, Paolo Bonzini wrote:
> Il 07/03/2012 17:36, Luiz Capitulino ha scritto:
>> Hi there,
>>
>> In the last few weeks we've had some proposals for new QMP commands that need
>> to be asynchronous. As we lack a standard asynchronous API today, each command
>> ends up adding its own way to execute in the background.
>>
>> This multiplies the API complexity as each command has to be implemented and
>> learned by clients separately, with their own way of doing more or less the
>> same things.
>>
>> The solution for this, envisioned for us for a long time now, is to introduce
>> an unified QMP API for asynchronous commands.
>>
>> But before doing this we have to:
>>
>>    1. Finish the commands conversion to the QAPI
>>
>>       This is almost done, the only missing commands are: add_graphics_client,
>>       do_closefd, do_device_add, do_device_del, do_getfd, do_migrate,
>>       do_netdev_add, do_netdev_del, do_qmp_capabilities and do_screen_dump.
>>
>>       Note that do_migrate has already been posted to the list, and I have
>>       the screendump more or less done. Also, Anthony has an old branch where most
>>       of the conversions are already done, they just need to be rebased&  tested.
>>
>>    2. Integrate the new QAPI server
>>
>>       Implemented by Anthony, may have missing pieces.
>>
>>    3. Implement async command support
>>
>>
>> I think the missing commands to be converted can be done in around one week,
>> but unfortunately I've been busy at other things and will need a few days to
>> resume this work. Then there's the new QAPI server&  async support, which I'm
>> not sure how much time we'll need to integrate them, but we should have this
>> done for 1.1.
>>
>> The main question is: what should we do for the already posted async commands?
>> Should we hold them until we finish this work?
>
> I think yes, and we could even have a list of features without which 1.1
> should not ship.  QOM buses, drive mirroring and QAPI async command
> support may be them.  Perhaps qtest too.

Okay, let's get serious about what we can and can't do.

Hard freeze for 1.1 is May 1st which is roughly 6 weeks from now.

I think QOM buses can go in no problem along with qtest.  I would be okay 
considering QOM buses a release blocker but probably not qtest.

I'm not really sure about drive mirroring.  Is the work already done such that 
we just need to talk about merging it?

With QAPI async command, I don't think 1.1 is a viable target.  We're not just 
talking about converting existing commands to QAPI, but also replacing the QMP 
server infrastructure.  I don't think that is a change that should be made at 
the tail end of the development cycle.

Regards,

Anthony Liguori

>
> Paolo
>
>
>

Re: [Qemu-devel] QAPI conversion status and async commands support

[Luiz Capitulino]

On Wed, 07 Mar 2012 11:36:06 -0600
Anthony Liguori <anthony@codemonkey.ws> wrote:

> On 03/07/2012 11:29 AM, Paolo Bonzini wrote:
> > Il 07/03/2012 17:36, Luiz Capitulino ha scritto:
> >> Hi there,
> >>
> >> In the last few weeks we've had some proposals for new QMP commands that need
> >> to be asynchronous. As we lack a standard asynchronous API today, each command
> >> ends up adding its own way to execute in the background.
> >>
> >> This multiplies the API complexity as each command has to be implemented and
> >> learned by clients separately, with their own way of doing more or less the
> >> same things.
> >>
> >> The solution for this, envisioned for us for a long time now, is to introduce
> >> an unified QMP API for asynchronous commands.
> >>
> >> But before doing this we have to:
> >>
> >>    1. Finish the commands conversion to the QAPI
> >>
> >>       This is almost done, the only missing commands are: add_graphics_client,
> >>       do_closefd, do_device_add, do_device_del, do_getfd, do_migrate,
> >>       do_netdev_add, do_netdev_del, do_qmp_capabilities and do_screen_dump.
> >>
> >>       Note that do_migrate has already been posted to the list, and I have
> >>       the screendump more or less done. Also, Anthony has an old branch where most
> >>       of the conversions are already done, they just need to be rebased&  tested.
> >>
> >>    2. Integrate the new QAPI server
> >>
> >>       Implemented by Anthony, may have missing pieces.
> >>
> >>    3. Implement async command support
> >>
> >>
> >> I think the missing commands to be converted can be done in around one week,
> >> but unfortunately I've been busy at other things and will need a few days to
> >> resume this work. Then there's the new QAPI server&  async support, which I'm
> >> not sure how much time we'll need to integrate them, but we should have this
> >> done for 1.1.
> >>
> >> The main question is: what should we do for the already posted async commands?
> >> Should we hold them until we finish this work?
> >
> > I think yes, and we could even have a list of features without which 1.1
> > should not ship.  QOM buses, drive mirroring and QAPI async command
> > support may be them.  Perhaps qtest too.
> 
> Okay, let's get serious about what we can and can't do.
> 
> Hard freeze for 1.1 is May 1st which is roughly 6 weeks from now.
> 
> I think QOM buses can go in no problem along with qtest.  I would be okay 
> considering QOM buses a release blocker but probably not qtest.
> 
> I'm not really sure about drive mirroring.  Is the work already done such that 
> we just need to talk about merging it?
> 
> With QAPI async command, I don't think 1.1 is a viable target.  We're not just 
> talking about converting existing commands to QAPI, but also replacing the QMP 
> server infrastructure.  I don't think that is a change that should be made at 
> the tail end of the development cycle.

I will have to agree with this, specially because I think the new server is
going to generate some fun discussions because iirc it has some novel
concepts. And those discussions burn a lot of time.

On top of that we also have the async API to design. I have some ideas in mind,
but again, design & review cycles take some time.

Having said that, I wonder if postponing want-to-be-async commands to 1.2 is
a good thing. Two solutions other than waiting:

 1. Add synchronous versions of them, if possible (I think this is possible
    for the dump command)

 2. Accept the ad-hoc async mechanism they introduce

Btw, probably needless to say but, I'd greatly appreciate and encourage anyone who
wants to join this effort and work on the conversions, new QMP server, etc...

Re: [Qemu-devel] QAPI conversion status and async commands support

[Alon Levy]

On Wed, Mar 07, 2012 at 11:36:06AM -0600, Anthony Liguori wrote:
> On 03/07/2012 11:29 AM, Paolo Bonzini wrote:
> >Il 07/03/2012 17:36, Luiz Capitulino ha scritto:
> >>Hi there,
> >>
> >>In the last few weeks we've had some proposals for new QMP commands that need
> >>to be asynchronous. As we lack a standard asynchronous API today, each command
> >>ends up adding its own way to execute in the background.
> >>
> >>This multiplies the API complexity as each command has to be implemented and
> >>learned by clients separately, with their own way of doing more or less the
> >>same things.
> >>
> >>The solution for this, envisioned for us for a long time now, is to introduce
> >>an unified QMP API for asynchronous commands.
> >>
> >>But before doing this we have to:
> >>
> >>   1. Finish the commands conversion to the QAPI
> >>
> >>      This is almost done, the only missing commands are: add_graphics_client,
> >>      do_closefd, do_device_add, do_device_del, do_getfd, do_migrate,
> >>      do_netdev_add, do_netdev_del, do_qmp_capabilities and do_screen_dump.
> >>
> >>      Note that do_migrate has already been posted to the list, and I have
> >>      the screendump more or less done. Also, Anthony has an old branch where most
> >>      of the conversions are already done, they just need to be rebased&  tested.
> >>
> >>   2. Integrate the new QAPI server
> >>
> >>      Implemented by Anthony, may have missing pieces.
> >>
> >>   3. Implement async command support
> >>
> >>
> >>I think the missing commands to be converted can be done in around one week,
> >>but unfortunately I've been busy at other things and will need a few days to
> >>resume this work. Then there's the new QAPI server&  async support, which I'm
> >>not sure how much time we'll need to integrate them, but we should have this
> >>done for 1.1.
> >>
> >>The main question is: what should we do for the already posted async commands?
> >>Should we hold them until we finish this work?
> >
> >I think yes, and we could even have a list of features without which 1.1
> >should not ship.  QOM buses, drive mirroring and QAPI async command
> >support may be them.  Perhaps qtest too.
> 
> Okay, let's get serious about what we can and can't do.
> 
> Hard freeze for 1.1 is May 1st which is roughly 6 weeks from now.
> 
> I think QOM buses can go in no problem along with qtest.  I would be
> okay considering QOM buses a release blocker but probably not qtest.
> 
> I'm not really sure about drive mirroring.  Is the work already done
> such that we just need to talk about merging it?
> 
> With QAPI async command, I don't think 1.1 is a viable target.
> We're not just talking about converting existing commands to QAPI,
> but also replacing the QMP server infrastructure.  I don't think
> that is a change that should be made at the tail end of the
> development cycle.

... so, Avi, Anthony, as:
1. screendump is broken for qxl
2. there will not be QAPI async for 1.1
3. monitor async is unacceptable by maintainer

Is screendump-async to be accepted?

> 
> Regards,
> 
> Anthony Liguori
> 
> >
> >Paolo
> >
> >
> >
> 
> 

Re: [Qemu-devel] QAPI conversion status and async commands support

[Kevin Wolf]

Am 07.03.2012 18:36, schrieb Anthony Liguori:
> On 03/07/2012 11:29 AM, Paolo Bonzini wrote:
>> Il 07/03/2012 17:36, Luiz Capitulino ha scritto:
>>> Hi there,
>>>
>>> In the last few weeks we've had some proposals for new QMP commands that need
>>> to be asynchronous. As we lack a standard asynchronous API today, each command
>>> ends up adding its own way to execute in the background.
>>>
>>> This multiplies the API complexity as each command has to be implemented and
>>> learned by clients separately, with their own way of doing more or less the
>>> same things.
>>>
>>> The solution for this, envisioned for us for a long time now, is to introduce
>>> an unified QMP API for asynchronous commands.
>>>
>>> But before doing this we have to:
>>>
>>>    1. Finish the commands conversion to the QAPI
>>>
>>>       This is almost done, the only missing commands are: add_graphics_client,
>>>       do_closefd, do_device_add, do_device_del, do_getfd, do_migrate,
>>>       do_netdev_add, do_netdev_del, do_qmp_capabilities and do_screen_dump.
>>>
>>>       Note that do_migrate has already been posted to the list, and I have
>>>       the screendump more or less done. Also, Anthony has an old branch where most
>>>       of the conversions are already done, they just need to be rebased&  tested.
>>>
>>>    2. Integrate the new QAPI server
>>>
>>>       Implemented by Anthony, may have missing pieces.
>>>
>>>    3. Implement async command support
>>>
>>>
>>> I think the missing commands to be converted can be done in around one week,
>>> but unfortunately I've been busy at other things and will need a few days to
>>> resume this work. Then there's the new QAPI server&  async support, which I'm
>>> not sure how much time we'll need to integrate them, but we should have this
>>> done for 1.1.
>>>
>>> The main question is: what should we do for the already posted async commands?
>>> Should we hold them until we finish this work?
>>
>> I think yes, and we could even have a list of features without which 1.1
>> should not ship.  QOM buses, drive mirroring and QAPI async command
>> support may be them.  Perhaps qtest too.
> 
> Okay, let's get serious about what we can and can't do.
> 
> Hard freeze for 1.1 is May 1st which is roughly 6 weeks from now.
> 
> I think QOM buses can go in no problem along with qtest.  I would be okay 
> considering QOM buses a release blocker but probably not qtest.
> 
> I'm not really sure about drive mirroring.  Is the work already done such that 
> we just need to talk about merging it?

There are patches, but they still need review. I think it's doable for
1.1. But in any case I don't think there's any justification for it to
be a release blocker. If anything in the block layer should be one, I'd
consider basic qcow3 support closest.

Kevin

Re: [Qemu-devel] QAPI conversion status and async commands support

[Paolo Bonzini]

Il 08/03/2012 13:34, Kevin Wolf ha scritto:
> > I'm not really sure about drive mirroring.  Is the work already done such that 
> > we just need to talk about merging it?
> There are patches, but they still need review. I think it's doable for
> 1.1. But in any case I don't think there's any justification for it to
> be a release blocker. 

However, defining the API for group snapshots should be a release blocker.

Paolo

Re: [Qemu-devel] QAPI conversion status and async commands support

[Kevin Wolf]

Am 08.03.2012 15:12, schrieb Paolo Bonzini:
> Il 08/03/2012 13:34, Kevin Wolf ha scritto:
>>> I'm not really sure about drive mirroring.  Is the work already done such that 
>>> we just need to talk about merging it?
>> There are patches, but they still need review. I think it's doable for
>> 1.1. But in any case I don't think there's any justification for it to
>> be a release blocker. 
> 
> However, defining the API for group snapshots should be a release blocker.

Agreed. And I think the design problem for this one is solved, so we
should easily be able to make it until May 1.

Kevin

Re: [Qemu-devel] QAPI conversion status and async commands support

[Alon Levy]

On Wed, Mar 07, 2012 at 10:06:31PM +0200, Alon Levy wrote:
> On Wed, Mar 07, 2012 at 11:36:06AM -0600, Anthony Liguori wrote:
> > On 03/07/2012 11:29 AM, Paolo Bonzini wrote:
> > >Il 07/03/2012 17:36, Luiz Capitulino ha scritto:
> > >>Hi there,
> > >>
> > >>In the last few weeks we've had some proposals for new QMP commands that need
> > >>to be asynchronous. As we lack a standard asynchronous API today, each command
> > >>ends up adding its own way to execute in the background.
> > >>
> > >>This multiplies the API complexity as each command has to be implemented and
> > >>learned by clients separately, with their own way of doing more or less the
> > >>same things.
> > >>
> > >>The solution for this, envisioned for us for a long time now, is to introduce
> > >>an unified QMP API for asynchronous commands.
> > >>
> > >>But before doing this we have to:
> > >>
> > >>   1. Finish the commands conversion to the QAPI
> > >>
> > >>      This is almost done, the only missing commands are: add_graphics_client,
> > >>      do_closefd, do_device_add, do_device_del, do_getfd, do_migrate,
> > >>      do_netdev_add, do_netdev_del, do_qmp_capabilities and do_screen_dump.
> > >>
> > >>      Note that do_migrate has already been posted to the list, and I have
> > >>      the screendump more or less done. Also, Anthony has an old branch where most
> > >>      of the conversions are already done, they just need to be rebased&  tested.
> > >>
> > >>   2. Integrate the new QAPI server
> > >>
> > >>      Implemented by Anthony, may have missing pieces.
> > >>
> > >>   3. Implement async command support
> > >>
> > >>
> > >>I think the missing commands to be converted can be done in around one week,
> > >>but unfortunately I've been busy at other things and will need a few days to
> > >>resume this work. Then there's the new QAPI server&  async support, which I'm
> > >>not sure how much time we'll need to integrate them, but we should have this
> > >>done for 1.1.
> > >>
> > >>The main question is: what should we do for the already posted async commands?
> > >>Should we hold them until we finish this work?
> > >
> > >I think yes, and we could even have a list of features without which 1.1
> > >should not ship.  QOM buses, drive mirroring and QAPI async command
> > >support may be them.  Perhaps qtest too.
> > 
> > Okay, let's get serious about what we can and can't do.
> > 
> > Hard freeze for 1.1 is May 1st which is roughly 6 weeks from now.
> > 
> > I think QOM buses can go in no problem along with qtest.  I would be
> > okay considering QOM buses a release blocker but probably not qtest.
> > 
> > I'm not really sure about drive mirroring.  Is the work already done
> > such that we just need to talk about merging it?
> > 
> > With QAPI async command, I don't think 1.1 is a viable target.
> > We're not just talking about converting existing commands to QAPI,
> > but also replacing the QMP server infrastructure.  I don't think
> > that is a change that should be made at the tail end of the
> > development cycle.
> 
> ... so, Avi, Anthony, as:
> 1. screendump is broken for qxl
> 2. there will not be QAPI async for 1.1
> 3. monitor async is unacceptable by maintainer
> 
> Is screendump-async to be accepted?

I've send a patchset that works for hmp and I intend to fix libvirt by
using the hmp screendump command instead of the qmp one for now. See:
 http://lists.gnu.org/archive/html/qemu-devel/2012-03/msg01802.html

> 
> > 
> > Regards,
> > 
> > Anthony Liguori
> > 
> > >
> > >Paolo
> > >
> > >
> > >
> > 
> > 
> 

Re: [Qemu-devel] QAPI conversion status and async commands support

[Wen Congyang]

At 03/08/2012 02:12 AM, Luiz Capitulino Wrote:
> On Wed, 07 Mar 2012 11:36:06 -0600
> Anthony Liguori <anthony@codemonkey.ws> wrote:
> 
>> On 03/07/2012 11:29 AM, Paolo Bonzini wrote:
>>> Il 07/03/2012 17:36, Luiz Capitulino ha scritto:
>>>> Hi there,
>>>>
>>>> In the last few weeks we've had some proposals for new QMP commands that need
>>>> to be asynchronous. As we lack a standard asynchronous API today, each command
>>>> ends up adding its own way to execute in the background.
>>>>
>>>> This multiplies the API complexity as each command has to be implemented and
>>>> learned by clients separately, with their own way of doing more or less the
>>>> same things.
>>>>
>>>> The solution for this, envisioned for us for a long time now, is to introduce
>>>> an unified QMP API for asynchronous commands.
>>>>
>>>> But before doing this we have to:
>>>>
>>>>    1. Finish the commands conversion to the QAPI
>>>>
>>>>       This is almost done, the only missing commands are: add_graphics_client,
>>>>       do_closefd, do_device_add, do_device_del, do_getfd, do_migrate,
>>>>       do_netdev_add, do_netdev_del, do_qmp_capabilities and do_screen_dump.
>>>>
>>>>       Note that do_migrate has already been posted to the list, and I have
>>>>       the screendump more or less done. Also, Anthony has an old branch where most
>>>>       of the conversions are already done, they just need to be rebased&  tested.
>>>>
>>>>    2. Integrate the new QAPI server
>>>>
>>>>       Implemented by Anthony, may have missing pieces.
>>>>
>>>>    3. Implement async command support
>>>>
>>>>
>>>> I think the missing commands to be converted can be done in around one week,
>>>> but unfortunately I've been busy at other things and will need a few days to
>>>> resume this work. Then there's the new QAPI server&  async support, which I'm
>>>> not sure how much time we'll need to integrate them, but we should have this
>>>> done for 1.1.
>>>>
>>>> The main question is: what should we do for the already posted async commands?
>>>> Should we hold them until we finish this work?
>>>
>>> I think yes, and we could even have a list of features without which 1.1
>>> should not ship.  QOM buses, drive mirroring and QAPI async command
>>> support may be them.  Perhaps qtest too.
>>
>> Okay, let's get serious about what we can and can't do.
>>
>> Hard freeze for 1.1 is May 1st which is roughly 6 weeks from now.
>>
>> I think QOM buses can go in no problem along with qtest.  I would be okay 
>> considering QOM buses a release blocker but probably not qtest.
>>
>> I'm not really sure about drive mirroring.  Is the work already done such that 
>> we just need to talk about merging it?
>>
>> With QAPI async command, I don't think 1.1 is a viable target.  We're not just 
>> talking about converting existing commands to QAPI, but also replacing the QMP 
>> server infrastructure.  I don't think that is a change that should be made at 
>> the tail end of the development cycle.
> 
> I will have to agree with this, specially because I think the new server is
> going to generate some fun discussions because iirc it has some novel
> concepts. And those discussions burn a lot of time.
> 
> On top of that we also have the async API to design. I have some ideas in mind,
> but again, design & review cycles take some time.
> 
> Having said that, I wonder if postponing want-to-be-async commands to 1.2 is
> a good thing. Two solutions other than waiting:
> 
>  1. Add synchronous versions of them, if possible (I think this is possible
>     for the dump command)

I agree with this. libvirt uses migrate command to do dump now, and it cannot work
when the guest uses host's device. We need to fix this bug as quickly as possible.
So we need dump command recently. So I think we can add synchronous versions of
dump command. Is it OK? If it is OK, I will send the synchronous versions of
dump command.

Thanks
Wen Congyang

> 
>  2. Accept the ad-hoc async mechanism they introduce
> 
> Btw, probably needless to say but, I'd greatly appreciate and encourage anyone who
> wants to join this effort and work on the conversions, new QMP server, etc...
> 
> 

Re: [Qemu-devel] QAPI conversion status and async commands support

[Luiz Capitulino]

On Mon, 12 Mar 2012 16:43:22 +0800
Wen Congyang <wency@cn.fujitsu.com> wrote:

> > Having said that, I wonder if postponing want-to-be-async commands to 1.2 is
> > a good thing. Two solutions other than waiting:
> > 
> >  1. Add synchronous versions of them, if possible (I think this is possible
> >     for the dump command)
> 
> I agree with this. libvirt uses migrate command to do dump now, and it cannot work
> when the guest uses host's device. We need to fix this bug as quickly as possible.
> So we need dump command recently. So I think we can add synchronous versions of
> dump command. Is it OK? If it is OK, I will send the synchronous versions of
> dump command.

I'd be fine with this because this command is expected to be used when the guest
crashes, right? In this case, I _guess_ that holding the global mutex for a long
time is not the end of the world, as the world has ended already :) Far from ideal,
of course, but is the best we can do right now.

Anthony, do you have objections?