[Qemu-devel] [RFC] libqblock OOM issue

[Qemu-devel] [RFC] libqblock OOM issue

[Wenchao Xia]

Hello,Paolo and Stefanha
  In order to resolve OOM issue, I am trying wrap all APIs using
sunrpc, need some suggestion before coding.

  There are some different way to implement, not sure which would be
better:
  1 keep client as thin as possible, client stores opaque pointer used
in server side, for eg, QBlockContext *ctx, client only get a pointer
pointing to the address where server stores really the object. This
have risk when server/client crash and reconnect.
  2 client and server maintains index for QBlockContext and QBlockState.
  3 thick client and server layer, expose all structure details in .x
file, each API have a correspond rpc call. .x file may be complex.
  4 define a custom protocol on XDR, like libvirt, this may need many
code in server/client side.

  also with method 1-3, Consider wrapping following API:
int qb_context_new(QBlockContext **context);

  The parameter context is a pointer that will be modified, it seems
sunrpc does not transfer back modified parameter by server to client, so
I need to define a structure as
struct qb_context_new_ret {
  int ret;
  int opaque_len;
  char *opaque_val;
}
and use that as rpc call's return structure. In this way each API
wrapped need a new defined internal structure make things complicate.
so I am wondering if there is a better way to do it.

Re: [Qemu-devel] [RFC] libqblock OOM issue

[Stefan Hajnoczi]

On Wed, Nov 14, 2012 at 11:50:06AM +0800, Wenchao Xia wrote:
>   In order to resolve OOM issue, I am trying wrap all APIs using
> sunrpc, need some suggestion before coding.

Is the client/server approach really necessary or can you write a
library that invokes qemu-nbd/qemu-img?

If there is a startup cost problem with qemu-img it may be possible to
add an interactive mode (like qemu-io) where qemu-img stays open and
responds to commands (maybe in JSON encoding).

The difference between this and the RPC approach is that you can write a
relatively thin NBD and qemu-img library with the tools that already
exist today.

Stefan

Re: [Qemu-devel] [RFC] libqblock OOM issue

[Paolo Bonzini]

Il 14/11/2012 04:50, Wenchao Xia ha scritto:
>   There are some different way to implement, not sure which would be
> better:
>   1 keep client as thin as possible, client stores opaque pointer used
> in server side, for eg, QBlockContext *ctx, client only get a pointer
> pointing to the address where server stores really the object. This
> have risk when server/client crash and reconnect.
>   2 client and server maintains index for QBlockContext and QBlockState.
>   3 thick client and server layer, expose all structure details in .x
> file, each API have a correspond rpc call. .x file may be complex.
>   4 define a custom protocol on XDR, like libvirt, this may need many
> code in server/client side.
> 
>   also with method 1-3, Consider wrapping following API:
> int qb_context_new(QBlockContext **context);

What is the return value of qb_context_new?  Can it simply return
QBlockContext*?

>   The parameter context is a pointer that will be modified, it seems
> sunrpc does not transfer back modified parameter by server to client, so
> I need to define a structure as
> struct qb_context_new_ret {
>   int ret;
>   int opaque_len;
>   char *opaque_val;
> }
> and use that as rpc call's return structure. In this way each API
> wrapped need a new defined internal structure make things complicate.
> so I am wondering if there is a better way to do it.

Surely not all of the APIs return structs this way, however...

Paolo

Re: [Qemu-devel] [RFC] libqblock OOM issue

[Wenchao Xia]

 > On Wed, Nov 14, 2012 at 11:50:06AM +0800, Wenchao Xia wrote:
>>    In order to resolve OOM issue, I am trying wrap all APIs using
>> sunrpc, need some suggestion before coding.
>
> Is the client/server approach really necessary or can you write a
> library that invokes qemu-nbd/qemu-img?
>
> If there is a startup cost problem with qemu-img it may be possible to
> add an interactive mode (like qemu-io) where qemu-img stays open and
> responds to commands (maybe in JSON encoding).
>
> The difference between this and the RPC approach is that you can write a
> relatively thin NBD and qemu-img library with the tools that already
> exist today.
>
> Stefan
>

   I understand your reason about using qemu-img and qemu-nbd and
appreciate for the suggestion. Currently libqblock already have the
capabilities of r/w images as qemu-nbd, and have info retrieving
capabilities as qemu-img. So the difference will be:
   using qemu-img qemu-nbd, it needs nbd client/json parser code, 
modification for qemu-img is needed for each new function added in
libqblock.
   using rpc plus native lib, it needs good framework to transparently
pass the information to client side for every API.
   IMHO, considering now libqblock have all the capabilities required
except OOM, it is not hard to get same effect as using qemu-img and
qemu-nbd, by wrapping read/write/info retrieve APIs if I keep the
file stays open in server side and make server/client not mirrored.
How to write a good enough framework which can wrap different API in an
unified style, and keep this layer as simple, as transparent as
possible, whether I should break the mirror between client/server, are
some question I am not quite sure.



-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] [RFC] libqblock OOM issue

[Wenchao Xia]

> Il 14/11/2012 04:50, Wenchao Xia ha scritto:
>>    There are some different way to implement, not sure which would be
>> better:
>>    1 keep client as thin as possible, client stores opaque pointer used
>> in server side, for eg, QBlockContext *ctx, client only get a pointer
>> pointing to the address where server stores really the object. This
>> have risk when server/client crash and reconnect.
>>    2 client and server maintains index for QBlockContext and QBlockState.
>>    3 thick client and server layer, expose all structure details in .x
>> file, each API have a correspond rpc call. .x file may be complex.
>>    4 define a custom protocol on XDR, like libvirt, this may need many
>> code in server/client side.
>>
>>    also with method 1-3, Consider wrapping following API:
>> int qb_context_new(QBlockContext **context);
> 
> What is the return value of qb_context_new?  Can it simply return
> QBlockContext*?
> 
  Yes it can return QBlockContext*. There are more APIs take 3 or 4
parameters, which may be used to retrieve result. In that case I am
afraid a return structure can't be avoided, this may result .x file
looks strange.

>>    The parameter context is a pointer that will be modified, it seems
>> sunrpc does not transfer back modified parameter by server to client, so
>> I need to define a structure as
>> struct qb_context_new_ret {
>>    int ret;
>>    int opaque_len;
>>    char *opaque_val;
>> }
>> and use that as rpc call's return structure. In this way each API
>> wrapped need a new defined internal structure make things complicate.
>> so I am wondering if there is a better way to do it.
> 
> Surely not all of the APIs return structs this way, however...
> 
> Paolo
> 


-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] [RFC] libqblock OOM issue

[Paolo Bonzini]

Il 14/11/2012 10:55, Wenchao Xia ha scritto:
>>    In order to resolve OOM issue, I am trying wrap all APIs using
>> sunrpc, need some suggestion before coding.
> 
> Is the client/server approach really necessary or can you write a
> library that invokes qemu-nbd/qemu-img?
> 
> If there is a startup cost problem with qemu-img it may be possible to
> add an interactive mode (like qemu-io) where qemu-img stays open and
> responds to commands (maybe in JSON encoding).
> 
> The difference between this and the RPC approach is that you can write a
> relatively thin NBD and qemu-img library with the tools that already
> exist today.

In fact, I think this is not our issue.  If libvirt wants to use
libqblock but have a problem with OOM exit, they can write their own
wrappers to do the simple tasks they need, or just keep on using
qemu-img with JSON output (possibly extending it and keeping the
functionality upstream).  For many of those tasks, it may turn out that
qemu-img extensions would be useful anyway.

Paolo

Re: [Qemu-devel] [RFC] libqblock OOM issue

[Wenchao Xia]

于 2012-11-14 18:32, Paolo Bonzini 写道:
> Il 14/11/2012 10:55, Wenchao Xia ha scritto:
>>>     In order to resolve OOM issue, I am trying wrap all APIs using
>>> sunrpc, need some suggestion before coding.
>>
>> Is the client/server approach really necessary or can you write a
>> library that invokes qemu-nbd/qemu-img?
>>
>> If there is a startup cost problem with qemu-img it may be possible to
>> add an interactive mode (like qemu-io) where qemu-img stays open and
>> responds to commands (maybe in JSON encoding).
>>
>> The difference between this and the RPC approach is that you can write a
>> relatively thin NBD and qemu-img library with the tools that already
>> exist today.
>
> In fact, I think this is not our issue.  If libvirt wants to use
> libqblock but have a problem with OOM exit, they can write their own
> wrappers to do the simple tasks they need, or just keep on using
> qemu-img with JSON output (possibly extending it and keeping the
> functionality upstream).  For many of those tasks, it may turn out that
> qemu-img extensions would be useful anyway.
>
> Paolo
>
   Personally agree, but I want to add a simple wrapper to let libqblock
get user faster. In this way I guess best choice now is making rpc
client and server not mirrored in implemention, server provides
r/w/info retrieving capabilities via XDR protocol, client keeps
the API unchanged but implement a bit different than server, which
may archieve the same effect as invoking qemu-img inside without string
parsing.
   Not sure if libqblock now is blocked by OOM issue as 1st version to
be reviewed. If you think it must be resolved first, I would add this
wrapper quickly.

-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] [RFC] libqblock OOM issue

[Paolo Bonzini]

Il 15/11/2012 05:18, Wenchao Xia ha scritto:
>   Personally agree, but I want to add a simple wrapper to let libqblock
> get user faster. In this way I guess best choice now is making rpc
> client and server not mirrored in implemention, server provides
> r/w/info retrieving capabilities via XDR protocol, client keeps
> the API unchanged but implement a bit different than server, which
> may archieve the same effect as invoking qemu-img inside without string
> parsing.

I think the result would really not be simple...

>   Not sure if libqblock now is blocked by OOM issue as 1st version to
> be reviewed. If you think it must be resolved first, I would add this
> wrapper quickly.

No, I was lost with the build problems.  Please repost the last version
you have.

Paolo

Re: [Qemu-devel] [RFC] libqblock OOM issue

[Wenchao Xia]

 > Il 15/11/2012 05:18, Wenchao Xia ha scritto:
>>    Personally agree, but I want to add a simple wrapper to let libqblock
>> get user faster. In this way I guess best choice now is making rpc
>> client and server not mirrored in implemention, server provides
>> r/w/info retrieving capabilities via XDR protocol, client keeps
>> the API unchanged but implement a bit different than server, which
>> may archieve the same effect as invoking qemu-img inside without string
>> parsing.
>
> I think the result would really not be simple...
>
   yes it would not be very simple, I guess reimplement API on client
side with RPC for only R/W/info APIs,  would be relative easier than
design a transparent and balanced RPC layer for every API.

>>    Not sure if libqblock now is blocked by OOM issue as 1st version to
>> be reviewed. If you think it must be resolved first, I would add this
>> wrapper quickly.
>
> No, I was lost with the build problems.  Please repost the last version
> you have.
>
> Paolo
>
   OK, I'll rebase and repost it.


-- 
Best Regards

Wenchao Xia

Re: [Qemu-devel] [RFC] libqblock OOM issue

[Paolo Bonzini]

Il 15/11/2012 13:21, Wenchao Xia ha scritto:
> 
>>>    Personally agree, but I want to add a simple wrapper to let libqblock
>>> get user faster. In this way I guess best choice now is making rpc
>>> client and server not mirrored in implemention, server provides
>>> r/w/info retrieving capabilities via XDR protocol, client keeps
>>> the API unchanged but implement a bit different than server, which
>>> may archieve the same effect as invoking qemu-img inside without string
>>> parsing.
>>
>> I think the result would really not be simple...
>>
>   yes it would not be very simple, I guess reimplement API on client
> side with RPC for only R/W/info APIs,  would be relative easier than
> design a transparent and balanced RPC layer for every API.

So just let the libvirt people use it.  Then you will understand their
requirements.

Do not try to solve the world's problems at the first attempt.

Paolo