Re: Xen Platform QoS design discussion

["Xu, Dongxiao" <dongxiao.xu@intel.com>]

> -----Original Message-----
> From: Andrew Cooper [mailto:andrew.cooper3@citrix.com]
> Sent: Tuesday, May 06, 2014 6:06 PM
> To: Xu, Dongxiao
> Cc: Jan Beulich; Ian Campbell; xen-devel@lists.xen.org
> Subject: Re: Xen Platform QoS design discussion
> 
> On 06/05/14 02:40, Xu, Dongxiao wrote:
> >> -----Original Message-----
> >> From: Xu, Dongxiao
> >> Sent: Sunday, May 04, 2014 8:46 AM
> >> To: Jan Beulich
> >> Cc: Andrew Cooper(andrew.cooper3@citrix.com); Ian Campbell;
> >> xen-devel@lists.xen.org
> >> Subject: RE: Xen Platform QoS design discussion
> >>
> >>> -----Original Message-----
> >>> From: Jan Beulich [mailto:JBeulich@suse.com]
> >>> Sent: Friday, May 02, 2014 8:40 PM
> >>> To: Xu, Dongxiao
> >>> Cc: Andrew Cooper(andrew.cooper3@citrix.com); Ian Campbell;
> >>> xen-devel@lists.xen.org
> >>> Subject: RE: Xen Platform QoS design discussion
> >>>
> >>>>>> On 02.05.14 at 14:30, <dongxiao.xu@intel.com> wrote:
> >>>>>  -----Original Message-----
> >>>>> From: Jan Beulich [mailto:JBeulich@suse.com]
> >>>>> Sent: Friday, May 02, 2014 5:24 PM
> >>>>> To: Xu, Dongxiao
> >>>>> Cc: Andrew Cooper(andrew.cooper3@citrix.com); Ian Campbell;
> >>>>> xen-devel@lists.xen.org
> >>>>> Subject: RE: Xen Platform QoS design discussion
> >>>>>
> >>>>>>>> On 01.05.14 at 02:56, <dongxiao.xu@intel.com> wrote:
> >>>>>>> From: Ian Campbell [mailto:Ian.Campbell@citrix.com]
> >>>>>>> Have you asked yourself whether this information even needs to be
> >>>>>>> exposed all the way up to libxl? Who are the expected consumers of
> this
> >>>>>>> interface? Are they low-level CLI tools (i.e. like xenpm is) or are you
> >>>>>>> expecting toolstacks to plumb this information all the way up to their
> >>>>>>> GUI or CLI (e.g. xl or virsh)?
> >>>>>> The information returned to libxl users is the cache utilization for a
> >>>>>> certain domain in certain socket, and the main consumers are cloud
> users
> >>>> like
> >>>>>> openstack, etc. Of course, we will also provide an xl command to present
> >>>> such
> >>>>>> information.
> >>>>> To me this doesn't really address the question Ian asked, yet knowing
> >>>>> who's going to be the consumer of the data is also quite relevant for
> >>>>> answering your original question on the method to obtain that data.
> >>>>> Obviously, if the main use of it is per-domain, a domctl would seem like
> >>>>> a suitable approach despite the data being more of sysctl kind. But if
> >>>>> a global view would be more important, that model would seem to make
> >>>>> life needlessly hard for the consumers. In turn, if using a domctl, I tend
> >>>>> to agree that not using shared pages would be preferable; iirc their use
> >>>>> was mainly suggested because of the size of the data.
> >>>> From the discussion with openstack developers, on certain cloud host, all
> >>>> running VM's information (e.g., domain ID) will be stored in a database, and
> >>>> openstack software will use libvirt/XenAPI to query specific domain
> >>>> information. That libvirt/XenAPI API interface basically accepts the domain
> >>>> ID as input parameter and get the domain information, including the
> platform
> >>>> QoS one.
> >>>>
> >>>> Based on above information, I think we'd better design the QoS hypercall
> >>>> per-domain.
> >>> If you think that this is going to be the only (or at least prevalent)
> >>> usage model, that's probably okay then. But I'm a little puzzled that
> >>> all this effort is just for a single, rather specific consumer. I thought
> >>> that if this is so important to Intel there would be wider interested
> >>> audience.
> > Since there is no further comments, I suppose we all agreed on making the
> hypercall per-domain and use data copying mechanism between hypervisor and
> Dom0 tool stack?
> >
> 

Reply previous Ian and Andrew's comments in this mail.

> No - the onus is very much on you to prove that your API will *not* be
> used in the following way:
> 
> every $TIMEPERIOD
>   for each domain
>     for each type of information
>       get-$TYPE-information-for-$DOMAIN

The "for loop" mentioned here does exist in certain software levels, and there are several options:
1. For loop in libvirt/openstack layer (likely):
In this case, domctl would be better which returns per-domain's QoS info. Otherwise it will repeatedly call sysctl hypercall to get the entire data structure but only returns one domain's info to user space.

2. For loop within libxl API function and returns whole QoS data (unlikely):
If we return such entire PQoS info to Dom0 user space via libxl API, then this API will be changing once new PQoS feature comes out. As Ian mentioned, we need certain compatibility for libxl API.

> 
> Which is the source of my concerns regarding overhead.
> 
> As far as I can see, as soon as you provide access to this QoS
> information, higher level toolstacks are going to want all information
> for all domains.  Given your proposed domctl, they will have exactly one
> (bad) way of getting this information.

I understand your point.
I think the final decision should be a compromise between:
1) Overhead.
2) Compatibility.
3) Code flexibility and simplicity.

Thanks,
Dongxiao

> 
> ~Andrew