From mboxrd@z Thu Jan  1 00:00:00 1970
From: "Xu, Dongxiao" <dongxiao.xu@intel.com>
Subject: Re: Xen Platform QoS design discussion
Date: Fri, 9 May 2014 02:41:08 +0000
Message-ID: <40776A41FC278F40B59438AD47D147A911A01733@SHSMSX104.ccr.corp.intel.com>
References: <40776A41FC278F40B59438AD47D147A9119F3FEA@SHSMSX104.ccr.corp.intel.com>
	<1398877340.5166.13.camel@kazak.uk.xensource.com>
	<40776A41FC278F40B59438AD47D147A9119F42A2@SHSMSX104.ccr.corp.intel.com>
	<5363804B020000780000E604@mail.emea.novell.com>
	<40776A41FC278F40B59438AD47D147A9119F4EF4@SHSMSX104.ccr.corp.intel.com>
	<5363AE54020000780000E7A2@mail.emea.novell.com>
	<40776A41FC278F40B59438AD47D147A9119FE6BB@SHSMSX104.ccr.corp.intel.com>
	<5368B418.9000307@citrix.com>
	<CAFLBxZZkPMLBBAieLP6abpwOaFBccy9U6yQfSShwAEP4tBtE7A@mail.gmail.com>
	<536AA342.8030003@citrix.com>
	<40776A41FC278F40B59438AD47D147A911A00A4C@SHSMSX104.ccr.corp.intel.com>
	<536B69AB.7010005@citrix.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Return-path: <xen-devel-bounces@lists.xen.org>
In-Reply-To: <536B69AB.7010005@citrix.com>
Content-Language: en-US
List-Unsubscribe: <http://lists.xen.org/cgi-bin/mailman/options/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xen.org>
List-Help: <mailto:xen-devel-request@lists.xen.org?subject=help>
List-Subscribe: <http://lists.xen.org/cgi-bin/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=subscribe>
Sender: xen-devel-bounces@lists.xen.org
Errors-To: xen-devel-bounces@lists.xen.org
To: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: George Dunlap <George.Dunlap@eu.citrix.com>, Ian Campbell <Ian.Campbell@citrix.com>, Jan Beulich <JBeulich@suse.com>, "xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
List-Id: xen-devel@lists.xenproject.org

> -----Original Message-----
> From: Andrew Cooper [mailto:andrew.cooper3@citrix.com]
> Sent: Thursday, May 08, 2014 7:26 PM
> To: Xu, Dongxiao
> Cc: George Dunlap; Ian Campbell; Jan Beulich; xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Xen Platform QoS design discussion
> 
> On 08/05/14 06:21, Xu, Dongxiao wrote:
> 
> <massive snip>
> 
> >>
> >>> We have two different hypercalls right now for getting "dominfo": a
> >>> domctl and a sysctl.  You use the domctl if you want information about
> >>> a single domain, you use sysctl if you want information about all
> >>> domains.  The sysctl implementation calls the domctl implementation
> >>> internally.
> >> It is not a fair comparison, given the completely different nature of
> >> the domctls in question.  XEN_DOMCTL_getdomaininfo is doing very little
> >> more than reading specific bits of data out the appropriate struct
> >> domain and its struct vcpu's which can trivially be done by the cpu
> >> handling the hypercall.
> >>
> >>> Is there a problem with doing the same thing here?  Or, with starting
> >>> with a domctl, and then creating a sysctl if iterating over all
> >>> domains (and calling the domctl internally) if we measure the domctl
> >>> to be too slow for many callers?
> >>>
> >>>  -George
> >> My problem is not with the domctl per-se.
> >>
> >> My problem is that this is not a QoS design discussion;  this is an
> >> email thread about a specific QoS implementation which is not answering
> >> the concerns raised against it to the satisfaction of people raising the
> >> concerns.
> >>
> >> The core argument here is that a statement of "OpenStack want to get a
> >> piece of QoS data back from libvirt/xenapi when querying a specific
> >> domain" is being used to justify implementing the hypercall in an
> >> identical fashion.
> >>
> >> This is not a libxl design; this is a single user story forming part of
> >> the requirement "I as a cloud service provider would like QoS
> >> information for each VM to be available to my
> >> $CHOSEN_ORCHESTRATION_SOFTWARE so I can {differentially charge
> >> customers, balance my load more evenly, etc}".
> >>
> >> The only valid justification for implementing a brand new hypercall in a
> >> certain way is "Because $THIS_CERTAIN_WAY is the $MOST_SENSIBLE way to
> >> perform the actions I need to perform", for appropriately
> >> substitutions.  Not "because it is the same way I want to hand this
> >> information off at the higher level".
> >>
> >> As part of this design discussion. I have raised a concern saying "I
> >> believe the usecase of having a stats gathering daemon in dom0 has not
> >> been appropriately considered", qualified with "If you were to use the
> >> domctl as currently designed from a stats gathering daemon, you will
> >> cripple Xen with the overhead".
> >>
> >> Going back to the original use, xenapi has a stats daemon for these
> >> things.  It has an rpc interface so a query given a specific domain can
> >> return some or all data for that domain, but it very definitely does not
> >> translate each request into a hypercall for the requested information.
> >> I have no real experience with libvirt, so can't comment on stats
> >> gathering in that context.
> >>
> >> I have proposed an alternative Xen->libxc interface designed with a
> >> stats daemon in mind, explaining why I believe it has lower overheads to
> >> Xen and why is more in line with what I expect ${VENDOR}Stack to
> >> actually want.
> >>
> >> I am now waiting for a reasoned rebuttal which has more content than
> >> "because there are a set of patches which already implement it in this way".
> > No, I don't have the patch for domctl implementation.
> >
> > In the past half year, all previous v1-v10 patches are implemented in sysctl way,
> however based on that, people raised a lot of comments (large size of memory,
> runtime non-0 order of memory allocation, page sharing with user space, CPU
> online/offline special logic, etc.), and these make the platform QoS
> implementation more and more complex in Xen. That's why I am proposing the
> domctl method that can make things easier.
> >
> > I don't have more things to argue or rebuttal, and if you prefer sysctl, I can
> continue to work out a v11, v12 or more, to present the big 2-dimension array to
> end user and let them withdraw their real required data, still includes the extra
> CPU online/offline logics to handle the QoS resource runtime allocation.
> >
> > Thanks,
> > Dongxiao
> 
> I am sorry - I was not trying to make an argument for one of the
> proposed mechanisms over the other.  The point I was trying to make
> (which on further consideration isn't as clear as I was hoping) is that
> you cannot possibly design the hypercall interface before knowing the
> library usecases, and there is a clear lack of understanding (or at
> least communication) in this regard.
> 
> 
> So, starting from the top. OpenStack want QoS information, and want to
> get it from libvirt/XenAPI.  I think libvirt/XenAPI is the correct level
> to do this at, and think exactly the same would apply to CloudStack as
> well.  The relevant part of this is the question "how does
> libvirt/XenAPI collect stats".
> 
> XenAPI collects stats with the RRD Daemon, running in dom0.  It has an
> internal database of statistics, and hands data from this database out
> upon RPC requests.  It also has threads whose purpose is to periodically
> refresh the data in the database.  This provides a disconnect between
> ${FOO}Stack requesting stats for a domain and the logic to obtain stats
> for that domain.
> 
> I am however unfamiliar with libvirt in this regard.  Could you please
> explain how the libvirt daemon deals with stats?

I am not the libvirt expert either.
Consult from other guys who work in libvirt that, libvirt doesn't maintain the domain status itself, but just expose the APIs for upper cloud/openstack to query, and these APIs accept the domain id as input parameter.

Thanks,
Dongxiao

> 
> ~Andrew