From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Tian, Kevin" Subject: Re: (v2) Design proposal for RMRR fix Date: Thu, 15 Jan 2015 09:36:38 +0000 Message-ID: References: <54B515E6020000780005444A@mail.emea.novell.com> <54B54D48020000780005473E@mail.emea.novell.com> <54B540AA.1010905@eu.citrix.com> <54B63E250200007800054A36@mail.emea.novell.com> <54B651EE0200007800054ADD@mail.emea.novell.com> <54B65AB4.5060504@eu.citrix.com> <54B68C180200007800054E84@mail.emea.novell.com> <54B67F32.8050902@eu.citrix.com> <54B68FA10200007800054F01@mail.emea.novell.com> <54B6B578.60106@eu.citrix.com> <54B78A5A02000078000552C2@mail.emea.novell.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <54B78A5A02000078000552C2@mail.emea.novell.com> Content-Language: en-US List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Jan Beulich , George Dunlap Cc: "wei.liu2@citrix.com" , "ian.campbell@citrix.com" , "stefano.stabellini@eu.citrix.com" , "tim@xen.org" , "ian.jackson@eu.citrix.com" , "xen-devel@lists.xen.org" , "Zhang, Yang Z" , "Chen, Tiejun" List-Id: xen-devel@lists.xenproject.org > From: Jan Beulich [mailto:JBeulich@suse.com] > Sent: Thursday, January 15, 2015 4:38 PM > > >>> On 14.01.15 at 19:29, wrote: > > Just to be clear -- what we're talking about here is that at the > > do_domain_create() level (called by libxl_domain_create_new()), it will > > take a list of pci devices, and the rmrr list above (including "host" > > and individual ranges), and generate a list of RMRRs to pass to the > > lower layer. The lower layer will simply see the range, and a "force / > > no force" flag, and behave appropriately. The determination of which > > RMRRs to force will be done at the domain_create level. > > > > Is that about right? > > That's certainly a sensible model. > It's really a mess for my outlook to sort multiple threads under same topic... so I'll reply to this one after reading through all previous good discussions. First, sorry that I used '0/1' as a bad example for user options, and thanks for your suggestion on a right way defining them. I also agree above model. Policy is all decided at domain_create while lower layer only reacts to specified regions which have individual policy to indicate 'force' or 'not force'. We'll need to make that skeleton ready first. Then regarding to config interface, how about making some simplification by only considering statically-assigned device and hotplug now (leaving migration to future based on necessity, and extending that doesn't change low level skeleton)? pci option can be extended as: pci = [ 'bdf, rmrr_check=force/try' ] domain_create queries Xen hypervisor about RMRRs associated with assigned devices, and then mark each region with user specified override. User can also specify a rmrr_host option as: rmrr = [ 'host, check=force/try' ] when rmrr_host is specified, domain_create queries all RMRRs reported in this platform, and mark per-region policy accordingly. per-device override is always favored if a conflicting setting in rmrr_host. The composed reserved region list is then passed to domain builder, which tries to detect and avoid conflicts when populating guest RAM. To avoid breaking lowmem/highmem layout, we can define a lowmem_guard so if making hole for a region would make lowmem_top below lowmem_guard we'll treat this region as a conflict. We may either just hardcode the value like 2G (or other reasonable value in your mind), or allow user to config e.g.: rmrr = [ 'host, check=force/try', 'lowmem_boundary=2G' ] after domain builder, libxl will request actual device assignment to Xen hypervisor. At that point, current assignment hypercall needs to be extended to include the override value, so Xen will handle conflict accordingly when setting up identity map. last step is in hvmloader, which is passed with all the necessary RMRR information, and then handles potential conflicts accordingly. In the future when we think necessary to allow user specify random regions for migration usage, it can be easily extended and we can argue whether to introduce same override. If above high level flow can be agreed, then we can move forward to discuss next level detail e.g. how to pass the rmrr list cross different components. :-) Thanks Kevin