From mboxrd@z Thu Jan 1 00:00:00 1970 From: Manish Jaggi Subject: Re: PCI Passthrough ARM Design : Draft1 Date: Fri, 26 Jun 2015 07:37:26 +0530 Message-ID: <558CB3DE.7000004@caviumnetworks.com> References: <55806109.7040808@caviumnetworks.com> <1434548605.13744.391.camel@citrix.com> <558180C9.9020904@caviumnetworks.com> <1434551364.13744.403.camel@citrix.com> <558BB174.40204@caviumnetworks.com> <1435223510.32500.2.camel@citrix.com> <558BED2D.3080600@caviumnetworks.com> <1435234888.32500.80.camel@citrix.com> <20150625172647.GB2470@x230.dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20150625172647.GB2470@x230.dumpdata.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Konrad Rzeszutek Wilk , Ian Campbell Cc: Vijay Kilari , Stefano Stabellini , Prasun Kapoor , "Kumar, Vijaya" , "xen-devel@lists.xen.org" , Julien Grall , Stefano Stabellini , "Kulkarni, Ganapatrao" , =?windows-1252?Q?Roger_Pau_Monn=E9?= List-Id: xen-devel@lists.xenproject.org On Thursday 25 June 2015 10:56 PM, Konrad Rzeszutek Wilk wrote: > On Thu, Jun 25, 2015 at 01:21:28PM +0100, Ian Campbell wrote: >> On Thu, 2015-06-25 at 17:29 +0530, Manish Jaggi wrote: >>> On Thursday 25 June 2015 02:41 PM, Ian Campbell wrote: >>>> On Thu, 2015-06-25 at 13:14 +0530, Manish Jaggi wrote: >>>>> On Wednesday 17 June 2015 07:59 PM, Ian Campbell wrote: >>>>>> On Wed, 2015-06-17 at 07:14 -0700, Manish Jaggi wrote: >>>>>>> On Wednesday 17 June 2015 06:43 AM, Ian Campbell wrote: >>>>>>>> On Wed, 2015-06-17 at 13:58 +0100, Stefano Stabellini wrote: >>>>>>>>> Yes, pciback is already capable of doing that, see >>>>>>>>> drivers/xen/xen-pciback/conf_space.c >>>>>>>>> >>>>>>>>>> I am not sure if the pci-back driver can query the guest memory map. Is there an existing hypercall ? >>>>>>>>> No, that is missing. I think it would be OK for the virtual BAR to be >>>>>>>>> initialized to the same value as the physical BAR. But I would let the >>>>>>>>> guest change the virtual BAR address and map the MMIO region wherever it >>>>>>>>> wants in the guest physical address space with >>>>>>>>> XENMEM_add_to_physmap_range. >>>>>>>> I disagree, given that we've apparently survived for years with x86 PV >>>>>>>> guests not being able to right to the BARs I think it would be far >>>>>>>> simpler to extend this to ARM and x86 PVH too than to allow guests to >>>>>>>> start writing BARs which has various complex questions around it. >>>>>>>> All that's needed is for the toolstack to set everything up and write >>>>>>>> some new xenstore nodes in the per-device directory with the BAR >>>>>>>> address/size. >>>>>>>> >>>>>>>> Also most guests apparently don't reassign the PCI bus by default, so >>>>>>>> using a 1:1 by default and allowing it to be changed would require >>>>>>>> modifying the guests to reasssign. Easy on Linux, but I don't know about >>>>>>>> others and I imagine some OSes (especially simpler/embedded ones) are >>>>>>>> assuming the firmware sets up something sane by default. >>>>>>> Does the Flow below captures all points >>>>>>> a) When assigning a device to domU, toolstack creates a node in per >>>>>>> device directory with virtual BAR address/size >>>>>>> >>>>>>> Option1: >>>>>>> b) toolstack using some hypercall ask xen to create p2m mapping { >>>>>>> virtual BAR : physical BAR } for domU >>>>> While implementing I think rather than the toolstack, pciback driver in >>>>> dom0 can send the >>>>> hypercall by to map the physical bar to virtual bar. >>>>> Thus no xenstore entry is required for BARs. >>>> pciback doesn't (and shouldn't) have sufficient knowledge of the guest >>>> address space layout to determine what the virtual BAR should be. The >>>> toolstack is the right place for that decision to be made. >>> Yes, the point is the pciback driver reads the physical BAR regions on >>> request from domU. >>> So it sends a hypercall to map the physical bars into stage2 translation >>> for the domU through xen. >>> Xen would use the holes left in IPA for MMIO. >> I still think it is the toolstack which should do this, that's whewre >> these sorts of layout decisions belong. can the xl tools read pci conf space ? Using some xen hypercall or a xl-dom0 ioctl ? If not then there is no otherway but xenpciback Also I need to introduce a hypercall which would tell toolkit the available holes for virtualBAR mapping. Much simpler is let xen allocate a virtualBAR and return to the caller. > At init - sure. But when the guest is running and doing those sort > of things. Unless you want guest -> pciback -> xenstore -> libxl -> > hypercall -> send ack on xenstore -> pciback -> guest. > > That would entail adding some pcibkack -> user-space tickle mechanism > and another back. Much simpler to do all of this in xenpciback I think? I agree. If the xenpciback sends a hypercall whenever a BAR read access, the mapping in xen would already have been done, so xen would simply be doing PA->IPA lookup. No xenstore lookup is required. >>> Xen would return the IPA for pci-back to return to the request to domU. >>>>> Moreover a pci driver would read BARs only once. >>>> You can't assume that though, a driver can do whatever it likes, or the >>>> module might be unloaded and reloaded in the guest etc etc. >>>> >>>> Are you going to send out a second draft based on the discussion so far? >>> yes, I was working on that only. I was traveling this week 24 hour >>> flights jetlag... >>>> Ian. >>>> >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xen.org >>>> http://lists.xen.org/xen-devel >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel