From mboxrd@z Thu Jan  1 00:00:00 1970
From: Ian Campbell <ian.campbell@citrix.com>
Subject: Re: RFC: [PATCH 1/3] Enhance platform support for PCI
Date: Mon, 23 Feb 2015 15:46:37 +0000
Message-ID: <1424706397.27930.178.camel@citrix.com>
References: <54E71BDE.5020106@caviumnetworks.com>
	<54E7229C.7000301@linaro.org> <54E72452.3090801@caviumnetworks.com>
	<54E72688.9010005@linaro.org> <54E729F1.6000804@caviumnetworks.com>
	<54E73010.2050902@caviumnetworks.com>
	<1424439941.30924.243.camel@citrix.com>
	<54E74EA802000078000620B7@mail.emea.novell.com>
	<1424442385.30924.265.camel@citrix.com>
	<54E7552F020000780006215A@mail.emea.novell.com>
	<1424444504.30924.282.camel@citrix.com>
	<54E75D8702000078000621DD@mail.emea.novell.com>
	<1424453619.30924.381.camel@citrix.com>
	<54EAF6390200007800062593@mail.emea.novell.com>
	<1424695518.27930.74.camel@citrix.com>
	<54EB42320200007800062871@mail.emea.novell.com>
	<1424702013.27930.118.camel@citrix.com>
	<54EB4AFE02000078000628B3@mail.emea.novell.com>
	<1424703761.27930.140.camel@citrix.com>
	<54EB54D8020000780006296F@mail.emea.novell.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Return-path: <xen-devel-bounces@lists.xen.org>
In-Reply-To: <54EB54D8020000780006296F@mail.emea.novell.com>
List-Unsubscribe: <http://lists.xen.org/cgi-bin/mailman/options/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xen.org>
List-Help: <mailto:xen-devel-request@lists.xen.org?subject=help>
List-Subscribe: <http://lists.xen.org/cgi-bin/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=subscribe>
Sender: xen-devel-bounces@lists.xen.org
Errors-To: xen-devel-bounces@lists.xen.org
To: Jan Beulich <JBeulich@suse.com>
Cc: Prasun.kapoor@cavium.com, Manish Jaggi <mjaggi@caviumnetworks.com>, Vijaya Kumar <Vijaya.Kumar@caviumnetworks.com>, Julien Grall <julien.grall@linaro.org>, xen-devel@lists.xen.org, "StefanoStabellini(Stefano.Stabellini@citrix.com)" <stefano.stabellini@citrix.com>
List-Id: xen-devel@lists.xenproject.org

On Mon, 2015-02-23 at 15:27 +0000, Jan Beulich wrote:
> >>> On 23.02.15 at 16:02, <ian.campbell@citrix.com> wrote:
> > On Mon, 2015-02-23 at 14:45 +0000, Jan Beulich wrote:
> >> In which case the Dom0 OS doing so would need to communicate
> >> its decisions to the hypervisor, as you suggest further down.
> > 
> > So more concretely something like:
> >         #define PHYSDEVOP_pci_host_bridge_add <XX>
> >         struct physdev_pci_host_bridge_add {
> >             /* IN */
> >             uint16_t seg;
> >             uint8_t bus;
> >             uint64_t address;
> >         };
> >         typedef struct physdev_pci_host_bridge_add physdev_pci_host_bridge_add_t;
> >         DEFINE_XEN_GUEST_HANDLE(physdev_pci_host_bridge_add_t);
> > 
> > Where seg+bus are enumerated/assigned by dom0 and address is some unique
> > property of the host bridge -- most likely its pci cfg space base
> > address (which is what physdev_pci_mmcfg_reserved also takes, I think?)
> 
> Right.
> 
> > Do you think we would need start_bus + end_bus here? Xen could enumerate
> > this itself I think, and perhaps should even if dom0 tells us something?
> 
> That depends - if what you get presented here by Dom0 is a PCI
> device at <seg>:<bus>:00.0, and if all other setup was already
> done on it, then you could read the secondary and subordinate
> bus numbers from its config space. If that's not possible, then
> Dom0 handing you these values would seem to be necessary.
> 
> As a result you may also need a hook from PCI device registration,
> allowing to associate it with the right host bridge (and refusing to
> add any for which there's none).

Right.

My thinking was that PHYSDEVOP_pci_host_bridge_add would add an entry
into some mapping data structure from (segment,bus) to a handle
associated with the associated pci host bridge driver in Xen.

PHYSDEVOP_manage_pci_add would have to lookup the host bridge driver
from the (segment,bus) I think to construct the necessary linkage for
use later when we try to do things to the device, and it should indeed
fail if it can't find one.

> As an alternative, extending PHYSDEVOP_manage_pci_add_ext in
> a suitable manner may be worth considering, provided (like on x86
> and ia64) the host bridges get surfaced as distinct PCI devices.
> 
> >> This
> >> basically replaces the bus scan (on segment 0) that Xen does on
> >> x86 (which topology information gets derived from).
> > 
> > Is the reason for the scan being of segment 0 only is that it is the one
> > which lives at the legacy PCI CFG addresses (or those magic I/O ports)? 
> 
> Right - ideally we would scan all segments, but we need Dom0 to
> tell us which MMCFG regions are safe to access,

Is this done via PHYSDEVOP_pci_mmcfg_reserved?

>  and hence can't
> do that scan at boot time. But we also won't get away without
> scanning, as we need to set up the IOMMU(s) to at least cover
> the devices used for booting the system.

Which hopefully are all segment 0 or aren't needed until after dom0
tells Xen about them I suppose.

> > What about other host bridges in segment 0 which aren't at that address?
> 
> At which address?

I meant this to be a back reference to "the legacy PCI CFG addresses (or
those magic I/O ports)".

>  (All devices on segment zero are supposed to
> be accessible via config space access method 1.)

Is that "the legacy ....  or magic ..." again?

> > You could do the others based on MMCFG tables if you wanted, right?
> 
> Yes, with the above mentioned caveat.
> 
> Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel