On Tue, 28 Jul 2020, Roger Pau Monné wrote: > On Mon, Jul 27, 2020 at 05:06:25PM -0700, Stefano Stabellini wrote: > > On Mon, 27 Jul 2020, Roger Pau Monné wrote: > > > On Sat, Jul 25, 2020 at 10:59:50AM +0100, Julien Grall wrote: > > > > On Sat, 25 Jul 2020 at 00:46, Stefano Stabellini wrote: > > > > > > > > > > On Fri, 24 Jul 2020, Julien Grall wrote: > > > > > > On Fri, 24 Jul 2020 at 19:32, Stefano Stabellini wrote: > > > > > > > > If they are not equal, then I fail to see why it would be useful to have this > > > > > > > > value in Xen. > > > > > > > > > > > > > > I think that's because the domain is actually more convenient to use > > > > > > > because a segment can span multiple PCI host bridges. So my > > > > > > > understanding is that a segment alone is not sufficient to identify a > > > > > > > host bridge. From a software implementation point of view it would be > > > > > > > better to use domains. > > > > > > > > > > > > AFAICT, this would be a matter of one check vs two checks in Xen :). > > > > > > But... looking at Linux, they will also use domain == segment for ACPI > > > > > > (see [1]). So, I think, they still have to use (domain, bus) to do the lookup. > > > > > > You have to use the (segment, bus) tuple when doing a lookup because > > > MMCFG regions on ACPI are defined for a segment and a bus range, you > > > can have a MMCFG region that covers segment 0 bus [0, 20) and another > > > MMCFG region that covers segment 0 bus [20, 255], and those will use > > > different addresses in the MMIO space. > > > > Thanks for the clarification! > > > > > > > > > > > > In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so > > > > > > > > Dom0 and Xen can synchronize on the segment number. > > > > > > > > > > > > > > I was hoping we could write down the assumption somewhere that for the > > > > > > > cases we care about domain == segment, and error out if it is not the > > > > > > > case. > > > > > > > > > > > > Given that we have only the domain in hand, how would you enforce that? > > > > > > > > > > > > >From this discussion, it also looks like there is a mismatch between the > > > > > > implementation and the understanding on QEMU devel. So I am a bit > > > > > > concerned that this is not stable and may change in future Linux version. > > > > > > > > > > > > IOW, we are know tying Xen to Linux. So could we implement > > > > > > PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that > > > > > > really represent the segment? > > > > > > > > > > I don't think we are tying Xen to Linux. Rob has already said that > > > > > linux,pci-domain is basically a generic device tree property. > > > > > > > > My concern is not so much the name of the property, but the definition of it. > > > > > > > > AFAICT, from this thread there can be two interpretation: > > > > - domain == segment > > > > - domain == (segment, bus) > > > > > > I think domain is just an alias for segment, the difference seems to > > > be that when using DT all bridges get a different segment (or domain) > > > number, and thus you will always end up starting numbering at bus 0 > > > for each bridge? > > > > > > Ideally you would need a way to specify the segment and start/end bus > > > numbers of each bridge, if not you cannot match what ACPI does. Albeit > > > it might be fine as long as the OS and Xen agree on the segments and > > > bus numbers that belong to each bridge (and thus each ECAM region). > > > > That is what I thought and it is why I was asking to clarify the naming > > and/or writing a document to explain the assumptions, if any. > > > > Then after Julien's email I followed up in the Linux codebase and > > clearly there is a different assumption baked in the Linux kernel for > > architectures that have CONFIG_PCI_DOMAINS enabled (including ARM64). > > > > The assumption is that segment == domain == unique host bridge. It > > looks like it is coming from IEEE Std 1275-1994 but I am not certain. > > In fact, it seems that ACPI MCFG and IEEE Std 1275-1994 don't exactly > > match. So I am starting to think that domain == segment for IEEE Std > > 1275-1994 compliant device tree based systems. > > I don't think the ACPI MCFG spec contains the notion of bridges, it > just describes ECAM (or MMCFG) regions, but those could be made up by > concatenating different bridge ECAM regions by the firmware itself, so > you could AFAICT end up with multiple bridges being aggregated into a > single ECAM region, and thus using the same segment number, which > seems not possible with the DT spec, where each bridge must get a > different segment number? Yes, that's my understanding too > If you could assign both a segment number and a bus start and end > values to a bridge then I think it would be kind of equivalent to ACPI > MCFG. > > I assume we would never support a system where Xen is getting the > hardware description from a DT and the hardware domain is using ACPI > (or the other way around)? Yeah, I think it is a good assumption > If so, I don't think we care that enumeration when using DT is > different than when using ACPI, as we can only guarantee consistency > when both Xen and the hardware domain use the same source for the > hardware description. > > If when using DT each bridge has a unique segment number that's fine > as long as Xen and the OS agree to not change such values. I agree