RE: [EXTERNAL] Re: [RFC 1/1] irqchip/gic-v3-its: Add irq domain and chip for Direct LPI without ITS

[Sunil Muthuswamy <sunilmut@microsoft.com>]

On Wednesday, August 4, 2021 2:21 AM,
Marc Zyngier <maz@kernel.org> wrote:
>
> On Tue, 03 Aug 2021 09:35:12 +0100,
> Robin Murphy <robin.murphy@arm.com> wrote:
> >
> > On 2021-08-03 03:11, Sunil Muthuswamy wrote:
> > >   On Saturday, July 31, 2021 2:52 AM,
> > >   Marc Zyngier <maz@kernel.org> wrote:
> > >>
> > >> [...]
> > >>
> > >>>> I also want to understand *how* you are going to plumb this into both
> > >>>> ACPI and DT, given that neither understand how to link a PCI endpoint
> > >>>> to a set of RDs.
> > >>>>
> > >>>> 	M.
> > >>>
> > >>> One way to do this for NUMA-aware systems would be to use the NUMA
> > >>> related information that is available with PCI endpoints or root complex, to
> > >>> pick a Redistributor/CPU that is in the NUMA node, as specified by the PCI
> > >>> endpoint/root complex. In DT PCI devices can specify this using
> > >>> 'numa-node-id' and in ACPI using the '_PXM (Proximity)'. For systems that
> > >>> are not NUMA-aware, we can go with *any* Redistributor/CPU.
> > >>
> > >> This makes zero sense. From the point of view of a device, all the RDs
> > >> should be reachable, and firmware has no say in it. Dealing with
> > >> interrupt affinity is the responsibility of the endpoint driver, and
> > >> NUMA affinity is only a performance optimisation.
> > >>
> > >>> Is there any additional information we would be able to gather from ACPI
> > >>> or DT that's not there currently, that would be useful here?
> > >>
> > >> You will need some firmware information describing that a given set of
> > >> devices must use the RDs for their MSIs. Just like we currently
> > >> describe it in IORT for the ITS. You cannot /assume/ things. At the
> > >> moment, there is nothing at all, because no-one (including Microsoft)
> > >> thought it would be a good idea not to have an ITS, which is also why
> > >> ACPI doesn't describe MBIs as a potential MSI provider.
> > >>
> > > I am a little bit confused by your above comment. Maybe you can help me
> > > understand the ask. You indicate that from the point of the view of the
> > > device, all the RDs should be reachable. But, then if we define a mapping
> > > between PCI endpoint and RD in the firmware, we would be doing exactly
> > > the opposite. i.e. restricting the RDs that are reachable by the device. Can
> > > you please clarify?
> 
> Not at all. What I am saying is that you need to describe that the
> MSIs have to be routed to the RDs, and there is no way to express this
> at the moment.
> 
> > >
> > > Is your concern that the device should be able to only DMA to a subset of
> > > GIC Redistributor, for the MSIs? If so, in the IORT, there is "memory address
> > > size limit" for both device and root complex nodes. In the implementation,
> > > we can enforce that the GICR is within that range. And, if a device deviates
> > > further than that (ex: by having accessibility gaps within the GICR range),
> > > then that is out of scope for support.
> >
> > No, please don't try to abuse the Memory Address Size Limit - that has
> > far more chance of adversely affecting normal DMA operation than of
> > being any use here.
> >
> > I believe the point Marc was trying to make is that firmware should
> > not associate a device with any one *specific* redistributor, however
> > ACPI currently has no way to describe that MSIs can target
> > redistributors *at all*, only ITS groups - there is no such concept as
> > a "redistributor group".
> 
> Thanks Robin.
> 
> That is exactly my point. There is no linkage whatsoever between a
> device generating MSIs and the redistributors. In that respect, this
> is the same issue we have with DT, as none of the firmware interfaces
> can currently describe MSIs directly targeting the RDs. The only way
> to describe MSIs with GICv3 in ACPI is to describe an ITS, and you
> cannot use the *absence* of an ITS to decide and use DirectLPIs.
> 
> Given the state of things, using DirectLPI means that you need to
> extend the firmware interfaces. This means both an extension to the DT
> binding, and an update to the ACPI spec. There is no way around this.
> 
> Thanks,
> 
> 	M.
> 
Thanks Marc and Robin for clarifying. I see and understand the point
about having explicit MSI mappings in the firmware specification for
Direct LPIs for generic hardware support.

Hey Mark, would you be willing to consider a scoped down implementation of
GIC Direct LPI with just an IRQ chip implementation and no
Direct LPI PCI-MSI IRQ chip. This will allow a MSI provider (such as Hyper-V vPCI) to
provide a PCI-MSI IRQ chip on top of the Direct LPI IRQ chip and enable
PCI-MSI scenarios, and avoid building in assumptions in other cases (like PCI) where
firmware specification is not available.
- This scoped down implementation would allow Microsoft to build virtual
PCI on top, to enable our business needs.
- If there's a need for a generic support for a Direct LPI PCI MSI IRQ, that could
drive firmware revision efforts, and we are happy to assist there.

Looking forward to hearing back.

Thanks,
Sunil