Re: [patch V2 06/40] PCI/MSI: Provide static key for parent mask/unmask

[Marc Zyngier <maz@kernel.org>]

On Mon, 22 May 2023 15:19:39 +0100,
Thomas Gleixner <tglx@linutronix.de> wrote:
> 
> On Fri, Nov 25 2022 at 01:11, Thomas Gleixner wrote:
> > On Thu, Nov 24 2022 at 13:38, Marc Zyngier wrote:
> >> On Thu, 24 Nov 2022 13:17:00 +0000,
> >> Thomas Gleixner <tglx@linutronix.de> wrote:
> >>> > I find this a bit odd. If anything, I'd rather drop the masking at the
> >>> > PCI level and keep it local to the interrupt controller, because this
> >>> > is likely to be more universal than the equivalent PCI operation
> >>> > (think multi-MSI, for example, which cannot masks individual MSIs).
> >>> >
> >>> > Another thing is that the static key is a global state. Nothing says
> >>> > that masking one way or the other is a universal thing, specially when
> >>> > you have multiple interrupt controllers dealing with MSIs in different
> >>> > ways. For example, GICv3 can use both the ITS and the GICv3-MBI frame
> >>> > at the same time for different PCI RC. OK, they happen to deal with
> >>> > MSIs in the same way, but you hopefully get my point.
> >>> 
> >>> I'm fine with dropping that. I did this because basically all of the
> >>> various ARM PCI/MSI domain implementation have a copy of the same
> >>> functions. Some of them have pointlessly the wrong order because copy &
> >>> pasta is so wonderful....
> >>> 
> >>> So the alternative solution is to provide _ONE_ set of correct callbacks
> >>> and let the domain initialization code override the irq chip callbacks
> >>> of the default PCI/MSI template.
> >>
> >> If the various irqchips can tell the core code whether they want
> >> things to be masked at the PCI level or at the irqchip level, this
> >> would be a move in the right direction. For the GIC, I'd definitely
> >> want things masked locally.
> >>
> >> What I'd like to get rid off is the double masking, as I agree it is
> >> on the "pretty dumb" side of things.
> >
> > Not necessarily. It mitigates the problem of MSI interrupts which can't
> > be masked because the implementers decided to spare the gates. MSI
> > allows that as masking is opt-in...
> >
> > Let me think about it.
> 
> That really took a while to think about it :)
> 
> We have the following cases on the PCI/MSI side:
> 
>  1) The MSI[X] entry can be masked
> 
>  2) The MSI[X] entry cannot be masked because hardware did not implement
>     it, masking is globally disabled due to XEN, masking does not exist
>     for this horrible virtual MSI hackery

And as a bonus the case of non-PCI MSIs, which are definitely a thing,
and I'd like them to fit in the same model (because life is too short
to do anything else). As for the Xen side, I hope to never have to
care about it for the architecture I care about (I've long proclaimed
Xen/arm64 dead and buried).

> 
> Now you said:
> 
>  "For the GIC, I'd definitely want things masked locally."
> 
> I decoded this, that you want to have these interrupts masked at the GIC
> level too independent of #1 or #2 above. And then:
> 
>  "What I'd like to get rid off is the double masking."
> 
> But relying on the GIC alone is not really a good thing IMO. There is no
> point to let some confused device send unwanted MSI messages around
> without a way to shut it up from the generic code via the regular
> mask/unmask callbacks.

I have a slightly different view of the problem. The device masking is
somehow orthogonal with the masking at the GIC level:

- can the interrupt be generated: this is a device property

- can the interrupt be signalled: this is an interrupt controller
  property

In a way, this is no different from your basic device, such as a
timer: you need both the interrupt generation to be enabled at the
timer level, and the interrupt signalling to be enabled (unmasked) at
the irqchip level.

Today, we conflate the two, because we have either:

- devices that cannot selectively mask interrupts

- interrupt controllers that are limited in what they can mask

and this results in the terrible pattern that's all over the
GIC-related stuff.

> On the other hand for PCI/MSI[x] the mask/unmask operations are not in
> the hot path as PCI/MSI[x] are strictly edge. Mask/unmask is only
> happening on startup, shutdown and when an interrupt arrives after
> disable_irq() incremented the lazy disable counter.
> 
> For regular interrupt handling mask/unmask is not involved.
> 
> So to avoid that global key we can let the parent domain set a new flag,
> e.g. MSI_FLAG_PCI_MSI_MASK_PARENT, in msi_parent_ops::supported_flags
> and let the PCI/MSI core code query that information when the per device
> domain is created and select the appropriate template or fixup the
> callbacks after the domain is created.
> 
> Does that address your concerns?

It does to a certain extent.

But what I'd really like is that in the most common case where the
interrupt controller is capable of masking MSIs, the PCI/MSI
*enabling* becomes the responsibility of the PCI core code and not the
IRQ code.

The IRQ code should ideally only be concerned with the masking of the
interrupt at the irqchip level, and not beyond that. And that'd solve
the Xen problem by merely ignoring it.

If we have HW out there that cannot mask MSIs at the interrupt
controller level, then we'd have to fallback to device-side masking,
which doesn't really work in general (MultiMSI being my favourite
example). My gut feeling is that this is rare, but I'm pretty sure it
exists.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.