Re: [PATCH v4 05/17] xen/arm: ITS: implement hw_irq_controller for LPIs

[Marc Zyngier <marc.zyngier@arm.com>]

On 16/07/15 15:49, Ian Campbell wrote:
> On Wed, 2015-07-15 at 22:31 +0530, Vijay Kilari wrote:
>> Sorry. I may not be clear.
>>
>> In Linux when MSIx is enabled.
>> Device is created first and its_device->its_collection is set for
>> first onine cpu
>> and mapvi is called with collection set in its_create_device() as below.
> [...]
>> When affinity is set, movi is sent with collection id selected
>> for the cpu_mask.
> 
> OK, so at start of day all events for a given device end up mapped to a
> single processor, but as individual interrupts are rebalanced they will
> then become spread out among different CPUs, that makes sense.
> 
> I'm not sure I follow the scheme which Linux is using to achieve that
> behaviour though, so I've CC'd Marc.
> 
> It seems to me from looking at the Linux code that its_dev->collection
> doesn't, as one might expect, contain the collection associated with the
> device (and therefore all Events of that device), but rather it contains
> the collection corresponding to a single Event which is the one which
> most recently had its affinity changed, i.e. the one with an potentially
> outstanding INV.
> 
> So its_send_movi sends the MOVI command which ends up calling INV on
> that collection, which at this point is the _old_ collection. Then it
> stores the new collection for that Event in its_dev->collection.
> 
> What I don't follow is how set_lpi_config copes with this, since it
> always sends the INV for an Event to the corresponding its_dev's
> collection, but after a bunch of Events/IRQs have been rebalanced there
> doesn't seem to be any guarantee that this would corresponding to the
> current collection associated with the interrupt which has been
> reconfigured.
> 
> Marc, have I misunderstood what the Linux ITS driver is trying to
> achieve or how it goes about it?

Seems like there is a lot of terminology related confusion here, so
let's start from the beginning:

The ITS maps a [DevID, EventID] pair to a [LPI, Collection] pair, where
DevID is the device writing the MSI, EventID is the payload being
written, LPI is the actual interrupt number, and Collection is roughly
equivalent to a target CPU.

The DevID is not directly associated to a Collection; it is the LPI that
is mapped to a Collection (through the MAPVI command). When you perform
a MOVI, you are moving the *result* of the [DevID, EventID] pair
(effectively an LPI) from a Collection to another.

> Perhaps lpi_set_config (and by extension its_(un)mask_irq) is only
> called at start of day?

lpi_set_config is indeed only used when we put the interrupt in service.
It updates the configuration table for the redistributor (the thing
pointed to by the Collection) in charge of this LPI (*LPI*, not device
or event) and issue an INV so that the redistributor (gets the new
value). And yes, this is always done by [DevID, EventID] pair.

When you perform a MOVI, this implicitly contains an INV for the target
redistributor (MOVI will also move the pending state, and the old
redistributor will be left alone).

> I'm not sure if Xen is going to (want to) follow the way Linux arranges
> this stuff, but I would very much like to at least understand it, since
> it may have implications...

Hope this helps,

	M.
-- 
Jazz is not dead. It just smells funny...