On 2016-07-31 05:59, Peter Xu wrote:
> On Sat, Jul 30, 2016 at 09:52:48AM +0200, Jan Kiszka wrote:
> 
> [...]
> 
>>> +/**
>>> + * x86_iommu_iec_notify_all - Notify IEC invalidations
>>> + * @iommu: IOMMU device that sends the notification
>>> + * @global: whether this is a global invalidation. If true, @index
>>> + *          and @mask are undefined.
>>> + * @index: starting index of interrupt entry to invalidate
>>> + * @mask: index mask for the invalidation
>>
>> This is Intel'ish: index and mask refer to the single Intel IR table.
>> AMD has per-device tables.
> 
> Actually I was trying to consider this before when designing about the
> interface...
> 
>>
>> But even for Intel: Would the index make any sense to the callbacks? KVM
>> uses (virtual and real) GSIs to address its routing entries, no?
>>
>> I suspect we will have to redesign this once we want to make use of
>> non-global invalidation.
> 
> IIUC here the problem is how we should manage the mapping between GSI
> routes and IRTE index (or device IDs, but let's talk later about
> device IDs, since we can map a device-id invalidation into several
> standalone index invalidations)? Or say, who should maintain this? IEC
> invalidation consumers (e.g., IRQFD logic, IOAPIC, ...), or IOMMU?
> 
> IMHO, I would prefer the consumers to maintain this, not IOMMU. So I
> would prefer a raw notification interface (index, mask, device-id,
> etc. rather than GSI route index), and the consumers are responsible
> to translate this message for their own sake.
> 
> The reason is simple: what if we have some other components (besides
> GSI routes) that will register to this notifier? Though I am not sure
> whether there would be one in the future, but letting IOMMU knowing
> about something like GSI route index is a little bit odd to me.
> 
> Take irqfd as an example, currently MSIRouteEntry is defined as:
> 
>     struct MSIRouteEntry {
>         PCIDevice *dev;             /* Device pointer */
>         int vector;                 /* MSI/MSIX vector index */
>         int virq;                   /* Virtual IRQ index */
>         QLIST_ENTRY(MSIRouteEntry) list;
>     };
> 
> When we want to support explicit IEC invalidation, we may need an
> extra field like:
> 
>         uint32_t index;             /* IRTE index */
> 
> So when MSI routes are invalidated, we can translated the raw index
> information into virq by simply looking up the MSIRouteEntry list.
> 
> Regarding to AMD's device-id interface...
> 
> I see that AMD IOMMUs do not have a global IRTE index table, not sure
> whether we can "define" one? E.g. IIUC AMD IOMMU IRTE will have 13
> bits index for each device, so how about making a global index like:
> 
>   device-id (16 bits) + 000b (3 bits) + index_per_device (13 bits)
> 
> to form a 32 bit index. So when AMD IOMMUs got a invalidation request,
> IOMMU can translate this per-device invalidation into several
> invalidations for specific IRTE entries? Not sure whether this would
> work.

Yes, there has to be a generic handle for each translation result an
IOMMU generated. This handle can be stored on the consumer side along
with the translation request. How a handle is generated should be
completely up to the IOMMU.

The consumer should receive a 32-bit (or more) opaque value with each
translation request (separate parameter) and then again on specific
invalidation. The latter case may also report a range of handles, to
make things more efficient (provided the consumer store those handles
close to each other).

Jan