Re: [RFC PATCH 21/24] ARM: vITS: handle INVALL command

[George Dunlap <George.Dunlap@citrix.com>]

> On Dec 10, 2016, at 4:18 AM, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Fri, 9 Dec 2016, Andre Przywara wrote:
>> On 07/12/16 20:20, Stefano Stabellini wrote:
>>> On Tue, 6 Dec 2016, Julien Grall wrote:
>>>> On 06/12/2016 22:01, Stefano Stabellini wrote:
>>>>> On Tue, 6 Dec 2016, Stefano Stabellini wrote:
>>>>>> moving a vCPU with interrupts assigned to it is slower than moving a
>>>>>> vCPU without interrupts assigned to it. You could say that the
>>>>>> slowness is directly proportional do the number of interrupts assigned
>>>>>> to the vCPU.
>>>>> 
>>>>> To be pedantic, by "assigned" I mean that a physical interrupt is routed
>>>>> to a given pCPU and is set to be forwarded to a guest vCPU running on it
>>>>> by the _IRQ_GUEST flag. The guest could be dom0. Upon receiving one of
>>>>> these physical interrupts, a corresponding virtual interrupt (could be a
>>>>> different irq) will be injected into the guest vCPU.
>>>>> 
>>>>> When the vCPU is migrated to a new pCPU, the physical interrupts that
>>>>> are configured to be injected as virtual interrupts into the vCPU, are
>>>>> migrated with it. The physical interrupt migration has a cost. However,
>>>>> receiving physical interrupts on the wrong pCPU has an higher cost.
>>>> 
>>>> I don't understand why it is a problem for you to receive the first interrupt
>>>> to the wrong pCPU and moving it if necessary.
>>>> 
>>>> While this may have an higher cost (I don't believe so) on the first received
>>>> interrupt, migrating thousands of interrupts at the same time is very
>>>> expensive and will likely get Xen stuck for a while (think about ITS with a
>>>> single command queue).
>>>> 
>>>> Furthermore, the current approach will move every single interrupt routed a
>>>> the vCPU, even those disabled. That's pointless and a waste of resource. You
>>>> may argue that we can skip the ones disabled, but in that case what would be
>>>> the benefits to migrate the IRQs while migrate the vCPUs?
>>>> 
>>>> So I would suggest to spread it over the time. This also means less headache
>>>> for the scheduler developers.
>>> 
>>> The most important aspect of interrupts handling in Xen is latency,
>>> measured as the time between Xen receiving a physical interrupt and the
>>> guest receiving it. This latency should be both small and deterministic.
>>> 
>>> We all agree so far, right?
>>> 
>>> 
>>> The issue with spreading interrupts migrations over time is that it makes
>>> interrupt latency less deterministic. It is OK, in the uncommon case of
>>> vCPU migration with interrupts, to take a hit for a short time. This
>>> "hit" can be measured. It can be known. If your workload cannot tolerate
>>> it, vCPUs can be pinned. It should be a rare event anyway. On the other
>>> hand, by spreading interrupts migrations, we make it harder to predict
>>> latency. Aside from determinism, another problem with this approach is
>>> that it ensures that every interrupt assigned to a vCPU will first hit
>>> the wrong pCPU, then it will be moved. It guarantees the worst-case
>>> scenario for interrupt latency for the vCPU that has been moved. If we
>>> migrated all interrupts as soon as possible, we would minimize the
>>> amount of interrupts delivered to the wrong pCPU. Most interrupts would
>>> be delivered to the new pCPU right away, reducing interrupt latency.

OK, so ultimately for each interrupt we can take an “eager” approach and move it as soon as the vcpu moves, or a “lazy” approach and move it after it fires.

The two options which have been discussed are:
1. Always take an eager approach, and try to tell the scheduler to limit the migration frequency for these vcpus more than others
2. Always take a lazy approach, and leave the scheduler the way it is.

Another approach which one might take:
3. Eagerly migrate a subset of the interrupts and lazily migrate the others.  For instance, we could eagerly migrate all the interrupts which have fired since the last vcpu migration.  In a system where migrations happen frequently, this should only be a handful; in a system that migrates infrequently, this will be more, but it won’t matter, because it will happen less often.

Workloads which need predictable IRQ latencies should probably be pinning their vcpus anyway.

So at the moment, the scheduler already tries to avoid migrating things *a little bit* if it can (see migrate_resist).  It’s not clear to me at the moment whether this is enough or not.  Or to put it a different way — how long should the scheduler try to wait before moving one of these vcpus?  At the moment I haven’t seen a good way of calculating this.

#3 to me has the feeling of being somewhat more satisfying, but also potentially fairly complicated.  Since the scheduler already does migration resistance somewhat, #1 would be a simpler to implement in the sort run.  If it turns out that #1 has other drawbacks, we can implement #3 as and when needed.

Thoughts?

 -George
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel