Re: [RFC PATCH 21/24] ARM: vITS: handle INVALL command

[Dario Faggioli <dario.faggioli@citrix.com>]

[-- Attachment #1.1: Type: text/plain, Size: 5013 bytes --]

On Wed, 2016-12-14 at 03:39 +0100, George Dunlap wrote:
> > On Dec 10, 2016, at 4:18 AM, Stefano Stabellini <sstabellini@kernel
> > .org> wrote:
> > > > The issue with spreading interrupts migrations over time is
> > > > that it makes
> > > > interrupt latency less deterministic. It is OK, in the uncommon
> > > > case of
> > > > vCPU migration with interrupts, to take a hit for a short time.
> > > > This
> > > > "hit" can be measured. It can be known. If your workload cannot
> > > > tolerate
> > > > it, vCPUs can be pinned. It should be a rare event anyway. On
> > > > the other
> > > > hand, by spreading interrupts migrations, we make it harder to
> > > > predict
> > > > latency. Aside from determinism, another problem with this
> > > > approach is
> > > > that it ensures that every interrupt assigned to a vCPU will
> > > > first hit
> > > > the wrong pCPU, then it will be moved. It guarantees the worst-
> > > > case
> > > > scenario for interrupt latency for the vCPU that has been
> > > > moved. If we
> > > > migrated all interrupts as soon as possible, we would minimize
> > > > the
> > > > amount of interrupts delivered to the wrong pCPU. Most
> > > > interrupts would
> > > > be delivered to the new pCPU right away, reducing interrupt
> > > > latency.
> 
> Another approach which one might take:
> 3. Eagerly migrate a subset of the interrupts and lazily migrate the
> others.  For instance, we could eagerly migrate all the interrupts
> which have fired since the last vcpu migration.  In a system where
> migrations happen frequently, this should only be a handful; in a
> system that migrates infrequently, this will be more, but it won’t
> matter, because it will happen less often.
> 
Yes, if doable (e.g., I don't know how easy and practical is to know
and keep track of fired interrupts) this looks a good solution to me
too.

> So at the moment, the scheduler already tries to avoid migrating
> things *a little bit* if it can (see migrate_resist).  It’s not clear
> to me at the moment whether this is enough or not.  
>
Well, true, but migration resistance, in Credit2, is just a fixed value
which:
 1. is set at boot time;
 2. is always the same for all vcpus;
 3. is always the same, no matter what a vcpu is doing.

And even if we make it tunable and changeable at runtime (which I
intend to do), it's still something pretty "static" because of 2 and 3.

And even if we make it tunable per-vcpu (which is doable), it would be
rather hard to decide to what value to set it, for each vcpu. And, of
course, 3 would still apply (i.e., it would change according to the
vcpu workload or characteristics).

So, it's guessing. More or less fine grained, but always guessing.

On the other hand, using something proportional to nr. of routed
interrupt as the migration resistance threshold would overcome all 1, 2
and 3. It would give us a migrate_resist value which is adaptive, and
is determined according to actual workload of properties of a specific
vcpu.
Feeding routed interrupt info to the load balancer comes from similar
reasoning (and we actually may want to do both).

FTR, Credit1 has a similar mechanism, i.e., it *even wilded guesses*
whether a vcpu could still have some of its data in cache, and tries
not to migrate it if it's likely (see __csched_vcpu_is_cache_hot()).
We can improve that too, although it is a lot more complex and less
predictable, as usual with Credit1.

> Or to put it a different way — how long should the scheduler try to
> wait before moving one of these vcpus?  
>
Yep, it's similar to the "anticipation" problem in I/O schedulers
(where "excessive seeks" ~= "too frequent migrations").

 https://en.wikipedia.org/wiki/Anticipatory_scheduling

> At the moment I haven’t seen a good way of calculating this.
> 
Exactly, and basing the calculation on the number of routed interrupt
--and, if possible, other metrics too-- could be that "good way" we're
looking for.

It would need experimenting, of course, but I like the idea.

> #3 to me has the feeling of being somewhat more satisfying, but also
> potentially fairly complicated.  Since the scheduler already does
> migration resistance somewhat, #1 would be a simpler to implement in
> the sort run.  If it turns out that #1 has other drawbacks, we can
> implement #3 as and when needed.
> 
> Thoughts?
> 
Yes, we can do things incrementally, which is always good. I like your
#1 proposal because it has the really positive side effect of bringing
us in the camp of adaptive migration resistance, which is something
pretty advanced and pretty cool, if we manage to do it right. :-)

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel