On Mon, 2016-07-18 at 22:36 +0100, Andrew Cooper wrote: > On 18/07/2016 19:55, Dario Faggioli wrote: > > > > On Mon, 2016-07-18 at 19:10 +0100, Andrew Cooper wrote: > > > > > > On 16/07/16 15:12, Dario Faggioli wrote: > > > > > > > > On Fri, 2016-07-15 at 19:07 +0100, Andrew Cooper wrote: > > > > So you have to always keep IRQ enabled, for all scheduling > > > > operations, > > > > which is ok for _almost_ all of them, with the only exception > > > > of > > > > the > > > > wakeup of a vcpu. > > > I know that it is all or nothing.  What specific action about > > > waking > > > a > > > vcpu requires holding a lock? > > > > > > If it is simply re-queueing the vcpu onto the runable queue, > > > there > > > are a > > > number of lockless queuing algorithms which can be used. > > > > > Yes, it's vcpu_wake() that does vcpu_schedule_lock_irqsave() > > before, > > among other things, calling SCHED_OP(wake, v). > Right - this looks easy to fix.  Use a per-pcpu single linked list, > which can be mutated safely with cmpxchg() rather than locks, have > vcpu_wake() add v to the linked list, and schedule itself a > SCHEDULE_SOFTIRQ, (possibly a new SCHEDULE_WAKE_SOFTIRQ with higher > priority than SCHEDULE_SOFTIRQ). > That's exactly what I'm doing in one of the variant I've implemented so far (with a dedicated lock, rather than cmpxchg, but I was indeed looking at removing that): http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=blobdiff;f=xen/include/xen/softirq.h;h=e62c247fdc41dd4e36ceb5f8094647fa62530c56;hp=0895a162031b64ad6acff6be9d17707dad7a89bf;hb=431423e69c9fe4503d26bd4c657699284e0223b7;hpb=a282bcd6f7751d2ff428ca6c1e14120aa6fcc5fc :-) > Then in the schedule softirq can take the vcpu schedule lock without > disabling interrupts and run the internals of what is currently > vcpu_wake().  The current content of vcpu_wake() very large for what > is > typically executed from interrupt context. > Totally agree. Something I was reasoning on, and trying to assess by means of benchmarks is, for instance, when taking out the vcpus from the queue and waking them, whether to do that always "to completion" (considering that it's probably even possible that new vcpus are added to the queue itself while I'm trying to drain it), or in batches (and after each batch, let Xen do something else and re-trigger the softirq). And this (and other similar "subtleties") is where the numbers I could get were not conclusive. I'll dust off the code and let you know. :-) Regards, Dario -- <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)