From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: [PATCH 2/3] xen: Have schedulers revise initial placement Date: Tue, 19 Jul 2016 09:14:48 +0200 Message-ID: <1468912488.13039.180.camel@citrix.com> References: <1468605722-24239-1-git-send-email-george.dunlap@citrix.com> <1468605722-24239-2-git-send-email-george.dunlap@citrix.com> <1468678323.13039.132.camel@citrix.com> <6bfba010-3712-8d43-f866-38b3bfc1444f@citrix.com> <1468868106.13039.170.camel@citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5300620687471159228==" Return-path: Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bPPF8-00026X-AQ for xen-devel@lists.xenproject.org; Tue, 19 Jul 2016 07:15:02 +0000 In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Andrew Cooper , George Dunlap , xen-devel@lists.xenproject.org Cc: Anshul Makkar , Meng Xu , Jan Beulich List-Id: xen-devel@lists.xenproject.org --===============5300620687471159228== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="=-5k/D4LPqZ1jtenQ/QcCz" --=-5k/D4LPqZ1jtenQ/QcCz Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, 2016-07-18 at 22:36 +0100, Andrew Cooper wrote: > On 18/07/2016 19:55, Dario Faggioli wrote: > >=20 > > On Mon, 2016-07-18 at 19:10 +0100, Andrew Cooper wrote: > > >=20 > > > On 16/07/16 15:12, Dario Faggioli wrote: > > > >=20 > > > > On Fri, 2016-07-15 at 19:07 +0100, Andrew Cooper wrote: > > > > So you have to always keep IRQ enabled, for all scheduling > > > > operations, > > > > which is ok for _almost_ all of them, with the only exception > > > > of > > > > the > > > > wakeup of a vcpu. > > > I know that it is all or nothing.=C2=A0=C2=A0What specific action abo= ut > > > waking > > > a > > > vcpu requires holding a lock? > > >=20 > > > If it is simply re-queueing the vcpu onto the runable queue, > > > there > > > are a > > > number of lockless queuing algorithms which can be used. > > >=20 > > Yes, it's vcpu_wake() that does vcpu_schedule_lock_irqsave() > > before, > > among other things, calling SCHED_OP(wake, v). > Right - this looks easy to fix.=C2=A0=C2=A0Use a per-pcpu single linked l= ist, > which can be mutated safely with cmpxchg() rather than locks, have > vcpu_wake() add v to the linked list, and schedule itself a > SCHEDULE_SOFTIRQ, (possibly a new SCHEDULE_WAKE_SOFTIRQ with higher > priority than SCHEDULE_SOFTIRQ). >=20 That's exactly what I'm doing in one of the variant I've implemented so far (with a dedicated lock, rather than cmpxchg, but I was indeed looking at removing that): http://xenbits.xen.org/gitweb/?p=3Dpeople/dariof/xen.git;a=3Dblobdiff;f=3Dx= en/include/xen/softirq.h;h=3De62c247fdc41dd4e36ceb5f8094647fa62530c56;hp=3D= 0895a162031b64ad6acff6be9d17707dad7a89bf;hb=3D431423e69c9fe4503d26bd4c65769= 9284e0223b7;hpb=3Da282bcd6f7751d2ff428ca6c1e14120aa6fcc5fc :-) > Then in the schedule softirq can take the vcpu schedule lock without > disabling interrupts and run the internals of what is currently > vcpu_wake().=C2=A0=C2=A0The current content of vcpu_wake() very large for= what > is > typically executed from interrupt context. >=20 Totally agree. Something I was reasoning on, and trying to assess by means of benchmarks is, for instance, when taking out the vcpus from the queue and waking them, whether to do that always "to completion" (considering that it's probably even possible that new vcpus are added to the queue itself while I'm trying to drain it), or in batches (and after each batch, let Xen do something else and re-trigger the softirq). And this (and other similar "subtleties") is where the numbers I could get were not conclusive. I'll dust off the code and let you know. :-) Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-5k/D4LPqZ1jtenQ/QcCz Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJXjdNqAAoJEBZCeImluHPu0ikP/2IBmogrEmTI8L99bxaqJyBn 7B/rg/i2ID4SWQx41dQbzc3nOrIB3XJQW9VE9PF0xL7KHnyRMAnAn1NJPf8v92K+ A1wWq1oC76vr2bq/5HkD0Ha96Ari6ySzbAmwxiGmx8RdjQeut+BghaccOHfCXnA3 JcbdvNUTFMh84BKeg+RxIxVZssSL1AYFQ9TtoPYSk6KWOgaSyzG7XRdr8nz8Pduy zKs2du0/gX4h4Ex607wmHTLWexYfpLYZjkc65itsWdf5Ssqt3rBsI2XjJxhWSRZL WuHrcyZqKXEFT1ppuv1IT1/zMtGNfmMmMeagGWDKVgpN5HpRocHiaEOyHugWvHMU p2NWX9YSBbIrCGsuOZw8TBAPQoEtdIH3SsiA/GrwUMlp6GN9vdNrD2a7gcjBDmny cxdapYaoSoK/u0BNkeDMvF+3O1ouN2MZuHERilADCORuOxg328fpnWAoaOOMlrEO bPj7d31ov3xRyokbUXEuj1d5v8EW9N1yVZK0Mf1tzOLa7rahg2YfIug5l90ZEAuM ixqRWaGepPn672wtEo+e4qB+xaK5984ikrT3hwXKnzNIiKaMlb7+VhnTvdx+1HP7 eFmBI/+8iV5aHxk81RdaV5elLwq4Du7E8ol63MdPTGL4u4B7fUazmn3kAYXAYc5c ryPIihb5iWOc4iqhCD/f =d2N5 -----END PGP SIGNATURE----- --=-5k/D4LPqZ1jtenQ/QcCz-- --===============5300620687471159228== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============5300620687471159228==--