From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: [PATCH 2/3] xen: Have schedulers revise initial placement Date: Sat, 16 Jul 2016 16:12:03 +0200 Message-ID: <1468678323.13039.132.camel@citrix.com> References: <1468605722-24239-1-git-send-email-george.dunlap@citrix.com> <1468605722-24239-2-git-send-email-george.dunlap@citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5740463975293191034==" Return-path: Received: from mail6.bemta14.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bOQKA-0001h3-TR for xen-devel@lists.xenproject.org; Sat, 16 Jul 2016 14:12:11 +0000 In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Andrew Cooper , George Dunlap , xen-devel@lists.xenproject.org Cc: Anshul Makkar , Meng Xu , Jan Beulich List-Id: xen-devel@lists.xenproject.org --===============5740463975293191034== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="=-EeA59SYOB2oyY5qG4UKx" --=-EeA59SYOB2oyY5qG4UKx Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, 2016-07-15 at 19:07 +0100, Andrew Cooper wrote: > On 15/07/16 19:02, George Dunlap wrote: > >=20 > > diff --git a/xen/common/sched_credit2.c > > b/xen/common/sched_credit2.c > > index 3b9aa27..5a04985 100644 > > --- a/xen/common/sched_credit2.c > > +++ b/xen/common/sched_credit2.c > > @@ -1620,15 +1620,23 @@ csched2_vcpu_insert(const struct scheduler > > *ops, struct vcpu *vc) > > =C2=A0 > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0BUG_ON(is_idle_vcpu(vc)); > > =C2=A0 > > +=C2=A0=C2=A0=C2=A0=C2=A0/* Locks in cpu_pick expect irqs to be disable= d */ > > +=C2=A0=C2=A0=C2=A0=C2=A0local_irq_disable(); > This doesn't make the problem much worse, but is there a plan to fix > this issue? >=20 There's a little bit more than a plan. I've got a proof of concept implementation which was working (now I need to refresh it), but for which I never really managed to evaluate the performance impact as accurately as I wanted to. In fact, I actually have a couple of variants implemented, that I was comparing against each others, in addition to against 'vanilla'. The problem was that I really was not seeing any impact at all, which looked strange (I was expecting improvement, at least on some workloads), and I wanted to investigate further. I'm leaving here the link to two branches, where I stashed some of the code that I have come up so far. As I said, it's WIP and needs refreshing and reworking. > None of the scheduler-accounting functions should be disabling > interrupts. >=20 They don't. But you can't keep irq disabled for some operations and enabled for others, on the same lock (because of the irq-safety spinlock checks/enforcement). So you have to always keep IRQ enabled, for all scheduling operations, which is ok for _almost_ all of them, with the only exception of the wakeup of a vcpu. So, the idea was to treat that one case specially, i.e., put the waking vcpus in a queue, and then drain the queue somehow. The insertion in the queue needs to be done disabling interrupts, but the draining --which is where the actual scheduling related hooks and operations are done-- can be done with IRQs on, which is what we want. What I was experimenting on was trying different ways of managing such a queue, e.g., only one queue for all CPUs or per-CPU queues; or whether to always drain the queue or only pick a couple of vcpu and defer the rest again; or whether to allow concurrent draining of the queue, or only have one CPU (at a time) doing that; etc etc. The "1 queue for all" and "per-CPU queues" is what's in the following two branches: git://xenbits.xen.org/people/dariof/xen.git =C2=A0wip/sched/irq-enabled http://xenbits.xen.org/gitweb/?p=3Dpeople/dariof/xen.git;a=3Dshortlog;h=3Dr= efs/heads/wip/sched/irq-enabled git://xenbits.xen.org/people/dariof/xen.git =C2=A0wip/sched/irq-enabled-per= cpu http://xenbits.xen.org/gitweb/?p=3Dpeople/dariof/xen.git;a=3Dshortlog;h=3Dr= efs/heads/wip/sched/irq-enabled-percpu I'll get back to this soon. In the meanwhile, feel free to comment, toss ideas, criticize, whatever. :-D Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-EeA59SYOB2oyY5qG4UKx Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJXikCzAAoJEBZCeImluHPu5tcP/jt1AaoFlwWa9xCVq+GbriB7 gSi9WcgmVFuYtoF78QKKW+fz4dglTBVFiB3u2JbP52C2AjT6Yh/MGr9sITrpqssP syCMr29TDUTGHo1dhqw0Z1FhhGlJE5P9RPHCefGKHIlu9U4ca2ImRCB4qiqWIZ9j urB/fHZknJcgQ4Vha0q3OkGEyO1m0dOFYdwylg9nUX/XcB6iXIEJeDqn4acJeQ82 UG458cafI5Eg/l6PwSeKs9RiMa+kNH2UnSQxYvVHsj8b1+8mfJNIPFFOvX7NHeLc IILO4WjsAEiByx5mRGO5bD3SbKw5UCyU+5W0gpAlipm1QFVRPXOAGv3ZaeCKWMh/ muFgQ7bIYX+23iW45Po8oJSielSQRPfWhzeSIR61YeKNgBT+15uLilVg8BTUGb5c jdGTLq70M0BFHQqO5G2hR5iYpg+9VeygJHxoLz+uAlGRCXXfYHNi6P6SPO/u+kkq 6jZBCrxSjJpSlo7Yyfz/0r3D8ry2xtwJDg38ca1VVXwtWL5BKPPca9+Tx0F7EPwP Yz5aLKl+5pgqXgj179dzG5NlmoeGNCfAFiQvTlvKjCJGh8gv3v06aYMYppbTAxGC gZK4ApUsmwfTHizMnyTF1hVgFHxaZicVDSuWB1Gg/6b2g2O6kXHzuxEgvsPY2iFD tTVnYaiSEBuQNgcV6XJC =QbeD -----END PGP SIGNATURE----- --=-EeA59SYOB2oyY5qG4UKx-- --===============5740463975293191034== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============5740463975293191034==--