From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: Strange interdependace between domains Date: Fri, 14 Feb 2014 00:13:18 +0100 Message-ID: <1392333198.32038.153.camel@Solace> References: <1646915994.20140213165604@gmail.com> <1392313015.32038.112.camel@Solace> <295276356.20140213222507@gmail.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8656291078796531798==" Return-path: In-Reply-To: <295276356.20140213222507@gmail.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Simon Martin Cc: Andrew Cooper , Nate Studer , Don Slutz , xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org --===============8656291078796531798== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-R32yVRGz99RkEPtHT6pT" --=-R32yVRGz99RkEPtHT6pT Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On gio, 2014-02-13 at 22:25 +0000, Simon Martin wrote: > Thanks for all the replies guys. >=20 :-) > Don> How many instruction per second a thread gets does depend on the > Don> "idleness" of other threads (no longer just the hyperThread's > Don> parther). >=20 > This seems a bit strange to me. In my case I have time > critical PV running by itself in a CPU pool. So Xen should not be > scheduling it, so I can't see how this Hypervisor thread would be affecte= d. >=20 I think Don is referring to the idleness of the other _hardware_ threads in the chip, rather than software threads of execution, either in Xen or in Dom0/DomU. I checked his original e-mail and, AFAIUI, he seems to confirm that the throughput you get on, say, core 3, depends on what it's sibling core (which really is his sibling hyperthread, again in the hardware sense... Gah, the terminology is just a mess! :-P). He seems to also add the fact that there is a similar kind of inter-dependency between all the hardware hyperthread, not just between siblings. Does this make sense Don? > >> 6.- All VCPUs are pinned: > >>=20 > Dario> Right, although, if you use cpupools, and if I've understood what = you're > Dario> up to, you really should not require pinning. I mean, the isolatio= n > Dario> between the RT-ish domain and the rest of the world should be alre= ady in > Dario> place thanks to cpupools. >=20 > This is what I thought, however when running looking at the vcpu-list > I CPU affinity was "all" until I starting pinning. As I wasn't sure > whether that was "all inside this cpu pool" or "all" I felt it was > safer to do it explicitly. >=20 Actually, you are right, we could put things in a way that results more clear, when one observes the output! So, I confirm that, despite the fact that you see "all", that all is relative to the cpupool the domain is assigned to. I'll try to think on how to make this more evident... A note in the manpage and/or the various sources of documentation, is the easy (but still necessary, I agree) part, and I'll add this to my TODO list. Actually modifying the output is more tricky, as affinity and cpupools are orthogonal by design, and that is the right (IMHO) thing. I guess trying to tweak the printf()-s in `xl vcpu-list' would not be that hard... I'll have a look and see if I can come up with a proposal. > Dario> So, if you ask me, you're restricting too much things in > Dario> pool-0, where dom0 and the Windows VM runs. In fact, is there a > Dario> specific reason why you need all their vcpus to be statically > Dario> pinned each one to only one pcpu? If not, I'd leave them a > Dario> little bit more of freedom. >=20 > I agree with you here, however when I don't pin CPU affinity is "all". > Is this "all in the CPU pool"? I couldn't find that info. >=20 Again, yes: once a domain is in a cpupool, no matter what its affinity says, it won't ever reach a pcpu assigned to another cpupool. The technical reason is that each cpupool is ruled by it's own (copy of a) scheduler, even if you use, e.g., credit, for both/all the pools. In that case, what you will get are two full instances of credit, completely independent between each other, each one in charge only of a very specific subset of pcpus (as mandated by cpupools). So, different runqueues, different data structures, different anything. > Dario> What I'd try is: > Dario> 1. all dom0 and win7 vcpus free, so no pinning in pool0. > Dario> 2. pinning as follows: > Dario> * all vcpus of win7 --> pcpus 1,2 > Dario> * all vcpus of dom0 --> no pinning > Dario> this way, what you get is the following: win7 could suffer some= times, > Dario> if all its 3 vcpus gets busy, but that, I think is acceptable, = at > Dario> least up to a certain extent, is that the case? > Dario> At the same time, you > Dario> are making sure dom0 always has a chance to run, as pcpu#0 woul= d be > Dario> his exclusive playground, in case someone, including your pv499 > Dario> domain, needs its services. >=20 > This is what I had when I started :-). Thanks for the confirmation > that I was doing it right. However if the hyperthreading is the issue, > then I will only have 2 PCPU available, and I will assign them both to > dom0 and win7. >=20 Yes, with hyperthreading in mind, that is what you should do. Once we will have confirmed that hyperthreading is the issue, we'll see what we can do. I mean, if, in your case, it's fine to 'waste' a cpu, then ok, but I think we need a general solution for this... Perhaps with a little worse performances than just leaving one core/hyperthread completely idle, but at the same time more resource efficient. I wonder how tweaking the sched_smt_power_savings would deal with this... > Dario> Right. Are you familiar with tracing what happens inside Xen > Dario> with xentrace and, perhaps, xenalyze? It takes a bit of time to > Dario> get used to it but, once you dominate it, it is a good mean for > Dario> getting out really useful info! >=20 > Dario> There is a blog post about that here: > Dario> http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and= -xenalyze/ > Dario> and it should have most of the info, or the links to where to > Dario> find them. >=20 > Thanks for this. If this problem is more than the hyperthreading then > I will definitely use it. Also looks like it might be useful when I > start looking at the jitter on the singleshot timer (which should be > in a couple of weeks). >=20 It will reveal to be very useful for that, I'm sure! :-) Let us know how the re-testing goes. Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-R32yVRGz99RkEPtHT6pT Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEABECAAYFAlL9UY4ACgkQk4XaBE3IOsR6WgCcCh5aPVng7s4eO3IKhoLrmYX4 x0gAnikeDy2TZdlfkjncpJNCipcShxQR =0ti7 -----END PGP SIGNATURE----- --=-R32yVRGz99RkEPtHT6pT-- --===============8656291078796531798== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============8656291078796531798==--