Strange interdependace between domains

Strange interdependace between domains

[Simon Martin]

Hi all,

Sorry, I just sent this without a subject. Here it is with the subject
as it should be.

I  am  now successfully running my little operating system inside Xen.
It  is  fully  preemptive and working a treat, but I have just noticed
something  I  wasn't expecting, and will really be a problem for me if
I can't work around it.

My configuration is as follows:

1.- Hardware: Intel i3, 4GB RAM, 64GB SSD.

2.- Xen: 4.4 (just pulled from repository)

3.- Dom0: Debian Wheezy (Kernel 3.2)

4.- 2 cpu pools:

# xl cpupool-list
Name               CPUs   Sched     Active   Domain count
Pool-0               3    credit       y          2
pv499                1  arinc653       y          1

5.- 2 domU:

# xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   984     3     r-----      39.7
win7x64                                      1  2046     3     -b----     143.0
pv499                                        3   128     1     -b----      61.2

6.- All VCPUs are pinned:

# xl vcpu-list
Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
Domain-0                             0     0    0   -b-      27.5  0
Domain-0                             0     1    1   -b-       7.2  1
Domain-0                             0     2    2   r--       5.1  2
win7x64                              1     0    0   -b-      71.6  0
win7x64                              1     1    1   -b-      37.7  1
win7x64                              1     2    2   -b-      34.5  2
pv499                                3     0    3   -b-      62.1  3

7.- pv499 is the domU that I am testing. It has no disk or vif devices
(yet). I am running a little test program in pv499 and the timing I
see is varies depending on disk activity.

My test program runs prints up the time taken in milliseconds for a
million cycles. With no disk activity I see 940 ms, with disk activity
I see 1200 ms.

I can't understand this as disk activity should be running on cores 0,
1  and 2, but never on core 3. The only thing running on core 3 should
by my paravirtual machine and the hypervisor stub.

Any idea what's going on?


-- 
Best regards,
 Simon                          mailto:furryfuttock@gmail.com

Re: Strange interdependace between domains

[Ian Campbell]

On Thu, 2014-02-13 at 16:56 +0000, Simon Martin wrote:
> I can't understand this as disk activity should be running on cores 0,
> 1  and 2, but never on core 3. The only thing running on core 3 should
> by my paravirtual machine and the hypervisor stub.
> 
> Any idea what's going on?

Is core 3 actual a hyperthread -- IOW it is sharing processor execution
resources with e.g. core 2. Or shared L2 caches etc.

Ian.

Re: Strange interdependace between domains

[Simon Martin]

> On Thu, 2014-02-13 at 16:56 +0000, Simon Martin wrote:
>> I can't understand this as disk activity should be running on cores 0,
>> 1  and 2, but never on core 3. The only thing running on core 3 should
>> by my paravirtual machine and the hypervisor stub.
>> 
>> Any idea what's going on?

> Is core 3 actual a hyperthread -- IOW it is sharing processor execution
> resources with e.g. core 2. Or shared L2 caches etc.

> Ian.

Thanks  Ian.  Very good point. It is a hyperthread. I will reconfigure
tomorrow, retest and let you know, but that make sense.

Regards.


-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Strange interdependace between domains

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 989 bytes --]

On gio, 2014-02-13 at 17:28 +0000, Simon Martin wrote:
> > On Thu, 2014-02-13 at 16:56 +0000, Simon Martin wrote:

> > Is core 3 actual a hyperthread -- IOW it is sharing processor execution
> > resources with e.g. core 2. Or shared L2 caches etc.
> 
> > Ian.
> 
> Thanks  Ian.  Very good point. It is a hyperthread. I will reconfigure
> tomorrow, retest and let you know, but that make sense.
> 
It does indeed make sense (much more than what I was saying,
probably :-D)...

Make sure you leave the sibling thread of the one where you pin the vcpu
of your domain completely free (by pinning the other domains' vcpus
everywhere but there), and yes, let us know if that change the results.

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Strange interdependace between domains

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 5440 bytes --]

On gio, 2014-02-13 at 16:56 +0000, Simon Martin wrote:
> Hi all,
> 
Hey Simon!

First of all, as you're using ARINC, I'm adding Nate, as he's ARINC's
maintainer, let's see if he can help us! ;-P

> I  am  now successfully running my little operating system inside Xen.
> It  is  fully  preemptive and working a treat, 
>
Aha, this is great! :-)

> but I have just noticed
> something  I  wasn't expecting, and will really be a problem for me if
> I can't work around it.
> 
Well, let's see...

> My configuration is as follows:
> 
> 1.- Hardware: Intel i3, 4GB RAM, 64GB SSD.
> 
> 2.- Xen: 4.4 (just pulled from repository)
> 
> 3.- Dom0: Debian Wheezy (Kernel 3.2)
> 
> 4.- 2 cpu pools:
> 
> # xl cpupool-list
> Name               CPUs   Sched     Active   Domain count
> Pool-0               3    credit       y          2
> pv499                1  arinc653       y          1
> 
Ok, I think I figured this out from the other information, but it would
be useful to know what pcpus are assigned to what cpupool. I think it's
`xl cpupool-list -c'.

> 5.- 2 domU:
> 
> # xl list
> Name                                        ID   Mem VCPUs      State   Time(s)
> Domain-0                                     0   984     3     r-----      39.7
> win7x64                                      1  2046     3     -b----     143.0
> pv499                                        3   128     1     -b----      61.2
> 
> 6.- All VCPUs are pinned:
> 
Right, although, if you use cpupools, and if I've understood what you're
up to, you really should not require pinning. I mean, the isolation
between the RT-ish domain and the rest of the world should be already in
place thanks to cpupools.

Actually, pinning can help, but meybe not in the exact way you're using
it...

> # xl vcpu-list
> Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
> Domain-0                             0     0    0   -b-      27.5  0
> Domain-0                             0     1    1   -b-       7.2  1
> Domain-0                             0     2    2   r--       5.1  2
> win7x64                              1     0    0   -b-      71.6  0
> win7x64                              1     1    1   -b-      37.7  1
> win7x64                              1     2    2   -b-      34.5  2
> pv499                                3     0    3   -b-      62.1  3
> 
...as it can be seen here.

So, if you ask me, you're restricting too much things in pool-0, where
dom0 and the Windows VM runs. In fact, is there a specific reason why
you need all their vcpus to be statically pinned each one to only one
pcpu? If not, I'd leave them a little bit more of freedom.

What I'd try is:
 1. all dom0 and win7 vcpus free, so no pinning in pool0.
 2. pinning as follows:
     * all vcpus of win7 --> pcpus 1,2
     * all vcpus of dom0 --> no pinning
   this way, what you get is the following: win7 could suffer sometimes,
   if all its 3 vcpus gets busy, but that, I think is acceptable, at
   least up to a certain extent, is that the case?
   At the same time, you
   are making sure dom0 always has a chance to run, as pcpu#0 would be
   his exclusive playground, in case someone, including your pv499
   domain, needs its services.

> 7.- pv499 is the domU that I am testing. It has no disk or vif devices
> (yet). I am running a little test program in pv499 and the timing I
> see is varies depending on disk activity.
> 
> My test program runs prints up the time taken in milliseconds for a
> million cycles. With no disk activity I see 940 ms, with disk activity
> I see 1200 ms.
> 
Wow, it's very hard to tell. What I first thought is that your domain
may need something from dom0, and the suboptimal (IMHO) pinning
configuration you're using could be slowing that down. The bug in this
theory is that dom0 services are mostly PV drivers for disk and network,
which you say you don't have...

I still think your pinning setup is unnecessary restrictive, so I'd give
it a try, but it's probably not the root cause of your issue.

> I can't understand this as disk activity should be running on cores 0,
> 1  and 2, but never on core 3. The only thing running on core 3 should
> by my paravirtual machine and the hypervisor stub.
> 
Right. Are you familiar with tracing what happens inside Xen with
xentrace and, perhaps, xenalyze? It takes a bit of time to get used to
it but, once you dominate it, it is a good mean for getting out really
useful info!

There is a blog post about that here:
http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/
and it should have most of the info, or the links to where to find them.

It's going to be a lot of data, but if you trace one run without disk IO
and one run with disk IO, it should be doable to compare the
differences, for instance, in terms of when the vcpus of your domain are
active, as well as when they get scheduled, and from that we hopefully
can try to narrow down a bit more the real root cause of the thing.

Let us know if you think you need help with that.

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Strange interdependace between domains

[Nate Studer]

On 2/13/2014 12:36 PM, Dario Faggioli wrote:
> On gio, 2014-02-13 at 16:56 +0000, Simon Martin wrote:
>> Hi all,
>>
> Hey Simon!
> 
> First of all, as you're using ARINC, I'm adding Nate, as he's ARINC's
> maintainer, let's see if he can help us! ;-P
> 
>> I  am  now successfully running my little operating system inside Xen.
>> It  is  fully  preemptive and working a treat, 
>>
> Aha, this is great! :-)
> 
>> but I have just noticed
>> something  I  wasn't expecting, and will really be a problem for me if
>> I can't work around it.
>>
> Well, let's see...
> 
>> My configuration is as follows:
>>
>> 1.- Hardware: Intel i3, 4GB RAM, 64GB SSD.
>>
>> 2.- Xen: 4.4 (just pulled from repository)
>>
>> 3.- Dom0: Debian Wheezy (Kernel 3.2)
>>
>> 4.- 2 cpu pools:
>>
>> # xl cpupool-list
>> Name               CPUs   Sched     Active   Domain count
>> Pool-0               3    credit       y          2
>> pv499                1  arinc653       y          1
>>
> Ok, I think I figured this out from the other information, but it would
> be useful to know what pcpus are assigned to what cpupool. I think it's
> `xl cpupool-list -c'.
> 
>> 5.- 2 domU:
>>
>> # xl list
>> Name                                        ID   Mem VCPUs      State   Time(s)
>> Domain-0                                     0   984     3     r-----      39.7
>> win7x64                                      1  2046     3     -b----     143.0
>> pv499                                        3   128     1     -b----      61.2
>>
>> 6.- All VCPUs are pinned:
>>
> Right, although, if you use cpupools, and if I've understood what you're
> up to, you really should not require pinning. I mean, the isolation
> between the RT-ish domain and the rest of the world should be already in
> place thanks to cpupools.
> 
> Actually, pinning can help, but meybe not in the exact way you're using
> it...
> 
>> # xl vcpu-list
>> Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
>> Domain-0                             0     0    0   -b-      27.5  0
>> Domain-0                             0     1    1   -b-       7.2  1
>> Domain-0                             0     2    2   r--       5.1  2
>> win7x64                              1     0    0   -b-      71.6  0
>> win7x64                              1     1    1   -b-      37.7  1
>> win7x64                              1     2    2   -b-      34.5  2
>> pv499                                3     0    3   -b-      62.1  3
>>
> ...as it can be seen here.
> 
> So, if you ask me, you're restricting too much things in pool-0, where
> dom0 and the Windows VM runs. In fact, is there a specific reason why
> you need all their vcpus to be statically pinned each one to only one
> pcpu? If not, I'd leave them a little bit more of freedom.
> 
> What I'd try is:
>  1. all dom0 and win7 vcpus free, so no pinning in pool0.
>  2. pinning as follows:
>      * all vcpus of win7 --> pcpus 1,2
>      * all vcpus of dom0 --> no pinning
>    this way, what you get is the following: win7 could suffer sometimes,
>    if all its 3 vcpus gets busy, but that, I think is acceptable, at
>    least up to a certain extent, is that the case?
>    At the same time, you
>    are making sure dom0 always has a chance to run, as pcpu#0 would be
>    his exclusive playground, in case someone, including your pv499
>    domain, needs its services.
> 
>> 7.- pv499 is the domU that I am testing. It has no disk or vif devices
>> (yet). I am running a little test program in pv499 and the timing I
>> see is varies depending on disk activity.
>>
>> My test program runs prints up the time taken in milliseconds for a
>> million cycles. With no disk activity I see 940 ms, with disk activity
>> I see 1200 ms.
>>
> Wow, it's very hard to tell. What I first thought is that your domain
> may need something from dom0, and the suboptimal (IMHO) pinning
> configuration you're using could be slowing that down. The bug in this
> theory is that dom0 services are mostly PV drivers for disk and network,
> which you say you don't have...

  Any shared resource between domains could cause one domain to affect the
timing of another domain:  shared cache, shared memory controller, interrupts,
shared I/O interface, domain-0, etc...

  Giving the small size and nature of your test application, the cache, which
Ian mentioned, is a likely culprit, but if your application gets more complex
some of these other sources could show up.

  What kind of variation (jitter) on this measurement can your application handle?

  You can never eliminate all the jitter, but you can get rid of a lot of it by
carefully partitioning your system.  The pinning suggested by Dario looks to be
a good step towards this goal.

     Nate

> 
> I still think your pinning setup is unnecessary restrictive, so I'd give
> it a try, but it's probably not the root cause of your issue.
> 
>> I can't understand this as disk activity should be running on cores 0,
>> 1  and 2, but never on core 3. The only thing running on core 3 should
>> by my paravirtual machine and the hypervisor stub.
>>
> Right. Are you familiar with tracing what happens inside Xen with
> xentrace and, perhaps, xenalyze? It takes a bit of time to get used to
> it but, once you dominate it, it is a good mean for getting out really
> useful info!
> 
> There is a blog post about that here:
> http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/
> and it should have most of the info, or the links to where to find them.
> 
> It's going to be a lot of data, but if you trace one run without disk IO
> and one run with disk IO, it should be doable to compare the
> differences, for instance, in terms of when the vcpus of your domain are
> active, as well as when they get scheduled, and from that we hopefully
> can try to narrow down a bit more the real root cause of the thing.
> 
> Let us know if you think you need help with that.
> 
> Regards,
> Dario
> 

Re: Strange interdependace between domains

[Simon Martin]

Thanks for all the replies guys.

With respect your comments/queries I'll put them all in here.

>> 1.- Hardware: Intel i3, 4GB RAM, 64GB SSD.
Andrew> Can you be more specific - this covers 4 generations of Intel CPUs.

This is a 4th gen (Haswell) processor.

>> # xl cpupool-list
>> Name               CPUs   Sched     Active   Domain count
>> Pool-0               3    credit       y          2
>> pv499                1  arinc653       y          1
Dario> Ok, I think I figured this out from the other information, but it would
Dario> be useful to know what pcpus are assigned to what cpupool. I think it's
Dario> `xl cpupool-list -c'.

Pool-0: 0,1,2
Dom0: 3

Don> How many instruction per second a thread gets does depend on the
Don> "idleness" of other threads (no longer just the hyperThread's
Don> parther).

This    seems    a    bit    strange   to   me. In my case I have time
critical  PV  running  by  itself  in a CPU pool. So Xen should not be
scheduling it, so I can't see how this Hypervisor thread would be affected.

>> 6.- All VCPUs are pinned:
>> 
Dario> Right, although, if you use cpupools, and if I've understood what you're
Dario> up to, you really should not require pinning. I mean, the isolation
Dario> between the RT-ish domain and the rest of the world should be already in
Dario> place thanks to cpupools.

This  is what I thought, however when running looking at the vcpu-list
I  CPU  affinity  was "all" until I starting pinning. As I wasn't sure
whether  that  was  "all  inside this cpu pool" or "all" I felt it was
safer to do it explicitly.

Dario> So, if you ask me, you're restricting too much things in
Dario> pool-0, where dom0 and the Windows VM runs. In fact, is there a
Dario> specific reason why you need all their vcpus to be statically
Dario> pinned each one to only one pcpu? If not, I'd leave them a
Dario> little bit more of freedom.

I agree with you here, however when I don't pin CPU affinity is "all".
Is this "all in the CPU pool"? I couldn't find that info.

Dario> What I'd try is:
Dario>  1. all dom0 and win7 vcpus free, so no pinning in pool0.
Dario>  2. pinning as follows:
Dario>      * all vcpus of win7 --> pcpus 1,2
Dario>      * all vcpus of dom0 --> no pinning
Dario>    this way, what you get is the following: win7 could suffer sometimes,
Dario>    if all its 3 vcpus gets busy, but that, I think is acceptable, at
Dario>    least up to a certain extent, is that the case?
Dario>    At the same time, you
Dario>    are making sure dom0 always has a chance to run, as pcpu#0 would be
Dario>    his exclusive playground, in case someone, including your pv499
Dario>    domain, needs its services.

This  is  what  I  had when I started :-). Thanks for the confirmation
that I was doing it right. However if the hyperthreading is the issue,
then I will only have 2 PCPU available, and I will assign them both to
dom0 and win7.

Dario> Right. Are you familiar with tracing what happens inside Xen
Dario> with xentrace and, perhaps, xenalyze? It takes a bit of time to
Dario> get used to it but, once you dominate it, it is a good mean for
Dario> getting out really useful info!

Dario> There is a blog post about that here:
Dario> http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/
Dario> and it should have most of the info, or the links to where to
Dario> find them.

Thanks for this. If this problem is more than the hyperthreading then
I will definitely use it. Also looks like it might be useful when I
start looking at the jitter on the singleshot timer (which should be
in a couple of weeks).

Andrew> How are you measuring time in pv499?

In  two  ways: counting the periodic interrupts (125 microsecond) and
from the monotonic_clock. They both agree on the time.

Andrew> What is your Cstates and Pstates looking like?

No  idea.  If the hyperthreading suggestion doesn't work out I'll look
at this.

Andrew> If you can, try disabling turbo?

If the hyperthreading suggestion doesn't work out I'll look at this.
-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Strange interdependace between domains

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 5971 bytes --]

On gio, 2014-02-13 at 22:25 +0000, Simon Martin wrote:
> Thanks for all the replies guys.
> 
:-)

> Don> How many instruction per second a thread gets does depend on the
> Don> "idleness" of other threads (no longer just the hyperThread's
> Don> parther).
> 
> This    seems    a    bit    strange   to   me. In my case I have time
> critical  PV  running  by  itself  in a CPU pool. So Xen should not be
> scheduling it, so I can't see how this Hypervisor thread would be affected.
> 
I think Don is referring to the idleness of the other _hardware_ threads
in the chip, rather than software threads of execution, either in Xen or
in Dom0/DomU. I checked his original e-mail and, AFAIUI, he seems to
confirm that the throughput you get on, say, core 3, depends on what
it's sibling core (which really is his sibling hyperthread, again in the
hardware sense... Gah, the terminology is just a mess! :-P). He seems to
also add the fact that there is a similar kind of inter-dependency
between all the hardware hyperthread, not just between siblings.

Does this make sense Don?

> >> 6.- All VCPUs are pinned:
> >> 
> Dario> Right, although, if you use cpupools, and if I've understood what you're
> Dario> up to, you really should not require pinning. I mean, the isolation
> Dario> between the RT-ish domain and the rest of the world should be already in
> Dario> place thanks to cpupools.
> 
> This  is what I thought, however when running looking at the vcpu-list
> I  CPU  affinity  was "all" until I starting pinning. As I wasn't sure
> whether  that  was  "all  inside this cpu pool" or "all" I felt it was
> safer to do it explicitly.
> 
Actually, you are right, we could put things in a way that results more
clear, when one observes the output! So, I confirm that, despite the
fact that you see "all", that all is relative to the cpupool the domain
is assigned to.

I'll try to think on how to make this more evident... A note in the
manpage and/or the various sources of documentation, is the easy (but
still necessary, I agree) part, and I'll add this to my TODO list.
Actually modifying the output is more tricky, as affinity and cpupools
are orthogonal by design, and that is the right (IMHO) thing.

I guess trying to tweak the printf()-s in `xl vcpu-list' would not be
that hard... I'll have a look and see if I can come up with a proposal.

> Dario> So, if you ask me, you're restricting too much things in
> Dario> pool-0, where dom0 and the Windows VM runs. In fact, is there a
> Dario> specific reason why you need all their vcpus to be statically
> Dario> pinned each one to only one pcpu? If not, I'd leave them a
> Dario> little bit more of freedom.
> 
> I agree with you here, however when I don't pin CPU affinity is "all".
> Is this "all in the CPU pool"? I couldn't find that info.
> 
Again, yes: once a domain is in a cpupool, no matter what its affinity
says, it won't ever reach a pcpu assigned to another cpupool. The
technical reason is that each cpupool is ruled by it's own (copy of a)
scheduler, even if you use, e.g., credit, for both/all the pools. In
that case, what you will get are two full instances of credit,
completely independent between each other, each one in charge only of a
very specific subset of pcpus (as mandated by cpupools). So, different
runqueues, different data structures, different anything.

> Dario> What I'd try is:
> Dario>  1. all dom0 and win7 vcpus free, so no pinning in pool0.
> Dario>  2. pinning as follows:
> Dario>      * all vcpus of win7 --> pcpus 1,2
> Dario>      * all vcpus of dom0 --> no pinning
> Dario>    this way, what you get is the following: win7 could suffer sometimes,
> Dario>    if all its 3 vcpus gets busy, but that, I think is acceptable, at
> Dario>    least up to a certain extent, is that the case?
> Dario>    At the same time, you
> Dario>    are making sure dom0 always has a chance to run, as pcpu#0 would be
> Dario>    his exclusive playground, in case someone, including your pv499
> Dario>    domain, needs its services.
> 
> This  is  what  I  had when I started :-). Thanks for the confirmation
> that I was doing it right. However if the hyperthreading is the issue,
> then I will only have 2 PCPU available, and I will assign them both to
> dom0 and win7.
> 
Yes, with hyperthreading in mind, that is what you should do.

Once we will have confirmed that hyperthreading is the issue, we'll see
what we can do. I mean, if, in your case, it's fine to 'waste' a cpu,
then ok, but I think we need a general solution for this... Perhaps with
a little worse performances than just leaving one core/hyperthread
completely idle, but at the same time more resource efficient.

I wonder how tweaking the sched_smt_power_savings would deal with
this...

> Dario> Right. Are you familiar with tracing what happens inside Xen
> Dario> with xentrace and, perhaps, xenalyze? It takes a bit of time to
> Dario> get used to it but, once you dominate it, it is a good mean for
> Dario> getting out really useful info!
> 
> Dario> There is a blog post about that here:
> Dario> http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/
> Dario> and it should have most of the info, or the links to where to
> Dario> find them.
> 
> Thanks for this. If this problem is more than the hyperthreading then
> I will definitely use it. Also looks like it might be useful when I
> start looking at the jitter on the singleshot timer (which should be
> in a couple of weeks).
> 
It will reveal to be very useful for that, I'm sure! :-)

Let us know how the re-testing goes.

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Strange interdependace between domains

[Don Slutz]

On 02/13/14 18:13, Dario Faggioli wrote:
> On gio, 2014-02-13 at 22:25 +0000, Simon Martin wrote:
>> Thanks for all the replies guys.
>>
> :-)
>
>> Don> How many instruction per second a thread gets does depend on the
>> Don> "idleness" of other threads (no longer just the hyperThread's
>> Don> parther).
>>
>> This    seems    a    bit    strange   to   me. In my case I have time
>> critical  PV  running  by  itself  in a CPU pool. So Xen should not be
>> scheduling it, so I can't see how this Hypervisor thread would be affected.
>>
> I think Don is referring to the idleness of the other _hardware_ threads
> in the chip, rather than software threads of execution, either in Xen or
> in Dom0/DomU. I checked his original e-mail and, AFAIUI, he seems to
> confirm that the throughput you get on, say, core 3, depends on what
> it's sibling core (which really is his sibling hyperthread, again in the
> hardware sense... Gah, the terminology is just a mess! :-P). He seems to
> also add the fact that there is a similar kind of inter-dependency
> between all the hardware hyperthread, not just between siblings.
>
> Does this make sense Don?
>

Yes, but the results I am getting vary based on the disto (most likely 
the microcode version).


Linux (and I think that xen) both have a CPU scheduler that picks core 
before threads:

top - 04:06:29 up 66 days, 15:31, 11 users,  load average: 2.43, 0.72, 0.29
Tasks: 250 total,   1 running, 249 sleeping,   0 stopped,   0 zombie
Cpu0  : 99.7%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.2%hi, 0.1%si,  
0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi, 0.2%si,  
0.0%st
Cpu2  : 99.9%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.1%hi, 0.0%si,  
0.0%st
Cpu3  :  1.6%us,  0.1%sy,  0.0%ni, 98.3%id,  0.0%wa,  0.0%hi, 0.0%si,  
0.0%st
Cpu4  : 99.9%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.1%hi, 0.0%si,  
0.0%st
Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi, 0.0%si,  
0.0%st
Cpu6  :  1.4%us,  0.0%sy,  0.0%ni, 98.6%id,  0.0%wa,  0.0%hi, 0.0%si,  
0.0%st
Cpu7  : 99.9%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.1%hi, 0.0%si,  
0.0%st
Mem:  32940640k total, 18008576k used, 14932064k free,   285740k buffers
Swap: 10223612k total,     4696k used, 10218916k free, 16746224k cached


Is an example without xen involved and Fedora 17

Linux dcs-xen-50 3.8.11-100.fc17.x86_64 #1 SMP Wed May 1 19:31:26 UTC 
2013 x86_64 x86_64 x86_64 GNU/Linux

On this machine:

Just 7:
         start                     done
thr 0:  14 Feb 14 04:11:08.944566  14 Feb 14 04:13:20.874764
        +02:11.930198 ~= 131.93 and 9.10 GiI/Sec


6 & 7:
         start                     done
thr 0:  14 Feb 14 04:14:31.010426  14 Feb 14 04:18:55.404116
        +04:24.393690 ~= 264.39 and 4.54 GiI/Sec
thr 1:  14 Feb 14 04:14:31.010426  14 Feb 14 04:18:55.415561
        +04:24.405135 ~= 264.41 and 4.54 GiI/Sec


5 & 7:
         start                     done
thr 0:  14 Feb 14 04:20:28.902831  14 Feb 14 04:22:45.563511
        +02:16.660680 ~= 136.66 and 8.78 GiI/Sec
thr 1:  14 Feb 14 04:20:28.902831  14 Feb 14 04:22:46.182159
        +02:17.279328 ~= 137.28 and 8.74 GiI/Sec


1 & 3 & 5 & 7:
         start                     done
thr 0:  14 Feb 14 04:32:24.353302  14 Feb 14 04:35:16.870558
        +02:52.517256 ~= 172.52 and 6.96 GiI/Sec
thr 1:  14 Feb 14 04:32:24.353301  14 Feb 14 04:35:17.371155
        +02:53.017854 ~= 173.02 and 6.94 GiI/Sec
thr 2:  14 Feb 14 04:32:24.353302  14 Feb 14 04:35:17.225871
        +02:52.872569 ~= 172.87 and 6.94 GiI/Sec
thr 3:  14 Feb 14 04:32:24.353302  14 Feb 14 04:35:16.655362
        +02:52.302060 ~= 172.30 and 6.96 GiI/Sec



This is from:
Feb 14 04:29:21 dcs-xen-51 kernel: [   41.921367] microcode: CPU3 
updated to revision 0x28, date = 2012-04-24


On CentOS 5.10:
Linux dcs-xen-53 2.6.18-371.el5 #1 SMP Tue Oct 1 08:35:08 EDT 2013 
x86_64 x86_64 x86_64 GNU/Linux

only 7:
         start                     done
thr 0:  14 Feb 14 09:43:10.903549  14 Feb 14 09:46:04.925463
        +02:54.021914 ~= 174.02 and 6.90 GiI/Sec


6 & 7:
         start                     done
thr 0:  14 Feb 14 09:49:17.804633  14 Feb 14 09:55:02.473549
        +05:44.668916 ~= 344.67 and 3.48 GiI/Sec
thr 1:  14 Feb 14 09:49:17.804618  14 Feb 14 09:55:02.533243
        +05:44.728625 ~= 344.73 and 3.48 GiI/Sec


5 & 7:
         start                     done
thr 0:  14 Feb 14 10:01:30.566603  14 Feb 14 10:04:23.024858
        +02:52.458255 ~= 172.46 and 6.96 GiI/Sec
thr 1:  14 Feb 14 10:01:30.566603  14 Feb 14 10:04:23.069964
        +02:52.503361 ~= 172.50 and 6.96 GiI/Sec


1 & 3 & 5 & 7:
         start                     done
thr 0:  14 Feb 14 10:05:58.359646  14 Feb 14 10:08:50.984629
        +02:52.624983 ~= 172.62 and 6.95 GiI/Sec
thr 1:  14 Feb 14 10:05:58.359646  14 Feb 14 10:08:50.993064
        +02:52.633418 ~= 172.63 and 6.95 GiI/Sec
thr 2:  14 Feb 14 10:05:58.359645  14 Feb 14 10:08:50.857982
        +02:52.498337 ~= 172.50 and 6.96 GiI/Sec
thr 3:  14 Feb 14 10:05:58.359645  14 Feb 14 10:08:50.905031
        +02:52.545386 ~= 172.55 and 6.95 GiI/Sec




Feb 14 09:41:42 dcs-xen-53 kernel: microcode: CPU3 updated from revision 
0x17 to 0x29, date = 06122013


Hope this helps.
     -Don Slutz

>>>> 6.- All VCPUs are pinned:
>>>>
>> Dario> Right, although, if you use cpupools, and if I've understood what you're
>> Dario> up to, you really should not require pinning. I mean, the isolation
>> Dario> between the RT-ish domain and the rest of the world should be already in
>> Dario> place thanks to cpupools.
>>
>> This  is what I thought, however when running looking at the vcpu-list
>> I  CPU  affinity  was "all" until I starting pinning. As I wasn't sure
>> whether  that  was  "all  inside this cpu pool" or "all" I felt it was
>> safer to do it explicitly.
>>
> Actually, you are right, we could put things in a way that results more
> clear, when one observes the output! So, I confirm that, despite the
> fact that you see "all", that all is relative to the cpupool the domain
> is assigned to.
>
> I'll try to think on how to make this more evident... A note in the
> manpage and/or the various sources of documentation, is the easy (but
> still necessary, I agree) part, and I'll add this to my TODO list.
> Actually modifying the output is more tricky, as affinity and cpupools
> are orthogonal by design, and that is the right (IMHO) thing.
>
> I guess trying to tweak the printf()-s in `xl vcpu-list' would not be
> that hard... I'll have a look and see if I can come up with a proposal.
>
>> Dario> So, if you ask me, you're restricting too much things in
>> Dario> pool-0, where dom0 and the Windows VM runs. In fact, is there a
>> Dario> specific reason why you need all their vcpus to be statically
>> Dario> pinned each one to only one pcpu? If not, I'd leave them a
>> Dario> little bit more of freedom.
>>
>> I agree with you here, however when I don't pin CPU affinity is "all".
>> Is this "all in the CPU pool"? I couldn't find that info.
>>
> Again, yes: once a domain is in a cpupool, no matter what its affinity
> says, it won't ever reach a pcpu assigned to another cpupool. The
> technical reason is that each cpupool is ruled by it's own (copy of a)
> scheduler, even if you use, e.g., credit, for both/all the pools. In
> that case, what you will get are two full instances of credit,
> completely independent between each other, each one in charge only of a
> very specific subset of pcpus (as mandated by cpupools). So, different
> runqueues, different data structures, different anything.
>
>> Dario> What I'd try is:
>> Dario>  1. all dom0 and win7 vcpus free, so no pinning in pool0.
>> Dario>  2. pinning as follows:
>> Dario>      * all vcpus of win7 --> pcpus 1,2
>> Dario>      * all vcpus of dom0 --> no pinning
>> Dario>    this way, what you get is the following: win7 could suffer sometimes,
>> Dario>    if all its 3 vcpus gets busy, but that, I think is acceptable, at
>> Dario>    least up to a certain extent, is that the case?
>> Dario>    At the same time, you
>> Dario>    are making sure dom0 always has a chance to run, as pcpu#0 would be
>> Dario>    his exclusive playground, in case someone, including your pv499
>> Dario>    domain, needs its services.
>>
>> This  is  what  I  had when I started :-). Thanks for the confirmation
>> that I was doing it right. However if the hyperthreading is the issue,
>> then I will only have 2 PCPU available, and I will assign them both to
>> dom0 and win7.
>>
> Yes, with hyperthreading in mind, that is what you should do.
>
> Once we will have confirmed that hyperthreading is the issue, we'll see
> what we can do. I mean, if, in your case, it's fine to 'waste' a cpu,
> then ok, but I think we need a general solution for this... Perhaps with
> a little worse performances than just leaving one core/hyperthread
> completely idle, but at the same time more resource efficient.
>
> I wonder how tweaking the sched_smt_power_savings would deal with
> this...
>
>> Dario> Right. Are you familiar with tracing what happens inside Xen
>> Dario> with xentrace and, perhaps, xenalyze? It takes a bit of time to
>> Dario> get used to it but, once you dominate it, it is a good mean for
>> Dario> getting out really useful info!
>>
>> Dario> There is a blog post about that here:
>> Dario> http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/
>> Dario> and it should have most of the info, or the links to where to
>> Dario> find them.
>>
>> Thanks for this. If this problem is more than the hyperthreading then
>> I will definitely use it. Also looks like it might be useful when I
>> start looking at the jitter on the singleshot timer (which should be
>> in a couple of weeks).
>>
> It will reveal to be very useful for that, I'm sure! :-)
>
> Let us know how the re-testing goes.
>
> Regards,
> Dario
>

Re: Strange interdependace between domains

[Simon Martin]

Hello Simon,

Thanks everyone and especially Ian! It was the hyperthreading that was
causing the problem.

Here's my current configuration:

# xl cpupool-list -c
Name               CPU list
Pool-0             0,1
pv499              2,3
# xl vcpu-list
Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
Domain-0                             0     0    0   r--      16.6  0
Domain-0                             0     1    1   -b-       7.3  1
win7x64                              1     0    1   -b-      82.5  all
win7x64                              1     1    0   -b-      18.6  all
pv499                                2     0    3   r--     226.1  3

I  have  pinned  dom0 as I wasn't sure whether it belongs to Pool-0 (I
assume it does, can you confirm please).

Dario, if you are going to look at the

Looking at my timings with this configuration I am seeing a 1%
variation (945 milliseconds +/- 5). I think this can be attributable
to  RAM  contention, at the end of the day all cores are competing for
the same bus.

-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Strange interdependace between domains

[Andrew Cooper]

On 14/02/14 12:02, Simon Martin wrote:
> Hello Simon,
>
> Thanks everyone and especially Ian! It was the hyperthreading that was
> causing the problem.
>
> Here's my current configuration:
>
> # xl cpupool-list -c
> Name               CPU list
> Pool-0             0,1
> pv499              2,3
> # xl vcpu-list
> Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
> Domain-0                             0     0    0   r--      16.6  0
> Domain-0                             0     1    1   -b-       7.3  1
> win7x64                              1     0    1   -b-      82.5  all
> win7x64                              1     1    0   -b-      18.6  all
> pv499                                2     0    3   r--     226.1  3
>
> I  have  pinned  dom0 as I wasn't sure whether it belongs to Pool-0 (I
> assume it does, can you confirm please).
>
> Dario, if you are going to look at the
>
> Looking at my timings with this configuration I am seeing a 1%
> variation (945 milliseconds +/- 5). I think this can be attributable
> to  RAM  contention, at the end of the day all cores are competing for
> the same bus.
>

There is also things such as the Xen time calibration rendezvous which a
synchronisation point of all online cpus, once per second.  Having any
single cpu slow to enter the rendezvous will delay all others which have
already entered.

This will likely add a bit of jitter if one cpu in xen is doing a
lengthy operation with interrupts disabled at the point at which the
rendezvous is triggered.

~Andrew

Re: Strange interdependace between domains

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3465 bytes --]

On ven, 2014-02-14 at 12:02 +0000, Simon Martin wrote:
> Thanks everyone and especially Ian! It was the hyperthreading that was
> causing the problem.
> 
Good to hear, and at the same time, sorry to hear that. :-)

I mean, I'm glad you nailed it, but at the same time, I'm sorry that the
solution is to 'waste' a core! :-( I reiterate and restate here, without
any problem doing so, the fact that Xen should be doing at least a bit
better in these circumstances, if we want to properly address use cases
like yours. However, there are limits on how far we can go, and hardware
design is certainly among them!

All this to say that, it should be possible to get a bit more of
isolation, by tweaking the proper Xen code path appropriately, but if
the amount of interference that comes from two hypethreads sharing
registers, pipeline stages, and whatever it is that they share, is
enough for disturbing your workload, then, I'm afraid we never get much
farther from the 'don't use hyperthread' solution! :-(

Anyways, with respect to the first part of this reasoning, would you
mind (when you've got the time of course) one more test? If no, I'd say,
configure the system as I was suggesting in my first reply, i.e., using
also core #2 (or, in general, all the cores). Also, make sure you add
this parameter, to the Xen boot command line:

 sched_smt_power_savings=1

(some background here:
http://lists.xen.org/archives/html/xen-devel/2009-03/msg01335.html)

And then run the bench with disk activity on.

> Here's my current configuration:
> 
> # xl cpupool-list -c
> Name               CPU list
> Pool-0             0,1
> pv499              2,3
> # xl vcpu-list
> Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
> Domain-0                             0     0    0   r--      16.6  0
> Domain-0                             0     1    1   -b-       7.3  1
> win7x64                              1     0    1   -b-      82.5  all
> win7x64                              1     1    0   -b-      18.6  all
> pv499                                2     0    3   r--     226.1  3
> 
> I  have  pinned  dom0 as I wasn't sure whether it belongs to Pool-0 (I
> assume it does, can you confirm please).
> 
Actually, you are right. It looks like there is no command or command
parameter telling explicitly to which pool a domain belong [BTW, adding
Juergen, who knows that for sure].

If that is the case, we really should add one.

BTW, If you boot the system and then create the (new) pool(s), all the
existing domains, including Dom0, at pool(s) creation time will stay in
the "original" pool, while the new pool(s) will be empty. To change
that, you'd have to either migrate existing domain into specific pools
with `xl cpupool-migrate', or create them specifying the proper option
and the name of the target pool in the config file (which is probably
what you're doing for your DomUs).

I guess a workaround to confirm where a domain is, you can (try to)
migrate it around with `xl cpupool-migrate', and see what happens.

> Dario, if you are going to look at the
> 
Is something missing here... ?

Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Strange interdependace between domains

[Simon Martin]

Hello Dario,

> All this to say that, it should be possible to get a bit more of
> isolation, by tweaking the proper Xen code path appropriately, but if
> the amount of interference that comes from two hypethreads sharing
> registers, pipeline stages, and whatever it is that they share, is
> enough for disturbing your workload, then, I'm afraid we never get much
> farther from the 'don't use hyperthread' solution! :-(

Hyperthreading is just a way to improve CPU resource utilization. Even
if you are doing a CPU intensive operation, a lot of the processor
circuits are actually idle, so adding 2 pipelines to feed one
processor is a good way to improve total throughput, but it does have
it have it's caveats. I totally forgot this.

Given the way that this works there isn't much that Xen can do. It is
a physical restriction.

The only thing I can think of would be to add an option to only run
hardware interaction on a sub-set of the available hyperthreads, but
that would require hacking the Dom0 kernel to lock device drivers onto
a given set of CPU and might end up worst than when it started.

Another way might be to have overlapping CPU pools. That way I have
Dom0 running on only one core and then I can run DomUs spread over the
available cores. AFAIU it is the hardware interaction that is causing
the interdependency.

> Anyways, with respect to the first part of this reasoning, would you
> mind (when you've got the time of course) one more test? If no, I'd say,
> configure the system as I was suggesting in my first reply, i.e., using
> also core #2 (or, in general, all the cores). Also, make sure you add
> this parameter, to the Xen boot command line:

>  sched_smt_power_savings=1

> (some background here:
> http://lists.xen.org/archives/html/xen-devel/2009-03/msg01335.html)

> And then run the bench with disk activity on.

OK. This is my current configuration:

Dom0     PCPU 0,1,2   no pinning
win7x64  PCPU   1,2   pinned
pv499    PCPU       3 pinned

And I get the same interdependence.

root@smartin-xen:~# xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  1253     3     r-----      37.1
win7x64                                      4  2046     2     ------     106.8
pv499                                        5   128     1     r-----      66.8
root@smartin-xen:~# xl vcpu-list 
Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
Domain-0                             0     0    1   -b-      21.2  all
Domain-0                             0     1    2   r--       8.8  all
Domain-0                             0     2    0   -b-       8.2  all
win7x64                              4     0    1   -b-      59.8  1
win7x64                              4     1    2   -b-      49.0  2
pv499                                5     0    3   r--      70.2  3
root@smartin-xen:~# xl cpupool-list -c
Name               CPU list
Pool-0             0,1,2
pv499              3

I have gone back to my working settings.


-- 
Best regards,
 Simon                            mailto:furryfuttock@gmail.com

Re: Strange interdependace between domains

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3156 bytes --]

On lun, 2014-02-17 at 12:46 +0000, Simon Martin wrote:
> Hyperthreading is just a way to improve CPU resource utilization. Even
> if you are doing a CPU intensive operation, a lot of the processor
> circuits are actually idle, so adding 2 pipelines to feed one
> processor is a good way to improve total throughput, but it does have
> it have it's caveats. I totally forgot this.
> 
> Given the way that this works there isn't much that Xen can do. It is
> a physical restriction.
> 
I know, and my point was not that we should try to "fix" in Xen, what's
impossible to fix, because it's just how hardware works... and that is
by design!

I was only saying that there are cases, when you still need isolation,
but you can afford a bit more of uncertainty, there is the possibility
for Xen to at least try to do the right thing automagically.

Basically, I'm ok with people looking for hard real-time response time
and jitter to have to pick up the proper hw platform, as a first step,
and properly fine tune it. For more _soft_ real-time workloads, I wish
we were (think we should be) able to do better, perhaps still with some
user required tweaks, but nothing equally intrusive, that's it. :-)

> OK. This is my current configuration:
> 
> Dom0     PCPU 0,1,2   no pinning
> win7x64  PCPU   1,2   pinned
> pv499    PCPU       3 pinned
> 
> And I get the same interdependence.
> 
> root@smartin-xen:~# xl list
> Name                                        ID   Mem VCPUs      State   Time(s)
> Domain-0                                     0  1253     3     r-----      37.1
> win7x64                                      4  2046     2     ------     106.8
> pv499                                        5   128     1     r-----      66.8
> root@smartin-xen:~# xl vcpu-list 
> Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
> Domain-0                             0     0    1   -b-      21.2  all
> Domain-0                             0     1    2   r--       8.8  all
> Domain-0                             0     2    0   -b-       8.2  all
> win7x64                              4     0    1   -b-      59.8  1
> win7x64                              4     1    2   -b-      49.0  2
> pv499                                5     0    3   r--      70.2  3
> root@smartin-xen:~# xl cpupool-list -c
> Name               CPU list
> Pool-0             0,1,2
> pv499              3
> 
> I have gone back to my working settings.
> 
Ok, thanks a lot for trying. I was hoping for that tweak, although
developed for completely different reasons, to be helpful in this case,
but it appears it is not.

Probably, I wouldn't have pinned the two win7 vcpus either, but anyway,
I don't want to eat any more of your time... Glad you find a
configuration that is working out! :-)

Thanks again and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Strange interdependace between domains

[Don Slutz]

On 02/18/14 11:55, Dario Faggioli wrote:
> On lun, 2014-02-17 at 12:46 +0000, Simon Martin wrote:
>> Hyperthreading is just a way to improve CPU resource utilization. Even
>> if you are doing a CPU intensive operation, a lot of the processor
>> circuits are actually idle, so adding 2 pipelines to feed one
>> processor is a good way to improve total throughput, but it does have
>> it have it's caveats. I totally forgot this.
>>
>> Given the way that this works there isn't much that Xen can do. It is
>> a physical restriction.
>>
> I know, and my point was not that we should try to "fix" in Xen, what's
> impossible to fix, because it's just how hardware works... and that is
> by design!
>
> I was only saying that there are cases, when you still need isolation,
> but you can afford a bit more of uncertainty, there is the possibility
> for Xen to at least try to do the right thing automagically.
>
> Basically, I'm ok with people looking for hard real-time response time
> and jitter to have to pick up the proper hw platform, as a first step,
> and properly fine tune it. For more _soft_ real-time workloads, I wish
> we were (think we should be) able to do better, perhaps still with some
> user required tweaks, but nothing equally intrusive, that's it. :-)

I would thing that a warning message from the xl command(s) dealing with cpupool would help here.  See below

>> OK. This is my current configuration:
>>
>> Dom0     PCPU 0,1,2   no pinning
>> win7x64  PCPU   1,2   pinned
>> pv499    PCPU       3 pinned
>>
>> And I get the same interdependence.
>>
>> root@smartin-xen:~# xl list
>> Name                                        ID   Mem VCPUs      State   Time(s)
>> Domain-0                                     0  1253     3     r-----      37.1
>> win7x64                                      4  2046     2     ------     106.8
>> pv499                                        5   128     1     r-----      66.8
>> root@smartin-xen:~# xl vcpu-list
>> Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
>> Domain-0                             0     0    1   -b-      21.2  all
>> Domain-0                             0     1    2   r--       8.8  all
>> Domain-0                             0     2    0   -b-       8.2  all
>> win7x64                              4     0    1   -b-      59.8  1
>> win7x64                              4     1    2   -b-      49.0  2
>> pv499                                5     0    3   r--      70.2  3
>> root@smartin-xen:~# xl cpupool-list -c
>> Name               CPU list
>> Pool-0             0,1,2

Change to something like:


Pool-0             0,1,2 (and part of 3)




>> pv499              3

Or add:

Warning: cpupool's share hyperthreaded cpus.

    -Don Slutz

>> I have gone back to my working settings.
>>
> Ok, thanks a lot for trying. I was hoping for that tweak, although
> developed for completely different reasons, to be helpful in this case,
> but it appears it is not.
>
> Probably, I wouldn't have pinned the two win7 vcpus either, but anyway,
> I don't want to eat any more of your time... Glad you find a
> configuration that is working out! :-)
>
> Thanks again and Regards,
> Dario
>

Re: Strange interdependace between domains

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1181 bytes --]

On mar, 2014-02-18 at 12:58 -0500, Don Slutz wrote:
> >> root@smartin-xen:~# xl cpupool-list -c
> >> Name               CPU list
> >> Pool-0             0,1,2
> 
> Change to something like:
> 
> 
> Pool-0             0,1,2 (and part of 3)
> 
This would be cool, and I personally would be all for it... but it
perhaps will not be that clear at pointing the user to think to
hyperthreading. :-P

> Or add:
> 
> Warning: cpupool's share hyperthreaded cpus.
> 
While this one, although a bit more "boring" than the above, would
probably be something quite valuable to have!

I can only think of rather expensive ways of implementing it, involving
going through all the cpupools and, for each cpupool, through all its
cpus and check the topology relationships, but perhaps there are others
(I'll think harder).

Also, we are certainly not talking about hot paths.

Juergen?

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Strange interdependace between domains

[Juergen Gross]

On 18.02.2014 19:06, Dario Faggioli wrote:
> On mar, 2014-02-18 at 12:58 -0500, Don Slutz wrote:
>>>> root@smartin-xen:~# xl cpupool-list -c
>>>> Name               CPU list
>>>> Pool-0             0,1,2
>>
>> Change to something like:
>>
>>
>> Pool-0             0,1,2 (and part of 3)
>>
> This would be cool, and I personally would be all for it... but it
> perhaps will not be that clear at pointing the user to think to
> hyperthreading. :-P
>
>> Or add:
>>
>> Warning: cpupool's share hyperthreaded cpus.
>>
> While this one, although a bit more "boring" than the above, would
> probably be something quite valuable to have!
>
> I can only think of rather expensive ways of implementing it, involving
> going through all the cpupools and, for each cpupool, through all its
> cpus and check the topology relationships, but perhaps there are others
> (I'll think harder).
>
> Also, we are certainly not talking about hot paths.
>
> Juergen?

Adding some information like this would be nice, indeed. But I think we should
not limit this to just hyperthreads. There are more levels of shared resources,
like caches or memory interfaces on the same socket. In case we want to add
information about potential performance influences due to shared resources, we
should be more generic.

And what about some NUMA information? Wouldn't it be worthwhile to show memory
locality information as well? This should be considered to be displayed by
"xl list", too.


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
PBG PDG ES&S SWE OS6                   Telephone: +49 (0) 89 62060 2932
Fujitsu                                   e-mail: juergen.gross@ts.fujitsu.com
Mies-van-der-Rohe-Str. 8                Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

Re: Strange interdependace between domains

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1716 bytes --]

On gio, 2014-02-20 at 07:07 +0100, Juergen Gross wrote:
> On 18.02.2014 19:06, Dario Faggioli wrote:

> > While this one, although a bit more "boring" than the above, would
> > probably be something quite valuable to have!
> >
> > I can only think of rather expensive ways of implementing it, involving
> > going through all the cpupools and, for each cpupool, through all its
> > cpus and check the topology relationships, but perhaps there are others
> > (I'll think harder).
> >
> Adding some information like this would be nice, indeed. But I think we should
> not limit this to just hyperthreads. There are more levels of shared resources,
> like caches or memory interfaces on the same socket. In case we want to add
> information about potential performance influences due to shared resources, we
> should be more generic.
> 
All true... To the point that I know also wonder what a suitable
interface and a not too verbose output configuration could be...

> And what about some NUMA information? Wouldn't it be worthwhile to show memory
> locality information as well? This should be considered to be displayed by
> "xl list", too.
> 
Indeed. I actually have an half backed series doing right that. It's a
bit more complicated (still from an interface point of view), as that
info resides in Xen, and some new hcall or similar is required to
retrieve that.

I'll post it in early 4.5 dev cycle.

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Strange interdependace between domains

[Juergen Gross]

On 20.02.2014 19:22, Dario Faggioli wrote:
> On gio, 2014-02-20 at 07:07 +0100, Juergen Gross wrote:
>> On 18.02.2014 19:06, Dario Faggioli wrote:
>
>>> While this one, although a bit more "boring" than the above, would
>>> probably be something quite valuable to have!
>>>
>>> I can only think of rather expensive ways of implementing it, involving
>>> going through all the cpupools and, for each cpupool, through all its
>>> cpus and check the topology relationships, but perhaps there are others
>>> (I'll think harder).
>>>
>> Adding some information like this would be nice, indeed. But I think we should
>> not limit this to just hyperthreads. There are more levels of shared resources,
>> like caches or memory interfaces on the same socket. In case we want to add
>> information about potential performance influences due to shared resources, we
>> should be more generic.
>>
> All true... To the point that I know also wonder what a suitable
> interface and a not too verbose output configuration could be...

Well, looking at the available topology information I think it should look like
the following example:

# xl cpupool-list --shareinfo
Name          CPUs   Sched     Active  Domain count  shared resources
Pool-0          1    credit       y         1        core:   lw_pool
lw_pool         1    credit       y         0        core:   Pool-0
bs2_pool        2    credit       y         1        socket: Pool-0,lw_pool

What do you think?

>> And what about some NUMA information? Wouldn't it be worthwhile to show memory
>> locality information as well? This should be considered to be displayed by
>> "xl list", too.
>>
> Indeed. I actually have an half backed series doing right that. It's a
> bit more complicated (still from an interface point of view), as that
> info resides in Xen, and some new hcall or similar is required to
> retrieve that.
>
> I'll post it in early 4.5 dev cycle.

Please do!


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
PBG PDG ES&S SWE OS6                   Telephone: +49 (0) 89 62060 2932
Fujitsu                                   e-mail: juergen.gross@ts.fujitsu.com
Mies-van-der-Rohe-Str. 8                Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

Re: Strange interdependace between domains

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1099 bytes --]

On ven, 2014-02-21 at 07:31 +0100, Juergen Gross wrote:
> On 20.02.2014 19:22, Dario Faggioli wrote:
> > All true... To the point that I know also wonder what a suitable
> > interface and a not too verbose output configuration could be...
> 
> Well, looking at the available topology information I think it should look like
> the following example:
> 
> # xl cpupool-list --shareinfo
> Name          CPUs   Sched     Active  Domain count  shared resources
> Pool-0          1    credit       y         1        core:   lw_pool
> lw_pool         1    credit       y         0        core:   Pool-0
> bs2_pool        2    credit       y         1        socket: Pool-0,lw_pool
> 
> What do you think?
> 
Looks reasonable.

> > I'll post it in early 4.5 dev cycle.
> 
> Please do!
> 
I will. :-)

regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Strange interdependace between domains

[Juergen Gross]

On 21.02.2014 18:24, Dario Faggioli wrote:
> On ven, 2014-02-21 at 07:31 +0100, Juergen Gross wrote:
>> On 20.02.2014 19:22, Dario Faggioli wrote:
>>> All true... To the point that I know also wonder what a suitable
>>> interface and a not too verbose output configuration could be...
>>
>> Well, looking at the available topology information I think it should look like
>> the following example:
>>
>> # xl cpupool-list --shareinfo
>> Name          CPUs   Sched     Active  Domain count  shared resources
>> Pool-0          1    credit       y         1        core:   lw_pool
>> lw_pool         1    credit       y         0        core:   Pool-0
>> bs2_pool        2    credit       y         1        socket: Pool-0,lw_pool
>>
>> What do you think?
>>
> Looks reasonable.

Another solution would be to add a --long option. This would have the advantage
of not having to choose between clobbering the table output or not being able
to show multiple optional information items.

So we could do something like:

# xl cpupool-list --long
Pool-0
   n-cpus:        1
   cpu-list:      0
   scheduler:     credit
   n-domains:     1
   domain-list:   Dom0
   res-share-lvl: core
   res-sharers:   lw_pool
lw_pool
   n-cpus:        1
   cpu-list:      1
   scheduler:     credit
   n-domains:     0
   domain-list:
   res-share-lvl: core
   res-sharers:   Pool-0
bs2_pool
   n-cpus:        2
   cpu-list:      2,3
   scheduler:     credit
   n-domains:     1
   domain-list:   BS2000
   res-share-lvl: socket
   res-sharers:   Pool-0,lw_pool

We could add scheduler parameters, NUMA-information, ... as well.


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
PBG PDG ES&S SWE OS6                   Telephone: +49 (0) 89 62060 2932
Fujitsu                                   e-mail: juergen.gross@ts.fujitsu.com
Mies-van-der-Rohe-Str. 8                Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

Re: Strange interdependace between domains

[Juergen Gross]

On 14.02.2014 18:21, Dario Faggioli wrote:
>
> Actually, you are right. It looks like there is no command or command
> parameter telling explicitly to which pool a domain belong [BTW, adding
> Juergen, who knows that for sure].

You didn't add me, but I just stumbled over this message. :-)

When I added cpupools the information could be obtained via "xm list -l".
In the moment I haven't got a xen-unstable system up. And on my 4.2.3
machine "xl list -l" isn't giving any information at all.

With "xenstore-ls /vm" the information can be retrieved: it is listed
under <uuid>/pool_name (with <uuid> being the UUID of the domain in
question).

> If that is the case, we really should add one.

Indeed. I think "xl cpupool-list" should have another option to show the
domains in the cpupool. I'll prepare a patch.


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
PBG PDG ES&S SWE OS6                   Telephone: +49 (0) 89 62060 2932
Fujitsu                                   e-mail: juergen.gross@ts.fujitsu.com
Mies-van-der-Rohe-Str. 8                Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

Re: Strange interdependace between domains

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1836 bytes --]

On lun, 2014-02-17 at 14:19 +0100, Juergen Gross wrote:
> On 14.02.2014 18:21, Dario Faggioli wrote:
> >
> > Actually, you are right. It looks like there is no command or command
> > parameter telling explicitly to which pool a domain belong [BTW, adding
> > Juergen, who knows that for sure].
> 
> You didn't add me, but I just stumbled over this message. :-)
> 
Oh, sorry... I could have sworn I did! :-P

Glad you've noticed the conversation anyway... and sorry.

> When I added cpupools the information could be obtained via "xm list -l".
> In the moment I haven't got a xen-unstable system up. And on my 4.2.3
> machine "xl list -l" isn't giving any information at all.
> 
Yeah, I remember something about a discussion on this. On my -unstable
test box, I still get no output for dom0, while, if you have a domU, I
do see something, and the poolid is there.

root@Zhaman:~# xl list -l | grep -i pool
                "poolid": 0,

> With "xenstore-ls /vm" the information can be retrieved: it is listed
> under <uuid>/pool_name (with <uuid> being the UUID of the domain in
> question).
> 
Funny:

root@Zhaman:~# xenstore-ls /vm | grep -i pool
root@Zhaman:~#

root@Zhaman:~# xenstore-ls | grep -i pool
root@Zhaman:~#

:-O

Is that because I didn't really add any pool (i.e., I'm running the
above with only Pool-0) ?

> > If that is the case, we really should add one.
> 
> Indeed. I think "xl cpupool-list" should have another option to show the
> domains in the cpupool. I'll prepare a patch.
> 
Cool!

Thanks,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Strange interdependace between domains

[Juergen Gross]

On 17.02.2014 16:08, Dario Faggioli wrote:
> On lun, 2014-02-17 at 14:19 +0100, Juergen Gross wrote:

>> With "xenstore-ls /vm" the information can be retrieved: it is listed
>> under <uuid>/pool_name (with <uuid> being the UUID of the domain in
>> question).
>>
> Funny:
>
> root@Zhaman:~# xenstore-ls /vm | grep -i pool
> root@Zhaman:~#
>
> root@Zhaman:~# xenstore-ls | grep -i pool
> root@Zhaman:~#
>
> :-O
>
> Is that because I didn't really add any pool (i.e., I'm running the
> above with only Pool-0) ?

Hmm. Could be another difference between xm and xl (on my 4.2 box I created
the domU via xm):

root@nehalem1 # xenstore-ls /vm | grep -i pool
  pool_name = "Pool-0"
  pool_name = "bs2_pool"


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
PBG PDG ES&S SWE OS6                   Telephone: +49 (0) 89 62060 2932
Fujitsu                                   e-mail: juergen.gross@ts.fujitsu.com
Mies-van-der-Rohe-Str. 8                Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

Re: Strange interdependace between domains

[Nate Studer]

On 2/14/2014 12:21 PM, Dario Faggioli wrote:
> On ven, 2014-02-14 at 12:02 +0000, Simon Martin wrote:
>> Thanks everyone and especially Ian! It was the hyperthreading that was
>> causing the problem.
>>
> Good to hear, and at the same time, sorry to hear that. :-)
> 
> I mean, I'm glad you nailed it, but at the same time, I'm sorry that the
> solution is to 'waste' a core! :-( I reiterate and restate here, without
> any problem doing so, the fact that Xen should be doing at least a bit
> better in these circumstances, if we want to properly address use cases
> like yours. However, there are limits on how far we can go, and hardware
> design is certainly among them!
> 
> All this to say that, it should be possible to get a bit more of
> isolation, by tweaking the proper Xen code path appropriately, but if
> the amount of interference that comes from two hypethreads sharing
> registers, pipeline stages, and whatever it is that they share, is
> enough for disturbing your workload, then, I'm afraid we never get much
> farther from the 'don't use hyperthread' solution! :-(

Which, as you say, unfortunately is the solution unless there is some way to
configure the hardware to eliminate this interference.  If it's any consolation,
the only multi-core ARINC653 implementations I know of have enacted these two
restrictions:
1.  # of cores enabled = # of memory controllers.
2.  Each enabled core must be configured to not share a memory controller,
cache, registers, etc...

It is practically an AMP system at that point, but without these restrictions
you can get some unpredictable behavior unless you have some specialized or
exotic hardware to make things more deterministic.

-- 
Nathan Studer
DornerWorks, Ltd.
Embedded Systems Engineering

Re: Strange interdependace between domains

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1680 bytes --]

On lun, 2014-02-17 at 09:13 -0500, Nate Studer wrote:
> On 2/14/2014 12:21 PM, Dario Faggioli wrote:

> > All this to say that, it should be possible to get a bit more of
> > isolation, by tweaking the proper Xen code path appropriately, but if
> > the amount of interference that comes from two hypethreads sharing
> > registers, pipeline stages, and whatever it is that they share, is
> > enough for disturbing your workload, then, I'm afraid we never get much
> > farther from the 'don't use hyperthread' solution! :-(
> 
> Which, as you say, unfortunately is the solution unless there is some way to
> configure the hardware to eliminate this interference.  
>
Yeah, I know!

> If it's any consolation,
> the only multi-core ARINC653 implementations I know of have enacted these two
> restrictions:
> 1.  # of cores enabled = # of memory controllers.
> 2.  Each enabled core must be configured to not share a memory controller,
> cache, registers, etc...
> 
Nice. :-)

> It is practically an AMP system at that point, but without these restrictions
> you can get some unpredictable behavior unless you have some specialized or
> exotic hardware to make things more deterministic.
> 
Sure, when you really need to be serious about isolation, treats are
there well before software (whatever is just OS, virt, whatever) comes
into play!

Thanks for sharing this and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel