CAP and performance problem

CAP and performance problem

[Massimo Canonico]

Hi,
I sent the following problem on xen-user ML without an answer. I hope 
I'll get one in this ML.

My application is written in std C++ and it makes a matrix 
multiplication: so it uses only CPU and memory (no I/O, no network).

I'm quite surprise that with CAP = 100% I got my results in about 600 
seconds and with CAP = 50% I got my results in about 1800 seconds 
(around 3 times longer).

For this kind of application I was expecting to get results in about 
1200 seconds (2 times longer) for the second scenario with respect to 
the first one.

Of course, the HW and SW are exactly the same for the 2 experiments.

Am I wrong or the CAP mechanism is not working well?

Thanks,
  Massimo

Re: CAP and performance problem

[George Dunlap]

On Tue, May 21, 2013 at 10:02 AM, Massimo Canonico <mex@di.unipmn.it> wrote:
> Hi,
> I sent the following problem on xen-user ML without an answer. I hope I'll
> get one in this ML.
>
> My application is written in std C++ and it makes a matrix multiplication:
> so it uses only CPU and memory (no I/O, no network).
>
>
> I'm quite surprise that with CAP = 100% I got my results in about 600
> seconds and with CAP = 50% I got my results in about 1800 seconds (around 3
> times longer).
>
> For this kind of application I was expecting to get results in about 1200
> seconds (2 times longer) for the second scenario with respect to the first
> one.
>
> Of course, the HW and SW are exactly the same for the 2 experiments.
>
> Am I wrong or the CAP mechanism is not working well?

How many vcpus do you have?

I'm not sure the CAP stuff actually does anything sensible with
multiple cpus -- e.g., a VM with 2 vcpus maxed out will typically
actually get 200%, so will a CAP force both of them to get only 25%
each?

The other possibility if you've got multiple vcpus is that you're
running into something I call "concurrency hazard": if one vcpu grabs
a spinlock and then is paused in order to enforce the cap, and another
vcpu tries to grab that spinlock, it will just spin until the first
vcpu manages to wake up and release it again.

 -George

Re: CAP and performance problem

[George Dunlap]

On Tue, May 21, 2013 at 11:41 AM, George Dunlap
<George.Dunlap@eu.citrix.com> wrote:
> On Tue, May 21, 2013 at 10:02 AM, Massimo Canonico <mex@di.unipmn.it> wrote:
>> Hi,
>> I sent the following problem on xen-user ML without an answer. I hope I'll
>> get one in this ML.
>>
>> My application is written in std C++ and it makes a matrix multiplication:
>> so it uses only CPU and memory (no I/O, no network).
>>
>>
>> I'm quite surprise that with CAP = 100% I got my results in about 600
>> seconds and with CAP = 50% I got my results in about 1800 seconds (around 3
>> times longer).
>>
>> For this kind of application I was expecting to get results in about 1200
>> seconds (2 times longer) for the second scenario with respect to the first
>> one.
>>
>> Of course, the HW and SW are exactly the same for the 2 experiments.
>>
>> Am I wrong or the CAP mechanism is not working well?
>
> How many vcpus do you have?
>
> I'm not sure the CAP stuff actually does anything sensible with
> multiple cpus -- e.g., a VM with 2 vcpus maxed out will typically
> actually get 200%, so will a CAP force both of them to get only 25%
> each?
>
> The other possibility if you've got multiple vcpus is that you're
> running into something I call "concurrency hazard": if one vcpu grabs
> a spinlock and then is paused in order to enforce the cap, and another
> vcpu tries to grab that spinlock, it will just spin until the first
> vcpu manages to wake up and release it again.

Sorry, meant to add: if you have more than one vcpu and you want to
limit the cpu footprint, you're much better off halving the number of
vcpus rather than giving each vcpu only 50% of a cpu.

I gave a presentation on why here, if you're interested:

http://www.citrix.com/tv/#videos/2930

 -George

Re: CAP and performance problem

[Massimo Canonico]

On 05/21/2013 12:48 PM, George Dunlap wrote:
> On Tue, May 21, 2013 at 11:41 AM, George Dunlap
> <George.Dunlap@eu.citrix.com> wrote:
>> On Tue, May 21, 2013 at 10:02 AM, Massimo Canonico <mex@di.unipmn.it> wrote:
>>> Hi,
>>> I sent the following problem on xen-user ML without an answer. I hope I'll
>>> get one in this ML.
>>>
>>> My application is written in std C++ and it makes a matrix multiplication:
>>> so it uses only CPU and memory (no I/O, no network).
>>>
>>>
>>> I'm quite surprise that with CAP = 100% I got my results in about 600
>>> seconds and with CAP = 50% I got my results in about 1800 seconds (around 3
>>> times longer).
>>>
>>> For this kind of application I was expecting to get results in about 1200
>>> seconds (2 times longer) for the second scenario with respect to the first
>>> one.
>>>
>>> Of course, the HW and SW are exactly the same for the 2 experiments.
>>>
>>> Am I wrong or the CAP mechanism is not working well?
>> How many vcpus do you have?
>>
>> I'm not sure the CAP stuff actually does anything sensible with
>> multiple cpus -- e.g., a VM with 2 vcpus maxed out will typically
>> actually get 200%, so will a CAP force both of them to get only 25%
>> each?
>>
>> The other possibility if you've got multiple vcpus is that you're
>> running into something I call "concurrency hazard": if one vcpu grabs
>> a spinlock and then is paused in order to enforce the cap, and another
>> vcpu tries to grab that spinlock, it will just spin until the first
>> vcpu manages to wake up and release it again.
> Sorry, meant to add: if you have more than one vcpu and you want to
> limit the cpu footprint, you're much better off halving the number of
> vcpus rather than giving each vcpu only 50% of a cpu.
>
> I gave a presentation on why here, if you're interested:
>
> http://www.citrix.com/tv/#videos/2930
>
>   -George
Hi George,
thanks for your answer.

Our physical machine has two processors and two cores for each processors.

The virtual machine where our application run has just one VCPU that is 
pinned to a single core.
Instead, dom0 was not pinned to any particular core.

So I think that my problem could not be related to the presence of 4 vcpus.

Any idea?
M

Re: CAP and performance problem

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1352 bytes --]

On mar, 2013-05-21 at 13:54 +0200, Massimo Canonico wrote:
> Hi George,
> thanks for your answer.
> 
> Our physical machine has two processors and two cores for each processors.
> 
> The virtual machine where our application run has just one VCPU that is 
> pinned to a single core.
> Instead, dom0 was not pinned to any particular core.
> 
Ok, and what are you doing in Dom0? I guess almost nothing, is that the
case?

Also, when inspecting the vCPU utilization, do you see the VM vCPU busy
up to 100% without cap (or with cap set to 100) and, OTOH, up to 50%
with cap set to 50?

> So I think that my problem could not be related to the presence of 4 vcpus.
> 
Well, looks like a scheduling issue, or, in any case, one where
something is interacting with the scheduling. Can you perhaps boot Dom0
so that it uses only 2 or 3 cores (or arrange for that later, e.g., with
cpupools) and pin the vCPU of the VM on the 4th one?

I'm not sure what could be going on, but that sounds a reasonable way to
rule out as much room for interference as possible...

Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: CAP and performance problem

[Massimo Canonico]

Hi Dario,
inline my answers
On 05/21/2013 03:06 PM, Dario Faggioli wrote:
> On mar, 2013-05-21 at 13:54 +0200, Massimo Canonico wrote:
>> Hi George,
>> thanks for your answer.
>>
>> Our physical machine has two processors and two cores for each processors.
>>
>> The virtual machine where our application run has just one VCPU that is
>> pinned to a single core.
>> Instead, dom0 was not pinned to any particular core.
>>
> Ok, and what are you doing in Dom0? I guess almost nothing, is that the
> case?
yes
>
> Also, when inspecting the vCPU utilization, do you see the VM vCPU busy
> up to 100% without cap (or with cap set to 100) and, OTOH, up to 50%
> with cap set to 50?
I have monitored the CPU usage with xentop during the experiment and
I got what you said: my application always use the total amonut of CPU 
available.
> Well, looks like a scheduling issue, or, in any case, one where
> something is interacting with the scheduling. Can you perhaps boot Dom0
> so that it uses only 2 or 3 cores (or arrange for that later, e.g., with
> cpupools) and pin the vCPU of the VM on the 4th one?
In my experiments, I pin the vCPU of the VM on one core and other cores 
are pinned to Dom0.
[root@csitest ~]# xl vcpu-list
Name                                ID  VCPU   CPU State   Time(s) CPU 
Affinity
Domain-0                             0     0    0   ---     383.3  0
Domain-0                             0     1    0   -b-     358.1  0
Domain-0                             0     2    0   -b-     224.3  0
Domain-0                             0     3    0   r--     256.8  0
rubis-web                            1     0    2   -b-    9250.9  2

(rubis-web is, of course, the VM where my application run)

Is this configuration what you meant?

Thanks,
  Massimo



>
> I'm not sure what could be going on, but that sounds a reasonable way to
> rule out as much room for interference as possible...
>
> Dario
>

Re: CAP and performance problem

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2120 bytes --]

On mar, 2013-05-21 at 15:28 +0100, Massimo Canonico wrote:
> On 05/21/2013 03:06 PM, Dario Faggioli wrote:
> > Also, when inspecting the vCPU utilization, do you see the VM vCPU busy
> > up to 100% without cap (or with cap set to 100) and, OTOH, up to 50%
> > with cap set to 50?
> I have monitored the CPU usage with xentop during the experiment and
> I got what you said: my application always use the total amonut of CPU 
> available.
>
Ok.

> > Well, looks like a scheduling issue, or, in any case, one where
> > something is interacting with the scheduling. Can you perhaps boot Dom0
> > so that it uses only 2 or 3 cores (or arrange for that later, e.g., with
> > cpupools) and pin the vCPU of the VM on the 4th one?
> In my experiments, I pin the vCPU of the VM on one core and other cores 
> are pinned to Dom0.
> [root@csitest ~]# xl vcpu-list
> Name                                ID  VCPU   CPU State   Time(s) CPU 
> Affinity
> Domain-0                             0     0    0   ---     383.3  0
> Domain-0                             0     1    0   -b-     358.1  0
> Domain-0                             0     2    0   -b-     224.3  0
> Domain-0                             0     3    0   r--     256.8  0
> rubis-web                            1     0    2   -b-    9250.9  2
> 
> (rubis-web is, of course, the VM where my application run)
> 
> Is this configuration what you meant?
> 
Sort of. However, since (you said) you have 4 cores, what I was thinking
was more a situation where you have the 4 Dom0 vCPUs pinned to cores
0-2, and the VM vCPU pinned to core 3.

You should be able to achieve that by doing right this:

# xl vcpu-pin 0 all 0-2
# xl vcpu-pin rubis-web all 3

Does it make sense? Oh, and also, you have 4 _actual_cores_, right? Or
is there any hyperthreading involved?

Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: CAP and performance problem

[Massimo Canonico]

Hi dario,
> Sort of. However, since (you said) you have 4 cores, what I was thinking
> was more a situation where you have the 4 Dom0 vCPUs pinned to cores
> 0-2, and the VM vCPU pinned to core 3.
>
> You should be able to achieve that by doing right this:
>
> # xl vcpu-pin 0 all 0-2
> # xl vcpu-pin rubis-web all 3
I have set pin as you suggested:

[root@csitest ~]# xl vcpu-pin 0 all 0-2
[root@csitest ~]# xl vcpu-pin rubis-web all 3
[root@csitest ~]# xl vcpu-list
Name                                ID  VCPU   CPU State   Time(s) CPU 
Affinity
Domain-0                             0     0    0   -b-     384.5 0-2
Domain-0                             0     1    2   -b-     358.9 0-2
Domain-0                             0     2    2   -b-     224.6 0-2
Domain-0                             0     3    1   r--     257.0 0-2
rubis-web                            1     0    3   -b-    9251.6  3


> Does it make sense? Oh, and also, you have 4 _actual_cores_, right? Or
> is there any hyperthreading involved?
No hyperthreading are involved. At the end of this email I've pasted 
what /proc/cpuinfo says about cores (they are actual cores).

The experiment just finished said again that the service time is 3 times 
longer when CAP=50% with respect to the scenario with CAP=100%.

If you want me do more experiment, just let me know, I'll be happy to 
collaborate. I can also provide you my code: it's a very simple C++ code 
with a makefile that make you ready to run it in few seconds.

Cheers,
  Massimo

[root@csitest ~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU            5160  @ 3.00GHz
stepping        : 6
microcode       : 0xd2
cpu MHz         : 3000.186
cache size      : 4096 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu de tsc msr pae mce cx8 apic sep mca cmov pat 
clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good 
nopl pni monitor est ssse3 cx16 hypervisor lahf_lm dtherm
bogomips        : 6000.37
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU            5160  @ 3.00GHz
stepping        : 6
microcode       : 0xd2
cpu MHz         : 3000.186
cache size      : 4096 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu de tsc msr pae mce cx8 apic sep mca cmov pat 
clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good 
nopl pni monitor est ssse3 cx16 hypervisor lahf_lm dtherm
bogomips        : 6000.37
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU            5160  @ 3.00GHz
stepping        : 6
microcode       : 0xd2
cpu MHz         : 3000.186
cache size      : 4096 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu de tsc msr pae mce cx8 apic sep mca cmov pat 
clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good 
nopl pni monitor est ssse3 cx16 hypervisor lahf_lm dtherm
bogomips        : 6000.37
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU            5160  @ 3.00GHz
stepping        : 6
microcode       : 0xd2
cpu MHz         : 3000.186
cache size      : 4096 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu de tsc msr pae mce cx8 apic sep mca cmov pat 
clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good 
nopl pni monitor est ssse3 cx16 hypervisor lahf_lm dtherm
bogomips        : 6000.37
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

Re: CAP and performance problem

[Massimo Canonico]

I have run other tests, changing the run duration but
it seems that CAP mechanism has some problem.

I always get response time too much longer (3 times instead of 2).

Any idea about how to fix the problem?

Did someone try to make a similar experiment?

Thanks,
  M


On 05/21/2013 04:47 PM, Dario Faggioli wrote:
> On mar, 2013-05-21 at 15:28 +0100, Massimo Canonico wrote:
>> On 05/21/2013 03:06 PM, Dario Faggioli wrote:
>>> Also, when inspecting the vCPU utilization, do you see the VM vCPU busy
>>> up to 100% without cap (or with cap set to 100) and, OTOH, up to 50%
>>> with cap set to 50?
>> I have monitored the CPU usage with xentop during the experiment and
>> I got what you said: my application always use the total amonut of CPU
>> available.
>>
> Ok.
>
>>> Well, looks like a scheduling issue, or, in any case, one where
>>> something is interacting with the scheduling. Can you perhaps boot Dom0
>>> so that it uses only 2 or 3 cores (or arrange for that later, e.g., with
>>> cpupools) and pin the vCPU of the VM on the 4th one?
>> In my experiments, I pin the vCPU of the VM on one core and other cores
>> are pinned to Dom0.
>> [root@csitest ~]# xl vcpu-list
>> Name                                ID  VCPU   CPU State   Time(s) CPU
>> Affinity
>> Domain-0                             0     0    0   ---     383.3  0
>> Domain-0                             0     1    0   -b-     358.1  0
>> Domain-0                             0     2    0   -b-     224.3  0
>> Domain-0                             0     3    0   r--     256.8  0
>> rubis-web                            1     0    2   -b-    9250.9  2
>>
>> (rubis-web is, of course, the VM where my application run)
>>
>> Is this configuration what you meant?
>>
> Sort of. However, since (you said) you have 4 cores, what I was thinking
> was more a situation where you have the 4 Dom0 vCPUs pinned to cores
> 0-2, and the VM vCPU pinned to core 3.
>
> You should be able to achieve that by doing right this:
>
> # xl vcpu-pin 0 all 0-2
> # xl vcpu-pin rubis-web all 3
>
> Does it make sense? Oh, and also, you have 4 _actual_cores_, right? Or
> is there any hyperthreading involved?
>
> Dario
>

Re: CAP and performance problem

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 998 bytes --]

On mer, 2013-05-22 at 16:42 +0200, Massimo Canonico wrote:
> I have run other tests, changing the run duration but
> it seems that CAP mechanism has some problem.
> 
> I always get response time too much longer (3 times instead of 2).
> 
I see.

> Any idea about how to fix the problem?
> 
Not really, no... Although, I admit, it's a bit since I looked at how
cap was implemented in details and, even worse, I really never had the
chance to use or play with it.

> Did someone try to make a similar experiment?
> 
Well, you said you can share info and code for this right? If you're
keen to give me a pointer to the source and the instructions on how you
run it, I guess I can give it a try.

Let me know,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: CAP and performance problem

[Massimo Canonico]

Hi Dario,
first of thanks for your help.
>> Did someone try to make a similar experiment?
>>
> Well, you said you can share info and code for this right? If you're
> keen to give me a pointer to the source and the instructions on how you
> run it, I guess I can give it a try.
In [1] you can find the source code.
The steps to do in order to run my application follow:
tar xvfz code.tgz
cd code
make cliSer/server
cd cliSer/
./server

The application make 20 times a matrix mutiplication. At the end, it 
shows the time elapsed.

If you need any info, please do not esitate.

M

[1] 
https://docs.google.com/file/d/0B1wfqsFZVqJlbWxrbGhMYnJmS1k/edit?usp=sharing




> Let me know,
> Dario
>

Re: CAP and performance problem

[Massimo Canonico]

Hi Dario and hi all,
is there any news about this issue?

I do not want switch to another VMM hypervisor but I need this cap 
mechanism working.

Thanks,
  Massimo

On 05/24/2013 09:48 AM, Massimo Canonico wrote:
> Hi Dario,
> first of thanks for your help.
>>> Did someone try to make a similar experiment?
>>>
>> Well, you said you can share info and code for this right? If you're
>> keen to give me a pointer to the source and the instructions on how you
>> run it, I guess I can give it a try.
> In [1] you can find the source code.
> The steps to do in order to run my application follow:
> tar xvfz code.tgz
> cd code
> make cliSer/server
> cd cliSer/
> ./server
>
> The application make 20 times a matrix mutiplication. At the end, it 
> shows the time elapsed.
>
> If you need any info, please do not esitate.
>
> M
>
> [1] 
> https://docs.google.com/file/d/0B1wfqsFZVqJlbWxrbGhMYnJmS1k/edit?usp=sharing
>
>
>
>
>> Let me know,
>> Dario
>>

Re: CAP and performance problem

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 772 bytes --]

On mar, 2013-06-04 at 16:03 +0200, Massimo Canonico wrote:
> Hi Dario and hi all,
>
Hi,

> is there any news about this issue?
>
Not really I'm afraid.

> I do not want switch to another VMM hypervisor but I need this cap 
> mechanism working.
> 
I see. Well, I definitely want to give your scripts a try, even if it
were just for personal curiosity. I hoped to be able to do that last
week, let me see if I can find a slot for running your tests by today or
tomorrow, ok?

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: CAP and performance problem

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3643 bytes --]

On mar, 2013-05-21 at 11:02 +0200, Massimo Canonico wrote:
> Hi,
>
Hi again,

> I sent the following problem on xen-user ML without an answer. I hope 
> I'll get one in this ML.
> 
> My application is written in std C++ and it makes a matrix 
> multiplication: so it uses only CPU and memory (no I/O, no network).
> 
> I'm quite surprise that with CAP = 100% I got my results in about 600 
> seconds and with CAP = 50% I got my results in about 1800 seconds 
> (around 3 times longer).
> 
> For this kind of application I was expecting to get results in about 
> 1200 seconds (2 times longer) for the second scenario with respect to 
> the first one.
> 
> Of course, the HW and SW are exactly the same for the 2 experiments.
> 
> Am I wrong or the CAP mechanism is not working well?
> 
Ok, I found a minute to run your code myself on my test box. It's quite
a large one, but since the VM has only 1 vcpu, that shouldn't really
make much difference.

I configured vcpu-pinning in such a way that there should be no room for
interference of any kind, i.e., dedicating a core to the VM, and making
sure even his fellow thread is not busy (which matters in an
hyperthreaded system):

# xl vcpu-list 
Name                                ID  VCPU   CPU State   Time(s) CPU
Affinity
Domain-0                             0     0    7   -b-      38.7  0-7
Domain-0                             0     1    3   -b-       2.3  0-7
Domain-0                             0     2    2   -b-       3.3  0-7
Domain-0                             0     3    6   -b-       6.8  0-7
Domain-0                             0     4    4   -b-       3.2  0-7
Domain-0                             0     5    2   -b-       3.6  0-7
Domain-0                             0     6    4   -b-       2.1  0-7
Domain-0                             0     7    1   -b-       1.8  0-7
Domain-0                             0     8    0   -b-       2.2  0-7
Domain-0                             0     9    7   -b-       1.7  0-7
Domain-0                             0    10    1   -b-       1.8  0-7
Domain-0                             0    11    5   r--      10.4  0-7
Domain-0                             0    12    1   -b-       3.5  0-7
Domain-0                             0    13    2   -b-       3.5  0-7
Domain-0                             0    14    3   -b-       2.7  0-7
Domain-0                             0    15    0   -b-       1.9  0-7
vm1                                  1     0   11   -b-     677.0  11

The numbers I'm getting are, I think, much more consistent with the
expectations:

 * no cap:
  Client served in 299.024
  Client served in 298.783
  Client served in 298.445
 * cap 50%:
  Client served in 643.668
  Client served in 643.372
  Client served in 644.342

Which means time roughly doubles.

I tried without pinning as well, and I'm getting pretty much the same
values.

At this point, I'm not sure what could be going on on your side. If you
want to try producing some traces, we can help inspect them, looking for
something weird. You can find some information about how to produce and
better interpret traces in this blog post:

http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/

Perhaps you can share your VM config file and Dom0 configuration
(basically, Xen and Linux boot command lines), to check whether there is
something strange there. Also, you might have said this already (in
which case I forgot), what versions of Xen and Linux are we talking
about?

I really am out of good ideas... George, any clue?

Regards,
Dario


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: CAP and performance problem

[Massimo Canonico]

Hi Dario,
and thanks for these test.

I forgot to ask you which xen version has beed used for your experiments.

Cheers,
  Massimo

On 06/05/2013 07:11 PM, Dario Faggioli wrote:
> On mer, 2013-06-05 at 18:50 +0200, Dario Faggioli wrote:
>> The numbers I'm getting are, I think, much more consistent with the
>> expectations:
>>
>>   * no cap:
>>    Client served in 299.024
>>    Client served in 298.783
>>    Client served in 298.445
>>   * cap 50%:
>>    Client served in 643.668
>>    Client served in 643.372
>>    Client served in 644.342
>>
>> Which means time roughly doubles.
>>
> Oh, I've just gave it a go with cap=25%, which produced this "Client
> served in 1353.14"... So still slightly more, but close enough to 4
> times the times it get when at 100%...
>
> Dario
>

Re: CAP and performance problem

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 856 bytes --]

On mer, 2013-06-05 at 18:50 +0200, Dario Faggioli wrote:
> The numbers I'm getting are, I think, much more consistent with the
> expectations:
> 
>  * no cap:
>   Client served in 299.024
>   Client served in 298.783
>   Client served in 298.445
>  * cap 50%:
>   Client served in 643.668
>   Client served in 643.372
>   Client served in 644.342
> 
> Which means time roughly doubles.
> 
Oh, I've just gave it a go with cap=25%, which produced this "Client
served in 1353.14"... So still slightly more, but close enough to 4
times the times it get when at 100%...

Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: CAP and performance problem

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 702 bytes --]

On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote:
> Hi Dario,
> and thanks for these test.
> 
> I forgot to ask you which xen version has beed used for your experiments.
> 
Yeah, me too! :-)

I'm using xen-unstable, pulled yesterday (commit id
e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed on a
Debian unstable system. Dom0 kernel is a bit old, as it's a 3.6.

What about you?

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: CAP and performance problem

[Massimo Canonico]

On 06/06/2013 10:37 AM, Dario Faggioli wrote:
> On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote:
>> Hi Dario,
>> and thanks for these test.
>>
>> I forgot to ask you which xen version has beed used for your experiments.
>>
> Yeah, me too! :-)
>
> I'm using xen-unstable, pulled yesterday (commit id
> e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed on a
> Debian unstable system. Dom0 kernel is a bit old, as it's a 3.6.
>
> What about you?
xen 4.2.2
kernel dom0: 3.8.11-200.fc18.x86_64

M


>
> Regards,
> Dario
>

Re: CAP and performance problem

[George Dunlap]

On 05/06/13 17:50, Dario Faggioli wrote:
> On mar, 2013-05-21 at 11:02 +0200, Massimo Canonico wrote:
>> Hi,
>>
> Hi again,
>
>> I sent the following problem on xen-user ML without an answer. I hope
>> I'll get one in this ML.
>>
>> My application is written in std C++ and it makes a matrix
>> multiplication: so it uses only CPU and memory (no I/O, no network).
>>
>> I'm quite surprise that with CAP = 100% I got my results in about 600
>> seconds and with CAP = 50% I got my results in about 1800 seconds
>> (around 3 times longer).
>>
>> For this kind of application I was expecting to get results in about
>> 1200 seconds (2 times longer) for the second scenario with respect to
>> the first one.
>>
>> Of course, the HW and SW are exactly the same for the 2 experiments.
>>
>> Am I wrong or the CAP mechanism is not working well?
>>
> Ok, I found a minute to run your code myself on my test box. It's quite
> a large one, but since the VM has only 1 vcpu, that shouldn't really
> make much difference.
>
> I configured vcpu-pinning in such a way that there should be no room for
> interference of any kind, i.e., dedicating a core to the VM, and making
> sure even his fellow thread is not busy (which matters in an
> hyperthreaded system):
>
> # xl vcpu-list
> Name                                ID  VCPU   CPU State   Time(s) CPU
> Affinity
> Domain-0                             0     0    7   -b-      38.7  0-7
> Domain-0                             0     1    3   -b-       2.3  0-7
> Domain-0                             0     2    2   -b-       3.3  0-7
> Domain-0                             0     3    6   -b-       6.8  0-7
> Domain-0                             0     4    4   -b-       3.2  0-7
> Domain-0                             0     5    2   -b-       3.6  0-7
> Domain-0                             0     6    4   -b-       2.1  0-7
> Domain-0                             0     7    1   -b-       1.8  0-7
> Domain-0                             0     8    0   -b-       2.2  0-7
> Domain-0                             0     9    7   -b-       1.7  0-7
> Domain-0                             0    10    1   -b-       1.8  0-7
> Domain-0                             0    11    5   r--      10.4  0-7
> Domain-0                             0    12    1   -b-       3.5  0-7
> Domain-0                             0    13    2   -b-       3.5  0-7
> Domain-0                             0    14    3   -b-       2.7  0-7
> Domain-0                             0    15    0   -b-       1.9  0-7
> vm1                                  1     0   11   -b-     677.0  11
>
> The numbers I'm getting are, I think, much more consistent with the
> expectations:
>
>   * no cap:
>    Client served in 299.024
>    Client served in 298.783
>    Client served in 298.445
>   * cap 50%:
>    Client served in 643.668
>    Client served in 643.372
>    Client served in 644.342
>
> Which means time roughly doubles.
>
> I tried without pinning as well, and I'm getting pretty much the same
> values.
>
> At this point, I'm not sure what could be going on on your side. If you
> want to try producing some traces, we can help inspect them, looking for
> something weird. You can find some information about how to produce and
> better interpret traces in this blog post:
>
> http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/
>
> Perhaps you can share your VM config file and Dom0 configuration
> (basically, Xen and Linux boot command lines), to check whether there is
> something strange there. Also, you might have said this already (in
> which case I forgot), what versions of Xen and Linux are we talking
> about?
>
> I really am out of good ideas... George, any clue?

Well for one, from the scheduler's perspective, the promise isn't that 
you'll get 50% of the *performance*, but 50% of the *cpu time*.  I 
haven't been following the thread terribly closely, but I don't remember 
seeing any xentop or xentrace reports.  The first question is, other 
than performance, do you have any reason to believe that the VM is not 
getting 50% of the cpu time?

At some point while your test is running, could you execute the 
following command in dom0:

xentrace -D -e 0x21000 -T 10 /tmp/test.trace

This will take a 10-second trace of just the scheduling events, placing 
the result in /tmp/test.trace

Then download and build xenalyze from the hg repo here:

http://xenbits.xen.org/ext/xenalyze

and run he following command:

xenalyze -s /tmp/test.trace > /tmp/test.summary

And post the results here?

Thanks,
  -George

Re: CAP and performance problem

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1992 bytes --]

On gio, 2013-06-06 at 10:02 +0100, George Dunlap wrote:
> Well for one, from the scheduler's perspective, the promise isn't that 
> you'll get 50% of the *performance*, but 50% of the *cpu time*.  I 
> haven't been following the thread terribly closely, but I don't remember 
> seeing any xentop or xentrace reports.  
>
Yep, I suggested monitoring the vcpu usage, and Massimo reported a
correct 50% utilization for the vcpu in question in xentop.

> The first question is, other 
> than performance, do you have any reason to believe that the VM is not 
> getting 50% of the cpu time?
> 
It correctly gets 50% for me, and for Massimo too, as per what he said
in one other e-mail in the thread.

I agree that it's vcpu utilization and non guest performance we're in
control of, but for a purely CPU bound workload, I guess one can expect
some sort of relationship between the two things, can't him? Besides,
that's right what I get in my tests with Massimo's code. :-)

> At some point while your test is running, could you execute the 
> following command in dom0:
> 
> xentrace -D -e 0x21000 -T 10 /tmp/test.trace
> 
> This will take a 10-second trace of just the scheduling events, placing 
> the result in /tmp/test.trace
> 
> Then download and build xenalyze from the hg repo here:
> 
> http://xenbits.xen.org/ext/xenalyze
> 
> and run he following command:
> 
> xenalyze -s /tmp/test.trace > /tmp/test.summary
> 
> And post the results here?
> 
Yes, I suggested trying to produce some traces too. However, having seen
that I get numbers consistent with the expectations, I now wonder what
can interact with the scheduler and make Massimo's results different...

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: CAP and performance problem

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1314 bytes --]

On gio, 2013-06-06 at 10:57 +0200, Massimo Canonico wrote:
> On 06/06/2013 10:37 AM, Dario Faggioli wrote:
> > On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote:
> >> Hi Dario,
> >> and thanks for these test.
> >>
> >> I forgot to ask you which xen version has beed used for your experiments.
> >>
> > Yeah, me too! :-)
> >
> > I'm using xen-unstable, pulled yesterday (commit id
> > e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed on a
> > Debian unstable system. Dom0 kernel is a bit old, as it's a 3.6.
> >
> > What about you?
> xen 4.2.2
> kernel dom0: 3.8.11-200.fc18.x86_64
> 
I see. There shouldn't be much differences, from a credit scheduler
point of view (and even more so wrt the caps) between 4.2 and unstable.
Anyway, just to rule out all the possible doubts, can you build and
install unstable on your box?

What about the guest? PV or HVM? What's running in there? In my case it
was a PV guest, also running a Debian unstable, and using the same
kernel dom0 does.

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: CAP and performance problem

[Juergen Gross]

On 06.06.2013 10:57, Massimo Canonico wrote:
>
> On 06/06/2013 10:37 AM, Dario Faggioli wrote:
>> On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote:
>>> Hi Dario,
>>> and thanks for these test.
>>>
>>> I forgot to ask you which xen version has beed used for your experiments.
>>>
>> Yeah, me too! :-)
>>
>> I'm using xen-unstable, pulled yesterday (commit id
>> e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed on a
>> Debian unstable system. Dom0 kernel is a bit old, as it's a 3.6.
>>
>> What about you?
> xen 4.2.2
> kernel dom0: 3.8.11-200.fc18.x86_64

Just had an idea: is there any other load on the system during your test (other
domains, dom0 load)? If not, it could be that the power management is reducing
the cpu speed during idle (when the cap applies). This could lead to reduced
performance overall.

You can test this by setting the xen hypervisor boot option

cpufreq=none

and run your test again (with and without cap).


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
PBG PDG ES&S SWE OS6                   Telephone: +49 (0) 89 3222 2967
Fujitsu Technology Solutions              e-mail: juergen.gross@ts.fujitsu.com
Domagkstr. 28                           Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

Re: CAP and performance problem

[George Dunlap]

On 06/06/13 11:39, Juergen Gross wrote:
> On 06.06.2013 10:57, Massimo Canonico wrote:
>>
>> On 06/06/2013 10:37 AM, Dario Faggioli wrote:
>>> On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote:
>>>> Hi Dario,
>>>> and thanks for these test.
>>>>
>>>> I forgot to ask you which xen version has beed used for your 
>>>> experiments.
>>>>
>>> Yeah, me too! :-)
>>>
>>> I'm using xen-unstable, pulled yesterday (commit id
>>> e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed on a
>>> Debian unstable system. Dom0 kernel is a bit old, as it's a 3.6.
>>>
>>> What about you?
>> xen 4.2.2
>> kernel dom0: 3.8.11-200.fc18.x86_64
>
> Just had an idea: is there any other load on the system during your 
> test (other
> domains, dom0 load)? If not, it could be that the power management is 
> reducing
> the cpu speed during idle (when the cap applies). This could lead to 
> reduced
> performance overall.
>
> You can test this by setting the xen hypervisor boot option
>
> cpufreq=none
>
> and run your test again (with and without cap).

Ah, genius Juergen!  That would make total sense.

  -George

Re: CAP and performance problem

[Massimo Canonico]

On 06/06/2013 12:44 PM, George Dunlap wrote:
> On 06/06/13 11:39, Juergen Gross wrote:
>> On 06.06.2013 10:57, Massimo Canonico wrote:
>>>
>>> On 06/06/2013 10:37 AM, Dario Faggioli wrote:
>>>> On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote:
>>>>> Hi Dario,
>>>>> and thanks for these test.
>>>>>
>>>>> I forgot to ask you which xen version has beed used for your 
>>>>> experiments.
>>>>>
>>>> Yeah, me too! :-)
>>>>
>>>> I'm using xen-unstable, pulled yesterday (commit id
>>>> e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed on a
>>>> Debian unstable system. Dom0 kernel is a bit old, as it's a 3.6.
>>>>
>>>> What about you?
>>> xen 4.2.2
>>> kernel dom0: 3.8.11-200.fc18.x86_64
>>
>> Just had an idea: is there any other load on the system during your 
>> test (other
>> domains, dom0 load)? If not, it could be that the power management is 
>> reducing
>> the cpu speed during idle (when the cap applies). This could lead to 
>> reduced
>> performance overall.
>>
>> You can test this by setting the xen hypervisor boot option
>>
>> cpufreq=none
>>
>> and run your test again (with and without cap).
>
> Ah, genius Juergen!  That would make total sense.
>
>  -George
>
Unfortunately, this did not change much. I set "cpufreq=none" in the 
boot line and restart my experiment.
With no cap I got 298.029
with cap=50% I got 910.272
(average values of 3 experiments for each cap setting)

dom0 load during the experiment is less than 1% (that says xentop)

Thanks,
  M

Re: CAP and performance problem

[George Dunlap]

On 06/06/13 13:52, Massimo Canonico wrote:
>
> On 06/06/2013 12:44 PM, George Dunlap wrote:
>> On 06/06/13 11:39, Juergen Gross wrote:
>>> On 06.06.2013 10:57, Massimo Canonico wrote:
>>>>
>>>> On 06/06/2013 10:37 AM, Dario Faggioli wrote:
>>>>> On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote:
>>>>>> Hi Dario,
>>>>>> and thanks for these test.
>>>>>>
>>>>>> I forgot to ask you which xen version has beed used for your 
>>>>>> experiments.
>>>>>>
>>>>> Yeah, me too! :-)
>>>>>
>>>>> I'm using xen-unstable, pulled yesterday (commit id
>>>>> e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed 
>>>>> on a
>>>>> Debian unstable system. Dom0 kernel is a bit old, as it's a 3.6.
>>>>>
>>>>> What about you?
>>>> xen 4.2.2
>>>> kernel dom0: 3.8.11-200.fc18.x86_64
>>>
>>> Just had an idea: is there any other load on the system during your 
>>> test (other
>>> domains, dom0 load)? If not, it could be that the power management 
>>> is reducing
>>> the cpu speed during idle (when the cap applies). This could lead to 
>>> reduced
>>> performance overall.
>>>
>>> You can test this by setting the xen hypervisor boot option
>>>
>>> cpufreq=none
>>>
>>> and run your test again (with and without cap).
>>
>> Ah, genius Juergen!  That would make total sense.
>>
>>  -George
>>
> Unfortunately, this did not change much. I set "cpufreq=none" in the 
> boot line and restart my experiment.
> With no cap I got 298.029
> with cap=50% I got 910.272
> (average values of 3 experiments for each cap setting)
>
> dom0 load during the experiment is less than 1% (that says xentop)

Have you checked your BIOS for performance settings?

  -George

Re: CAP and performance problem

[Juergen Gross]

On 06.06.2013 14:52, Massimo Canonico wrote:
>
> On 06/06/2013 12:44 PM, George Dunlap wrote:
>> On 06/06/13 11:39, Juergen Gross wrote:
>>> On 06.06.2013 10:57, Massimo Canonico wrote:
>>>>
>>>> On 06/06/2013 10:37 AM, Dario Faggioli wrote:
>>>>> On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote:
>>>>>> Hi Dario,
>>>>>> and thanks for these test.
>>>>>>
>>>>>> I forgot to ask you which xen version has beed used for your experiments.
>>>>>>
>>>>> Yeah, me too! :-)
>>>>>
>>>>> I'm using xen-unstable, pulled yesterday (commit id
>>>>> e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed on a
>>>>> Debian unstable system. Dom0 kernel is a bit old, as it's a 3.6.
>>>>>
>>>>> What about you?
>>>> xen 4.2.2
>>>> kernel dom0: 3.8.11-200.fc18.x86_64
>>>
>>> Just had an idea: is there any other load on the system during your test (other
>>> domains, dom0 load)? If not, it could be that the power management is reducing
>>> the cpu speed during idle (when the cap applies). This could lead to reduced
>>> performance overall.
>>>
>>> You can test this by setting the xen hypervisor boot option
>>>
>>> cpufreq=none
>>>
>>> and run your test again (with and without cap).
>>
>> Ah, genius Juergen! That would make total sense.
>>
>> -George
>>
> Unfortunately, this did not change much. I set "cpufreq=none" in the boot line

You added the boot parameter for the hypervisor, not dom0? And (please forgive
my paranoia) you rebooted the complete system after that?

> and restart my experiment.
> With no cap I got 298.029
> with cap=50% I got 910.272
> (average values of 3 experiments for each cap setting)
>
> dom0 load during the experiment is less than 1% (that says xentop)

What was the load reported by xentop for your domu?

Could you try:

xl vcpu-list; sleep 10; xl vcpu-list

when the test is running and post the output?


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
PBG PDG ES&S SWE OS6                   Telephone: +49 (0) 89 3222 2967
Fujitsu Technology Solutions              e-mail: juergen.gross@ts.fujitsu.com
Domagkstr. 28                           Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

Re: CAP and performance problem

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1163 bytes --]

On gio, 2013-06-06 at 14:52 +0200, Massimo Canonico wrote:
> On 06/06/2013 12:44 PM, George Dunlap wrote:
> >> You can test this by setting the xen hypervisor boot option
> >>
> >> cpufreq=none
> >>
> >> and run your test again (with and without cap).
> >
> > Ah, genius Juergen!  That would make total sense.
> >
> >  -George
> >
> Unfortunately, this did not change much. I set "cpufreq=none" in the 
> boot line and restart my experiment.
> With no cap I got 298.029
> with cap=50% I got 910.272
> (average values of 3 experiments for each cap setting)
> 
Mmm... 298.029 is a lot faster than what you reported before, though...
Are this the exact same experiments?

In fact, in the first e-mail of this thread you said that "with CAP =
100% I got my results in about 600 seconds". So, have you changed
something or the cpufred=none did at least have some effect?

Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: CAP and performance problem

[Massimo Canonico]

On 06/06/2013 03:03 PM, Dario Faggioli wrote:
> On gio, 2013-06-06 at 14:52 +0200, Massimo Canonico wrote:
>> On 06/06/2013 12:44 PM, George Dunlap wrote:
>>>> You can test this by setting the xen hypervisor boot option
>>>>
>>>> cpufreq=none
>>>>
>>>> and run your test again (with and without cap).
>>> Ah, genius Juergen!  That would make total sense.
>>>
>>>   -George
>>>
>> Unfortunately, this did not change much. I set "cpufreq=none" in the
>> boot line and restart my experiment.
>> With no cap I got 298.029
>> with cap=50% I got 910.272
>> (average values of 3 experiments for each cap setting)
>>
> Mmm... 298.029 is a lot faster than what you reported before, though...
> Are this the exact same experiments?
>
> In fact, in the first e-mail of this thread you said that "with CAP =
> 100% I got my results in about 600 seconds". So, have you changed
> something or the cpufred=none did at least have some effect?
>
> Dario

Hi Dario,
I only changed the workload from "40 times matrix multiplications" to 
"20 times matrix multiplications" to get the results faster, since I do 
not think that this makes any difference in perfomance comparisin 
between cap/no cap.

M

Re: CAP and performance problem

[Massimo Canonico]

On 06/06/2013 03:02 PM, Juergen Gross wrote:
> On 06.06.2013 14:52, Massimo Canonico wrote:
>>
>> On 06/06/2013 12:44 PM, George Dunlap wrote:
>>> On 06/06/13 11:39, Juergen Gross wrote:
>>>> On 06.06.2013 10:57, Massimo Canonico wrote:
>>>>>
>>>>> On 06/06/2013 10:37 AM, Dario Faggioli wrote:
>>>>>> On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote:
>>>>>>> Hi Dario,
>>>>>>> and thanks for these test.
>>>>>>>
>>>>>>> I forgot to ask you which xen version has beed used for your 
>>>>>>> experiments.
>>>>>>>
>>>>>> Yeah, me too! :-)
>>>>>>
>>>>>> I'm using xen-unstable, pulled yesterday (commit id
>>>>>> e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed 
>>>>>> on a
>>>>>> Debian unstable system. Dom0 kernel is a bit old, as it's a 3.6.
>>>>>>
>>>>>> What about you?
>>>>> xen 4.2.2
>>>>> kernel dom0: 3.8.11-200.fc18.x86_64
>>>>
>>>> Just had an idea: is there any other load on the system during your 
>>>> test (other
>>>> domains, dom0 load)? If not, it could be that the power management 
>>>> is reducing
>>>> the cpu speed during idle (when the cap applies). This could lead 
>>>> to reduced
>>>> performance overall.
>>>>
>>>> You can test this by setting the xen hypervisor boot option
>>>>
>>>> cpufreq=none
>>>>
>>>> and run your test again (with and without cap).
>>>
>>> Ah, genius Juergen! That would make total sense.
>>>
>>> -George
>>>
>> Unfortunately, this did not change much. I set "cpufreq=none" in the 
>> boot line
>
> You added the boot parameter for the hypervisor, not dom0? 
Fedora, after few seconds, asks you which kernel do you want to use. You 
can add some parameter in the command line who launches the kernel. So, 
I add "cpufreq=none" in the command line.
> And (please forgive
> my paranoia) you rebooted the complete system after that?
Your paranoia is also mine, I always reboot my machines. Thanks for asking.
>> and restart my experiment.
>> With no cap I got 298.029
>> with cap=50% I got 910.272
>> (average values of 3 experiments for each cap setting)
>>
>> dom0 load during the experiment is less than 1% (that says xentop)
>
> What was the load reported by xentop for your domu?
My virtual machines is called rubis-web and during "cap=50%" experiment 
I can see this values in xentop for this specific domain.
>
> Could you try:
>
> xl vcpu-list; sleep 10; xl vcpu-list
>
> when the test is running and post the output?
>
>
> Juergen
>
here we go:

[root@csitest ~]# xl vcpu-list; sleep 10; xl vcpu-list
Name                                ID  VCPU   CPU State   Time(s) CPU 
Affinity
Domain-0                             0     0    0   ---      23.0  0
Domain-0                             0     1    0   ---      10.5  0
Domain-0                             0     2    0   ---       8.5  0
Domain-0                             0     3    0   r--       6.8  0
rubis-web                            1     0    2   r--    2968.5  2
Name                                ID  VCPU   CPU State   Time(s) CPU 
Affinity
Domain-0                             0     0    0   ---      23.0  0
Domain-0                             0     1    0   r--      10.6  0
Domain-0                             0     2    0   ---       8.5  0
Domain-0                             0     3    0   ---       6.8  0
rubis-web                            1     0    2   r--    2973.5  2

Concerning the George's question:
> Have you checked your BIOS for performance settings?
I'm not sure what you mean for "BIOS perfomance settings". To my best 
knowledge, in the BIOS I have to be sure that the "hw virtualization" is 
enabled.

Thanks for your time,
  M

Re: CAP and performance problem

[George Dunlap]

On 06/06/13 14:55, Massimo Canonico wrote:
>
> On 06/06/2013 03:02 PM, Juergen Gross wrote:
>> On 06.06.2013 14:52, Massimo Canonico wrote:
>>>
>>> On 06/06/2013 12:44 PM, George Dunlap wrote:
>>>> On 06/06/13 11:39, Juergen Gross wrote:
>>>>> On 06.06.2013 10:57, Massimo Canonico wrote:
>>>>>>
>>>>>> On 06/06/2013 10:37 AM, Dario Faggioli wrote:
>>>>>>> On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote:
>>>>>>>> Hi Dario,
>>>>>>>> and thanks for these test.
>>>>>>>>
>>>>>>>> I forgot to ask you which xen version has beed used for your 
>>>>>>>> experiments.
>>>>>>>>
>>>>>>> Yeah, me too! :-)
>>>>>>>
>>>>>>> I'm using xen-unstable, pulled yesterday (commit id
>>>>>>> e430510e5cbbfcdc1077739292def633e70fedea), compiled and 
>>>>>>> installed on a
>>>>>>> Debian unstable system. Dom0 kernel is a bit old, as it's a 3.6.
>>>>>>>
>>>>>>> What about you?
>>>>>> xen 4.2.2
>>>>>> kernel dom0: 3.8.11-200.fc18.x86_64
>>>>>
>>>>> Just had an idea: is there any other load on the system during 
>>>>> your test (other
>>>>> domains, dom0 load)? If not, it could be that the power management 
>>>>> is reducing
>>>>> the cpu speed during idle (when the cap applies). This could lead 
>>>>> to reduced
>>>>> performance overall.
>>>>>
>>>>> You can test this by setting the xen hypervisor boot option
>>>>>
>>>>> cpufreq=none
>>>>>
>>>>> and run your test again (with and without cap).
>>>>
>>>> Ah, genius Juergen! That would make total sense.
>>>>
>>>> -George
>>>>
>>> Unfortunately, this did not change much. I set "cpufreq=none" in the 
>>> boot line
>>
>> You added the boot parameter for the hypervisor, not dom0? 
> Fedora, after few seconds, asks you which kernel do you want to use. 
> You can add some parameter in the command line who launches the 
> kernel. So, I add "cpufreq=none" in the command line.
>> And (please forgive
>> my paranoia) you rebooted the complete system after that?
> Your paranoia is also mine, I always reboot my machines. Thanks for 
> asking.
>>> and restart my experiment.
>>> With no cap I got 298.029
>>> with cap=50% I got 910.272
>>> (average values of 3 experiments for each cap setting)
>>>
>>> dom0 load during the experiment is less than 1% (that says xentop)
>>
>> What was the load reported by xentop for your domu?
> My virtual machines is called rubis-web and during "cap=50%" 
> experiment I can see this values in xentop for this specific domain.
>>
>> Could you try:
>>
>> xl vcpu-list; sleep 10; xl vcpu-list
>>
>> when the test is running and post the output?
>>
>>
>> Juergen
>>
> here we go:
>
> [root@csitest ~]# xl vcpu-list; sleep 10; xl vcpu-list
> Name                                ID  VCPU   CPU State   Time(s) CPU 
> Affinity
> Domain-0                             0     0    0   ---      23.0 0
> Domain-0                             0     1    0   ---      10.5 0
> Domain-0                             0     2    0   ---       8.5 0
> Domain-0                             0     3    0   r--       6.8 0
> rubis-web                            1     0    2   r--    2968.5 2
> Name                                ID  VCPU   CPU State   Time(s) CPU 
> Affinity
> Domain-0                             0     0    0   ---      23.0 0
> Domain-0                             0     1    0   r--      10.6 0
> Domain-0                             0     2    0   ---       8.5 0
> Domain-0                             0     3    0   ---       6.8 0
> rubis-web                            1     0    2   r--    2973.5 2
>
> Concerning the George's question:
>> Have you checked your BIOS for performance settings?
> I'm not sure what you mean for "BIOS perfomance settings". To my best 
> knowledge, in the BIOS I have to be sure that the "hw virtualization" 
> is enabled.

Some BIOSes have settings that will automatically mess around with the 
performance settings of the processor.  This could be automatically 
slowing the core down because it's only 50% busy. This would be 
different on each BIOS.  Just take a quick look for anything that seems 
to say something about performance, and set it to "max" or "performance" 
mode (as opposed to say, power-saving or balanced mode).

You might also try playing around with the "turbo" mode if it's 
available -- maybe the "turbo" mode takes a bit of time to get going, 
and running at 50% never kicks it in.  If you disable "turbo" mode, you 
may find that with no cap you get 450s instead of 300s.

  -George

Re: CAP and performance problem

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3168 bytes --]

On gio, 2013-06-06 at 15:55 +0200, Massimo Canonico wrote:
> On 06/06/2013 03:02 PM, Juergen Gross wrote:
> > You added the boot parameter for the hypervisor, not dom0? 
> Fedora, after few seconds, asks you which kernel do you want to use. You 
> can add some parameter in the command line who launches the kernel. So, 
> I add "cpufreq=none" in the command line.
>
IIRC, that won't do, as you're most likely modifying Dom0 command line,
you need to be sure you're adding that bit to the Xen boot command
line. 

I think you can do that dynamically, by editing the grub2 parameters
on-line, as you're saying you did, but you have to make sure you affect
Xen and not Linux. IIRC

Otherwise, you should go to /etc/default/grub (I think), look for
something like GRUB_CMDLINE_XEN (I think) add the "cpufreq=none" string
there and regenerate grub.cfg by running:

# grub2-mkconfig -o /boot/grub2/grub.cfg

After that, I recommend double checking that /boot/grub2/grub.cfg has
it, and has it in the right place.

Feel free to post it here, if you're not sure about that...

> > Could you try:
> >
> > xl vcpu-list; sleep 10; xl vcpu-list
> >
> > when the test is running and post the output?
> >
> >
> > Juergen
> >
> here we go:
> 
> [root@csitest ~]# xl vcpu-list; sleep 10; xl vcpu-list
> Name                                ID  VCPU   CPU State   Time(s) CPU 
> Affinity
> Domain-0                             0     0    0   ---      23.0  0
> Domain-0                             0     1    0   ---      10.5  0
> Domain-0                             0     2    0   ---       8.5  0
> Domain-0                             0     3    0   r--       6.8  0
> rubis-web                            1     0    2   r--    2968.5  2
> Name                                ID  VCPU   CPU State   Time(s) CPU 
> Affinity
> Domain-0                             0     0    0   ---      23.0  0
> Domain-0                             0     1    0   r--      10.6  0
> Domain-0                             0     2    0   ---       8.5  0
> Domain-0                             0     3    0   ---       6.8  0
> rubis-web                            1     0    2   r--    2973.5  2
> 
So, rubis-web's vcpu ran for 5 secs, which sounds right. I see why
Jurgen asked you to do that, but not how he plans to squeeze anything
more than that from these numbers... Waiting for him to enlighten us
again! ;-P

> Concerning the George's question:
> > Have you checked your BIOS for performance settings?
> I'm not sure what you mean for "BIOS perfomance settings". To my best 
> knowledge, in the BIOS I have to be sure that the "hw virtualization" is 
> enabled.
> 
I think he was referring to something related to poerformance vs. power
management, i.e., cpufreq/speedstep tweaks, and stuff like that, if ou
have any in the BIOS... Do you?

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: CAP and performance problem

[Massimo Canonico]

On 06/06/2013 04:12 PM, Dario Faggioli wrote:
> On gio, 2013-06-06 at 15:55 +0200, Massimo Canonico wrote:
>> On 06/06/2013 03:02 PM, Juergen Gross wrote:
>>> You added the boot parameter for the hypervisor, not dom0?
>> Fedora, after few seconds, asks you which kernel do you want to use. You
>> can add some parameter in the command line who launches the kernel. So,
>> I add "cpufreq=none" in the command line.
>>
> IIRC, that won't do, as you're most likely modifying Dom0 command line,
> you need to be sure you're adding that bit to the Xen boot command
> line.
>
> I think you can do that dynamically, by editing the grub2 parameters
> on-line, as you're saying you did, but you have to make sure you affect
> Xen and not Linux. IIRC
>
> Otherwise, you should go to /etc/default/grub (I think), look for
> something like GRUB_CMDLINE_XEN (I think) add the "cpufreq=none" string
> there and regenerate grub.cfg by running:
>
> # grub2-mkconfig -o /boot/grub2/grub.cfg
>
> After that, I recommend double checking that /boot/grub2/grub.cfg has
> it, and has it in the right place.
>
> Feel free to post it here, if you're not sure about that...
>
>>> Could you try:
>>>
>>> xl vcpu-list; sleep 10; xl vcpu-list
>>>
>>> when the test is running and post the output?
>>>
>>>
>>> Juergen
>>>
>> here we go:
>>
>> [root@csitest ~]# xl vcpu-list; sleep 10; xl vcpu-list
>> Name                                ID  VCPU   CPU State   Time(s) CPU
>> Affinity
>> Domain-0                             0     0    0   ---      23.0  0
>> Domain-0                             0     1    0   ---      10.5  0
>> Domain-0                             0     2    0   ---       8.5  0
>> Domain-0                             0     3    0   r--       6.8  0
>> rubis-web                            1     0    2   r--    2968.5  2
>> Name                                ID  VCPU   CPU State   Time(s) CPU
>> Affinity
>> Domain-0                             0     0    0   ---      23.0  0
>> Domain-0                             0     1    0   r--      10.6  0
>> Domain-0                             0     2    0   ---       8.5  0
>> Domain-0                             0     3    0   ---       6.8  0
>> rubis-web                            1     0    2   r--    2973.5  2
>>
> So, rubis-web's vcpu ran for 5 secs, which sounds right. I see why
> Jurgen asked you to do that, but not how he plans to squeeze anything
> more than that from these numbers... Waiting for him to enlighten us
> again! ;-P
>
>> Concerning the George's question:
>>> Have you checked your BIOS for performance settings?
>> I'm not sure what you mean for "BIOS perfomance settings". To my best
>> knowledge, in the BIOS I have to be sure that the "hw virtualization" is
>> enabled.
>>
> I think he was referring to something related to poerformance vs. power
> management, i.e., cpufreq/speedstep tweaks, and stuff like that, if ou
> have any in the BIOS... Do you?
>
> Regards,
> Dario
>
Actually it seems that the problem is related to a BIOS setting. In my 
HP server, there were a parameter called "HP power regulator" that I set 
to "HP Static High-performance power" (before it was set to "dynamic 
performance power") and now I get what it is expected: cap 50% make my 
application twice longer than when cap is 100%.

Thanks to all of you guys. I think that this should be reported in the 
doc where the cap mechanism is explained.

Cheer,
  Massimo

Re: CAP and performance problem

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1424 bytes --]

On gio, 2013-06-06 at 17:27 +0200, Massimo Canonico wrote:
> Actually it seems that the problem is related to a BIOS setting. In my 
> HP server, there were a parameter called "HP power regulator" that I set 
> to "HP Static High-performance power" (before it was set to "dynamic 
> performance power") and now I get what it is expected: cap 50% make my 
> application twice longer than when cap is 100%.
> 
Cool! :-)

> Thanks to all of you guys. I think that this should be reported in the 
> doc where the cap mechanism is explained.
> 
Well, yes, I guess we can put there something like the following:

"Beware of bugous BIOSes, providing obscure options with fancy names,
 promising to solve world hunger and all the other nasty problems out
 there if you check them, while all they do is just messing up with
 your OS power management strategies (for your ease and convenience)"

:-D :-D

Jokes apart, we may want to add a line there, perhaps on the Wiki, but I
honestly don't think this is by any means the cap mechanism's. :-)

Glad you have it working... I guess that means you can stay on Xen,
right? :-P

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: CAP and performance problem

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 618 bytes --]

On gio, 2013-06-06 at 17:42 +0200, Dario Faggioli wrote:
> Jokes apart, we may want to add a line there, perhaps on the Wiki, but I
> honestly don't think this is by any means the cap mechanism's. :-)
                                                               ^fault

What I meant is: I don't think this can be considered cap's fault.

Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: CAP and performance problem

[George Dunlap]

On 06/06/13 16:54, Dario Faggioli wrote:
> On gio, 2013-06-06 at 17:42 +0200, Dario Faggioli wrote:
>> Jokes apart, we may want to add a line there, perhaps on the Wiki, but I
>> honestly don't think this is by any means the cap mechanism's. :-)
>                                                                 ^fault
>
> What I meant is: I don't think this can be considered cap's fault.

It's certainly not cap's fault, but it wouldn't hurt to just put a 
comment saying something like,

"NB that many systems have features that will scale down the computing 
power of a cpu that is not 100% utilized.  This can be in the operating 
system, but can also sometimes be below the operating system in the 
BIOS.  If you set a cap such that individual cores are running at less 
than 100%, this may have an impact on the performance of your workload 
over and above the impact of the cap. For example, a workload that may 
take 300s with a cap of 100% may take 900s with a cap of 50%, because in 
addition to being run only 50% of the time, the cpu is running in a 
lower state and so the 50% you get is less powerful.  If this is a 
problem, look at performance and cpufreq options in our operating system 
and your BIOS."

  -George

Re: CAP and performance problem

[Juergen Gross]

On 06.06.2013 16:12, Dario Faggioli wrote:
> On gio, 2013-06-06 at 15:55 +0200, Massimo Canonico wrote:
>> On 06/06/2013 03:02 PM, Juergen Gross wrote:
>>> Could you try:
>>>
>>> xl vcpu-list; sleep 10; xl vcpu-list
>>>
>>> when the test is running and post the output?
>>>
>>>
>>> Juergen
>>>
>> here we go:
>>
>> [root@csitest ~]# xl vcpu-list; sleep 10; xl vcpu-list
>> Name                                ID  VCPU   CPU State   Time(s) CPU
>> Affinity
>> Domain-0                             0     0    0   ---      23.0  0
>> Domain-0                             0     1    0   ---      10.5  0
>> Domain-0                             0     2    0   ---       8.5  0
>> Domain-0                             0     3    0   r--       6.8  0
>> rubis-web                            1     0    2   r--    2968.5  2
>> Name                                ID  VCPU   CPU State   Time(s) CPU
>> Affinity
>> Domain-0                             0     0    0   ---      23.0  0
>> Domain-0                             0     1    0   r--      10.6  0
>> Domain-0                             0     2    0   ---       8.5  0
>> Domain-0                             0     3    0   ---       6.8  0
>> rubis-web                            1     0    2   r--    2973.5  2
>>
> So, rubis-web's vcpu ran for 5 secs, which sounds right. I see why
> Jurgen asked you to do that, but not how he plans to squeeze anything
> more than that from these numbers... Waiting for him to enlighten us
> again! ;-P

Just for the records: I only wanted to be sure the problem isn't in the xen
scheduler. And this seems to be proved by the numbers.


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
PBG PDG ES&S SWE OS6                   Telephone: +49 (0) 89 3222 2967
Fujitsu Technology Solutions              e-mail: juergen.gross@ts.fujitsu.com
Domagkstr. 28                           Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

Re: CAP and performance problem

[Massimo Canonico]

On 06/06/2013 06:27 PM, George Dunlap wrote:
> On 06/06/13 16:54, Dario Faggioli wrote:
>> On gio, 2013-06-06 at 17:42 +0200, Dario Faggioli wrote:
>>> Jokes apart, we may want to add a line there, perhaps on the Wiki, 
>>> but I
>>> honestly don't think this is by any means the cap mechanism's. :-)
>> ^fault
>>
>> What I meant is: I don't think this can be considered cap's fault.
>
> It's certainly not cap's fault, but it wouldn't hurt to just put a 
> comment saying something like,
>
> "NB that many systems have features that will scale down the computing 
> power of a cpu that is not 100% utilized.  This can be in the 
> operating system, but can also sometimes be below the operating system 
> in the BIOS.  If you set a cap such that individual cores are running 
> at less than 100%, this may have an impact on the performance of your 
> workload over and above the impact of the cap. For example, a workload 
> that may take 300s with a cap of 100% may take 900s with a cap of 50%, 
> because in addition to being run only 50% of the time, the cpu is 
> running in a lower state and so the 50% you get is less powerful.  If 
> this is a problem, look at performance and cpufreq options in our 
> operating system and your BIOS."
>
>  -George

Thanks George, this is exactly what I meant and I think it is quite 
important to put this "warning" in evidence.
M