Re: AW: Re: AW: Re: Xen 3.4 strange behaviour as compared to Xen 3.3.1

Re: AW: Re: AW: Re: Xen 3.4 strange behaviour as compared to Xen 3.3.1

[Carsten Schiers]

Sorry, I think it could also be that the CPU mask is somehow modified in the
hypercall itself. Too much code for me to realy understand. 

Just a guess, but does Changeset 18898 take care of cpufreq=dom0-kernel case? It's
patching the hypercall 52 and the cpufreq_ondemand.c of Xen near a modfication of
a CPU mask variable, but no change in the cpufreq_ondemand.c of the Dom0 kernel. 
It also seems to fit into the time window, doesn't it?

BR,
Carsten.

----- Originalnachricht -----
Von: Carsten Schiers <carsten@schiers.de>
Gesendet: Die, 2.6.2009 17:28
An: Carsten Schiers <carsten@schiers.de> ; keir.fraser <keir.fraser@eu.citrix.com> ; xen-devel <xen-devel@lists.xensource.com> ; mark.langsdorf@amd.com
Betreff: AW: Re: AW: Re: AW: Re: [Xen-devel] Xen 3.4 strange behaviour as compared to Xen 3.3.1

I managed to narrowed the problem down to the fact that inside dbs_calculate_load, which 
submits a hypercall to receive the load of all CPUs and then loops through all of them
to calculate the combined load (which is a max of all (wall-idle)/wall scaled to 100).

It will not loop as policy->cpus, which is a bitmap of CPUs is zero. I am currently stuck 
and not able to find out, where this is set. It seems that in the drivers/cpufreq dir, no 
change was made at all, so it must be somewhere else.

I thought, maybe somebody that changed something that has to do with this bitmap might
come up with something like: uuuh, I missed that one...

I honestly have no idea what CPU hotpluging is, as I only have a simple mainstream board
with a normal dual-core CPU, but the only change that is somehow related to CPUs that 
I recently came accross was that. 

Comments, ideas, help?

BR,
Carsten.



----- Originalnachricht -----
Von: Carsten Schiers <carsten@schiers.de>
Gesendet: Sam, 30.5.2009 15:29
An: keir.fraser <keir.fraser@eu.citrix.com> ; xen-devel <xen-devel@lists.xensource.com>
Betreff: AW: Re: AW: Re: AW: Re: [Xen-devel] Xen 3.4 strange behaviour as compared to Xen 3.3.1

No it's the kernel of Xen 3.4.0 with the same config. I will try later with the 
older kernel, which is the one of 06.04.2009 (I have not realy understood the
tagging or versioning of the Xen kernel). The other combination would be Xen 3.3.1
with the latest kernel. That's how we narrow it down to hypervisor vs. kernel.

As I use cpufreq=dom0-kernel and enable cpufreq modules in drivers/cpufreq/Kconfig,
I guess it should use the hypercall for idle time. But to double check, I generated
load in dom0. No difference. It simply doesn't step upwards. The kernel with cpufreq
debugging is enabled. I will also try it, maybe it's loging something usefull.

BR,
Carsten.

-----Ursprüngliche Nachricht-----
Von: Keir Fraser [mailto:keir.fraser@eu.citrix.com] 
Gesendet: Samstag, 30. Mai 2009 11:30
An: Carsten Schiers; xen-devel
Betreff: Re: AW: Re: AW: Re: [Xen-devel] Xen 3.4 strange behaviour as compared to Xen 3.3.1

Are you using the exact same dom0 kernel as before, when it worked? One
theory would be that dom0 is looking at its own idle stats, and it probably
is pretty idle. So then it steps down the CPUs and keeps them down. When you
work the CPUs, are you working dom0 hard?

 -- Keir

On 30/05/2009 07:59, "Carsten Schiers" <carsten@schiers.de> wrote:

> Sorry, it's lowest frequency (1.0 out of 1.0, 1.8, 2.0, and 2.1 GHz).
> When booting,
> the CPU will be at 2.1 GHz, when switching the governor from performance
> to ondemand,
> this one will set it to 1.0 GHz, where it's sort of fixed. I can set it
> manually with
> cpufreq-set when switching to userspace governor, though.
> 
> So my guess is that the ondemand governor donesn't get te right
> information about idle
> time though the according hypercall. I recompiled with CPUFREQ DEBUG
> set. But as said
> earlier, I don't have that much knowledge about how to debug kernels.
> 
> BR,
> Carsten.
> 
> -----Ursprüngliche Nachricht-----
> Von: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Gesendet: Freitag, 29. Mai 2009 23:31
> An: Carsten Schiers; xen-devel
> Betreff: Re: AW: Re: [Xen-devel] Xen 3.4 strange behaviour as compared
> to Xen 3.3.1
> 
> Is lowest p-state lowest or highest frequency/voltage?
> 
>  -- Keir
> 
> On 29/05/2009 17:47, "Carsten Schiers" <carsten@schiers.de> wrote:
> 
>> Thanks Keir, I tried it out but there is no difference. And by the
> way:
>> it
>> is in lowest p-state and doesn't come up, even if under heavy load.
> Hmm.
>> 
>> BR,
>> Carsten.
>> 
>> -----Ursprüngliche Nachricht-----
>> Von: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
>> Gesendet: Freitag, 29. Mai 2009 17:02
>> An: Carsten Schiers; xen-devel
>> Betreff: Re: [Xen-devel] Xen 3.4 strange behaviour as compared to Xen
>> 3.3.1
>> 
>> On 29/05/2009 15:03, "Carsten Schiers" <carsten@schiers.de> wrote:
>> 
>>>   - as already reported, ondemand manager in dom0-kernel doesn't
> step,
>>> manual
>>>     setting works so it seems a bit like the communication between
>> dom0
>>> and
>>>     hypervisor regarding idle time is not working
>> 
>> Could be an interaction with C-state support, preferring deep sleep to
>> running at lower voltage/frequency? You could try no-cpuidle on Xen's
>> command line at boot time and see if that changes things.
>> 
>>  -- Keir
>> 
>>>   - all beside one domu use Xen 3.4.0 kernel, the one who uses it's
>>> customized
>>>     kernel won't start up as first domu. It simply hangs and this
>>> prevents also
>>>     all other domus (I all auto start them, no save/restore) don't
>> come
>>> up. When
>>>     I start the chain with a different one and this (with the
>> different
>>> kernel)
>>>     is started as #2 or #3, not problem
>>> 
>>>   - one domu is for vdr with three dvb pci cards passed trough. This
>>> one, when
>>>     started as the first one, will cause xentop to show 20% load.
> When
>>> restarted
>>>     or started as #2, the load is like with 3.3.1 at roughly 3-5%.
>> 
>> 
>> 
>> 
> 
> 
> 
> 

Re: AW: Re: AW: Re: Xen 3.4 strange behaviour as compared to Xen 3.3.1

[Keir Fraser]

Have you tried rolling back to use the dom0 kernel that you used with 3.3.1?
It will be compatible. My guess would be that the large amounts of code
churn in the Linux tree to support cpufreq=xen better has perhaps broken
cpufreq=dom0-kernel.

 -- Keir

On 02/06/2009 17:18, "Carsten Schiers" <carsten@schiers.de> wrote:

> Sorry, I think it could also be that the CPU mask is somehow modified in the
> hypercall itself. Too much code for me to realy understand.
> 
> Just a guess, but does Changeset 18898 take care of cpufreq=dom0-kernel case?
> It's
> patching the hypercall 52 and the cpufreq_ondemand.c of Xen near a modfication
> of
> a CPU mask variable, but no change in the cpufreq_ondemand.c of the Dom0
> kernel. 
> It also seems to fit into the time window, doesn't it?
> 
> BR,
> Carsten.
> 
> ----- Originalnachricht -----
> Von: Carsten Schiers <carsten@schiers.de>
> Gesendet: Die, 2.6.2009 17:28
> An: Carsten Schiers <carsten@schiers.de> ; keir.fraser
> <keir.fraser@eu.citrix.com> ; xen-devel <xen-devel@lists.xensource.com> ;
> mark.langsdorf@amd.com
> Betreff: AW: Re: AW: Re: AW: Re: [Xen-devel] Xen 3.4 strange behaviour as
> compared to Xen 3.3.1
> 
> I managed to narrowed the problem down to the fact that inside
> dbs_calculate_load, which
> submits a hypercall to receive the load of all CPUs and then loops through all
> of them
> to calculate the combined load (which is a max of all (wall-idle)/wall scaled
> to 100).
> 
> It will not loop as policy->cpus, which is a bitmap of CPUs is zero. I am
> currently stuck 
> and not able to find out, where this is set. It seems that in the
> drivers/cpufreq dir, no
> change was made at all, so it must be somewhere else.
> 
> I thought, maybe somebody that changed something that has to do with this
> bitmap might
> come up with something like: uuuh, I missed that one...
> 
> I honestly have no idea what CPU hotpluging is, as I only have a simple
> mainstream board
> with a normal dual-core CPU, but the only change that is somehow related to
> CPUs that 
> I recently came accross was that.
> 
> Comments, ideas, help?
> 
> BR,
> Carsten.
> 
> 
> 
> ----- Originalnachricht -----
> Von: Carsten Schiers <carsten@schiers.de>
> Gesendet: Sam, 30.5.2009 15:29
> An: keir.fraser <keir.fraser@eu.citrix.com> ; xen-devel
> <xen-devel@lists.xensource.com>
> Betreff: AW: Re: AW: Re: AW: Re: [Xen-devel] Xen 3.4 strange behaviour as
> compared to Xen 3.3.1
> 
> No it's the kernel of Xen 3.4.0 with the same config. I will try later with
> the 
> older kernel, which is the one of 06.04.2009 (I have not realy understood the
> tagging or versioning of the Xen kernel). The other combination would be Xen
> 3.3.1
> with the latest kernel. That's how we narrow it down to hypervisor vs. kernel.
> 
> As I use cpufreq=dom0-kernel and enable cpufreq modules in
> drivers/cpufreq/Kconfig,
> I guess it should use the hypercall for idle time. But to double check, I
> generated
> load in dom0. No difference. It simply doesn't step upwards. The kernel with
> cpufreq
> debugging is enabled. I will also try it, maybe it's loging something usefull.
> 
> BR,
> Carsten.
> 
> -----Ursprüngliche Nachricht-----
> Von: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Gesendet: Samstag, 30. Mai 2009 11:30
> An: Carsten Schiers; xen-devel
> Betreff: Re: AW: Re: AW: Re: [Xen-devel] Xen 3.4 strange behaviour as compared
> to Xen 3.3.1
> 
> Are you using the exact same dom0 kernel as before, when it worked? One
> theory would be that dom0 is looking at its own idle stats, and it probably
> is pretty idle. So then it steps down the CPUs and keeps them down. When you
> work the CPUs, are you working dom0 hard?
> 
>  -- Keir
> 
> On 30/05/2009 07:59, "Carsten Schiers" <carsten@schiers.de> wrote:
> 
>> Sorry, it's lowest frequency (1.0 out of 1.0, 1.8, 2.0, and 2.1 GHz).
>> When booting,
>> the CPU will be at 2.1 GHz, when switching the governor from performance
>> to ondemand,
>> this one will set it to 1.0 GHz, where it's sort of fixed. I can set it
>> manually with
>> cpufreq-set when switching to userspace governor, though.
>> 
>> So my guess is that the ondemand governor donesn't get te right
>> information about idle
>> time though the according hypercall. I recompiled with CPUFREQ DEBUG
>> set. But as said
>> earlier, I don't have that much knowledge about how to debug kernels.
>> 
>> BR,
>> Carsten.
>> 
>> -----Ursprüngliche Nachricht-----
>> Von: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
>> Gesendet: Freitag, 29. Mai 2009 23:31
>> An: Carsten Schiers; xen-devel
>> Betreff: Re: AW: Re: [Xen-devel] Xen 3.4 strange behaviour as compared
>> to Xen 3.3.1
>> 
>> Is lowest p-state lowest or highest frequency/voltage?
>> 
>>  -- Keir
>> 
>> On 29/05/2009 17:47, "Carsten Schiers" <carsten@schiers.de> wrote:
>> 
>>> Thanks Keir, I tried it out but there is no difference. And by the
>> way:
>>> it
>>> is in lowest p-state and doesn't come up, even if under heavy load.
>> Hmm.
>>> 
>>> BR,
>>> Carsten.
>>> 
>>> -----Ursprüngliche Nachricht-----
>>> Von: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
>>> Gesendet: Freitag, 29. Mai 2009 17:02
>>> An: Carsten Schiers; xen-devel
>>> Betreff: Re: [Xen-devel] Xen 3.4 strange behaviour as compared to Xen
>>> 3.3.1
>>> 
>>> On 29/05/2009 15:03, "Carsten Schiers" <carsten@schiers.de> wrote:
>>> 
>>>>   - as already reported, ondemand manager in dom0-kernel doesn't
>> step,
>>>> manual
>>>>     setting works so it seems a bit like the communication between
>>> dom0
>>>> and
>>>>     hypervisor regarding idle time is not working
>>> 
>>> Could be an interaction with C-state support, preferring deep sleep to
>>> running at lower voltage/frequency? You could try no-cpuidle on Xen's
>>> command line at boot time and see if that changes things.
>>> 
>>>  -- Keir
>>> 
>>>>   - all beside one domu use Xen 3.4.0 kernel, the one who uses it's
>>>> customized
>>>>     kernel won't start up as first domu. It simply hangs and this
>>>> prevents also
>>>>     all other domus (I all auto start them, no save/restore) don't
>>> come
>>>> up. When
>>>>     I start the chain with a different one and this (with the
>>> different
>>>> kernel)
>>>>     is started as #2 or #3, not problem
>>>> 
>>>>   - one domu is for vdr with three dvb pci cards passed trough. This
>>>> one, when
>>>>     started as the first one, will cause xentop to show 20% load.
>> When
>>>> restarted
>>>>     or started as #2, the load is like with 3.3.1 at roughly 3-5%.
>>> 
>>> 
>>> 
>>> 
>> 
>> 
>> 
>> 
> 
> 
> 
> 

RE: AW: Re: AW: Re: Xen 3.4 strange behaviour as compared to Xen 3.3.1

[Tian, Kevin]

[-- Attachment #1: Type: text/plain, Size: 1390 bytes --]

>From: Carsten Schiers
>Sent: 2009年6月3日 0:18
>
>Sorry, I think it could also be that the CPU mask is somehow 
>modified in the
>hypercall itself. Too much code for me to realy understand. 
>
>Just a guess, but does Changeset 18898 take care of 
>cpufreq=dom0-kernel case? It's
>patching the hypercall 52 and the cpufreq_ondemand.c of Xen 
>near a modfication of
>a CPU mask variable, but no change in the cpufreq_ondemand.c 
>of the Dom0 kernel. 
>It also seems to fit into the time window, doesn't it?
>

That's really a silly typo. Could you try whether below fixes for you?

diff -r e8b74e981bfb xen/arch/x86/platform_hypercall.c
--- a/xen/arch/x86/platform_hypercall.c Tue Jun 02 18:58:09 2009 +0800
+++ b/xen/arch/x86/platform_hypercall.c Tue Jun 02 18:59:18 2009 +0800
@@ -313,7 +313,6 @@ ret_t do_platform_op(XEN_GUEST_HANDLE(xe
     {
         uint32_t cpu;
         uint64_t idletime, now = NOW();
-        struct vcpu *v;
         struct xenctl_cpumap ctlmap;
         cpumask_t cpumap;
         XEN_GUEST_HANDLE(uint8) cpumap_bitmap;
@@ -336,7 +335,7 @@ ret_t do_platform_op(XEN_GUEST_HANDLE(xe

         for_each_cpu_mask ( cpu, cpumap )
         {
-            if ( (v = idle_vcpu[cpu]) != NULL )
+            if ( !idle_vcpu[cpu] )
                 cpu_clear(cpu, cpumap);
             idletime = get_cpu_idle_time(cpu);

Thanks
Kevin

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: AW: Re: AW: Re: Xen 3.4 strange behaviour as compared to Xen 3.3.1

[Keir Fraser]

Are you using the exact same dom0 kernel as before, when it worked? One
theory would be that dom0 is looking at its own idle stats, and it probably
is pretty idle. So then it steps down the CPUs and keeps them down. When you
work the CPUs, are you working dom0 hard?

 -- Keir

On 30/05/2009 07:59, "Carsten Schiers" <carsten@schiers.de> wrote:

> Sorry, it's lowest frequency (1.0 out of 1.0, 1.8, 2.0, and 2.1 GHz).
> When booting,
> the CPU will be at 2.1 GHz, when switching the governor from performance
> to ondemand,
> this one will set it to 1.0 GHz, where it's sort of fixed. I can set it
> manually with
> cpufreq-set when switching to userspace governor, though.
> 
> So my guess is that the ondemand governor donesn't get te right
> information about idle
> time though the according hypercall. I recompiled with CPUFREQ DEBUG
> set. But as said
> earlier, I don't have that much knowledge about how to debug kernels.
> 
> BR,
> Carsten.
> 
> -----Ursprüngliche Nachricht-----
> Von: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Gesendet: Freitag, 29. Mai 2009 23:31
> An: Carsten Schiers; xen-devel
> Betreff: Re: AW: Re: [Xen-devel] Xen 3.4 strange behaviour as compared
> to Xen 3.3.1
> 
> Is lowest p-state lowest or highest frequency/voltage?
> 
>  -- Keir
> 
> On 29/05/2009 17:47, "Carsten Schiers" <carsten@schiers.de> wrote:
> 
>> Thanks Keir, I tried it out but there is no difference. And by the
> way:
>> it
>> is in lowest p-state and doesn't come up, even if under heavy load.
> Hmm.
>> 
>> BR,
>> Carsten.
>> 
>> -----Ursprüngliche Nachricht-----
>> Von: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
>> Gesendet: Freitag, 29. Mai 2009 17:02
>> An: Carsten Schiers; xen-devel
>> Betreff: Re: [Xen-devel] Xen 3.4 strange behaviour as compared to Xen
>> 3.3.1
>> 
>> On 29/05/2009 15:03, "Carsten Schiers" <carsten@schiers.de> wrote:
>> 
>>>   - as already reported, ondemand manager in dom0-kernel doesn't
> step,
>>> manual
>>>     setting works so it seems a bit like the communication between
>> dom0
>>> and
>>>     hypervisor regarding idle time is not working
>> 
>> Could be an interaction with C-state support, preferring deep sleep to
>> running at lower voltage/frequency? You could try no-cpuidle on Xen's
>> command line at boot time and see if that changes things.
>> 
>>  -- Keir
>> 
>>>   - all beside one domu use Xen 3.4.0 kernel, the one who uses it's
>>> customized
>>>     kernel won't start up as first domu. It simply hangs and this
>>> prevents also
>>>     all other domus (I all auto start them, no save/restore) don't
>> come
>>> up. When
>>>     I start the chain with a different one and this (with the
>> different
>>> kernel)
>>>     is started as #2 or #3, not problem
>>> 
>>>   - one domu is for vdr with three dvb pci cards passed trough. This
>>> one, when
>>>     started as the first one, will cause xentop to show 20% load.
> When
>>> restarted
>>>     or started as #2, the load is like with 3.3.1 at roughly 3-5%.
>> 
>> 
>> 
>> 
> 
> 
> 
> 

AW: Re: AW: Re: Xen 3.4 strange behaviour as compared to Xen 3.3.1

[Carsten Schiers]

Sorry, it's lowest frequency (1.0 out of 1.0, 1.8, 2.0, and 2.1 GHz). 
When booting,
the CPU will be at 2.1 GHz, when switching the governor from performance 
to ondemand,
this one will set it to 1.0 GHz, where it's sort of fixed. I can set it 
manually with
cpufreq-set when switching to userspace governor, though.

So my guess is that the ondemand governor donesn't get te right 
information about idle
time though the according hypercall. I recompiled with CPUFREQ DEBUG 
set. But as said
earlier, I don't have that much knowledge about how to debug kernels.

BR,
Carsten.

-----Ursprüngliche Nachricht-----
Von: Keir Fraser [mailto:keir.fraser@eu.citrix.com] 
Gesendet: Freitag, 29. Mai 2009 23:31
An: Carsten Schiers; xen-devel
Betreff: Re: AW: Re: [Xen-devel] Xen 3.4 strange behaviour as compared 
to Xen 3.3.1

Is lowest p-state lowest or highest frequency/voltage?

 -- Keir

On 29/05/2009 17:47, "Carsten Schiers" <carsten@schiers.de> wrote:

> Thanks Keir, I tried it out but there is no difference. And by the 
way:
> it
> is in lowest p-state and doesn't come up, even if under heavy load. 
Hmm.
> 
> BR,
> Carsten.
> 
> -----Ursprüngliche Nachricht-----
> Von: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Gesendet: Freitag, 29. Mai 2009 17:02
> An: Carsten Schiers; xen-devel
> Betreff: Re: [Xen-devel] Xen 3.4 strange behaviour as compared to Xen
> 3.3.1
> 
> On 29/05/2009 15:03, "Carsten Schiers" <carsten@schiers.de> wrote:
> 
>>   - as already reported, ondemand manager in dom0-kernel doesn't 
step,
>> manual
>>     setting works so it seems a bit like the communication between
> dom0
>> and
>>     hypervisor regarding idle time is not working
> 
> Could be an interaction with C-state support, preferring deep sleep to
> running at lower voltage/frequency? You could try no-cpuidle on Xen's
> command line at boot time and see if that changes things.
> 
>  -- Keir
> 
>>   - all beside one domu use Xen 3.4.0 kernel, the one who uses it's
>> customized
>>     kernel won't start up as first domu. It simply hangs and this
>> prevents also
>>     all other domus (I all auto start them, no save/restore) don't
> come
>> up. When
>>     I start the chain with a different one and this (with the
> different
>> kernel)
>>     is started as #2 or #3, not problem
>> 
>>   - one domu is for vdr with three dvb pci cards passed trough. This
>> one, when
>>     started as the first one, will cause xentop to show 20% load. 
When
>> restarted
>>     or started as #2, the load is like with 3.3.1 at roughly 3-5%.
> 
> 
> 
>