[PATCH] cpu-freq: add troubleshooting section for FSB changes

[PATCH] cpu-freq: add troubleshooting section for FSB changes

[Luis R. Rodriguez]

With cpu frequency scaling you may run into some system
performance issues when the FSB changes speed due to the
throughput constraints this brings on the system.

Document how to rule out whether or not FSB changes
affect general system issues.

Cc: Matthew Garrett <mjg@redhat.com>
Cc: Arjan van de Ven" <arjan@linux.intel.com>
Cc: Reinette Chatre <reinette.chatre@intel.com>
Cc: Aeolus Yang <Aeolus.Yang@Atheros.com>
Cc: Amod Bodas <Amod.Bodas@Atheros.com>
Cc: David Quan <David.Quan@Atheros.com>
Cc: Kishore Jotwani <Kishore.Jotwani@Atheros.com>
Signed-off-by: Luis R. Rodriguez <lrodriguez@atheros.com>
---
 Documentation/cpu-freq/user-guide.txt |   45 +++++++++++++++++++++++++++++++++
 1 files changed, 45 insertions(+), 0 deletions(-)

diff --git a/Documentation/cpu-freq/user-guide.txt b/Documentation/cpu-freq/user-guide.txt
index 2a5b850..846729d 100644
--- a/Documentation/cpu-freq/user-guide.txt
+++ b/Documentation/cpu-freq/user-guide.txt
@@ -32,6 +32,7 @@ Contents:
 3. How to change the CPU cpufreq policy and/or speed
 3.1 Preferred interface: sysfs
 
+4. Troubleshooting
 
 
 1. Supported Architectures and Processors
@@ -211,3 +212,47 @@ scaling_setspeed.		By "echoing" a new frequency into this
 				you can change the speed of the CPU,
 				but only within the limits of
 				scaling_min_freq and scaling_max_freq.
+
+4. Troubleshooting
+==================
+
+If you are not sure if frequency scaling is the culprit to your
+issues you can try to disable the feature on the BIOS if possible.
+
+Alternatively it helps to understand that on some CPUs (Intel)
+the Front Side Bus (FSB) speed is changed when you happen to change
+the active CPU frequency to the lowest supported frequency, which
+happens to be the frequency used on the highest supported P-n state.
+The P0 state is the highest Performance state with the highest
+CPU frequency and system power. The highest P-n state would be
+the one with the lowest supported frequency and power.
+
+On Intel CPUs the FSB will not be modified with SpeedStep enabled if the
+second to last lowest frequency on the available scaling frequencies
+is used. Testing this would rule out whether or not the FSB changes
+made by SpeedStep would be the main issue.
+
+Modifying the FSB will alter your general system throughput
+on your Northbridge so to prevent general throughput issues
+with devices on it you can modify frequency scaling so that
+it nevers uses the lowest supported frequency.
+
+For example:
+
+cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_available_frequencies
+1801000 1800000 1200000 800000
+1801000 1800000 1200000 800000
+
+cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq
+800000
+800000
+
+To disable changing the FSB select the second to last CPU freq
+on the available freq for each CPU and set the scaling_min_freq
+for each CPU to that value. For the example output this would be
+echo'ing 1200000 to each scaling_min_freq:
+
+echo 1200000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
+echo 1200000 > /sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq
+
+The first line is for the 1st CPU, the second line for the second line.
-- 
1.6.5.2.143.g8cc62

Re: [PATCH] cpu-freq: add troubleshooting section for FSB changes

[Thomas Renninger]

On Friday 06 November 2009 09:01:00 pm Luis R. Rodriguez wrote:
> With cpu frequency scaling you may run into some system
> performance issues when the FSB changes speed due to the
> throughput constraints this brings on the system.
Interesting.
Could you be a bit more detailed how and what kind of perfomance issues are
seen, please.
I expect IO based issues like WLAN package throughput or even lost connections :)
"wireless" is also a good word to grep for...
If this gets documented also a bit more info about the HW should get
added. The problem sounds rather restricted to specific models (atom based, laptop,
desktop chipsets?)?
Otherwise everyone adding a 10G network card into his Intel machine will
start measuring or excluding the lowest freq when reading this.

Thanks for the valuable info,

      Thomas

> Document how to rule out whether or not FSB changes
> affect general system issues.
>
> Cc: Matthew Garrett <mjg@redhat.com>
> Cc: Arjan van de Ven" <arjan@linux.intel.com>
> Cc: Reinette Chatre <reinette.chatre@intel.com>
> Cc: Aeolus Yang <Aeolus.Yang@Atheros.com>
> Cc: Amod Bodas <Amod.Bodas@Atheros.com>
> Cc: David Quan <David.Quan@Atheros.com>
> Cc: Kishore Jotwani <Kishore.Jotwani@Atheros.com>
> Signed-off-by: Luis R. Rodriguez <lrodriguez@atheros.com>
> ---
>  Documentation/cpu-freq/user-guide.txt |   45
> +++++++++++++++++++++++++++++++++ 1 files changed, 45 insertions(+), 0
> deletions(-)
>
> diff --git a/Documentation/cpu-freq/user-guide.txt
> b/Documentation/cpu-freq/user-guide.txt index 2a5b850..846729d 100644
> --- a/Documentation/cpu-freq/user-guide.txt
> +++ b/Documentation/cpu-freq/user-guide.txt
> @@ -32,6 +32,7 @@ Contents:
>  3. How to change the CPU cpufreq policy and/or speed
>  3.1 Preferred interface: sysfs
>
> +4. Troubleshooting
>
>
>  1. Supported Architectures and Processors
> @@ -211,3 +212,47 @@ scaling_setspeed.		By "echoing" a new frequency into
> this you can change the speed of the CPU,
>  				but only within the limits of
>  				scaling_min_freq and scaling_max_freq.
> +
> +4. Troubleshooting
> +==================
> +
> +If you are not sure if frequency scaling is the culprit to your
> +issues you can try to disable the feature on the BIOS if possible.
> +
> +Alternatively it helps to understand that on some CPUs (Intel)
> +the Front Side Bus (FSB) speed is changed when you happen to change
> +the active CPU frequency to the lowest supported frequency, which
> +happens to be the frequency used on the highest supported P-n state.
> +The P0 state is the highest Performance state with the highest
> +CPU frequency and system power. The highest P-n state would be
> +the one with the lowest supported frequency and power.
> +
> +On Intel CPUs the FSB will not be modified with SpeedStep enabled if the
> +second to last lowest frequency on the available scaling frequencies
> +is used. Testing this would rule out whether or not the FSB changes
> +made by SpeedStep would be the main issue.
> +
> +Modifying the FSB will alter your general system throughput
> +on your Northbridge so to prevent general throughput issues
> +with devices on it you can modify frequency scaling so that
> +it nevers uses the lowest supported frequency.
> +
> +For example:
> +
> +cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_available_frequencies
> +1801000 1800000 1200000 800000
> +1801000 1800000 1200000 800000
> +
> +cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq
> +800000
> +800000
> +
> +To disable changing the FSB select the second to last CPU freq
> +on the available freq for each CPU and set the scaling_min_freq
> +for each CPU to that value. For the example output this would be
> +echo'ing 1200000 to each scaling_min_freq:
> +
> +echo 1200000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
> +echo 1200000 > /sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq
> +
> +The first line is for the 1st CPU, the second line for the second line.

Re: [PATCH] cpu-freq: add troubleshooting section for FSB changes

[Thomas Renninger]

On Saturday 07 November 2009 09:52:56 pm Thomas Renninger wrote:
> On Friday 06 November 2009 09:01:00 pm Luis R. Rodriguez wrote:
> > With cpu frequency scaling you may run into some system
> > performance issues when the FSB changes speed due to the
> > throughput constraints this brings on the system.
>
> Interesting.
> Could you be a bit more detailed how and what kind of perfomance issues are
> seen, please.
> I expect IO based issues like WLAN package throughput or even lost
> connections :) "wireless" is also a good word to grep for...
> If this gets documented also a bit more info about the HW should get
> added. The problem sounds rather restricted to specific models (atom based,
> laptop, desktop chipsets?)?
Or whatabout adding a list of affected machine/chipset models. The ordinary
user will hardly find his IO throughput problem in:
Documentation/cpu-freq/user-guide.txt
Then a lot of people will be happy when starting to google for their HW.

     Thomas

Re: [PATCH] cpu-freq: add troubleshooting section for FSB changes

[Arjan van de Ven]

Thomas Renninger wrote:

[please cut away the part of the mail you're not responding to; your mail was 90% like that]

> On Friday 06 November 2009 09:01:00 pm Luis R. Rodriguez wrote:
>> With cpu frequency scaling you may run into some system
>> performance issues when the FSB changes speed due to the
>> throughput constraints this brings on the system.
> Interesting.
> Could you be a bit more detailed how and what kind of perfomance issues are
> seen, please.
> I expect IO based issues like WLAN package throughput or even lost connections :)

the bus doesn't get THAT slow.....
in addition, most FSB systems have the memory controller in the chipset, next to the PCI
logic... so that the FSB bus for DMA transactions only carries the snoop traffic, not the whole data.

Re: [PATCH] cpu-freq: add troubleshooting section for FSB changes

[Thomas Renninger]

On Saturday 07 November 2009 11:28:12 pm Arjan van de Ven wrote:
> Thomas Renninger wrote:
>
> > On Friday 06 November 2009 09:01:00 pm Luis R. Rodriguez wrote:
> >> With cpu frequency scaling you may run into some system
> >> performance issues when the FSB changes speed due to the
> >> throughput constraints this brings on the system.
> >
> > Interesting.
> > Could you be a bit more detailed how and what kind of perfomance issues
> > are seen, please.
> > I expect IO based issues like WLAN package throughput or even lost
> > connections :)
>
> the bus doesn't get THAT slow.....
I expected wireless problems because 5 people with @atheros.com are in CC.
There is zero information about the problem itself and possibly affected 
devices/machines.
> in addition, most FSB systems have the memory controller in the chipset,
> next to the PCI logic... so that the FSB bus for DMA transactions only
> carries the snoop traffic, not the whole data.
So when should people look at this?

    Thomas

Re: [PATCH] cpu-freq: add troubleshooting section for FSB changes

[Arjan van de Ven]

>> in addition, most FSB systems have the memory controller in the chipset,
>> next to the PCI logic... so that the FSB bus for DMA transactions only
>> carries the snoop traffic, not the whole data.
> So when should people look at this?

at this point the atheros folks haven't even confirmed that this is the cause...

I'm not saying that the linux behavior is optimal with P states (I have a rather
sizeable algorithm rewrite in the queue) but to blame anything and everything on
a half-speed FSB during a very idle system?
I'm still somewhat skeptical. Again.. the CPU is basically idle here (otherwise
ondemand would ramp the freq up quickly); at which point the FSB traffic mostly is
just cache coherency traffic.... much less bandwidth intensive.

Re: [PATCH] cpu-freq: add troubleshooting section for FSB changes

[Matthew Garrett]

The described problem was one of significant throughput reduction when 
speedstep was enabled in the BIOS on a mobile system - the idea here was 
just to document how to identify whether the issue is down to cpu 
frequency scaling or some platform side effect.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [PATCH] cpu-freq: add troubleshooting section for FSB changes

[Luis R. Rodriguez]

On Sat, Nov 7, 2009 at 2:59 PM, Arjan van de Ven <arjan@linux.intel.com> wrote:
>>> in addition, most FSB systems have the memory controller in the chipset,
>>> next to the PCI logic... so that the FSB bus for DMA transactions only
>>> carries the snoop traffic, not the whole data.
>>
>> So when should people look at this?
>
> at this point the atheros folks haven't even confirmed that this is the
> cause...

And we tested this (reducing the min cpu freq to one less than the
highest supported P state to avoid an FSB speed change) and it seems
doing the steps described here did not fix the issue. But at least now
if anyone else wants to verify this they can with some sort of
documentaiton.

So to confirm though -- we are seeing a huge performance depredation
mainly on RX on an Intel Pine Trail platform with SpeedStep enabled on
the BIOS.

Let me get into the specifics in case anyone is able to help. The
issue is with ath9k on RX and the CPU on C3 state requesting DMA over
PCI-E. We typically would get about 110 Mbps with an AR9285 (single
stream) but when SpeedStep is enabled it goes down to 25 Mbps. At the
PCI-E level we are seeing huge latencies introduced when SpeedStep is
used for DMA requests to the Intel root complex on the Intel Pine
trail platform. Latencies are about 20-60 us.

Is there a timeout threshold change that will cause the Intel chipset
wait for some time after completion before going into a C3 state? Are
there any other explanations for seeing such huge latencies on C3
state?

Thanks for your review and help with this. Any suggestions are greatly
appreciated.

  Luis

Re: [PATCH] cpu-freq: add troubleshooting section for FSB changes

[Corrado Zoccolo]

Hi,
On Mon, Nov 9, 2009 at 5:32 PM, Luis R. Rodriguez <mcgrof@gmail.com> wrote:
>
> And we tested this (reducing the min cpu freq to one less than the
> highest supported P state to avoid an FSB speed change) and it seems
> doing the steps described here did not fix the issue. But at least now
> if anyone else wants to verify this they can with some sort of
> documentaiton.
>
> So to confirm though -- we are seeing a huge performance depredation
> mainly on RX on an Intel Pine Trail platform with SpeedStep enabled on
> the BIOS.
>
> Let me get into the specifics in case anyone is able to help. The
> issue is with ath9k on RX and the CPU on C3 state requesting DMA over
> PCI-E. We typically would get about 110 Mbps with an AR9285 (single
> stream) but when SpeedStep is enabled it goes down to 25 Mbps. At the
> PCI-E level we are seeing huge latencies introduced when SpeedStep is
> used for DMA requests to the Intel root complex on the Intel Pine
> trail platform. Latencies are about 20-60 us.

You mentioned speedstep and cpufreq, but the problem is with C3 state
and cpuidle (probably the BIOS mixes the two concepts, but we should
keep them separated).
C3 is not related to the core or FSB frequency, it is an idle state.
When in C3, the CPU is not ready to perform any operation, not just
slower, and depending on the CPU hw, it may take several us to wake up
(even 85us, on an Atom).

>
> Is there a timeout threshold change that will cause the Intel chipset
> wait for some time after completion before going into a C3 state? Are
> there any other explanations for seeing such huge latencies on C3
> state?
There is a patch from Arjan for the cpuidle menu governor, that may fix it.
http://linux.derkeiler.com/Mailing-Lists/Kernel/2009-09/msg05276.html
It is already present in 2.6.32.

Thanks,
Corrado

>
> Thanks for your review and help with this. Any suggestions are greatly
> appreciated.
>
>  Luis
> --
> To unsubscribe from this list: send the line "unsubscribe cpufreq" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
__________________________________________________________________________

dott. Corrado Zoccolo                          mailto:czoccolo@gmail.com
PhD - Department of Computer Science - University of Pisa, Italy
--------------------------------------------------------------------------
The self-confidence of a warrior is not the self-confidence of the average
man. The average man seeks certainty in the eyes of the onlooker and calls
that self-confidence. The warrior seeks impeccability in his own eyes and
calls that humbleness.
                               Tales of Power - C. Castaneda

Re: [PATCH] cpu-freq: add troubleshooting section for FSB changes

[Luis R. Rodriguez]

On Tue, Nov 10, 2009 at 4:12 AM, Corrado Zoccolo <czoccolo@gmail.com> wrote:
> Hi,
> On Mon, Nov 9, 2009 at 5:32 PM, Luis R. Rodriguez <mcgrof@gmail.com> wrote:
>>
>> And we tested this (reducing the min cpu freq to one less than the
>> highest supported P state to avoid an FSB speed change) and it seems
>> doing the steps described here did not fix the issue. But at least now
>> if anyone else wants to verify this they can with some sort of
>> documentaiton.
>>
>> So to confirm though -- we are seeing a huge performance depredation
>> mainly on RX on an Intel Pine Trail platform with SpeedStep enabled on
>> the BIOS.
>>
>> Let me get into the specifics in case anyone is able to help. The
>> issue is with ath9k on RX and the CPU on C3 state requesting DMA over
>> PCI-E. We typically would get about 110 Mbps with an AR9285 (single
>> stream) but when SpeedStep is enabled it goes down to 25 Mbps. At the
>> PCI-E level we are seeing huge latencies introduced when SpeedStep is
>> used for DMA requests to the Intel root complex on the Intel Pine
>> trail platform. Latencies are about 20-60 us.
>
> You mentioned speedstep and cpufreq, but the problem is with C3 state
> and cpuidle (probably the BIOS mixes the two concepts, but we should
> keep them separated).
> C3 is not related to the core or FSB frequency, it is an idle state.
> When in C3, the CPU is not ready to perform any operation, not just
> slower, and depending on the CPU hw, it may take several us to wake up
> (even 85us, on an Atom).
>
>>
>> Is there a timeout threshold change that will cause the Intel chipset
>> wait for some time after completion before going into a C3 state? Are
>> there any other explanations for seeing such huge latencies on C3
>> state?
> There is a patch from Arjan for the cpuidle menu governor, that may fix it.
> http://linux.derkeiler.com/Mailing-Lists/Kernel/2009-09/msg05276.html
> It is already present in 2.6.32.

Thanks all for your feedback -- this was determined to be a BIOS bug
:) Anyway, hope you do consider the patch for inclusion.

  Luis