[Xenomai-help] linux 2.6.11

[Xenomai-help] linux 2.6.11

[xenophile]

Hi,

I am trying to see why a trivial driver reading an ADC on an EP9312 ARM 
board has slowed by a factor of three. Since I installed it under 2.6.29 
a couple fo years ago.

It's looking like this is due to a kernel update I did in the mean time 
to 2.6.32


I have just built a new toolchain and rebuilt the kernel to 2.6.33 and 
the result is about the same.

I recall a number of years ago running a 2.6.11 kernel and the system 
was incredibily responsive. I could be building kde-libs, downloading at 
full speed browsing and listening to mp3 decoded music all without the 
slightest lack of responsiveness in the browser not a glitch in the mp3 
playback.

It was truly impressive.

Now I find if I am rebuilding the toolchain , the browser or any other 
window can take over a second to respond to a redraw ( 2.6.32 on this 
machine).


Now I've seen comments on audio forums that there was nothing better 
than linux 2.6.11 for real time audio work. I'm wondering  whether there 
is not a general degradation in the linux kernel as time goes on.  It 
certainly seems to be getting forever bigger and slower.

Would there be any value in trying to rebuild a 2.6.11 based system for 
realtime work ?

TIA,

Re: [Xenomai-help] linux 2.6.11

[Gilles Chanteperdrix]

On 04/21/2012 10:12 PM, xenophile wrote:
> Hi,
> 
> I am trying to see why a trivial driver reading an ADC on an EP9312 ARM 
> board has slowed by a factor of three. Since I installed it under 2.6.29 
> a couple fo years ago.
> 
> It's looking like this is due to a kernel update I did in the mean time 
> to 2.6.32

If the kernel is patched with Xenomai, did you check that you have the
same setting for FCSE ? Other than that, you can have an idea of how the
time is spent in both cases using the I-pipe tracer.

> Would there be any value in trying to rebuild a 2.6.11 based system for 
> realtime work ?

xenomai comes with a latency test that you can run on any two
configuration to compare their latencies. If you go on the adeos project
download area:
http://download.gna.org/adeos/patches/

You will find kernel patches for versions such as 2.6.13 or 2.6.14, they
are supposed to work with xenomai latest stable release. You can even
find patches for 2.4 kernels for i386 and ppc.

Chances are however that these patches have bugs which have been fixed
since then (after all, 2.6.11 is 7 years old).

-- 
                                                                Gilles.

Re: [Xenomai-help] linux 2.6.11

[xenophile]

On 21/04/12 23:03, Gilles Chanteperdrix wrote:
> On 04/21/2012 10:12 PM, xenophile wrote:
>> Hi,
>>
>> I am trying to see why a trivial driver reading an ADC on an EP9312 ARM
>> board has slowed by a factor of three. Since I installed it under 2.6.29
>> a couple fo years ago.
>>
>> It's looking like this is due to a kernel update I did in the mean time
>> to 2.6.32
>
> If the kernel is patched with Xenomai, did you check that you have the
> same setting for FCSE ? Other than that, you can have an idea of how the
> time is spent in both cases using the I-pipe tracer.
>

I'm interested in exploring RT on this hardware and part of the change 
of kernel was to move to more recent kernel which has better support for 
this board and to find a kernel that has been used RT with this type of 
hardware so that I don't have to start from scratch.

Yesterday I tested 2.6.33 and it was equally slow. I need to find out if 
this is due to kernel version or gcc. (4.3.4)

I need to resolve the cause of this factor of 2 or 3 slow down before I 
get involved in xenomia patching so as to start from a sound base.

>> Would there be any value in trying to rebuild a 2.6.11 based system for
>> realtime work ?
>
> xenomai comes with a latency test that you can run on any two
> configuration to compare their latencies. If you go on the adeos project
> download area:
> http://download.gna.org/adeos/patches/
>
> You will find kernel patches for versions such as 2.6.13 or 2.6.14, they
> are supposed to work with xenomai latest stable release. You can even
> find patches for 2.4 kernels for i386 and ppc.
>
> Chances are however that these patches have bugs which have been fixed
> since then (after all, 2.6.11 is 7 years old).
>

Thanks, I was hoping to find some comments on which kernels are more 
responsive. I'm sure there's good and bad since RT is not one of the 
main priorities for kernel development, this may not be a case of the 
most recent being the best.

 From a simple user experience circa 2.6.11 was hugely more responsive 
than the more recent offerings. Maybe 2.6.13 or 2.6.14 are close enough 
to be in the same league.

I would have thought this sort of thing would have been thoroughly 
investigated by RT people and would be documented.

regards.

Re: [Xenomai-help] linux 2.6.11

[Wolfgang Grandegger]

On 04/21/2012 10:12 PM, xenophile wrote:
> Hi,
> 
> I am trying to see why a trivial driver reading an ADC on an EP9312 ARM
> board has slowed by a factor of three. Since I installed it under 2.6.29
> a couple fo years ago.
> 
> It's looking like this is due to a kernel update I did in the mean time
> to 2.6.32
> 
> 
> I have just built a new toolchain and rebuilt the kernel to 2.6.33 and
> the result is about the same.
> 
> I recall a number of years ago running a 2.6.11 kernel and the system
> was incredibily responsive. I could be building kde-libs, downloading at
> full speed browsing and listening to mp3 decoded music all without the
> slightest lack of responsiveness in the browser not a glitch in the mp3
> playback.
> 
> It was truly impressive.
> 
> Now I find if I am rebuilding the toolchain , the browser or any other
> window can take over a second to respond to a redraw ( 2.6.32 on this
> machine).

What premption mode does the kernel use? CONFIG_PREEMPT should be enabled.

> Now I've seen comments on audio forums that there was nothing better
> than linux 2.6.11 for real time audio work. I'm wondering  whether there
> is not a general degradation in the linux kernel as time goes on.  It
> certainly seems to be getting forever bigger and slower.

Yes, there is definitively a degradation which will hurt especially
low-end systems. Just compare the size of the kernel. It's getting
bigger and bigger. Also the timer now runs at 250 Hz and there are many
more features, interfaces, etc.. You may regain some degradation by
optimizing the kernel configuration. On low-end systems, if it's an
option, choose a 2.4 kernel.

> Would there be any value in trying to rebuild a 2.6.11 based system for
> realtime work ?

Well, then I would prefer 2.4.x but for both Xenomai support (for your
board) is not available, I guess. But normally it's not an option as
people do want more recent kernel features and drivers.

But all this does not explain a factor of 3 with your driver. I doubt
that it's due to a move from 2.6.29 -> 2.6.32. Try using ftrace to find
out what's going on (if it's already available).

Wolfgang.

Re: [Xenomai-help] linux 2.6.11

[Gilles Chanteperdrix]

On 04/22/2012 08:23 AM, xenophile wrote:
> I'm interested in exploring RT on this hardware and part of the change 
> of kernel was to move to more recent kernel which has better support for 
> this board and to find a kernel that has been used RT with this type of 
> hardware so that I don't have to start from scratch.
> 
> Yesterday I tested 2.6.33 and it was equally slow. I need to find out if 
> this is due to kernel version or gcc. (4.3.4)
> 
> I need to resolve the cause of this factor of 2 or 3 slow down before I 
> get involved in xenomia patching so as to start from a sound base.

Well, given the way xenomai works, we do not depend much on the linux
performances. So, if you want to compare something, it should be the
xenomai latency differences between the two kernel versions. Using linux
performance to get an idea of xenomai performance does not really make
any sense.

To give you just an example, at some point around 2.6.30, the linux
kernel introduced threaded interrupts, which greatly increase interrupt
latencies. Xenomai does not use threaded interrupts, so it is
essentially unaffected by the change.

Of course there are some indirect effect such as the size of the kernel,
which has an impact on I-cache and so on the latency.

> Thanks, I was hoping to find some comments on which kernels are more 
> responsive. I'm sure there's good and bad since RT is not one of the 
> main priorities for kernel development, this may not be a case of the 
> most recent being the best.
> 
>  From a simple user experience circa 2.6.11 was hugely more responsive 
> than the more recent offerings. Maybe 2.6.13 or 2.6.14 are close enough 
> to be in the same league.
> 
> I would have thought this sort of thing would have been thoroughly 
> investigated by RT people and would be documented.

We run xenomai, so, what we investigate are xenomai performances. On a
system running xenomai, the performance-sensitive job is done by
xenomai, not by linux anyway. And we check between two releases that we
do not get differences in performances. So, I am pretty sure that on
at91rm9200 for instance (the platform I have which should be the closest
to an EP9312), the xenomai user-space latency from 2.6.29 to 3.2.1 is
around 220us.

Besides, in terms of performance, the relation performance/version
probably also depends on the platform which is running the kernel. So
for instance, you may have found that 2.6.11 was good on x86, but maybe
it sucked on ARM.

Apart from that, as I already said, we provide you with:
- the latest version of xenomai which should supports all past kernel
versions
- the tools to do the performance measurement.

So, you have everything you need to do the comparison.

-- 
                                                                Gilles.

Re: [Xenomai-help] linux 2.6.11

[Gilles Chanteperdrix]

On 04/22/2012 11:10 AM, Wolfgang Grandegger wrote:
> But all this does not explain a factor of 3 with your driver. I doubt
> that it's due to a move from 2.6.29 -> 2.6.32. Try using ftrace to find
> out what's going on (if it's already available).

Threaded interrupts were introduced around this time. And threaded
interrupts DO have a significant impact on latencies.

-- 
                                                                Gilles.

Re: [Xenomai-help] linux 2.6.11

[Gilles Chanteperdrix]

On 04/22/2012 01:24 PM, Gilles Chanteperdrix wrote:
> On 04/22/2012 11:10 AM, Wolfgang Grandegger wrote:
>> But all this does not explain a factor of 3 with your driver. I doubt
>> that it's due to a move from 2.6.29 -> 2.6.32. Try using ftrace to find
>> out what's going on (if it's already available).
> 
> Threaded interrupts were introduced around this time. And threaded
> interrupts DO have a significant impact on latencies.
> 
But I agree: tracing with ftrace or the I-pipe tracer if using xenomai
to see where the time is spent is the way to go.

-- 
                                                                Gilles.

Re: [Xenomai-help] linux 2.6.11

[Wolfgang Grandegger]

On 04/22/2012 01:34 PM, Gilles Chanteperdrix wrote:
> On 04/22/2012 01:24 PM, Gilles Chanteperdrix wrote:
>> On 04/22/2012 11:10 AM, Wolfgang Grandegger wrote:
>>> But all this does not explain a factor of 3 with your driver. I doubt
>>> that it's due to a move from 2.6.29 -> 2.6.32. Try using ftrace to find
>>> out what's going on (if it's already available).
>>
>> Threaded interrupts were introduced around this time. And threaded
>> interrupts DO have a significant impact on latencies.

Yes, but thread interrupts are normally not enabled/used by default. I'm
not speaking about CONFIG_PREEMPT_RT. Well, do we speak about latencies?
He just said "reading an ADC on an EP9312 ARM board has slowed by a
factor of three". I'm not even sure if it's related to Xenomai at all.
xenophile (&$#^?) , could you please be more precise.

> But I agree: tracing with ftrace or the I-pipe tracer if using xenomai
> to see where the time is spent is the way to go.

Yep.

Wolfgang.

Re: [Xenomai-help] linux 2.6.11

[Gilles Chanteperdrix]

On 04/22/2012 03:53 PM, Wolfgang Grandegger wrote:
> On 04/22/2012 01:34 PM, Gilles Chanteperdrix wrote:
>> On 04/22/2012 01:24 PM, Gilles Chanteperdrix wrote:
>>> On 04/22/2012 11:10 AM, Wolfgang Grandegger wrote:
>>>> But all this does not explain a factor of 3 with your driver. I doubt
>>>> that it's due to a move from 2.6.29 -> 2.6.32. Try using ftrace to find
>>>> out what's going on (if it's already available).
>>>
>>> Threaded interrupts were introduced around this time. And threaded
>>> interrupts DO have a significant impact on latencies.
> 
> Yes, but thread interrupts are normally not enabled/used by default. I'm
> not speaking about CONFIG_PREEMPT_RT. Well, do we speak about latencies?
> He just said "reading an ADC on an EP9312 ARM board has slowed by a
> factor of three". I'm not even sure if it's related to Xenomai at all.
> xenophile (&$#^?) , could you please be more precise.

Yes, without answers to these questions, xenophile's post looks a lot
like a troll attempt.

-- 
                                                                Gilles.