[Xenomai-help] milli-second overruns under heavy load on x86, running Xenomai 2.5.6

[Xenomai-help] milli-second overruns under heavy load on x86, running Xenomai 2.5.6

[George Broz]

[-- Attachment #1.1: Type: text/plain, Size: 2605 bytes --]

Hello,

I'm experiencing very large overruns in the neighborhood of 10 to 50 
*milli-seconds* as reported by the Xenomai latency tool in user-task mode 
whenever the Linux OS is heavily loaded. Under lighter loads these are in 
the 30 to 70 micro-second (normal) range. 

My system is:
- Linux 2.6.37.6 (32-bit), Ubuntu 10.10 distribution
- Xenomai 2.5.6 using I-pipe patch adeos-ipipe-2.6.37.6-x86-2.9-02
- Intel x86 Atom (dual-core) CPU D510 @ 1.66GHz 
- Emb. dev board from Intel (ICH8M ctrlr, Ethernet h/w for e1000e)
- Xenomai configured w&w/o --enable-smp --enable-x86-tsc options
- gcc version 4.4.5

I've followed all of the recommendations for kernel configuration and have 
disabled CPU sleep features in the BIOS such as SpeedStep and C-States. 
Legacy USB has been disabled in the BIOS. Hyperthreading has also been 
disabled, although I don't think it matters.

The heavy load was to compile a new Linux kernel while the latency test 
ran.

I've captured several traces with the I-pipe tracer - here are some of the 
milli-second overrun excerpts:

snippet:
    Type   User Val. Time        Delay Function (Parent)
:| +begin 0x80000001 -14384      0.551 __ipipe_handle_exception+0x3d3 
(error_code+0x5f)
:| +end   0x80000001 -14383! 14344.649 __ipipe_handle_exception+0x2d9 
(error_code+0x5f)
:| +begin 0xffffff02 -39         0.995 reschedule_interrupt+0x30 
(<40116e89>)

snippet:
:  +func             -17242      0.545 _raw_spin_lock+0x3 (ipc_lock+0x2e)
:  +func             -17241! 17188.986 ipcperms+0x10 
(sys_semtimedop+0x2f8)
:| +begin 0xffffff09 -52         0.797 ipipe_ipi3+0x30 
(sys_semtimedop+0x396)
:| +func             -51         0.701 __ipipe_handle_irq+0xa 
(ipipe_ipi3+0x37)

snippet:
: #func              -39426      0.605 scheduler_tick+0xa 
(update_process_times+0x5d)
: #func              -39425      0.797 ktime_get+0xa 
(sched_clock_tick+0x5a)
: #func              -39424! 39377.690 acpi_pm_read+0x3 (ktime_get+0x6a)
:| #begin 0xffffff09 -47         0.923 ipipe_ipi3+0x30 (acpi_pm_read+0xf)
:| #func             -46         0.707 __ipipe_handle_irq+0xa 
(ipipe_ipi3+0x37)


Attached below is a zipped text file with a more complete dump of these 
traces, plus more system info, and the complete .config file used to build 
this kernel.

If I run the exact same kernel and load on a single-core (Atom N450) 
version of this development board, these large overruns do not occur.

Would much appreciate any insight in figuring out why these overruns are 
occurring and what I can try to get rid of them.


Thanks in advance!
---
George Broz
Moog, Inc. Industrial Group



[-- Attachment #1.2: Type: text/html, Size: 4641 bytes --]

[-- Attachment #2: HL-ovrruns-xen2_5_6-AtomD510-txt.zip --]
[-- Type: application/zip, Size: 26065 bytes --]

Re: [Xenomai-help] milli-second overruns under heavy load on x86, running Xenomai 2.5.6

[Gilles Chanteperdrix]

On 08/26/2011 10:42 PM, George Broz wrote:
> Hello,
> 
> I'm experiencing very large overruns in the neighborhood of 10 to 50 
> *milli-seconds* as reported by the Xenomai latency tool in user-task mode 
> whenever the Linux OS is heavily loaded. Under lighter loads these are in 
> the 30 to 70 micro-second (normal) range. 
> 
> My system is:
> - Linux 2.6.37.6 (32-bit), Ubuntu 10.10 distribution
> - Xenomai 2.5.6 using I-pipe patch adeos-ipipe-2.6.37.6-x86-2.9-02
> - Intel x86 Atom (dual-core) CPU D510 @ 1.66GHz 
> - Emb. dev board from Intel (ICH8M ctrlr, Ethernet h/w for e1000e)
> - Xenomai configured w&w/o --enable-smp --enable-x86-tsc options
> - gcc version 4.4.5
> 
> I've followed all of the recommendations for kernel configuration and have 
> disabled CPU sleep features in the BIOS such as SpeedStep and C-States. 
> Legacy USB has been disabled in the BIOS. Hyperthreading has also been 
> disabled, although I don't think it matters.

Have you tried to enable the SMI workaround?

-- 
                                                                Gilles.

Re: [Xenomai-help] milli-second overruns under heavy load on x86, running Xenomai 2.5.6

[George Broz]

>Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> wrote on 
08/26/2011 02:18:05 PM:
> On 08/26/2011 10:42 PM, George Broz wrote:
> > Hello,
> > 
> > I'm experiencing very large overruns in the neighborhood of 10 to 50 
> > *milli-seconds* as reported by the Xenomai latency tool in user-task 
mode 
> > whenever the Linux OS is heavily loaded. Under lighter loads these are 
in 
> > the 30 to 70 micro-second (normal) range. 
> > 
> > My system is:
> > - Linux 2.6.37.6 (32-bit), Ubuntu 10.10 distribution
> > - Xenomai 2.5.6 using I-pipe patch adeos-ipipe-2.6.37.6-x86-2.9-02
> > - Intel x86 Atom (dual-core) CPU D510 @ 1.66GHz 
> > - Emb. dev board from Intel (ICH8M ctrlr, Ethernet h/w for e1000e)
> > - Xenomai configured w&w/o --enable-smp --enable-x86-tsc options
> > - gcc version 4.4.5
> > 
> > I've followed all of the recommendations for kernel configuration and 
have 
> > disabled CPU sleep features in the BIOS such as SpeedStep and 
C-States. 
> > Legacy USB has been disabled in the BIOS. Hyperthreading has also been 

> > disabled, although I don't think it matters.
> 
> Have you tried to enable the SMI workaround?
> 
> -- 
>                                                                 Gilles.
> 

Yes - SMI workaround was enabled when the overruns were observed.


# CONFIG_XENO_HW_SMI_DETECT_DISABLE is not set
CONFIG_XENO_HW_SMI_DETECT=y
CONFIG_XENO_HW_SMI_WORKAROUND=y
CONFIG_XENO_HW_SMI_ALL=y

Aug 25 11:32:33 nxtgenhd kernel: [    2.492709] I-pipe: Domain Xenomai 
registered.
  :
Aug 25 11:32:33 nxtgenhd kernel: [   11.579015] Xenomai: SMI-enabled 
chipset found
Aug 25 11:32:33 nxtgenhd kernel: [   11.579044] Xenomai: SMI workaround 
enabled


--George

Re: [Xenomai-help] milli-second overruns under heavy load on x86, running Xenomai 2.5.6

[Gilles Chanteperdrix]

On 08/26/2011 11:49 PM, George Broz wrote:
>> Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> wrote on 
> 08/26/2011 02:18:05 PM:
>> On 08/26/2011 10:42 PM, George Broz wrote:
>>> Hello,
>>>
>>> I'm experiencing very large overruns in the neighborhood of 10 to 50 
>>> *milli-seconds* as reported by the Xenomai latency tool in user-task 
> mode 
>>> whenever the Linux OS is heavily loaded. Under lighter loads these are 
> in 
>>> the 30 to 70 micro-second (normal) range. 
>>>
>>> My system is:
>>> - Linux 2.6.37.6 (32-bit), Ubuntu 10.10 distribution
>>> - Xenomai 2.5.6 using I-pipe patch adeos-ipipe-2.6.37.6-x86-2.9-02
>>> - Intel x86 Atom (dual-core) CPU D510 @ 1.66GHz 
>>> - Emb. dev board from Intel (ICH8M ctrlr, Ethernet h/w for e1000e)
>>> - Xenomai configured w&w/o --enable-smp --enable-x86-tsc options
>>> - gcc version 4.4.5
>>>
>>> I've followed all of the recommendations for kernel configuration and 
> have 
>>> disabled CPU sleep features in the BIOS such as SpeedStep and 
> C-States. 
>>> Legacy USB has been disabled in the BIOS. Hyperthreading has also been 
> 
>>> disabled, although I don't think it matters.
>>
>> Have you tried to enable the SMI workaround?
>>
>> -- 
>>                                                                 Gilles.
>>
> 
> Yes - SMI workaround was enabled when the overruns were observed.

Could you try disabling CONFIG_SMP to see if you get the same behaviour,
and take a few traces?
If you processor supports 64 bits mode, could you try running a 64 bits
kernel?


-- 
                                                                Gilles.

Re: [Xenomai-help] milli-second overruns under heavy load on x86, running Xenomai 2.5.6

[Jan Kiszka]

On 2011-08-26 22:42, George Broz wrote:
> Hello,
> 
> I'm experiencing very large overruns in the neighborhood of 10 to 50 
> *milli-seconds* as reported by the Xenomai latency tool in user-task mode 
> whenever the Linux OS is heavily loaded. Under lighter loads these are in 
> the 30 to 70 micro-second (normal) range. 
> 
> My system is:
> - Linux 2.6.37.6 (32-bit), Ubuntu 10.10 distribution
> - Xenomai 2.5.6 using I-pipe patch adeos-ipipe-2.6.37.6-x86-2.9-02
> - Intel x86 Atom (dual-core) CPU D510 @ 1.66GHz 
> - Emb. dev board from Intel (ICH8M ctrlr, Ethernet h/w for e1000e)
> - Xenomai configured w&w/o --enable-smp --enable-x86-tsc options
> - gcc version 4.4.5
> 
> I've followed all of the recommendations for kernel configuration and have 
> disabled CPU sleep features in the BIOS such as SpeedStep and C-States. 
> Legacy USB has been disabled in the BIOS. Hyperthreading has also been 
> disabled, although I don't think it matters.
> 
> The heavy load was to compile a new Linux kernel while the latency test 
> ran.
> 
> I've captured several traces with the I-pipe tracer - here are some of the 
> milli-second overrun excerpts:
> 
> snippet:
>     Type   User Val. Time        Delay Function (Parent)
> :| +begin 0x80000001 -14384      0.551 __ipipe_handle_exception+0x3d3 
> (error_code+0x5f)
> :| +end   0x80000001 -14383! 14344.649 __ipipe_handle_exception+0x2d9 
> (error_code+0x5f)
> :| +begin 0xffffff02 -39         0.995 reschedule_interrupt+0x30 
> (<40116e89>)
> 
> snippet:
> :  +func             -17242      0.545 _raw_spin_lock+0x3 (ipc_lock+0x2e)
> :  +func             -17241! 17188.986 ipcperms+0x10 
> (sys_semtimedop+0x2f8)
> :| +begin 0xffffff09 -52         0.797 ipipe_ipi3+0x30 
> (sys_semtimedop+0x396)
> :| +func             -51         0.701 __ipipe_handle_irq+0xa 
> (ipipe_ipi3+0x37)
> 
> snippet:
> : #func              -39426      0.605 scheduler_tick+0xa 
> (update_process_times+0x5d)
> : #func              -39425      0.797 ktime_get+0xa 
> (sched_clock_tick+0x5a)
> : #func              -39424! 39377.690 acpi_pm_read+0x3 (ktime_get+0x6a)
> :| #begin 0xffffff09 -47         0.923 ipipe_ipi3+0x30 (acpi_pm_read+0xf)
> :| #func             -46         0.707 __ipipe_handle_irq+0xa 
> (ipipe_ipi3+0x37)

Specifically this last snippet suggests that your hardware has some
serious latency problems. There is nothing in that code path that allows
a software explanation of those 39 ms.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Xenomai-help] milli-second overruns under heavy load on x86, running Xenomai 2.5.6

[George Broz]

[-- Attachment #1: Type: text/plain, Size: 2709 bytes --]

Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> wrote on 
08/29/2011 12:13:48 AM:

> On 08/26/2011 11:49 PM, George Broz wrote:
> >> Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> wrote on 
> > 08/26/2011 02:18:05 PM:
> >> On 08/26/2011 10:42 PM, George Broz wrote:
> >>> Hello,
> >>>
> >>> I'm experiencing very large overruns in the neighborhood of 10 to 50 

> >>> *milli-seconds* as reported by the Xenomai latency tool in user-task 

> > mode 
> >>> whenever the Linux OS is heavily loaded. Under lighter loads these 
are 
> > in 
> >>> the 30 to 70 micro-second (normal) range. 
> >>>
> >>> My system is:
> >>> - Linux 2.6.37.6 (32-bit), Ubuntu 10.10 distribution
> >>> - Xenomai 2.5.6 using I-pipe patch adeos-ipipe-2.6.37.6-x86-2.9-02
> >>> - Intel x86 Atom (dual-core) CPU D510 @ 1.66GHz 
> >>> - Emb. dev board from Intel (ICH8M ctrlr, Ethernet h/w for e1000e)
> >>> - Xenomai configured w&w/o --enable-smp --enable-x86-tsc options
> >>> - gcc version 4.4.5
> >>>
> >>> I've followed all of the recommendations for kernel configuration 
and 
> > have 
> >>> disabled CPU sleep features in the BIOS such as SpeedStep and 
> > C-States. 
> >>> Legacy USB has been disabled in the BIOS. Hyperthreading has also 
been 
> > 
> >>> disabled, although I don't think it matters.
> >>
> >> Have you tried to enable the SMI workaround?
> >>
> >> -- 
> >> Gilles.
> >>
> > 
> > Yes - SMI workaround was enabled when the overruns were observed.
> 
> Could you try disabling CONFIG_SMP to see if you get the same behaviour,
> and take a few traces?
> If you processor supports 64 bits mode, could you try running a 64 bits
> kernel?
> 
> 
> -- 
>                                                                 Gilles.
> 

I disabled CONFIG_SMP and did not get the same behaviour - latencies were 
normal (albeit now slightly increased). The processor supports 64 bit 
mode, but my initial attempt to get that working failed (for unrelated 
reasons).

I then started looking at the hardware some more after Jan's comment, and 
found another CPU frequency throttling mechanism that exists in the Atom 
N450 and D510. This is the "thermal monitor" (TM1) as described in this 
Intel document: 
http://download.intel.com/embedded/processors/thermalguide/323439.pdf

Even though my dev. board had a fan, adding even more cooling 
significantly reduced the occurrence of the overruns (but they would still 
occur occasionally). I then disabled the thermal monitor (TM1) feature 
from the BIOS, and since then have not seen any more happen (after about 2 
hours of testing). I will test more tomorrow, and post if I see the 
problem still exists.

Thanks very much for your (and Jan's) help. 

--George

[-- Attachment #2: Type: text/html, Size: 3894 bytes --]