[Xenomai-help] NMI watchdog: Loading of xeno_native leads to reboot of PC

[Xenomai-help] NMI watchdog: Loading of xeno_native leads to reboot of PC

[M. Koehrer]

Hi!

Before digging deeper into the issue I have mailed yesterday (see below), I have to solve and
to understand the NMI watchdog feature.
I have enabled the NMI watchdog in the kernel configuration and set the time value to 100 now (100us).
Also, I passed the nmi_watchdog=1 kernel parameter to GRUB.
In dmesg's output I see the line:
Testing NMI watchdog ... OK.
 
The Xenomai functionality is compiled as modules (as far as it is possible).

Now, I do a 
modprobe xeno_nucleus
This looks fine.

Now, I do a 
modprobe xeno_native
to load the native skin (I need it for my application to run).
Then the PC reboots directly.
This means I have no chance to  start my application as the PC reboots before
I can start it...

I have the impression that some important thing is missing in my test.
The xeno timer is not started after modprobe xenu_nucleus
The contents of /proc/timer is:
status=off:setup=120:tickval=0:jiffies=0

Could this be a reason for the behaviour?
Can I start the timer before loading xeno_native to avoid the NMIs?

Thanks for all help on this topic as without this feature it seems to be impossible
to detect the bug/issue with a freezing system with the latest Xenomai/rtnet and 
kernel 2.6.19.1. (see https://mail.gna.org/public/xenomai-help/2006-12/msg00109.html)

Regards

Mathias



----- Original Nachricht ----
Von:     Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
An:      "M. Koehrer" <mathias_koehrer@domain.hid>
Datum:   18.12.2006 16:32
Betreff: Re: Aw: Re: Aw: Re: [RTnet-users] [Xenomai-help] rtnet / Xenomai:
 Kernel

> M. Koehrer wrote:
> > O.k,
> > 
> > I tried once more with the  NMI watchdog stuff.
> > However, it looks as if I do not understand the NMI watchdog correctly...
> > I passed nmi_watchdog=1 as kernel parameter.
> > Now, the NMI watchdog seems to be o.k.
> > I have set the kernel parameter "NMI watchdog latency threshold (us)" to
> 1000000 (1 second).
> > Now I do a modprobe xeno_nucleus.
> > This is o.k.
> > Then I do a modprobe xeno_native.
> > This leads to a watchdog NMI on the console after 1s.
> > "NMI watchdog detected timer latency above 100000us"
> > CPU 1
> > EIP is at mwait_idle 0x23/0x37
> > 
> > When I compile the Xenomai functionality directly into the kernel (no
> modules), I never reach
> > the login prompt at my PC as the NMI watchdog from above came first...
> > 
> > Well, somehow the NMI stuff seems to work. However, I am not able to start
> my application
> > as my systems gets the NMI before I have the chance to start the
> application...
> > What is wrong here? I think, I miss one piece in the puzzle...
> > 
> > Thanks for any support on that strange behaviour.
> 
> 1 second is probably way to much and overflow a 32 bits value when
> converted to a processor ticks count. The defaults of 100 us is more
> reasonable.
> 
> -- 
>                                                  Gilles Chanteperdrix
> 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: [Xenomai-help] NMI watchdog: Loading of xeno_native leads to reboot of PC

[Jan Kiszka]

M. Koehrer wrote:
> Hi!
> 
> Before digging deeper into the issue I have mailed yesterday (see below), I have to solve and
> to understand the NMI watchdog feature.
> I have enabled the NMI watchdog in the kernel configuration and set the time value to 100 now (100us).
> Also, I passed the nmi_watchdog=1 kernel parameter to GRUB.
> In dmesg's output I see the line:
> Testing NMI watchdog ... OK.
>  
> The Xenomai functionality is compiled as modules (as far as it is possible).
> 
> Now, I do a 
> modprobe xeno_nucleus
> This looks fine.
> 
> Now, I do a 
> modprobe xeno_native
> to load the native skin (I need it for my application to run).
> Then the PC reboots directly.
> This means I have no chance to  start my application as the PC reboots before
> I can start it...
> 
> I have the impression that some important thing is missing in my test.
> The xeno timer is not started after modprobe xenu_nucleus
> The contents of /proc/timer is:
> status=off:setup=120:tickval=0:jiffies=0
> 
> Could this be a reason for the behaviour?

That's normal, it is started on loading the first skin.

> Can I start the timer before loading xeno_native to avoid the NMIs?

Nope, the NMI must work as you tried to use it. I have basically the
same setup here (all modular, NMI on by default), but even with latest
kernel/ipipe everything is fine.

Could you post your .config to compare details? Also the output of
/proc/interrupts over .19 would be interesting.

Jan

Re: Re: [Xenomai-help] NMI watchdog: Loading of xeno_native leads to

[M. Koehrer]

[-- Attachment #1.1: Type: text/plain, Size: 2431 bytes --]

Hi Jan,

here is my /proc/interrupts 

           CPU0
  0:     331260   IO-APIC-edge      timer
  1:          8   IO-APIC-edge      i8042
  7:          0   IO-APIC-edge      parport0
  9:          0   IO-APIC-fasteoi   acpi
 14:         12   IO-APIC-edge      ide0
 16:       1022   IO-APIC-fasteoi   eth0
 19:       2358   IO-APIC-fasteoi   libata
NMI:     331248
LOC:     331233
ERR:          0
MIS:          0

I have enclosed a config.gz file of my 2.6.19.1 kernel.

Mathias

> > Before digging deeper into the issue I have mailed yesterday (see below),
> I have to solve and
> > to understand the NMI watchdog feature.
> > I have enabled the NMI watchdog in the kernel configuration and set the
> time value to 100 now (100us).
> > Also, I passed the nmi_watchdog=1 kernel parameter to GRUB.
> > In dmesg's output I see the line:
> > Testing NMI watchdog ... OK.
> >  
> > The Xenomai functionality is compiled as modules (as far as it is
> possible).
> > 
> > Now, I do a 
> > modprobe xeno_nucleus
> > This looks fine.
> > 
> > Now, I do a 
> > modprobe xeno_native
> > to load the native skin (I need it for my application to run).
> > Then the PC reboots directly.
> > This means I have no chance to  start my application as the PC reboots
> before
> > I can start it...
> > 
> > I have the impression that some important thing is missing in my test.
> > The xeno timer is not started after modprobe xenu_nucleus
> > The contents of /proc/timer is:
> > status=off:setup=120:tickval=0:jiffies=0
> > 
> > Could this be a reason for the behaviour?
> 
> That's normal, it is started on loading the first skin.
> 
> > Can I start the timer before loading xeno_native to avoid the NMIs?
> 
> Nope, the NMI must work as you tried to use it. I have basically the
> same setup here (all modular, NMI on by default), but even with latest
> kernel/ipipe everything is fine.
> 
> Could you post your .config to compare details? Also the output of
> /proc/interrupts over .19 would be interesting.
> 
> Jan
> 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[-- Attachment #2: config.gz --]
[-- Type: application/octetstream, Size: 8526 bytes --]

Re: [Xenomai-help] NMI watchdog: Loading of xeno_native leads to

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 788 bytes --]

M. Koehrer wrote:
> Hi Jan,
> 
> here is my /proc/interrupts 
> 
>            CPU0
>   0:     331260   IO-APIC-edge      timer
>   1:          8   IO-APIC-edge      i8042
>   7:          0   IO-APIC-edge      parport0
>   9:          0   IO-APIC-fasteoi   acpi
>  14:         12   IO-APIC-edge      ide0
>  16:       1022   IO-APIC-fasteoi   eth0
>  19:       2358   IO-APIC-fasteoi   libata
> NMI:     331248
> LOC:     331233
> ERR:          0
> MIS:          0

Nothing unusual on first sight.

> 
> I have enclosed a config.gz file of my 2.6.19.1 kernel.

Two things to try:
 - CONFIG_XENO_HW_NMI_DEBUG_LATENCY_MAX=100 (i.e. default again)
 - CONFIG_PROFILING=y (I'm not sure ATM if it may interfere with
                       Xenomai's watchdog)

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Aw: Re: [Xenomai-help] NMI watchdog: Loading of xeno_native leads to

[M. Koehrer]

Hi!

I have modified the configuration to disable profiling and to set the NMI time value to 100.
However, there is no difference.
Could this be a compiler issue?
I am using gcc 4.1.2 (debian etch prerelease)
I try to recompile the kernel using an older gcc-3.3 to see if this helps.

Regards

Mathias
> > here is my /proc/interrupts 
> > 
> >            CPU0
> >   0:     331260   IO-APIC-edge      timer
> >   1:          8   IO-APIC-edge      i8042
> >   7:          0   IO-APIC-edge      parport0
> >   9:          0   IO-APIC-fasteoi   acpi
> >  14:         12   IO-APIC-edge      ide0
> >  16:       1022   IO-APIC-fasteoi   eth0
> >  19:       2358   IO-APIC-fasteoi   libata
> > NMI:     331248
> > LOC:     331233
> > ERR:          0
> > MIS:          0
> 
> Nothing unusual on first sight.
> 
> > 
> > I have enclosed a config.gz file of my 2.6.19.1 kernel.
> 
> Two things to try:
>  - CONFIG_XENO_HW_NMI_DEBUG_LATENCY_MAX=100 (i.e. default again)
>  - CONFIG_PROFILING=y (I'm not sure ATM if it may interfere with
>                        Xenomai's watchdog)
> 
> Jan
> 
> 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: [Xenomai-help] NMI watchdog: Loading of xeno_native leads...

[M. Koehrer]

Hi!

It was the compiler!!! Using gcc-3.3 solved the issue.
That means, that somehow gcc-4.1.2 must not be used with Xenomai currently...
 
This allowed me to catch the system hang I was hunting for since yesterday...
And I got it.
I write the information I found on the console (I have not tried with console redirection yet...).
CPU 0, eip c010122d
EIP 0060:[<c010122d>]
EFLAGS 00000246
EIP is at mwait_idle_with_hints+0x2c/0x2e
eax:0 ebx:c0498000 ecx:0 edx:0

Call Trace
mwait_idle+0x0/0x2e
mwait_idle+0x1d/0x2e
cpu_idle+0x44/0x86
start_kernel+0x1f1/0x1f5
unknown_bootoption+0x0/0x191

I'll try to reproduce the same with console redirection, however, I hope this could help already!

Regards

Mathias
> Hi!
> 
> I have modified the configuration to disable profiling and to set the NMI
> time value to 100.
> However, there is no difference.
> Could this be a compiler issue?
> I am using gcc 4.1.2 (debian etch prerelease)
> I try to recompile the kernel using an older gcc-3.3 to see if this helps.
> 
> Regards
> 
> Mathias
> > > here is my /proc/interrupts 
> > > 
> > >            CPU0
> > >   0:     331260   IO-APIC-edge      timer
> > >   1:          8   IO-APIC-edge      i8042
> > >   7:          0   IO-APIC-edge      parport0
> > >   9:          0   IO-APIC-fasteoi   acpi
> > >  14:         12   IO-APIC-edge      ide0
> > >  16:       1022   IO-APIC-fasteoi   eth0
> > >  19:       2358   IO-APIC-fasteoi   libata
> > > NMI:     331248
> > > LOC:     331233
> > > ERR:          0
> > > MIS:          0
> > 
> > Nothing unusual on first sight.
> > 
> > > 
> > > I have enclosed a config.gz file of my 2.6.19.1 kernel.
> > 
> > Two things to try:
> >  - CONFIG_XENO_HW_NMI_DEBUG_LATENCY_MAX=100 (i.e. default again)
> >  - CONFIG_PROFILING=y (I'm not sure ATM if it may interfere with
> >                        Xenomai's watchdog)
> > 
> > Jan
> > 
> > 
> 
> -- 
> Mathias Koehrer
> mathias_koehrer@domain.hid
> 
> 
> Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
> ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
> und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
> nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
> http://www.arcor.de/rd/emf-dsl-2
> 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: Aw: Re: [Xenomai-help] NMI watchdog: Loading of xeno_native leads to

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 590 bytes --]

M. Koehrer wrote:
> Hi!
> 
> I have modified the configuration to disable profiling and to set the NMI time value to 100.
> However, there is no difference.
> Could this be a compiler issue?
> I am using gcc 4.1.2 (debian etch prerelease)

My feeling is no (I'm using 4.1 here, .2 should only fix regressions in
theory).

> I try to recompile the kernel using an older gcc-3.3 to see if this helps.

After that let us forget about the NMI for a while and check if the soft
watchdog + CONFIG_XENO_OPT_DEBUG_NUCLEUS may already reveal some detail
of the RTnet issue.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] NMI watchdog: Loading of xeno_native leads...

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1267 bytes --]

M. Koehrer wrote:
> Hi!
> 
> It was the compiler!!! Using gcc-3.3 solved the issue.
> That means, that somehow gcc-4.1.2 must not be used with Xenomai currently...

OK, this needs some examination then. Can anyone reproduce this issue?
/me is currently lacking the compiler.

>  
> This allowed me to catch the system hang I was hunting for since yesterday...
> And I got it.
> I write the information I found on the console (I have not tried with console redirection yet...).
> CPU 0, eip c010122d
> EIP 0060:[<c010122d>]
> EFLAGS 00000246
> EIP is at mwait_idle_with_hints+0x2c/0x2e
> eax:0 ebx:c0498000 ecx:0 edx:0
> 
> Call Trace
> mwait_idle+0x0/0x2e
> mwait_idle+0x1d/0x2e
> cpu_idle+0x44/0x86
> start_kernel+0x1f1/0x1f5
> unknown_bootoption+0x0/0x191

Well, this caught your box in the idle loop - not that informative yet.
We must probably look beyond the last context switch. Please switch on
the I-pipe tracer (kernel hacking -> I-pipe debugging) and configure
back_trace_points to, hmm, say 200. On NMI alarm, we should then see a
function call trace.

> 
> I'll try to reproduce the same with console redirection, however, I hope this could help already!
> 
> Regards
> 
> Mathias

Thanks for your effort,

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Aw: Re: [Xenomai-help] NMI watchdog: Loading of xeno_native leads...

[M. Koehrer]

Hi Jan,

I tried to enable the kernel hacking parameters you proposed.
However, now I get a "NMI early shots: 0" message many times per second.
These seems to slow down everything dramatically, it is hard to work with the
system. What are the early shots messages and how can I avoid them?

Regards

Mathias


> > It was the compiler!!! Using gcc-3.3 solved the issue.
> > That means, that somehow gcc-4.1.2 must not be used with Xenomai
> currently...
> 
> OK, this needs some examination then. Can anyone reproduce this issue?
> /me is currently lacking the compiler.
> 
> >  
> > This allowed me to catch the system hang I was hunting for since
> yesterday...
> > And I got it.
> > I write the information I found on the console (I have not tried with
> console redirection yet...).
> > CPU 0, eip c010122d
> > EIP 0060:[<c010122d>]
> > EFLAGS 00000246
> > EIP is at mwait_idle_with_hints+0x2c/0x2e
> > eax:0 ebx:c0498000 ecx:0 edx:0
> > 
> > Call Trace
> > mwait_idle+0x0/0x2e
> > mwait_idle+0x1d/0x2e
> > cpu_idle+0x44/0x86
> > start_kernel+0x1f1/0x1f5
> > unknown_bootoption+0x0/0x191
> 
> Well, this caught your box in the idle loop - not that informative yet.
> We must probably look beyond the last context switch. Please switch on
> the I-pipe tracer (kernel hacking -> I-pipe debugging) and configure
> back_trace_points to, hmm, say 200. On NMI alarm, we should then see a
> function call trace.
> 
> > 
> > I'll try to reproduce the same with console redirection, however, I hope
> this could help already!
> > 
> > Regards
> > 
> > Mathias
> 
> Thanks for your effort,
> 
> Jan
> 
> 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: Aw: Re: [Xenomai-help] NMI watchdog: Loading of xeno_native leads...

[Gilles Chanteperdrix]

M. Koehrer wrote:
> Hi Jan,
> 
> I tried to enable the kernel hacking parameters you proposed.
> However, now I get a "NMI early shots: 0" message many times per second.
> These seems to slow down everything dramatically, it is hard to work with the
> system. What are the early shots messages and how can I avoid them?

Normally, when you see "NMI early shots: 0", your system is about to
panic. I guess the reason why your system is able to recover from this
situation is because you enabled the watchdog. What would be interesting
would be to know what is printed right after "NMI early shots: 0", it
may be significant, if not, simply disable the NMI watchdog, maybe it
does not work properly on your machine.

-- 
                                                 Gilles Chanteperdrix

Re: Aw: Re: [Xenomai-help] NMI watchdog: Loading of xeno_native leads...

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 812 bytes --]

M. Koehrer wrote:
> Hi Jan,
> 
> I tried to enable the kernel hacking parameters you proposed.
> However, now I get a "NMI early shots: 0" message many times per second.
> These seems to slow down everything dramatically, it is hard to work with the
> system. What are the early shots messages and how can I avoid them?

My feeling is that the issue you saw with gcc-4.1.2 is still unsolved,
just the recompilation made some stuff move around. Now, with additional
features activated, the issue seems to pop up again in a different way.

So again my suggestion: forget about the NMI for a while, check if we
can make progress with other debugging switches.

Further question to tighten the test case: do the basic Xenomai
testsuite programs run without noticeable problems on your box?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Aw: Re: Aw: Re: [Xenomai-help] NMI watchdog: Loading of xeno_native leads...

[M. Koehrer]

Hi Jan,

I think you are right.
The NMI watchdog (Realtime) kernel parameter in the kernel configuration was
accidentally disabled (perhaps because I tried in between with a non-APIC IRQ).
Now I have again the error situation of this morning:
Even with gcc-3.3 I am not able to load the xeno_native module when NMI watchdog
is enabled, the system directly reboots (without printing to the serial console)
The printout I have sent before was the printout of the standard linux NMI but
not of the Xenomai NMI.

I.e., it seems _not_ to be the reason of the compiler that leads to the behaviour!

Regards

Mathias
> M. Koehrer wrote:
> > Hi Jan,
> > 
> > I tried to enable the kernel hacking parameters you proposed.
> > However, now I get a "NMI early shots: 0" message many times per second.
> > These seems to slow down everything dramatically, it is hard to work with
> the
> > system. What are the early shots messages and how can I avoid them?
> 
> My feeling is that the issue you saw with gcc-4.1.2 is still unsolved,
> just the recompilation made some stuff move around. Now, with additional
> features activated, the issue seems to pop up again in a different way.
> 
> So again my suggestion: forget about the NMI for a while, check if we
> can make progress with other debugging switches.
> 
> Further question to tighten the test case: do the basic Xenomai
> testsuite programs run without noticeable problems on your box?
> 
> Jan
> 
> 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[Xenomai-help] A fairly small rtnet/Xenomai application that freezes the PC

[M. Koehrer]

[-- Attachment #1.1: Type: text/plain, Size: 1786 bytes --]

Hi Jan and Gilles,

after a long debug-reboot-try-again-session I was able to reduce the problem I had
to a very short application that leads to a complete system freeze on a 2.6.19.1 kernel.
As with my PC I am not able to run the NMI stuff within Xenomai, I forward the application to you
and the list. Perhaps somebody can try out the enclosed application with the latest Xenomai version
on 2.6.19.1.
Please adjust the IP settings in the C file and in the xenorun script to your setup.
It is sufficient to send out UDP frames, if the remote device does not answer this does not hurt.
However the remote device should listen on a specified port to avoid ICMP complains about
no-opened UDP ports...
Using Ethereal I monitored the traffic.
Here is the summary of it:
ARP request/response (from the xenorun script)
ARP request/response (from the applicaion)
UDP message to port 18765 (no answer)
2 seconds later: UDP message to port 18765 (no answer)
2+5=7 seconds later: ARP request/response (from the application)
2 seconds later: UDP message to port 18765
Here the system freezes. I do not see the message "Step A".

One important piece is the system() call out of the realtime application to do a rtroute.
When I remove this call, there is no error...


I am using Xenomai SVN #1969 and the included 2.6.19.1 patch. Pentium 4 UP.

Any feedback on this is highly welcome!

Regards

Mathias



-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[-- Attachment #2: rtnet_crashtest.tgz --]
[-- Type: application/octet-stream, Size: 1889 bytes --]

[Xenomai-help] Re: A fairly small rtnet/Xenomai application that freezes the PC

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1939 bytes --]

M. Koehrer wrote:
> Hi Jan and Gilles,
> 
> after a long debug-reboot-try-again-session I was able to reduce the problem I had
> to a very short application that leads to a complete system freeze on a 2.6.19.1 kernel.
> As with my PC I am not able to run the NMI stuff within Xenomai, I forward the application to you
> and the list. Perhaps somebody can try out the enclosed application with the latest Xenomai version
> on 2.6.19.1.

Thanks for the test case, I will give this a try ASAP.

> Please adjust the IP settings in the C file and in the xenorun script to your setup.
> It is sufficient to send out UDP frames, if the remote device does not answer this does not hurt.
> However the remote device should listen on a specified port to avoid ICMP complains about
> no-opened UDP ports...
> Using Ethereal I monitored the traffic.
> Here is the summary of it:
> ARP request/response (from the xenorun script)
> ARP request/response (from the applicaion)
> UDP message to port 18765 (no answer)
> 2 seconds later: UDP message to port 18765 (no answer)
> 2+5=7 seconds later: ARP request/response (from the application)
> 2 seconds later: UDP message to port 18765
> Here the system freezes. I do not see the message "Step A".

Could you set me the Ethereal dump for reference? Just in case the test
does not kick immediately for me and I need to check the event flow.

> 
> One important piece is the system() call out of the realtime application to do a rtroute.
> When I remove this call, there is no error...
> 
> 
> I am using Xenomai SVN #1969 and the included 2.6.19.1 patch. Pentium 4 UP.

Do you have a different (slower) execution platform at hand to check if
the CPU speed as influence on the lock-up? I hope it is not the case -
makes tracking easier. But it wouldn't be the first time.

> 
> Any feedback on this is highly welcome!
> 

You will get it.

Thanks again,
Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

[Xenomai-help] Aw: Re: A fairly small rtnet/Xenomai application that freezes the PC

[M. Koehrer]

[-- Attachment #1.1: Type: text/plain, Size: 2569 bytes --]

Hi Jan,

enclosed is the ethereal dump file.
Unfortunately I do not have a slow machine... (We try to get the fastest machines running...).

Thanks for checking the application.

Mathias
> > after a long debug-reboot-try-again-session I was able to reduce the
> problem I had
> > to a very short application that leads to a complete system freeze on a
> 2.6.19.1 kernel.
> > As with my PC I am not able to run the NMI stuff within Xenomai, I forward
> the application to you
> > and the list. Perhaps somebody can try out the enclosed application with
> the latest Xenomai version
> > on 2.6.19.1.
> 
> Thanks for the test case, I will give this a try ASAP.
> 
> > Please adjust the IP settings in the C file and in the xenorun script to
> your setup.
> > It is sufficient to send out UDP frames, if the remote device does not
> answer this does not hurt.
> > However the remote device should listen on a specified port to avoid ICMP
> complains about
> > no-opened UDP ports...
> > Using Ethereal I monitored the traffic.
> > Here is the summary of it:
> > ARP request/response (from the xenorun script)
> > ARP request/response (from the applicaion)
> > UDP message to port 18765 (no answer)
> > 2 seconds later: UDP message to port 18765 (no answer)
> > 2+5=7 seconds later: ARP request/response (from the application)
> > 2 seconds later: UDP message to port 18765
> > Here the system freezes. I do not see the message "Step A".
> 
> Could you set me the Ethereal dump for reference? Just in case the test
> does not kick immediately for me and I need to check the event flow.
> 
> > 
> > One important piece is the system() call out of the realtime application
> to do a rtroute.
> > When I remove this call, there is no error...
> > 
> > 
> > I am using Xenomai SVN #1969 and the included 2.6.19.1 patch. Pentium 4
> UP.
> 
> Do you have a different (slower) execution platform at hand to check if
> the CPU speed as influence on the lock-up? I hope it is not the case -
> makes tracking easier. But it wouldn't be the first time.
> 
> > 
> > Any feedback on this is highly welcome!
> > 
> 
> You will get it.
> 
> Thanks again,
> Jan
> 
> 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[-- Attachment #2: rtnet_crashtest.tcpdump --]
[-- Type: application/octet-stream, Size: 714 bytes --]

[Xenomai-help] Re: Re: A fairly small rtnet/Xenomai application that freezes the

[M. Koehrer]

Hi Jan,

meanwhile I have done a couple of additional tests.
With one of the PCs I have, I have disabled the Memory Cache to slow it down.
The effect was the very same. 
I have reproduced the behaviour on another PC (also P4, but different mainboard).
The system freezes as well.
And finally, I have found out that printing out a line with printf() directly after the system() call provides
a workaround.Then the system is stable.

I have replaced the system() call where I called rtroute with a simple call to "ls -l" (i.e system("ls -l") ).
Then the system freezes as well.
It looks to me as if a system() call out of the realtime task is not properly handled.
An printf() after the system() call seems to move the system back on track...

Hope that helps a little bit to identify the issue...

Regards

Mathias 

> Hi Jan,
> 
> enclosed is the ethereal dump file.
> Unfortunately I do not have a slow machine... (We try to get the fastest
> machines running...).
> 
> Thanks for checking the application.
> 
> Mathias
> > > after a long debug-reboot-try-again-session I was able to reduce the
> > problem I had
> > > to a very short application that leads to a complete system freeze on a
> > 2.6.19.1 kernel.
> > > As with my PC I am not able to run the NMI stuff within Xenomai, I
> forward
> > the application to you
> > > and the list. Perhaps somebody can try out the enclosed application
> with
> > the latest Xenomai version
> > > on 2.6.19.1.
> > 
> > Thanks for the test case, I will give this a try ASAP.
> > 
> > > Please adjust the IP settings in the C file and in the xenorun script
> to
> > your setup.
> > > It is sufficient to send out UDP frames, if the remote device does not
> > answer this does not hurt.
> > > However the remote device should listen on a specified port to avoid
> ICMP
> > complains about
> > > no-opened UDP ports...
> > > Using Ethereal I monitored the traffic.
> > > Here is the summary of it:
> > > ARP request/response (from the xenorun script)
> > > ARP request/response (from the applicaion)
> > > UDP message to port 18765 (no answer)
> > > 2 seconds later: UDP message to port 18765 (no answer)
> > > 2+5=7 seconds later: ARP request/response (from the application)
> > > 2 seconds later: UDP message to port 18765
> > > Here the system freezes. I do not see the message "Step A".
> > 
> > Could you set me the Ethereal dump for reference? Just in case the test
> > does not kick immediately for me and I need to check the event flow.
> > 
> > > 
> > > One important piece is the system() call out of the realtime
> application
> > to do a rtroute.
> > > When I remove this call, there is no error...
> > > 
> > > 
> > > I am using Xenomai SVN #1969 and the included 2.6.19.1 patch. Pentium 4
> > UP.
> > 
> > Do you have a different (slower) execution platform at hand to check if
> > the CPU speed as influence on the lock-up? I hope it is not the case -
> > makes tracking easier. But it wouldn't be the first time.
> > 
> > > 
> > > Any feedback on this is highly welcome!
> > > 
> > 
> > You will get it.
> > 
> > Thanks again,
> > Jan
> > 
> > 
> 
> -- 
> Mathias Koehrer
> mathias_koehrer@domain.hid
> 


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[Xenomai-help] Re: A fairly small rtnet/Xenomai application that freezes the

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1450 bytes --]

M. Koehrer wrote:
> Hi Jan,
> 
> meanwhile I have done a couple of additional tests.
> With one of the PCs I have, I have disabled the Memory Cache to slow it down.
> The effect was the very same. 
> I have reproduced the behaviour on another PC (also P4, but different mainboard).
> The system freezes as well.
> And finally, I have found out that printing out a line with printf() directly after the system() call provides
> a workaround.Then the system is stable.
> 
> I have replaced the system() call where I called rtroute with a simple call to "ls -l" (i.e system("ls -l") ).
> Then the system freezes as well.
> It looks to me as if a system() call out of the realtime task is not properly handled.
> An printf() after the system() call seems to move the system back on track...
> 
> Hope that helps a little bit to identify the issue...

Unfortunately not yet. I tried with exactly the same configuration you
once mailed on a Pentium M 1.3 GHz - but all worked fine. The obvious
differences are the CPU speed (I may have access to a crash box with
more GHz next week) and the NIC (rt_eepro100 in my case).

As my hope of being able to reproduce it on my own is not that high, I
would like to ask you to try nailing down the lock-up on kernel level.
That means trying to identify (via printk e.g. - if this doesn't make
the bug jump around) which function is still executed and where we do
not get.

TIA,
Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

[Xenomai-help] Re: A fairly small rtnet/Xenomai application that freezes the

[M. Koehrer]

[-- Attachment #1.1: Type: text/plain, Size: 2547 bytes --]

Hi Jan, hi everybody,

I have stripped down my program that is crashing Xenomai even further.
(I have attached the complete source code).
No rtnet is required.
Now I have the following real time task:

static void realtimetask(void *arg)
{
    system("ls -l");
    rt_task_sleep(1000000000ULL);
    printf("rt_task_sleep done...\n");
}

This leads to a complete freeze of the PC on a 2.6.19.1 kernel using the latest
Xenomai (from SVN) and the included adeos-patch.
I never had this issue on 2.6.17.* kernel.

Unfortunately this seems to occur only on fast Pentium 4 machines...
Is there anybody out there that can reproduce it?

Thanks for all feedback on this!

Mathias

> > meanwhile I have done a couple of additional tests.
> > With one of the PCs I have, I have disabled the Memory Cache to slow it
> down.
> > The effect was the very same. 
> > I have reproduced the behaviour on another PC (also P4, but different
> mainboard).
> > The system freezes as well.
> > And finally, I have found out that printing out a line with printf()
> directly after the system() call provides
> > a workaround.Then the system is stable.
> > 
> > I have replaced the system() call where I called rtroute with a simple
> call to "ls -l" (i.e system("ls -l") ).
> > Then the system freezes as well.
> > It looks to me as if a system() call out of the realtime task is not
> properly handled.
> > An printf() after the system() call seems to move the system back on
> track...
> > 
> > Hope that helps a little bit to identify the issue...
> 
> Unfortunately not yet. I tried with exactly the same configuration you
> once mailed on a Pentium M 1.3 GHz - but all worked fine. The obvious
> differences are the CPU speed (I may have access to a crash box with
> more GHz next week) and the NIC (rt_eepro100 in my case).
> 
> As my hope of being able to reproduce it on my own is not that high, I
> would like to ask you to try nailing down the lock-up on kernel level.
> That means trying to identify (via printk e.g. - if this doesn't make
> the bug jump around) which function is still executed and where we do
> not get.
> 
> TIA,
> Jan


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[-- Attachment #2: xeno_crash.tgz --]
[-- Type: application/octet-stream, Size: 954 bytes --]

Re: [Xenomai-help] Re: A fairly small rtnet/Xenomai application that freezes the

[Dmitry Adamushko]

On 21/12/06, M. Koehrer <mathias_koehrer@domain.hid> wrote:
> Hi Jan, hi everybody,
>
> I have stripped down my program that is crashing Xenomai even further.
> (I have attached the complete source code).
> No rtnet is required.
> Now I have the following real time task:
>
> static void realtimetask(void *arg)
> {
>     system("ls -l");
>     rt_task_sleep(1000000000ULL);
>     printf("rt_task_sleep done...\n");
> }


Is it still true that when you place printf() right after the system()
call, it works?

What happens when you try different sleep intervals : 0, say 1000 ?

Just to be sure where we are stuck. Insert exit() (or rt_task_delete(NULL)) :

(0) after system() --- if said above about printf() is not true;
(1) after rt_task_sleep();
(2) after printf("rt_task_sleep done...\n").

At which step does a hang start occuring?

>
> This leads to a complete freeze of the PC on a 2.6.19.1 kernel using the latest
> Xenomai (from SVN) and the included adeos-patch.

I have almost the same config (only 2.6.19) at home, although it's p3
750 so I'll try if it won't be solved by this everning (or we are sure
that only high-end machines are involved).


-- 
Best regards,
Dmitry Adamushko

[Xenomai-help] Re: A fairly small rtnet/Xenomai application that freezes the

[Gilles Chanteperdrix]

M. Koehrer wrote:
> Hi Jan, hi everybody,
> 
> I have stripped down my program that is crashing Xenomai even further.
> (I have attached the complete source code).
> No rtnet is required.
> Now I have the following real time task:
> 
> static void realtimetask(void *arg)
> {
>     system("ls -l");

If you want to use system (or any function calling fork, eg popen or
vfork) with Xenomai, you have to make sure to fault all pages mapped
with write permission after the fork before trying to use services in
priimary mode, because fork write protects all pages with write
permission and notably the threads stacks. A piece of code that faults
all pages follows.

static void fault_vm(void)
{
        FILE *maps = fopen("/proc/self/maps", "r");
        unsigned begin, end, pagesize=getpagesize();
        char buffer[128];
        int rc;

        if (!maps) {
                perror("fopen");
                exit(EXIT_FAILURE);
        }

        while ((rc = fscanf(maps, "%x-%x",&begin, &end) == 2)) {
                fgets(buffer, 128, maps);

                if(buffer[2] != 'w')
                        continue;

                for (; begin != end; begin += pagesize)
                        *(volatile int *) begin = *(volatile int *) begin;
        }

        fclose(maps);
}


-- 
                                                 Gilles Chanteperdrix

[Xenomai-help] Re: Re: A fairly small rtnet/Xenomai application that freezes the

[M. Koehrer]

Hi Dmitry, 

thanks for your response.
> 
> 
> Is it still true that when you place printf() right after the system()
> call, it works?
Yes, a printf() directly after the system() fixes the issue.

> 
> What happens when you try different sleep intervals : 0, say 1000 ?
> 
> Just to be sure where we are stuck. Insert exit() (or rt_task_delete(NULL))
> :
> 
> (0) after system() --- if said above about printf() is not true;
> (1) after rt_task_sleep();
> (2) after printf("rt_task_sleep done...\n").
I have added a hacky trace mechanism in the kernel to see what's happening
(whenever the nmi_watchdog with the kernel oops appears).
What I can see is that rt_task_sleep(); is completely passed.
If I modify my application to add a second rt_task_sleep() directly after the
existing one, I see the both rt_task_sleep() calls are passed.
The freeze seems to happen with the printf() call after the rt_task_sleep.
Unfortunately, I do not know where I can trace here...

Best regards

Mathias


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: [Xenomai-help] Re: Re: A fairly small rtnet/Xenomai application that freezes the

[Philippe Gerum]

On Thu, 2006-12-21 at 12:19 +0100, M. Koehrer wrote:
> Hi Dmitry, 
> 
> thanks for your response.
> > 
> > 
> > Is it still true that when you place printf() right after the system()
> > call, it works?
> Yes, a printf() directly after the system() fixes the issue.
> 
> > 
> > What happens when you try different sleep intervals : 0, say 1000 ?
> > 
> > Just to be sure where we are stuck. Insert exit() (or rt_task_delete(NULL))
> > :

The important issue to check is the above one. Please try calling
rt_task_delete(NULL), and in a second test, rt_task_suspend(NULL),
instead of letting the thread routine return. TIA,

-- 
Philippe.

[Xenomai-help] Re: Re: A fairly small rtnet/Xenomai application that freezes the

[M. Koehrer]

Hi Gilles,

thanks for your answer.
That is quite interesting. 
I have added your function and placed at call to it directly after
the system() call. Then, everything works fine.
However, a couple of questions come up:

1.) Why does a printf() directly after the system() seems to help aswell?
I do not think, that printf cares about write protection of pages...

2.)  I think, I never saw this behaviour with the 2.6.17.* kernel, only with the
2.6.19.1. Is there any difference related to that?
I will doublecheck this.

3) On Jan's PC, the issue never appears, as he has a slower PC.
How can that be related to that?

4.) Could this somehow be handled by Xenomai? Well, adding the function
is - in my eyes - a workaround around the issue, but I expect Xenomai to 
do this things for me. It needs a lot of "magic knowledge" to get this running...
Another approach could be to encapsulate this somehow in the native library
(and in the other libraries) and to document clearly, that fork and related functions
should not be called out of the realtime task without additional cleanup code.

Anyway, this seems to be the right track!

Regards

Mathias

> > I have stripped down my program that is crashing Xenomai even further.
> > (I have attached the complete source code).
> > No rtnet is required.
> > Now I have the following real time task:
> > 
> > static void realtimetask(void *arg)
> > {
> >     system("ls -l");
> 
> If you want to use system (or any function calling fork, eg popen or
> vfork) with Xenomai, you have to make sure to fault all pages mapped
> with write permission after the fork before trying to use services in
> priimary mode, because fork write protects all pages with write
> permission and notably the threads stacks. A piece of code that faults
> all pages follows.
> 


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[Xenomai-help] Re: A fairly small rtnet/Xenomai application that freezes the

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1121 bytes --]

Gilles Chanteperdrix wrote:
> M. Koehrer wrote:
>> Hi Jan, hi everybody,
>>
>> I have stripped down my program that is crashing Xenomai even further.
>> (I have attached the complete source code).
>> No rtnet is required.
>> Now I have the following real time task:
>>
>> static void realtimetask(void *arg)
>> {
>>     system("ls -l");
> 
> If you want to use system (or any function calling fork, eg popen or
> vfork) with Xenomai, you have to make sure to fault all pages mapped
> with write permission after the fork before trying to use services in
> priimary mode, because fork write protects all pages with write
> permission and notably the threads stacks. A piece of code that faults
> all pages follows.

Let me check if I got this correctly: fork calls mark all pages
write-protected - also those of the father process, or only those of the
new child? That would mean we see unexpected page faults, right?
Anything more (probably not a lock-up)?

I think we should document this special property somewhere, e.g. in the
wiki, maybe also a demo program in the examples repos.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

[Xenomai-help] Re: Re: Re: A fairly small rtnet/Xenomai

[M. Koehrer]

Hello Philippe, 

> The important issue to check is the above one. Please try calling
> rt_task_delete(NULL), and in a second test, rt_task_suspend(NULL),
> instead of letting the thread routine return. TIA,
O.K, here are the results:
I have placed rt_task_delete() at all possible positions in realtimetask().
The system freezes when I place rt_task_delete() after the printf() statement.
At all other positions (after system(), after rt_task_sleep()) the system does not freeze.
I get the very same results when I use rt_task_suspend() instead of rt_task_delete().

It means, that the printf() call causes the freeze.

Regards

Mathias

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[Xenomai-help] Re: Re: A fairly small rtnet/Xenomai application that freezes the

[M. Koehrer]

Hi all,

I have doublechecked the behaviour for the 2.6.17.14 kernel.
With this kernel (and the related Adeos patch provided by Xenomai) 
my xeno_crash application works fine without any modification or
additional workaround.
Thus, I think this is an issue that is related to 2.6.19.* kernel only. 

Regards

Mathias
> 2.)  I think, I never saw this behaviour with the 2.6.17.* kernel, only with
> the
> 2.6.19.1. Is there any difference related to that?
> I will doublecheck this.
> 


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[Xenomai-help] Re: A fairly small rtnet/Xenomai application that freezes the

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 890 bytes --]

M. Koehrer wrote:
> Hi Jan, hi everybody,
> 
> I have stripped down my program that is crashing Xenomai even further.
> (I have attached the complete source code).
> No rtnet is required.
> Now I have the following real time task:
> 
> static void realtimetask(void *arg)
> {
>     system("ls -l");
>     rt_task_sleep(1000000000ULL);
>     printf("rt_task_sleep done...\n");
> }

Does this code freeze your box as well:

#include <native/task.h>
#include <sys/mman.h>

main()
{
    RT_TASK task;

    mlockall(MCL_CURRENT|MCL_FUTURE);
    rt_task_shadow(&task, "crash", 1, 0);
    rt_task_sleep(1000000000ULL);
    system("ls -l");
    rt_task_sleep(1000000000ULL);
    printf("rt_task_sleep done...\n");
}

Or what is different with your test case? I got hold of P4 2.8 GHz
meanwhile, but the box still prefers to run the code without troubles.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] Re: A fairly small rtnet/Xenomai application that freezes the

[Dmitry Adamushko]

On 21/12/06, Jan Kiszka <jan.kiszka@domain.hid> wrote:
> Gilles Chanteperdrix wrote:
...
> >> static void realtimetask(void *arg)
> >> {
> >>     system("ls -l");
> >
> > If you want to use system (or any function calling fork, eg popen or
> > vfork) with Xenomai, you have to make sure to fault all pages mapped
> > with write permission after the fork before trying to use services in
> > priimary mode, because fork write protects all pages with write
> > permission and notably the threads stacks. A piece of code that faults
> > all pages follows.
>
> Let me check if I got this correctly: fork calls mark all pages
> write-protected - also those of the father process, or only those of the
> new child?

It's copy-on-write policy. fork() creates a separate address space for
a child (vm_struct, vm_area(s), all the page tables) but it still
points to the same physical pages as parent's address space. Now what
if either a child or parent want to modify writeable pages? That's why
copy-on-write. A new page will be allocated when a write request is
commited. And yep, to address this issue, the vm areas of (in fact,
related areas in page tables) both parent and child are marked as
copy-on-write.

But I don't think this is an issue here :

1) by the moment system() returns -> the child has finished and there
is no need to allocate new pages on parent's write requests (not sure
how linux handles it - have to check);

2) (I'd expect logically) the parent did mlockall() in advance for
even future allocations, so I'd expect copy-on-write is not applicable
fot it -> all the pages should have been duplicated for a child. Well,
it's a brief idea. Have to check.

so it's likely that it's just a side - effect.

Mathias: if printf() helps (after system()) so could you try e.g.
fopen(), fclose() instead? Also, please, do what previously was
suggested with rt_task_delete(NULL) at the end.

Another situation would be if one does fork() in real-time app. and
then both parent and child continue their execution (worth checking).

p.s. vfork() is a different beast. Here a child is supposed to call
exec*() rigth after fork() so it's allowed to "borrow" parent's
address space (vm_struct -> all vm_area(s)) and the parent is blocked
in the mean time. As a result, no need for copy-on-write.


-- 
Best regards,
Dmitry Adamushko

[Xenomai-help] Re: Re: Re: A fairly small rtnet/Xenomai

[Dmitry Adamushko]

On 21/12/06, M. Koehrer <mathias_koehrer@domain.hid> wrote:
> Hello Philippe,
>
> > The important issue to check is the above one. Please try calling
> > rt_task_delete(NULL), and in a second test, rt_task_suspend(NULL),
> > instead of letting the thread routine return. TIA,
> O.K, here are the results:
> I have placed rt_task_delete() at all possible positions in realtimetask().
> The system freezes when I place rt_task_delete() after the printf() statement.
> At all other positions (after system(), after rt_task_sleep()) the system does not freeze.
> I get the very same results when I use rt_task_suspend() instead of rt_task_delete().
>
> It means, that the printf() call causes the freeze.

just to be sure.

Could you try with other secondary domain calls (instead of printf() )
: write() (open/close as I suggested before) or even getpid() ?

(1) instead of the last code line (printf() after rt_task_sleep() ) ;

(2) put it after system(). Does it still prevent a PC from hanging?


-- 
Best regards,
Dmitry Adamushko

Re: [Xenomai-help] Re: A fairly small rtnet/Xenomai application that freezes the

[Gilles Chanteperdrix]

Dmitry Adamushko wrote:
> On 21/12/06, Jan Kiszka <jan.kiszka@domain.hid> wrote:
> 
>>Gilles Chanteperdrix wrote:
> 
> ...
> 
>>>>static void realtimetask(void *arg)
>>>>{
>>>>    system("ls -l");
>>>
>>>If you want to use system (or any function calling fork, eg popen or
>>>vfork) with Xenomai, you have to make sure to fault all pages mapped
>>>with write permission after the fork before trying to use services in
>>>priimary mode, because fork write protects all pages with write
>>>permission and notably the threads stacks. A piece of code that faults
>>>all pages follows.
>>
>>Let me check if I got this correctly: fork calls mark all pages
>>write-protected - also those of the father process, or only those of the
>>new child?
> 
> 
> It's copy-on-write policy. fork() creates a separate address space for
> a child (vm_struct, vm_area(s), all the page tables) but it still
> points to the same physical pages as parent's address space. Now what
> if either a child or parent want to modify writeable pages? That's why
> copy-on-write. A new page will be allocated when a write request is
> commited. And yep, to address this issue, the vm areas of (in fact,
> related areas in page tables) both parent and child are marked as
> copy-on-write.
> 
> But I don't think this is an issue here :
> 
> 1) by the moment system() returns -> the child has finished and there
> is no need to allocate new pages on parent's write requests (not sure
> how linux handles it - have to check);

After the fork and the child has returned, the pages remain write
protected. A fault that would occur after that would do almost nothing,
but still causes a migration to secondary mode.

> 
> 2) (I'd expect logically) the parent did mlockall() in advance for
> even future allocations, so I'd expect copy-on-write is not applicable
> fot it -> all the pages should have been duplicated for a child. Well,
> it's a brief idea. Have to check.

I have checked, mlockall has no effect on copy-on-write.

> 
> so it's likely that it's just a side - effect.
> 
> Mathias: if printf() helps (after system()) so could you try e.g.
> fopen(), fclose() instead? Also, please, do what previously was
> suggested with rt_task_delete(NULL) at the end.
> 
> Another situation would be if one does fork() in real-time app. and
> then both parent and child continue their execution (worth checking).
> 
> p.s. vfork() is a different beast. Here a child is supposed to call
> exec*() rigth after fork() so it's allowed to "borrow" parent's
> address space (vm_struct -> all vm_area(s)) and the parent is blocked
> in the mean time. As a result, no need for copy-on-write.

I think to remember that glibc vfork is just fork.

-- 
                                                 Gilles Chanteperdrix

[Xenomai-help] Re: Re: Re: Re: A fairly small rtnet/Xenomai

[M. Koehrer]

Hi all,

some more interesing measurements:
1) Even when I remove the printf() completely, the system freezes.
My realtimetask is then
static void realtimetask(void *arg)
{
    system("ls -l");
    rt_task_sleep(1000000000ULL);

}

2) When I replace the printf() by a gepid() the behaviour is the same.

3) Jan, your application using the rt_task_shadow behaves the very same (it freezes)!

4) A getpid() directly after system does not help - however, a printf at this position helps to
prevent the freeze.

But now, I found one very interesting thing:
When I rename my /lib/tls to /lib/tls.disabled, it works!!!!
It seems to be (once more) a ugly thread local storage stuff.
The ldd dependency of xeno_crash show libc and libpthread that are take from 
/lib/tls/i686/cmov directory.
Perhaps that could give a hint!

Regards

Mathias

> Hello Philippe,
> >
> > > The important issue to check is the above one. Please try calling
> > > rt_task_delete(NULL), and in a second test, rt_task_suspend(NULL),
> > > instead of letting the thread routine return. TIA,
> > O.K, here are the results:
> > I have placed rt_task_delete() at all possible positions in
> realtimetask().
> > The system freezes when I place rt_task_delete() after the printf()
> statement.
> > At all other positions (after system(), after rt_task_sleep()) the system
> does not freeze.
> > I get the very same results when I use rt_task_suspend() instead of
> rt_task_delete().
> >
> > It means, that the printf() call causes the freeze.
> 
> just to be sure.
> 
> Could you try with other secondary domain calls (instead of printf() )
> : write() (open/close as I suggested before) or even getpid() ?
> 
> (1) instead of the last code line (printf() after rt_task_sleep() ) ;
> 
> (2) put it after system(). Does it still prevent a PC from hanging?
> 
> 
> -- 
> Best regards,
> Dmitry Adamushko
> 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[Xenomai-help] Re: Re: Re: Re: A fairly small rtnet/Xenomai

[Philippe Gerum]

On Thu, 2006-12-21 at 14:36 +0100, M. Koehrer wrote:
> Hi all,
> 
> some more interesing measurements:
> 1) Even when I remove the printf() completely, the system freezes.
> My realtimetask is then
> static void realtimetask(void *arg)
> {
>     system("ls -l");
>     rt_task_sleep(1000000000ULL);
> 
> }
> 
> 2) When I replace the printf() by a gepid() the behaviour is the same.
> 
> 3) Jan, your application using the rt_task_shadow behaves the very same (it freezes)!
> 
> 4) A getpid() directly after system does not help - however, a printf at this position helps to
> prevent the freeze.
> 
> But now, I found one very interesting thing:
> When I rename my /lib/tls to /lib/tls.disabled, it works!!!!
> It seems to be (once more) a ugly thread local storage stuff.
> The ldd dependency of xeno_crash show libc and libpthread that are take from 
> /lib/tls/i686/cmov directory.
> Perhaps that could give a hint!

Please check the following assertions on your setup:

- does enabling the debug option for the Xenomai nucleus cause Xenomai
warnings to appear (messages about forced switches of the crashtest task
to secondary mode), even over 2.6.17 with the very same test code?

- does enabling CONFIG_DEBUG_SPINLOCK and CONFIG_DEBUG_SPINLOCK_SLEEP in
the kernel hacking section cause Linux warnings to appear while the test
code runs over 2.6.19, before the box crashes?

- does the bug still occurs after the call to the system() routine has
been replaced by the following frag?

    if (vfork() == 0)
            execlp("/bin/ls", "ls", "-l", NULL);
    else
            wait(NULL);

-- 
Philippe.

Re: [Xenomai-help] Re: A fairly small rtnet/Xenomai application that freezes the

[Gilles Chanteperdrix]

Gilles Chanteperdrix wrote:
> Dmitry Adamushko wrote:
>>p.s. vfork() is a different beast. Here a child is supposed to call
>>exec*() rigth after fork() so it's allowed to "borrow" parent's
>>address space (vm_struct -> all vm_area(s)) and the parent is blocked
>>in the mean time. As a result, no need for copy-on-write.
> 
> 
> I think to remember that glibc vfork is just fork.

At least, linuxthread's vfork is the same as linuxthread's fork.

-- 
                                                 Gilles Chanteperdrix

[Xenomai-help] Re: Re: Re: Re: Re: A fairly small rtnet/Xenomai

[M. Koehrer]

Hi Philippe,

here are some of the results: 
> Please check the following assertions on your setup:
> 
> - does enabling the debug option for the Xenomai nucleus cause Xenomai
> warnings to appear (messages about forced switches of the crashtest task
> to secondary mode), even over 2.6.17 with the very same test code?
Yes, on 2.6.17 I get the line
Xenomai: Switching crashtest to secondary mode after exception #14 from user-space at 0xb7ff2c7c (pid 1616)
On 2.6.19.1, the system freezes, i.e. I cannot see any message.

> 
> - does enabling CONFIG_DEBUG_SPINLOCK and CONFIG_DEBUG_SPINLOCK_SLEEP in
> the kernel hacking section cause Linux warnings to appear while the test
> code runs over 2.6.19, before the box crashes?
I am about to recompile the kernel, this takes a while...

> - does the bug still occurs after the call to the system() routine has
> been replaced by the following frag?
> 
>     if (vfork() == 0)
>             execlp("/bin/ls", "ls", "-l", NULL);
>     else
>             wait(NULL);
> 
I have replaced the system() call by your code fragement. And yes, this works!
No freeze!

Regards

Mathias

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: [Xenomai-help] Re: A fairly small rtnet/Xenomai application that freezes the

[Dmitry Adamushko]

On 21/12/06, Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> wrote:
> Gilles Chanteperdrix wrote:
> > Dmitry Adamushko wrote:
> >>p.s. vfork() is a different beast. Here a child is supposed to call
> >>exec*() rigth after fork() so it's allowed to "borrow" parent's
> >>address space (vm_struct -> all vm_area(s)) and the parent is blocked
> >>in the mean time. As a result, no need for copy-on-write.
> >
> >
> > I think to remember that glibc vfork is just fork.
>
> At least, linuxthread's vfork is the same as linuxthread's fork.
>

Well, I'm looking at glibc-2.3 :

1) ./sysdeps/unix/sysv/linux/i386/vfork.S

  - if there is a vfork syscall - it's used, otherwise - fork.

  - If per-arch code doesn't provide an implementation, the generic (
== notmal fork() ) is used.

2) linux (I'm looking at 2.6.12.6) does provide a separate vfork()
(sys_vfork => do_fork() with additional flags CLONE_VFORK | CLONE_VM).

And it looks like it works as it should be in theory (at least briefly
following the CLONE_VFORK logic in do_fork() and knowing that CLONE_VM
is used for threads to share a vm_struct).

Anyway, I was refering to vfork() as it should be in theory in the
context of copy-on-write policy.

Indeed, your remarks make sense. We should know when Markus conducts
the tests suggested by Philippe. Although, the use of fork() in
multi-threaded applications is sometimes tricky. I do remember once I
had a problem but just can't recall any details (just that it was
memory leaks :)


> --
>                                                  Gilles Chanteperdrix

-- 
Best regards,
Dmitry Adamushko

Re: [Xenomai-help] Re: Re: Re: Re: Re: A fairly small rtnet/Xenomai

[Dmitry Adamushko]

On 21/12/06, M. Koehrer <mathias_koehrer@domain.hid> wrote:
>
> > - does the bug still occurs after the call to the system() routine has
> > been replaced by the following frag?
> >
> >     if (vfork() == 0)
> >             execlp("/bin/ls", "ls", "-l", NULL);
> >     else
> >             wait(NULL);
> >
> I have replaced the system() call by your code fragement. And yes, this works!
> No freeze!

Ok, could you let me know what happens if you use fork() instead of vfork()?

system() is the same thing : fork -> exec + but it also deals with signals.


-- 
Best regards,
Dmitry Adamushko

[Xenomai-help] Re: Re: Re: Re: Re: A fairly small rtnet/Xenomai

[M. Koehrer]

Hi Philippe,

> - does enabling CONFIG_DEBUG_SPINLOCK and CONFIG_DEBUG_SPINLOCK_SLEEP in
> the kernel hacking section cause Linux warnings to appear while the test
> code runs over 2.6.19, before the box crashes?

Unfortunately, I am not able to get the xeno_nucleus compiled when I have the 
kernel hacking configuration set as proposed. 
The kernel linker complains about an undefined "tasklist_lock" in xeno_nucleus.ko
Thus, I am not able to perform this test.

Regards
Mathias

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[Xenomai-help] Re: Re: A fairly small rtnet/Xenomai...

[M. Koehrer]

Hi Dmitry,

when I use fork() instead of vfork() I have the freeze again. 
I.e. using vfork() it works fine, using fork() freezes the system.

Regards

Mathias
> >
> > > - does the bug still occurs after the call to the system() routine has
> > > been replaced by the following frag?
> > >
> > >     if (vfork() == 0)
> > >             execlp("/bin/ls", "ls", "-l", NULL);
> > >     else
> > >             wait(NULL);
> > >
> > I have replaced the system() call by your code fragement. And yes, this
> works!
> > No freeze!
> 
> Ok, could you let me know what happens if you use fork() instead of
> vfork()?
> 
> system() is the same thing : fork -> exec + but it also deals with signals.
> 


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[Xenomai-help] Re: Re: A fairly small rtnet/Xenomai...

[Dmitry Adamushko]

> when I use fork() instead of vfork() I have the freeze again.
> I.e. using vfork() it works fine, using fork() freezes the system.

Gilles: is this not the end point of the fork vs. vfork epopee ? :)

Anyway, it looks your remarks about fork() make sense. But if it's
true indeed, a more nicer solution would be required.


-- 
Best regards,
Dmitry Adamushko

Re: [Xenomai-help] Re: Re: Re: Re: Re: A fairly small rtnet/Xenomai

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 756 bytes --]

M. Koehrer wrote:
> Hi Philippe,
> 
>> - does enabling CONFIG_DEBUG_SPINLOCK and CONFIG_DEBUG_SPINLOCK_SLEEP in
>> the kernel hacking section cause Linux warnings to appear while the test
>> code runs over 2.6.19, before the box crashes?
> 
> Unfortunately, I am not able to get the xeno_nucleus compiled when I have the 
> kernel hacking configuration set as proposed. 
> The kernel linker complains about an undefined "tasklist_lock" in xeno_nucleus.ko
> Thus, I am not able to perform this test.

Since 2.6.18, tasklist_lock is no longer exported. Either i-pipe has to
take care for this or some other wakeup/locking mechanism in
xnshadow_signal_completion needs to be found. Building the nucleus
non-modular works around this.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

[Xenomai-help] Re: Re: Re: Re: Re: A fairly small rtnet/Xenomai

[Philippe Gerum]

On Thu, 2006-12-21 at 16:19 +0100, M. Koehrer wrote:
> Hi Philippe,
> 
> > - does enabling CONFIG_DEBUG_SPINLOCK and CONFIG_DEBUG_SPINLOCK_SLEEP in
> > the kernel hacking section cause Linux warnings to appear while the test
> > code runs over 2.6.19, before the box crashes?
> 
> Unfortunately, I am not able to get the xeno_nucleus compiled when I have the 
> kernel hacking configuration set as proposed. 
> The kernel linker complains about an undefined "tasklist_lock" in xeno_nucleus.ko
> Thus, I am not able to perform this test.
> 

This patch against the linux tree fixes the issue:

--- arch/i386/kernel/ipipe.c~	2006-12-18 22:03:37.000000000 +0100
+++ arch/i386/kernel/ipipe.c	2006-12-21 17:52:22.000000000 +0100
@@ -901,6 +901,7 @@
 EXPORT_SYMBOL_GPL(show_stack);
 EXPORT_PER_CPU_SYMBOL_GPL(init_tss);
 #ifdef CONFIG_SMP
+EXPORT_SYMBOL(tasklist_lock);
 EXPORT_SYMBOL(__ipipe_logical_cpuid);
 EXPORT_PER_CPU_SYMBOL_GPL(cpu_tlbstate);
 #endif /* CONFIG_SMP */

> Regards
> Mathias
> 
-- 
Philippe.

[Xenomai-help] Re: A fairly small rtnet/Xenomai

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 435 bytes --]

M. Koehrer wrote:
> Hi all,
> 
> some more interesing measurements:
> 1) Even when I remove the printf() completely, the system freezes.
> My realtimetask is then
> static void realtimetask(void *arg)
> {
>     system("ls -l");
>     rt_task_sleep(1000000000ULL);
> 
> }
> 
> 2) When I replace the printf() by a gepid() the behaviour is the same.

Hah, it finally oopses! Hurray!! Now the funny part begins...

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

[Xenomai-help] Re: A fairly small rtnet/Xenomai

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1383 bytes --]

Jan Kiszka wrote:
> M. Koehrer wrote:
>> Hi all,
>>
>> some more interesing measurements:
>> 1) Even when I remove the printf() completely, the system freezes.
>> My realtimetask is then
>> static void realtimetask(void *arg)
>> {
>>     system("ls -l");
>>     rt_task_sleep(1000000000ULL);
>>
>> }
>>
>> 2) When I replace the printf() by a gepid() the behaviour is the same.
> 
> Hah, it finally oopses! Hurray!! Now the funny part begins...
> 

It must be a tight race window. My oops disappears once logging goes to
serial console, so I only caught the following from the stalled console:

BUG: sleeping function called from invalid context at
include/asm/semaphore.h
in_atomic():0, irqs_disabled():1
 [...] lock_kernel+0x28
       default_llseek+0x15
       sys_llseek+0x4e
       sysenter_past_esp+0x65

This all happens shortly after the test case terminated. I'm able scroll
for a short while up in the log, but then the box goes South. With the
serial console attached I only get that trap-#14 reports by Xenomai that
the test program got switched to secondary mode.

I'm not able to continue my test on this particular system for the rest
of the year. I will try to reproduce it again elsewhere (the SPINLOCK
debugging switches seem to reveal something - sometime), but there is no
guarantee that I can do anything about it.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

[Xenomai-help] Re: A fairly small rtnet/Xenomai...

[Gilles Chanteperdrix]

Dmitry Adamushko wrote:
>>when I use fork() instead of vfork() I have the freeze again.
>>I.e. using vfork() it works fine, using fork() freezes the system.
> 
> 
> Gilles: is this not the end point of the fork vs. vfork epopee ? :)

Right. I am currently using linuxthreads pthread implementation, that is
why I had a look at it.

> 
> Anyway, it looks your remarks about fork() make sense. But if it's
> true indeed, a more nicer solution would be required.

The simple solution is to only use fork in non time-critical part of
code (like initializations and cleanup) and to fault the pages after the
fork. It make sense to limit fork use to non time-critical code, I mean,
I would already not recommend to create threads in time-critical code,
creating a process is even worse.

A more complex solution would be to really duplicate the mlocked pages
at fork time.

-- 
                                                 Gilles Chanteperdrix

[Xenomai-help] Re: Re: A fairly small rtnet/Xenomai...

[M. Koehrer]

Good morning everybody,

here are some more news concerning the Xenomai crash issue. 
1) I have tried to enable the CONFIG_DEBUG_SPINLOCK and CONFIG_DEBUG_SPINLOCK_SLEEP
kernel config parameters. But I did not get any messages before the kernel freezes.

2) I have tried to link a static application (using -lstatic). When I pass
the -L/usr/lib/nptl  option to force the linker to use the nptl libs, the effect is the same.
Well, I have now a static binary that freezes the system.
I can mails you the binary file (about 280kByte bz2 file) if you like.
I think this file is too large to be posted to the list.

One additional question comes in my mind:
As I learned from all that things that are related to the issue, it seems to be fairly
critical to fork out of a real time task or to create new processes out of it.
My question is now:
Is it possible to force a real time task back to standard linux task behaviour,
in this state the critical calls can be made. After it a move back to the origin
real time state would be nice. In the native skin there is the rt_task_shadow()
call that allows to move a standard linux task to a real time task.
For moving a real time task to a standard task (and back) something like
rt_task_movetostandard(&state) that saves the real time state in a variable
and a rt_task_movebacktorealtime(&state) would be helpful.

>From the user point all that would be something like a critical region, 
I enter a region where I can do really everything and then I move back
to the real time domain.

Regards

Mathias


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[Xenomai-help] Re: A fairly small rtnet/Xenomai application that freezes the

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
> 
>>M. Koehrer wrote:
>>
>>>Hi Jan, hi everybody,
>>>
>>>I have stripped down my program that is crashing Xenomai even further.
>>>(I have attached the complete source code).
>>>No rtnet is required.
>>>Now I have the following real time task:
>>>
>>>static void realtimetask(void *arg)
>>>{
>>>    system("ls -l");
>>
>>If you want to use system (or any function calling fork, eg popen or
>>vfork) with Xenomai, you have to make sure to fault all pages mapped
>>with write permission after the fork before trying to use services in
>>priimary mode, because fork write protects all pages with write
>>permission and notably the threads stacks. A piece of code that faults
>>all pages follows.
> 
> 
> Let me check if I got this correctly: fork calls mark all pages
> write-protected - also those of the father process, or only those of the
> new child? That would mean we see unexpected page faults, right?
> Anything more (probably not a lock-up)?

Not all page faults are inocuous. I discovered this behaviour of fork
when chasing a lock-up that was caused by an xnshadow_relax that did not
work. It turned out that a page fault occured because of the thread
stack page protection in an xn_copy_to_user, after the calling thread
had been kicked. Trying to relax a kicked thread can not work, and
triggers a nucleus panic.

There may be other pathes where a page fault may be deadly.

Now, in Mathias case, the fact that printf helps is probably the sign
that the deadly fault occurs on the thread stack, because printf touches
the thread stack (remember?, printf segfaults when the thread stack is
too small).

The fact that the bug only occurs with 2.6.19 shows that maybe 2.6.19 is
accessing the thread stack whereas previous versions did not.

> 
> I think we should document this special property somewhere, e.g. in the
> wiki, maybe also a demo program in the examples repos.

I agree.

-- 
                                                 Gilles Chanteperdrix

[Xenomai-help] Re: Re: A fairly small rtnet/Xenomai...

[Dmitry Adamushko]

> One additional question comes in my mind:
> As I learned from all that things that are related to the issue, it seems to be fairly
> critical to fork out of a real time task or to create new processes out of it.
> My question is now:
> Is it possible to force a real time task back to standard linux task behaviour,
> in this state the critical calls can be made. After it a move back to the origin
> real time state would be nice.

A switch is done automatically when you make a call to any
secondary-domain sys call, and the vice-versa. Nevertheless,
rt_task_set_mode() does it excplicitly. Look for the T_PRIMARY flag.

But, well, it's not something that might help you in this particular
case (I mean a problem with fork() :)

So far, I would suggest to use a separate (preliminary) forked context
(and make some communication channel to it from your application) to
send arp requests in your scenario (or anything else that requires
fork()).


-- 
Best regards,
Dmitry Adamushko

[Xenomai-help] Re: A fairly small rtnet/Xenomai...

[Gilles Chanteperdrix]

M. Koehrer wrote:
> Good morning everybody,
> 
> here are some more news concerning the Xenomai crash issue. 
> 1) I have tried to enable the CONFIG_DEBUG_SPINLOCK and CONFIG_DEBUG_SPINLOCK_SLEEP
> kernel config parameters. But I did not get any messages before the kernel freezes.
> 
> 2) I have tried to link a static application (using -lstatic). When I pass
> the -L/usr/lib/nptl  option to force the linker to use the nptl libs, the effect is the same.
> Well, I have now a static binary that freezes the system.
> I can mails you the binary file (about 280kByte bz2 file) if you like.
> I think this file is too large to be posted to the list.
> 
> One additional question comes in my mind:
> As I learned from all that things that are related to the issue, it seems to be fairly
> critical to fork out of a real time task or to create new processes out of it.

In absence of a better solution, I would recommend to use the "fault_vm"
 function after each fork.

-- 
                                                 Gilles Chanteperdrix

[Xenomai-help] Re: Re: A fairly small rtnet/Xenomai...

[M. Koehrer]

Hi Gilles and Dmitry,

I actually prefer Gilles proposal to add the fault_vm() function after calling
a forking function. 
Dmitry's idea with a different context and inter process communication will surely
work but it makes the code less transparent as when using the fault_vm() function.
Are there situations where the fault_vm() could fail or is this fine in all situations?

Regards

Mathias
> M. Koehrer wrote:
> > Good morning everybody,
> > 
> > here are some more news concerning the Xenomai crash issue. 
> > 1) I have tried to enable the CONFIG_DEBUG_SPINLOCK and
> CONFIG_DEBUG_SPINLOCK_SLEEP
> > kernel config parameters. But I did not get any messages before the kernel
> freezes.
> > 
> > 2) I have tried to link a static application (using -lstatic). When I
> pass
> > the -L/usr/lib/nptl  option to force the linker to use the nptl libs, the
> effect is the same.
> > Well, I have now a static binary that freezes the system.
> > I can mails you the binary file (about 280kByte bz2 file) if you like.
> > I think this file is too large to be posted to the list.
> > 
> > One additional question comes in my mind:
> > As I learned from all that things that are related to the issue, it seems
> to be fairly
> > critical to fork out of a real time task or to create new processes out of
> it.
> 
> In absence of a better solution, I would recommend to use the "fault_vm"
>  function after each fork.
> 


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[Xenomai-help] Re: A fairly small rtnet/Xenomai...

[Dmitry Adamushko]

On 22/12/06, Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> wrote:
>
> In absence of a better solution, I would recommend to use the "fault_vm"
>  function after each fork.

But it's not safe. It doesn't stop any other rt threads (if there are
a few in this app.) from touching the wp-pages (it's not only about
stacks after all) in the mean time. So fault_vm() only increases the
probability of not-crashing but doesn't eliminate it completely.

So all the contextes have to be blocked starting from the moment
fork() is about to be called and till the moment a subsequent
fault_vm() is done.
It's ugly and that suggests fork() is not ok here at all.

And btw, vfork() should be a funny thing being called from
multi-threaded app. It blocks a calling context and borrows its
address space while other threads are continuing to run, well, with
the same context (maybe it's handled somehow, have to check).


>
> --
>                                                  Gilles Chanteperdrix
>

-- 
Best regards,
Dmitry Adamushko

[Xenomai-help] Re: A fairly small rtnet/Xenomai...

[Gilles Chanteperdrix]

Dmitry Adamushko wrote:
> On 22/12/06, Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> wrote:
> 
>>In absence of a better solution, I would recommend to use the "fault_vm"
>> function after each fork.
> 
> 
> But it's not safe. It doesn't stop any other rt threads (if there are
> a few in this app.) from touching the wp-pages (it's not only about
> stacks after all) in the mean time. So fault_vm() only increases the
> probability of not-crashing but doesn't eliminate it completely.
> 
> So all the contextes have to be blocked starting from the moment
> fork() is about to be called and till the moment a subsequent
> fault_vm() is done.
> It's ugly and that suggests fork() is not ok here at all.
> 
> And btw, vfork() should be a funny thing being called from
> multi-threaded app. It blocks a calling context and borrows its
> address space while other threads are continuing to run, well, with
> the same context (maybe it's handled somehow, have to check).

fault_vm is safe to use only if you are calling fork at a time when
there is only one thread. So, if your application is forking at init, it
should be OK.

-- 
                                                 Gilles Chanteperdrix

[Xenomai-help] Re: Re: A fairly small rtnet/Xenomai...

[M. Koehrer]

Hi Gilles,

> fault_vm is safe to use only if you are calling fork at a time when
> there is only one thread. So, if your application is forking at init, it
> should be OK.
Do you mean there must be only one real time thread? 
That means, when I have an application that creates multiple real time threads,
I can not rely on fault_vm() ?
In this case I have to do the "hard" way by using a different (non real time)
context to do the forks and system() calls.
As this is hard to understand, I strongly recommend that there is Xenomai support
for this! I.e. a Xenomai API that can be called with a (callback-)function pointer and
a user data pointer. 
When a (realtime) thread calls this function, the real time thread is blocked. 
The callback function is then called from a safe context and
after exit of the callback function the real time thread is resumed.

Regards

Mathias




-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[Xenomai-help] Re: Re: A fairly small rtnet/Xenomai...

[Dmitry Adamushko]

On 22/12/06, M. Koehrer <mathias_koehrer@domain.hid> wrote:
> Hi Gilles,
>
> > fault_vm is safe to use only if you are calling fork at a time when
> > there is only one thread. So, if your application is forking at init, it
> > should be OK.
> Do you mean there must be only one real time thread?
> That means, when I have an application that creates multiple real time threads,
> I can not rely on fault_vm() ?

Yes, read my previous mail.

> In this case I have to do the "hard" way by using a different (non real time)
> context to do the forks and system() calls.

That's much safer indeed. Both in respect of this particular problem
and others that may be encountered using fork() from a multithreaded
application.

e.g. look here

http://www.gnu.org/software/libc/manual/html_node/Threads-and-Fork.html#Threads%20and%20Fork

> As this is hard to understand, I strongly recommend that there is Xenomai support
> for this! I.e. a Xenomai API that can be called with a (callback-)function pointer and
> a user data pointer.
> When a (realtime) thread calls this function, the real time thread is blocked.
> The callback function is then called from a safe context and
> after exit of the callback function the real time thread is resumed.

I haven't got your idea. Did you get the cause of the problem with fork() ?

>
> Regards
>
> Mathias

-- 
Best regards,
Dmitry Adamushko

Re: [Xenomai-help] Re: A fairly small rtnet/Xenomai...

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1387 bytes --]

Gilles Chanteperdrix wrote:
> Dmitry Adamushko wrote:
>> On 22/12/06, Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> wrote:
>>
>>> In absence of a better solution, I would recommend to use the "fault_vm"
>>> function after each fork.
>>
>> But it's not safe. It doesn't stop any other rt threads (if there are
>> a few in this app.) from touching the wp-pages (it's not only about
>> stacks after all) in the mean time. So fault_vm() only increases the
>> probability of not-crashing but doesn't eliminate it completely.
>>
>> So all the contextes have to be blocked starting from the moment
>> fork() is about to be called and till the moment a subsequent
>> fault_vm() is done.
>> It's ugly and that suggests fork() is not ok here at all.
>>
>> And btw, vfork() should be a funny thing being called from
>> multi-threaded app. It blocks a calling context and borrows its
>> address space while other threads are continuing to run, well, with
>> the same context (maybe it's handled somehow, have to check).
> 
> fault_vm is safe to use only if you are calling fork at a time when
> there is only one thread. So, if your application is forking at init, it
> should be OK.
> 

Isn't there something to do about it at kernel level, i.e. on cloning
the mm-context of a process? Could we somehow force an immediate copy of
a real-time process's mm?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]

[Xenomai-help] Re: Re: A fairly small rtnet/Xenomai...

[Philippe Gerum]

On Fri, 2006-12-22 at 11:27 +0100, M. Koehrer wrote:
> Hi Gilles,
> 
> > fault_vm is safe to use only if you are calling fork at a time when
> > there is only one thread. So, if your application is forking at init, it
> > should be OK.
> Do you mean there must be only one real time thread? 
> That means, when I have an application that creates multiple real time threads,
> I can not rely on fault_vm() ?
> In this case I have to do the "hard" way by using a different (non real time)
> context to do the forks and system() calls.
> As this is hard to understand, I strongly recommend that there is Xenomai support
> for this! I.e. a Xenomai API that can be called with a (callback-)function pointer and
> a user data pointer. 
> When a (realtime) thread calls this function, the real time thread is blocked. 
> The callback function is then called from a safe context and
> after exit of the callback function the real time thread is resumed.
> 

Sorry, but no, no way, I won't merge anything like this, ever. This is
the wrong way to go. The right way is to fix the COW issue at kernel
level - probably the I-pipe has to provide the required support -
because this is where those dirty details belong to. This is definitely
not an API issue, because you just cannot tell application developers to
care about arch-specific VM issues when using a so-called generic API
that has to work the same way on all archs (e.g. MMU-less platforms
don't care about this, others would). What could be considered as a
bearable limitation right now must not have any impact on long-term
principles, and the API stuff belongs to that category of issues.

> Regards
> 
> Mathias
> 
> 
> 
> 
-- 
Philippe.

[Xenomai-help] Re: Re: Re: A fairly small rtnet/Xenomai...

[M. Koehrer]

Hi Philippe, 

I agree. To fix the root cause is actually the very best to do!
This eases the life of users and developers.

Regards

Mathias

> 
> Sorry, but no, no way, I won't merge anything like this, ever. This is
> the wrong way to go. The right way is to fix the COW issue at kernel
> level - probably the I-pipe has to provide the required support -
> because this is where those dirty details belong to. This is definitely
> not an API issue, because you just cannot tell application developers to
> care about arch-specific VM issues when using a so-called generic API
> that has to work the same way on all archs (e.g. MMU-less platforms
> don't care about this, others would). What could be considered as a
> bearable limitation right now must not have any impact on long-term
> principles, and the API stuff belongs to that category of issues.


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[Xenomai-help] Re: Re: Re: A fairly small rtnet/Xenomai...

[Philippe Gerum]

On Fri, 2006-12-22 at 12:40 +0100, M. Koehrer wrote:
> Hi Philippe, 
> 
> I agree. To fix the root cause is actually the very best to do!
> This eases the life of users and developers.

Definitely, yes. The point is that once you go down the
"rt_task_system()" path, you end up being trapped into API
proliferation, which would continue with rt_task_fork, rt_task_vfork and
so on. This would a dead end, unfortunately.

Asking people to use rt_task_system() is already asking them to
understand why they should not use system() in the first place, so this
can't solve the root issue. The problem comes entirely from the fact
that we don't expect any more faults after mlockall, and COW proved us
wrong. The fact that the machine freezes is only a side-effect due to
the co-kernel constraints, the first and foremost problem is that
gracefully handling a page fault would induce uncontrolable latencies
anyway.

IOW, the problem has to be fixed at kernel level, because it's
fundamentally an arch-dependent core issue.

> 
> Regards
> 
> Mathias
> 
> > 
> > Sorry, but no, no way, I won't merge anything like this, ever. This is
> > the wrong way to go. The right way is to fix the COW issue at kernel
> > level - probably the I-pipe has to provide the required support -
> > because this is where those dirty details belong to. This is definitely
> > not an API issue, because you just cannot tell application developers to
> > care about arch-specific VM issues when using a so-called generic API
> > that has to work the same way on all archs (e.g. MMU-less platforms
> > don't care about this, others would). What could be considered as a
> > bearable limitation right now must not have any impact on long-term
> > principles, and the API stuff belongs to that category of issues.
> 
> 
-- 
Philippe.

[Xenomai-help] Re: A fairly small rtnet/Xenomai...

[Gilles Chanteperdrix]

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

M. Koehrer wrote:
> Hi Philippe, 
> 
> I agree. To fix the root cause is actually the very best to do!
> This eases the life of users and developers.
> 
> Regards
> 
> Mathias

Hi Mathias,

here comes a workaround for the COW issue on Linux 2.6.19. The patch
relies on a new VM_NOCOW flag which should be set for real-time
application if you use Xenomai trunk.

It would be nice if you could test it.

Thanks in advance.

-- 
                                                 Gilles Chanteperdrix


[-- Attachment #2: vm-nocow-2.6.19.patch --]
[-- Type: text/x-patch, Size: 4658 bytes --]

diff -x '*~' -Naurdp linux-2.6.19/include/linux/mm.h linux-2.6.19-nocow/include/linux/mm.h
--- linux-2.6.19/include/linux/mm.h	2006-11-30 11:04:08.000000000 +0100
+++ linux-2.6.19-nocow/include/linux/mm.h	2006-12-27 10:03:09.000000000 +0100
@@ -166,6 +166,7 @@ extern unsigned int kobjsize(const void 
 #define VM_NONLINEAR	0x00800000	/* Is non-linear (remap_file_pages) */
 #define VM_MAPPED_COPY	0x01000000	/* T if mapped copy of data (nommu mmap) */
 #define VM_INSERTPAGE	0x02000000	/* The vma has had "vm_insert_page()" done on it */
+#define VM_NOCOW	0x10000000	/* Disable COW mapping for the vma */
 
 #ifndef VM_STACK_DEFAULT_FLAGS		/* arch can override this */
 #define VM_STACK_DEFAULT_FLAGS VM_DATA_DEFAULT_FLAGS
diff -x '*~' -Naurdp linux-2.6.19/mm/memory.c linux-2.6.19-nocow/mm/memory.c
--- linux-2.6.19/mm/memory.c	2006-11-30 11:04:14.000000000 +0100
+++ linux-2.6.19-nocow/mm/memory.c	2006-12-27 10:06:37.000000000 +0100
@@ -418,13 +418,41 @@ struct page *vm_normal_page(struct vm_ar
 	return pfn_to_page(pfn);
 }
 
+static inline void cow_user_page(struct page *dst, struct page *src, unsigned long va)
+{
+	/*
+	 * If the source page was a PFN mapping, we don't have
+	 * a "struct page" for it. We do a best-effort copy by
+	 * just copying from the original user address. If that
+	 * fails, we just zero-fill it. Live with it.
+	 */
+	if (unlikely(!src)) {
+		void *kaddr = kmap_atomic(dst, KM_USER0);
+		void __user *uaddr = (void __user *)(va & PAGE_MASK);
+
+		/*
+		 * This really shouldn't fail, because the page is there
+		 * in the page tables. But it might just be unreadable,
+		 * in which case we just give up and fill the result with
+		 * zeroes.
+		 */
+		if (__copy_from_user_inatomic(kaddr, uaddr, PAGE_SIZE))
+			memset(kaddr, 0, PAGE_SIZE);
+		kunmap_atomic(kaddr, KM_USER0);
+		flush_dcache_page(dst);
+		return;
+		
+	}
+	copy_user_highpage(dst, src, va);
+}
+
 /*
  * copy one vm_area from one task to the other. Assumes the page tables
  * already present in the new task to be cleared in the whole range
  * covered by this vma.
  */
 
-static inline void
+static inline int
 copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 		pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *vma,
 		unsigned long addr, int *rss)
@@ -466,6 +494,25 @@ copy_one_pte(struct mm_struct *dst_mm, s
 	 * in the parent and the child
 	 */
 	if (is_cow_mapping(vm_flags)) {
+#ifdef CONFIG_IPIPE
+		if (((vm_flags|src_mm->def_flags) & (VM_LOCKED|VM_NOCOW)) == (VM_LOCKED|VM_NOCOW)) {
+			struct page *old_page = vm_normal_page(vma, addr, pte);
+			page = alloc_page_vma(GFP_HIGHUSER, vma, addr);
+			if (!page)
+				return -ENOMEM;
+
+			cow_user_page(page, old_page, addr);
+			pte = mk_pte(page, vma->vm_page_prot);
+			
+			if (vm_flags & VM_SHARED)
+				pte = pte_mkclean(pte);
+			pte = pte_mkold(pte);
+
+			page_dup_rmap(page);
+			rss[!!PageAnon(page)]++;
+			goto out_set_pte;
+		}
+#endif /* CONFIG_IPIPE */
 		ptep_set_wrprotect(src_mm, addr, src_pte);
 		pte = pte_wrprotect(pte);
 	}
@@ -487,6 +534,7 @@ copy_one_pte(struct mm_struct *dst_mm, s
 
 out_set_pte:
 	set_pte_at(dst_mm, addr, dst_pte, pte);
+	return 0;
 }
 
 static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,
@@ -524,7 +572,9 @@ again:
 			progress++;
 			continue;
 		}
-		copy_one_pte(dst_mm, src_mm, dst_pte, src_pte, vma, addr, rss);
+		if (copy_one_pte(dst_mm, src_mm, dst_pte,
+				 src_pte, vma, addr, rss))
+			return -ENOMEM;
 		progress += 8;
 	} while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr != end);
 
@@ -1431,34 +1481,6 @@ static inline pte_t maybe_mkwrite(pte_t 
 	return pte;
 }
 
-static inline void cow_user_page(struct page *dst, struct page *src, unsigned long va)
-{
-	/*
-	 * If the source page was a PFN mapping, we don't have
-	 * a "struct page" for it. We do a best-effort copy by
-	 * just copying from the original user address. If that
-	 * fails, we just zero-fill it. Live with it.
-	 */
-	if (unlikely(!src)) {
-		void *kaddr = kmap_atomic(dst, KM_USER0);
-		void __user *uaddr = (void __user *)(va & PAGE_MASK);
-
-		/*
-		 * This really shouldn't fail, because the page is there
-		 * in the page tables. But it might just be unreadable,
-		 * in which case we just give up and fill the result with
-		 * zeroes.
-		 */
-		if (__copy_from_user_inatomic(kaddr, uaddr, PAGE_SIZE))
-			memset(kaddr, 0, PAGE_SIZE);
-		kunmap_atomic(kaddr, KM_USER0);
-		flush_dcache_page(dst);
-		return;
-		
-	}
-	copy_user_highpage(dst, src, va);
-}
-
 /*
  * This routine handles present pages, when users try to write
  * to a shared page. It is done by copying the page to a new address

[Xenomai-help] Aw: Re: A fairly small rtnet/Xenomai...

[M. Koehrer]

HI Gilles,

thanks for the patch.
However, as I am not in the office until 2nd of january, I will do the tests early next year.

Regards

Mathias

> M. Koehrer wrote:
> > Hi Philippe, 
> > 
> > I agree. To fix the root cause is actually the very best to do!
> > This eases the life of users and developers.
> > 
> > Regards
> > 
> > Mathias
> 
> Hi Mathias,
> 
> here comes a workaround for the COW issue on Linux 2.6.19. The patch
> relies on a new VM_NOCOW flag which should be set for real-time
> application if you use Xenomai trunk.
> 
> It would be nice if you could test it.
> 
> Thanks in advance.
> 
> -- 
>                                                  Gilles Chanteperdrix
> 
> 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: [Xenomai-help] Re: A fairly small rtnet/Xenomai...

[Philippe Gerum]

On Wed, 2006-12-27 at 10:25 +0100, Gilles Chanteperdrix wrote:
> M. Koehrer wrote:
> > Hi Philippe, 
> > 
> > I agree. To fix the root cause is actually the very best to do!
> > This eases the life of users and developers.
> > 
> > Regards
> > 
> > Mathias
> 
> Hi Mathias,
> 
> here comes a workaround for the COW issue on Linux 2.6.19. The patch
> relies on a new VM_NOCOW flag which should be set for real-time
> application if you use Xenomai trunk.
> 
> It would be nice if you could test it.
> 

You will additionally need to apply this patch to the Xenomai tree in
order to activate the COW-disable feature:

--- ksrc/nucleus/shadow.c.orig	2006-12-27 11:42:50.000000000 +0100
+++ ksrc/nucleus/shadow.c	2006-12-27 11:42:55.000000000 +0100
@@ -839,6 +839,8 @@
 #ifdef CONFIG_MMU
 	if (!(current->mm->def_flags & VM_LOCKED))
 		send_sig(SIGXCPU, current, 1);
+	else
+	    current->mm->def_flags |= VM_NOCOW;
 #endif /* CONFIG_MMU */
 
 	current->cap_effective |=

-- 

Philippe.

[Xenomai-help] COW-disable patch

[Philippe Gerum]

On Wed, 2006-12-27 at 10:25 +0100, Gilles Chanteperdrix wrote:
> M. Koehrer wrote:
> > Hi Philippe, 
> > 
> > I agree. To fix the root cause is actually the very best to do!
> > This eases the life of users and developers.
> > 
> > Regards
> > 
> > Mathias
> 
> Hi Mathias,
> 
> here comes a workaround for the COW issue on Linux 2.6.19. The patch
> relies on a new VM_NOCOW flag which should be set for real-time
> application if you use Xenomai trunk.
> 
> It would be nice if you could test it.
> 

Here is the same patch backported to 2.6.14 for the ppc people who
experienced the same kind of issues. When enabling the nucleus debug
option (full, with queue checks), no more warning should appear in the
kernel log about threads being switched to secondary mode while the
latency test runs. The last patch is to be applied against a Xenomai
2.3.x tree to activate the feature. Feedback highly recommended if you
want this patch to be merged in. TIA,

--- 2.6.14/include/linux/mm.h	2005-10-28 02:02:08.000000000 +0200
+++ 2.6.14-ipipe/include/linux/mm.h	2006-12-26 19:38:10.000000000 +0100
@@ -162,6 +162,7 @@
 #define VM_HUGETLB	0x00400000	/* Huge TLB Page VM */
 #define VM_NONLINEAR	0x00800000	/* Is non-linear (remap_file_pages) */
 #define VM_MAPPED_COPY	0x01000000	/* T if mapped copy of data (nommu mmap) */
+#define VM_NOCOW	0x10000000	/* Disable COW mapping for the vma */
 
 #ifndef VM_STACK_DEFAULT_FLAGS		/* arch can override this */
 #define VM_STACK_DEFAULT_FLAGS VM_DATA_DEFAULT_FLAGS
--- 2.6.14/mm/memory.c	2006-12-26 19:46:40.000000000 +0100
+++ 2.6.14-ipipe/mm/memory.c	2006-12-26 19:46:54.000000000 +0100
@@ -341,10 +341,10 @@
  * but may be dropped within p[mg]d_alloc() and pte_alloc_map().
  */
 
-static inline void
+static inline int
 copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
-		pte_t *dst_pte, pte_t *src_pte, unsigned long vm_flags,
-		unsigned long addr)
+	     pte_t *dst_pte, pte_t *src_pte, unsigned long vm_flags,
+	     unsigned long addr, struct vm_area_struct *vma)
 {
 	pte_t pte = *src_pte;
 	struct page *page;
@@ -362,7 +362,7 @@
 			}
 		}
 		set_pte_at(dst_mm, addr, dst_pte, pte);
-		return;
+		return 0;
 	}
 
 	pfn = pte_pfn(pte);
@@ -377,7 +377,7 @@
 
 	if (!page || PageReserved(page)) {
 		set_pte_at(dst_mm, addr, dst_pte, pte);
-		return;
+		return 0;
 	}
 
 	/*
@@ -385,6 +385,27 @@
 	 * in the parent and the child
 	 */
 	if ((vm_flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE) {
+#ifdef CONFIG_IPIPE
+		if (((vm_flags|src_mm->def_flags) & (VM_NOCOW|VM_LOCKED)) == (VM_NOCOW|VM_LOCKED)) {
+			struct page *old_page = page;
+			page = alloc_page_vma(GFP_HIGHUSER, vma, addr);
+			if (!page)
+				return -ENOMEM;
+
+			copy_user_highpage(page, old_page, addr);
+			pte = mk_pte(page, vma->vm_page_prot);
+
+			if (vm_flags & VM_SHARED)
+			    pte = pte_mkclean(pte);
+			pte = pte_mkold(pte);
+			inc_mm_counter(dst_mm, rss);
+			if (PageAnon(page))
+				inc_mm_counter(dst_mm, anon_rss);
+			set_pte_at(dst_mm, addr, dst_pte, pte);
+			page_dup_rmap(page);
+			return 0;
+		}
+#endif /* CONFIG_IPIPE */
 		ptep_set_wrprotect(src_mm, addr, src_pte);
 		pte = *src_pte;
 	}
@@ -402,6 +423,8 @@
 		inc_mm_counter(dst_mm, anon_rss);
 	set_pte_at(dst_mm, addr, dst_pte, pte);
 	page_dup_rmap(page);
+
+	return 0;
 }
 
 static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,
@@ -433,7 +456,8 @@
 			progress++;
 			continue;
 		}
-		copy_one_pte(dst_mm, src_mm, dst_pte, src_pte, vm_flags, addr);
+		if (copy_one_pte(dst_mm, src_mm, dst_pte, src_pte, vm_flags, addr, vma))
+			return -ENOMEM;
 		progress += 8;
 	} while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr != end);
 	spin_unlock(&src_mm->page_table_lock);

--- ksrc/nucleus/shadow.c.orig  2006-12-27 11:42:50.000000000 +0100
+++ ksrc/nucleus/shadow.c       2006-12-27 11:42:55.000000000 +0100
@@ -839,6 +839,8 @@
 #ifdef CONFIG_MMU
        if (!(current->mm->def_flags & VM_LOCKED))
                send_sig(SIGXCPU, current, 1);
+       else
+           current->mm->def_flags |= VM_NOCOW;
 #endif /* CONFIG_MMU */
 
        current->cap_effective |=

-- 
Philippe.

Re: Re: [Xenomai-help] Re: A fairly small rtnet/Xenomai...

[M. Koehrer]

Happy new year to all!

Back in the office, I tried out the patches from Chilles and Philippe.
And yes, it is working now! No crash!
Using these patches, I can now call a system() call out of a rea ltime task that is followed 
by a rt_task_sleep() and a printf() without crash on a 2.6.19 kernel.

Thanks for the support!

Best regards

Mathias
> > here comes a workaround for the COW issue on Linux 2.6.19. The patch
> > relies on a new VM_NOCOW flag which should be set for real-time
> > application if you use Xenomai trunk.
> > 
> > It would be nice if you could test it.
> > 
> 
> You will additionally need to apply this patch to the Xenomai tree in
> order to activate the COW-disable feature:


-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: Re: [Xenomai-help] Re: A fairly small rtnet/Xenomai...

[Philippe Gerum]

On Tue, 2007-01-02 at 09:23 +0100, M. Koehrer wrote:
> Happy new year to all!
> 
> Back in the office, I tried out the patches from Chilles and Philippe.
> And yes, it is working now! No crash!
> Using these patches, I can now call a system() call out of a rea ltime task that is followed 
> by a rt_task_sleep() and a printf() without crash on a 2.6.19 kernel.
> 
> Thanks for the support!
> 
> Best regards
> 
> Mathias
> > > here comes a workaround for the COW issue on Linux 2.6.19. The patch
> > > relies on a new VM_NOCOW flag which should be set for real-time
> > > application if you use Xenomai trunk.
> > > 
> > > It would be nice if you could test it.
> > > 
> > 
> > You will additionally need to apply this patch to the Xenomai tree in
> > order to activate the COW-disable feature:
> 
> 

Great, thanks. Niklaus, could you check the vm-nocow patch for 2.6.14 on
the ppc board (the hcu, IIRC) that exhibited strange faults in
user-space while running the latency test, especially when enabling the
nucleus debug option (the one which also used to cause secondary switch
warnings being sent by Xenomai to the kernel log)? TIA,

-- 
Philippe.

Re: [Xenomai-help] Re: A fairly small rtnet/Xenomai...

[Niklaus Giger]

Am Dienstag, 2. Januar 2007 10:53 schrieb Philippe Gerum:
> On Tue, 2007-01-02 at 09:23 +0100, M. Koehrer wrote:
> > Happy new year to all!
> >
> > Back in the office, I tried out the patches from Chilles and Philippe.
> > And yes, it is working now! No crash!
> > Using these patches, I can now call a system() call out of a rea ltime
> > task that is followed by a rt_task_sleep() and a printf() without crash
> > on a 2.6.19 kernel.
> >
> > Thanks for the support!
> >
> > Best regards
> >
> > Mathias
> >
> > > > here comes a workaround for the COW issue on Linux 2.6.19. The patch
> > > > relies on a new VM_NOCOW flag which should be set for real-time
> > > > application if you use Xenomai trunk.
> > > >
> > > > It would be nice if you could test it.
> > >
> > > You will additionally need to apply this patch to the Xenomai tree in
> > > order to activate the COW-disable feature:
>
> Great, thanks. Niklaus, could you check the vm-nocow patch for 2.6.14 on
> the ppc board (the hcu, IIRC) that exhibited strange faults in
> user-space while running the latency test, especially when enabling the
> nucleus debug option (the one which also used to cause secondary switch
> warnings being sent by Xenomai to the kernel log)? TIA,
I will check it, but I must first upgrade my board specific patch from Linux 
2.6.17 to 2.6.19. Don't know how much time it will take me.

Best regards
-- 
Niklaus Giger

Re: [Xenomai-help] Re: A fairly small rtnet/Xenomai...

[Niklaus Giger]

[-- Attachment #1: Type: text/plain, Size: 1525 bytes --]

Am Dienstag, 2. Januar 2007 10:53 schrieb Philippe Gerum:
> On Tue, 2007-01-02 at 09:23 +0100, M. Koehrer wrote:
> > Happy new year to all!
> >
> > Back in the office, I tried out the patches from Chilles and Philippe.
> > And yes, it is working now! No crash!
> > Using these patches, I can now call a system() call out of a rea ltime
> > task that is followed by a rt_task_sleep() and a printf() without crash
> > on a 2.6.19 kernel.
> >
> > Thanks for the support!
> >
> > Best regards
> >
> > Mathias
> >
> > > > here comes a workaround for the COW issue on Linux 2.6.19. The patch
> > > > relies on a new VM_NOCOW flag which should be set for real-time
> > > > application if you use Xenomai trunk.
> > > >
> > > > It would be nice if you could test it.
> > >
> > > You will additionally need to apply this patch to the Xenomai tree in
> > > order to activate the COW-disable feature:
>
> Great, thanks. Niklaus, could you check the vm-nocow patch for 2.6.14 on
> the ppc board (the hcu, IIRC) that exhibited strange faults in
> user-space while running the latency test, especially when enabling the
> nucleus debug option (the one which also used to cause secondary switch
> warnings being sent by Xenomai to the kernel log)? TIA,
Here the results. I had no problems running xeno-test.

I did set in my .config
CONFIG_XENO_OPT_DEBUG=y
CONFIG_XENO_OPT_DEBUG_NUCLEUS=y

Attached is my .config and the output of xeno-test.

If you need the results of another combination just let me know.

Best regards
-- 
Niklaus Giger

[-- Attachment #2: .config.zip --]
[-- Type: application/x-zip, Size: 6049 bytes --]

PK\x03\x04\x14\0\0\0\b\0·­$6i\x7fÍü(\x17\0\0U\0\0\a\0\0\0.config•<is㸎ßçW¨z¶jº«&\x13Û¹§6[EI”Ͷ®\x16);™/*·£$Þvì<\x1f3\x7f¿ )Y\x17hϾz=±\bð\x02\x01\x10\0Aþú˯\x16ÙïÖo³Ýb>[.?¬—|•of»üÉz›ýÈ­ùzõ¼xùÓzZ¯~ÛYùÓb\a5üÅjÿÓú‘oVùÒú;ßl\x17ëÕŸÖàë?úw\x7fô\x01A¼î­¯³•e]ZƒþŸƒ»?{\x17Ö ×»ùå×_œ(ôØ0\v‚ôþ£ü\x18Ґ&ÌÉF$qYòWdÊi=8£!qÝŒøÃ(ab\x14 U§”\rG¢\vpˆÏì„\bš¹Ô'\x15B\x1c;‹A·r8I\bÒ™ÇB7\véƒÈlVë’;#
å‘ü\x7f\x140‘yP›fqÄBA“
$Î(\vÈ#Ìw\x02`hÏ­\rÅ¥^ñËg\Ü\x7f:_.¾Ÿ¿­ŸöË|{þ_i(›L¨O	§ç\x7fèåùô\v\x10öWËY?å°j»ýf±û°–ùß°:ë÷\x1d,ζ"<}ˆa\x0e\x01\r\x05ñ«Ní$\x1aÓ0‹ÂŒ\aqUÌB˜\x05\r'0d9\x1c˜ÓýÅ@w6T|²´¶ùnÿ^5ïG@ð	M8‹ÂûOŸ°âŒ¤"‚>`À\x05ݦ$¶\x16[kµÞÉæ\x0eä|ä\x13¦\x16é€	ŸYÈë¸\aP\x1cqö\x05ßRšR\x14Áæn\x16'‘C9ψã\b\x14I\x10>æ‚\b¼TpS÷$u\x19Þ$^[ë\x1f(0‘<‰Ì]\x12\x1e˜Çã\x19ÒÄ¡’”‡JŽ“E±€Õø‹f^”d\x1c~àôsDmi`S×¥na\x15†úCü\x1aæ\x18¾øcÀë¤/Ë@\b}t*\a\x04‹„d1á8¥F‘ˆýtˆ\f8N@PÆ5¦L‡µÑû^æD	­A\x022/­\x0fÜK\x05}¨šÆQs¸åÌG\x01­	6÷‰]}M‚ŒN@@ ¿TJn]\x1f‰,}Ð\x06m\x04\v\x1fu“HWj˜<\x04îÕxޏì:²’)\x7f={š}_‚\x10+q·¶û÷÷õfWÓš‘›ú´6\x1e]¥¡\x1f\x11·>¤\x02\0ìá”`t5\0¯\x10K|µŠvxâ\x1c¤\x17%è\x18\x10¡\x7f5\r{¹žÿ°–³|S\rÝ\x06\x150®Ð·ñ\x11Ùþ\x18tà$cQ\x06|äàÂìs¯C=\x16Y|þšKÊmj
E¥^Žêš­(%¼[æRâú,¤]ˆã}«O\x01T5I}\x01 ƒ,Áe{\bÙJ\x14CÃrÌGj\x15úÿ4\x7fþO±\a$ùly¶[¼Iæù~¶ýØîò·Š\x16\x0f4Œ\x02ª‰É\x02©L²0u|šr\x04\x02ÛÅ„p6¡õ\x01\x1e Œ\x18õñ•< %1s\x19'¶‘à€Å©“Âîþ(µ3(il(,¦’Ùˆ\x12×\07\x14ga\x02L%w¯Îàè\x10öÙä\x11©X‚ 2È«à÷Wýn}Оr<1ÿë¾?¸í‚ån‚´íÒ†zk–×–£KM¡6:~ŒžS"œ‘^[\rK±\x14‹·ÅêE[
-T¹™„C¹Ô,r™s¬Õ\x06*h4\x03\x1cþÐÄ'ÂŒ¡XWc¨1q°?gß\x17K°]š\f«É\b;”d ]\r¡›êP›L\x18Ñ4X®ÓqVÕxÓ\x11¥~GÁð×Ù\x06ìâÅj—o6û÷Ý\x16¥%\bDò-óa\aÁ÷É\x16"u‡´ÓS0›¿.V9üÖšMöø<›çÛ\x16eø˜…YH„–ÎŽL+€I*jPØi=`ßËÞݵ	툔\x14\x18¼¾¡¶j?5ƒƒbß6€¡\x18[ò\x02ª$Á\f–kn†\x02W%GÆ\x1dĘì\x16@°V’\x16T-ˆ2E1=¤LT>ÂúÓ0¤ÅJâËf›œ¤{ä\x11¾\x05U()\x18”Qx\x02i’ˆ\a“ðk„‡i”Œ»zI\x01\x13AØÑÚ‰p±¹Ëbð©|\x17,^[L…êé\x17••@¸›Åßz“ׂ™o\x16àŠ<\x1dJë-¸	“¦KÖ¿¾ºê‘û·B\x19æÛ\x1dhÃ\x13u”6°i茠\x1e†\0\x12|\f̧\ft°  ”Š~\x1d\x12žè\x130\f­\x01ä°m4É_Ç\x10\x0fàmE±Mœ±i1êèÉ\x03\b¿r!îû½Á¥\x11/ \x0fÒ,c°5ß\x1fÇJ¨CUÉ}ÿÚˆ8a‰¤‰y\x1aü+é÷z½ŽjŒ7kЃÛõ¦®|¯\x7fþDæzÙûY§ÔååO”ýo\x7fâåtÐì¾\x02\\x19\0\x01\x11#p³RØÕ˜AÖÆà;8(ĉÓÌKè7l"?^[\x13™\x12ðM(I|p:`^[&ݝŠÙ\x05•jÞ\x7fe^§6ì¾Ä0\x04‡]ö®ðIÇ-HéÎ9éE}p	u§Qäfx+%ô\x1a…òG‡\x04àã¡À)ñÃ\x14w±\x1f\x18X…àüû\x17†…9 \ö.0oÛ\x0e²È©í\x120ÕaœÔ§\x15Ç\x0e\x104s\x03‚Ô\a Œ\r~©šHI"z™\x10¼Wo§*î#\r…‘t:G4QÞ'\x01S§Ôxñr¶{^oÞ°%…ÞÇôÑŽHâb“+öÿÊ\x01gCƒ«ª\x02Sq\x14K\x06¦\x1cFãÂ\x0e¨vϪò_Y¿Iä
0¸êuQqÜ{À­Ü~J\x03¹«Facœeù$òÁ\r'É#º´\x05\x16æãSŸ:à¯S`©GéÈÒz¤\0¦\b\x10˜^˜ª@Xå{1\x0e¿\x04^[V`œYc’€Íƒ"5;iöZUÕk¶jPú\f¨\x15ÎcŸ­ ¤/-\x15†TÅ5ÙÒ±"ž]_Ú\f£\bH¾\x17\b\x19J©÷[”\x060o¤Ž\x13(ß5³£¨F¾¢ôþÓâý~½ªÅ¦bŒ¯À³s¢F\‘\x13)J\x19‰Ù!h°ß¢¼\rÌÍn\aWwøâ;ÌTž¹ÒÓmF5´$9Ž\x14”ϱ\x138Œœ;³Í\x13ôý¥\x16lQhäéïÙj\x0eæ~\x19à¬B}î„„\x0euU\x10.BzpóçÙ~©
¤±µ@p­YÙž\x0eÙî739×rÊ\x16x\x16Ö~›?UFA!¦’\x19`«ìý|Î{ê\x7fUHuªÀrçîý¼è5¡cš„Ô?Tž·À2ÎYV½mÁà\x0f\a[_ª CßÏ}#Šn¤×^[40€]D¦‚R\x12v©aŠ:a¾ûg½ù¡]ÑRïQQòA\x05ÆØ\x01\x10•\x05„¯:˜<\x14ãû4lØãž²ø«Í\x01¾³\x14vR|ëPžÚ \x17}æà:\bF%50\x1aÇ¥5·‡Åàéø Ú¤é¥Bù©’\b|¡\x04Õã`#ÛÙˆðQ£Å8Œ[mAIæŽ\x1c,rT@åê`µ\x12’àþ±œ¡ÜHŽ\0‡†}[NZM
›'1¶iñÇ\x10˜,\x1a³f U’3##¼\x1f	£\x1c\x1f"Óc”ZÈ\fWK-Ò04xî
©\vï4!·\x1a\x19»\fy\x1c%õåoa¨–Œ`›Òv]—‘a«H8qY\…£¡\f~Vš
\x19í\x01Ç\x01{°yú¤B‹%üþÓ|ÿ}1ÿT_Ï	n½Éñ\ÿ+"^\x1fC\0¸ô1å¦\x11dlÂ\x01\x7f±%'Z÷:À÷NCÅ~®\x1fy}©k\x13`M…ß	ô8âÿÕˆÂo7"€Ýþu#
\x19•\x0f\x11 åvÂZ£\x03hâƒûtÛ^[ô¿¡`—:!Åíhßw\x06\x06\x19~0ŒŽøø
=\fp\aÀ'±\x02(ü5\fk*ÝÔŽZTäû¶æÖlõt\x0e^[ìól±±þ³Ï÷y+Ê)ÅDE\r;µ‹½ÆÚé`@»R<\x16`Õã‡X$HˆË"œZ‰‹ûW¶á„R*׫ßåæüïÅ<o‡;ÔÑëb^\x14[Qûˆ\x17víÐ%~T?Á\03Y\x1diy,	¦$‘!?æ7NŒt”E»à\x06ïŒg£Gy\x1cÀxÔ]
`õU>ßÁJœYûÕây\x01¶\x0eX4›íû\f¦ðßgÿS$\fèïåbõ#oøðð'\x04{\x1di9ÈßÖ›\x0fKäó×Õz¹~ù(è²µ>\aÂýR;\x1e\x13n\x15:výP”Á\x1cíˆÓû‹:\b~:D4[\0›\0eæ5N×\0\x02®¬2r$F×\x06LaæV Oí-²z²Äf¶Ú.µ\x1eðåÑض1îÌ\x19‘¤Ùµ>1{köª
A\x1aÐ\x15ò„A¥š\0Ì\bI<×Ø\x1cçžëtfœà<‰‚so9Û¾Zó×Å{7Ð¥&ê±æ<¿RPIq\x12Ù´Y\x0e²—•Å\r\x1a@\vrkÃìï\x1a\x16xW™MÂq6e®\x18eý&^[´ ƒ£ÐËöÚ·à·x4\b\x19\x04¾s"˜\x17\x03ôääYk2ªlÐ\x1e¤*½4v¨À·Çz	\x05xÐ\x0f-PÄ\x0f\Þ\x165Y\x0eŠ‡tKSÁüö\x12\x02³\x18\a\x06Ld„\x11›ƒ\x163‚#›G>\x15\x14¤‰Å]¡\ffïïµ ¯òË\x14£Îæ2¦X×FZ\x1f\x041\x10@.MÌ¡‰ÑâÑ#—®ü^[Rxð nZNV\x1dǧ!`ôÍ\b’/\x14[Ü\x0fš#Œ}"Z¤,\x02âËç3p4w³Å
Ô0 ZOz+©D²Ñ\x10÷­H<:\x06…\x7fÇÀJc\rä\x10:»Ûbûã,Z9r	:[]£\x117r\x06=SÔõ\0Ç~ÚåŠ\x10vKKk­ãƒPˆíÚÑ*ÿ×\rÀŽŒ¶\x11Ï6³å2_Z2\0Q\vDÔ<ÛDº\x12ÝŠËý‹´\x7fdxð£\x1dÁÐi\x18zÝ›!à"ÃÂ;ž€áDS)\x02A3’ÝBòuVE·rò\x18‹Èoå/tÐB^[sNJh#ù®V¨\x13sjÇ\vu˜
Hԏ2ë@ÅŒÃŽCʉ̺j\x18EŽ\vÚH\x1a‚Ž;Á©\x05&p\x169‘Q1ê*œÅvŽ­€`^ ²™:5 ¹sø\x17³óÀƒýÔ÷1v`n·"w8+…\x1c©’\x10\x06ÚSˆ\x04Ób²n]CËoi\x01û,Äΐ\x14X¦Òe\x1e/Ã6ªû¢_k÷ñž[ŸŸ@¼\x7f·v³÷üwËqÏ€_0.ä8]Q¢Á¸¶/Á\x117 \x1cšÇÙCóx(é\0vº‹Ê×oyÜ`†æ\x7f¼ü\x01s´þwÿ#ÿ¾þy\b&Zoûånñ¾Ì-?\r^[\x1c h¨‚A\x19€p3K¢ÀoiÎ^[\x12\x12\x15Š\x1f\r‡°=á\x1c¡ÌN9”nï<f]Žh¢xÎ)\f¦þ{\x02‰\x13þoP|fsÒÕÏj\x1eËõ?g:£\x15ÙÁô „ƒë\x1a\x05ìD\rë\aÇ pÖç÷Íúé‹V¦¥:VåùÏwÀzËa3]~Áz—X\x1dÁ—ë~Ö\x14Gë³\x14BÕƒ?iÊBÐÝ\x13¼½Ìª¶äцYª½”·\x0e\x1aõñ\x1f8”VÿâîÒúì-6ù\x14þuâÛl\x10u^[.“O\x16«Ý\x1av3³Í0e¡ë‘¤ky”.u§éʯ.O‘ë¾g\x1a\x04xhÕŽB·ÅÜUÈà[J|Ð帄‹´K\x180¨ŽP\x13 À‚Jt÷*\x1dWX¿~Ï\x02‹\x11\fµàûb÷¥1)©þeȽ\x16²\vX-ÑN\x1e1Ò€8Ý\x12y\x02\x7f\x7f}Ù)\x16P|Û)•ù¬Y‘ë^%³ÕÚ\x02\x11?fb”P>ŠÀÃ\x1f\]cX|l«,º$Š‚z^ê\x01\a\x17˜:-À\x1c+é€ÀZ@±þ‘¯¬DY߇ýPA$w‚“¼µ|\x12ZŸWëÕÙëìm3{Z¬^[""iÜ	¹è\x06f+“UfHÕ\x03GøAAlÐ\x14¾!ÂÍG$6°\x19ŒKêèÈǃq\0n§ 7cLr=á[$ð\x03	p1î†2•óÉYSzn—Á\x05òýu½ú°8ÆßQØ5\x1cØê}¿3Ê+\vã´yV!\v2Ï“'˜>åxäc)•Ü\x02ÉZÓµƒ(åT¦c~àåYÌIú`„r'¡4Ì\x1e\x1a6\x1cŽóx\x7fs}Û\x1eü×è\x11P\f\x01\x11‰ øq8œ‚c²£	mö\x10t]ì$¿…¢&y\x04\x0eóã‚9x`¶˜a”:#M£c\x1dµŽ„ÕhG³ÍÓ?ò¸’GViY”r"³B^[gÚ² c·½K,ž¡¡ð_uNòÖ®\x06Ê8æx@Z#$dŠB‡$ ¨¿ä¼ÂÖ>\a¶ÄŒò	nHê\x14^[™ ãªIw¯6åŸ\x15Uo\aõĐZaVh\x0e\x1c\x18&*[¤–nU‡&à\x01±€\x1ePÚãUHôAÐÐÅ"ß f%\x06”¨¡·\x1cÏfSÍk\x17µÂîøåIëÝm\x16‹ÇZÞ¤O‡Äy4\x16\x16ޜܨ´œÄ\x01k\x1e°\x04,^[\x01é1ý8íæ¯Oë\x17Kžã·”¿N\x7fFøn4\x05Ö	ݨq\f¬îzáìDeê"~>ã
Ã\x01Hrqw}Ù\x19®§Ý!Ø)­çåúýýCùGMC«é¶M‘¼Võq€º–\x0eü‹»†\x19^[8ˆv\x1fÔ\r’£¢ÐZ1\x1f*‘åËz³Ø½¾5•\x15 Ëëx¶éÖS\x01\x1dï\x04¼k<Ë^\x0fZFæ…`.0Ô—¶
cøºHx`8S“°Hò/îHp\x11p\x01\x15\x04†–\x11‹‹\x14?ÌRM8Dæ¹ 1|	;d¾†©É¹ªø\x172GNAº\x12ž¹¼\x7fqqcPN\x15Ê\r\x1eë-Q(M‘Õ\x12\x05\x16ëöêD30­»«\v<\x1d§ÑÎ\x1d\x1e+q‚˾0\x04ô\x0e(äáúö\x06?G”\x14ÖG9í0
‚"\x19í\x04Šš™¡ègĐ“☡+\x17ãyIÒå\x0e\bG\x0f®Áåœm\x7fÛZý³\x7fÀ"¶¾ï›Ú¬{8x\x10‘`½Zì@FÁ¶îHøh\x1aDa#+O\x16d\x13†^[\x18%é	Èu:L“ô8–^[€	v|5\x0e¾ˆM\x1cCNh\x1dçâd_wƒËãLE¸Ý?ÁwD<ÄÇ;\x02q»>1oï¦\x7fۻ•ß\x01G\x1eK^[\x0e[+”ؐÛ_¢\fý«þ-?.Ù€3èÀaâö¸’ñƒë‹\x13\b\x06i­!œêâ\x06?L«\x10n/\x1e œ\x1aäí©Až¤ÃÝ©1Ü™ÌÖƒfë_÷O)ÑÛ›‹ëã\x1dñ€;—7ÁqN,‘NŒI£Ù\x17wÇ'?\x11ýV¦B\aez{qsÛ?®`$ÎÝ¿Á\x19œÄñon¯LñÑ
ëzp3:.\x1a‰\x1a°´Þ<²\x15¨ß[þ´˜¡Î\x05si”µÜÆâ\x14ìe±\x03\x13|²xÊ×–½YÏžæ3}9Åí¶ãNº\x01ªáföþº˜o»ÚÿíA]Ö8×ðp»E^\x1aQ&Oæ;nÆÓ\x18õ¢øz¿zjlw`Àw}Œ”ÛØÞ\bÅ:Ñ|Dx6rðåm E#Cšo\x03‹¶°´Ÿ\x13	ú§\x1c	ØjQB†ÔÊWò†õVE•\x7fWÁØßt,\x7f±ýQ\x0e÷·26%§0œ=½ä;,ž"›\x1d\x12wH»$
\x02眻Ê)Áh\0àN\x15Ÿº\x18ã„t
¾’ۍ³(üfb,\x12›ÅËK­Œ­ž\x17«…-'ÚŠóR—8Ö™•o6ké\x0fïò¹JN‘˜›\"J\x16üœ\x10þ¥›¸¦3>r`[!ï\x1a;ò¨±á·\bð%|‚„‡ä
ºzY¬Ð\x10¬L
§á!q*YÑY.òÕî07Yä6£‰ÞBÞ›WÁ²Æ£\x17b \x0f«ªà±.Ê\x1eäÁ\x04\x1e^.1\x18–I\vÐ\vÙä[«@7ØêI\x01ô\x15?âàÁÀ\x12«¼\x05mBºT'·Ø)ÞWÛ­ÇPà³^[Ä-Å\x1f¶ »¼_R-\x1aeàÜ{¼Õ|Õ¢\x19trz\x0fæºC\x0fL°È9\x02\fX\b=\x1a€àëtº,mzÐ
Ì{¬Ç\x17UJ\x02¯\x13ä[\x1a	쾏{hà€*ß41OQC/ÔKåK\x16]Yѧ¦çòèYòuÅÖµ ptw}Ý35œº\x1eÖ®^[ñsˆóP˜Ú\r8à˜Z@]\x13,\x14\x1d*è“—m¾\x7fZ[ύîŠ:ÕQr½`Ü\f;©²ö“&ð]¯'‚¸)äªà\x04gŽRPà¾\r.ÿ\x10{ž@½ÂÒH	P\x7fðY6}ûgœ¶Ä=Â(ž\x196:
’©-&°MÍUm3èH­¯ÝqÖ ƒ†b,KŠlÌ~§|
ú\x0e@ž×”>\rçi\x10˜î~\x1dZè(ñ\x06‚ÌãJ(W¯—\x149ƒ\x15Ïh”¿|f·Ë\x12\x19_í\x0e(‘\x17(\fÇQÙûa„¾÷QG‰\x13&_³zìv¡á†g}œ\x0eGNŽ(×Ql†}\v\x1f.ÍPùü’Q¯àì_\x1eþªM˜·Å<T•**ËïIãî¨.1Jª\x02ãÑ)	rYB\x1daÈÉ\x06\x04,Ó(‘\x19¼a}T2AÈm}B¯Íaë´¨š\x1eJäùR\x15|ÂNž\r9ÏƉm¸\f[áðx\x1c\x18ÜÖÀ6­‚ÃLJ؉u"—˜•‘\x19îÚ\rþRä­í\x16Êh\x14\x1fïõc¾CÖôá
Jc¯\x04Í\x1eV8øÆϽ\x13\x18$`Cr
G„ÀQ§êÇ1Ôfx\x1cÇwƒ\x13\x18|xj$©/\x12بN4“ž"ݘ$Á)ÂPÏ0\x18-dz\x1dø\x15–?[½ìg/9æû„~—\x1dÔ­ó^ýÖ¹öJ\x16ß7³Í‡µYïÁ¥hz7N\x02~‚Ä!É+qš\x19ÓUùE#Ý\x19´·*Ã.°JÕ.\x13ÝåæJbU©Kuéa«ŠÛïÇ€ÕáÉ»ãÝÓܱ~ýðu6ÿѺ:¢Ÿ1SoG`~C¨ž±	R.2°¿›ob\x05dÈ”•“àwwÒP\x1eo\x03B`G>.©êº·Ñð×Põ®V9’	ñSZ=Ó£#\x1cúúh]\rÊ\x10¹'߬ñÄ}ÿ²Â\x16òŽ5<u'9m^h–—n̯ééžäÓkGÀÕƒk¸
-À™|cÃ\x10J-ú‰Y(‡x\x04åXGÅhäs”GgSô\x02Ó¢\x14Ï%ш\x12I¾‚Db&/ª{rüÇ\x1a\x1eGöWê`—[5\x1c\x18¼qð®K\rÚ¼¸\x03ƒ‡ƒ‹ñ1Žu¦Þ“s¡7	\x18~ã`\x12W=Þ!™\x1a_,'ÍD”ˆ4¡j½pí5tq~x0$"»¬xf¥sD-ä\v­+@ò>¹~ê¡8¹Ÿë×2‘»ÆcúhŠ*v]t]qóñ¾[¿èp\x1c֤Ί®×û?PK\x01\x02\x14\x03\x14\0\0\0\b\0·­$6i\x7fÍü(\x17\0\0U\0\0\a\0	\0\0\0\0\0\0\0\0\0¤\0\0\0\0.configUT\x05\0\azgEPK\x05\x06\0\0\0\0\x01\0\x01\0>\0\0\0M\x17\0\0\0\0

[-- Attachment #3: hcu3.log.zip --]
[-- Type: application/x-zip, Size: 7669 bytes --]

PK\x03\x04\x14\0\0\0\b\0Ò®$6 \x0f÷´z\x1d\0\0¼b\0\0\b\0\0\0hcu3.logí<ÙrãF’ãFD\x7fÁ¼TtÃ’G„
÷1Á‰•(ª›k\x1d\x1c’²=Ûáá‚\0(b\x05\x02h\0ÔÑáßÌ*\< Ëöì8vè6ÉBfeeeeåQ•âÄO—Aä„$õ\x1dïA\x10„«Îq\x1cçD\x12%Qí\{¾d¹†5ëxAš?½£Õ5‘T"Sª“\x0e‘%›ê¶J÷\x05¡7¼²	!Gç½\x1e\x19Æw~:ì\x11•j\x1f†)\x19ù·"9&N\x0e\x0f(9ÿø…ì\rÏŽ»’¢\x1cËáq\x17?ûÇ=øDà¾p@†½\x01É\x1e"—¸aìÞ`O\x0e#ü%éäæ˜\f:=Ç]øEë„·„½+\x15\x19áÓ\x19Ƚºq2::‡Ö\x18?I/Žò4\x0eC\x1fù»\x0e²ÜO3Òé\bHŸaÐiïô\x034lBï-jR|5¡ãÉÑäjŒP\x0e[‡\x1eÓÞˆ÷\x05€\x0eâ’Ö R\x03ºÕw2"¤„šŽJ%­	\x1d10ƒªEßÜÏr›(29?†E¹KƒÜ‡Oœ4|Ä7\x02ƒ\b“8!ñœàäW™3\v}2SR,·\x03\x04¨\Ð^[ù™ŸÞ\x06\x11,µiÜl¡IsOÚ@“MŽ¶tBX±½ý\x12qF]ª6\x11•ã\aà—a\x1fÇNê‘A4kü™Ù$¬š\rô\x0fa<\x03E=qr§¯Qaœ; %Ã8ˆ`\x15^[ Å\x14.ü;²\x0e\x0e²&8¾#é*Šp¬ b²éìœëéÙÑø£Mæ¡“-¦I\x1aç¾›“Ë\vx’ÆKX‹ÓÓS¦\x06$yëH=9=\x15JÌ8"´Ùš\rYÐ$\x19tXøöÛoÉ\x0fNʸé\x19,_oÔ;€ÕÂ\až?wVaNüè6HãhéG¹ \f"To\x02â
œP¸\åvÝê§i£UL*õa}œÜ÷\x18£åìÞ¿\x7f\x0fK\x17DÓ0\x06%ñQN¾g“\x19ôðüÐyèjÂ:\x0e‡¹K¯û\x0e¿$„«\x0eY\r–εÿ\x17\x06]¾\x13>\x06 ÆèÜø\x0f8^–\x03ug•Ç\b\x06Þ4òÕW_\x11•½+ì]fï\x12{§Dè_ô'dœøÀn\x11	­Ç,mJN¯ÎΈ·JBÿž¸q\x14$ƒ8\x12Ž//'C2KcÇs\fŒ˜pò±7üèD\x1eìs›\ƒ\0\x12Ð\x05\x1f\x06ßËR·«^[\aÄËò®n\x1eÐº
¥ûÀ%È\a¶“p\x1a„ \b”}r““.¬à\x165|ð\x12’ˆoófwÜ?ë÷&ƒ‹\x0fd–LçA\b\bï¸üÞ	“ÑÑÅx0\x19\^ |rIFý¿]õÇ\f^[\rO_ÀÕäx\x1fádá&c?òFþç\x15؃á\rp„mä\x1f‡-(\b“Ô‰²eç^[€j\x1aÀ5LVQ•_ <õ×\x17žÊ\bàfÆV-\x1d’@<7\f@A\x11W\x11ÓuÇóR?\x03%2dQÖÀ¹iÂä\x14”…í_47`\x1aj˜ù\x17\x12¯R2\x18Vý‚µ®0;?r–>ù†ù(œÒ•ØhùŽ`A
6ùö«÷¿ô%¼!¿\x0f\x1a‚\x17G¾À-wŽú6÷S°( \x06’¤Z*xì=I2-ß$\vÿ~\x1fL\vÁEdF\x18¥‰\x0eŸ\x16Þˆˆ¢ˆî‰™\€ÌѨ\x05Ñê¾#‹º(Y¢TÃ'\x0f	ƒ—\x11\bÃ#ßùiä‡\x05ÊÞõ— \x01£±Lp¥|o\x1f{3Ÿ2\x0e¾°ÎŒIE+=U—`0„\x1e\x14\aÆU>ªV¹á½û\x10O<p/\x03Dšïa\x1fÌ\x1fpv½…ïÞd«%NŠ\~‡À«¨d\x05\x11Ö8-@:½\x15ˆ\x0f´9cž\f6uÆDtO¥ÓcjÉ”tÿ
:œ“U’€\x1a‡\x01loîˆh\x19¸\0\x11°Öa\x10ùEOj€“ââ­š†$Ì*\x1d&¸iù\bÇà“—þ\x12FOs”O·\x19¸ \0\x04G*@\x19E0_Yôé6\d\x01à}
R\r@<ŸãTÖÆÈR§\x1a}^[PŒ¾\x06@\x0f\x03䮳M\0ø]wžúŸy\x0fx\x151©„³él•1\x18è©¢ˆ
\x0f:…Ä\r\x1a\x06ÀÏ\x17(¬rê²M\x15›ª6Õ0*¦†P˜\x02Þ0\x1f3gå¥`ÇJ&,\x1d˜\0Ï\x06ñ\b„!Ažz¸`“rç f¸<^ÅeåJ½\aëX.U9»}¶[Þ\x06\x1exjw±·o\x13¶\x11ß¾=?¿²™[/¾/îØ8Eké$AT|\aÙ\a1|_¸+e
€i\x10“<poJ"÷¬\x17`­’©“º\v›”t›Ï˜ÿ÷—ð4v!X(z\x15ø¬Á'\x01V7\v0\x04*v2Ù‹‚›ÐYeÿ\x11]ïÃVuÝ
E\x15©HÉÞIÿâGÒ?;ù\x0e\x1fð‡ûûä=ä$“ÅŠü§\x03ˆ˜–È–­é¤\aá\x03¨£YB.ü\x1c”($çNæ.`'Däè\x03¹\vò\x05y,ç\x11…ÿ\x02\x19’áé\x05õ¸ö3^[¶íÉù\x11i¾(éü•}1%K\x06øEœB(\\0ñÙ\x1aÜwÒða\x1aÅž\x12þ$ýD\x1cˆ_n›ƒ°ˆŸÚ»é\x1f¯À¹B”ô\x05\x18\v!uÉDB&1N-qAÀ’M¡°%`^–àu	î}^[Õ(‹C¿›ç\x0fczÀô.…©w\x0f=ÿö0šg$½#ðÁž5Ü¡}¸ˆ—þ!(Å!‚\0/H\x1ap^[¿J\x05ªm\x03šjÛ°‘É_ˆ0\x1cœ\x05ìk’³œ\x03´%\rG`‘ìÅ©‡Ñ\0¸}\fgho÷…A'	\x12ÈëD­CA\x04Ø`†\v<.PðDáÄg¶Öeéß.Ú*µôŠ¸|\0)¢bª\x15y\x14|§½³LÕš3é€/_Ñ÷Ü_Æé\x03àX¦¢Cbzë\x04!ë½'™\x12<¸)eîù\aD×!+òÀ³\0\x03\x14¾âŽ; ô†,‚ë\x05ì\x0e \aÑIþ\b+ \x14\x01B`»JQÁ‡bâ\x10»`	æÎ2\b\x1f`nÂÛ5]\x14Þ>ÕEFË”Æ+0@ˆA«EØA\x19R¹\x12á\x04\x02,ˆ1¡C-`€ë‹ø%\x01zÀפÑ\x7f\x16D»;¢&T\x03ƒüÙzÔ\x1dmò±ê”¡>σë\x15Îr¯É\x11ç\x16‡\0r|@ð 1¼•R)ÔÌ&'1¦5äG\0×\x06†(\x14Ïlà3<L0¦H\Â\x1cÑ\x1a\x142í°“\a\x10\x0eF+7ôW\x19¹•ÁÀ–"{Ç~\x18färNÎœpŸ„\x10>¬ueÄоG\x0e³\0GÃ\x01‹F\x03×Ïv¡\r/ǃ\x1f\x1fÅ\x18MNÎw!\x04,ЀQPØÈ,dë ¦àG f\bb/p!Ɇ`ù€D˜o•\x0f÷^[$¾¿ÿ!No2’Ý€¼P›A¡ \x10†.¨ð°Xº(€»\0ëê{+<Y‰0lˆ|\rèDàW\x02à\bvT+’ç;<jiCpçŸ^[0²W¤Éû˜å½`\ve\x1eJº¦Qâ¥\x01†ù\x7f\x1aù·\x01ú\x14PRÑ¢äOà2’8ͳ\x032\x18ýd\v‡y\/`§$žÀ3h¤c\x13f71„:?\x1f\b\Aö‚\x14Ã\x02ð½"8„t´ÕGÚÕGZï3::?\x19Œ¿+ÙD\x01\x03\rˆn \x15‡\x19\x0f'€'\x7f<g»é;ÂB\x1f®êxP†M§ç\ÑÈÞÒ¹'&H‘iþ0ÄS¹û{r	û¡\x7f~Ô+Æ:¨¼¬"jª\0®‹ÚÍña÷“ɏÄ]8j‡\x196Gu\x13ã À÷!æ€}ŽT·C!ŽÂ\x14Œ|ð#ŽKÎ\a\x032üøw²‡aæ¾°\x04\x16m2\x1c\x1fÊ ˜«Ì/¸f\x0e\f8Ã\x13 '\f	¢	ì\x12tY¥Õ(æÁö¹»š\x01ñ†¾<iA\vî@ÏnpA¼ø.Úx´J\x0eØñÃéɏ`;;=fulòaUÄí‘Ÿ/ì\x06Œ¦&âÿ\x10•4ñz°óB\x1fòV›h>³.ŽqÀ‚¸†/= H©Û t@®ïªvñÿ\x01§´ˆ![Fw\vy33c]ؾAÖ)\x1a{\x11D\bû\x05.Æd<Ñí6Üõ\x01óÿçró9ìÎE\x17rØø\x06g¹JØ>a§ˆ HxtD\x15X.<Ъ{>Š®\x1dJ›èߟŽmÂ<!¬\r\x0e
A \x04\x18˜Òg\x0f°dK0A§©ï3’\x11(†WºÙeá+ï*¼\x1d\\f`±ë°ÒÔؼ¡ï[áÍ›\x1fü\x10´ÉÇ šÇ‘Ë™ïá>áÑh¿q´öæ͐…Öß\f¾AtŸŸ\x1dâ{\x11«1ƒ»JD\x16±S–|QúÓO\x04\x127B~z+à™WŠÉ\x1chÈ\x1d\x18O\x02‘?Ø/\fËk¤\x1eì#<pt]\x1c\x11É|tRïÎI}Òc\aශÃÏÑ"PM˜v¾ˆÁ‰\Á\x1eAä\x0exšÙêšÄ	\x1e‚±ƒ6ŸC<?\a\x03Í\f\x06\x1egd>Fà|'Eåp%µ±Ï„ø‘»]…ӰשdC F\rr\x0eñ´d\x19´1Á²^[j";\x11aÁ\x1fi`°¥E\x14<y\f^[ëš­¡õ1ºC´ìÎIH–8h\r*x¹²ý¨8UJWQèßú`à\x15ì[<\x05}ïìX$á\x18¡…N¢œa\x1eÍ\x10ç|\x1c'MØ)M\x16óã¨"Å/M\x04.O\x06ÙuŒž\x1f‚‘ʆà)k\fö3CµŠAæ)“\x04X*vç@@]³úìI6p?ƒÚ\vŸN\x06gý\x13\x14`鸹P€½ëk\f}\x1a".1
µ\x7f\x04\x03\x19„”ÂÞ%þ‹Óq«ð+$>…6´jœ{pɹ×\x1cf@œ%I]‘-² ü\x10„á\r\x18nH²œÕ\x1c\x141%,\aÃ/»ò0\x01o\x01l¦<"·ZdoÊ\x04B®*=|"ÏÛgï\x1d*u(ƒª²­\x19\x02\x12E¡\x05àðѾ\bgx6\<¨ÔœuP@Ó54PEžÄ§uèçî!¬8Š…í\x1eÜ[˜ \vg'Ó³Áñèhô÷éðhò\x11\E\x0eºrG\x0eKŽ\x0fWYzxÏ\x03¨Ã0˜	Ï@ƒ°ÕÞ~ÀÞ²
V}ÁGBº„ÔŽ[‘\x14Œ"hþ\x7f7)|c“\x01\x06\x1aÐú.Æ[B\bSßD)\b@ï”ß!Õè3\b¹;ŠHé{–¾\x7f\x05/‘õìàÝ\x13é€øß6pþøÉ\x1a‚Ùé„)C}\x02^[)þñÓwˆ\x06\x1f©›Ä_‘¯ª\x7fþ=s"(·îa\x14ü\x1d\x0fL§\0áÂÛ”Ô¡ÍûµKŒ¼e2kpàzä°8x˜î¦º.\x01Æяý‹ËÉ´÷÷ÞÙ ×í|nÅ\x18ÿ0˜ô>NúãI·³F\adò‡Z(ŸQ.’L…JPv™j4€\r9²èćüÆƳ?^[mT0\x7fÀ¸í\x16¢6¯t\ah”\x05<å8^e\x0f³ø\x1eÙµAÊû/le‚°¾!Tµ´ûì\x1c¥ÀµIÁ\aO¿À\x03ÎÌâÌ\x17v?ÅðƒµÐ+wßÉ"„™ô]ñØu»ï`“O!.í\»nù\x18]\x15\0xÒU>LR\x7f\x1eÜwß5—§„ñ¡Ù¡_÷]g°¶‚ü„+LQN¦\x1f.®¦ãË«Q¯­Q¿\x7f—-“5*¡W’9ÛÜÀ ÔI¾`w«\x15Sq\x16Ü?9òÎLJ¬ï#lacŠz4®\x0f÷,\x16§ „5¿¸\x1d+¦\x01/uÒ\a<ãZ\x17'Rx÷B5\bÀx\vƒ9÷‚óÀ\x0fñ€\x1a¢\x17\x7f™ä\x0f\x04\f&øÜ^[\x16¼­Àî?Ä+ˆq\x1f ©\x06;\x05CÄ¡WFl¢€¡2v…(†ÅSÅÉ÷\x12»%«Rø¼\x02«„!Z†7¶'±»Â†ƒº~Ø[°Ã3Q \x02w\x1eÌô?×{0WSþ\x0fëqÈh\0­\x0fÑŠôÈÖ‹\x1d>2àÒ¹ñ7€ŠhRa•\a!H\x1a9i¾ ø–\x13a‰>w›j	Äô¶ƒAm'ã0+©B²"Ï3pF×Ùz?ITL!q—nàtÜ\r˜"B$\ˆ\x04/EØÒ×\x03Ân\x14N\x1eÀP@Þ„\x19\x0føè½Ðóö+à0ÝdƒfAV\x13À7Y¬ùÑ©ðÝÌÛž!!¦e\vãE\aųI\x15²^[˜ã9›~Æ®[üšÄ.µ´v¨¥\vÉ6:l÷ÐMVL9±\x01!Xœ¾±ÁúÁÓ7oì¢>E`\x11/oS<ÖO‹ã\x01xb‰¦Dö’Û”hÔ‚@ØÒ QÅ×ñ2H2€+–)*Š€§í`‹‘F\x19Ü`´ƒw
d¶ÊxL\r@IQp€gÎbéixW´6\x11\x02Ϊø6\a$°\x10Ùæ‚&ë²ì«¦<—4ËÑ\x14ÃU\ÅœY®+ÏdWÒɆ8^,Ç\0ÏÊŠ\r1Lƒ\x18⃇.dt\adàu^[×+Aú™v ŠbÙE’ãiMúY‘ëö\x01$3\x10’CŒˆ\0¥£@²p:³Ù&H³Iâà\x05]ýø•\\x17çW\x10bÞ:iàÀ¦[€9,8×LUS}u^sÎG-ù¾\x03k³“éMæ
ž7ð_Éra\x0e\x05~\x06þ\*×x°âäþ\x14O\x7f Ý\x10<`Ò‰Í2–8 I\00E×YjŠ•\x15\x119\x04Á\x1c¢ŽÖÑ…(+ªˆ¨¯˜\x02æUé*É3¡±±{Ã+ZncÈØí\x1a\x02¯\x06=JÎ0ŸcOΏÎȸ?\x1a\x01¢T"ê–nìDœ@\x1e\x7fy\f¨r*QEÕw¢ŽJTåÉáêI\x1f0Õ'1G\x05¦ö(&ž¶	ÇG'M¤\x1d¢•è£¢Å•tn¯\x05È!\r\x02ÞÇÂ7…ȇ²N\x14C~9Á¥¿d\x06áÜ_²^[¤Š=Ù²T\x15‹’\0‚§/5ß\x125e\x13!Ç«ù\x1cô´1#6s€°R<¯\x01TMg}ƐÜoAYŸ#–±7‰\x19–‚¥}°o\r˜©è´¤¶ÎvI\r!ël—\x13¼Ô[ç\x19¥\x05\x1f0éÁ“-jG‘\x0eùÕZ9!Ces'IÖæ\x02’6\x18×ãЙ­\x0f#Ë\¢ã‘ï†\x0e„3³°äÏ0¹t®¢”ÃÊž(8„àè\x13våQ¯\x10/\x1aƒ´~z\x15±«\x0f¿q_ˆë\x03\x01†ëoL•¯\x0fdæA~†u\x02v1ŒeÈ5\x04ìÄôhÌAªF\x19×ßóÚ¾J܆lP.ƒ\x02r•5\x04Á¹. ½Å*ºá}tËdÔ^¬¨etêÌç\x18\x14=\b…Ç‘~\x01©Ä\x15Ž†À\x0eM”мvå\x02ƒ×\x1eZåé\x1d„y«\x04\vF¤\x1d(a\f‹píO\x17¼Æ	Ñä\x1dhü`:}˜ò$\x11Ñ”]\x03b”?Í‚ëÈ	³šæ«'ËnE2,,^[>g¾¬ &Æ\x13OàuŪëŠ\x03öé\x1f…À+†âm\x13?
q\x13»\x06Tצm“‡x\x13b‚%q±^\x06²i\f\x1d€\x03DWw Ÿóðk\x17º¶\x03ý*bç¨\bÖw€\a)Ï qºI°¦\a±\x1dØ£UÄïÑÖ†5w`Žae ¤Ïr?YG¶v _ÄQ\aL\x01žÅðÛÑF\aiׂãy~Çs®U蔘»\x16ê\x04\x0f‹÷^[þrÌ°¿¿á\x1f7×\x0fâ\x1ePàª(=K\x15\x17\x10\x17µ\x066Ï¥á;‰€÷]]I¡šNm¼\x17èêªÍ* ¾tñæüÕÄ!”\x13ð&°|ñ}¢Tò’%	2Éòõ©ˆõ~\x02”5u£;P^Í\x13¬©\x1f¹\x0f‚¥í\x16›ü,*ìÞ\x14kõ	Áº\fx\rG\x03Îã°?\x1a\â£Éà¼\x7fy5ÁgXjZLáâè¼/°9•ÓêH^[Ó¬çKF¯——“_À0döküž\x7f¨–¥þ:<Ý`÷kÖg‹é\x06‹[_)«§7±aY¢UpÞÒ[–\x14‰®÷æ/ö]\x04Ñ€\x02¡Æ|bWí;\x17þy\x12`ý\x05”Ã*ëÆ‘Ÿ-âüÏwNî.¼øÚfEW]ÙÆ\x12­['ìJöÿ\x04à!ý¬kêºf¨’jZ;wÂó\x06/󐢜a\x17!åy[ª¼kÉ\x0e]È:„:übR¼hâW"Æ<înZêKi¥¹·ü%¤°ÿá<\b=?\x13rŒ¸º’lÚqâG]jÏ!®Åö/$\aÃÞ´(\r\x10°¨æ\r–×¾×	»M\x7fó†õú…à ÓÓb*ƒÈóïßœá\x1d›÷愍¼‹ºö|êå=Üú<†ÅÓSv{gÙ¥\x1eÖ\x06?=³]cãŸ-tf‘ä
ì[‰ªãm¢„G“\axK
\x1e(…/,ÙvÐo_CìMEÅ8@ƒ à;Õ„‰“Ý`\f\x0f)\x14[Ø\x034&å\x19<4 ‘ËBßOx\vŒ\vþÑD‚y;6¾ÄËYà\v½dµ—íã_PˆÆ×0ð\x01xzÑúšd\x0f\ah}è×$
°pMT¾&Çži_“;§„.‚òY\x16`‚ǽ\x16Kùnj¶ðª×¤781·©¤\x03\x02êßAe;oÈŒç,ç²^[ÏkBŃŠPÑ®\bI*$@7üòÜ\x13\x04nì¯Æý\fáó\x02ÜÔ÷ƒ\x11Ø÷Q\x7f\f¦þ㈌¹•ÿú¼\x7fŽXè·þ\f^áòüüèâ„4^\x18eê:¯-à3ոݖtÈJ	Ö£\x13\x05rÍ\x11±dQÇ£KôîÔ¦ºˆUC\x1e	æ¬Úäð‹ŸÆx™Žq¨b4\bJ\x16'(ëà?$
Ù)1U\x13Ý „žD\x115N\x10¤¯×ê´Æ!ªPƒ`Í!ð¢AzH4`sÌ\x16\x10e)'(ãA\0+Åú¤üÔœò\x0eÕÖw\x1eàÃWÒéÔWL¤“Sòž”ñÆ·ø\x1f««'ÿèU\x7fÄ“/°p)IÂÀe±¦Èðº]2v–	»Åçõc6«šYe\bšà5\x17†ÊvU\ÆöL‡…Ï9ì\vÄ:
ñ?Ãd\x05O¦–›Æ\x19\x04‘—	\x10ß.ù%>–9—÷VŸWŸ‡\x0fìb\x16ù/±;ÎϝÆë\x19µ§k/a4\x19ÿ\j®¨I´lÈ¢¦\x18EC2EË”+´Ÿy\`c5\x14=,¿l­ŒnË»œñΕ‘~ë•	¢NQeP­Ñïge:’Håje°¡\x14\r™Š²õâ•1m°ÒÏ\\x19ùŸ·2¼x³8\x12ø¬ŒÑØ3ÐP%«\\fE4µ—®ŒFmeW\bR¯L±,\x11¬Jv\x17@°ŒgÚ/]˜	»7\x05oÍ<2X*JÒü\x06ò\x1e|\a§\r¡wGá\x1f*~d\x1d\x7fèø\x11w\fþa‚ &?óiÁ?	š\x1f\x7fvóû‚3?+EÈ\x1c&@OJq€[—µµF;T–5Ú\x0eU\x14c³¯¢V\rU£Öf_³jhºblö•ª†nèfû¸†i)í”-*K­”!ÚÐäÖ¾ø}s¾j½÷eUn—†¤hê¦4j…”Tð¹í”5“nBkIJº¥´KC2©¾)F_\vâ­ÖùÊTÙZýZV2ž\x13·÷•5³}F²bÈꚊÊò‹T´ž‚¬šÚæô^[PͲÚ\x17T6¤­%k0iÊzû‚Ê õö\x05U¨¶µd5e\x05bÚö\x05UdÃhŸ‘¢XRûŒ\x14\x18¶}FŠ.m)p=#Å\0®ÛÇ5UesÃ6 –f´o:•\x1atsÓÕ\©’©nBkIª²µµ
¾ªD7¡5Wª&+›r®ç«êŠÖ>#ÕP™\x19©UTU^fEëé›ú#6GµŒm›S1©QÓhß¡\x1aø¢ö\x1dª)’Ò¾C5U~Äæhšº¥À\r®tMÞ\x14]½dš¡«í\x16Iƒ	mY¤šgËzd\]¢[ã6 ²¤µo+]‘Ív®`CJ[
\C5mËÆ6(ëºÑ¾%uHhÛ\x15X7­­\r[S6 Ôh—†\x012·\x1a**ÙLt¯±¢†¬(íÓ7\x14õ\x11^[k¨Ú#6ÖÐŒGl,8ò­ÝÝ lXÖ#Ó·`þ­}M*kí\x16É”\x14³}Kš²öˆ5\x15ý\x11^[kªÆ#6ÖÔ¬Gl¬iÐ-^[ÛèkJ[^[¶Ñ×’­võ†M%·‡\x01–¤iíFÆ’õ-OWÏ\x17ÜàVP›\x02KµTuME\x19òkbQp\x03úÖrWKf–m‰½ž‚	›¿²¥n©J5Að/ÚV<YM\x10\x02Fc[U´
*›íq¬\x04‹½\x1dÇš\x15\x14\x02ä-ë]SÖåvŸ"QCic%jªíq¬\x04£nmØJU$ ÝncA0涍­Æ•\x14*mΨ\x1eWR¥­x²A\x19\x1cÎ#\骼e(êq\rmK\x1aqM™¯ZEUýuV\x14Ô\x02BÆV¨L·#™šIY¦[¢«™”\x15yK^[”UEßœ`\x03ª©ÛŠT«ë[¢kŒk\x18í6\a¨˜[6§†*\x14ò‡Öq\x15X´v%Td¥=®“\x14EÝŠë\x1aP,üYƒB8nÕP\f¨6“{ÙVv\x15˜ÕÉý¢Êêg~ä.~Å󖓘ý©OUÁƉ$¼\x1c)"c<tXErÝpn¯^[\x10羚-\x16…4¦/Q‘Jåi‡¤‰”VR¶Dèóf‚åaÅé
ûE\0ö-CÎñ’c\f\x04%J: kÒè£\x15\x10	!ò.ˆ‚\x10µ\x01‘h\x01Q\x11¢•\x10­*‡h\bÑK\bä<Ô, ø^[p’AšZS@\f„˜Íq¬\x02b"Ī ’\f&›C,¼Z¡\rˆ‰}Š#$\v^:ÿ*©¢ª•\x1aFESÚ>\x1f’áߣ*”0ñÁ÷ý]"\x1eå½'îƒ\vÊòâ“¢‰M(ÙCÿjì“!\x16ä\føÏÈ á^ýõ<ˆÊ
E•\x1c¹yÙÐÉÑíuÝ8w²us‹\x13Ûu$ù[\x17_Bš+?³øRüò÷X|ù´hËâKI„U¢ø—ãTT\r^|iAöúb‚¯(¾„Øï\x1d¾¨ø\x12²<ý¥Å—–ÊK\x12w\x14_J¼ððÿ²ø’\x17†¾¬øR“[Š/MNmWñ¥ôÿ¡øòiE}vñå\vHý\x1aÅ—’FÁŸç\x14_²z­_·ø\x12'ûx%ü‹‹/ùdþ]|ùïâË\rÐ\x13Å—ÏWÅöâË\x17Ðxyñåó‰?^|©˜&ü3+è®âKÈåecW}æ&OŠMŸW<ýXñåó©ü‹\x14_¾€á߬øRÖ\fÙ2”úi³ø’U¸<V|	\x1a RÍhö®‹/ñgáž(¾|¾\x04^[|©Kª©Éš©é†¶kôçÕü>Q}ù|BOV_¾ŠT[õ嫈µT_¾ŠÖîêK$õ¼ª×WU_¾ük«/_8Ì\v«/_HýW­¾Ä±w\x15íDT•U_ÒÇË/%Q¡Xxi©ø®š¿nù¥¢c%–_j\x12\x16b®—_ò’Ì¢üRÚQ~)=Q~)S°!ͪI‹ZòoR~©™Xçù/P~©Š`Õyù¥ik°b-å—ì7Ï^U~©þsÊ/ù”ù¯Å\x11~–\x13‚Ù4\x7fz\x03³)ü¹Â毂À®p¢^[þ‹÷ü—ß“À…í´\x14ŠOr\v\x02\x12\x04ö\v#Aõçó6\x17\x11Œx5>ƟҍïÊŸŒePüMúwgy?›ýð¬8ø×Ü\x15­\x02\x17\x7f¸r\x16äü‡\x1cð©)ø™ë$~\x03±×q„(fóoö†g©?â[<,ž±\x1f–ZÃËüÈ›ºK¯âÇÉÜ ˜ÞÏSÒÉHç–tB‰
\x10Šû`7JD›¤_\0\x06îð\x7f\x01PK\x01\x02\x14\x03\x14\0\0\0\b\0Ò®$6 \x0f÷´z\x1d\0\0¼b\0\0\b\0	\0\0\0\0\0\0\0\0\0¤\0\0\0\0hcu3.logUT\x05\0\aŒiEPK\x05\x06\0\0\0\0\x01\0\x01\0?\0\0\0 \x1d\0\0\0\0

Re: [Xenomai-help] Re: A fairly small rtnet/Xenomai...

[Philippe Gerum]

On Thu, 2007-01-04 at 21:57 +0100, Niklaus Giger wrote:
> Am Dienstag, 2. Januar 2007 10:53 schrieb Philippe Gerum:
> > On Tue, 2007-01-02 at 09:23 +0100, M. Koehrer wrote:
> > > Happy new year to all!
> > >
> > > Back in the office, I tried out the patches from Chilles and Philippe.
> > > And yes, it is working now! No crash!
> > > Using these patches, I can now call a system() call out of a rea ltime
> > > task that is followed by a rt_task_sleep() and a printf() without crash
> > > on a 2.6.19 kernel.
> > >
> > > Thanks for the support!
> > >
> > > Best regards
> > >
> > > Mathias
> > >
> > > > > here comes a workaround for the COW issue on Linux 2.6.19. The patch
> > > > > relies on a new VM_NOCOW flag which should be set for real-time
> > > > > application if you use Xenomai trunk.
> > > > >
> > > > > It would be nice if you could test it.
> > > >
> > > > You will additionally need to apply this patch to the Xenomai tree in
> > > > order to activate the COW-disable feature:
> >
> > Great, thanks. Niklaus, could you check the vm-nocow patch for 2.6.14 on
> > the ppc board (the hcu, IIRC) that exhibited strange faults in
> > user-space while running the latency test, especially when enabling the
> > nucleus debug option (the one which also used to cause secondary switch
> > warnings being sent by Xenomai to the kernel log)? TIA,
> Here the results. I had no problems running xeno-test.
> 

Nice.

> I did set in my .config
> CONFIG_XENO_OPT_DEBUG=y
> CONFIG_XENO_OPT_DEBUG_NUCLEUS=y
> 
> Attached is my .config and the output of xeno-test.
> 
> If you need the results of another combination just let me know.

That's fine, thanks.

-- 
Philippe.