[Xenomai-help] How do I force a core dump on a page fault event?

[Xenomai-help] How do I force a core dump on a page fault event?

[Bob Feretich]

[-- Attachment #1: Type: text/plain, Size: 5172 bytes --]

  I am seeing various Oops reports referencing my rt user task, but they 
don't provide any useful information regarding my program's state at the 
time of the Oops.

The most common Oops is...
Unable to handle kernel NULL pointer dereference at virtual address 0000000c
pgd = cf02c000
[0000000c] *pgd=8f8a5031, *pte=00000000, *ppte=00000000
Internal error: Oops: 17 [#2]
last sysfs file: /sys/devices/virtual/gpio/gpio7/value
Modules linked in: rtservo_driver rtasuspidvr [last unloaded: 
rtservo_driver]
CPU: 0    Tainted: G      D     (2.6.33 #10)
PC is at do_page_fault+0x40/0x26c
LR is at do_DataAbort+0x34/0x11c
pc : [<c002d9d8>]    lr : [<c002733c>]    psr: 60000113
sp : ce8620d8  ip : 00000007  fp : 00000000
r10: ce862218  r9 : 00000017  r8 : 0000000c
r7 : ce862218  r6 : 0000000c  r5 : 00000017  r4 : ffffffff
r3 : 00000000  r2 : 00000003  r1 : 00000017  r0 : c0429100
Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Control: 10c5387d  Table: 8f02c019  DAC: 00000015
Process navigator (pid: 666, stack limit = 0xce8602e8)
Stack: (0xce8620d8 to 0xce862000)
[<c002d9d8>] (do_page_fault+0x40/0x26c) from [<c002733c>] 
(do_DataAbort+0x34/0x11c)
[<c002733c>] (do_DataAbort+0x34/0x11c) from [<c0027bec>] 
(__dabt_svc+0x4c/0x60)
Exception stack(0xce862218 to 0xce862260)
2200:                                                       c0429100 
00000017
2220: 00000003 00000000 ffffffff 00000017 0000000c ce8623a0 0000000c 
00000017
2240: ce8623a0 00000000 00000007 ce862260 c002733c c002d9d8 60000113 
ffffffff
[<c0027bec>] (__dabt_svc+0x4c/0x60) from [<c002d9d8>] 
(do_page_fault+0x40/0x26c)
[<c002d9d8>] (do_page_fault+0x40/0x26c) from [<c002733c>] 
(do_DataAbort+0x34/0x11c)
[<c002733c>] (do_DataAbort+0x34/0x11c) from [<c0027bec>] 
(__dabt_svc+0x4c/0x60)
Exception stack(0xce8623a0 to 0xce8623e8)
23a0: c0429100 00000017 00000003 00000000 ffffffff 00000017 0000000c 
ce862528
23c0: 0000000c 00000017 ce862528 00000000 00000007 ce8623e8 c002733c 
c002d9d8
23e0: 60000113 ffffffff
[<c0027bec>] (__dabt_svc+0x4c/0x60) from [<c002d9d8>] 
(do_page_fault+0x40/0x26c)
[<c002d9d8>] (do_page_fault+0x40/0x26c) from [<c002733c>] 
(do_DataAbort+0x34/0x11c)
[<c002733c>] (do_DataAbort+0x34/0x11c) from [<c0027bec>] 
(__dabt_svc+0x4c/0x60)
Exception stack(0xce862528 to 0xce862570)
2520:                   c0429100 00000017 00000003 00000000 ffffffff 
00000017
2540: 0000000c ce8626b0 0000000c 00000017 ce8626b0 00000000 00000007 
ce862570
2560: c002733c c002d9d8 60000113 ffffffff
[<c0027bec>] (__dabt_svc+0x4c/0x60) from [<c002d9d8>] 
(do_page_fault+0x40/0x26c)
[<c002d9d8>] (do_page_fault+0x40/0x26c) from [<c002733c>] 
(do_DataAbort+0x34/0x11c)
[<c002733c>] (do_DataAbort+0x34/0x11c) from [<c0027bec>] 
(__dabt_svc+0x4c/0x60)
Exception stack(0xce8626b0 to 0xce8626f8)
26a0:                                     c0429100 80000007 00000003 
00000000
26c0: ffffffff c03e3ecc 00000000 ce862838 00000000 80000007 ce862838 
00000000
26e0: 00000007 ce8626f8 c00272a4 c002d9d8 60000113 ffffffff
[<c0027bec>] (__dabt_svc+0x4c/0x60) from [<c002d9d8>] 
(do_page_fault+0x40/0x26c)
[<c002d9d8>] (do_page_fault+0x40/0x26c) from [<c00272a4>] 
(do_PrefetchAbort+0x34/0x98)
[<c00272a4>] (do_PrefetchAbort+0x34/0x98) from [<c0027d70>] 
(__pabt_svc+0x50/0xa0)
Exception stack(0xce862838 to 0xce862880)
2820:                                                       fffffff2 
fffffff2
2840: fffffff2 00000000 00000000 00000000 00000000 00000000 00000000 
00000000
2860: 00000000 00000000 40044f74 ce862880 40038968 00000000 a0000113 
ffffffff
[<c0027d70>] (__pabt_svc+0x50/0xa0) from [<40038968>] (0x40038968)
Code: 1a000005 e3cd3d7f e3c3303f e593300c (e593300c)
---[ end trace 2f957df7c098c760 ]---

Another is...
Unable to handle kernel paging request at virtual address 70000049
pgd = cf034000
[70000049] *pgd=00000000
Internal error: Oops: 805 [#1]
last sysfs file: /sys/devices/virtual/gpio/gpio7/value
Modules linked in: rtservo_driver rtasuspidvr
CPU: 0    Not tainted  (2.6.33 #10)
PC is at 0x40038998
LR is at 0x40038984
pc : [<40038998>]    lr : [<40038984>]    psr: 60000113
sp : cf0f3ff8  ip : 00000000  fp : 00000001
r10: 40242c3c  r9 : 00000000  r8 : 40242c40
r7 : 000f0042  r6 : 40242c40  r5 : 402434b0  r4 : 00000000
r3 : 00000a64  r2 : 70000049  r1 : ffffffab  r0 : 00000000
Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Control: 10c5387d  Table: 8f034019  DAC: 00000015
Process navigator (pid: 646, stack limit = 0xcf0f22e8)
Stack: (0xcf0f3ff8 to 0xcf0f4000)
3fe0:                                                       00000000 
00000000
Code: 0affffe9 e3500000 059d3014 059d2008 (05823000)
---[ end trace 6d46aff735536a73 ]---

Both Oops reports reference paging and of course none should be 
occurring. (mlockall(MCL_CURRENT | MCL_FUTURE)); was called.

They also reference "navigator" which is the name of my rt user task 
which should be running in primary mode until it is told to terminate. 
(No explicit memory allocations are being performed.)
How can I force a core dump when the page fault occurs?
Can I configure the SIGXCPU signal (as generated from 
pthread_set_mode_np(0, PTHREAD_WARNSW); ) to core dump?

Regards,
Bob Feretich



[-- Attachment #2: Type: text/html, Size: 8345 bytes --]

Re: [Xenomai-help] How do I force a core dump on a page fault event?

[Gilles Chanteperdrix]

Bob Feretich wrote:
>   I am seeing various Oops reports referencing my rt user task, but they 
> don't provide any useful information regarding my program's state at the 
> time of the Oops.
> 
> The most common Oops is...
> Unable to handle kernel NULL pointer dereference at virtual address 0000000c
> pgd = cf02c000
> [0000000c] *pgd=8f8a5031, *pte=00000000, *ppte=00000000
> Internal error: Oops: 17 [#2]
> last sysfs file: /sys/devices/virtual/gpio/gpio7/value
> Modules linked in: rtservo_driver rtasuspidvr [last unloaded: 
> rtservo_driver]
> CPU: 0    Tainted: G      D     (2.6.33 #10)
> PC is at do_page_fault+0x40/0x26c
> LR is at do_DataAbort+0x34/0x11c
> pc : [<c002d9d8>]    lr : [<c002733c>]    psr: 60000113
> sp : ce8620d8  ip : 00000007  fp : 00000000
> r10: ce862218  r9 : 00000017  r8 : 0000000c
> r7 : ce862218  r6 : 0000000c  r5 : 00000017  r4 : ffffffff
> r3 : 00000000  r2 : 00000003  r1 : 00000017  r0 : c0429100
> Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
> Control: 10c5387d  Table: 8f02c019  DAC: 00000015
> Process navigator (pid: 666, stack limit = 0xce8602e8)
> Stack: (0xce8620d8 to 0xce862000)
> [<c002d9d8>] (do_page_fault+0x40/0x26c) from [<c002733c>] 
> (do_DataAbort+0x34/0x11c)
> [<c002733c>] (do_DataAbort+0x34/0x11c) from [<c0027bec>] 
> (__dabt_svc+0x4c/0x60)

This tells us that a bug happens in kernel-space for some reason, while
trying to handle a user-space fault.

Do you have a simple piece of code which I can run to reproduce this issue?

> Another is...
> Unable to handle kernel paging request at virtual address 70000049
> pgd = cf034000
> [70000049] *pgd=00000000
> Internal error: Oops: 805 [#1]
> last sysfs file: /sys/devices/virtual/gpio/gpio7/value
> Modules linked in: rtservo_driver rtasuspidvr
> CPU: 0    Not tainted  (2.6.33 #10)
> PC is at 0x40038998
> LR is at 0x40038984
> pc : [<40038998>]    lr : [<40038984>]    psr: 60000113
> sp : cf0f3ff8  ip : 00000000  fp : 00000001
> r10: 40242c3c  r9 : 00000000  r8 : 40242c40
> r7 : 000f0042  r6 : 40242c40  r5 : 402434b0  r4 : 00000000
> r3 : 00000a64  r2 : 70000049  r1 : ffffffab  r0 : 00000000
> Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
> Control: 10c5387d  Table: 8f034019  DAC: 00000015
> Process navigator (pid: 646, stack limit = 0xcf0f22e8)
> Stack: (0xcf0f3ff8 to 0xcf0f4000)
> 3fe0:                                                       00000000 
> 00000000
> Code: 0affffe9 e3500000 059d3014 059d2008 (05823000)
> ---[ end trace 6d46aff735536a73 ]---

This is the real user-space fault. Happening at pc 0x40038998 which
corresponds to an address in your process. However, the stack pointer is
invalid here. So, the most probablie reason for such fault is that you
overwrote some piece of stack, which caused the return from a function
to try and use cf0f3ff8 as a stack address, causing the fault.

> 
> Both Oops reports reference paging and of course none should be 
> occurring. (mlockall(MCL_CURRENT | MCL_FUTURE)); was called.

No. do_page_fault means that a fault occurs, it has nothing to do with
whether memory is locked or not. You are running with an MMU, if you try
and reference an invalid address, you get an MMU fault, and end up in
do_page_fault.

> 
> They also reference "navigator" which is the name of my rt user task 
> which should be running in primary mode until it is told to terminate. 

Well, from the traces, it seems that you are referencing
/sys/devices/virtual/gpio/gpio7/value. If you do that, you leave primary
mode.

> (No explicit memory allocations are being performed.)
> How can I force a core dump when the page fault occurs?

with ulimit. But there is a simpler way to know where the fault occurs,
simply run your application inside gdb.

> Can I configure the SIGXCPU signal (as generated from 
> pthread_set_mode_np(0, PTHREAD_WARNSW); ) to core dump?

You do not need to do that. Simply install a handler which prints the
backtrace, as in the "sigxcpu.c" example.

-- 
					    Gilles.

Re: [Xenomai-help] How do I force a core dump on a page fault event?

[Bob Feretich]

  Comments inline...

On 9/7/2010 5:04 AM, Gilles Chanteperdrix wrote:
> Bob Feretich wrote:
>>    I am seeing various Oops reports referencing my rt user task, but they
>> don't provide any useful information regarding my program's state at the
>> time of the Oops.
>>
>> The most common Oops is...
>> Unable to handle kernel NULL pointer dereference at virtual address 0000000c
>> ... snipped...
> This tells us that a bug happens in kernel-space for some reason, while
> trying to handle a user-space fault.
>
> Do you have a simple piece of code which I can run to reproduce this issue?
>
No. This problem started occurring when I integrated the whole system 
together. I'm going to have to work on it a bit to reduce it to a 
suitable code segment .
The Xenomai content of the loop is:
while (!end) {
     ...
     rc = rt_event_wait(&event1,..);
     ...
     rc = rt_event_clear(&event1,..);
     ...
     rc = rt_event_wait(&event2,..);
     ...
     rc = rt_event_clear(&event2,..);
     ...
     rc = rt_event_wait(&event3, TM_NONBLOC, ...);
     if (rc==0) end = 1;
}

I changed the way my system of programs worked and replaced the first 
two rt_event_wait()s with ioctls, which execute rtdm_event_wait()s.  
This fix seemed to work around the problem. Initial testing show no Oopses.

>> Another is...
>> Unable to handle kernel paging request at virtual address 70000049
>> pgd = cf034000
>> [70000049] *pgd=00000000
>> Internal error: Oops: 805 [#1]
>> last sysfs file: /sys/devices/virtual/gpio/gpio7/value
>> Modules linked in: rtservo_driver rtasuspidvr
>> CPU: 0    Not tainted  (2.6.33 #10)
>> PC is at 0x40038998
>> LR is at 0x40038984
>> pc : [<40038998>]    lr : [<40038984>]    psr: 60000113
>> sp : cf0f3ff8  ip : 00000000  fp : 00000001
>> r10: 40242c3c  r9 : 00000000  r8 : 40242c40
>> r7 : 000f0042  r6 : 40242c40  r5 : 402434b0  r4 : 00000000
>> r3 : 00000a64  r2 : 70000049  r1 : ffffffab  r0 : 00000000
>> Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
>> Control: 10c5387d  Table: 8f034019  DAC: 00000015
>> Process navigator (pid: 646, stack limit = 0xcf0f22e8)
>> Stack: (0xcf0f3ff8 to 0xcf0f4000)
>> 3fe0:                                                       00000000
>> 00000000
>> Code: 0affffe9 e3500000 059d3014 059d2008 (05823000)
>> ---[ end trace 6d46aff735536a73 ]---
> This is the real user-space fault. Happening at pc 0x40038998 which
> corresponds to an address in your process. However, the stack pointer is
> invalid here. So, the most probablie reason for such fault is that you
> overwrote some piece of stack, which caused the return from a function
> to try and use cf0f3ff8 as a stack address, causing the fault.
0x40038998 is somewhere in /usr/xenomai/lib/native.so.3.0.0. A load map 
that I captured from one small iteration after the above Oops showed the 
executable section of native.so starting at 0x40035000.

I have a deadline in a few days, then some travel, so I can't dig 
further into the rt_event Oopses for two weeks.

Regards,
Bob Feretich

Re: [Xenomai-help] How do I force a core dump on a page fault event?

[Gilles Chanteperdrix]

Bob Feretich wrote:
>   Comments inline...
> 
> On 9/7/2010 5:04 AM, Gilles Chanteperdrix wrote:
>> Bob Feretich wrote:
>>>    I am seeing various Oops reports referencing my rt user task, but they
>>> don't provide any useful information regarding my program's state at the
>>> time of the Oops.
>>>
>>> The most common Oops is...
>>> Unable to handle kernel NULL pointer dereference at virtual address 0000000c
>>> ... snipped...
>> This tells us that a bug happens in kernel-space for some reason, while
>> trying to handle a user-space fault.
>>
>> Do you have a simple piece of code which I can run to reproduce this issue?
>>
> No. This problem started occurring when I integrated the whole system 
> together. I'm going to have to work on it a bit to reduce it to a 
> suitable code segment .
> The Xenomai content of the loop is:
> while (!end) {
>      ...
>      rc = rt_event_wait(&event1,..);
>      ...
>      rc = rt_event_clear(&event1,..);
>      ...
>      rc = rt_event_wait(&event2,..);
>      ...
>      rc = rt_event_clear(&event2,..);
>      ...
>      rc = rt_event_wait(&event3, TM_NONBLOC, ...);
>      if (rc==0) end = 1;
> }
> 
> I changed the way my system of programs worked and replaced the first 
> two rt_event_wait()s with ioctls, which execute rtdm_event_wait()s.  
> This fix seemed to work around the problem. Initial testing show no Oopses.
> 
>>> Another is...
>>> Unable to handle kernel paging request at virtual address 70000049
>>> pgd = cf034000
>>> [70000049] *pgd=00000000
>>> Internal error: Oops: 805 [#1]
>>> last sysfs file: /sys/devices/virtual/gpio/gpio7/value
>>> Modules linked in: rtservo_driver rtasuspidvr
>>> CPU: 0    Not tainted  (2.6.33 #10)
>>> PC is at 0x40038998
>>> LR is at 0x40038984
>>> pc : [<40038998>]    lr : [<40038984>]    psr: 60000113
>>> sp : cf0f3ff8  ip : 00000000  fp : 00000001
>>> r10: 40242c3c  r9 : 00000000  r8 : 40242c40
>>> r7 : 000f0042  r6 : 40242c40  r5 : 402434b0  r4 : 00000000
>>> r3 : 00000a64  r2 : 70000049  r1 : ffffffab  r0 : 00000000
>>> Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
>>> Control: 10c5387d  Table: 8f034019  DAC: 00000015
>>> Process navigator (pid: 646, stack limit = 0xcf0f22e8)
>>> Stack: (0xcf0f3ff8 to 0xcf0f4000)
>>> 3fe0:                                                       00000000
>>> 00000000
>>> Code: 0affffe9 e3500000 059d3014 059d2008 (05823000)
>>> ---[ end trace 6d46aff735536a73 ]---
>> This is the real user-space fault. Happening at pc 0x40038998 which
>> corresponds to an address in your process. However, the stack pointer is
>> invalid here. So, the most probablie reason for such fault is that you
>> overwrote some piece of stack, which caused the return from a function
>> to try and use cf0f3ff8 as a stack address, causing the fault.
> 0x40038998 is somewhere in /usr/xenomai/lib/native.so.3.0.0. A load map 
> that I captured from one small iteration after the above Oops showed the 
> executable section of native.so starting at 0x40035000.
> 
> I have a deadline in a few days, then some travel, so I can't dig 
> further into the rt_event Oopses for two weeks.

Chances are high that the faults come from some memory corruption in the
application, or stack overflows, or things like that. If you pass an
invalid address to a Xenomai service, it will cause a segmentation
fault, but that does not make the Xenomai service buggy.

-- 

                                                                Gilles.

Re: [Xenomai-help] How do I force a core dump on a page fault event?

[Gilles Chanteperdrix]

Bob Feretich wrote:
>   Comments inline...
> 
> On 9/7/2010 5:04 AM, Gilles Chanteperdrix wrote:
>> Bob Feretich wrote:
>>>    I am seeing various Oops reports referencing my rt user task, but they
>>> don't provide any useful information regarding my program's state at the
>>> time of the Oops.
>>>
>>> The most common Oops is...
>>> Unable to handle kernel NULL pointer dereference at virtual address 0000000c
>>> ... snipped...
>> This tells us that a bug happens in kernel-space for some reason, while
>> trying to handle a user-space fault.
>>
>> Do you have a simple piece of code which I can run to reproduce this issue?
>>
> No. This problem started occurring when I integrated the whole system 
> together. I'm going to have to work on it a bit to reduce it to a 
> suitable code segment .
> The Xenomai content of the loop is:
> while (!end) {
>      ...
>      rc = rt_event_wait(&event1,..);
>      ...
>      rc = rt_event_clear(&event1,..);
>      ...
>      rc = rt_event_wait(&event2,..);
>      ...
>      rc = rt_event_clear(&event2,..);
>      ...
>      rc = rt_event_wait(&event3, TM_NONBLOC, ...);
>      if (rc==0) end = 1;
> }

Is this is a loop using all the CPU (all the events with TM_NONBLOCK) ?

-- 
					    Gilles.

Re: [Xenomai-help] Oops during rt_event_wait(); formerly "How do I force..."

[Bob Feretich]

[-- Attachment #1: Type: text/plain, Size: 1717 bytes --]

  I changed the subject from "How do I force a core dump on a page fault 
event?", because the thread's topic has drifted.
Comments inline...
On 9/8/2010 12:55 AM, Gilles Chanteperdrix wrote:
> ...Snipped...
>> No. This problem started occurring when I integrated the whole system
>> together. I'm going to have to work on it a bit to reduce it to a
>> suitable code segment .
>> The Xenomai content of the loop is:
>> while (!end) {
>>       ...
>>       rc = rt_event_wait(&event1,..);
>>       ...
>>       rc = rt_event_clear(&event1,..);
>>       ...
>>       rc = rt_event_wait(&event2,..);
>>       ...
>>       rc = rt_event_clear(&event2,..);
>>       ...
>>       rc = rt_event_wait(&event3, TM_NONBLOC, ...);
>>       if (rc==0) end = 1;
>> }
> Is this is a loop using all the CPU (all the events with TM_NONBLOCK) ?
No, just the last wait has TM_NONBLOCK. CPU utilization for the task is 
about 2%.

I examined my use of the rt_event mechanism, trying to identify how my 
use might be unusual.

    * The above calls are in a rt-user-task and are bracketed with
      bind/unbind, outside the loop. The task starts in secondary mode,
      performs the loop in primary mode, then finishes in secondary mode.
    * rt_event_create and rt_event_delete occur in a rtdm driver init
      and exit routines.
    * However, rt_event_signal is invoked in interrupt handlers. I think
      that this may be unusual. Normally, interrupt handlers would use
      the rtdm_event mechanism.

Could rt_event_signal be sensitive to being invoked from the context of 
a real time interrupt?

Interrupt handler stack space is usually quite small. Does 
rt_event_signal make heavy use of the stack?

Regards,
Bob Feretich

[-- Attachment #2: Type: text/html, Size: 2313 bytes --]

Re: [Xenomai-help] Oops during rt_event_wait(); formerly "How do I force..."

[Gilles Chanteperdrix]

Bob Feretich wrote:
>   I changed the subject from "How do I force a core dump on a page fault 
> event?", because the thread's topic has drifted.
> Comments inline...
> On 9/8/2010 12:55 AM, Gilles Chanteperdrix wrote:
>> ...Snipped...
>>> No. This problem started occurring when I integrated the whole system
>>> together. I'm going to have to work on it a bit to reduce it to a
>>> suitable code segment .
>>> The Xenomai content of the loop is:
>>> while (!end) {
>>>       ...
>>>       rc = rt_event_wait(&event1,..);
>>>       ...
>>>       rc = rt_event_clear(&event1,..);
>>>       ...
>>>       rc = rt_event_wait(&event2,..);
>>>       ...
>>>       rc = rt_event_clear(&event2,..);
>>>       ...
>>>       rc = rt_event_wait(&event3, TM_NONBLOC, ...);
>>>       if (rc==0) end = 1;
>>> }
>> Is this is a loop using all the CPU (all the events with TM_NONBLOCK) ?
> No, just the last wait has TM_NONBLOCK. CPU utilization for the task is 
> about 2%.

Ok.

> 
> I examined my use of the rt_event mechanism, trying to identify how my 
> use might be unusual.

Well, the first unusual thing I see, is that you wait sequentially for
several events. Since each event has 32 bits, you should not really
require more than one event.

> 
>     * The above calls are in a rt-user-task and are bracketed with
>       bind/unbind, outside the loop. The task starts in secondary mode,
>       performs the loop in primary mode, then finishes in secondary mode.
>     * rt_event_create and rt_event_delete occur in a rtdm driver init
>       and exit routines.
>     * However, rt_event_signal is invoked in interrupt handlers. I think
>       that this may be unusual. Normally, interrupt handlers would use
>       the rtdm_event mechanism.
> 
> Could rt_event_signal be sensitive to being invoked from the context of 
> a real time interrupt?

rt_event_signal is documented as being able to be invoked in interrupt
handlers.

> 
> Interrupt handler stack space is usually quite small. Does 
> rt_event_signal make heavy use of the stack?

The interrupts handler do not use separated stacks on this architecture.
The stack of whatever kernel thread is preempted by the interrupt
handler is used. But no, rt_event_signal does not require special a lot
of stack space.

Anway, have you tried gdb to see where the error points?

-- 
					    Gilles.