[Xenomai] powerpc 440 userspace latency test does nothing

[Xenomai] powerpc 440 userspace latency test does nothing

[Steven Seeger]

Hey guys.

I am running a virtex5 ml507 board from Xilinx configured to use the internal powerpc 440 processor in the FPGA. I compiled a 3.18.20 kernel and patched it and build xenomai 3.0.1.

I am able to run latency -t 1 and latency -t 2

When I try to run just latency, however, I get no results. It seems like userspace threads are not ever waking up. Here's the result of cat /proc/xenomai/sched/stat:

SCLINUX: / # cat /proc/xenomai/sched/stat
CPU  PID    MSW        CSW        XSC        PF    STAT       %CPU  NAME
  0  0      0          3240214    0          0     00018000  100.0  [ROOT]
  0  898    1          1          5          0     000680c0    0.0  latency
  0  900    1          2          5          0     00068042    0.0  display-898
  0  901    2          3          7          0     0004c042    0.0  sampling-898
  0  0      0          3245052    0          0     00000000    0.0  [IRQ512: [timer]]

Before I spend too much time tearing into this I wanted to ask if anyone's seen this issue before.

It should be noted that clocktest runs fine. switchtest runs. xeno-test just sits there though. 

Thanks,
Steven

Re: [Xenomai] powerpc 440 userspace latency test does nothing

[Steven Seeger]

Just to add more info, the issue is that the read() call from the timerfd 
never returns. I've never seen this occur before. 

Steven

On Saturday, December 05, 2015 03:02:38 Steven Seeger wrote:
> Hey guys.
> 
> I am running a virtex5 ml507 board from Xilinx configured to use the
> internal powerpc 440 processor in the FPGA. I compiled a 3.18.20 kernel and
> patched it and build xenomai 3.0.1.
> 
> I am able to run latency -t 1 and latency -t 2
> 
> When I try to run just latency, however, I get no results. It seems like
> userspace threads are not ever waking up. Here's the result of cat
> /proc/xenomai/sched/stat:
> 
> SCLINUX: / # cat /proc/xenomai/sched/stat
> CPU  PID    MSW        CSW        XSC        PF    STAT       %CPU  NAME
>   0  0      0          3240214    0          0     00018000  100.0  [ROOT]
>   0  898    1          1          5          0     000680c0    0.0  latency
>   0  900    1          2          5          0     00068042    0.0 
> display-898 0  901    2          3          7          0     0004c042   
> 0.0  sampling-898 0  0      0          3245052    0          0     00000000
>    0.0  [IRQ512: [timer]]
> 
> Before I spend too much time tearing into this I wanted to ask if anyone's
> seen this issue before.
> 
> It should be noted that clocktest runs fine. switchtest runs. xeno-test just
> sits there though.
> 
> Thanks,
> Steven

Steven Seeger
Software Engineer Codes 443/444/582
Embedded Flight Systems, Inc.
304-550-8800 (cell)
301-286-5641 (office)

Re: [Xenomai] powerpc 440 userspace latency test does nothing

[Steven Seeger]

More info on this.

timer_gettime() on latency's timerfd returns 0 seconds and 1 nanosecond, 
indicating the timer has not expired. No matter how much time I sleep before 
this the result is the same. 

The output of cat /proc/xenomai/timer/coreclk:

SCLINUX: / # cat /proc/xenomai/timer/coreclk 
CPU  SCHED/SHOT            TIMEOUT     INTERVAL    NAME
0    2320/1334             1ms523us    -           [host-timer]
0    2/2                   -           -           latency
0    1/0                   -           100us       timerfd_handler

The "latency" with 2/2 is a couple of sleep(1) I added for debugging purposes 
during the beginning of the program.

I've modified latency to look like this:

133     err = clock_gettime(CLOCK_MONOTONIC, &expected);
134     if (err)
135         error(1, errno, "clock_gettime()");
136 
137     printf("expected is: %Ld %ld\n", (long long int)expected.tv_sec, 
expected.tv_nsec);
138     sleep(1);
139     err = clock_gettime(CLOCK_MONOTONIC, &expected);
140     printf("expected is: %Ld %ld\n", (long long int)expected.tv_sec, 
expected.tv_nsec);
141 
142     fault_threshold = CONFIG_XENO_DEFAULT_PERIOD;
143     nsamples = (long long)ONE_BILLION / period_ns;
144     /* start time: one millisecond from now. */
145     expected.tv_nsec += 1000000;
146     if (expected.tv_nsec > ONE_BILLION) {
147         expected.tv_nsec -= ONE_BILLION;
148         expected.tv_sec++;
149     }
150     timer_conf.it_value = expected;
151     timer_conf.it_interval.tv_sec = period_ns / ONE_BILLION;
152     timer_conf.it_interval.tv_nsec = period_ns % ONE_BILLION;
153 
154     printf("expected is: %Ld %ld\n", (long long int)expected.tv_sec, 
expected.tv_nsec);
155     err = timerfd_settime(tfd, TFD_TIMER_ABSTIME, &timer_conf, NULL);
156     if (err)
157         error(1, errno, "timerfd_settime()");
158 
159 sleep(1);
160     err = timerfd_gettime(tfd, &timer_conf);
161     if(err) error(1, errno, "steven temp");
162     printf("timer got time %Ld %ld\n", (long long 
int)timer_conf.it_value.tv_sec, timer_conf.it_value.tv_n    sec);

Which yields the output:

expected is: 1582352598 771936946
expected is: 1582352599 772083521
expected is: 1582352599 773083521
timer got time 0 1

I've also confirmed that timerfd_read() reaches xnsynch_sleep_on

Seeing as how coreclk is also used for the host-timer and that works correctly 
I am totally stumped. Anybody have any ideas? :)

Steven

Re: [Xenomai] powerpc 440 userspace latency test does nothing

[Steven Seeger]

Another piece of info for you. I changed the latency absolute start time from 
1ms to 10 seconds. I see the correct value in __cobalt_timer_setval() but cat 
/proc/xenomai/timer/coreclk never shows any value for TIMEOUT for the 
timerfd_handler.

It's almost like the timer never really starts.

Steven

Re: [Xenomai] powerpc 440 userspace latency test does nothing

[Steven Seeger]

Me again. The issue appears to be that there's a discrepancy between the value 
returned by clock_gettime(CLOCK_MONOTONIC... and the cobalt kernel's internal 
notion of time. Changing latency to start with a 100 nanosecond relative time 
(which adds some constant latency to the expected result since both the 
expected and actual are no longer based on the same point in time) allows it 
to run successfully.

It seems if I telnet into the board and generate too much text on the screen 
the board locks up. I don't know if the board has any thermal cutoff (it gets 
very hot) or if something is wrong with the kernel. I manually patched in some 
vendor code from a 3.6 kernel into the 3.18.20 kernel to use the xenomai patch 
so I could be missing something, too.

I'm not sure if this makes a difference, but here is clock_gettime's returned 
tv_sec and tv_nsec (with a space in the middle) and the line after that is an 
immediate read of ticks: from /proc/xenomai/clock/coreclk:

1582347100 704906148
 ticks: 15136959121354131671

You can see there's an extra digit in the coreclk output compared to what 
comes out of clock_gettime.

I went ahead and used the ipipe-core-3.14.39-powerpc-8 patch and built the 
vendor's 3.14.2 kernel against it. I didn't have to do as much work as I did 
with the 3.18.20 patch. I had the same results (both with latency and with 
freezing.)

At this point I've got the answers I need from this exercise, but it would be 
good to understand the problem and possibly fix it. If anyone can point me in 
the right direction maybe I can take a look at it.

Steven

Re: [Xenomai] powerpc 440 userspace latency test does nothing

[Philippe Gerum]

On 12/06/2015 04:46 AM, Steven Seeger wrote:
> I went ahead and used the ipipe-core-3.14.39-powerpc-8 patch and built the 
> vendor's 3.14.2 kernel against it. I didn't have to do as much work as I did 
> with the 3.18.20 patch. I had the same results (both with latency and with 
> freezing.)
> 
> At this point I've got the answers I need from this exercise, but it would be 
> good to understand the problem and possibly fix it. If anyone can point me in 
> the right direction maybe I can take a look at it.
> 

For the Virtex5 family, we likely need the xilinx-intc controller, and
that one has not been made pipeline-aware in the kernel tree. I checked
Xenomai 3.0.1 over a yosemite 44xFP SoC and all went well, so this is
consistent with a SoC-dependent issue on the ml507 (AMCC's 440EP sports
a single UIC).

I would suggest to have a look at arch/powerpc/sysdev/xilinx_intc.c, and
make sure it implements the I-pipe helpers and tweaks.

/proc/interrupts should reveal all interrupt controllers in use for a
given configuration (make sure to enable GPIOs to have a complete
report). From that point you should be able to cross-check that each of
them is supported by the pipeline.

-- 
Philippe.

Re: [Xenomai] powerpc 440 userspace latency test does nothing

[Philippe Gerum]

On 12/05/2015 07:53 PM, Steven Seeger wrote:
> More info on this.
> 
> timer_gettime() on latency's timerfd returns 0 seconds and 1 nanosecond, 
> indicating the timer has not expired.

This value usually indicates an expired but not yet handled timer shot;
that would match your description of an elapsed timer that did not
trigger the associated action.

-- 
Philippe.

Re: [Xenomai] powerpc 440 userspace latency test does nothing

[Seeger, Steven D. (GSFC-444.0)[Embedded Flight Systems, Inc]]

Thanks for the suggestion. I will look at the interrupt controller and then get back to you.
I will have access to the board again on Friday.

It does sound like we should be able to get decent numbers with this board.
________________________________________
From: Philippe Gerum [rpm@xenomai.org]
Sent: Sunday, December 06, 2015 10:43 AM
To: Seeger, Steven D. (GSFC-444.0)[Embedded Flight Systems, Inc]; xenomai@xenomai.org
Subject: Re: [Xenomai] powerpc 440 userspace latency test does nothing

On 12/06/2015 04:46 AM, Steven Seeger wrote:
> I went ahead and used the ipipe-core-3.14.39-powerpc-8 patch and built the
> vendor's 3.14.2 kernel against it. I didn't have to do as much work as I did
> with the 3.18.20 patch. I had the same results (both with latency and with
> freezing.)
>
> At this point I've got the answers I need from this exercise, but it would be
> good to understand the problem and possibly fix it. If anyone can point me in
> the right direction maybe I can take a look at it.
>

For the Virtex5 family, we likely need the xilinx-intc controller, and
that one has not been made pipeline-aware in the kernel tree. I checked
Xenomai 3.0.1 over a yosemite 44xFP SoC and all went well, so this is
consistent with a SoC-dependent issue on the ml507 (AMCC's 440EP sports
a single UIC).

I would suggest to have a look at arch/powerpc/sysdev/xilinx_intc.c, and
make sure it implements the I-pipe helpers and tweaks.

/proc/interrupts should reveal all interrupt controllers in use for a
given configuration (make sure to enable GPIOs to have a complete
report). From that point you should be able to cross-check that each of
them is supported by the pipeline.

--
Philippe.

Re: [Xenomai] powerpc 440 userspace latency test does nothing

[Seeger, Steven D. (GSFC-444.0)[Embedded Flight Systems, Inc]]

What would keep it from firing? The latency -t 1 works fine. At its core both user thread and kernel
thread use xntimers so they should all fire.
________________________________________
From: Philippe Gerum [rpm@xenomai.org]
Sent: Sunday, December 06, 2015 10:48 AM
To: Seeger, Steven D. (GSFC-444.0)[Embedded Flight Systems, Inc]; xenomai@xenomai.org
Subject: Re: [Xenomai] powerpc 440 userspace latency test does nothing

On 12/05/2015 07:53 PM, Steven Seeger wrote:
> More info on this.
>
> timer_gettime() on latency's timerfd returns 0 seconds and 1 nanosecond,
> indicating the timer has not expired.

This value usually indicates an expired but not yet handled timer shot;
that would match your description of an elapsed timer that did not
trigger the associated action.

--
Philippe.

Re: [Xenomai] powerpc 440 userspace latency test does nothing

[Philippe Gerum]

On 12/06/2015 07:22 PM, Seeger, Steven D. (GSFC-444.0)[Embedded Flight
Systems, Inc] wrote:
> What would keep it from firing? The latency -t 1 works fine. At its core both user thread and kernel
> thread use xntimers so they should all fire.

It does fire kernel-wise, but the system seems to be confused enough
that the userland task does not switch in. Not that I'm 100% sure that
the IC issue will fix this, but that bug is serious enough to wreck the
interrupt state of the whole kernel, so this may be related, and such
consequence has already been observed in similar cases (e.g. lockups,
unexpected hangs).

> ________________________________________
> From: Philippe Gerum [rpm@xenomai.org]
> Sent: Sunday, December 06, 2015 10:48 AM
> To: Seeger, Steven D. (GSFC-444.0)[Embedded Flight Systems, Inc]; xenomai@xenomai.org
> Subject: Re: [Xenomai] powerpc 440 userspace latency test does nothing
> 
> On 12/05/2015 07:53 PM, Steven Seeger wrote:
>> More info on this.
>>
>> timer_gettime() on latency's timerfd returns 0 seconds and 1 nanosecond,
>> indicating the timer has not expired.
> 
> This value usually indicates an expired but not yet handled timer shot;
> that would match your description of an elapsed timer that did not
> trigger the associated action.
> 
> --
> Philippe.
> 


-- 
Philippe.