[Xenomai] I-Pipe Tracer and linux ftrace

[Xenomai] I-Pipe Tracer and linux ftrace

[Antoine Durand]

Hi,

I would like to check my RT threads scheduling, so I started here :
http://xenomai.org/2014/06/using-the-i-pipe-tracer/

I put 100000 in /proc/ipipe/trace/back_trace_points
I ran my tasks, i write 1 in frozen, and then I read the frozen file.

I'm not sure what kind of datas can be found in I-Pipe Tracer
(/proc/ipipe/trace/frozen) and what is in linux ftrace
(/sys/kernel/debug/tracing). For instance, I "think" I'm interested in the
__xnsched_run trace points, but there isn't informations about the CPU on
which the task is running on in I-Pipe tracer output.

So I checked in linux ftrace.

But when i activate linux ftrace by writing 1 to
/sys/kernel/debug/tracing/tracing_on, the RT tasks behaviors became weird
(the task with the higher priority just toggle an output that I check on an
oscilloscope, this task is unable to make a clean 10ms periodic square
signal with tracing_on activated !).

so I did not find any source to be able to observe my tasks without
crippling side effect.

So here are my questions !



Are linux ftrace datas relevant for xenomai tasks (there is a lot of
cobalt_core entries so I guess yes) ?

Is there any tool to plot I-Pipe tracer datas from /proc/ipipe/trace/frozen
?
(I tried to use trace-cmd and kernelshark to plot linux ftrace but if what
is ploted is true, the side effect with xenomai is awful !)

What datas must be checked from I-Pipe Tracer and what must be checked from
linux ftrace ?



That's a lot of questions in one, I'm sorry, it's more that I'am a bit lost.

Thank you for your help.

Bob

x86_64
linux-3.16.7
xenomai-3.0-rc4 with I-Pipe tracer Options On

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Jan Kiszka]

On 2015-06-22 18:06, Antoine Durand wrote:
> Hi,
> 
> I would like to check my RT threads scheduling, so I started here :
> http://xenomai.org/2014/06/using-the-i-pipe-tracer/
> 
> I put 100000 in /proc/ipipe/trace/back_trace_points
> I ran my tasks, i write 1 in frozen, and then I read the frozen file.
> 
> I'm not sure what kind of datas can be found in I-Pipe Tracer
> (/proc/ipipe/trace/frozen) and what is in linux ftrace
> (/sys/kernel/debug/tracing). For instance, I "think" I'm interested in the
> __xnsched_run trace points, but there isn't informations about the CPU on
> which the task is running on in I-Pipe tracer output.
> 
> So I checked in linux ftrace.
> 
> But when i activate linux ftrace by writing 1 to
> /sys/kernel/debug/tracing/tracing_on, the RT tasks behaviors became weird
> (the task with the higher priority just toggle an output that I check on an
> oscilloscope, this task is unable to make a clean 10ms periodic square
> signal with tracing_on activated !).

Can you share more details of of your setup (patch version, kernel
config, detailed tracing setup, ...)?

> 
> so I did not find any source to be able to observe my tasks without
> crippling side effect.
> 
> So here are my questions !
> 
> 
> 
> Are linux ftrace datas relevant for xenomai tasks (there is a lot of
> cobalt_core entries so I guess yes) ?

Yes, they are. The Xenomai 3 cobalt core is pretty well instrumented for
event tracing.

> 
> Is there any tool to plot I-Pipe tracer datas from /proc/ipipe/trace/frozen
> ?

No, that data is so far for textual consumption only.

> (I tried to use trace-cmd and kernelshark to plot linux ftrace but if what
> is ploted is true, the side effect with xenomai is awful !)

We need to understand that side effects, they should not be that
massive, rather much less invasive than that of the I-pipe tracer.

> 
> What datas must be checked from I-Pipe Tracer and what must be checked from
> linux ftrace ?

The I-pipe tracer is for detailed analysis, specifically on kernel
function level. If you have a latency spot or a bug on which you can
trigger a trace freeze, then you can look into the details around this
event. The I-pipe tracer is actually quite similar to ftrace's function
tracer, but it has some additional data about the context (RT or Linux).
Maybe we can merge both one day.

Event tracing via ftrace with additional Xenomai tracepoints is for a
broader overview. For analyzing scheduling orders and domain migrations,
identifying application activities based on their syscalls and driver
interactions. It's the first stop when you want to understand your
system activities as a whole - probably what you are after here.

Jan

> 
> 
> 
> That's a lot of questions in one, I'm sorry, it's more that I'am a bit lost.
> 
> Thank you for your help.
> 
> Bob
> 
> x86_64
> linux-3.16.7
> xenomai-3.0-rc4 with I-Pipe tracer Options On
> 

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Antoine Durand]

Thank you,

I really better understand the differences between the two traces sources.
Indeed I am more interrested in ftrace datas for the moment.

I use a x86_64 CPCI cpu board with linux-3.16.7 and
xenomai-3.0-rc4 with I-Pipe tracer Options On

My problem may comes from trace-cmd actually, because it uses nearly two
cores full time during ftrace collect.
I don't known what for ! I used it to be able to plot datas in kernelshark
but its overhead is too much.

I will try with no external tools, (just /sys/kernel/debug/tracing/*
controls).

I had to disable nmi (echo 0 > /proc/sys/kernel/nmi_watchdog) at boot to
avoid hard Lockup kernel crash when system load is high (running dohell for
exemple). I don't think it can be linked.



2015-06-22 19:20 GMT+02:00 Jan Kiszka <jan.kiszka@siemens.com>:

> On 2015-06-22 18:06, Antoine Durand wrote:
> > Hi,
> >
> > I would like to check my RT threads scheduling, so I started here :
> > http://xenomai.org/2014/06/using-the-i-pipe-tracer/
> >
> > I put 100000 in /proc/ipipe/trace/back_trace_points
> > I ran my tasks, i write 1 in frozen, and then I read the frozen file.
> >
> > I'm not sure what kind of datas can be found in I-Pipe Tracer
> > (/proc/ipipe/trace/frozen) and what is in linux ftrace
> > (/sys/kernel/debug/tracing). For instance, I "think" I'm interested in
> the
> > __xnsched_run trace points, but there isn't informations about the CPU on
> > which the task is running on in I-Pipe tracer output.
> >
> > So I checked in linux ftrace.
> >
> > But when i activate linux ftrace by writing 1 to
> > /sys/kernel/debug/tracing/tracing_on, the RT tasks behaviors became weird
> > (the task with the higher priority just toggle an output that I check on
> an
> > oscilloscope, this task is unable to make a clean 10ms periodic square
> > signal with tracing_on activated !).
>
> Can you share more details of of your setup (patch version, kernel
> config, detailed tracing setup, ...)?
>
> >
> > so I did not find any source to be able to observe my tasks without
> > crippling side effect.
> >
> > So here are my questions !
> >
> >
> >
> > Are linux ftrace datas relevant for xenomai tasks (there is a lot of
> > cobalt_core entries so I guess yes) ?
>
> Yes, they are. The Xenomai 3 cobalt core is pretty well instrumented for
> event tracing.
>
> >
> > Is there any tool to plot I-Pipe tracer datas from
> /proc/ipipe/trace/frozen
> > ?
>
> No, that data is so far for textual consumption only.
>
> > (I tried to use trace-cmd and kernelshark to plot linux ftrace but if
> what
> > is ploted is true, the side effect with xenomai is awful !)
>
> We need to understand that side effects, they should not be that
> massive, rather much less invasive than that of the I-pipe tracer.
>
> >
> > What datas must be checked from I-Pipe Tracer and what must be checked
> from
> > linux ftrace ?
>
> The I-pipe tracer is for detailed analysis, specifically on kernel
> function level. If you have a latency spot or a bug on which you can
> trigger a trace freeze, then you can look into the details around this
> event. The I-pipe tracer is actually quite similar to ftrace's function
> tracer, but it has some additional data about the context (RT or Linux).
> Maybe we can merge both one day.
>
> Event tracing via ftrace with additional Xenomai tracepoints is for a
> broader overview. For analyzing scheduling orders and domain migrations,
> identifying application activities based on their syscalls and driver
> interactions. It's the first stop when you want to understand your
> system activities as a whole - probably what you are after here.
>
> Jan
>
> >
> >
> >
> > That's a lot of questions in one, I'm sorry, it's more that I'am a bit
> lost.
> >
> > Thank you for your help.
> >
> > Bob
> >
> > x86_64
> > linux-3.16.7
> > xenomai-3.0-rc4 with I-Pipe tracer Options On
> >
>
> --
> Siemens AG, Corporate Technology, CT RTC ITP SES-DE
> Corporate Competence Center Embedded Linux
>

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Jan Kiszka]

On 2015-06-22 22:47, Antoine Durand wrote:
> Thank you,
> 
> I really better understand the differences between the two traces sources.
> Indeed I am more interrested in ftrace datas for the moment.
> 
> I use a x86_64 CPCI cpu board with linux-3.16.7 and
> xenomai-3.0-rc4 with I-Pipe tracer Options On

The first thing is to switch the I-pipe tracer off, at least during
runtime. You don't need that overhead if you do ftrace event tracing.
But I suspect the issue elsewhere.

> 
> My problem may comes from trace-cmd actually, because it uses nearly two
> cores full time during ftrace collect.
> I don't known what for ! I used it to be able to plot datas in kernelshark
> but its overhead is too much.
> 
> I will try with no external tools, (just /sys/kernel/debug/tracing/*
> controls).

I'm using trace-cmd without such problems. Can you share your .config?
Could you also try 3.14.44-x86-9 (that's where I ran ftrace recently).

> 
> I had to disable nmi (echo 0 > /proc/sys/kernel/nmi_watchdog) at boot to
> avoid hard Lockup kernel crash when system load is high (running dohell for
> exemple). I don't think it can be linked.

Shouldn't happen either. Maybe related, and the system is in a general
bad shape due to configuration or patch issues.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Antoine Durand]

Hi,

I found a new problem that must be the root cause of the others (I hope)
the linux system time is wrong, one second last 31 real seconds !

ftrace timing are wrong and may be nmi watchdog is fired because of that
wrong time too.

however the time run normally in /proc/driver/rtc
(the CPU board get a Dallas DS12887 / Motorola MC146818 compatible RTC)

I use the CLOCK_MONOTONIC clock in xenomai periodic task and it works well.
I-Pipe tracer timing are correct too.

in /proc/interrupts :
IRQ0 (timer) counter is not counting
LOC (local timer interrupts) is counting

in /proc/xenomai/irq :
[timer/0] is counting (following LOC value)

I'm trying to find why IRQ0 never happen.

Thanks.



2015-06-23 8:19 GMT+02:00 Jan Kiszka <jan.kiszka@siemens.com>:

> On 2015-06-22 22:47, Antoine Durand wrote:
> > Thank you,
> >
> > I really better understand the differences between the two traces
> sources.
> > Indeed I am more interrested in ftrace datas for the moment.
> >
> > I use a x86_64 CPCI cpu board with linux-3.16.7 and
> > xenomai-3.0-rc4 with I-Pipe tracer Options On
>
> The first thing is to switch the I-pipe tracer off, at least during
> runtime. You don't need that overhead if you do ftrace event tracing.
> But I suspect the issue elsewhere.
>
> >
> > My problem may comes from trace-cmd actually, because it uses nearly two
> > cores full time during ftrace collect.
> > I don't known what for ! I used it to be able to plot datas in
> kernelshark
> > but its overhead is too much.
> >
> > I will try with no external tools, (just /sys/kernel/debug/tracing/*
> > controls).
>
> I'm using trace-cmd without such problems. Can you share your .config?
> Could you also try 3.14.44-x86-9 (that's where I ran ftrace recently).
>
> >
> > I had to disable nmi (echo 0 > /proc/sys/kernel/nmi_watchdog) at boot to
> > avoid hard Lockup kernel crash when system load is high (running dohell
> for
> > exemple). I don't think it can be linked.
>
> Shouldn't happen either. Maybe related, and the system is in a general
> bad shape due to configuration or patch issues.
>
> Jan
>
> --
> Siemens AG, Corporate Technology, CT RTC ITP SES-DE
> Corporate Competence Center Embedded Linux
>

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Lennart Sorensen]

On Tue, Jun 23, 2015 at 12:09:09PM +0200, Antoine Durand wrote:
> I found a new problem that must be the root cause of the others (I hope)
> the linux system time is wrong, one second last 31 real seconds !
> 
> ftrace timing are wrong and may be nmi watchdog is fired because of that
> wrong time too.
> 
> however the time run normally in /proc/driver/rtc
> (the CPU board get a Dallas DS12887 / Motorola MC146818 compatible RTC)

Normally the RTC is only used to get time at boot and save it when the
system is off.  It usually has nothing at all to do with system time
while the system is running.

So having the rtc correct but system time wrong is perfectly plausible
if whatever is used for system time is broken.

> I use the CLOCK_MONOTONIC clock in xenomai periodic task and it works well.
> I-Pipe tracer timing are correct too.
> 
> in /proc/interrupts :
> IRQ0 (timer) counter is not counting
> LOC (local timer interrupts) is counting
> 
> in /proc/xenomai/irq :
> [timer/0] is counting (following LOC value)
> 
> I'm trying to find why IRQ0 never happen.

Are you running NO_HZ config?

-- 
Len Sorensen

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Jan Kiszka]

On 2015-06-23 16:11, Lennart Sorensen wrote:
> On Tue, Jun 23, 2015 at 12:09:09PM +0200, Antoine Durand wrote:
>> I found a new problem that must be the root cause of the others (I hope)
>> the linux system time is wrong, one second last 31 real seconds !
>>
>> ftrace timing are wrong and may be nmi watchdog is fired because of that
>> wrong time too.
>>
>> however the time run normally in /proc/driver/rtc
>> (the CPU board get a Dallas DS12887 / Motorola MC146818 compatible RTC)
> 
> Normally the RTC is only used to get time at boot and save it when the
> system is off.  It usually has nothing at all to do with system time
> while the system is running.
> 
> So having the rtc correct but system time wrong is perfectly plausible
> if whatever is used for system time is broken.
> 
>> I use the CLOCK_MONOTONIC clock in xenomai periodic task and it works well.
>> I-Pipe tracer timing are correct too.
>>
>> in /proc/interrupts :
>> IRQ0 (timer) counter is not counting
>> LOC (local timer interrupts) is counting
>>
>> in /proc/xenomai/irq :
>> [timer/0] is counting (following LOC value)
>>
>> I'm trying to find why IRQ0 never happen.
> 
> Are you running NO_HZ config?
> 

Huh, that would be interesting...

Also check your Linux clocksource
(/sys/devices/system/clocksource/clocksource0/current_clocksource). It
should be TSC on sane systems. If not, check the kernel log for
clocksource switches and related reason reports.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> On 2015-06-23 16:11, Lennart Sorensen wrote:
>> On Tue, Jun 23, 2015 at 12:09:09PM +0200, Antoine Durand wrote:
>>> I found a new problem that must be the root cause of the others (I
>>> hope)
>>> the linux system time is wrong, one second last 31 real seconds !
>>>
>>> ftrace timing are wrong and may be nmi watchdog is fired because of
>>> that
>>> wrong time too.
>>>
>>> however the time run normally in /proc/driver/rtc
>>> (the CPU board get a Dallas DS12887 / Motorola MC146818 compatible RTC)
>>
>> Normally the RTC is only used to get time at boot and save it when the
>> system is off.  It usually has nothing at all to do with system time
>> while the system is running.
>>
>> So having the rtc correct but system time wrong is perfectly plausible
>> if whatever is used for system time is broken.
>>
>>> I use the CLOCK_MONOTONIC clock in xenomai periodic task and it works
>>> well.
>>> I-Pipe tracer timing are correct too.
>>>
>>> in /proc/interrupts :
>>> IRQ0 (timer) counter is not counting
>>> LOC (local timer interrupts) is counting
>>>
>>> in /proc/xenomai/irq :
>>> [timer/0] is counting (following LOC value)
>>>
>>> I'm trying to find why IRQ0 never happen.
>>
>> Are you running NO_HZ config?
>>
>
> Huh, that would be interesting...
>
> Also check your Linux clocksource
> (/sys/devices/system/clocksource/clocksource0/current_clocksource). It
> should be TSC on sane systems. If not, check the kernel log for
> clocksource switches and related reason reports.

Using another clocksource than TSC should not be a problem, neither for
Linux, nor for Xenomai. That said I have not tested this in a long time.

-- 
                                            Gilles.
https://click-hack.org

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Antoine Durand]

The clocksource switching is due to the config option

CONFIG_NR_CPUS=2

the CPU is quad core and i would like to use only two of them to emulate a
cheaper board.

now the time run normally

i will check if the other problems are resolved.

Thanks


2015-06-23 16:27 GMT+02:00 Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org>:

>
> Jan Kiszka wrote:
> > On 2015-06-23 16:11, Lennart Sorensen wrote:
> >> On Tue, Jun 23, 2015 at 12:09:09PM +0200, Antoine Durand wrote:
> >>> I found a new problem that must be the root cause of the others (I
> >>> hope)
> >>> the linux system time is wrong, one second last 31 real seconds !
> >>>
> >>> ftrace timing are wrong and may be nmi watchdog is fired because of
> >>> that
> >>> wrong time too.
> >>>
> >>> however the time run normally in /proc/driver/rtc
> >>> (the CPU board get a Dallas DS12887 / Motorola MC146818 compatible RTC)
> >>
> >> Normally the RTC is only used to get time at boot and save it when the
> >> system is off.  It usually has nothing at all to do with system time
> >> while the system is running.
> >>
> >> So having the rtc correct but system time wrong is perfectly plausible
> >> if whatever is used for system time is broken.
> >>
> >>> I use the CLOCK_MONOTONIC clock in xenomai periodic task and it works
> >>> well.
> >>> I-Pipe tracer timing are correct too.
> >>>
> >>> in /proc/interrupts :
> >>> IRQ0 (timer) counter is not counting
> >>> LOC (local timer interrupts) is counting
> >>>
> >>> in /proc/xenomai/irq :
> >>> [timer/0] is counting (following LOC value)
> >>>
> >>> I'm trying to find why IRQ0 never happen.
> >>
> >> Are you running NO_HZ config?
> >>
> >
> > Huh, that would be interesting...
> >
> > Also check your Linux clocksource
> > (/sys/devices/system/clocksource/clocksource0/current_clocksource). It
> > should be TSC on sane systems. If not, check the kernel log for
> > clocksource switches and related reason reports.
>
> Using another clocksource than TSC should not be a problem, neither for
> Linux, nor for Xenomai. That said I have not tested this in a long time.
>
> --
>                                             Gilles.
> https://click-hack.org
>
>

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Jan Kiszka]

On 2015-06-23 16:41, Antoine Durand wrote:
> The clocksource switching is due to the config option
> 
> CONFIG_NR_CPUS=2
> 
> the CPU is quad core and i would like to use only two of them to emulate a
> cheaper board.
> 
> now the time run normally

Interesting. Maybe the kernel parameter maxcpus works better (it is
definitely more handy).

Jan

> 
> i will check if the other problems are resolved.
> 
> Thanks
> 
> 
> 2015-06-23 16:27 GMT+02:00 Gilles Chanteperdrix <
> gilles.chanteperdrix@xenomai.org>:
> 
>>
>> Jan Kiszka wrote:
>>> On 2015-06-23 16:11, Lennart Sorensen wrote:
>>>> On Tue, Jun 23, 2015 at 12:09:09PM +0200, Antoine Durand wrote:
>>>>> I found a new problem that must be the root cause of the others (I
>>>>> hope)
>>>>> the linux system time is wrong, one second last 31 real seconds !
>>>>>
>>>>> ftrace timing are wrong and may be nmi watchdog is fired because of
>>>>> that
>>>>> wrong time too.
>>>>>
>>>>> however the time run normally in /proc/driver/rtc
>>>>> (the CPU board get a Dallas DS12887 / Motorola MC146818 compatible RTC)
>>>>
>>>> Normally the RTC is only used to get time at boot and save it when the
>>>> system is off.  It usually has nothing at all to do with system time
>>>> while the system is running.
>>>>
>>>> So having the rtc correct but system time wrong is perfectly plausible
>>>> if whatever is used for system time is broken.
>>>>
>>>>> I use the CLOCK_MONOTONIC clock in xenomai periodic task and it works
>>>>> well.
>>>>> I-Pipe tracer timing are correct too.
>>>>>
>>>>> in /proc/interrupts :
>>>>> IRQ0 (timer) counter is not counting
>>>>> LOC (local timer interrupts) is counting
>>>>>
>>>>> in /proc/xenomai/irq :
>>>>> [timer/0] is counting (following LOC value)
>>>>>
>>>>> I'm trying to find why IRQ0 never happen.
>>>>
>>>> Are you running NO_HZ config?
>>>>
>>>
>>> Huh, that would be interesting...
>>>
>>> Also check your Linux clocksource
>>> (/sys/devices/system/clocksource/clocksource0/current_clocksource). It
>>> should be TSC on sane systems. If not, check the kernel log for
>>> clocksource switches and related reason reports.
>>
>> Using another clocksource than TSC should not be a problem, neither for
>> Linux, nor for Xenomai. That said I have not tested this in a long time.
>>
>> --
>>                                             Gilles.
>> https://click-hack.org
>>
>>
> 

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Gilles Chanteperdrix]

Antoine Durand wrote:
> The clocksource switching is due to the config option
>
> CONFIG_NR_CPUS=2
>
> the CPU is quad core and i would like to use only two of them to emulate a
> cheaper board.
>
> now the time run normally
>
> i will check if the other problems are resolved.

It would be interesting to know what clocksource your system switches to
when it does not work, because again, it should not be a problem.


-- 
                                            Gilles.
https://click-hack.org

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Antoine Durand]

It switched to refined-jiffies :

dmesg :

tsc: Fast TSC calibration using PIT
tsc: Detected 2100.042 MHz processor
PTP clock support registered
Switched to clocksource refined-jiffies
Switche to clocksource tsc
rtc_cmos: setting system clock to 2015-06-23 17:28:56 UTC (1435080536)
Clocksource tsc unstable (delta = 128069998 ns)
Switched to clocksource refined-jiffies



2015-06-23 16:46 GMT+02:00 Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org>:

>
> Antoine Durand wrote:
> > The clocksource switching is due to the config option
> >
> > CONFIG_NR_CPUS=2
> >
> > the CPU is quad core and i would like to use only two of them to emulate
> a
> > cheaper board.
> >
> > now the time run normally
> >
> > i will check if the other problems are resolved.
>
> It would be interesting to know what clocksource your system switches to
> when it does not work, because again, it should not be a problem.
>
>
> --
>                                             Gilles.
> https://click-hack.org
>
>

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Gilles Chanteperdrix]

Antoine Durand wrote:
> It switched to refined-jiffies :

Ok, indeed, this one is not supported by Xenomai, and will likely never
be. We support HPET and 8254 only.


-- 
                                            Gilles.
https://click-hack.org

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Jan Kiszka]

On 2015-06-23 16:49, Gilles Chanteperdrix wrote:
> 
> Antoine Durand wrote:
>> It switched to refined-jiffies :
> 
> Ok, indeed, this one is not supported by Xenomai, and will likely never
> be. We support HPET and 8254 only.

acpi_pm is fine as well.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Antoine Durand]

I don't need do disable nmi_watchdog anymore,
I can run dohell without any Hard Lockup kernel crash.
That's a good point !



2015-06-23 16:49 GMT+02:00 Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org>:

>
> Antoine Durand wrote:
> > It switched to refined-jiffies :
>
> Ok, indeed, this one is not supported by Xenomai, and will likely never
> be. We support HPET and 8254 only.
>
>
> --
>                                             Gilles.
> https://click-hack.org
>
>

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Lennart Sorensen]

On Tue, Jun 23, 2015 at 04:41:06PM +0200, Antoine Durand wrote:
> The clocksource switching is due to the config option
> 
> CONFIG_NR_CPUS=2
> 
> the CPU is quad core and i would like to use only two of them to emulate a
> cheaper board.
> 
> now the time run normally
> 
> i will check if the other problems are resolved.

Wouldn't booting with 'maxcpus=2' be good enough for that test and avoid
any breaking of the system?

-- 
Len Sorensen

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Antoine Durand]

booting with 'maxcpus=2' in my case has the same behavior than
CONFIG_NR_CPUS=2
the clocksource switche to refined-jiffies and system time is wrong (very
very slow)

I only get
refined-jiffies jiffies and tsc
in /sys/devices/system/clocksource/clocksource0/available_clocksource

the 8254 isn't seen.


2015-06-23 18:07 GMT+02:00 Lennart Sorensen <lsorense@csclub.uwaterloo.ca>:

> On Tue, Jun 23, 2015 at 04:41:06PM +0200, Antoine Durand wrote:
> > The clocksource switching is due to the config option
> >
> > CONFIG_NR_CPUS=2
> >
> > the CPU is quad core and i would like to use only two of them to emulate
> a
> > cheaper board.
> >
> > now the time run normally
> >
> > i will check if the other problems are resolved.
>
> Wouldn't booting with 'maxcpus=2' be good enough for that test and avoid
> any breaking of the system?
>
> --
> Len Sorensen
>

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Jan Kiszka]

On 2015-06-23 18:18, Antoine Durand wrote:
> booting with 'maxcpus=2' in my case has the same behavior than
> CONFIG_NR_CPUS=2
> the clocksource switche to refined-jiffies and system time is wrong (very
> very slow)

Ugh, reproduced something with a 4-core VM and maxcpus=2. I get a kernel
crash in __ipipe_ack_hrtimer_irq. Probably an online vs. possible cpus
issue. Will look into this.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Jan Kiszka]

On 2015-06-23 19:00, Jan Kiszka wrote:
> On 2015-06-23 18:18, Antoine Durand wrote:
>> booting with 'maxcpus=2' in my case has the same behavior than
>> CONFIG_NR_CPUS=2
>> the clocksource switche to refined-jiffies and system time is wrong (very
>> very slow)
> 
> Ugh, reproduced something with a 4-core VM and maxcpus=2. I get a kernel
> crash in __ipipe_ack_hrtimer_irq. Probably an online vs. possible cpus
> issue. Will look into this.

Interesting: maxcpus limits the number of cores the kernel will bring
online on its own, but it does not stop the kernel from detecting more,
keeping them recorded as offline. Now userland jumps in and may decide
to bring them online as well, which happens with my recent OpenSUSE
system, not with a much older one. And that means cpu hotplugging, which
does not work when Xenomai considers those CPUs as part of the RT set.
Still, excluding the additional ones per supported_cpus mask is not
sufficient.

Anyway, what works fine here is nr_cpus=2 - clocksource TSC is chosen,
and no crash. Could you check that as well?

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Antoine Durand]

 nr_cpus=2 and maxcpus=2 have the same consequences on my system,
tsc is considered unstable and linux switch to refined-jiffies
because I've nothing better available as clocksource than refined-jiffies
(except for tsc) on that board.

2015-06-23 19:23 GMT+02:00 Jan Kiszka <jan.kiszka@siemens.com>:

> On 2015-06-23 19:00, Jan Kiszka wrote:
> > On 2015-06-23 18:18, Antoine Durand wrote:
> >> booting with 'maxcpus=2' in my case has the same behavior than
> >> CONFIG_NR_CPUS=2
> >> the clocksource switche to refined-jiffies and system time is wrong
> (very
> >> very slow)
> >
> > Ugh, reproduced something with a 4-core VM and maxcpus=2. I get a kernel
> > crash in __ipipe_ack_hrtimer_irq. Probably an online vs. possible cpus
> > issue. Will look into this.
>
> Interesting: maxcpus limits the number of cores the kernel will bring
> online on its own, but it does not stop the kernel from detecting more,
> keeping them recorded as offline. Now userland jumps in and may decide
> to bring them online as well, which happens with my recent OpenSUSE
> system, not with a much older one. And that means cpu hotplugging, which
> does not work when Xenomai considers those CPUs as part of the RT set.
> Still, excluding the additional ones per supported_cpus mask is not
> sufficient.
>
> Anyway, what works fine here is nr_cpus=2 - clocksource TSC is chosen,
> and no crash. Could you check that as well?
>
> Jan
>
> --
> Siemens AG, Corporate Technology, CT RTC ITP SES-DE
> Corporate Competence Center Embedded Linux
>

Re: [Xenomai] I-Pipe Tracer and linux ftrace

[Jan Kiszka]

[restoring cc list]

On 2015-06-23 16:25, Antoine Durand wrote:
> I can "repair" the linux system time by forcing clock source from
> refined-jiffies to tsc
> 
> $ echo tsc >
> /sys/devices/system/clocksource/clocksource0/current_clocksource
> 
> Linux used refined-jiffies due to those messages at boot :
> 
> tsc: Fast TSC calibration using PIT
> tsc: Detected 2100.042 MHz processor
> PTP clock support registered
> Switched to clocksource refined-jiffies
> Switche to clocksource tsc
> rtc_cmos: setting system clock to 2015-06-23 17:28:56 UTC (1435080536)
> Clocksource tsc unstable (delta = 128069998 ns)
> Switched to clocksource refined-jiffies

jiffies - that is not good and explains the unstable timebase.

> 
> I don't know why.
> I've a modern core i7 CPU with constant_tsc flag (in /proc/cpuinfo)
> 
> 
> here are my HZ config:
> 
> CONFIG_NO_HZ_COMMON=y
> # CONFIG_HZ_PERIODIC is not set
> CONFIG_NO_HZ_IDLE=y
> CONFIG_NO_HZ=y

OK, CONFIG_NO_HZ, but that is set here as well. I was thinking of
NO_HZ_FULL when Lennart mentioned it. That's not tested and may cause
troubles.

> # CONFIG_RCU_FAST_NO_HZ is not set
> # CONFIG_HZ_100 is not set
> # CONFIG_HZ_250 is not set
> # CONFIG_HZ_300 is not set
> CONFIG_HZ_1000=y
> CONFIG_HZ=1000
> 

We don't have the full picture of your config yet: Do you use CPU
frequency scaling? Or some specific powersaving modes?

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux