* [Xenomai] I-Pipe Tracer and linux ftrace @ 2015-06-22 16:06 Antoine Durand 2015-06-22 17:20 ` Jan Kiszka 0 siblings, 1 reply; 21+ messages in thread From: Antoine Durand @ 2015-06-22 16:06 UTC (permalink / raw) To: xenomai Hi, I would like to check my RT threads scheduling, so I started here : http://xenomai.org/2014/06/using-the-i-pipe-tracer/ I put 100000 in /proc/ipipe/trace/back_trace_points I ran my tasks, i write 1 in frozen, and then I read the frozen file. I'm not sure what kind of datas can be found in I-Pipe Tracer (/proc/ipipe/trace/frozen) and what is in linux ftrace (/sys/kernel/debug/tracing). For instance, I "think" I'm interested in the __xnsched_run trace points, but there isn't informations about the CPU on which the task is running on in I-Pipe tracer output. So I checked in linux ftrace. But when i activate linux ftrace by writing 1 to /sys/kernel/debug/tracing/tracing_on, the RT tasks behaviors became weird (the task with the higher priority just toggle an output that I check on an oscilloscope, this task is unable to make a clean 10ms periodic square signal with tracing_on activated !). so I did not find any source to be able to observe my tasks without crippling side effect. So here are my questions ! Are linux ftrace datas relevant for xenomai tasks (there is a lot of cobalt_core entries so I guess yes) ? Is there any tool to plot I-Pipe tracer datas from /proc/ipipe/trace/frozen ? (I tried to use trace-cmd and kernelshark to plot linux ftrace but if what is ploted is true, the side effect with xenomai is awful !) What datas must be checked from I-Pipe Tracer and what must be checked from linux ftrace ? That's a lot of questions in one, I'm sorry, it's more that I'am a bit lost. Thank you for your help. Bob x86_64 linux-3.16.7 xenomai-3.0-rc4 with I-Pipe tracer Options On ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-22 16:06 [Xenomai] I-Pipe Tracer and linux ftrace Antoine Durand @ 2015-06-22 17:20 ` Jan Kiszka 2015-06-22 20:47 ` Antoine Durand 0 siblings, 1 reply; 21+ messages in thread From: Jan Kiszka @ 2015-06-22 17:20 UTC (permalink / raw) To: Antoine Durand, xenomai On 2015-06-22 18:06, Antoine Durand wrote: > Hi, > > I would like to check my RT threads scheduling, so I started here : > http://xenomai.org/2014/06/using-the-i-pipe-tracer/ > > I put 100000 in /proc/ipipe/trace/back_trace_points > I ran my tasks, i write 1 in frozen, and then I read the frozen file. > > I'm not sure what kind of datas can be found in I-Pipe Tracer > (/proc/ipipe/trace/frozen) and what is in linux ftrace > (/sys/kernel/debug/tracing). For instance, I "think" I'm interested in the > __xnsched_run trace points, but there isn't informations about the CPU on > which the task is running on in I-Pipe tracer output. > > So I checked in linux ftrace. > > But when i activate linux ftrace by writing 1 to > /sys/kernel/debug/tracing/tracing_on, the RT tasks behaviors became weird > (the task with the higher priority just toggle an output that I check on an > oscilloscope, this task is unable to make a clean 10ms periodic square > signal with tracing_on activated !). Can you share more details of of your setup (patch version, kernel config, detailed tracing setup, ...)? > > so I did not find any source to be able to observe my tasks without > crippling side effect. > > So here are my questions ! > > > > Are linux ftrace datas relevant for xenomai tasks (there is a lot of > cobalt_core entries so I guess yes) ? Yes, they are. The Xenomai 3 cobalt core is pretty well instrumented for event tracing. > > Is there any tool to plot I-Pipe tracer datas from /proc/ipipe/trace/frozen > ? No, that data is so far for textual consumption only. > (I tried to use trace-cmd and kernelshark to plot linux ftrace but if what > is ploted is true, the side effect with xenomai is awful !) We need to understand that side effects, they should not be that massive, rather much less invasive than that of the I-pipe tracer. > > What datas must be checked from I-Pipe Tracer and what must be checked from > linux ftrace ? The I-pipe tracer is for detailed analysis, specifically on kernel function level. If you have a latency spot or a bug on which you can trigger a trace freeze, then you can look into the details around this event. The I-pipe tracer is actually quite similar to ftrace's function tracer, but it has some additional data about the context (RT or Linux). Maybe we can merge both one day. Event tracing via ftrace with additional Xenomai tracepoints is for a broader overview. For analyzing scheduling orders and domain migrations, identifying application activities based on their syscalls and driver interactions. It's the first stop when you want to understand your system activities as a whole - probably what you are after here. Jan > > > > That's a lot of questions in one, I'm sorry, it's more that I'am a bit lost. > > Thank you for your help. > > Bob > > x86_64 > linux-3.16.7 > xenomai-3.0-rc4 with I-Pipe tracer Options On > -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-22 17:20 ` Jan Kiszka @ 2015-06-22 20:47 ` Antoine Durand 2015-06-23 6:19 ` Jan Kiszka 0 siblings, 1 reply; 21+ messages in thread From: Antoine Durand @ 2015-06-22 20:47 UTC (permalink / raw) To: Jan Kiszka, xenomai Thank you, I really better understand the differences between the two traces sources. Indeed I am more interrested in ftrace datas for the moment. I use a x86_64 CPCI cpu board with linux-3.16.7 and xenomai-3.0-rc4 with I-Pipe tracer Options On My problem may comes from trace-cmd actually, because it uses nearly two cores full time during ftrace collect. I don't known what for ! I used it to be able to plot datas in kernelshark but its overhead is too much. I will try with no external tools, (just /sys/kernel/debug/tracing/* controls). I had to disable nmi (echo 0 > /proc/sys/kernel/nmi_watchdog) at boot to avoid hard Lockup kernel crash when system load is high (running dohell for exemple). I don't think it can be linked. 2015-06-22 19:20 GMT+02:00 Jan Kiszka <jan.kiszka@siemens.com>: > On 2015-06-22 18:06, Antoine Durand wrote: > > Hi, > > > > I would like to check my RT threads scheduling, so I started here : > > http://xenomai.org/2014/06/using-the-i-pipe-tracer/ > > > > I put 100000 in /proc/ipipe/trace/back_trace_points > > I ran my tasks, i write 1 in frozen, and then I read the frozen file. > > > > I'm not sure what kind of datas can be found in I-Pipe Tracer > > (/proc/ipipe/trace/frozen) and what is in linux ftrace > > (/sys/kernel/debug/tracing). For instance, I "think" I'm interested in > the > > __xnsched_run trace points, but there isn't informations about the CPU on > > which the task is running on in I-Pipe tracer output. > > > > So I checked in linux ftrace. > > > > But when i activate linux ftrace by writing 1 to > > /sys/kernel/debug/tracing/tracing_on, the RT tasks behaviors became weird > > (the task with the higher priority just toggle an output that I check on > an > > oscilloscope, this task is unable to make a clean 10ms periodic square > > signal with tracing_on activated !). > > Can you share more details of of your setup (patch version, kernel > config, detailed tracing setup, ...)? > > > > > so I did not find any source to be able to observe my tasks without > > crippling side effect. > > > > So here are my questions ! > > > > > > > > Are linux ftrace datas relevant for xenomai tasks (there is a lot of > > cobalt_core entries so I guess yes) ? > > Yes, they are. The Xenomai 3 cobalt core is pretty well instrumented for > event tracing. > > > > > Is there any tool to plot I-Pipe tracer datas from > /proc/ipipe/trace/frozen > > ? > > No, that data is so far for textual consumption only. > > > (I tried to use trace-cmd and kernelshark to plot linux ftrace but if > what > > is ploted is true, the side effect with xenomai is awful !) > > We need to understand that side effects, they should not be that > massive, rather much less invasive than that of the I-pipe tracer. > > > > > What datas must be checked from I-Pipe Tracer and what must be checked > from > > linux ftrace ? > > The I-pipe tracer is for detailed analysis, specifically on kernel > function level. If you have a latency spot or a bug on which you can > trigger a trace freeze, then you can look into the details around this > event. The I-pipe tracer is actually quite similar to ftrace's function > tracer, but it has some additional data about the context (RT or Linux). > Maybe we can merge both one day. > > Event tracing via ftrace with additional Xenomai tracepoints is for a > broader overview. For analyzing scheduling orders and domain migrations, > identifying application activities based on their syscalls and driver > interactions. It's the first stop when you want to understand your > system activities as a whole - probably what you are after here. > > Jan > > > > > > > > > That's a lot of questions in one, I'm sorry, it's more that I'am a bit > lost. > > > > Thank you for your help. > > > > Bob > > > > x86_64 > > linux-3.16.7 > > xenomai-3.0-rc4 with I-Pipe tracer Options On > > > > -- > Siemens AG, Corporate Technology, CT RTC ITP SES-DE > Corporate Competence Center Embedded Linux > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-22 20:47 ` Antoine Durand @ 2015-06-23 6:19 ` Jan Kiszka 2015-06-23 10:09 ` Antoine Durand 0 siblings, 1 reply; 21+ messages in thread From: Jan Kiszka @ 2015-06-23 6:19 UTC (permalink / raw) To: Antoine Durand, xenomai On 2015-06-22 22:47, Antoine Durand wrote: > Thank you, > > I really better understand the differences between the two traces sources. > Indeed I am more interrested in ftrace datas for the moment. > > I use a x86_64 CPCI cpu board with linux-3.16.7 and > xenomai-3.0-rc4 with I-Pipe tracer Options On The first thing is to switch the I-pipe tracer off, at least during runtime. You don't need that overhead if you do ftrace event tracing. But I suspect the issue elsewhere. > > My problem may comes from trace-cmd actually, because it uses nearly two > cores full time during ftrace collect. > I don't known what for ! I used it to be able to plot datas in kernelshark > but its overhead is too much. > > I will try with no external tools, (just /sys/kernel/debug/tracing/* > controls). I'm using trace-cmd without such problems. Can you share your .config? Could you also try 3.14.44-x86-9 (that's where I ran ftrace recently). > > I had to disable nmi (echo 0 > /proc/sys/kernel/nmi_watchdog) at boot to > avoid hard Lockup kernel crash when system load is high (running dohell for > exemple). I don't think it can be linked. Shouldn't happen either. Maybe related, and the system is in a general bad shape due to configuration or patch issues. Jan -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-23 6:19 ` Jan Kiszka @ 2015-06-23 10:09 ` Antoine Durand 2015-06-23 14:11 ` Lennart Sorensen 0 siblings, 1 reply; 21+ messages in thread From: Antoine Durand @ 2015-06-23 10:09 UTC (permalink / raw) To: Jan Kiszka, xenomai Hi, I found a new problem that must be the root cause of the others (I hope) the linux system time is wrong, one second last 31 real seconds ! ftrace timing are wrong and may be nmi watchdog is fired because of that wrong time too. however the time run normally in /proc/driver/rtc (the CPU board get a Dallas DS12887 / Motorola MC146818 compatible RTC) I use the CLOCK_MONOTONIC clock in xenomai periodic task and it works well. I-Pipe tracer timing are correct too. in /proc/interrupts : IRQ0 (timer) counter is not counting LOC (local timer interrupts) is counting in /proc/xenomai/irq : [timer/0] is counting (following LOC value) I'm trying to find why IRQ0 never happen. Thanks. 2015-06-23 8:19 GMT+02:00 Jan Kiszka <jan.kiszka@siemens.com>: > On 2015-06-22 22:47, Antoine Durand wrote: > > Thank you, > > > > I really better understand the differences between the two traces > sources. > > Indeed I am more interrested in ftrace datas for the moment. > > > > I use a x86_64 CPCI cpu board with linux-3.16.7 and > > xenomai-3.0-rc4 with I-Pipe tracer Options On > > The first thing is to switch the I-pipe tracer off, at least during > runtime. You don't need that overhead if you do ftrace event tracing. > But I suspect the issue elsewhere. > > > > > My problem may comes from trace-cmd actually, because it uses nearly two > > cores full time during ftrace collect. > > I don't known what for ! I used it to be able to plot datas in > kernelshark > > but its overhead is too much. > > > > I will try with no external tools, (just /sys/kernel/debug/tracing/* > > controls). > > I'm using trace-cmd without such problems. Can you share your .config? > Could you also try 3.14.44-x86-9 (that's where I ran ftrace recently). > > > > > I had to disable nmi (echo 0 > /proc/sys/kernel/nmi_watchdog) at boot to > > avoid hard Lockup kernel crash when system load is high (running dohell > for > > exemple). I don't think it can be linked. > > Shouldn't happen either. Maybe related, and the system is in a general > bad shape due to configuration or patch issues. > > Jan > > -- > Siemens AG, Corporate Technology, CT RTC ITP SES-DE > Corporate Competence Center Embedded Linux > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-23 10:09 ` Antoine Durand @ 2015-06-23 14:11 ` Lennart Sorensen 2015-06-23 14:15 ` Jan Kiszka 0 siblings, 1 reply; 21+ messages in thread From: Lennart Sorensen @ 2015-06-23 14:11 UTC (permalink / raw) To: Antoine Durand; +Cc: Jan Kiszka, xenomai On Tue, Jun 23, 2015 at 12:09:09PM +0200, Antoine Durand wrote: > I found a new problem that must be the root cause of the others (I hope) > the linux system time is wrong, one second last 31 real seconds ! > > ftrace timing are wrong and may be nmi watchdog is fired because of that > wrong time too. > > however the time run normally in /proc/driver/rtc > (the CPU board get a Dallas DS12887 / Motorola MC146818 compatible RTC) Normally the RTC is only used to get time at boot and save it when the system is off. It usually has nothing at all to do with system time while the system is running. So having the rtc correct but system time wrong is perfectly plausible if whatever is used for system time is broken. > I use the CLOCK_MONOTONIC clock in xenomai periodic task and it works well. > I-Pipe tracer timing are correct too. > > in /proc/interrupts : > IRQ0 (timer) counter is not counting > LOC (local timer interrupts) is counting > > in /proc/xenomai/irq : > [timer/0] is counting (following LOC value) > > I'm trying to find why IRQ0 never happen. Are you running NO_HZ config? -- Len Sorensen ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-23 14:11 ` Lennart Sorensen @ 2015-06-23 14:15 ` Jan Kiszka 2015-06-23 14:27 ` Gilles Chanteperdrix [not found] ` <CAOL1OyNELX_6g8fOSS8wuJ+6-TzrBXjoxn3wH-Gb8u-HKHFqSw@mail.gmail.com> 0 siblings, 2 replies; 21+ messages in thread From: Jan Kiszka @ 2015-06-23 14:15 UTC (permalink / raw) To: Lennart Sorensen, Antoine Durand; +Cc: xenomai On 2015-06-23 16:11, Lennart Sorensen wrote: > On Tue, Jun 23, 2015 at 12:09:09PM +0200, Antoine Durand wrote: >> I found a new problem that must be the root cause of the others (I hope) >> the linux system time is wrong, one second last 31 real seconds ! >> >> ftrace timing are wrong and may be nmi watchdog is fired because of that >> wrong time too. >> >> however the time run normally in /proc/driver/rtc >> (the CPU board get a Dallas DS12887 / Motorola MC146818 compatible RTC) > > Normally the RTC is only used to get time at boot and save it when the > system is off. It usually has nothing at all to do with system time > while the system is running. > > So having the rtc correct but system time wrong is perfectly plausible > if whatever is used for system time is broken. > >> I use the CLOCK_MONOTONIC clock in xenomai periodic task and it works well. >> I-Pipe tracer timing are correct too. >> >> in /proc/interrupts : >> IRQ0 (timer) counter is not counting >> LOC (local timer interrupts) is counting >> >> in /proc/xenomai/irq : >> [timer/0] is counting (following LOC value) >> >> I'm trying to find why IRQ0 never happen. > > Are you running NO_HZ config? > Huh, that would be interesting... Also check your Linux clocksource (/sys/devices/system/clocksource/clocksource0/current_clocksource). It should be TSC on sane systems. If not, check the kernel log for clocksource switches and related reason reports. Jan -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-23 14:15 ` Jan Kiszka @ 2015-06-23 14:27 ` Gilles Chanteperdrix 2015-06-23 14:41 ` Antoine Durand [not found] ` <CAOL1OyNELX_6g8fOSS8wuJ+6-TzrBXjoxn3wH-Gb8u-HKHFqSw@mail.gmail.com> 1 sibling, 1 reply; 21+ messages in thread From: Gilles Chanteperdrix @ 2015-06-23 14:27 UTC (permalink / raw) To: Jan Kiszka; +Cc: xenomai, Antoine Durand Jan Kiszka wrote: > On 2015-06-23 16:11, Lennart Sorensen wrote: >> On Tue, Jun 23, 2015 at 12:09:09PM +0200, Antoine Durand wrote: >>> I found a new problem that must be the root cause of the others (I >>> hope) >>> the linux system time is wrong, one second last 31 real seconds ! >>> >>> ftrace timing are wrong and may be nmi watchdog is fired because of >>> that >>> wrong time too. >>> >>> however the time run normally in /proc/driver/rtc >>> (the CPU board get a Dallas DS12887 / Motorola MC146818 compatible RTC) >> >> Normally the RTC is only used to get time at boot and save it when the >> system is off. It usually has nothing at all to do with system time >> while the system is running. >> >> So having the rtc correct but system time wrong is perfectly plausible >> if whatever is used for system time is broken. >> >>> I use the CLOCK_MONOTONIC clock in xenomai periodic task and it works >>> well. >>> I-Pipe tracer timing are correct too. >>> >>> in /proc/interrupts : >>> IRQ0 (timer) counter is not counting >>> LOC (local timer interrupts) is counting >>> >>> in /proc/xenomai/irq : >>> [timer/0] is counting (following LOC value) >>> >>> I'm trying to find why IRQ0 never happen. >> >> Are you running NO_HZ config? >> > > Huh, that would be interesting... > > Also check your Linux clocksource > (/sys/devices/system/clocksource/clocksource0/current_clocksource). It > should be TSC on sane systems. If not, check the kernel log for > clocksource switches and related reason reports. Using another clocksource than TSC should not be a problem, neither for Linux, nor for Xenomai. That said I have not tested this in a long time. -- Gilles. https://click-hack.org ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-23 14:27 ` Gilles Chanteperdrix @ 2015-06-23 14:41 ` Antoine Durand 2015-06-23 14:43 ` Jan Kiszka ` (2 more replies) 0 siblings, 3 replies; 21+ messages in thread From: Antoine Durand @ 2015-06-23 14:41 UTC (permalink / raw) To: Gilles Chanteperdrix, Jan Kiszka, Lennart Sorensen, xenomai The clocksource switching is due to the config option CONFIG_NR_CPUS=2 the CPU is quad core and i would like to use only two of them to emulate a cheaper board. now the time run normally i will check if the other problems are resolved. Thanks 2015-06-23 16:27 GMT+02:00 Gilles Chanteperdrix < gilles.chanteperdrix@xenomai.org>: > > Jan Kiszka wrote: > > On 2015-06-23 16:11, Lennart Sorensen wrote: > >> On Tue, Jun 23, 2015 at 12:09:09PM +0200, Antoine Durand wrote: > >>> I found a new problem that must be the root cause of the others (I > >>> hope) > >>> the linux system time is wrong, one second last 31 real seconds ! > >>> > >>> ftrace timing are wrong and may be nmi watchdog is fired because of > >>> that > >>> wrong time too. > >>> > >>> however the time run normally in /proc/driver/rtc > >>> (the CPU board get a Dallas DS12887 / Motorola MC146818 compatible RTC) > >> > >> Normally the RTC is only used to get time at boot and save it when the > >> system is off. It usually has nothing at all to do with system time > >> while the system is running. > >> > >> So having the rtc correct but system time wrong is perfectly plausible > >> if whatever is used for system time is broken. > >> > >>> I use the CLOCK_MONOTONIC clock in xenomai periodic task and it works > >>> well. > >>> I-Pipe tracer timing are correct too. > >>> > >>> in /proc/interrupts : > >>> IRQ0 (timer) counter is not counting > >>> LOC (local timer interrupts) is counting > >>> > >>> in /proc/xenomai/irq : > >>> [timer/0] is counting (following LOC value) > >>> > >>> I'm trying to find why IRQ0 never happen. > >> > >> Are you running NO_HZ config? > >> > > > > Huh, that would be interesting... > > > > Also check your Linux clocksource > > (/sys/devices/system/clocksource/clocksource0/current_clocksource). It > > should be TSC on sane systems. If not, check the kernel log for > > clocksource switches and related reason reports. > > Using another clocksource than TSC should not be a problem, neither for > Linux, nor for Xenomai. That said I have not tested this in a long time. > > -- > Gilles. > https://click-hack.org > > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-23 14:41 ` Antoine Durand @ 2015-06-23 14:43 ` Jan Kiszka 2015-06-23 14:46 ` Gilles Chanteperdrix 2015-06-23 16:07 ` Lennart Sorensen 2 siblings, 0 replies; 21+ messages in thread From: Jan Kiszka @ 2015-06-23 14:43 UTC (permalink / raw) To: Antoine Durand, Gilles Chanteperdrix, Lennart Sorensen, xenomai On 2015-06-23 16:41, Antoine Durand wrote: > The clocksource switching is due to the config option > > CONFIG_NR_CPUS=2 > > the CPU is quad core and i would like to use only two of them to emulate a > cheaper board. > > now the time run normally Interesting. Maybe the kernel parameter maxcpus works better (it is definitely more handy). Jan > > i will check if the other problems are resolved. > > Thanks > > > 2015-06-23 16:27 GMT+02:00 Gilles Chanteperdrix < > gilles.chanteperdrix@xenomai.org>: > >> >> Jan Kiszka wrote: >>> On 2015-06-23 16:11, Lennart Sorensen wrote: >>>> On Tue, Jun 23, 2015 at 12:09:09PM +0200, Antoine Durand wrote: >>>>> I found a new problem that must be the root cause of the others (I >>>>> hope) >>>>> the linux system time is wrong, one second last 31 real seconds ! >>>>> >>>>> ftrace timing are wrong and may be nmi watchdog is fired because of >>>>> that >>>>> wrong time too. >>>>> >>>>> however the time run normally in /proc/driver/rtc >>>>> (the CPU board get a Dallas DS12887 / Motorola MC146818 compatible RTC) >>>> >>>> Normally the RTC is only used to get time at boot and save it when the >>>> system is off. It usually has nothing at all to do with system time >>>> while the system is running. >>>> >>>> So having the rtc correct but system time wrong is perfectly plausible >>>> if whatever is used for system time is broken. >>>> >>>>> I use the CLOCK_MONOTONIC clock in xenomai periodic task and it works >>>>> well. >>>>> I-Pipe tracer timing are correct too. >>>>> >>>>> in /proc/interrupts : >>>>> IRQ0 (timer) counter is not counting >>>>> LOC (local timer interrupts) is counting >>>>> >>>>> in /proc/xenomai/irq : >>>>> [timer/0] is counting (following LOC value) >>>>> >>>>> I'm trying to find why IRQ0 never happen. >>>> >>>> Are you running NO_HZ config? >>>> >>> >>> Huh, that would be interesting... >>> >>> Also check your Linux clocksource >>> (/sys/devices/system/clocksource/clocksource0/current_clocksource). It >>> should be TSC on sane systems. If not, check the kernel log for >>> clocksource switches and related reason reports. >> >> Using another clocksource than TSC should not be a problem, neither for >> Linux, nor for Xenomai. That said I have not tested this in a long time. >> >> -- >> Gilles. >> https://click-hack.org >> >> > -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-23 14:41 ` Antoine Durand 2015-06-23 14:43 ` Jan Kiszka @ 2015-06-23 14:46 ` Gilles Chanteperdrix 2015-06-23 14:47 ` Antoine Durand 2015-06-23 16:07 ` Lennart Sorensen 2 siblings, 1 reply; 21+ messages in thread From: Gilles Chanteperdrix @ 2015-06-23 14:46 UTC (permalink / raw) To: Antoine Durand; +Cc: Jan Kiszka, xenomai Antoine Durand wrote: > The clocksource switching is due to the config option > > CONFIG_NR_CPUS=2 > > the CPU is quad core and i would like to use only two of them to emulate a > cheaper board. > > now the time run normally > > i will check if the other problems are resolved. It would be interesting to know what clocksource your system switches to when it does not work, because again, it should not be a problem. -- Gilles. https://click-hack.org ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-23 14:46 ` Gilles Chanteperdrix @ 2015-06-23 14:47 ` Antoine Durand 2015-06-23 14:49 ` Gilles Chanteperdrix 0 siblings, 1 reply; 21+ messages in thread From: Antoine Durand @ 2015-06-23 14:47 UTC (permalink / raw) To: Gilles Chanteperdrix; +Cc: Jan Kiszka, xenomai It switched to refined-jiffies : dmesg : tsc: Fast TSC calibration using PIT tsc: Detected 2100.042 MHz processor PTP clock support registered Switched to clocksource refined-jiffies Switche to clocksource tsc rtc_cmos: setting system clock to 2015-06-23 17:28:56 UTC (1435080536) Clocksource tsc unstable (delta = 128069998 ns) Switched to clocksource refined-jiffies 2015-06-23 16:46 GMT+02:00 Gilles Chanteperdrix < gilles.chanteperdrix@xenomai.org>: > > Antoine Durand wrote: > > The clocksource switching is due to the config option > > > > CONFIG_NR_CPUS=2 > > > > the CPU is quad core and i would like to use only two of them to emulate > a > > cheaper board. > > > > now the time run normally > > > > i will check if the other problems are resolved. > > It would be interesting to know what clocksource your system switches to > when it does not work, because again, it should not be a problem. > > > -- > Gilles. > https://click-hack.org > > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-23 14:47 ` Antoine Durand @ 2015-06-23 14:49 ` Gilles Chanteperdrix 2015-06-23 14:59 ` Jan Kiszka 2015-06-23 15:01 ` Antoine Durand 0 siblings, 2 replies; 21+ messages in thread From: Gilles Chanteperdrix @ 2015-06-23 14:49 UTC (permalink / raw) To: Antoine Durand; +Cc: Jan Kiszka, xenomai Antoine Durand wrote: > It switched to refined-jiffies : Ok, indeed, this one is not supported by Xenomai, and will likely never be. We support HPET and 8254 only. -- Gilles. https://click-hack.org ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-23 14:49 ` Gilles Chanteperdrix @ 2015-06-23 14:59 ` Jan Kiszka 2015-06-23 15:01 ` Antoine Durand 1 sibling, 0 replies; 21+ messages in thread From: Jan Kiszka @ 2015-06-23 14:59 UTC (permalink / raw) To: Gilles Chanteperdrix, Antoine Durand; +Cc: xenomai On 2015-06-23 16:49, Gilles Chanteperdrix wrote: > > Antoine Durand wrote: >> It switched to refined-jiffies : > > Ok, indeed, this one is not supported by Xenomai, and will likely never > be. We support HPET and 8254 only. acpi_pm is fine as well. Jan -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-23 14:49 ` Gilles Chanteperdrix 2015-06-23 14:59 ` Jan Kiszka @ 2015-06-23 15:01 ` Antoine Durand 1 sibling, 0 replies; 21+ messages in thread From: Antoine Durand @ 2015-06-23 15:01 UTC (permalink / raw) To: Gilles Chanteperdrix; +Cc: Jan Kiszka, xenomai I don't need do disable nmi_watchdog anymore, I can run dohell without any Hard Lockup kernel crash. That's a good point ! 2015-06-23 16:49 GMT+02:00 Gilles Chanteperdrix < gilles.chanteperdrix@xenomai.org>: > > Antoine Durand wrote: > > It switched to refined-jiffies : > > Ok, indeed, this one is not supported by Xenomai, and will likely never > be. We support HPET and 8254 only. > > > -- > Gilles. > https://click-hack.org > > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-23 14:41 ` Antoine Durand 2015-06-23 14:43 ` Jan Kiszka 2015-06-23 14:46 ` Gilles Chanteperdrix @ 2015-06-23 16:07 ` Lennart Sorensen 2015-06-23 16:18 ` Antoine Durand 2 siblings, 1 reply; 21+ messages in thread From: Lennart Sorensen @ 2015-06-23 16:07 UTC (permalink / raw) To: Antoine Durand; +Cc: Jan Kiszka, xenomai On Tue, Jun 23, 2015 at 04:41:06PM +0200, Antoine Durand wrote: > The clocksource switching is due to the config option > > CONFIG_NR_CPUS=2 > > the CPU is quad core and i would like to use only two of them to emulate a > cheaper board. > > now the time run normally > > i will check if the other problems are resolved. Wouldn't booting with 'maxcpus=2' be good enough for that test and avoid any breaking of the system? -- Len Sorensen ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-23 16:07 ` Lennart Sorensen @ 2015-06-23 16:18 ` Antoine Durand 2015-06-23 17:00 ` Jan Kiszka 0 siblings, 1 reply; 21+ messages in thread From: Antoine Durand @ 2015-06-23 16:18 UTC (permalink / raw) To: Lennart Sorensen; +Cc: Jan Kiszka, xenomai booting with 'maxcpus=2' in my case has the same behavior than CONFIG_NR_CPUS=2 the clocksource switche to refined-jiffies and system time is wrong (very very slow) I only get refined-jiffies jiffies and tsc in /sys/devices/system/clocksource/clocksource0/available_clocksource the 8254 isn't seen. 2015-06-23 18:07 GMT+02:00 Lennart Sorensen <lsorense@csclub.uwaterloo.ca>: > On Tue, Jun 23, 2015 at 04:41:06PM +0200, Antoine Durand wrote: > > The clocksource switching is due to the config option > > > > CONFIG_NR_CPUS=2 > > > > the CPU is quad core and i would like to use only two of them to emulate > a > > cheaper board. > > > > now the time run normally > > > > i will check if the other problems are resolved. > > Wouldn't booting with 'maxcpus=2' be good enough for that test and avoid > any breaking of the system? > > -- > Len Sorensen > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-23 16:18 ` Antoine Durand @ 2015-06-23 17:00 ` Jan Kiszka 2015-06-23 17:23 ` Jan Kiszka 0 siblings, 1 reply; 21+ messages in thread From: Jan Kiszka @ 2015-06-23 17:00 UTC (permalink / raw) To: Antoine Durand, Lennart Sorensen; +Cc: xenomai On 2015-06-23 18:18, Antoine Durand wrote: > booting with 'maxcpus=2' in my case has the same behavior than > CONFIG_NR_CPUS=2 > the clocksource switche to refined-jiffies and system time is wrong (very > very slow) Ugh, reproduced something with a 4-core VM and maxcpus=2. I get a kernel crash in __ipipe_ack_hrtimer_irq. Probably an online vs. possible cpus issue. Will look into this. Jan -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-23 17:00 ` Jan Kiszka @ 2015-06-23 17:23 ` Jan Kiszka 2015-06-24 7:30 ` Antoine Durand 0 siblings, 1 reply; 21+ messages in thread From: Jan Kiszka @ 2015-06-23 17:23 UTC (permalink / raw) To: Antoine Durand, Lennart Sorensen; +Cc: xenomai On 2015-06-23 19:00, Jan Kiszka wrote: > On 2015-06-23 18:18, Antoine Durand wrote: >> booting with 'maxcpus=2' in my case has the same behavior than >> CONFIG_NR_CPUS=2 >> the clocksource switche to refined-jiffies and system time is wrong (very >> very slow) > > Ugh, reproduced something with a 4-core VM and maxcpus=2. I get a kernel > crash in __ipipe_ack_hrtimer_irq. Probably an online vs. possible cpus > issue. Will look into this. Interesting: maxcpus limits the number of cores the kernel will bring online on its own, but it does not stop the kernel from detecting more, keeping them recorded as offline. Now userland jumps in and may decide to bring them online as well, which happens with my recent OpenSUSE system, not with a much older one. And that means cpu hotplugging, which does not work when Xenomai considers those CPUs as part of the RT set. Still, excluding the additional ones per supported_cpus mask is not sufficient. Anyway, what works fine here is nr_cpus=2 - clocksource TSC is chosen, and no crash. Could you check that as well? Jan -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [Xenomai] I-Pipe Tracer and linux ftrace 2015-06-23 17:23 ` Jan Kiszka @ 2015-06-24 7:30 ` Antoine Durand 0 siblings, 0 replies; 21+ messages in thread From: Antoine Durand @ 2015-06-24 7:30 UTC (permalink / raw) To: Jan Kiszka; +Cc: xenomai nr_cpus=2 and maxcpus=2 have the same consequences on my system, tsc is considered unstable and linux switch to refined-jiffies because I've nothing better available as clocksource than refined-jiffies (except for tsc) on that board. 2015-06-23 19:23 GMT+02:00 Jan Kiszka <jan.kiszka@siemens.com>: > On 2015-06-23 19:00, Jan Kiszka wrote: > > On 2015-06-23 18:18, Antoine Durand wrote: > >> booting with 'maxcpus=2' in my case has the same behavior than > >> CONFIG_NR_CPUS=2 > >> the clocksource switche to refined-jiffies and system time is wrong > (very > >> very slow) > > > > Ugh, reproduced something with a 4-core VM and maxcpus=2. I get a kernel > > crash in __ipipe_ack_hrtimer_irq. Probably an online vs. possible cpus > > issue. Will look into this. > > Interesting: maxcpus limits the number of cores the kernel will bring > online on its own, but it does not stop the kernel from detecting more, > keeping them recorded as offline. Now userland jumps in and may decide > to bring them online as well, which happens with my recent OpenSUSE > system, not with a much older one. And that means cpu hotplugging, which > does not work when Xenomai considers those CPUs as part of the RT set. > Still, excluding the additional ones per supported_cpus mask is not > sufficient. > > Anyway, what works fine here is nr_cpus=2 - clocksource TSC is chosen, > and no crash. Could you check that as well? > > Jan > > -- > Siemens AG, Corporate Technology, CT RTC ITP SES-DE > Corporate Competence Center Embedded Linux > ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <CAOL1OyNELX_6g8fOSS8wuJ+6-TzrBXjoxn3wH-Gb8u-HKHFqSw@mail.gmail.com>]
* Re: [Xenomai] I-Pipe Tracer and linux ftrace [not found] ` <CAOL1OyNELX_6g8fOSS8wuJ+6-TzrBXjoxn3wH-Gb8u-HKHFqSw@mail.gmail.com> @ 2015-06-23 14:42 ` Jan Kiszka 0 siblings, 0 replies; 21+ messages in thread From: Jan Kiszka @ 2015-06-23 14:42 UTC (permalink / raw) To: Antoine Durand; +Cc: Xenomai [restoring cc list] On 2015-06-23 16:25, Antoine Durand wrote: > I can "repair" the linux system time by forcing clock source from > refined-jiffies to tsc > > $ echo tsc > > /sys/devices/system/clocksource/clocksource0/current_clocksource > > Linux used refined-jiffies due to those messages at boot : > > tsc: Fast TSC calibration using PIT > tsc: Detected 2100.042 MHz processor > PTP clock support registered > Switched to clocksource refined-jiffies > Switche to clocksource tsc > rtc_cmos: setting system clock to 2015-06-23 17:28:56 UTC (1435080536) > Clocksource tsc unstable (delta = 128069998 ns) > Switched to clocksource refined-jiffies jiffies - that is not good and explains the unstable timebase. > > I don't know why. > I've a modern core i7 CPU with constant_tsc flag (in /proc/cpuinfo) > > > here are my HZ config: > > CONFIG_NO_HZ_COMMON=y > # CONFIG_HZ_PERIODIC is not set > CONFIG_NO_HZ_IDLE=y > CONFIG_NO_HZ=y OK, CONFIG_NO_HZ, but that is set here as well. I was thinking of NO_HZ_FULL when Lennart mentioned it. That's not tested and may cause troubles. > # CONFIG_RCU_FAST_NO_HZ is not set > # CONFIG_HZ_100 is not set > # CONFIG_HZ_250 is not set > # CONFIG_HZ_300 is not set > CONFIG_HZ_1000=y > CONFIG_HZ=1000 > We don't have the full picture of your config yet: Do you use CPU frequency scaling? Or some specific powersaving modes? Jan -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2015-06-24 7:30 UTC | newest] Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-06-22 16:06 [Xenomai] I-Pipe Tracer and linux ftrace Antoine Durand 2015-06-22 17:20 ` Jan Kiszka 2015-06-22 20:47 ` Antoine Durand 2015-06-23 6:19 ` Jan Kiszka 2015-06-23 10:09 ` Antoine Durand 2015-06-23 14:11 ` Lennart Sorensen 2015-06-23 14:15 ` Jan Kiszka 2015-06-23 14:27 ` Gilles Chanteperdrix 2015-06-23 14:41 ` Antoine Durand 2015-06-23 14:43 ` Jan Kiszka 2015-06-23 14:46 ` Gilles Chanteperdrix 2015-06-23 14:47 ` Antoine Durand 2015-06-23 14:49 ` Gilles Chanteperdrix 2015-06-23 14:59 ` Jan Kiszka 2015-06-23 15:01 ` Antoine Durand 2015-06-23 16:07 ` Lennart Sorensen 2015-06-23 16:18 ` Antoine Durand 2015-06-23 17:00 ` Jan Kiszka 2015-06-23 17:23 ` Jan Kiszka 2015-06-24 7:30 ` Antoine Durand [not found] ` <CAOL1OyNELX_6g8fOSS8wuJ+6-TzrBXjoxn3wH-Gb8u-HKHFqSw@mail.gmail.com> 2015-06-23 14:42 ` Jan Kiszka
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.