* Xenomai crash in timer IRQ
@ 2022-12-02 20:45 Sri Subramanian
2022-12-03 16:06 ` Philippe Gerum
0 siblings, 1 reply; 13+ messages in thread
From: Sri Subramanian @ 2022-12-02 20:45 UTC (permalink / raw)
To: xenomai
Hello,
I have an application waking up every 500us (waiting with an
oob_read() on an EVL monotonic clock) and taking action.
The application also has an interface to enable/disable NTP and set
the clock manually. The system will run over days without any issues.
But if NTP is disabled, and the clock is set manually, then NTP is
re-enabled, a short time after (order of minutes), the OS crashes and
the system reboots. It is reproducible every time.
Should the monotonic clock be affected by an RTC clock setting? I was
assuming not, but I am not so sure now.
Hardware environment: Raspberry PI CM4 with a standard CM4-IO board.
Xenomai tag: 5.15.77-evl-rebase
Stack and config as below:
Thanks for any help,
Sri
==========
[ 363.094612] CPU: 3 PID: 706 Comm: EcatCyclic-tid_ Tainted: G
C 5.15.35-xeno-v5.15.77-evl1-rebase #1
[ 363.094617] Hardware name: Raspberry Pi Compute Module 4 Rev 1.0 (DT)
[ 363.094619] IRQ stage: Linux
[ 363.094621] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 363.094626] pc : evl_get_timer_overruns+0x1b8/0x1c4
[ 363.094630] lr : evl_get_timer_overruns+0x54/0x1c4
[ 363.094634] sp : ffffffc00a283c40
[ 363.094635] x29: ffffffc00a283c40 x28: 0000000000000000 x27: ffffff810103acc0
[ 363.094642] x26: ffffff810103ac00 x25: ffffffc008deb320 x24: ffffffc008c6aae8
[ 363.094647] x23: 0000000000000000 x22: ffffff81fefde3e0 x21: 0000000000000001
[ 363.094652] x20: 000000548931dc32 x19: ffffff810103ac00 x18: 0000000000000000
[ 363.094658] x17: 0000000000000000 x16: 0000000000000001 x15: 0000000000000000
[ 363.094662] x14: 0000000000000000 x13: ffffffc00898a790 x12: 00000000fa83b2da
[ 363.094667] x11: 0000000000000029 x10: 0000000000000930 x9 : ffffffc00a283c30
[ 363.094673] x8 : ffffff811c496810 x7 : 000000000000b7df x6 : 00000004a520e245
[ 363.094678] x5 : 00ffffffffffffff x4 : 0011bdb637657ad8 x3 : 000000000007a120
[ 363.094683] x2 : 0000000000000001 x1 : 0000000000000005 x0 : 0000000000000005
[ 363.094688] Call trace:
[ 363.094691] evl_get_timer_overruns+0x1b8/0x1c4
[ 363.094695] timerfd_oob_read+0x200/0x290
[ 363.094701] EVL_read+0x50/0xd0
[ 363.094703] handle_pipelined_syscall+0x218/0x260
[ 363.094707] __pipeline_syscall+0xe8/0x2a0
[ 363.094710] pipeline_syscall+0x64/0x110
[ 363.094713] el0_svc_common.constprop.0+0x48/0x12c
[ 363.094719] do_el0_svc+0x28/0x90
[ 363.094722] el0_svc+0x4c/0xd0
[ 363.094728] el0t_64_sync_handler+0xa4/0x130
[ 363.094733] el0t_64_sync+0x1a0/0x1a4
[ 363.094736] ---[ end trace ac73e88b51a9ac19 ]---
[ 363.094750] EVL: EcatCyclic-tid_706 resuming out-of-band [pid=706,
excpt=3, evl_get_timer_overruns+0x1bc/0x1c4]
[ 363.094764] Unable to handle kernel paging request at virtual
address dead000000000108
[ 363.094767] Mem abort info:
[ 363.094768] ESR = 0x96000044
[ 363.094769] EC = 0x25: DABT (current EL), IL = 32 bits
[ 363.094772] SET = 0, FnV = 0
[ 363.094774] EA = 0, S1PTW = 0
[ 363.094775] FSC = 0x04: level 0 translation fault
[ 363.094778] Data abort info:
[ 363.094779] ISV = 0, ISS = 0x00000044
[ 363.094780] CM = 0, WnR = 1
[ 363.094782] [dead000000000108] address between user and kernel address ranges
[ 363.094786] Internal error: Oops: 96000044 [#1] PREEMPT SMP
[ 363.094789] Modules linked in: cdc_mbim cdc_wdm cdc_ncm cdc_ether
bcm2835_v4l2(C) bcm2835_mmal_vchiq(C) videobuf2_vmalloc
videobuf2_memops videobuf2_v4l2 videobuf2_common raspberrypi_hwmon
videodev mc vc4 cec drm_kms_helper snd_soc_core snd_pcm_dmaengine
snd_pcm snd_timer snd syscopyarea sysfillrect sysimgblt fb_sys_fops
i2c_gpio i2c_algo_bit ax88179_178a uio_pdrv_genirq uio nvmem_rmem
cfg80211 rfkill sch_fq_codel fuse drm drm_panel_orientation_quirks
backlight configfs ipv6
[ 363.094828] CPU: 3 PID: 706 Comm: EcatCyclic-tid_ Tainted: G
WC 5.15.35-xeno-v5.15.77-evl1-rebase #1
[ 363.094832] Hardware name: Raspberry Pi Compute Module 4 Rev 1.0 (DT)
[ 363.094834] IRQ stage: EVL
[ 363.094835] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 363.094839] pc : evl_get_timer_overruns+0xfc/0x1c4
[ 363.094843] lr : evl_get_timer_overruns+0x54/0x1c4
[ 363.094846] sp : ffffffc00a283c40
[ 363.094847] x29: ffffffc00a283c40 x28: 0000000000000000 x27: ffffff810103acc0
[ 363.094853] x26: ffffff810103ac00 x25: ffffffc008deb320 x24: ffffffc008c6aae8
[ 363.094858] x23: 0000000000000000 x22: ffffff81fefde3e0 x21: 0000000000000001
[ 363.094863] x20: 000000548931dc32 x19: ffffff810103ac00 x18: 0000000000000000
[ 363.094868] x17: 0000000000000000 x16: 0000000000000001 x15: 0000000000000000
[ 363.094873] x14: 0000000000000000 x13: ffffffc00898a790 x12: 00000000fa83b2da
[ 363.094878] x11: 0000000000000029 x10: 0000000000000930 x9 : ffffffc00a283c30
[ 363.094883] x8 : ffffff811c496810 x7 : 000000000000b7df x6 : 00000004a520e245
[ 363.094887] x5 : 00ffffffffffffff x4 : 0011bdb637657ad8 x3 : 000000000007a120
[ 363.094892] x2 : dead000000000100 x1 : 0000000000000005 x0 : dead000000000122
[ 363.094897] Call trace:
[ 363.094899] evl_get_timer_overruns+0xfc/0x1c4
[ 363.094903] timerfd_oob_read+0x200/0x290
[ 363.094907] EVL_read+0x50/0xd0
[ 363.094910] handle_pipelined_syscall+0x218/0x260
[ 363.094913] __pipeline_syscall+0xe8/0x2a0
[ 363.094916] pipeline_syscall+0x64/0x110
[ 363.094919] el0_svc_common.constprop.0+0x48/0x12c
[ 363.094923] do_el0_svc+0x28/0x90
[ 363.094926] el0_svc+0x4c/0xd0
[ 363.094931] el0t_64_sync_handler+0xa4/0x130
[ 363.094935] el0t_64_sync+0x1a0/0x1a4
[ 363.094940] Code: 0a010000 7100101f 54000621 a9410262 (f9000440)
========
CONFIG_HAVE_ARCH_EVL=y
CONFIG_EVL=y
CONFIG_EVL_SCHED_QUOTA=y
# CONFIG_EVL_SCHED_TP is not set
# CONFIG_EVL_HIGH_PERCPU_CONCURRENCY is not set
CONFIG_EVL_RUNSTATS=y
# CONFIG_EVL_NET is not set
CONFIG_EVL_COREMEM_SIZE=2048
CONFIG_EVL_NR_THREADS=256
CONFIG_EVL_NR_MONITORS=512
CONFIG_EVL_NR_CLOCKS=8
CONFIG_EVL_NR_XBUFS=16
CONFIG_EVL_NR_PROXIES=64
CONFIG_EVL_NR_OBSERVABLES=64
CONFIG_EVL_LATENCY_USER=1500
CONFIG_EVL_LATENCY_KERNEL=1500
CONFIG_EVL_LATENCY_IRQ=500
CONFIG_EVL_DEBUG=y
CONFIG_EVL_DEBUG_CORE=y
CONFIG_EVL_DEBUG_MEMORY=y
CONFIG_EVL_DEBUG_WOLI=y
# CONFIG_EVL_DEBUG_NET is not set
CONFIG_EVL_WATCHDOG=y
CONFIG_EVL_WATCHDOG_TIMEOUT=4
CONFIG_EVL_LATMUS=y
CONFIG_EVL_HECTIC=y
==========
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Xenomai crash in timer IRQ
2022-12-02 20:45 Xenomai crash in timer IRQ Sri Subramanian
@ 2022-12-03 16:06 ` Philippe Gerum
2022-12-03 16:33 ` Philippe Gerum
0 siblings, 1 reply; 13+ messages in thread
From: Philippe Gerum @ 2022-12-03 16:06 UTC (permalink / raw)
To: Sri Subramanian; +Cc: xenomai
Sri Subramanian <sridhar.subramanian@gmail.com> writes:
> Hello,
>
> I have an application waking up every 500us (waiting with an
> oob_read() on an EVL monotonic clock) and taking action.
>
> The application also has an interface to enable/disable NTP and set
> the clock manually. The system will run over days without any issues.
> But if NTP is disabled, and the clock is set manually, then NTP is
> re-enabled, a short time after (order of minutes), the OS crashes and
> the system reboots. It is reproducible every time.
>
> Should the monotonic clock be affected by an RTC clock setting? I was
> assuming not, but I am not so sure now.
>
Not EVL_CLOCK_MONOTONIC.
> Hardware environment: Raspberry PI CM4 with a standard CM4-IO board.
> Xenomai tag: 5.15.77-evl-rebase
> Stack and config as below:
>
> Thanks for any help,
> Sri
> ==========
> [ 363.094612] CPU: 3 PID: 706 Comm: EcatCyclic-tid_ Tainted: G
> C 5.15.35-xeno-v5.15.77-evl1-rebase #1
> [ 363.094617] Hardware name: Raspberry Pi Compute Module 4 Rev 1.0 (DT)
> [ 363.094619] IRQ stage: Linux
> [ 363.094621] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> [ 363.094626] pc : evl_get_timer_overruns+0x1b8/0x1c4
> [ 363.094630] lr : evl_get_timer_overruns+0x54/0x1c4
> [ 363.094634] sp : ffffffc00a283c40
> [ 363.094635] x29: ffffffc00a283c40 x28: 0000000000000000 x27: ffffff810103acc0
> [ 363.094642] x26: ffffff810103ac00 x25: ffffffc008deb320 x24: ffffffc008c6aae8
> [ 363.094647] x23: 0000000000000000 x22: ffffff81fefde3e0 x21: 0000000000000001
> [ 363.094652] x20: 000000548931dc32 x19: ffffff810103ac00 x18: 0000000000000000
> [ 363.094658] x17: 0000000000000000 x16: 0000000000000001 x15: 0000000000000000
> [ 363.094662] x14: 0000000000000000 x13: ffffffc00898a790 x12: 00000000fa83b2da
> [ 363.094667] x11: 0000000000000029 x10: 0000000000000930 x9 : ffffffc00a283c30
> [ 363.094673] x8 : ffffff811c496810 x7 : 000000000000b7df x6 : 00000004a520e245
> [ 363.094678] x5 : 00ffffffffffffff x4 : 0011bdb637657ad8 x3 : 000000000007a120
> [ 363.094683] x2 : 0000000000000001 x1 : 0000000000000005 x0 : 0000000000000005
> [ 363.094688] Call trace:
> [ 363.094691] evl_get_timer_overruns+0x1b8/0x1c4
> [ 363.094695] timerfd_oob_read+0x200/0x290
> [ 363.094701] EVL_read+0x50/0xd0
> [ 363.094703] handle_pipelined_syscall+0x218/0x260
> [ 363.094707] __pipeline_syscall+0xe8/0x2a0
> [ 363.094710] pipeline_syscall+0x64/0x110
> [ 363.094713] el0_svc_common.constprop.0+0x48/0x12c
> [ 363.094719] do_el0_svc+0x28/0x90
> [ 363.094722] el0_svc+0x4c/0xd0
> [ 363.094728] el0t_64_sync_handler+0xa4/0x130
> [ 363.094733] el0t_64_sync+0x1a0/0x1a4
> [ 363.094736] ---[ end trace ac73e88b51a9ac19 ]---
> [ 363.094750] EVL: EcatCyclic-tid_706 resuming out-of-band [pid=706,
> excpt=3, evl_get_timer_overruns+0x1bc/0x1c4]
> [ 363.094764] Unable to handle kernel paging request at virtual
> address dead000000000108
> [ 363.094767] Mem abort info:
> [ 363.094768] ESR = 0x96000044
> [ 363.094769] EC = 0x25: DABT (current EL), IL = 32 bits
> [ 363.094772] SET = 0, FnV = 0
> [ 363.094774] EA = 0, S1PTW = 0
> [ 363.094775] FSC = 0x04: level 0 translation fault
> [ 363.094778] Data abort info:
> [ 363.094779] ISV = 0, ISS = 0x00000044
> [ 363.094780] CM = 0, WnR = 1
> [ 363.094782] [dead000000000108] address between user and kernel address ranges
> [ 363.094786] Internal error: Oops: 96000044 [#1] PREEMPT SMP
> [ 363.094789] Modules linked in: cdc_mbim cdc_wdm cdc_ncm cdc_ether
> bcm2835_v4l2(C) bcm2835_mmal_vchiq(C) videobuf2_vmalloc
> videobuf2_memops videobuf2_v4l2 videobuf2_common raspberrypi_hwmon
> videodev mc vc4 cec drm_kms_helper snd_soc_core snd_pcm_dmaengine
> snd_pcm snd_timer snd syscopyarea sysfillrect sysimgblt fb_sys_fops
> i2c_gpio i2c_algo_bit ax88179_178a uio_pdrv_genirq uio nvmem_rmem
> cfg80211 rfkill sch_fq_codel fuse drm drm_panel_orientation_quirks
> backlight configfs ipv6
> [ 363.094828] CPU: 3 PID: 706 Comm: EcatCyclic-tid_ Tainted: G
> WC 5.15.35-xeno-v5.15.77-evl1-rebase #1
> [ 363.094832] Hardware name: Raspberry Pi Compute Module 4 Rev 1.0 (DT)
> [ 363.094834] IRQ stage: EVL
> [ 363.094835] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> [ 363.094839] pc : evl_get_timer_overruns+0xfc/0x1c4
> [ 363.094843] lr : evl_get_timer_overruns+0x54/0x1c4
> [ 363.094846] sp : ffffffc00a283c40
> [ 363.094847] x29: ffffffc00a283c40 x28: 0000000000000000 x27: ffffff810103acc0
> [ 363.094853] x26: ffffff810103ac00 x25: ffffffc008deb320 x24: ffffffc008c6aae8
> [ 363.094858] x23: 0000000000000000 x22: ffffff81fefde3e0 x21: 0000000000000001
> [ 363.094863] x20: 000000548931dc32 x19: ffffff810103ac00 x18: 0000000000000000
> [ 363.094868] x17: 0000000000000000 x16: 0000000000000001 x15: 0000000000000000
> [ 363.094873] x14: 0000000000000000 x13: ffffffc00898a790 x12: 00000000fa83b2da
> [ 363.094878] x11: 0000000000000029 x10: 0000000000000930 x9 : ffffffc00a283c30
> [ 363.094883] x8 : ffffff811c496810 x7 : 000000000000b7df x6 : 00000004a520e245
> [ 363.094887] x5 : 00ffffffffffffff x4 : 0011bdb637657ad8 x3 : 000000000007a120
> [ 363.094892] x2 : dead000000000100 x1 : 0000000000000005 x0 : dead000000000122
> [ 363.094897] Call trace:
> [ 363.094899] evl_get_timer_overruns+0xfc/0x1c4
> [ 363.094903] timerfd_oob_read+0x200/0x290
> [ 363.094907] EVL_read+0x50/0xd0
> [ 363.094910] handle_pipelined_syscall+0x218/0x260
> [ 363.094913] __pipeline_syscall+0xe8/0x2a0
> [ 363.094916] pipeline_syscall+0x64/0x110
> [ 363.094919] el0_svc_common.constprop.0+0x48/0x12c
> [ 363.094923] do_el0_svc+0x28/0x90
> [ 363.094926] el0_svc+0x4c/0xd0
> [ 363.094931] el0t_64_sync_handler+0xa4/0x130
> [ 363.094935] el0t_64_sync+0x1a0/0x1a4
> [ 363.094940] Code: 0a010000 7100101f 54000621 a9410262 (f9000440)
>
> ========
> CONFIG_HAVE_ARCH_EVL=y
> CONFIG_EVL=y
> CONFIG_EVL_SCHED_QUOTA=y
> # CONFIG_EVL_SCHED_TP is not set
> # CONFIG_EVL_HIGH_PERCPU_CONCURRENCY is not set
> CONFIG_EVL_RUNSTATS=y
> # CONFIG_EVL_NET is not set
> CONFIG_EVL_COREMEM_SIZE=2048
> CONFIG_EVL_NR_THREADS=256
> CONFIG_EVL_NR_MONITORS=512
> CONFIG_EVL_NR_CLOCKS=8
> CONFIG_EVL_NR_XBUFS=16
> CONFIG_EVL_NR_PROXIES=64
> CONFIG_EVL_NR_OBSERVABLES=64
> CONFIG_EVL_LATENCY_USER=1500
> CONFIG_EVL_LATENCY_KERNEL=1500
> CONFIG_EVL_LATENCY_IRQ=500
> CONFIG_EVL_DEBUG=y
> CONFIG_EVL_DEBUG_CORE=y
> CONFIG_EVL_DEBUG_MEMORY=y
> CONFIG_EVL_DEBUG_WOLI=y
> # CONFIG_EVL_DEBUG_NET is not set
> CONFIG_EVL_WATCHDOG=y
> CONFIG_EVL_WATCHDOG_TIMEOUT=4
> CONFIG_EVL_LATMUS=y
> CONFIG_EVL_HECTIC=y
> ==========
--
Philippe.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Xenomai crash in timer IRQ
2022-12-03 16:06 ` Philippe Gerum
@ 2022-12-03 16:33 ` Philippe Gerum
2022-12-07 21:00 ` Sri Subramanian
0 siblings, 1 reply; 13+ messages in thread
From: Philippe Gerum @ 2022-12-03 16:33 UTC (permalink / raw)
To: Sri Subramanian; +Cc: xenomai
Philippe Gerum <rpm@xenomai.org> writes:
> Sri Subramanian <sridhar.subramanian@gmail.com> writes:
>
>> Hello,
>>
>> I have an application waking up every 500us (waiting with an
>> oob_read() on an EVL monotonic clock) and taking action.
>>
>> The application also has an interface to enable/disable NTP and set
>> the clock manually. The system will run over days without any issues.
>> But if NTP is disabled, and the clock is set manually, then NTP is
>> re-enabled, a short time after (order of minutes), the OS crashes and
>> the system reboots. It is reproducible every time.
>>
I cannot reproduce this using a plain hwclock -s while latmus
runs.
>> Should the monotonic clock be affected by an RTC clock setting? I was
>> assuming not, but I am not so sure now.
>>
>
> Not EVL_CLOCK_MONOTONIC.
>
>> Hardware environment: Raspberry PI CM4 with a standard CM4-IO board.
>> Xenomai tag: 5.15.77-evl-rebase
>> Stack and config as below:
>>
>> Thanks for any help,
>> Sri
>> ==========
>> [ 363.094612] CPU: 3 PID: 706 Comm: EcatCyclic-tid_ Tainted: G
>> C 5.15.35-xeno-v5.15.77-evl1-rebase #1
>> [ 363.094617] Hardware name: Raspberry Pi Compute Module 4 Rev 1.0 (DT)
>> [ 363.094619] IRQ stage: Linux
>> [ 363.094621] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
>> [ 363.094626] pc : evl_get_timer_overruns+0x1b8/0x1c4
>> [ 363.094630] lr : evl_get_timer_overruns+0x54/0x1c4
>> [ 363.094634] sp : ffffffc00a283c40
>> [ 363.094635] x29: ffffffc00a283c40 x28: 0000000000000000 x27: ffffff810103acc0
>> [ 363.094642] x26: ffffff810103ac00 x25: ffffffc008deb320 x24: ffffffc008c6aae8
>> [ 363.094647] x23: 0000000000000000 x22: ffffff81fefde3e0 x21: 0000000000000001
>> [ 363.094652] x20: 000000548931dc32 x19: ffffff810103ac00 x18: 0000000000000000
>> [ 363.094658] x17: 0000000000000000 x16: 0000000000000001 x15: 0000000000000000
>> [ 363.094662] x14: 0000000000000000 x13: ffffffc00898a790 x12: 00000000fa83b2da
>> [ 363.094667] x11: 0000000000000029 x10: 0000000000000930 x9 : ffffffc00a283c30
>> [ 363.094673] x8 : ffffff811c496810 x7 : 000000000000b7df x6 : 00000004a520e245
>> [ 363.094678] x5 : 00ffffffffffffff x4 : 0011bdb637657ad8 x3 : 000000000007a120
>> [ 363.094683] x2 : 0000000000000001 x1 : 0000000000000005 x0 : 0000000000000005
>> [ 363.094688] Call trace:
>> [ 363.094691] evl_get_timer_overruns+0x1b8/0x1c4
>> [ 363.094695] timerfd_oob_read+0x200/0x290
Since the crash is easily reproducible, you should be able to trace it
directly into the kernel code, i.e. from evl_get_timer_overruns(), to
figure out which line is causing this. Or you could disassemble your
vmlinux image compiled with CONFIG_DEBUG_INFO set, and look for a
statement which matches $pc as reported in the register dump.
I pushed a change addressing a potential load tearing issue in the timer
code to the next-evl-rebase. I don't think this fixes the issue you
observed, but at the very least you may want that change in.
Next step for you would be to enable the ftracer (using next-evl-rebase,
it contains fixes you need for tracing), then post a snapshot of the
global trace buffer until the crash happens. ftrace_dump_on_oops may be
useful if you cannot access the machine after the oops, with a bit of
luck the snapshot should be sent to the console stream.
--
Philippe.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Xenomai crash in timer IRQ
2022-12-03 16:33 ` Philippe Gerum
@ 2022-12-07 21:00 ` Sri Subramanian
2022-12-08 7:53 ` Philippe Gerum
0 siblings, 1 reply; 13+ messages in thread
From: Sri Subramanian @ 2022-12-07 21:00 UTC (permalink / raw)
To: Philippe Gerum; +Cc: xenomai
Hi Philippe,
With next-evl-rebase, I am not able to reproduce the crash. Instead
the system reboots (from the watchdog) or hangs (if I disable the
watchdog).
I'll try going back to the original setup and try other experiments --
maybe try to reproduce this with a test program or attach a debugger
to get more data.
Sri
On Sat, Dec 3, 2022 at 8:43 AM Philippe Gerum <rpm@xenomai.org> wrote:
>
>
> Philippe Gerum <rpm@xenomai.org> writes:
>
> > Sri Subramanian <sridhar.subramanian@gmail.com> writes:
> >
> >> Hello,
> >>
> >> I have an application waking up every 500us (waiting with an
> >> oob_read() on an EVL monotonic clock) and taking action.
> >>
> >> The application also has an interface to enable/disable NTP and set
> >> the clock manually. The system will run over days without any issues.
> >> But if NTP is disabled, and the clock is set manually, then NTP is
> >> re-enabled, a short time after (order of minutes), the OS crashes and
> >> the system reboots. It is reproducible every time.
> >>
>
> I cannot reproduce this using a plain hwclock -s while latmus
> runs.
>
> >> Should the monotonic clock be affected by an RTC clock setting? I was
> >> assuming not, but I am not so sure now.
> >>
> >
> > Not EVL_CLOCK_MONOTONIC.
> >
> >> Hardware environment: Raspberry PI CM4 with a standard CM4-IO board.
> >> Xenomai tag: 5.15.77-evl-rebase
> >> Stack and config as below:
> >>
> >> Thanks for any help,
> >> Sri
> >> ==========
> >> [ 363.094612] CPU: 3 PID: 706 Comm: EcatCyclic-tid_ Tainted: G
> >> C 5.15.35-xeno-v5.15.77-evl1-rebase #1
> >> [ 363.094617] Hardware name: Raspberry Pi Compute Module 4 Rev 1.0 (DT)
> >> [ 363.094619] IRQ stage: Linux
> >> [ 363.094621] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> >> [ 363.094626] pc : evl_get_timer_overruns+0x1b8/0x1c4
> >> [ 363.094630] lr : evl_get_timer_overruns+0x54/0x1c4
> >> [ 363.094634] sp : ffffffc00a283c40
> >> [ 363.094635] x29: ffffffc00a283c40 x28: 0000000000000000 x27: ffffff810103acc0
> >> [ 363.094642] x26: ffffff810103ac00 x25: ffffffc008deb320 x24: ffffffc008c6aae8
> >> [ 363.094647] x23: 0000000000000000 x22: ffffff81fefde3e0 x21: 0000000000000001
> >> [ 363.094652] x20: 000000548931dc32 x19: ffffff810103ac00 x18: 0000000000000000
> >> [ 363.094658] x17: 0000000000000000 x16: 0000000000000001 x15: 0000000000000000
> >> [ 363.094662] x14: 0000000000000000 x13: ffffffc00898a790 x12: 00000000fa83b2da
> >> [ 363.094667] x11: 0000000000000029 x10: 0000000000000930 x9 : ffffffc00a283c30
> >> [ 363.094673] x8 : ffffff811c496810 x7 : 000000000000b7df x6 : 00000004a520e245
> >> [ 363.094678] x5 : 00ffffffffffffff x4 : 0011bdb637657ad8 x3 : 000000000007a120
> >> [ 363.094683] x2 : 0000000000000001 x1 : 0000000000000005 x0 : 0000000000000005
> >> [ 363.094688] Call trace:
> >> [ 363.094691] evl_get_timer_overruns+0x1b8/0x1c4
> >> [ 363.094695] timerfd_oob_read+0x200/0x290
>
> Since the crash is easily reproducible, you should be able to trace it
> directly into the kernel code, i.e. from evl_get_timer_overruns(), to
> figure out which line is causing this. Or you could disassemble your
> vmlinux image compiled with CONFIG_DEBUG_INFO set, and look for a
> statement which matches $pc as reported in the register dump.
>
> I pushed a change addressing a potential load tearing issue in the timer
> code to the next-evl-rebase. I don't think this fixes the issue you
> observed, but at the very least you may want that change in.
>
> Next step for you would be to enable the ftracer (using next-evl-rebase,
> it contains fixes you need for tracing), then post a snapshot of the
> global trace buffer until the crash happens. ftrace_dump_on_oops may be
> useful if you cannot access the machine after the oops, with a bit of
> luck the snapshot should be sent to the console stream.
>
> --
> Philippe.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Xenomai crash in timer IRQ
2022-12-07 21:00 ` Sri Subramanian
@ 2022-12-08 7:53 ` Philippe Gerum
2022-12-09 17:05 ` Sri Subramanian
2023-05-31 20:40 ` Sri Subramanian
0 siblings, 2 replies; 13+ messages in thread
From: Philippe Gerum @ 2022-12-08 7:53 UTC (permalink / raw)
To: Sri Subramanian; +Cc: xenomai
Sri Subramanian <sridhar.subramanian@gmail.com> writes:
> Hi Philippe,
>
> With next-evl-rebase, I am not able to reproduce the crash. Instead
> the system reboots (from the watchdog) or hangs (if I disable the
> watchdog).
>
> I'll try going back to the original setup and try other experiments --
> maybe try to reproduce this with a test program or attach a debugger
> to get more data.
>
Ack, at this point it should be possible to reproduce this bug with a
simple test case if you think the core is doing something wrong when the
kernel wallclock is set.
If you have custom drivers, you may want to check the lifetime of any
standalone evl_timer structs it might define for correctness, to make
sure there is no stale timers being reprogrammed (unlikely, but worth a
check anyway).
--
Philippe.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Xenomai crash in timer IRQ
2022-12-08 7:53 ` Philippe Gerum
@ 2022-12-09 17:05 ` Sri Subramanian
2022-12-19 21:51 ` Sri Subramanian
2023-05-31 20:40 ` Sri Subramanian
1 sibling, 1 reply; 13+ messages in thread
From: Sri Subramanian @ 2022-12-09 17:05 UTC (permalink / raw)
To: Philippe Gerum; +Cc: xenomai
[-- Attachment #1: Type: text/plain, Size: 1374 bytes --]
Hi Philippe,
I was able to reproduce one crash with a trace on evl events! If there
are other specific events you want me to include, please let me know.
I can only get it to hang with no output (after an overnight run) if I
include all events.
To answer your questions, we don't have any custom drivers. I'll check
the application for stale timers. I had no luck with reproducing the
crash with a simple test but will continue to pursue it.
Sri
On Wed, Dec 7, 2022 at 11:58 PM Philippe Gerum <rpm@xenomai.org> wrote:
>
>
> Sri Subramanian <sridhar.subramanian@gmail.com> writes:
>
> > Hi Philippe,
> >
> > With next-evl-rebase, I am not able to reproduce the crash. Instead
> > the system reboots (from the watchdog) or hangs (if I disable the
> > watchdog).
> >
> > I'll try going back to the original setup and try other experiments --
> > maybe try to reproduce this with a test program or attach a debugger
> > to get more data.
> >
>
> Ack, at this point it should be possible to reproduce this bug with a
> simple test case if you think the core is doing something wrong when the
> kernel wallclock is set.
>
> If you have custom drivers, you may want to check the lifetime of any
> standalone evl_timer structs it might define for correctness, to make
> sure there is no stale timers being reprogrammed (unlikely, but worth a
> check anyway).
>
> --
> Philippe.
[-- Attachment #2: evl-event-trace-timer-crash.log --]
[-- Type: text/x-log, Size: 147571 bytes --]
[ 4230.829970] audit: type=1334 audit(1670538127.858:9160): prog-id=4588 op=LOAD
[ 4230.856002] audit: type=1334 audit(1670538127.858:9161): prog-id=4589 op=LOAD
[ 4231.892422] systemd-journald[122]: Time jumped backwards, rotating.
** 108627 printk messages dropped **
[ 4235.290994] <idle>-0 0dNh1. 4234454360us : evl_inband_wakeup: pid=364 comm=CommandChannelS
[ 4235.291001] CommandC-364 0*..2. 4234454364us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.291007] CommandC-364 0..... 4234454365us : evl_switched_inband: state=0x4080 info=0x0
[ 4235.291014] T5Server-366 0*..2. 4234454370us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.291020] T5Server-366 0..... 4234454370us : evl_switched_inband: state=0x4880 info=0x0
[ 4235.291026] <idle>-0 2*.~1. 4234454656us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.291033] <idle>-0 0*.~1. 4234454656us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291040] <idle>-0 0*.~1. 4234454657us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 436996 us, 23597831 cycles)
[ 4235.291047] <idle>-0 2*.h1. 4234454661us : evl_timer_start: timer=[proxy-timer/2] value=4234448000129 interval=0
[ 4235.291053] <idle>-0 0*.h1. 4234454662us : evl_timer_start: timer=[proxy-timer/0] value=4234444678585 interval=0
[ 4235.291061] <idle>-0 2*.h1. 4234454663us : evl_timer_shot: [proxy-timer/2] at 4234.458655 (delay: 3992 us, 215588 cycles)
[ 4235.291068] <idle>-0 0*.h1. 4234454663us : evl_timer_shot: [proxy-timer/0] at 4234.455333 (delay: 669 us, 36171 cycles)
[ 4235.291074] <idle>-0 0*.~1. 4234455334us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291082] <idle>-0 0*.~1. 4234455335us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 436318 us, 23561198 cycles)
[ 4235.291089] <idle>-0 0*Nh1. 4234455339us : evl_timer_start: timer=[proxy-timer/0] value=4234446711287 interval=0
[ 4235.291096] <idle>-0 0*Nh1. 4234455340us : evl_timer_shot: [proxy-timer/0] at 4234.457366 (delay: 2025 us, 109403 cycles)
[ 4235.291103] T5Server-367 0*..1. 4234455351us : evl_timer_start: timer=[proxy-timer/0] value=4234446695267 interval=0
[ 4235.291110] T5Server-367 0*..1. 4234455352us : evl_timer_shot: [proxy-timer/0] at 4234.457350 (delay: 1997 us, 107884 cycles)
[ 4235.291117] <idle>-0 0*.~1. 4234457351us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291124] <idle>-0 0*.~1. 4234457352us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 434301 us, 23452304 cycles)
[ 4235.291131] <idle>-0 0*Nh1. 4234457356us : evl_timer_start: timer=[proxy-timer/0] value=4234446711270 interval=0
[ 4235.291138] <idle>-0 0*Nh1. 4234457356us : evl_timer_shot: [proxy-timer/0] at 4234.457366 (delay: 9 us, 514 cycles)
[ 4235.291145] T5Server-367 0*.~2. 4234457367us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291152] T5Server-367 0*.~2. 4234457368us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 434285 us, 23451443 cycles)
[ 4235.291159] <idle>-0 0*Nh1. 4234457375us : evl_timer_start: timer=[proxy-timer/0] value=4234448000148 interval=0
[ 4235.291166] <idle>-0 0*Nh1. 4234457376us : evl_timer_shot: [proxy-timer/0] at 4234.458655 (delay: 1279 us, 69082 cycles)
[ 4235.291172] <idle>-0 2*.~1. 4234458655us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.291178] <idle>-0 0*.~1. 4234458656us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291185] <idle>-0 0*.~1. 4234458657us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 432997 us, 23381844 cycles)
[ 4235.291192] <idle>-0 2*.h1. 4234458659us : evl_timer_start: timer=[proxy-timer/2] value=4234452000112 interval=0
[ 4235.291200] <idle>-0 2*.h1. 4234458660us : evl_timer_shot: [proxy-timer/2] at 4234.462655 (delay: 3994 us, 215711 cycles)
[ 4235.291206] <idle>-0 0*.h1. 4234458661us : evl_timer_start: timer=[proxy-timer/0] value=4234448432981 interval=0
[ 4235.291214] <idle>-0 0*.h1. 4234458663us : evl_timer_shot: [proxy-timer/0] at 4234.459089 (delay: 425 us, 22999 cycles)
[ 4235.291221] EtherCAT-359 2..... 4234458677us : evl_mutex_destroy: mutex=0000000025dadac8
[ 4235.291227] <idle>-0 0*.~1. 4234459089us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291234] <idle>-0 0*.~1. 4234459090us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 432563 us, 23358434 cycles)
[ 4235.291241] <idle>-0 0*Nh1. 4234459096us : evl_timer_start: timer=[proxy-timer/0] value=4234448709634 interval=0
[ 4235.291248] <idle>-0 0*Nh1. 4234459097us : evl_timer_shot: [proxy-timer/0] at 4234.459364 (delay: 267 us, 14429 cycles)
[ 4235.291255] RT engin-361 0..... 4234459110us : evl_inband_sysentry: syscall=oob_ioctl
[ 4235.291261] RT engin-361 0..... 4234459111us : evl_switch_oob: state=0x4880 info=0x0
[ 4235.291268] <idle>-0 0*..2. 4234459118us : evl_release_thread: name=RT engine-tid_361 pid=361 mask=0x80 info=0x0
[ 4235.291275] <idle>-0 0*.~2. 4234459120us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.291281] <idle>-0 0*.~2. 4234459121us : evl_timer_start: timer=[rrb-timer/0] value=4234453465854 interval=0
[ 4235.291288] <idle>-0 0*.~2. 4234459122us : evl_pick_thread: { next=RT engine-tid_361[361] }
[ 4235.291296] <idle>-0 0*.~2. 4234459123us : evl_switch_context: { ROOT/0[0] prio=-1, state=0x1010 } => { RT engine-tid_361[361] prio=20 }
[ 4235.291303] RT engin-361 0d.~2. 4234459125us : evl_switched_oob: state=0x4800 info=0x0
[ 4235.291310] RT engin-361 0d.~2. 4234459127us : evl_mutex_lock: mutex=0000000041a7b4c3
[ 4235.291316] RT engin-361 0d.~2. 4234459128us : evl_inband_sysexit: result=0
[ 4235.291322] RT engin-361 0d.~2. 4234459136us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.291329] RT engin-361 0*.~2. 4234459138us : evl_wake_up: wq=&mon->wait_queue
[ 4235.291336] RT engin-361 0*.~2. 4234459140us : evl_wakeup_thread: name=VM1Thread-tid_363 pid=363 mask=0x2 info=0x0
[ 4235.291342] RT engin-361 0d.~2. 4234459141us : evl_schedule: flags=0x0, local_flags=0x10000000
[ 4235.291349] RT engin-361 0d.~2. 4234459143us : evl_oob_sysexit: result=0
[ 4235.291356] RT engin-361 0d.~2. 4234459146us : evl_switch_inband: cause=in-band syscall
[ 4235.291362] EtherCAT-359 2D.~.. 4234459146us : evl_reschedule_ipi: flags=0x10000000, local_flags=0x4000
[ 4235.291369] RT engin-361 0*.~2. 4234459147us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.291376] EtherCAT-359 2D.~.. 4234459148us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.291382] RT engin-361 0*.~2. 4234459149us : evl_timer_stop: timer=[rrb-timer/0]
[ 4235.291388] EtherCAT-359 2D.~.. 4234459149us : evl_pick_thread: { next=VM1Thread-tid_363[363] }
[ 4235.291395] RT engin-361 0*.~2. 4234459150us : evl_pick_thread: { next=ROOT/0[0] }
[ 4235.291402] EtherCAT-359 2D.~.. 4234459150us : evl_switch_context: { ROOT/2[0] prio=-1, state=0x1010 } => { VM1Thread-tid_363[363] prio=90 }
[ 4235.291410] RT engin-361 0*.~2. 4234459150us : evl_switch_context: { RT engine-tid_361[361] prio=20, state=0x4880 } => { ROOT/0[0] prio=-1 }
[ 4235.291416] <idle>-0 0*.~2. 4234459151us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.291423] <...>-363 2*.~2. 4234459152us : evl_switch_tail: { current=VM1Thread-tid_363[363] }
[ 4235.291430] <...>-363 2d.~2. 4234459153us : evl_finish_wait: wq=&mon->wait_queue
[ 4235.291436] <idle>-0 0d.h1. 4234459153us : evl_inband_wakeup: pid=361 comm=RT engine-tid_3
[ 4235.291442] <...>-363 2d.~2. 4234459154us : evl_oob_sysexit: result=0
[ 4235.291449] RT engin-361 0*..2. 4234459158us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.291455] RT engin-361 0..... 4234459159us : evl_switched_inband: state=0x4880 info=0x0
[ 4235.291462] <...>-363 2d.~2. 4234459187us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.291468] <...>-363 2*.~2. 4234459190us : evl_wait: wq=&mon->wait_queue
[ 4235.291476] <...>-363 2*.~2. 4234459191us : evl_sleep_on: pid=363 timeout=0 timeout_mode=0 clock=monotonic wchan=&mon->wait_queue(00000000ea5e79e5)
[ 4235.291483] <...>-363 2d.~2. 4234459192us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.291489] <...>-363 2*.~2. 4234459193us : evl_pick_thread: { next=ROOT/2[0] }
[ 4235.291497] <...>-363 2*.~2. 4234459194us : evl_switch_context: { VM1Thread-tid_363[363] prio=90, state=0x4002 } => { ROOT/2[0] prio=-1 }
[ 4235.291503] EtherCAT-359 2D.~.. 4234459195us : evl_switch_tail: { current=ROOT/2[0] }
[ 4235.291510] <idle>-0 0*.~1. 4234459366us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291517] <idle>-0 0*.~1. 4234459367us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 432286 us, 23343491 cycles)
[ 4235.291524] <idle>-0 0*Nh1. 4234459372us : evl_timer_start: timer=[proxy-timer/0] value=4234448716787 interval=0
[ 4235.291531] <idle>-0 0*Nh1. 4234459373us : evl_timer_shot: [proxy-timer/0] at 4234.459374 (delay: 1 us, 54 cycles)
[ 4235.291538] <idle>-0 0*N~1. 4234459375us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291545] <idle>-0 0*N~1. 4234459375us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 432278 us, 23343028 cycles)
[ 4235.291552] <idle>-0 0*Nh1. 4234459379us : evl_timer_start: timer=[proxy-timer/0] value=4234449725967 interval=0
[ 4235.291559] <idle>-0 0*Nh1. 4234459380us : evl_timer_shot: [proxy-timer/0] at 4234.460380 (delay: 1000 us, 54049 cycles)
[ 4235.291565] <idle>-0 0*.~1. 4234460382us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291572] <idle>-0 0*.~1. 4234460384us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 431270 us, 23288583 cycles)
[ 4235.291579] <idle>-0 0*Nh1. 4234460389us : evl_timer_start: timer=[proxy-timer/0] value=4234450739038 interval=0
[ 4235.291586] <idle>-0 0*Nh1. 4234460391us : evl_timer_shot: [proxy-timer/0] at 4234.461393 (delay: 1003 us, 54176 cycles)
[ 4235.291593] <idle>-0 0*.~1. 4234461395us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291600] <idle>-0 0*.~1. 4234461396us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 430258 us, 23233938 cycles)
[ 4235.291607] <idle>-0 0*Nh1. 4234461400us : evl_timer_start: timer=[proxy-timer/0] value=4234452745500 interval=0
[ 4235.291614] <idle>-0 0*Nh1. 4234461401us : evl_timer_shot: [proxy-timer/0] at 4234.463400 (delay: 1999 us, 107947 cycles)
[ 4235.291621] <idle>-0 0*N.2. 4234461404us : evl_timer_start: timer=[proxy-timer/0] value=4234452000130 interval=0
[ 4235.291628] <idle>-0 0*N.2. 4234461404us : evl_timer_shot: [proxy-timer/0] at 4234.462655 (delay: 1250 us, 67531 cycles)
[ 4235.291635] <idle>-0 0*..2. 4234461421us : evl_timer_start: timer=[proxy-timer/0] value=4234452745500 interval=0
[ 4235.291642] <idle>-0 0*..2. 4234461422us : evl_timer_shot: [proxy-timer/0] at 4234.463400 (delay: 1978 us, 106815 cycles)
[ 4235.291649] EtherCAT-359 2D.~.. 4234462661us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.291655] EtherCAT-359 2*.h.. 4234462684us : evl_timer_start: timer=[proxy-timer/2] value=4234456000222 interval=0
[ 4235.291662] EtherCAT-359 2*.h.. 4234462687us : evl_timer_shot: [proxy-timer/2] at 4234.466655 (delay: 3968 us, 214301 cycles)
[ 4235.291669] <idle>-0 0*.~1. 4234463401us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291676] <idle>-0 0*.~1. 4234463403us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 428251 us, 23125583 cycles)
[ 4235.291683] <idle>-0 0*Nh1. 4234463407us : evl_timer_start: timer=[proxy-timer/0] value=4234452759683 interval=0
[ 4235.291690] <idle>-0 0*Nh1. 4234463409us : evl_timer_shot: [proxy-timer/0] at 4234.463414 (delay: 5 us, 316 cycles)
[ 4235.291696] CommandC-364 0D.~.. 4234463415us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291703] CommandC-364 0D.~.. 4234463416us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 428237 us, 23124838 cycles)
[ 4235.291710] CommandC-364 0*.h.. 4234463419us : evl_timer_start: timer=[proxy-timer/0] value=4234453346955 interval=0
[ 4235.291717] CommandC-364 0*.h.. 4234463420us : evl_timer_shot: [proxy-timer/0] at 4234.464001 (delay: 581 us, 31387 cycles)
[ 4235.291724] <idle>-0 0*.~1. 4234464003us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291731] <idle>-0 0*.~1. 4234464004us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 427650 us, 23093103 cycles)
[ 4235.291738] <idle>-0 0*Nh1. 4234464008us : evl_timer_start: timer=[proxy-timer/0] value=4234453799149 interval=0
[ 4235.291745] <idle>-0 0*Nh1. 4234464009us : evl_timer_shot: [proxy-timer/0] at 4234.464454 (delay: 444 us, 24011 cycles)
[ 4235.291751] <idle>-0 0*.~1. 4234464455us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291758] <idle>-0 0*.~1. 4234464456us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 427197 us, 23068689 cycles)
[ 4235.291765] <idle>-0 0*Nh1. 4234464460us : evl_timer_start: timer=[proxy-timer/0] value=4234454434388 interval=0
[ 4235.291773] <idle>-0 0*Nh1. 4234464461us : evl_timer_shot: [proxy-timer/0] at 4234.465089 (delay: 628 us, 33918 cycles)
[ 4235.291779] <idle>-0 0*.~1. 4234465090us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291786] <idle>-0 0*.~1. 4234465091us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 426562 us, 23034368 cycles)
[ 4235.291793] <idle>-0 0*Nh1. 4234465096us : evl_timer_start: timer=[proxy-timer/0] value=4234454784994 interval=0
[ 4235.291800] <idle>-0 0*Nh1. 4234465097us : evl_timer_shot: [proxy-timer/0] at 4234.465439 (delay: 342 us, 18513 cycles)
[ 4235.291807] <idle>-0 0*.~1. 4234465441us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291814] <idle>-0 0*.~1. 4234465442us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 426212 us, 23015460 cycles)
[ 4235.291821] <idle>-0 0*Nh1. 4234465445us : evl_timer_start: timer=[proxy-timer/0] value=4234455772550 interval=0
[ 4235.291828] <idle>-0 0*Nh1. 4234465446us : evl_timer_shot: [proxy-timer/0] at 4234.466427 (delay: 981 us, 52974 cycles)
[ 4235.291834] <idle>-0 0*.~1. 4234466429us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291841] <idle>-0 0*.~1. 4234466430us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 425224 us, 22962108 cycles)
[ 4235.291848] <idle>-0 0*Nh1. 4234466434us : evl_timer_start: timer=[proxy-timer/0] value=4234456801029 interval=0
[ 4235.291855] <idle>-0 0*Nh1. 4234466435us : evl_timer_shot: [proxy-timer/0] at 4234.467455 (delay: 1021 us, 55135 cycles)
[ 4235.291862] <idle>-0 0*N.2. 4234466437us : evl_timer_start: timer=[proxy-timer/0] value=4234456000111 interval=0
[ 4235.291869] <idle>-0 0*N.2. 4234466438us : evl_timer_shot: [proxy-timer/0] at 4234.466655 (delay: 217 us, 11729 cycles)
[ 4235.291876] <idle>-0 0*..2. 4234466450us : evl_timer_start: timer=[proxy-timer/0] value=4234456801010 interval=0
[ 4235.291883] <idle>-0 0*..2. 4234466451us : evl_timer_shot: [proxy-timer/0] at 4234.467455 (delay: 1004 us, 54242 cycles)
[ 4235.291889] EtherCAT-359 2D.~.. 4234466657us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.291896] EtherCAT-359 2*.h.. 4234466667us : evl_timer_start: timer=[proxy-timer/2] value=4234456235302 interval=0
[ 4235.291903] EtherCAT-359 2*.h.. 4234466668us : evl_timer_shot: [proxy-timer/2] at 4234.466890 (delay: 221 us, 11980 cycles)
[ 4235.291909] EtherCAT-359 2D.~.. 4234466891us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.291916] EtherCAT-359 2*.h.. 4234466897us : evl_timer_start: timer=[proxy-timer/2] value=4234460000111 interval=0
[ 4235.291923] EtherCAT-359 2*.h.. 4234466898us : evl_timer_shot: [proxy-timer/2] at 4234.470655 (delay: 3757 us, 202884 cycles)
[ 4235.291929] <idle>-0 0*.~1. 4234467456us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291936] <idle>-0 0*.~1. 4234467457us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 424196 us, 22906609 cycles)
[ 4235.291943] <idle>-0 0*Nh1. 4234467461us : evl_timer_start: timer=[proxy-timer/0] value=4234458505668 interval=0
[ 4235.291950] <idle>-0 0*Nh1. 4234467462us : evl_timer_shot: [proxy-timer/0] at 4234.469160 (delay: 1698 us, 91706 cycles)
[ 4235.291957] <idle>-0 0*.~1. 4234469161us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.291964] <idle>-0 0*.~1. 4234469162us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 422491 us, 22814539 cycles)
[ 4235.291971] <idle>-0 0*Nh1. 4234469167us : evl_timer_start: timer=[proxy-timer/0] value=4234458788749 interval=0
[ 4235.291978] <idle>-0 0*Nh1. 4234469168us : evl_timer_shot: [proxy-timer/0] at 4234.469443 (delay: 275 us, 14880 cycles)
[ 4235.291984] RT engin-361 0..... 4234469183us : evl_inband_sysentry: syscall=oob_ioctl
[ 4235.291991] RT engin-361 0..... 4234469184us : evl_switch_oob: state=0x4880 info=0x0
[ 4235.291998] <idle>-0 0*..2. 4234469191us : evl_release_thread: name=RT engine-tid_361 pid=361 mask=0x80 info=0x0
[ 4235.292005] <idle>-0 0*.~2. 4234469194us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.292012] <idle>-0 0*.~2. 4234469199us : evl_timer_start: timer=[rrb-timer/0] value=4234463540745 interval=0
[ 4235.292019] <idle>-0 0*.~2. 4234469200us : evl_pick_thread: { next=RT engine-tid_361[361] }
[ 4235.292026] <idle>-0 0*.~2. 4234469201us : evl_switch_context: { ROOT/0[0] prio=-1, state=0x1010 } => { RT engine-tid_361[361] prio=20 }
[ 4235.292033] RT engin-361 0d.~2. 4234469204us : evl_switched_oob: state=0x4800 info=0x0
[ 4235.292040] RT engin-361 0d.~2. 4234469207us : evl_mutex_lock: mutex=0000000041a7b4c3
[ 4235.292046] RT engin-361 0d.~2. 4234469210us : evl_inband_sysexit: result=0
[ 4235.292053] RT engin-361 0d.~2. 4234469221us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.292059] RT engin-361 0*.~2. 4234469225us : evl_wake_up: wq=&mon->wait_queue
[ 4235.292066] RT engin-361 0*.~2. 4234469227us : evl_wakeup_thread: name=VM1Thread-tid_363 pid=363 mask=0x2 info=0x0
[ 4235.292073] RT engin-361 0d.~2. 4234469229us : evl_schedule: flags=0x0, local_flags=0x10000000
[ 4235.292079] RT engin-361 0d.~2. 4234469231us : evl_oob_sysexit: result=0
[ 4235.292086] EtherCAT-359 2D.~.. 4234469232us : evl_reschedule_ipi: flags=0x10000000, local_flags=0x4000
[ 4235.292092] EtherCAT-359 2D.~.. 4234469234us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.292099] RT engin-361 0d.~2. 4234469235us : evl_switch_inband: cause=in-band syscall
[ 4235.292105] EtherCAT-359 2D.~.. 4234469235us : evl_pick_thread: { next=VM1Thread-tid_363[363] }
[ 4235.292113] EtherCAT-359 2D.~.. 4234469236us : evl_switch_context: { ROOT/2[0] prio=-1, state=0x1010 } => { VM1Thread-tid_363[363] prio=90 }
[ 4235.292119] RT engin-361 0*.~2. 4234469237us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.292126] RT engin-361 0*.~2. 4234469239us : evl_timer_stop: timer=[rrb-timer/0]
[ 4235.292132] <...>-363 2*.~2. 4234469240us : evl_switch_tail: { current=VM1Thread-tid_363[363] }
[ 4235.292139] RT engin-361 0*.~2. 4234469240us : evl_pick_thread: { next=ROOT/0[0] }
[ 4235.292146] RT engin-361 0*.~2. 4234469241us : evl_switch_context: { RT engine-tid_361[361] prio=20, state=0x4880 } => { ROOT/0[0] prio=-1 }
[ 4235.292153] <...>-363 2d.~2. 4234469241us : evl_finish_wait: wq=&mon->wait_queue
[ 4235.292159] <idle>-0 0*.~2. 4234469242us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.292166] <...>-363 2d.~2. 4234469243us : evl_oob_sysexit: result=0
[ 4235.292172] <idle>-0 0d.h1. 4234469244us : evl_inband_wakeup: pid=361 comm=RT engine-tid_3
[ 4235.292179] RT engin-361 0*..2. 4234469249us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.292185] RT engin-361 0..... 4234469251us : evl_switched_inband: state=0x4880 info=0x0
[ 4235.292192] <...>-363 2d.~2. 4234469295us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.292198] <...>-363 2*.~2. 4234469299us : evl_wait: wq=&mon->wait_queue
[ 4235.292206] <...>-363 2*.~2. 4234469301us : evl_sleep_on: pid=363 timeout=0 timeout_mode=0 clock=monotonic wchan=&mon->wait_queue(00000000ea5e79e5)
[ 4235.292213] <...>-363 2d.~2. 4234469303us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.292219] <...>-363 2*.~2. 4234469304us : evl_pick_thread: { next=ROOT/2[0] }
[ 4235.292227] <...>-363 2*.~2. 4234469305us : evl_switch_context: { VM1Thread-tid_363[363] prio=90, state=0x4002 } => { ROOT/2[0] prio=-1 }
[ 4235.292233] EtherCAT-359 2D.~.. 4234469306us : evl_switch_tail: { current=ROOT/2[0] }
[ 4235.292240] <idle>-0 0*.~1. 4234469445us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292247] <idle>-0 0*.~1. 4234469446us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 422207 us, 22799218 cycles)
[ 4235.292254] <idle>-0 0*Nh1. 4234469451us : evl_timer_start: timer=[proxy-timer/0] value=4234458817563 interval=0
[ 4235.292261] <idle>-0 0*Nh1. 4234469452us : evl_timer_shot: [proxy-timer/0] at 4234.469472 (delay: 20 us, 1116 cycles)
[ 4235.292267] <idle>-0 0*.~1. 4234469473us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292274] <idle>-0 0*.~1. 4234469474us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 422179 us, 22797709 cycles)
[ 4235.292281] <idle>-0 0*Nh1. 4234469478us : evl_timer_start: timer=[proxy-timer/0] value=4234458822993 interval=0
[ 4235.292289] <idle>-0 0*Nh1. 4234469479us : evl_timer_shot: [proxy-timer/0] at 4234.469480 (delay: 1 us, 54 cycles)
[ 4235.292295] <idle>-0 0*N~1. 4234469481us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292302] <idle>-0 0*N~1. 4234469481us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 422172 us, 22797315 cycles)
[ 4235.292309] <idle>-0 0*Nh1. 4234469484us : evl_timer_start: timer=[proxy-timer/0] value=4234460000130 interval=0
[ 4235.292316] <idle>-0 0*Nh1. 4234469485us : evl_timer_shot: [proxy-timer/0] at 4234.470655 (delay: 1169 us, 63161 cycles)
[ 4235.292322] <idle>-0 0*.~1. 4234470656us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292329] EtherCAT-359 2D.~.. 4234470657us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.292336] <idle>-0 0*.~1. 4234470657us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 420996 us, 22733831 cycles)
[ 4235.292343] <idle>-0 0*.h1. 4234470663us : evl_timer_start: timer=[proxy-timer/0] value=4234460839966 interval=0
[ 4235.292350] <idle>-0 0*.h1. 4234470664us : evl_timer_shot: [proxy-timer/0] at 4234.471494 (delay: 830 us, 44857 cycles)
[ 4235.292357] EtherCAT-359 2*.h.. 4234470665us : evl_timer_start: timer=[proxy-timer/2] value=4234464000130 interval=0
[ 4235.292364] EtherCAT-359 2*.h.. 4234470666us : evl_timer_shot: [proxy-timer/2] at 4234.474655 (delay: 3988 us, 215380 cycles)
[ 4235.292370] <idle>-0 0*.~1. 4234471496us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292377] <idle>-0 0*.~1. 4234471497us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 420157 us, 22688486 cycles)
[ 4235.292384] <idle>-0 0*Nh1. 4234471501us : evl_timer_start: timer=[proxy-timer/0] value=4234461803022 interval=0
[ 4235.292391] <idle>-0 0*Nh1. 4234471502us : evl_timer_shot: [proxy-timer/0] at 4234.472457 (delay: 956 us, 51626 cycles)
[ 4235.292397] <idle>-0 0*.~1. 4234472459us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292404] <idle>-0 0*.~1. 4234472460us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 419194 us, 22636481 cycles)
[ 4235.292411] <idle>-0 0*Nh1. 4234472464us : evl_timer_start: timer=[proxy-timer/0] value=4234462857315 interval=0
[ 4235.292418] <idle>-0 0*Nh1. 4234472465us : evl_timer_shot: [proxy-timer/0] at 4234.473512 (delay: 1046 us, 56532 cycles)
[ 4235.292424] <idle>-0 0*.~1. 4234473513us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292431] <idle>-0 0*.~1. 4234473514us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 418139 us, 22579557 cycles)
[ 4235.292438] <idle>-0 0*Nh1. 4234473518us : evl_timer_start: timer=[proxy-timer/0] value=4234463363938 interval=0
[ 4235.292446] <idle>-0 0*Nh1. 4234473519us : evl_timer_shot: [proxy-timer/0] at 4234.474018 (delay: 499 us, 26982 cycles)
[ 4235.292452] <idle>-0 0*.~1. 4234474020us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292459] <idle>-0 0*.~1. 4234474021us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 417633 us, 22552193 cycles)
[ 4235.292466] <idle>-0 0*Nh1. 4234474025us : evl_timer_start: timer=[proxy-timer/0] value=4234463851874 interval=0
[ 4235.292474] <idle>-0 0*Nh1. 4234474026us : evl_timer_shot: [proxy-timer/0] at 4234.474506 (delay: 481 us, 25977 cycles)
[ 4235.292480] <idle>-0 0*.~1. 4234474507us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292487] <idle>-0 0*.~1. 4234474509us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 417145 us, 22525849 cycles)
[ 4235.292494] <idle>-0 0*Nh1. 4234474513us : evl_timer_start: timer=[proxy-timer/0] value=4234464000148 interval=0
[ 4235.292501] <idle>-0 0*Nh1. 4234474514us : evl_timer_shot: [proxy-timer/0] at 4234.474655 (delay: 140 us, 7610 cycles)
[ 4235.292508] <idle>-0 0*.~1. 4234474656us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292514] EtherCAT-359 2D.~.. 4234474658us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.292521] <idle>-0 0*.~1. 4234474658us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 416996 us, 22517791 cycles)
[ 4235.292528] <idle>-0 0*.h1. 4234474664us : evl_timer_start: timer=[proxy-timer/0] value=4234464269928 interval=0
[ 4235.292536] <idle>-0 0*.h1. 4234474665us : evl_timer_shot: [proxy-timer/0] at 4234.474924 (delay: 259 us, 13999 cycles)
[ 4235.292542] EtherCAT-359 2*.h.. 4234474668us : evl_timer_start: timer=[proxy-timer/2] value=4234468000130 interval=0
[ 4235.292550] EtherCAT-359 2*.h.. 4234474670us : evl_timer_shot: [proxy-timer/2] at 4234.478655 (delay: 3985 us, 215213 cycles)
[ 4235.292556] <...>-141 0D.~.. 4234474927us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292563] <...>-141 0D.~.. 4234474929us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 416724 us, 22503149 cycles)
[ 4235.292570] <...>-141 0*Nh.. 4234474935us : evl_timer_start: timer=[proxy-timer/0] value=4234464816554 interval=0
[ 4235.292577] <...>-141 0*Nh.. 4234474937us : evl_timer_shot: [proxy-timer/0] at 4234.475471 (delay: 534 us, 28877 cycles)
[ 4235.292584] systemd--11716 0D.~.. 4234475475us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292591] systemd--11716 0D.~.. 4234475478us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 416176 us, 22473537 cycles)
[ 4235.292597] systemd--11716 0*Nh.. 4234475485us : evl_timer_start: timer=[proxy-timer/0] value=4234464873052 interval=0
[ 4235.292605] systemd--11716 0*Nh.. 4234475487us : evl_timer_shot: [proxy-timer/0] at 4234.475528 (delay: 40 us, 2206 cycles)
[ 4235.292611] systemd--11716 0D.~.. 4234475529us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292618] systemd--11716 0D.~.. 4234475530us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 416123 us, 22470665 cycles)
[ 4235.292625] systemd--11716 0*Nh.. 4234475535us : evl_timer_start: timer=[proxy-timer/0] value=4234467845827 interval=0
[ 4235.292632] systemd--11716 0*Nh.. 4234475536us : evl_timer_shot: [proxy-timer/0] at 4234.478500 (delay: 2964 us, 160094 cycles)
[ 4235.292639] T5Server-367 0*..1. 4234475558us : evl_timer_start: timer=[proxy-timer/0] value=4234466901698 interval=0
[ 4235.292646] T5Server-367 0*..1. 4234475559us : evl_timer_shot: [proxy-timer/0] at 4234.477556 (delay: 1997 us, 107869 cycles)
[ 4235.292652] <idle>-0 0*.~1. 4234477562us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292660] <idle>-0 0*.~1. 4234477567us : evl_timer_shot: timerfd_handler at 4234.891655 (delay: 414087 us, 22360749 cycles)
[ 4235.292666] <idle>-0 0*Nh1. 4234477581us : evl_timer_start: timer=[proxy-timer/0] value=4234467845938 interval=0
[ 4235.292674] <idle>-0 0*Nh1. 4234477583us : evl_timer_shot: [proxy-timer/0] at 4234.478501 (delay: 917 us, 49541 cycles)
[ 4235.292680] <idle>-0 0*.~1. 4234478502us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292687] <idle>-0 0*.~1. 4234478503us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 413150 us, 22310147 cycles)
[ 4235.292694] <idle>-0 0*Nh1. 4234478507us : evl_timer_start: timer=[proxy-timer/0] value=4234468000130 interval=0
[ 4235.292701] <idle>-0 0*Nh1. 4234478508us : evl_timer_shot: [proxy-timer/0] at 4234.478655 (delay: 146 us, 7907 cycles)
[ 4235.292708] <idle>-0 0*.~1. 4234478656us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292714] <idle>-0 2*.~1. 4234478657us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.292721] <idle>-0 0*.~1. 4234478657us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 412996 us, 22301832 cycles)
[ 4235.292728] <idle>-0 0*.h1. 4234478668us : evl_timer_start: timer=[proxy-timer/0] value=4234468597467 interval=0
[ 4235.292735] <idle>-0 2*.h1. 4234478668us : evl_timer_start: timer=[proxy-timer/2] value=4234472000130 interval=0
[ 4235.292742] <idle>-0 0*.h1. 4234478669us : evl_timer_shot: [proxy-timer/0] at 4234.479252 (delay: 583 us, 31517 cycles)
[ 4235.292749] <idle>-0 2*.h1. 4234478670us : evl_timer_shot: [proxy-timer/2] at 4234.482655 (delay: 3985 us, 215205 cycles)
[ 4235.292756] <idle>-0 0*.~1. 4234479253us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.292763] <idle>-0 0*.~1. 4234479254us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 412399 us, 22269582 cycles)
[ 4235.292770] <idle>-0 0*Nh1. 4234479258us : evl_timer_start: timer=[proxy-timer/0] value=4234468873459 interval=0
[ 4235.292777] <idle>-0 0*Nh1. 4234479259us : evl_timer_shot: [proxy-timer/0] at 4234.479528 (delay: 268 us, 14511 cycles)
[ 4235.292783] RT engin-361 0..... 4234479279us : evl_inband_sysentry: syscall=oob_ioctl
[ 4235.292790] RT engin-361 0..... 4234479280us : evl_switch_oob: state=0x4880 info=0x0
[ 4235.292798] <idle>-0 0*..2. 4234479293us : evl_release_thread: name=RT engine-tid_361 pid=361 mask=0x80 info=0x0
[ 4235.292804] <idle>-0 0*.~2. 4234479296us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.292811] <idle>-0 0*.~2. 4234479298us : evl_timer_start: timer=[rrb-timer/0] value=4234473642839 interval=0
[ 4235.292818] <idle>-0 0*.~2. 4234479300us : evl_pick_thread: { next=RT engine-tid_361[361] }
[ 4235.292825] <idle>-0 0*.~2. 4234479301us : evl_switch_context: { ROOT/0[0] prio=-1, state=0x1010 } => { RT engine-tid_361[361] prio=20 }
[ 4235.292832] RT engin-361 0d.~2. 4234479304us : evl_switched_oob: state=0x4800 info=0x0
[ 4235.292839] RT engin-361 0d.~2. 4234479309us : evl_mutex_lock: mutex=0000000041a7b4c3
[ 4235.292845] RT engin-361 0d.~2. 4234479311us : evl_inband_sysexit: result=0
[ 4235.292852] RT engin-361 0d.~2. 4234479327us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.292858] RT engin-361 0*.~2. 4234479332us : evl_wake_up: wq=&mon->wait_queue
[ 4235.292865] RT engin-361 0*.~2. 4234479335us : evl_wakeup_thread: name=VM1Thread-tid_363 pid=363 mask=0x2 info=0x0
[ 4235.292872] RT engin-361 0d.~2. 4234479337us : evl_schedule: flags=0x0, local_flags=0x10000000
[ 4235.292879] RT engin-361 0d.~2. 4234479339us : evl_oob_sysexit: result=0
[ 4235.292885] <idle>-0 2*.~1. 4234479340us : evl_reschedule_ipi: flags=0x10000000, local_flags=0x4000
[ 4235.292892] <idle>-0 2*.~1. 4234479341us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.292898] <idle>-0 2*.~1. 4234479342us : evl_pick_thread: { next=VM1Thread-tid_363[363] }
[ 4235.292906] <idle>-0 2*.~1. 4234479343us : evl_switch_context: { ROOT/2[0] prio=-1, state=0x1010 } => { VM1Thread-tid_363[363] prio=90 }
[ 4235.292913] RT engin-361 0d.~2. 4234479345us : evl_switch_inband: cause=in-band syscall
[ 4235.292919] <...>-363 2*.~2. 4234479346us : evl_switch_tail: { current=VM1Thread-tid_363[363] }
[ 4235.292926] RT engin-361 0*.~2. 4234479348us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.292932] <...>-363 2d.~2. 4234479348us : evl_finish_wait: wq=&mon->wait_queue
[ 4235.292939] RT engin-361 0*.~2. 4234479350us : evl_timer_stop: timer=[rrb-timer/0]
[ 4235.292945] <...>-363 2d.~2. 4234479350us : evl_oob_sysexit: result=0
[ 4235.292951] RT engin-361 0*.~2. 4234479351us : evl_pick_thread: { next=ROOT/0[0] }
[ 4235.292959] RT engin-361 0*.~2. 4234479351us : evl_switch_context: { RT engine-tid_361[361] prio=20, state=0x4880 } => { ROOT/0[0] prio=-1 }
[ 4235.292965] <idle>-0 0*.~2. 4234479353us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.292972] <idle>-0 0d.h1. 4234479356us : evl_inband_wakeup: pid=361 comm=RT engine-tid_3
[ 4235.292978] RT engin-361 0*..2. 4234479361us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.292985] RT engin-361 0..... 4234479362us : evl_switched_inband: state=0x4880 info=0x0
[ 4235.292991] <...>-363 2d.~2. 4234479407us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.292998] <...>-363 2*.~2. 4234479411us : evl_wait: wq=&mon->wait_queue
[ 4235.293005] <...>-363 2*.~2. 4234479413us : evl_sleep_on: pid=363 timeout=0 timeout_mode=0 clock=monotonic wchan=&mon->wait_queue(00000000ea5e79e5)
[ 4235.293012] <...>-363 2d.~2. 4234479415us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.293018] <...>-363 2*.~2. 4234479416us : evl_pick_thread: { next=ROOT/2[0] }
[ 4235.293026] <...>-363 2*.~2. 4234479417us : evl_switch_context: { VM1Thread-tid_363[363] prio=90, state=0x4002 } => { ROOT/2[0] prio=-1 }
[ 4235.293032] <idle>-0 2*.~1. 4234479418us : evl_switch_tail: { current=ROOT/2[0] }
[ 4235.293039] <idle>-0 0*.~1. 4234479530us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.293046] <idle>-0 0*.~1. 4234479531us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 412123 us, 22254656 cycles)
[ 4235.293053] <idle>-0 0*Nh1. 4234479535us : evl_timer_start: timer=[proxy-timer/0] value=4234468956231 interval=0
[ 4235.293060] <idle>-0 0*Nh1. 4234479536us : evl_timer_shot: [proxy-timer/0] at 4234.479611 (delay: 75 us, 4052 cycles)
[ 4235.293066] T5Server-366 0D.~.. 4234479612us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.293073] T5Server-366 0D.~.. 4234479613us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 412040 us, 22250189 cycles)
[ 4235.293080] T5Server-366 0*NH.. 4234479618us : evl_timer_start: timer=[proxy-timer/0] value=4234470863286 interval=0
[ 4235.293087] T5Server-366 0*NH.. 4234479619us : evl_timer_shot: [proxy-timer/0] at 4234.481518 (delay: 1899 us, 102556 cycles)
[ 4235.293094] <idle>-0 0*.~1. 4234481519us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.293101] <idle>-0 0*.~1. 4234481521us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 410133 us, 22147190 cycles)
[ 4235.293108] <idle>-0 0*Nh1. 4234481525us : evl_timer_start: timer=[proxy-timer/0] value=4234470979560 interval=0
[ 4235.293115] <idle>-0 0*Nh1. 4234481527us : evl_timer_shot: [proxy-timer/0] at 4234.481634 (delay: 107 us, 5821 cycles)
[ 4235.293121] <idle>-0 0*.~1. 4234481635us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.293128] <idle>-0 0*.~1. 4234481636us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 410017 us, 22140954 cycles)
[ 4235.293135] <idle>-0 0*Nh1. 4234481640us : evl_timer_start: timer=[proxy-timer/0] value=4234472000130 interval=0
[ 4235.293142] <idle>-0 0*Nh1. 4234481641us : evl_timer_shot: [proxy-timer/0] at 4234.482655 (delay: 1013 us, 54749 cycles)
[ 4235.293149] <idle>-0 0*.~1. 4234482656us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.293155] <idle>-0 2*.~1. 4234482656us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.293162] <idle>-0 0*.~1. 4234482657us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 408997 us, 22085841 cycles)
[ 4235.293169] <idle>-0 0*.h1. 4234482662us : evl_timer_start: timer=[proxy-timer/0] value=4234472995150 interval=0
[ 4235.293176] <idle>-0 0*.h1. 4234482663us : evl_timer_shot: [proxy-timer/0] at 4234.483650 (delay: 987 us, 53318 cycles)
[ 4235.293183] <idle>-0 2*.h1. 4234482663us : evl_timer_start: timer=[proxy-timer/2] value=4234476000111 interval=0
[ 4235.293190] <idle>-0 2*.h1. 4234482664us : evl_timer_shot: [proxy-timer/2] at 4234.486655 (delay: 3991 us, 215525 cycles)
[ 4235.293197] <idle>-0 0*.~1. 4234483651us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.293204] <idle>-0 0*.~1. 4234483652us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 408001 us, 22032105 cycles)
[ 4235.293211] <idle>-0 0*Nh1. 4234483656us : evl_timer_start: timer=[proxy-timer/0] value=4234473377774 interval=0
[ 4235.293218] <idle>-0 0*Nh1. 4234483657us : evl_timer_shot: [proxy-timer/0] at 4234.484032 (delay: 375 us, 20278 cycles)
[ 4235.293224] <idle>-0 0*.~1. 4234484033us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.293231] <idle>-0 0*.~1. 4234484034us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 407619 us, 22011450 cycles)
[ 4235.293238] <idle>-0 0*Nh1. 4234484039us : evl_timer_start: timer=[proxy-timer/0] value=4234473879002 interval=0
[ 4235.293245] <idle>-0 0*Nh1. 4234484040us : evl_timer_shot: [proxy-timer/0] at 4234.484533 (delay: 493 us, 26659 cycles)
[ 4235.293252] <idle>-0 0*.~1. 4234484535us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.293259] <idle>-0 0*.~1. 4234484536us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 407118 us, 21984385 cycles)
[ 4235.293266] <idle>-0 0*Nh1. 4234484539us : evl_timer_start: timer=[proxy-timer/0] value=4234473983079 interval=0
[ 4235.293273] <idle>-0 0*Nh1. 4234484540us : evl_timer_shot: [proxy-timer/0] at 4234.484637 (delay: 97 us, 5261 cycles)
[ 4235.293279] <idle>-0 0*.~1. 4234484639us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.293286] <idle>-0 0*.~1. 4234484640us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 407014 us, 21978767 cycles)
[ 4235.293293] <idle>-0 0*Nh1. 4234484643us : evl_timer_start: timer=[proxy-timer/0] value=4234475011092 interval=0
[ 4235.293300] <idle>-0 0*Nh1. 4234484644us : evl_timer_shot: [proxy-timer/0] at 4234.485666 (delay: 1021 us, 55165 cycles)
[ 4235.293307] T5Server-366 0..... 4234484678us : evl_inband_sysentry: syscall=oob_ioctl
[ 4235.293314] T5Server-366 0..... 4234484679us : evl_switch_oob: state=0x4880 info=0x0
[ 4235.293320] <idle>-0 0*..2. 4234484686us : evl_release_thread: name=T5Server_K5T5-tid_366 pid=366 mask=0x80 info=0x0
[ 4235.293327] <idle>-0 0*.~2. 4234484687us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.293334] <idle>-0 0*.~2. 4234484688us : evl_timer_start: timer=[rrb-timer/0] value=4234479033016 interval=0
[ 4235.293340] <idle>-0 0*.~2. 4234484689us : evl_pick_thread: { next=T5Server_K5T5-tid_366[366] }
[ 4235.293348] <idle>-0 0*.~2. 4234484690us : evl_switch_context: { ROOT/0[0] prio=-1, state=0x1010 } => { T5Server_K5T5-tid_366[366] prio=20 }
[ 4235.293355] T5Server-366 0d.~2. 4234484691us : evl_switched_oob: state=0x4800 info=0x0
[ 4235.293361] T5Server-366 0d.~2. 4234484694us : evl_mutex_lock: mutex=000000005cfd6a29
[ 4235.293368] T5Server-366 0d.~2. 4234484695us : evl_inband_sysexit: result=0
[ 4235.293374] T5Server-366 0d.~2. 4234484701us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.293380] T5Server-366 0*.~2. 4234484703us : evl_wait: wq=&mon->wait_queue
[ 4235.293388] T5Server-366 0*.~2. 4234484704us : evl_sleep_on: pid=366 timeout=4244474044627 timeout_mode=1 clock=monotonic wchan=&mon->wait_queue(00000000c7be549f)
[ 4235.293395] T5Server-366 0*.~2. 4234484705us : evl_timer_start: timer=T5Server_K5T5-tid_366 value=4244474044627 interval=0
[ 4235.293401] T5Server-366 0d.~2. 4234484706us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.293408] T5Server-366 0*.~2. 4234484708us : evl_timer_stop: timer=[rrb-timer/0]
[ 4235.293414] T5Server-366 0*.~2. 4234484708us : evl_pick_thread: { next=ROOT/0[0] }
[ 4235.293421] T5Server-366 0*.~2. 4234484709us : evl_switch_context: { T5Server_K5T5-tid_366[366] prio=20, state=0x4806 } => { ROOT/0[0] prio=-1 }
[ 4235.293428] <idle>-0 0*.~2. 4234484710us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.293434] <idle>-0 0*.~1. 4234485667us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.293442] <idle>-0 0*.~1. 4234485668us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 405986 us, 21923255 cycles)
[ 4235.293449] <idle>-0 0*Nh1. 4234485672us : evl_timer_start: timer=[proxy-timer/0] value=4234476000148 interval=0
[ 4235.293456] <idle>-0 0*Nh1. 4234485673us : evl_timer_shot: [proxy-timer/0] at 4234.486655 (delay: 981 us, 53027 cycles)
[ 4235.293462] <idle>-0 2*.~1. 4234486655us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.293468] <idle>-0 0*.~1. 4234486656us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.293475] <idle>-0 0*.~1. 4234486657us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 404996 us, 21869832 cycles)
[ 4235.293483] <idle>-0 2*.h1. 4234486659us : evl_timer_start: timer=[proxy-timer/2] value=4234480000111 interval=0
[ 4235.293490] <idle>-0 2*.h1. 4234486660us : evl_timer_shot: [proxy-timer/2] at 4234.490655 (delay: 3994 us, 215709 cycles)
[ 4235.293497] <idle>-0 0*.h1. 4234486661us : evl_timer_start: timer=[proxy-timer/0] value=4234476891146 interval=0
[ 4235.293504] <idle>-0 0*.h1. 4234486662us : evl_timer_shot: [proxy-timer/0] at 4234.487546 (delay: 883 us, 47709 cycles)
[ 4235.293511] <idle>-0 0*.~1. 4234487546us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.293518] <idle>-0 0*.~1. 4234487547us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 404106 us, 21821755 cycles)
[ 4235.293525] <idle>-0 0*Nh1. 4234487551us : evl_timer_start: timer=[proxy-timer/0] value=4234477027847 interval=0
[ 4235.293532] <idle>-0 0*Nh1. 4234487552us : evl_timer_shot: [proxy-timer/0] at 4234.487682 (delay: 131 us, 7075 cycles)
[ 4235.293538] CommandC-364 0..... 4234487559us : evl_inband_sysentry: syscall=oob_ioctl
[ 4235.293545] CommandC-364 0..... 4234487559us : evl_switch_oob: state=0x4080 info=0x0
[ 4235.293552] <idle>-0 0*..2. 4234487565us : evl_release_thread: name=CommandChannelServer-tid_364 pid=364 mask=0x80 info=0x0
[ 4235.293559] <idle>-0 0*.~2. 4234487566us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.293565] <idle>-0 0*.~2. 4234487567us : evl_pick_thread: { next=CommandChannelServer-tid_364[364] }
[ 4235.293573] <idle>-0 0*.~2. 4234487568us : evl_switch_context: { ROOT/0[0] prio=-1, state=0x1010 } => { CommandChannelServer-tid_364[364] prio=70 }
[ 4235.293580] CommandC-364 0d.~2. 4234487569us : evl_switched_oob: state=0x4000 info=0x0
[ 4235.293586] CommandC-364 0d.~2. 4234487571us : evl_mutex_lock: mutex=00000000f3b83432
[ 4235.293593] CommandC-364 0d.~2. 4234487572us : evl_inband_sysexit: result=0
[ 4235.293599] CommandC-364 0d.~2. 4234487582us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.293606] CommandC-364 0*.~2. 4234487586us : evl_wake_up: wq=&mon->wait_queue
[ 4235.293613] CommandC-364 0*.~2. 4234487587us : evl_wakeup_thread: name=T5Server_K5T5-tid_366 pid=366 mask=0x2 info=0x0
[ 4235.293620] CommandC-364 0*.~2. 4234487587us : evl_timer_stop: timer=T5Server_K5T5-tid_366
[ 4235.293626] CommandC-364 0d.~2. 4234487588us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.293633] CommandC-364 0*.~2. 4234487589us : evl_pick_thread: { next=CommandChannelServer-tid_364[364] }
[ 4235.293640] CommandC-364 0d.~2. 4234487590us : evl_oob_sysexit: result=0
[ 4235.293646] CommandC-364 0d.~2. 4234487592us : evl_switch_inband: cause=in-band syscall
[ 4235.293652] CommandC-364 0*.~2. 4234487594us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.293659] CommandC-364 0*.~2. 4234487594us : evl_timer_start: timer=[rrb-timer/0] value=4234481939104 interval=0
[ 4235.293666] CommandC-364 0*.~2. 4234487595us : evl_pick_thread: { next=T5Server_K5T5-tid_366[366] }
[ 4235.293674] CommandC-364 0*.~2. 4234487596us : evl_switch_context: { CommandChannelServer-tid_364[364] prio=70, state=0x4080 } => { T5Server_K5T5-tid_366[366] prio=20 }
[ 4235.293681] T5Server-366 0*.~2. 4234487597us : evl_switch_tail: { current=T5Server_K5T5-tid_366[366] }
[ 4235.293687] T5Server-366 0d.~2. 4234487598us : evl_finish_wait: wq=&mon->wait_queue
[ 4235.293694] T5Server-366 0d.~2. 4234487599us : evl_oob_sysexit: result=0
[ 4235.293700] T5Server-366 0d.~2. 4234487603us : evl_switch_inband: cause=in-band syscall
[ 4235.293706] T5Server-366 0*.~2. 4234487604us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.293713] T5Server-366 0*.~2. 4234487605us : evl_timer_stop: timer=[rrb-timer/0]
[ 4235.293719] T5Server-366 0*.~2. 4234487606us : evl_pick_thread: { next=ROOT/0[0] }
[ 4235.293726] T5Server-366 0*.~2. 4234487606us : evl_switch_context: { T5Server_K5T5-tid_366[366] prio=20, state=0x4880 } => { ROOT/0[0] prio=-1 }
[ 4235.293733] <idle>-0 0*.~2. 4234487607us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.293740] <idle>-0 0d.h1. 4234487609us : evl_inband_wakeup: pid=366 comm=T5Server_K5T5-t
[ 4235.293746] <idle>-0 0dNh1. 4234487611us : evl_inband_wakeup: pid=364 comm=CommandChannelS
[ 4235.293753] CommandC-364 0*..2. 4234487615us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.293759] CommandC-364 0..... 4234487616us : evl_switched_inband: state=0x4080 info=0x0
[ 4235.293766] T5Server-366 0*..2. 4234487620us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.293772] T5Server-366 0..... 4234487621us : evl_switched_inband: state=0x4880 info=0x0
[ 4235.293779] <idle>-0 0*.~1. 4234487684us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.293786] <idle>-0 0*.~1. 4234487685us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 403969 us, 21814338 cycles)
[ 4235.293793] <idle>-0 0*Nh1. 4234487689us : evl_timer_start: timer=[proxy-timer/0] value=4234478708784 interval=0
[ 4235.293801] <idle>-0 0*Nh1. 4234487690us : evl_timer_shot: [proxy-timer/0] at 4234.489363 (delay: 1673 us, 90377 cycles)
[ 4235.293807] <idle>-0 0*.~1. 4234489364us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.293814] <idle>-0 0*.~1. 4234489365us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 402288 us, 21723575 cycles)
[ 4235.293821] <idle>-0 0*Nh1. 4234489369us : evl_timer_start: timer=[proxy-timer/0] value=4234479044085 interval=0
[ 4235.293829] <idle>-0 0*Nh1. 4234489370us : evl_timer_shot: [proxy-timer/0] at 4234.489699 (delay: 328 us, 17745 cycles)
[ 4235.293835] RT engin-361 0..... 4234489378us : evl_inband_sysentry: syscall=oob_ioctl
[ 4235.293842] RT engin-361 0..... 4234489378us : evl_switch_oob: state=0x4880 info=0x0
[ 4235.293848] <idle>-0 0*..2. 4234489383us : evl_release_thread: name=RT engine-tid_361 pid=361 mask=0x80 info=0x0
[ 4235.293855] <idle>-0 0*.~2. 4234489384us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.293862] <idle>-0 0*.~2. 4234489385us : evl_timer_start: timer=[rrb-timer/0] value=4234483729860 interval=0
[ 4235.293869] <idle>-0 0*.~2. 4234489386us : evl_pick_thread: { next=RT engine-tid_361[361] }
[ 4235.293876] <idle>-0 0*.~2. 4234489387us : evl_switch_context: { ROOT/0[0] prio=-1, state=0x1010 } => { RT engine-tid_361[361] prio=20 }
[ 4235.293883] RT engin-361 0d.~2. 4234489388us : evl_switched_oob: state=0x4800 info=0x0
[ 4235.293890] RT engin-361 0d.~2. 4234489389us : evl_mutex_lock: mutex=0000000041a7b4c3
[ 4235.293896] RT engin-361 0d.~2. 4234489390us : evl_inband_sysexit: result=0
[ 4235.293902] RT engin-361 0d.~2. 4234489396us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.293909] RT engin-361 0*.~2. 4234489397us : evl_wake_up: wq=&mon->wait_queue
[ 4235.293916] RT engin-361 0*.~2. 4234489398us : evl_wakeup_thread: name=VM1Thread-tid_363 pid=363 mask=0x2 info=0x0
[ 4235.293923] RT engin-361 0d.~2. 4234489399us : evl_schedule: flags=0x0, local_flags=0x10000000
[ 4235.293929] RT engin-361 0d.~2. 4234489400us : evl_oob_sysexit: result=0
[ 4235.293936] <idle>-0 2*.~1. 4234489401us : evl_reschedule_ipi: flags=0x10000000, local_flags=0x4000
[ 4235.293942] <idle>-0 2*.~1. 4234489402us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.293949] <idle>-0 2*.~1. 4234489403us : evl_pick_thread: { next=VM1Thread-tid_363[363] }
[ 4235.293956] RT engin-361 0d.~2. 4234489403us : evl_switch_inband: cause=in-band syscall
[ 4235.293963] <idle>-0 2*.~1. 4234489403us : evl_switch_context: { ROOT/2[0] prio=-1, state=0x1010 } => { VM1Thread-tid_363[363] prio=90 }
[ 4235.293970] RT engin-361 0*.~2. 4234489404us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.293977] <...>-363 2*.~2. 4234489405us : evl_switch_tail: { current=VM1Thread-tid_363[363] }
[ 4235.293983] RT engin-361 0*.~2. 4234489405us : evl_timer_stop: timer=[rrb-timer/0]
[ 4235.293990] RT engin-361 0*.~2. 4234489406us : evl_pick_thread: { next=ROOT/0[0] }
[ 4235.293996] <...>-363 2d.~2. 4234489406us : evl_finish_wait: wq=&mon->wait_queue
[ 4235.294004] RT engin-361 0*.~2. 4234489406us : evl_switch_context: { RT engine-tid_361[361] prio=20, state=0x4880 } => { ROOT/0[0] prio=-1 }
[ 4235.294010] <...>-363 2d.~2. 4234489407us : evl_oob_sysexit: result=0
[ 4235.294017] <idle>-0 0*.~2. 4234489407us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.294024] <idle>-0 0d.h1. 4234489409us : evl_inband_wakeup: pid=361 comm=RT engine-tid_3
[ 4235.294030] RT engin-361 0*..2. 4234489413us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.294036] RT engin-361 0..... 4234489414us : evl_switched_inband: state=0x4880 info=0x0
[ 4235.294043] <...>-363 2d.~2. 4234489422us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.294050] <...>-363 2*.~2. 4234489423us : evl_wait: wq=&mon->wait_queue
[ 4235.294057] <...>-363 2*.~2. 4234489424us : evl_sleep_on: pid=363 timeout=0 timeout_mode=0 clock=monotonic wchan=&mon->wait_queue(00000000ea5e79e5)
[ 4235.294064] <...>-363 2d.~2. 4234489425us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.294071] <...>-363 2*.~2. 4234489426us : evl_pick_thread: { next=ROOT/2[0] }
[ 4235.294078] <...>-363 2*.~2. 4234489427us : evl_switch_context: { VM1Thread-tid_363[363] prio=90, state=0x4002 } => { ROOT/2[0] prio=-1 }
[ 4235.294085] <idle>-0 2*.~1. 4234489428us : evl_switch_tail: { current=ROOT/2[0] }
[ 4235.294091] <idle>-0 0*.~1. 4234489700us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294098] <idle>-0 0*.~1. 4234489701us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 401952 us, 21705452 cycles)
[ 4235.294105] <idle>-0 0*Nh1. 4234489705us : evl_timer_start: timer=[proxy-timer/0] value=4234479962381 interval=0
[ 4235.294113] <idle>-0 0*Nh1. 4234489706us : evl_timer_shot: [proxy-timer/0] at 4234.490617 (delay: 911 us, 49196 cycles)
[ 4235.294119] <idle>-0 0*.~1. 4234490618us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294126] <idle>-0 0*.~1. 4234490619us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 401034 us, 21655873 cycles)
[ 4235.294133] <idle>-0 0*Nh1. 4234490623us : evl_timer_start: timer=[proxy-timer/0] value=4234480000130 interval=0
[ 4235.294140] <idle>-0 0*Nh1. 4234490624us : evl_timer_shot: [proxy-timer/0] at 4234.490655 (delay: 30 us, 1663 cycles)
[ 4235.294147] <idle>-0 0*.~1. 4234490656us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294153] <idle>-0 0*.~1. 4234490657us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 400997 us, 21653842 cycles)
[ 4235.294160] <idle>-0 2*.~1. 4234490657us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.294166] <idle>-0 0*.h1. 4234490661us : evl_timer_start: timer=[proxy-timer/0] value=4234481060508 interval=0
[ 4235.294174] <idle>-0 0*.h1. 4234490662us : evl_timer_shot: [proxy-timer/0] at 4234.491715 (delay: 1053 us, 56880 cycles)
[ 4235.294181] <idle>-0 2*.h1. 4234490662us : evl_timer_start: timer=[proxy-timer/2] value=4234484000130 interval=0
[ 4235.294188] <idle>-0 2*.h1. 4234490664us : evl_timer_shot: [proxy-timer/2] at 4234.494655 (delay: 3991 us, 215529 cycles)
[ 4235.294194] <idle>-0 0*.~1. 4234491716us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294201] <idle>-0 0*.~1. 4234491717us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 399937 us, 21596603 cycles)
[ 4235.294208] <idle>-0 0*Nh1. 4234491720us : evl_timer_start: timer=[proxy-timer/0] value=4234481967511 interval=0
[ 4235.294215] <idle>-0 0*Nh1. 4234491721us : evl_timer_shot: [proxy-timer/0] at 4234.492622 (delay: 901 us, 48669 cycles)
[ 4235.294222] <idle>-0 0*.~1. 4234492623us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294229] <idle>-0 0*.~1. 4234492624us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 399029 us, 21547592 cycles)
[ 4235.294236] <idle>-0 0*Nh1. 4234492628us : evl_timer_start: timer=[proxy-timer/0] value=4234482975006 interval=0
[ 4235.294243] <idle>-0 0*Nh1. 4234492629us : evl_timer_shot: [proxy-timer/0] at 4234.493629 (delay: 1000 us, 54039 cycles)
[ 4235.294249] <idle>-0 0*.~1. 4234493631us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294256] <idle>-0 0*.~1. 4234493633us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 398021 us, 21493148 cycles)
[ 4235.294263] <idle>-0 0*Nh1. 4234493637us : evl_timer_start: timer=[proxy-timer/0] value=4234483074801 interval=0
[ 4235.294270] <idle>-0 0*Nh1. 4234493638us : evl_timer_shot: [proxy-timer/0] at 4234.493729 (delay: 91 us, 4942 cycles)
[ 4235.294277] <idle>-0 0*.~1. 4234493730us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294284] <idle>-0 0*.~1. 4234493732us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 397922 us, 21487805 cycles)
[ 4235.294291] <idle>-0 0*Nh1. 4234493736us : evl_timer_start: timer=[proxy-timer/0] value=4234483394721 interval=0
[ 4235.294298] <idle>-0 0*Nh1. 4234493737us : evl_timer_shot: [proxy-timer/0] at 4234.494049 (delay: 312 us, 16901 cycles)
[ 4235.294304] <idle>-0 0*.~1. 4234494050us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294311] <idle>-0 0*.~1. 4234494052us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 397602 us, 21470513 cycles)
[ 4235.294318] <idle>-0 0*Nh1. 4234494056us : evl_timer_start: timer=[proxy-timer/0] value=4234484000112 interval=0
[ 4235.294325] <idle>-0 0*Nh1. 4234494057us : evl_timer_shot: [proxy-timer/0] at 4234.494655 (delay: 598 us, 32305 cycles)
[ 4235.294331] <idle>-0 2*.~1. 4234494655us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.294338] <idle>-0 0*.~1. 4234494656us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294345] <idle>-0 0*.~1. 4234494658us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 396996 us, 21437794 cycles)
[ 4235.294352] <idle>-0 2*.h1. 4234494659us : evl_timer_start: timer=[proxy-timer/2] value=4234488000111 interval=0
[ 4235.294359] <idle>-0 2*.h1. 4234494660us : evl_timer_shot: [proxy-timer/2] at 4234.498655 (delay: 3994 us, 215708 cycles)
[ 4235.294366] <idle>-0 0*.h1. 4234494662us : evl_timer_start: timer=[proxy-timer/0] value=4234485090668 interval=0
[ 4235.294373] <idle>-0 0*.h1. 4234494663us : evl_timer_shot: [proxy-timer/0] at 4234.495745 (delay: 1082 us, 58428 cycles)
[ 4235.294379] <idle>-0 0*.~1. 4234495746us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294386] <idle>-0 0*.~1. 4234495748us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 395906 us, 21378942 cycles)
[ 4235.294393] <idle>-0 0*Nh1. 4234495751us : evl_timer_start: timer=[proxy-timer/0] value=4234485989279 interval=0
[ 4235.294400] <idle>-0 0*Nh1. 4234495752us : evl_timer_shot: [proxy-timer/0] at 4234.496644 (delay: 891 us, 48147 cycles)
[ 4235.294407] <idle>-0 0*.~1. 4234496645us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294414] <idle>-0 0*.~1. 4234496646us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 395007 us, 21330418 cycles)
[ 4235.294421] <idle>-0 0*Nh1. 4234496650us : evl_timer_start: timer=[proxy-timer/0] value=4234487058021 interval=0
[ 4235.294428] <idle>-0 0*Nh1. 4234496652us : evl_timer_shot: [proxy-timer/0] at 4234.497714 (delay: 1061 us, 57335 cycles)
[ 4235.294434] <idle>-0 0*.~1. 4234497714us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294441] <idle>-0 0*.~1. 4234497715us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 393939 us, 21272713 cycles)
[ 4235.294448] <idle>-0 0*Nh1. 4234497719us : evl_timer_start: timer=[proxy-timer/0] value=4234487107776 interval=0
[ 4235.294456] <idle>-0 0*Nh1. 4234497719us : evl_timer_shot: [proxy-timer/0] at 4234.497762 (delay: 43 us, 2326 cycles)
[ 4235.294462] <idle>-0 0*.~1. 4234497764us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294469] <idle>-0 0*.~1. 4234497765us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 393889 us, 21270022 cycles)
[ 4235.294476] <idle>-0 0*Nh1. 4234497768us : evl_timer_start: timer=[proxy-timer/0] value=4234488000148 interval=0
[ 4235.294484] <idle>-0 0*Nh1. 4234497769us : evl_timer_shot: [proxy-timer/0] at 4234.498655 (delay: 885 us, 47813 cycles)
[ 4235.294490] <idle>-0 2*.~1. 4234498655us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.294496] <idle>-0 0*.~1. 4234498656us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294503] <idle>-0 0*.~1. 4234498657us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 392996 us, 21221837 cycles)
[ 4235.294510] <idle>-0 2*.h1. 4234498659us : evl_timer_start: timer=[proxy-timer/2] value=4234492000130 interval=0
[ 4235.294517] <idle>-0 2*.h1. 4234498660us : evl_timer_shot: [proxy-timer/2] at 4234.502655 (delay: 3994 us, 215724 cycles)
[ 4235.294524] <idle>-0 0*.h1. 4234498662us : evl_timer_start: timer=[proxy-timer/0] value=4234488760378 interval=0
[ 4235.294531] <idle>-0 0*.h1. 4234498663us : evl_timer_shot: [proxy-timer/0] at 4234.499415 (delay: 752 us, 40632 cycles)
[ 4235.294538] <idle>-0 0*.~1. 4234499416us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294545] <idle>-0 0*.~1. 4234499417us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 392236 us, 21180789 cycles)
[ 4235.294552] <idle>-0 0*Nh1. 4234499421us : evl_timer_start: timer=[proxy-timer/0] value=4234488985983 interval=0
[ 4235.294559] <idle>-0 0*Nh1. 4234499422us : evl_timer_shot: [proxy-timer/0] at 4234.499640 (delay: 218 us, 11812 cycles)
[ 4235.294566] RT engin-361 0..... 4234499432us : evl_inband_sysentry: syscall=oob_ioctl
[ 4235.294572] RT engin-361 0..... 4234499433us : evl_switch_oob: state=0x4880 info=0x0
[ 4235.294579] <idle>-0 0*..2. 4234499438us : evl_release_thread: name=RT engine-tid_361 pid=361 mask=0x80 info=0x0
[ 4235.294586] <idle>-0 0*.~2. 4234499439us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.294592] <idle>-0 0*.~2. 4234499441us : evl_timer_start: timer=[rrb-timer/0] value=4234493785307 interval=0
[ 4235.294599] <idle>-0 0*.~2. 4234499441us : evl_pick_thread: { next=RT engine-tid_361[361] }
[ 4235.294607] <idle>-0 0*.~2. 4234499442us : evl_switch_context: { ROOT/0[0] prio=-1, state=0x1010 } => { RT engine-tid_361[361] prio=20 }
[ 4235.294614] RT engin-361 0d.~2. 4234499444us : evl_switched_oob: state=0x4800 info=0x0
[ 4235.294620] RT engin-361 0d.~2. 4234499446us : evl_mutex_lock: mutex=0000000041a7b4c3
[ 4235.294627] RT engin-361 0d.~2. 4234499447us : evl_inband_sysexit: result=0
[ 4235.294633] RT engin-361 0d.~2. 4234499453us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.294640] RT engin-361 0*.~2. 4234499454us : evl_wake_up: wq=&mon->wait_queue
[ 4235.294647] RT engin-361 0*.~2. 4234499456us : evl_wakeup_thread: name=VM1Thread-tid_363 pid=363 mask=0x2 info=0x0
[ 4235.294653] RT engin-361 0d.~2. 4234499457us : evl_schedule: flags=0x0, local_flags=0x10000000
[ 4235.294660] RT engin-361 0d.~2. 4234499458us : evl_oob_sysexit: result=0
[ 4235.294667] <idle>-0 2*.~1. 4234499459us : evl_reschedule_ipi: flags=0x10000000, local_flags=0x4000
[ 4235.294673] <idle>-0 2*.~1. 4234499460us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.294680] RT engin-361 0d.~2. 4234499461us : evl_switch_inband: cause=in-band syscall
[ 4235.294686] <idle>-0 2*.~1. 4234499461us : evl_pick_thread: { next=VM1Thread-tid_363[363] }
[ 4235.294694] <idle>-0 2*.~1. 4234499462us : evl_switch_context: { ROOT/2[0] prio=-1, state=0x1010 } => { VM1Thread-tid_363[363] prio=90 }
[ 4235.294700] RT engin-361 0*.~2. 4234499462us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.294707] RT engin-361 0*.~2. 4234499463us : evl_timer_stop: timer=[rrb-timer/0]
[ 4235.294714] <...>-363 2*.~2. 4234499463us : evl_switch_tail: { current=VM1Thread-tid_363[363] }
[ 4235.294720] RT engin-361 0*.~2. 4234499464us : evl_pick_thread: { next=ROOT/0[0] }
[ 4235.294727] <...>-363 2d.~2. 4234499464us : evl_finish_wait: wq=&mon->wait_queue
[ 4235.294734] RT engin-361 0*.~2. 4234499465us : evl_switch_context: { RT engine-tid_361[361] prio=20, state=0x4880 } => { ROOT/0[0] prio=-1 }
[ 4235.294741] <...>-363 2d.~2. 4234499465us : evl_oob_sysexit: result=0
[ 4235.294747] <idle>-0 0*.~2. 4234499466us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.294754] <idle>-0 0d.h1. 4234499468us : evl_inband_wakeup: pid=361 comm=RT engine-tid_3
[ 4235.294760] RT engin-361 0*..2. 4234499472us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.294767] RT engin-361 0..... 4234499473us : evl_switched_inband: state=0x4880 info=0x0
[ 4235.294773] <...>-363 2d.~2. 4234499483us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.294779] <...>-363 2*.~2. 4234499485us : evl_wait: wq=&mon->wait_queue
[ 4235.294787] <...>-363 2*.~2. 4234499486us : evl_sleep_on: pid=363 timeout=0 timeout_mode=0 clock=monotonic wchan=&mon->wait_queue(00000000ea5e79e5)
[ 4235.294794] <...>-363 2d.~2. 4234499487us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.294800] <...>-363 2*.~2. 4234499488us : evl_pick_thread: { next=ROOT/2[0] }
[ 4235.294808] <...>-363 2*.~2. 4234499489us : evl_switch_context: { VM1Thread-tid_363[363] prio=90, state=0x4002 } => { ROOT/2[0] prio=-1 }
[ 4235.294814] <idle>-0 2*.~1. 4234499490us : evl_switch_tail: { current=ROOT/2[0] }
[ 4235.294821] <idle>-0 0*.~1. 4234499642us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294828] <idle>-0 0*.~1. 4234499643us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 392011 us, 21168599 cycles)
[ 4235.294835] <idle>-0 0*Nh1. 4234499647us : evl_timer_start: timer=[proxy-timer/0] value=4234488999613 interval=0
[ 4235.294842] <idle>-0 0*Nh1. 4234499649us : evl_timer_shot: [proxy-timer/0] at 4234.499654 (delay: 5 us, 316 cycles)
[ 4235.294849] ModbusTh-431 0D.~.. 4234499655us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294856] ModbusTh-431 0D.~.. 4234499656us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 391997 us, 21167881 cycles)
[ 4235.294863] ModbusTh-431 0*.h.. 4234499659us : evl_timer_start: timer=[proxy-timer/0] value=4234489003311 interval=0
[ 4235.294870] ModbusTh-431 0*.h.. 4234499661us : evl_timer_shot: [proxy-timer/0] at 4234.499661 (delay: 1 us, 54 cycles)
[ 4235.294876] ModbusTh-431 0D.~.. 4234499662us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294883] ModbusTh-431 0D.~.. 4234499663us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 391990 us, 21167494 cycles)
[ 4235.294890] ModbusTh-431 0*.h.. 4234499666us : evl_timer_start: timer=[proxy-timer/0] value=4234489011983 interval=0
[ 4235.294897] ModbusTh-431 0*.h.. 4234499667us : evl_timer_shot: [proxy-timer/0] at 4234.499667 (delay: 1 us, 54 cycles)
[ 4235.294903] ModbusTh-431 0D.~.. 4234499668us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.294910] ModbusTh-431 0D.~.. 4234499669us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 391984 us, 21167178 cycles)
[ 4235.294917] ModbusTh-431 0*.h.. 4234499672us : evl_timer_start: timer=[proxy-timer/0] value=4234489122920 interval=0
[ 4235.294924] ModbusTh-431 0*.h.. 4234499673us : evl_timer_shot: [proxy-timer/0] at 4234.499777 (delay: 104 us, 5669 cycles)
[ 4235.294930] ModbusTh-431 0..... 4234499678us : evl_inband_sysentry: syscall=oob_ioctl
[ 4235.294937] ModbusTh-431 0..... 4234499678us : evl_switch_oob: state=0x4080 info=0x0
[ 4235.294943] CommandC-364 0*..2. 4234499683us : evl_release_thread: name=ModbusThread-tid_431 pid=431 mask=0x80 info=0x0
[ 4235.294950] CommandC-364 0*.~2. 4234499684us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.294957] CommandC-364 0*.~2. 4234499685us : evl_pick_thread: { next=ModbusThread-tid_431[431] }
[ 4235.294964] CommandC-364 0*.~2. 4234499686us : evl_switch_context: { ROOT/0[0] prio=-1, state=0x1010 } => { ModbusThread-tid_431[431] prio=75 }
[ 4235.294971] ModbusTh-431 0d.~2. 4234499687us : evl_switched_oob: state=0x4000 info=0x0
[ 4235.294978] ModbusTh-431 0d.~2. 4234499689us : evl_mutex_lock: mutex=0000000030b10aed
[ 4235.294984] ModbusTh-431 0d.~2. 4234499690us : evl_inband_sysexit: result=0
[ 4235.294991] ModbusTh-431 0d.~2. 4234499694us : evl_switch_inband: cause=in-band syscall
[ 4235.294997] ModbusTh-431 0*.~2. 4234499695us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.295004] ModbusTh-431 0*.~2. 4234499696us : evl_pick_thread: { next=ROOT/0[0] }
[ 4235.295011] ModbusTh-431 0*.~2. 4234499697us : evl_switch_context: { ModbusThread-tid_431[431] prio=75, state=0x4080 } => { ROOT/0[0] prio=-1 }
[ 4235.295018] CommandC-364 0*.~2. 4234499698us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.295025] CommandC-364 0d.h1. 4234499700us : evl_inband_wakeup: pid=431 comm=ModbusThread-ti
[ 4235.295031] ModbusTh-431 0*..2. 4234499704us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.295038] ModbusTh-431 0..... 4234499705us : evl_switched_inband: state=0x4080 info=0x0
[ 4235.295045] <idle>-0 0*.~1. 4234499779us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295052] <idle>-0 0*.~1. 4234499780us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 391873 us, 21161185 cycles)
[ 4235.295059] <idle>-0 0*Nh1. 4234499784us : evl_timer_start: timer=[proxy-timer/0] value=4234492057510 interval=0
[ 4235.295066] <idle>-0 0*Nh1. 4234499786us : evl_timer_shot: [proxy-timer/0] at 4234.502712 (delay: 2926 us, 158044 cycles)
[ 4235.295073] <idle>-0 0*N.2. 4234499788us : evl_timer_start: timer=[proxy-timer/0] value=4234492000111 interval=0
[ 4235.295080] <idle>-0 0*N.2. 4234499788us : evl_timer_shot: [proxy-timer/0] at 4234.502655 (delay: 2866 us, 154788 cycles)
[ 4235.295087] T5Server-367 0*..1. 4234499799us : evl_timer_start: timer=[proxy-timer/0] value=4234491142973 interval=0
[ 4235.295094] T5Server-367 0*..1. 4234499800us : evl_timer_shot: [proxy-timer/0] at 4234.501797 (delay: 1998 us, 107895 cycles)
[ 4235.295101] <idle>-0 0*.~1. 4234501799us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295108] <idle>-0 0*.~1. 4234501800us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 389854 us, 21052133 cycles)
[ 4235.295115] <idle>-0 0*Nh1. 4234501803us : evl_timer_start: timer=[proxy-timer/0] value=4234492057491 interval=0
[ 4235.295122] <idle>-0 0*Nh1. 4234501804us : evl_timer_shot: [proxy-timer/0] at 4234.502712 (delay: 908 us, 49037 cycles)
[ 4235.295129] <idle>-0 0*N.2. 4234501806us : evl_timer_start: timer=[proxy-timer/0] value=4234492000130 interval=0
[ 4235.295136] <idle>-0 0*N.2. 4234501807us : evl_timer_shot: [proxy-timer/0] at 4234.502655 (delay: 847 us, 45785 cycles)
[ 4235.295143] <idle>-0 0*..2. 4234501821us : evl_timer_start: timer=[proxy-timer/0] value=4234492057510 interval=0
[ 4235.295150] <idle>-0 0*..2. 4234501823us : evl_timer_shot: [proxy-timer/0] at 4234.502712 (delay: 889 us, 48050 cycles)
[ 4235.295157] <idle>-0 2*.~1. 4234502656us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.295164] <idle>-0 2*.h1. 4234502662us : evl_timer_start: timer=[proxy-timer/2] value=4234496000130 interval=0
[ 4235.295171] <idle>-0 2*.h1. 4234502663us : evl_timer_shot: [proxy-timer/2] at 4234.506655 (delay: 3991 us, 215546 cycles)
[ 4235.295178] <idle>-0 0*.~1. 4234502713us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295185] <idle>-0 0*.~1. 4234502714us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 388939 us, 21002738 cycles)
[ 4235.295192] <idle>-0 0*Nh1. 4234502718us : evl_timer_start: timer=[proxy-timer/0] value=4234492077161 interval=0
[ 4235.295199] <idle>-0 0*Nh1. 4234502719us : evl_timer_shot: [proxy-timer/0] at 4234.502732 (delay: 12 us, 675 cycles)
[ 4235.295205] <idle>-0 0*.~1. 4234502733us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295212] <idle>-0 0*.~1. 4234502734us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 388920 us, 21001682 cycles)
[ 4235.295219] <idle>-0 0*Nh1. 4234502738us : evl_timer_start: timer=[proxy-timer/0] value=4234493160099 interval=0
[ 4235.295227] <idle>-0 0*Nh1. 4234502740us : evl_timer_shot: [proxy-timer/0] at 4234.503816 (delay: 1075 us, 58097 cycles)
[ 4235.295233] <idle>-0 0*.~1. 4234503816us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295240] <idle>-0 0*.~1. 4234503817us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 387836 us, 20943190 cycles)
[ 4235.295247] <idle>-0 0*Nh1. 4234503821us : evl_timer_start: timer=[proxy-timer/0] value=4234493407705 interval=0
[ 4235.295255] <idle>-0 0*Nh1. 4234503822us : evl_timer_shot: [proxy-timer/0] at 4234.504062 (delay: 240 us, 12980 cycles)
[ 4235.295261] <idle>-0 0*.~1. 4234504063us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295268] <idle>-0 0*.~1. 4234504064us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 387589 us, 20929828 cycles)
[ 4235.295275] <idle>-0 0*Nh1. 4234504068us : evl_timer_start: timer=[proxy-timer/0] value=4234494566694 interval=0
[ 4235.295283] <idle>-0 0*Nh1. 4234504069us : evl_timer_shot: [proxy-timer/0] at 4234.505221 (delay: 1152 us, 62214 cycles)
[ 4235.295289] <idle>-0 0*.~1. 4234505222us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295296] <idle>-0 0*.~1. 4234505223us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 386430 us, 20867252 cycles)
[ 4235.295303] <idle>-0 0*.h1. 4234505228us : evl_timer_start: timer=[proxy-timer/0] value=4234495071283 interval=0
[ 4235.295310] <idle>-0 0*.h1. 4234505229us : evl_timer_shot: [proxy-timer/0] at 4234.505726 (delay: 496 us, 26824 cycles)
[ 4235.295317] <idle>-0 0*.~1. 4234505727us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295324] <idle>-0 0*.~1. 4234505727us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 385926 us, 20840028 cycles)
[ 4235.295331] <idle>-0 0*Nh1. 4234505731us : evl_timer_start: timer=[proxy-timer/0] value=4234495176541 interval=0
[ 4235.295338] <idle>-0 0*Nh1. 4234505732us : evl_timer_shot: [proxy-timer/0] at 4234.505831 (delay: 99 us, 5369 cycles)
[ 4235.295344] <idle>-0 0*.~1. 4234505832us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295351] <idle>-0 0*.~1. 4234505833us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 385820 us, 20834317 cycles)
[ 4235.295358] <idle>-0 0*Nh1. 4234505837us : evl_timer_start: timer=[proxy-timer/0] value=4234497099542 interval=0
[ 4235.295366] <idle>-0 0*Nh1. 4234505838us : evl_timer_shot: [proxy-timer/0] at 4234.507754 (delay: 1916 us, 103480 cycles)
[ 4235.295372] <idle>-0 0*N.2. 4234505840us : evl_timer_start: timer=[proxy-timer/0] value=4234496000129 interval=0
[ 4235.295380] <idle>-0 0*N.2. 4234505841us : evl_timer_shot: [proxy-timer/0] at 4234.506655 (delay: 814 us, 43961 cycles)
[ 4235.295387] <idle>-0 0*..2. 4234505855us : evl_timer_start: timer=[proxy-timer/0] value=4234497099542 interval=0
[ 4235.295394] <idle>-0 0*..2. 4234505856us : evl_timer_shot: [proxy-timer/0] at 4234.507754 (delay: 1898 us, 102501 cycles)
[ 4235.295400] <idle>-0 2*.~1. 4234506656us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.295407] <idle>-0 2*.h1. 4234506660us : evl_timer_start: timer=[proxy-timer/2] value=4234500000111 interval=0
[ 4235.295415] <idle>-0 2*.h1. 4234506661us : evl_timer_shot: [proxy-timer/2] at 4234.510655 (delay: 3994 us, 215679 cycles)
[ 4235.295421] <idle>-0 0*.~1. 4234507755us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295428] <idle>-0 0*.~1. 4234507756us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 383897 us, 20730486 cycles)
[ 4235.295435] <idle>-0 0*Nh1. 4234507760us : evl_timer_start: timer=[proxy-timer/0] value=4234497194130 interval=0
[ 4235.295442] <idle>-0 0*Nh1. 4234507760us : evl_timer_shot: [proxy-timer/0] at 4234.507849 (delay: 88 us, 4777 cycles)
[ 4235.295449] <idle>-0 0*.~1. 4234507850us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295456] <idle>-0 0*.~1. 4234507851us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 383802 us, 20725358 cycles)
[ 4235.295463] <idle>-0 0*Nh1. 4234507855us : evl_timer_start: timer=[proxy-timer/0] value=4234498083333 interval=0
[ 4235.295470] <idle>-0 0*Nh1. 4234507856us : evl_timer_shot: [proxy-timer/0] at 4234.508738 (delay: 882 us, 47647 cycles)
[ 4235.295476] <idle>-0 0*.~1. 4234508739us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295483] <idle>-0 0*.~1. 4234508740us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 382913 us, 20677349 cycles)
[ 4235.295491] <idle>-0 0*Nh1. 4234508744us : evl_timer_start: timer=[proxy-timer/0] value=4234498819474 interval=0
[ 4235.295498] <idle>-0 0*Nh1. 4234508745us : evl_timer_shot: [proxy-timer/0] at 4234.509474 (delay: 729 us, 39382 cycles)
[ 4235.295504] <idle>-0 0*.~1. 4234509475us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295511] <idle>-0 0*.~1. 4234509476us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 382177 us, 20637602 cycles)
[ 4235.295518] <idle>-0 0*Nh1. 4234509480us : evl_timer_start: timer=[proxy-timer/0] value=4234499209108 interval=0
[ 4235.295525] <idle>-0 0*Nh1. 4234509481us : evl_timer_shot: [proxy-timer/0] at 4234.509864 (delay: 383 us, 20689 cycles)
[ 4235.295532] RT engin-361 0..... 4234509488us : evl_inband_sysentry: syscall=oob_ioctl
[ 4235.295538] RT engin-361 0..... 4234509489us : evl_switch_oob: state=0x4880 info=0x0
[ 4235.295545] <idle>-0 0*..2. 4234509493us : evl_release_thread: name=RT engine-tid_361 pid=361 mask=0x80 info=0x0
[ 4235.295552] <idle>-0 0*.~2. 4234509494us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.295559] <idle>-0 0*.~2. 4234509495us : evl_timer_start: timer=[rrb-timer/0] value=4234503839976 interval=0
[ 4235.295566] <idle>-0 0*.~2. 4234509496us : evl_pick_thread: { next=RT engine-tid_361[361] }
[ 4235.295573] <idle>-0 0*.~2. 4234509497us : evl_switch_context: { ROOT/0[0] prio=-1, state=0x1010 } => { RT engine-tid_361[361] prio=20 }
[ 4235.295580] RT engin-361 0d.~2. 4234509498us : evl_switched_oob: state=0x4800 info=0x0
[ 4235.295587] RT engin-361 0d.~2. 4234509500us : evl_mutex_lock: mutex=0000000041a7b4c3
[ 4235.295593] RT engin-361 0d.~2. 4234509501us : evl_inband_sysexit: result=0
[ 4235.295600] RT engin-361 0d.~2. 4234509505us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.295606] RT engin-361 0*.~2. 4234509506us : evl_wake_up: wq=&mon->wait_queue
[ 4235.295613] RT engin-361 0*.~2. 4234509507us : evl_wakeup_thread: name=VM1Thread-tid_363 pid=363 mask=0x2 info=0x0
[ 4235.295620] RT engin-361 0d.~2. 4234509508us : evl_schedule: flags=0x0, local_flags=0x10000000
[ 4235.295627] RT engin-361 0d.~2. 4234509509us : evl_oob_sysexit: result=0
[ 4235.295634] <idle>-0 2*.~1. 4234509510us : evl_reschedule_ipi: flags=0x10000000, local_flags=0x4000
[ 4235.295640] <idle>-0 2*.~1. 4234509511us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.295647] RT engin-361 0d.~2. 4234509511us : evl_switch_inband: cause=in-band syscall
[ 4235.295654] <idle>-0 2*.~1. 4234509512us : evl_pick_thread: { next=VM1Thread-tid_363[363] }
[ 4235.295661] <idle>-0 2*.~1. 4234509512us : evl_switch_context: { ROOT/2[0] prio=-1, state=0x1010 } => { VM1Thread-tid_363[363] prio=90 }
[ 4235.295668] RT engin-361 0*.~2. 4234509512us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.295675] RT engin-361 0*.~2. 4234509514us : evl_timer_stop: timer=[rrb-timer/0]
[ 4235.295681] <...>-363 2*.~2. 4234509514us : evl_switch_tail: { current=VM1Thread-tid_363[363] }
[ 4235.295688] RT engin-361 0*.~2. 4234509514us : evl_pick_thread: { next=ROOT/0[0] }
[ 4235.295695] <...>-363 2d.~2. 4234509515us : evl_finish_wait: wq=&mon->wait_queue
[ 4235.295702] RT engin-361 0*.~2. 4234509515us : evl_switch_context: { RT engine-tid_361[361] prio=20, state=0x4880 } => { ROOT/0[0] prio=-1 }
[ 4235.295709] <...>-363 2d.~2. 4234509516us : evl_oob_sysexit: result=0
[ 4235.295716] <idle>-0 0*.~2. 4234509516us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.295722] <idle>-0 0d.h1. 4234509518us : evl_inband_wakeup: pid=361 comm=RT engine-tid_3
[ 4235.295729] RT engin-361 0*..2. 4234509522us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.295735] RT engin-361 0..... 4234509523us : evl_switched_inband: state=0x4880 info=0x0
[ 4235.295742] <...>-363 2d.~2. 4234509529us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.295748] <...>-363 2*.~2. 4234509530us : evl_wait: wq=&mon->wait_queue
[ 4235.295756] <...>-363 2*.~2. 4234509531us : evl_sleep_on: pid=363 timeout=0 timeout_mode=0 clock=monotonic wchan=&mon->wait_queue(00000000ea5e79e5)
[ 4235.295763] <...>-363 2d.~2. 4234509532us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.295769] <...>-363 2*.~2. 4234509533us : evl_pick_thread: { next=ROOT/2[0] }
[ 4235.295776] <...>-363 2*.~2. 4234509534us : evl_switch_context: { VM1Thread-tid_363[363] prio=90, state=0x4002 } => { ROOT/2[0] prio=-1 }
[ 4235.295783] <idle>-0 2*.~1. 4234509535us : evl_switch_tail: { current=ROOT/2[0] }
[ 4235.295790] <idle>-0 0*.~1. 4234509865us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295796] <idle>-0 0*.~1. 4234509866us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 381787 us, 20616550 cycles)
[ 4235.295804] <idle>-0 0*Nh1. 4234509870us : evl_timer_start: timer=[proxy-timer/0] value=4234500000111 interval=0
[ 4235.295811] <idle>-0 0*Nh1. 4234509871us : evl_timer_shot: [proxy-timer/0] at 4234.510655 (delay: 783 us, 42332 cycles)
[ 4235.295817] <idle>-0 2*.~1. 4234510656us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.295823] <idle>-0 0*.~1. 4234510656us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295831] <idle>-0 0*.~1. 4234510657us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 380997 us, 20573838 cycles)
[ 4235.295838] <idle>-0 2*.h1. 4234510661us : evl_timer_start: timer=[proxy-timer/2] value=4234504000130 interval=0
[ 4235.295844] <idle>-0 0*.h1. 4234510662us : evl_timer_start: timer=[proxy-timer/0] value=4234501095551 interval=0
[ 4235.295852] <idle>-0 2*.h1. 4234510662us : evl_timer_shot: [proxy-timer/2] at 4234.514655 (delay: 3992 us, 215621 cycles)
[ 4235.295859] <idle>-0 0*.h1. 4234510663us : evl_timer_shot: [proxy-timer/0] at 4234.511750 (delay: 1087 us, 58721 cycles)
[ 4235.295866] <idle>-0 0*.~1. 4234511752us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295873] <idle>-0 0*.~1. 4234511753us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 379900 us, 20514639 cycles)
[ 4235.295880] <idle>-0 0*Nh1. 4234511758us : evl_timer_start: timer=[proxy-timer/0] value=4234501225049 interval=0
[ 4235.295887] <idle>-0 0*Nh1. 4234511759us : evl_timer_shot: [proxy-timer/0] at 4234.511880 (delay: 121 us, 6534 cycles)
[ 4235.295894] <idle>-0 0*.~1. 4234511881us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295901] <idle>-0 0*.~1. 4234511882us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 379772 us, 20507692 cycles)
[ 4235.295908] <idle>-0 0*Nh1. 4234511886us : evl_timer_start: timer=[proxy-timer/0] value=4234502118478 interval=0
[ 4235.295915] <idle>-0 0*Nh1. 4234511887us : evl_timer_shot: [proxy-timer/0] at 4234.512773 (delay: 886 us, 47862 cycles)
[ 4235.295921] <idle>-0 0*.~1. 4234512774us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.295928] <idle>-0 0*.~1. 4234512775us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 378878 us, 20459441 cycles)
[ 4235.295935] <idle>-0 0*Nh1. 4234512779us : evl_timer_start: timer=[proxy-timer/0] value=4234503242065 interval=0
[ 4235.295943] <idle>-0 0*Nh1. 4234512780us : evl_timer_shot: [proxy-timer/0] at 4234.513897 (delay: 1116 us, 60295 cycles)
[ 4235.295949] T5Server-366 0..... 4234512806us : evl_inband_sysentry: syscall=oob_ioctl
[ 4235.295956] T5Server-366 0..... 4234512807us : evl_switch_oob: state=0x4880 info=0x0
[ 4235.295963] <idle>-0 0*..2. 4234512813us : evl_release_thread: name=T5Server_K5T5-tid_366 pid=366 mask=0x80 info=0x0
[ 4235.295970] <idle>-0 0*.~2. 4234512814us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.295976] <idle>-0 0*.~2. 4234512815us : evl_timer_start: timer=[rrb-timer/0] value=4234507159692 interval=0
[ 4235.295983] <idle>-0 0*.~2. 4234512816us : evl_pick_thread: { next=T5Server_K5T5-tid_366[366] }
[ 4235.295990] <idle>-0 0*.~2. 4234512817us : evl_switch_context: { ROOT/0[0] prio=-1, state=0x1010 } => { T5Server_K5T5-tid_366[366] prio=20 }
[ 4235.295998] T5Server-366 0d.~2. 4234512818us : evl_switched_oob: state=0x4800 info=0x0
[ 4235.296004] T5Server-366 0d.~2. 4234512820us : evl_mutex_lock: mutex=000000005cfd6a29
[ 4235.296010] T5Server-366 0d.~2. 4234512821us : evl_inband_sysexit: result=0
[ 4235.296017] T5Server-366 0d.~2. 4234512825us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.296024] T5Server-366 0*.~2. 4234512828us : evl_wait: wq=&mon->wait_queue
[ 4235.296031] T5Server-366 0*.~2. 4234512829us : evl_sleep_on: pid=366 timeout=4244502168692 timeout_mode=1 clock=monotonic wchan=&mon->wait_queue(00000000c7be549f)
[ 4235.296039] T5Server-366 0*.~2. 4234512830us : evl_timer_start: timer=T5Server_K5T5-tid_366 value=4244502168692 interval=0
[ 4235.296045] T5Server-366 0d.~2. 4234512831us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.296052] T5Server-366 0*.~2. 4234512832us : evl_timer_stop: timer=[rrb-timer/0]
[ 4235.296058] T5Server-366 0*.~2. 4234512832us : evl_pick_thread: { next=ROOT/0[0] }
[ 4235.296065] T5Server-366 0*.~2. 4234512833us : evl_switch_context: { T5Server_K5T5-tid_366[366] prio=20, state=0x4806 } => { ROOT/0[0] prio=-1 }
[ 4235.296072] <idle>-0 0*.~2. 4234512834us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.296079] <idle>-0 0*.~1. 4234513898us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.296086] <idle>-0 0*.~1. 4234513899us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 377755 us, 20398781 cycles)
[ 4235.296093] <idle>-0 0*Nh1. 4234513903us : evl_timer_start: timer=[proxy-timer/0] value=4234503421188 interval=0
[ 4235.296100] <idle>-0 0*Nh1. 4234513904us : evl_timer_shot: [proxy-timer/0] at 4234.514076 (delay: 171 us, 9286 cycles)
[ 4235.296107] <idle>-0 0*.~1. 4234514077us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.296114] <idle>-0 0*.~1. 4234514078us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 377575 us, 20389096 cycles)
[ 4235.296120] <idle>-0 0*Nh1. 4234514082us : evl_timer_start: timer=[proxy-timer/0] value=4234504112045 interval=0
[ 4235.296128] <idle>-0 0*Nh1. 4234514083us : evl_timer_shot: [proxy-timer/0] at 4234.514767 (delay: 683 us, 36930 cycles)
[ 4235.296134] <idle>-0 0*N.2. 4234514085us : evl_timer_start: timer=[proxy-timer/0] value=4234504000111 interval=0
[ 4235.296142] <idle>-0 0*N.2. 4234514086us : evl_timer_shot: [proxy-timer/0] at 4234.514655 (delay: 568 us, 30725 cycles)
[ 4235.296149] <idle>-0 0*..2. 4234514098us : evl_timer_start: timer=[proxy-timer/0] value=4234504112027 interval=0
[ 4235.296156] <idle>-0 0*..2. 4234514099us : evl_timer_shot: [proxy-timer/0] at 4234.514767 (delay: 668 us, 36082 cycles)
[ 4235.296162] <idle>-0 2*.~1. 4234514656us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.296169] <idle>-0 2*.h1. 4234514659us : evl_timer_start: timer=[proxy-timer/2] value=4234508000130 interval=0
[ 4235.296176] <idle>-0 2*.h1. 4234514660us : evl_timer_shot: [proxy-timer/2] at 4234.518655 (delay: 3994 us, 215708 cycles)
[ 4235.296183] <idle>-0 0*.~1. 4234514768us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.296190] <idle>-0 0*.~1. 4234514769us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 376885 us, 20351804 cycles)
[ 4235.296197] <idle>-0 0*Nh1. 4234514772us : evl_timer_start: timer=[proxy-timer/0] value=4234504452344 interval=0
[ 4235.296204] <idle>-0 0*Nh1. 4234514773us : evl_timer_shot: [proxy-timer/0] at 4234.515107 (delay: 333 us, 18024 cycles)
[ 4235.296211] CommandC-364 0..... 4234514781us : evl_inband_sysentry: syscall=oob_ioctl
[ 4235.296218] CommandC-364 0..... 4234514782us : evl_switch_oob: state=0x4080 info=0x0
[ 4235.296225] <idle>-0 0*..2. 4234514786us : evl_release_thread: name=CommandChannelServer-tid_364 pid=364 mask=0x80 info=0x0
[ 4235.296231] <idle>-0 0*.~2. 4234514788us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.296238] <idle>-0 0*.~2. 4234514788us : evl_pick_thread: { next=CommandChannelServer-tid_364[364] }
[ 4235.296246] <idle>-0 0*.~2. 4234514789us : evl_switch_context: { ROOT/0[0] prio=-1, state=0x1010 } => { CommandChannelServer-tid_364[364] prio=70 }
[ 4235.296253] CommandC-364 0d.~2. 4234514791us : evl_switched_oob: state=0x4000 info=0x0
[ 4235.296259] CommandC-364 0d.~2. 4234514792us : evl_mutex_lock: mutex=00000000f3b83432
[ 4235.296266] CommandC-364 0d.~2. 4234514793us : evl_inband_sysexit: result=0
[ 4235.296272] CommandC-364 0d.~2. 4234514801us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.296279] CommandC-364 0*.~2. 4234514802us : evl_wake_up: wq=&mon->wait_queue
[ 4235.296286] CommandC-364 0*.~2. 4234514803us : evl_wakeup_thread: name=T5Server_K5T5-tid_366 pid=366 mask=0x2 info=0x0
[ 4235.296292] CommandC-364 0*.~2. 4234514804us : evl_timer_stop: timer=T5Server_K5T5-tid_366
[ 4235.296299] CommandC-364 0d.~2. 4234514805us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.296306] CommandC-364 0*.~2. 4234514806us : evl_pick_thread: { next=CommandChannelServer-tid_364[364] }
[ 4235.296312] CommandC-364 0d.~2. 4234514806us : evl_oob_sysexit: result=0
[ 4235.296319] CommandC-364 0d.~2. 4234514808us : evl_switch_inband: cause=in-band syscall
[ 4235.296325] CommandC-364 0*.~2. 4234514809us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.296332] CommandC-364 0*.~2. 4234514810us : evl_timer_start: timer=[rrb-timer/0] value=4234509154985 interval=0
[ 4235.296339] CommandC-364 0*.~2. 4234514811us : evl_pick_thread: { next=T5Server_K5T5-tid_366[366] }
[ 4235.296347] CommandC-364 0*.~2. 4234514812us : evl_switch_context: { CommandChannelServer-tid_364[364] prio=70, state=0x4080 } => { T5Server_K5T5-tid_366[366] prio=20 }
[ 4235.296354] T5Server-366 0*.~2. 4234514813us : evl_switch_tail: { current=T5Server_K5T5-tid_366[366] }
[ 4235.296360] T5Server-366 0d.~2. 4234514814us : evl_finish_wait: wq=&mon->wait_queue
[ 4235.296367] T5Server-366 0d.~2. 4234514815us : evl_oob_sysexit: result=0
[ 4235.296373] T5Server-366 0d.~2. 4234514819us : evl_switch_inband: cause=in-band syscall
[ 4235.296379] T5Server-366 0*.~2. 4234514819us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.296386] T5Server-366 0*.~2. 4234514821us : evl_timer_stop: timer=[rrb-timer/0]
[ 4235.296392] T5Server-366 0*.~2. 4234514821us : evl_pick_thread: { next=ROOT/0[0] }
[ 4235.296400] T5Server-366 0*.~2. 4234514822us : evl_switch_context: { T5Server_K5T5-tid_366[366] prio=20, state=0x4880 } => { ROOT/0[0] prio=-1 }
[ 4235.296407] <idle>-0 0*.~2. 4234514823us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.296413] <idle>-0 0d.h1. 4234514825us : evl_inband_wakeup: pid=366 comm=T5Server_K5T5-t
[ 4235.296420] <idle>-0 0dNh1. 4234514827us : evl_inband_wakeup: pid=364 comm=CommandChannelS
[ 4235.296426] CommandC-364 0*..2. 4234514830us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.296433] CommandC-364 0..... 4234514831us : evl_switched_inband: state=0x4080 info=0x0
[ 4235.296439] T5Server-366 0*..2. 4234514836us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.296446] T5Server-366 0..... 4234514837us : evl_switched_inband: state=0x4880 info=0x0
[ 4235.296452] <idle>-0 0*.~1. 4234515108us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.296460] <idle>-0 0*.~1. 4234515109us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 376544 us, 20333418 cycles)
[ 4235.296467] <idle>-0 0*Nh1. 4234515114us : evl_timer_start: timer=[proxy-timer/0] value=4234505259617 interval=0
[ 4235.296474] <idle>-0 0*Nh1. 4234515115us : evl_timer_shot: [proxy-timer/0] at 4234.515914 (delay: 799 us, 43167 cycles)
[ 4235.296480] <idle>-0 0*.~1. 4234515915us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.296488] <idle>-0 0*.~1. 4234515916us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 375737 us, 20289833 cycles)
[ 4235.296494] <idle>-0 0*Nh1. 4234515920us : evl_timer_start: timer=[proxy-timer/0] value=4234507177523 interval=0
[ 4235.296502] <idle>-0 0*Nh1. 4234515921us : evl_timer_shot: [proxy-timer/0] at 4234.517832 (delay: 1911 us, 103216 cycles)
[ 4235.296508] <idle>-0 0*.~1. 4234517833us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.296515] <idle>-0 0*.~1. 4234517834us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 373819 us, 20186252 cycles)
[ 4235.296522] <idle>-0 0*Nh1. 4234517838us : evl_timer_start: timer=[proxy-timer/0] value=4234507276151 interval=0
[ 4235.296529] <idle>-0 0*Nh1. 4234517839us : evl_timer_shot: [proxy-timer/0] at 4234.517931 (delay: 91 us, 4955 cycles)
[ 4235.296536] <idle>-0 0*.~1. 4234517932us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.296543] <idle>-0 0*.~1. 4234517933us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 373721 us, 20180939 cycles)
[ 4235.296550] <idle>-0 0*Nh1. 4234517937us : evl_timer_start: timer=[proxy-timer/0] value=4234508869291 interval=0
[ 4235.296557] <idle>-0 0*Nh1. 4234517938us : evl_timer_shot: [proxy-timer/0] at 4234.519524 (delay: 1586 us, 85668 cycles)
[ 4235.296564] <idle>-0 0*N.2. 4234517940us : evl_timer_start: timer=[proxy-timer/0] value=4234508000111 interval=0
[ 4235.296571] <idle>-0 0*N.2. 4234517941us : evl_timer_shot: [proxy-timer/0] at 4234.518655 (delay: 714 us, 38569 cycles)
[ 4235.296578] <idle>-0 0*..2. 4234517955us : evl_timer_start: timer=[proxy-timer/0] value=4234508869290 interval=0
[ 4235.296585] <idle>-0 0*..2. 4234517956us : evl_timer_shot: [proxy-timer/0] at 4234.519524 (delay: 1567 us, 84661 cycles)
[ 4235.296591] <idle>-0 2*.~1. 4234518656us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.296598] <idle>-0 2*.h1. 4234518662us : evl_timer_start: timer=[proxy-timer/2] value=4234512000111 interval=0
[ 4235.296605] <idle>-0 2*.h1. 4234518663us : evl_timer_shot: [proxy-timer/2] at 4234.522655 (delay: 3991 us, 215567 cycles)
[ 4235.296612] <idle>-0 0*.~1. 4234519525us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.296619] <idle>-0 0*.~1. 4234519526us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 372127 us, 20094909 cycles)
[ 4235.296626] <idle>-0 0*Nh1. 4234519530us : evl_timer_start: timer=[proxy-timer/0] value=4234509182727 interval=0
[ 4235.296633] <idle>-0 0*Nh1. 4234519531us : evl_timer_shot: [proxy-timer/0] at 4234.519837 (delay: 306 us, 16554 cycles)
[ 4235.296640] RT engin-361 0..... 4234519540us : evl_inband_sysentry: syscall=oob_ioctl
[ 4235.296646] RT engin-361 0..... 4234519541us : evl_switch_oob: state=0x4880 info=0x0
[ 4235.296653] <idle>-0 0*..2. 4234519546us : evl_release_thread: name=RT engine-tid_361 pid=361 mask=0x80 info=0x0
[ 4235.296660] <idle>-0 0*.~2. 4234519547us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.296667] <idle>-0 0*.~2. 4234519548us : evl_timer_start: timer=[rrb-timer/0] value=4234513892589 interval=0
[ 4235.296674] <idle>-0 0*.~2. 4234519549us : evl_pick_thread: { next=RT engine-tid_361[361] }
[ 4235.296681] <idle>-0 0*.~2. 4234519549us : evl_switch_context: { ROOT/0[0] prio=-1, state=0x1010 } => { RT engine-tid_361[361] prio=20 }
[ 4235.296688] RT engin-361 0d.~2. 4234519551us : evl_switched_oob: state=0x4800 info=0x0
[ 4235.296695] RT engin-361 0d.~2. 4234519552us : evl_mutex_lock: mutex=0000000041a7b4c3
[ 4235.296701] RT engin-361 0d.~2. 4234519553us : evl_inband_sysexit: result=0
[ 4235.296708] RT engin-361 0d.~2. 4234519558us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.296714] RT engin-361 0*.~2. 4234519559us : evl_wake_up: wq=&mon->wait_queue
[ 4235.296721] RT engin-361 0*.~2. 4234519560us : evl_wakeup_thread: name=VM1Thread-tid_363 pid=363 mask=0x2 info=0x0
[ 4235.296728] RT engin-361 0d.~2. 4234519561us : evl_schedule: flags=0x0, local_flags=0x10000000
[ 4235.296734] RT engin-361 0d.~2. 4234519562us : evl_oob_sysexit: result=0
[ 4235.296741] <idle>-0 2*.~1. 4234519563us : evl_reschedule_ipi: flags=0x10000000, local_flags=0x4000
[ 4235.296747] <idle>-0 2*.~1. 4234519564us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.296754] RT engin-361 0d.~2. 4234519565us : evl_switch_inband: cause=in-band syscall
[ 4235.296761] <idle>-0 2*.~1. 4234519565us : evl_pick_thread: { next=VM1Thread-tid_363[363] }
[ 4235.296768] <idle>-0 2*.~1. 4234519566us : evl_switch_context: { ROOT/2[0] prio=-1, state=0x1010 } => { VM1Thread-tid_363[363] prio=90 }
[ 4235.296775] RT engin-361 0*.~2. 4234519566us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.296781] RT engin-361 0*.~2. 4234519567us : evl_timer_stop: timer=[rrb-timer/0]
[ 4235.296788] <...>-363 2*.~2. 4234519567us : evl_switch_tail: { current=VM1Thread-tid_363[363] }
[ 4235.296795] <...>-363 2d.~2. 4234519568us : evl_finish_wait: wq=&mon->wait_queue
[ 4235.296802] RT engin-361 0*.~2. 4234519568us : evl_pick_thread: { next=ROOT/0[0] }
[ 4235.296808] <...>-363 2d.~2. 4234519569us : evl_oob_sysexit: result=0
[ 4235.296816] RT engin-361 0*.~2. 4234519569us : evl_switch_context: { RT engine-tid_361[361] prio=20, state=0x4880 } => { ROOT/0[0] prio=-1 }
[ 4235.296822] <idle>-0 0*.~2. 4234519570us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.296829] <idle>-0 0d.h1. 4234519572us : evl_inband_wakeup: pid=361 comm=RT engine-tid_3
[ 4235.296836] RT engin-361 0*..2. 4234519576us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.296842] RT engin-361 0..... 4234519577us : evl_switched_inband: state=0x4880 info=0x0
[ 4235.296849] <...>-363 2d.~2. 4234519586us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.296855] <...>-363 2*.~2. 4234519587us : evl_wait: wq=&mon->wait_queue
[ 4235.296863] <...>-363 2*.~2. 4234519588us : evl_sleep_on: pid=363 timeout=0 timeout_mode=0 clock=monotonic wchan=&mon->wait_queue(00000000ea5e79e5)
[ 4235.296869] <...>-363 2d.~2. 4234519589us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.296876] <...>-363 2*.~2. 4234519590us : evl_pick_thread: { next=ROOT/2[0] }
[ 4235.296883] <...>-363 2*.~2. 4234519591us : evl_switch_context: { VM1Thread-tid_363[363] prio=90, state=0x4002 } => { ROOT/2[0] prio=-1 }
[ 4235.296890] <idle>-0 2*.~1. 4234519592us : evl_switch_tail: { current=ROOT/2[0] }
[ 4235.296897] <idle>-0 0*.~1. 4234519839us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.296904] <idle>-0 0*.~1. 4234519839us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 371814 us, 20077979 cycles)
[ 4235.296911] <idle>-0 0*Nh1. 4234519844us : evl_timer_start: timer=[proxy-timer/0] value=4234509293944 interval=0
[ 4235.296918] <idle>-0 0*Nh1. 4234519845us : evl_timer_shot: [proxy-timer/0] at 4234.519948 (delay: 104 us, 5617 cycles)
[ 4235.296925] <idle>-0 0*.~1. 4234519950us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.296932] <idle>-0 0*.~1. 4234519951us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 371702 us, 20071938 cycles)
[ 4235.296939] <idle>-0 0*Nh1. 4234519956us : evl_timer_start: timer=[proxy-timer/0] value=4234510190925 interval=0
[ 4235.296946] <idle>-0 0*Nh1. 4234519957us : evl_timer_shot: [proxy-timer/0] at 4234.520845 (delay: 889 us, 48015 cycles)
[ 4235.296952] <idle>-0 0*.~1. 4234520847us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.296960] <idle>-0 0*.~1. 4234520848us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 370806 us, 20023535 cycles)
[ 4235.296967] <idle>-0 0*Nh1. 4234520852us : evl_timer_start: timer=[proxy-timer/0] value=4234511311996 interval=0
[ 4235.296974] <idle>-0 0*Nh1. 4234520853us : evl_timer_shot: [proxy-timer/0] at 4234.521966 (delay: 1114 us, 60170 cycles)
[ 4235.296980] <idle>-0 0*.~1. 4234521968us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.296987] <idle>-0 0*.~1. 4234521969us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 369685 us, 19963000 cycles)
[ 4235.296994] <idle>-0 0*Nh1. 4234521972us : evl_timer_start: timer=[proxy-timer/0] value=4234513205013 interval=0
[ 4235.297001] <idle>-0 0*Nh1. 4234521973us : evl_timer_shot: [proxy-timer/0] at 4234.523860 (delay: 1886 us, 101864 cycles)
[ 4235.297008] <idle>-0 0*N.2. 4234521976us : evl_timer_start: timer=[proxy-timer/0] value=4234512000129 interval=0
[ 4235.297015] <idle>-0 0*N.2. 4234521976us : evl_timer_shot: [proxy-timer/0] at 4234.522655 (delay: 678 us, 36649 cycles)
[ 4235.297022] <idle>-0 0*..2. 4234521990us : evl_timer_start: timer=[proxy-timer/0] value=4234513204994 interval=0
[ 4235.297029] <idle>-0 0*..2. 4234521992us : evl_timer_shot: [proxy-timer/0] at 4234.523860 (delay: 1868 us, 100884 cycles)
[ 4235.297036] <idle>-0 2*.~1. 4234522656us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.297042] <idle>-0 2*.h1. 4234522662us : evl_timer_start: timer=[proxy-timer/2] value=4234516000130 interval=0
[ 4235.297050] <idle>-0 2*.h1. 4234522663us : evl_timer_shot: [proxy-timer/2] at 4234.526655 (delay: 3992 us, 215574 cycles)
[ 4235.297056] <idle>-0 0*.~1. 4234523861us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.297063] <idle>-0 0*.~1. 4234523861us : evl_timer_shot: timerfd_handler at 4234.891653 (delay: 367792 us, 19860789 cycles)
[ 4235.297070] <idle>-0 0*Nh1. 4234523865us : evl_timer_start: timer=[proxy-timer/0] value=4234513329437 interval=0
[ 4235.297077] <idle>-0 0*Nh1. 4234523866us : evl_timer_shot: [proxy-timer/0] at 4234.523984 (delay: 118 us, 6384 cycles)
[ 4235.297083] <idle>-0 0*.~1. 4234523985us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.297090] <idle>-0 0*.~1. 4234523986us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 367667 us, 19854062 cycles)
[ 4235.297097] <idle>-0 0*Nh1. 4234523990us : evl_timer_start: timer=[proxy-timer/0] value=4234513436949 interval=0
[ 4235.297104] <idle>-0 0*Nh1. 4234523991us : evl_timer_shot: [proxy-timer/0] at 4234.524091 (delay: 100 us, 5441 cycles)
[ 4235.297111] <idle>-0 0*.~1. 4234524093us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.297118] <idle>-0 0*.~1. 4234524094us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 367560 us, 19848253 cycles)
[ 4235.297125] <idle>-0 0*Nh1. 4234524097us : evl_timer_start: timer=[proxy-timer/0] value=4234514241625 interval=0
[ 4235.297132] <idle>-0 0*Nh1. 4234524098us : evl_timer_shot: [proxy-timer/0] at 4234.524896 (delay: 798 us, 43097 cycles)
[ 4235.297139] <idle>-0 0*.~1. 4234524897us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.297146] <idle>-0 0*.~1. 4234524898us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 366755 us, 19804808 cycles)
[ 4235.297152] <idle>-0 0*Nh1. 4234524902us : evl_timer_start: timer=[proxy-timer/0] value=4234514296754 interval=0
[ 4235.297160] <idle>-0 0*Nh1. 4234524903us : evl_timer_shot: [proxy-timer/0] at 4234.524951 (delay: 48 us, 2609 cycles)
[ 4235.297166] <idle>-0 0*.~1. 4234524953us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.297173] <idle>-0 0*.~1. 4234524954us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 366700 us, 19801817 cycles)
[ 4235.297180] <idle>-0 0*Nh1. 4234524958us : evl_timer_start: timer=[proxy-timer/0] value=4234515344304 interval=0
[ 4235.297187] <idle>-0 0*Nh1. 4234524959us : evl_timer_shot: [proxy-timer/0] at 4234.525999 (delay: 1040 us, 56168 cycles)
[ 4235.297194] <idle>-0 0*.~1. 4234526000us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.297201] <idle>-0 0*.~1. 4234526001us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 365653 us, 19745263 cycles)
[ 4235.297208] <idle>-0 0*Nh1. 4234526005us : evl_timer_start: timer=[proxy-timer/0] value=4234516000148 interval=0
[ 4235.297215] <idle>-0 0*Nh1. 4234526006us : evl_timer_shot: [proxy-timer/0] at 4234.526655 (delay: 649 us, 35055 cycles)
[ 4235.297222] <idle>-0 2*.~1. 4234526656us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.297228] <idle>-0 0*.~1. 4234526656us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.297235] <idle>-0 0*.~1. 4234526657us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 364996 us, 19709836 cycles)
[ 4235.297242] <idle>-0 2*.h1. 4234526659us : evl_timer_start: timer=[proxy-timer/2] value=4234520000111 interval=0
[ 4235.297249] <idle>-0 2*.h1. 4234526660us : evl_timer_shot: [proxy-timer/2] at 4234.530655 (delay: 3994 us, 215709 cycles)
[ 4235.297256] <idle>-0 0*.h1. 4234526661us : evl_timer_start: timer=[proxy-timer/0] value=4234516217527 interval=0
[ 4235.297263] <idle>-0 0*.h1. 4234526662us : evl_timer_shot: [proxy-timer/0] at 4234.526872 (delay: 210 us, 11348 cycles)
[ 4235.297269] <idle>-0 0*.~1. 4234526873us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.297276] <idle>-0 0*.~1. 4234526874us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 364779 us, 19698101 cycles)
[ 4235.297283] <idle>-0 0*Nh1. 4234526878us : evl_timer_start: timer=[proxy-timer/0] value=4234517359190 interval=0
[ 4235.297290] <idle>-0 0*Nh1. 4234526879us : evl_timer_shot: [proxy-timer/0] at 4234.528014 (delay: 1134 us, 61288 cycles)
[ 4235.297297] <idle>-0 0*.~1. 4234528015us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.297304] <idle>-0 0*.~1. 4234528016us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 363638 us, 19636452 cycles)
[ 4235.297311] <idle>-0 0*Nh1. 4234528020us : evl_timer_start: timer=[proxy-timer/0] value=4234518923645 interval=0
[ 4235.297319] <idle>-0 0*Nh1. 4234528021us : evl_timer_shot: [proxy-timer/0] at 4234.529578 (delay: 1557 us, 84114 cycles)
[ 4235.297325] <idle>-0 0*.~1. 4234529579us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.297332] <idle>-0 0*.~1. 4234529580us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 362073 us, 19551968 cycles)
[ 4235.297339] <idle>-0 0*Nh1. 4234529584us : evl_timer_start: timer=[proxy-timer/0] value=4234519230910 interval=0
[ 4235.297346] <idle>-0 0*Nh1. 4234529585us : evl_timer_shot: [proxy-timer/0] at 4234.529885 (delay: 300 us, 16216 cycles)
[ 4235.297353] RT engin-361 0..... 4234529593us : evl_inband_sysentry: syscall=oob_ioctl
[ 4235.297360] RT engin-361 0..... 4234529594us : evl_switch_oob: state=0x4880 info=0x0
[ 4235.297367] <idle>-0 0*..2. 4234529598us : evl_release_thread: name=RT engine-tid_361 pid=361 mask=0x80 info=0x0
[ 4235.297374] <idle>-0 0*.~2. 4234529600us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.297381] <idle>-0 0*.~2. 4234529601us : evl_timer_start: timer=[rrb-timer/0] value=4234523945517 interval=0
[ 4235.297387] <idle>-0 0*.~2. 4234529602us : evl_pick_thread: { next=RT engine-tid_361[361] }
[ 4235.297395] <idle>-0 0*.~2. 4234529602us : evl_switch_context: { ROOT/0[0] prio=-1, state=0x1010 } => { RT engine-tid_361[361] prio=20 }
[ 4235.297402] RT engin-361 0d.~2. 4234529604us : evl_switched_oob: state=0x4800 info=0x0
[ 4235.297408] RT engin-361 0d.~2. 4234529605us : evl_mutex_lock: mutex=0000000041a7b4c3
[ 4235.297415] RT engin-361 0d.~2. 4234529606us : evl_inband_sysexit: result=0
[ 4235.297421] RT engin-361 0d.~2. 4234529611us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.297427] RT engin-361 0*.~2. 4234529612us : evl_wake_up: wq=&mon->wait_queue
[ 4235.297434] RT engin-361 0*.~2. 4234529613us : evl_wakeup_thread: name=VM1Thread-tid_363 pid=363 mask=0x2 info=0x0
[ 4235.297441] RT engin-361 0d.~2. 4234529614us : evl_schedule: flags=0x0, local_flags=0x10000000
[ 4235.297448] RT engin-361 0d.~2. 4234529615us : evl_oob_sysexit: result=0
[ 4235.297454] <idle>-0 2*.~1. 4234529616us : evl_reschedule_ipi: flags=0x10000000, local_flags=0x4000
[ 4235.297461] <idle>-0 2*.~1. 4234529617us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.297467] <idle>-0 2*.~1. 4234529617us : evl_pick_thread: { next=VM1Thread-tid_363[363] }
[ 4235.297474] RT engin-361 0d.~2. 4234529618us : evl_switch_inband: cause=in-band syscall
[ 4235.297481] <idle>-0 2*.~1. 4234529618us : evl_switch_context: { ROOT/2[0] prio=-1, state=0x1010 } => { VM1Thread-tid_363[363] prio=90 }
[ 4235.297488] RT engin-361 0*.~2. 4234529619us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.297495] <...>-363 2*.~2. 4234529620us : evl_switch_tail: { current=VM1Thread-tid_363[363] }
[ 4235.297501] RT engin-361 0*.~2. 4234529620us : evl_timer_stop: timer=[rrb-timer/0]
[ 4235.297508] <...>-363 2d.~2. 4234529620us : evl_finish_wait: wq=&mon->wait_queue
[ 4235.297514] RT engin-361 0*.~2. 4234529621us : evl_pick_thread: { next=ROOT/0[0] }
[ 4235.297521] <...>-363 2d.~2. 4234529621us : evl_oob_sysexit: result=0
[ 4235.297528] RT engin-361 0*.~2. 4234529622us : evl_switch_context: { RT engine-tid_361[361] prio=20, state=0x4880 } => { ROOT/0[0] prio=-1 }
[ 4235.297535] <idle>-0 0*.~2. 4234529623us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.297541] <idle>-0 0d.h1. 4234529625us : evl_inband_wakeup: pid=361 comm=RT engine-tid_3
[ 4235.297548] RT engin-361 0*..2. 4234529629us : evl_switch_tail: { current=ROOT/0[0] }
[ 4235.297554] RT engin-361 0..... 4234529630us : evl_switched_inband: state=0x4880 info=0x0
[ 4235.297561] <...>-363 2d.~2. 4234529634us : evl_oob_sysentry: syscall=oob_ioctl
[ 4235.297567] <...>-363 2*.~2. 4234529635us : evl_wait: wq=&mon->wait_queue
[ 4235.297575] <...>-363 2*.~2. 4234529636us : evl_sleep_on: pid=363 timeout=0 timeout_mode=0 clock=monotonic wchan=&mon->wait_queue(00000000ea5e79e5)
[ 4235.297582] <...>-363 2d.~2. 4234529637us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.297588] <...>-363 2*.~2. 4234529638us : evl_pick_thread: { next=ROOT/2[0] }
[ 4235.297596] <...>-363 2*.~2. 4234529639us : evl_switch_context: { VM1Thread-tid_363[363] prio=90, state=0x4002 } => { ROOT/2[0] prio=-1 }
[ 4235.297602] <idle>-0 2*.~1. 4234529640us : evl_switch_tail: { current=ROOT/2[0] }
[ 4235.297609] <idle>-0 0*.~1. 4234529887us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.297616] <idle>-0 0*.~1. 4234529888us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 361766 us, 19535374 cycles)
[ 4235.297623] <idle>-0 0*Nh1. 4234529892us : evl_timer_start: timer=[proxy-timer/0] value=4234519260247 interval=0
[ 4235.297630] <idle>-0 0*Nh1. 4234529893us : evl_timer_shot: [proxy-timer/0] at 4234.529915 (delay: 21 us, 1182 cycles)
[ 4235.297637] <idle>-0 0*.~1. 4234529916us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.297644] <idle>-0 0*.~1. 4234529918us : evl_timer_shot: timerfd_handler at 4234.891655 (delay: 361737 us, 19533799 cycles)
[ 4235.297651] <idle>-0 0*Nh1. 4234529922us : evl_timer_start: timer=[proxy-timer/0] value=4234519375113 interval=0
[ 4235.297659] <idle>-0 0*Nh1. 4234529923us : evl_timer_shot: [proxy-timer/0] at 4234.530030 (delay: 107 us, 5781 cycles)
[ 4235.297665] <idle>-0 0*.~1. 4234530031us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.297672] <idle>-0 0*.~1. 4234530032us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 361621 us, 19527582 cycles)
[ 4235.297679] <idle>-0 0*Nh1. 4234530036us : evl_timer_start: timer=[proxy-timer/0] value=4234522244905 interval=0
[ 4235.297686] <idle>-0 0*Nh1. 4234530037us : evl_timer_shot: [proxy-timer/0] at 4234.532899 (delay: 2862 us, 154594 cycles)
[ 4235.297694] <idle>-0 0*N.2. 4234530039us : evl_timer_start: timer=[proxy-timer/0] value=4234520000130 interval=0
[ 4235.297701] <idle>-0 0*N.2. 4234530040us : evl_timer_shot: [proxy-timer/0] at 4234.530655 (delay: 615 us, 33226 cycles)
[ 4235.297708] <idle>-0 0*..2. 4234530052us : evl_timer_start: timer=[proxy-timer/0] value=4234521392627 interval=0
[ 4235.297715] <idle>-0 0*..2. 4234530054us : evl_timer_shot: [proxy-timer/0] at 4234.532047 (delay: 1993 us, 107671 cycles)
[ 4235.297721] <idle>-0 2*.~1. 4234530656us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.297728] <idle>-0 2*.h1. 4234530661us : evl_timer_start: timer=[proxy-timer/2] value=4234524000111 interval=0
[ 4235.297735] <idle>-0 2*.h1. 4234530662us : evl_timer_shot: [proxy-timer/2] at 4234.534655 (delay: 3993 us, 215628 cycles)
[ 4235.297742] <idle>-0 2*..2. 4234530862us : evl_timer_start: timer=[proxy-timer/2] value=4234714073736 interval=0
[ 4235.297749] <idle>-0 2*..2. 4234530864us : evl_timer_shot: [proxy-timer/2] at 4234.724728 (delay: 193865 us, 10468713 cycles)
[ 4235.297756] <idle>-0 2*N.2. 4234530891us : evl_timer_start: timer=[proxy-timer/2] value=4234524000112 interval=0
[ 4235.297763] <idle>-0 2*N.2. 4234530892us : evl_timer_shot: [proxy-timer/2] at 4234.534655 (delay: 3762 us, 203183 cycles)
[ 4235.297770] EtherCAT-359 2..... 4234531575us : evl_inband_sysentry: syscall=oob_ioctl
[ 4235.297776] EtherCAT-359 2..... 4234531578us : evl_switch_oob: state=0x4080 info=0x0
[ 4235.297783] <idle>-0 2*..2. 4234531585us : evl_release_thread: name=EtherCATDrv-tid_359 pid=359 mask=0x80 info=0x0
[ 4235.297790] <idle>-0 2*.~2. 4234531587us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.297797] <idle>-0 2*.~2. 4234531589us : evl_pick_thread: { next=EtherCATDrv-tid_359[359] }
[ 4235.297804] <idle>-0 2*.~2. 4234531590us : evl_switch_context: { ROOT/2[0] prio=-1, state=0x1010 } => { EtherCATDrv-tid_359[359] prio=70 }
[ 4235.297811] EtherCAT-359 2d.~2. 4234531592us : evl_switched_oob: state=0x4000 info=0x0
[ 4235.297817] EtherCAT-359 2d.~2. 4234531596us : evl_mutex_lock: mutex=00000000da88c8bd
[ 4235.297824] EtherCAT-359 2d.~2. 4234531598us : evl_inband_sysexit: result=0
[ 4235.297830] EtherCAT-359 2d.~2. 4234531614us : evl_switch_inband: cause=in-band syscall
[ 4235.297837] EtherCAT-359 2*.~2. 4234531619us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.297843] EtherCAT-359 2*.~2. 4234531621us : evl_pick_thread: { next=ROOT/2[0] }
[ 4235.297851] EtherCAT-359 2*.~2. 4234531622us : evl_switch_context: { EtherCATDrv-tid_359[359] prio=70, state=0x4080 } => { ROOT/2[0] prio=-1 }
[ 4235.297858] <idle>-0 2*.~2. 4234531623us : evl_switch_tail: { current=ROOT/2[0] }
[ 4235.297864] <idle>-0 2d.h1. 4234531626us : evl_inband_wakeup: pid=359 comm=EtherCATDrv-tid
[ 4235.297871] EtherCAT-359 2*..2. 4234531631us : evl_switch_tail: { current=ROOT/2[0] }
[ 4235.297877] EtherCAT-359 2..... 4234531633us : evl_switched_inband: state=0x4080 info=0x0
[ 4235.297884] <idle>-0 0*n~1. 4234532051us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.297891] <idle>-0 0*n~1. 4234532055us : evl_timer_shot: timerfd_handler at 4234.891655 (delay: 359600 us, 19418428 cycles)
[ 4235.297898] <idle>-0 0*Nh1. 4234532065us : evl_timer_start: timer=[proxy-timer/0] value=4234522244997 interval=0
[ 4235.297905] <idle>-0 0*Nh1. 4234532068us : evl_timer_shot: [proxy-timer/0] at 4234.532900 (delay: 832 us, 44958 cycles)
[ 4235.297912] EtherCAT-359 2..... 4234532689us : evl_inband_sysentry: syscall=oob_ioctl
[ 4235.297918] EtherCAT-359 2..... 4234532691us : evl_switch_oob: state=0x4080 info=0x0
[ 4235.297925] <idle>-0 2*..2. 4234532698us : evl_release_thread: name=EtherCATDrv-tid_359 pid=359 mask=0x80 info=0x0
[ 4235.297932] <idle>-0 2*.~2. 4234532701us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.297938] <idle>-0 2*.~2. 4234532705us : evl_pick_thread: { next=EtherCATDrv-tid_359[359] }
[ 4235.297946] <idle>-0 2*.~2. 4234532706us : evl_switch_context: { ROOT/2[0] prio=-1, state=0x1010 } => { EtherCATDrv-tid_359[359] prio=70 }
[ 4235.297953] EtherCAT-359 2d.~2. 4234532709us : evl_switched_oob: state=0x4000 info=0x0
[ 4235.297959] EtherCAT-359 2d.~2. 4234532712us : evl_mutex_lock: mutex=00000000f8119351
[ 4235.297966] EtherCAT-359 2d.~2. 4234532714us : evl_inband_sysexit: result=0
[ 4235.297972] EtherCAT-359 2d.~2. 4234532753us : evl_switch_inband: cause=in-band syscall
[ 4235.297978] EtherCAT-359 2*.~2. 4234532755us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.297985] EtherCAT-359 2*.~2. 4234532757us : evl_pick_thread: { next=ROOT/2[0] }
[ 4235.297992] EtherCAT-359 2*.~2. 4234532757us : evl_switch_context: { EtherCATDrv-tid_359[359] prio=70, state=0x4080 } => { ROOT/2[0] prio=-1 }
[ 4235.297999] <idle>-0 2*.~2. 4234532759us : evl_switch_tail: { current=ROOT/2[0] }
[ 4235.298005] <idle>-0 2d.h1. 4234532762us : evl_inband_wakeup: pid=359 comm=EtherCATDrv-tid
[ 4235.298012] EtherCAT-359 2*..2. 4234532767us : evl_switch_tail: { current=ROOT/2[0] }
[ 4235.298019] EtherCAT-359 2..... 4234532769us : evl_switched_inband: state=0x4080 info=0x0
[ 4235.298025] systemd--11716 0D.~.. 4234532903us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.298032] systemd--11716 0D.~.. 4234532907us : evl_timer_shot: timerfd_handler at 4234.891655 (delay: 358747 us, 19372388 cycles)
[ 4235.298039] systemd--11716 0*Nh.. 4234532916us : evl_timer_start: timer=[proxy-timer/0] value=4234523437457 interval=0
[ 4235.298046] systemd--11716 0*Nh.. 4234532919us : evl_timer_shot: [proxy-timer/0] at 4234.534092 (delay: 1173 us, 63388 cycles)
[ 4235.298053] systemd--11716 0D.~.. 4234534100us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.298060] systemd--11716 0D.~.. 4234534106us : evl_timer_shot: timerfd_handler at 4234.891655 (delay: 357549 us, 19307669 cycles)
[ 4235.298066] systemd--11716 0*Nh.. 4234534120us : evl_timer_start: timer=[proxy-timer/0] value=4234524000148 interval=0
[ 4235.298073] systemd--11716 0*Nh.. 4234534122us : evl_timer_shot: [proxy-timer/0] at 4234.534655 (delay: 533 us, 28790 cycles)
[ 4235.298080] systemd--11716 0D.~.. 4234534659us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.298086] EtherCAT-359 2D.~.. 4234534659us : evl_timer_expire: timer=[proxy-timer/2]
[ 4235.298093] systemd--11716 0D.~.. 4234534663us : evl_timer_shot: timerfd_handler at 4234.891655 (delay: 356992 us, 19277569 cycles)
[ 4235.298100] EtherCAT-359 2*.h.. 4234534678us : evl_timer_start: timer=[proxy-timer/2] value=4234528000148 interval=0
[ 4235.298107] systemd--11716 0*.h.. 4234534678us : evl_timer_start: timer=[proxy-timer/0] value=4234524281776 interval=0
[ 4235.298114] systemd--11716 0*.h.. 4234534680us : evl_timer_shot: [proxy-timer/0] at 4234.534936 (delay: 256 us, 13876 cycles)
[ 4235.298121] EtherCAT-359 2*.h.. 4234534680us : evl_timer_shot: [proxy-timer/2] at 4234.538655 (delay: 3975 us, 214667 cycles)
[ 4235.298128] <idle>-0 3*N.2. 4234534811us : evl_timer_start: timer=[proxy-timer/3] value=4234528000129 interval=0
[ 4235.298136] <idle>-0 3*N.2. 4234534812us : evl_timer_shot: [proxy-timer/3] at 4234.538655 (delay: 3842 us, 207517 cycles)
[ 4235.298143] EtherCAT-359 2D.... 4234534837us : evl_timer_move: timer=timerfd_handler clock=monotonic cpu=2
[ 4235.298150] EtherCAT-359 2..... 4234534839us : evl_timer_start: timer=timerfd_handler value=4234525000001 interval=500000
[ 4235.298157] EtherCAT-359 2D.... 4234534840us : evl_timer_shot: timerfd_handler at 4234.535654 (delay: 813 us, 43938 cycles)
[ 4235.298167] EcatAcyc-11722 3D.... 4234534866us : evl_thread_set_current_prio: thread=0000000029e63da1 pid=0 prio=0
[ 4235.298175] EcatAcyc-11722 3..... 4234534868us : evl_init_thread: thread=0000000029e63da1 name=EcatAcyc-tid_11722 flags=0x4000 class=weak prio=0 status=0x0
[ 4235.298184] EcatAcyc-11722 3D.... 4234534876us : evl_thread_migrate: thread=0000000029e63da1 pid=0 cpu=3
[ 4235.298191] EcatAcyc-11722 3..... 4234534879us : evl_thread_map: thread=0000000029e63da1 pid=0 prio=0
[ 4235.298198] EcatAcyc-11722 3D.... 4234534881us : evl_release_thread: name=EcatAcyc-tid_11722 pid=0 mask=0x20 info=0x0
[ 4235.298205] <...>-31 0D.~.. 4234534938us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.298211] <idle>-0 3*..2. 4234534938us : evl_timer_stop: timer=[proxy-timer/3]
[ 4235.298219] <...>-31 0D.~.. 4234534941us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 356713 us, 19262536 cycles)
[ 4235.298226] <...>-31 0*Nh.. 4234534946us : evl_timer_start: timer=[proxy-timer/0] value=4234525281678 interval=0
[ 4235.298233] <...>-31 0*Nh.. 4234534948us : evl_timer_shot: [proxy-timer/0] at 4234.535936 (delay: 988 us, 53395 cycles)
[ 4235.298240] <idle>-0 3*N.2. 4234535001us : evl_timer_start: timer=[proxy-timer/3] value=4234528000186 interval=0
[ 4235.298248] <idle>-0 3*N.2. 4234535002us : evl_timer_shot: [proxy-timer/3] at 4234.538655 (delay: 3653 us, 197285 cycles)
[ 4235.298254] EcatCycl-11723 3D.... 4234535060us : evl_thread_set_current_prio: thread=00000000762154bd pid=0 prio=0
[ 4235.298262] EcatCycl-11723 3..... 4234535063us : evl_init_thread: thread=00000000762154bd name=EcatCyclic-tid_11723 flags=0x4000 class=weak prio=0 status=0x0
[ 4235.298269] EcatCycl-11723 3D.... 4234535067us : evl_thread_migrate: thread=00000000762154bd pid=0 cpu=3
[ 4235.298276] EcatCycl-11723 3..... 4234535068us : evl_thread_map: thread=00000000762154bd pid=0 prio=0
[ 4235.298282] EcatCycl-11723 3D.... 4234535069us : evl_release_thread: name=EcatCyclic-tid_11723 pid=0 mask=0x20 info=0x0
[ 4235.298289] EcatCycl-11723 3..... 4234535163us : evl_inband_sysentry: syscall=oob_ioctl
[ 4235.298295] EcatCycl-11723 3..... 4234535165us : evl_switch_oob: state=0x6080 info=0x0
[ 4235.298302] EcatAcyc-11722 3*..2. 4234535170us : evl_release_thread: name=EcatCyclic-tid_11723 pid=11723 mask=0x80 info=0x0
[ 4235.298309] EcatAcyc-11722 3*.~2. 4234535173us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.298316] EcatAcyc-11722 3*.~2. 4234535177us : evl_pick_thread: { next=EcatCyclic-tid_11723[11723] }
[ 4235.298323] EcatAcyc-11722 3*.~2. 4234535178us : evl_switch_context: { ROOT/3[0] prio=-1, state=0x1010 } => { EcatCyclic-tid_11723[11723] prio=0 }
[ 4235.298330] EcatCycl-11723 3d.~2. 4234535182us : evl_switched_oob: state=0x6000 info=0x0
[ 4235.298339] EcatCycl-11723 3d.~2. 4234535186us : evl_thread_setsched: thread=EcatCyclic-tid_11723 policy=fifo param={ priority=90 }
[ 4235.298346] EcatCycl-11723 3*.~2. 4234535188us : evl_thread_set_current_prio: thread=00000000762154bd pid=11723 prio=90
[ 4235.298353] EcatCycl-11723 3d.~2. 4234535189us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.298359] EcatCycl-11723 3*.~2. 4234535190us : evl_pick_thread: { next=EcatCyclic-tid_11723[11723] }
[ 4235.298366] EcatCycl-11723 3d.~2. 4234535192us : evl_inband_sysexit: result=0
[ 4235.298372] EcatCycl-11723 3d.~2. 4234535208us : evl_switch_inband: cause=in-band syscall
[ 4235.298379] EcatCycl-11723 3*.~2. 4234535210us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.298385] EcatCycl-11723 3*.~2. 4234535211us : evl_pick_thread: { next=ROOT/3[0] }
[ 4235.298392] EcatCycl-11723 3*.~2. 4234535212us : evl_switch_context: { EcatCyclic-tid_11723[11723] prio=90, state=0x4080 } => { ROOT/3[0] prio=-1 }
[ 4235.298399] EcatAcyc-11722 3*.~2. 4234535213us : evl_switch_tail: { current=ROOT/3[0] }
[ 4235.298406] EcatAcyc-11722 3d.h1. 4234535217us : evl_inband_wakeup: pid=11723 comm=EcatCyclic-tid_
[ 4235.298413] EcatCycl-11723 3*..2. 4234535222us : evl_switch_tail: { current=ROOT/3[0] }
[ 4235.298419] EcatCycl-11723 3..... 4234535224us : evl_switched_inband: state=0x4080 info=0x100
[ 4235.298425] <idle>-0 2*.~1. 4234535658us : evl_timer_expire: timer=timerfd_handler
[ 4235.298432] <idle>-0 2*.~1. 4234535663us : evl_flush_wait: wq=&timerfd->readers
[ 4235.298439] <idle>-0 2*.~1. 4234535666us : evl_timer_shot: timerfd_handler at 4234.536155 (delay: 489 us, 26423 cycles)
[ 4235.298445] systemd--11715 0D.~.. 4234535939us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.298453] systemd--11715 0D.~.. 4234535942us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 355712 us, 19208488 cycles)
[ 4235.298460] systemd--11715 0*Nh.. 4234535954us : evl_timer_start: timer=[proxy-timer/0] value=4234525510473 interval=0
[ 4235.298467] systemd--11715 0*Nh.. 4234535956us : evl_timer_shot: [proxy-timer/0] at 4234.536165 (delay: 209 us, 11294 cycles)
[ 4235.298474] <idle>-0 2*.~1. 4234536155us : evl_timer_expire: timer=timerfd_handler
[ 4235.298480] <idle>-0 2*.~1. 4234536155us : evl_flush_wait: wq=&timerfd->readers
[ 4235.298486] <idle>-0 2*.~1. 4234536156us : evl_timer_shot: timerfd_handler at 4234.536654 (delay: 497 us, 26875 cycles)
[ 4235.298493] <...>-141 0D.~.. 4234536168us : evl_timer_expire: timer=[proxy-timer/0]
[ 4235.298500] <...>-141 0D.~.. 4234536170us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 355484 us, 19196142 cycles)
[ 4235.298507] <...>-141 0*Nh.. 4234536176us : evl_timer_start: timer=[proxy-timer/0] value=4234528000148 interval=0
[ 4235.298514] <...>-141 0*Nh.. 4234536178us : evl_timer_shot: [proxy-timer/0] at 4234.538655 (delay: 2477 us, 133773 cycles)
[ 4235.298521] T5Server-367 0*..1. 4234536206us : evl_timer_start: timer=[proxy-timer/0] value=4234527549803 interval=0
[ 4235.298529] T5Server-367 0*..1. 4234536207us : evl_timer_shot: [proxy-timer/0] at 4234.538204 (delay: 1997 us, 107855 cycles)
[ 4235.298535] EcatCycl-11723 3..... 4234536630us : evl_inband_sysentry: syscall=oob_read
[ 4235.298542] EcatCycl-11723 3..... 4234536633us : evl_switch_oob: state=0x4080 info=0x0
[ 4235.298549] EcatAcyc-11722 3*..2. 4234536641us : evl_release_thread: name=EcatCyclic-tid_11723 pid=11723 mask=0x80 info=0x0
[ 4235.298555] EcatAcyc-11722 3*.~2. 4234536643us : evl_schedule: flags=0x10000000, local_flags=0x0
[ 4235.298562] EcatAcyc-11722 3*.~2. 4234536646us : evl_pick_thread: { next=EcatCyclic-tid_11723[11723] }
[ 4235.298569] EcatAcyc-11722 3*.~2. 4234536648us : evl_switch_context: { ROOT/3[0] prio=-1, state=0x1010 } => { EcatCyclic-tid_11723[11723] prio=90 }
[ 4235.298576] EcatCycl-11723 3d.~2. 4234536650us : evl_switched_oob: state=0x4000 info=0x0
[ 4235.298582] <idle>-0 2*.~1. 4234536654us : evl_timer_expire: timer=timerfd_handler
[ 4235.298588] <idle>-0 2*.~1. 4234536655us : evl_flush_wait: wq=&timerfd->readers
[ 4235.298590] ---------------------------------
[ 4235.298593] Modules linked in: cdc_mbim cdc_wdm cdc_ncm cdc_ether bcm2835_v4l2(C) bcm2835_mmal_vchiq(C) videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common raspberrypi_hwmon videodev mc vc4 cec drm_kms_helper i2c_gpio snd_soc_core i2c_algo_bit snd_pcm_dmaengine snd_pcm snd_timer snd syscopyarea sysfillrect sysimgblt fb_sys_fops ax88179_178a uio_pdrv_genirq uio nvmem_rmem cfg80211 rfkill sch_fq_codel fuse drm drm_panel_orientation_quirks backlight configfs ipv6
[ 4235.298650] CPU: 3 PID: 11723 Comm: EcatCyclic-tid_ Tainted: G C 5.15.35-xeno-v5.15.77-evl3-rebase #1
[ 4235.298656] Hardware name: Raspberry Pi Compute Module 4 Rev 1.0 (DT)
[ 4235.298658] IRQ stage: EVL
[ 4235.298660] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 4235.298665] pc : evl_get_timer_overruns+0xec/0x1b0
[ 4235.298678] lr : evl_get_timer_overruns+0x54/0x1b0
[ 4235.298683] sp : ffffffc009bcbc30
[ 4235.298684] x29: ffffffc009bcbc30 x28: ffffff81318008c0 x27: 0000000000000000
[ 4235.298690] x26: ffffffc008e03ae8 x25: ffffffc009070220 x24: ffffff8131800888
[ 4235.298696] x23: 0000000000000000 x22: ffffff81fefd6460 x21: 0000000000000001
[ 4235.298701] x20: 000003d9ed6b0122 x19: ffffff8131800800 x18: 0000000000000000
[ 4235.298706] x17: 0000000000000000 x16: 0000000000000001 x15: 00000143e8483fde
[ 4235.298711] x14: 00050fa120ff7800 x13: ffffffc008aaa908 x12: 00000000000001ea
[ 4235.298717] x11: 00000000fa83b2da x10: 0000000000000950 x9 : ffffff812c04d580
[ 4235.298722] x8 : 0000000000000001 x7 : 0000000000000adc x6 : 0000003552b84029
[ 4235.298727] x5 : 00ffffffffffffff x4 : 00283ce766d17a8f x3 : 000000000007a120
[ 4235.298732] x2 : dead000000000100 x1 : 00000000000f3fe1 x0 : dead000000000122
[ 4235.298738] Call trace:
[ 4235.298741] evl_get_timer_overruns+0xec/0x1b0
[ 4235.298746] timerfd_oob_read+0x1dc/0x254
[ 4235.298753] EVL_read+0x50/0xd0
[ 4235.298757] handle_pipelined_syscall+0x314/0x350
[ 4235.298762] __pipeline_syscall+0xe8/0x2a0
[ 4235.298766] pipeline_syscall+0x64/0x110
[ 4235.298770] el0_svc_common.constprop.0+0x48/0x12c
[ 4235.298777] do_el0_svc+0x28/0x90
[ 4235.298781] el0_svc+0x4c/0xd0
[ 4235.298787] el0t_64_sync_handler+0xa4/0x130
[ 4235.298791] el0t_64_sync+0x1a0/0x1a4
[ 4235.298798] Code: 8b150042 f9002662 362ffc80 a9410262 (f9000440)
^[[7mReally kill this window [y/n]^[[27m^[[K ^[[?1l^[>^[[55;1H
^[[?1049l^[[23;0;0t[screen is terminating]
^[[?2004h^[]0;sri@sri-linux: ~\a^[[01;32msri@sri-linux^[[00m:^[[01;34m~^[[00m$ less cm4-\a
cm4-ip cm4-uart.log
^[]0;sri@sri-linux: ~\a^[[01;32msri@sri-linux^[[00m:^[[01;34m~^[[00m$ less cm4-uart.log
^[[?2004l"cm4-uart.log" may be a binary file. See it anyway?
^[[?1049h^[[22;0;0t^[[?1h^[=Script started on 2022-12-08 12:26:35-08:00 [TERM="xterm-256color" TTY="/dev/pts/3" COLUMNS="203" LINES="55"]
^[[7mESC^[[27m[?2004h^[[7mESC^[[27m]0;sri@sri-linux: ~^[[7m^GESC^[[27m[01;32msri@sri-linux^[[7mESC^[[27m[00m:^[[7mESC^[[27m[01;34m~^[[7mESC^[[27m[00m$ cm4-uart
^[[7mESC^[[27m[?2004l^[[7m^M^[[27m[sudo] password for sri:
^[[7mESC^[[27m[!p^[[7mESC^[[27m[?3;4l^[[7mESC^[[27m[4l^[[7mESC^[[27m>^[[7mESC^[[27m[?1049h^[[7mESC^[[27m[22;0;0t^[[7mESC^[[27m[4l^[[7mESC^[[27m[?1h^[[7mESC^[[27m=^[[7mESC^[[27m[0m^[[7mESC^[[27m(B^[[7mESC^[[27m[1;55r^[[7mESC^[[27m[H^[[7mESC^[[27m[2J^[[7mESC^[[27m[H^[[7mESC^[[27m[2J
kas-rpi4 login: kas
Password:
Last login: Thu Dec 8 12:14:07 PST 2022 from 10.8.235.68 on pts/0
^[[7mESC^[[27m[H^[[7mESC^[[27m[55;203H^[[7mESC^[[27m[5;1HOS Version: 1.0.0.00159
kas-rpi4:~$ [ 1450.758188] audit: type=1334 audit(1670531261.301:308): prog-id=162 op=LOAD
[ 1450.767983] audit: type=1334 audit(1670531261.309:309): prog-id=163 op=LOAD
[ 1450.787833] audit: type=1334 audit(1670531261.309:310): prog-id=0 op=UNLOAD
[ 1450.810821] audit: type=1334 audit(1670531261.309:311): prog-id=0 op=UNLOAD
[ 1450.828759] audit: type=1334 audit(1670531261.325:312): prog-id=164 op=LOAD
[ 1450.855127] audit: type=1334 audit(1670531261.325:313): prog-id=165 op=LOAD
[ 1450.877848] audit: type=1334 audit(1670531261.325:314): prog-id=0 op=UNLOAD
[ 1450.893381] audit: type=1334 audit(1670531261.325:315): prog-id=0 op=UNLOAD
[ 1450.907996] audit: type=1334 audit(1670531261.345:316): prog-id=166 op=LOAD
[ 1450.924586] audit: type=1334 audit(1670531261.345:317): prog-id=167 op=LOAD
[ 1456.345933] systemd-journald[122]: Time jumped backwards, rotating.
[ 1459.454508] kauditd_printk_skb: 14 callbacks suppressed
[ 1459.454518] audit: type=1334 audit(1670529603.110:332): prog-id=174 op=LOAD
[ 1459.476090] audit: type=1334 audit(1670529603.126:333): prog-id=175 op=LOAD
[ 1459.499896] audit: type=1334 audit(1670529603.126:334): prog-id=0 op=UNLOAD
[ 1459.520558] audit: type=1334 audit(1670529603.126:335): prog-id=0 op=UNLOAD
[ 1459.538737] audit: type=1334 audit(1670529603.142:336): prog-id=176 op=LOAD
[ 1459.570435] audit: type=1334 audit(1670529603.142:337): prog-id=177 op=LOAD
[ 1459.585832] audit: type=1334 audit(1670529603.142:338): prog-id=0 op=UNLOAD
[ 1459.601311] audit: type=1334 audit(1670529603.142:339): prog-id=0 op=UNLOAD
[ 1459.619002] audit: type=1334 audit(1670529603.150:340): prog-id=178 op=LOAD
[ 1459.681376] audit: type=1334 audit(1670529603.150:341): prog-id=179 op=LOAD
[ 1464.587847] kauditd_printk_skb: 14 callbacks suppressed
[ 1464.587857] audit: type=1334 audit(1670531275.128:356): prog-id=186 op=LOAD
[ 1464.609708] audit: type=1334 audit(1670531275.140:357): prog-id=187 op=LOAD
[ 1464.633987] audit: type=1334 audit(1670531275.140:358): prog-id=0 op=UNLOAD
[ 1464.654442] audit: type=1334 audit(1670531275.140:359): prog-id=0 op=UNLOAD
[ 1464.669897] audit: type=1334 audit(1670531275.156:360): prog-id=188 op=LOAD
[ 1464.701537] audit: type=1334 audit(1670531275.156:361): prog-id=189 op=LOAD
[ 1464.717037] audit: type=1334 audit(1670531275.156:362): prog-id=0 op=UNLOAD
[ 1464.732589] audit: type=1334 audit(1670531275.156:363): prog-id=0 op=UNLOAD
[ 1464.747615] audit: type=1334 audit(1670531275.164:364): prog-id=190 op=LOAD
[ 1464.773327] audit: type=1334 audit(1670531275.168:365): prog-id=191 op=LOAD
[ 1468.390379] systemd-journald[122]: Time jumped backwards, rotating.
[ 1475.786272] kauditd_printk_skb: 14 callbacks suppressed
[ 1475.786283] audit: type=1334 audit(1670529607.398:380): prog-id=198 op=LOAD
[ 1475.809049] audit: type=1334 audit(1670529607.414:381): prog-id=199 op=LOAD
[ 1475.832456] audit: type=1334 audit(1670529607.414:382): prog-id=0 op=UNLOAD
[ 1475.854170] audit: type=1334 audit(1670529607.414:383): prog-id=0 op=UNLOAD
[ 1475.869767] audit: type=1334 audit(1670529607.418:384): prog-id=200 op=LOAD
[ 1475.901408] audit: type=1334 audit(1670529607.418:385): prog-id=201 op=LOAD
[ 1475.916919] audit: type=1334 audit(1670529607.418:386): prog-id=0 op=UNLOAD
[ 1475.932335] audit: type=1334 audit(1670529607.418:387): prog-id=0 op=UNLOAD
[ 1475.947959] audit: type=1334 audit(1670529607.438:388): prog-id=202 op=LOAD
[ 1476.011439] audit: type=1334 audit(1670529607.438:389): prog-id=203 op=LOAD
[ 1489.064040] kauditd_printk_skb: 14 callbacks suppressed
^[[7mcm4-uart.log^[[27m^[[K^[[K...skipping...
[ 4235.298678] lr : evl_get_timer_overruns+0x54/0x1b0
[ 4235.298683] sp : ffffffc009bcbc30
[ 4235.298684] x29: ffffffc009bcbc30 x28: ffffff81318008c0 x27: 0000000000000000
[ 4235.298690] x26: ffffffc008e03ae8 x25: ffffffc009070220 x24: ffffff8131800888
[ 4235.298696] x23: 0000000000000000 x22: ffffff81fefd6460 x21: 0000000000000001
[ 4235.298701] x20: 000003d9ed6b0122 x19: ffffff8131800800 x18: 0000000000000000
[ 4235.298706] x17: 0000000000000000 x16: 0000000000000001 x15: 00000143e8483fde
[ 4235.298711] x14: 00050fa120ff7800 x13: ffffffc008aaa908 x12: 00000000000001ea
[ 4235.298717] x11: 00000000fa83b2da x10: 0000000000000950 x9 : ffffff812c04d580
[ 4235.298722] x8 : 0000000000000001 x7 : 0000000000000adc x6 : 0000003552b84029
[ 4235.298727] x5 : 00ffffffffffffff x4 : 00283ce766d17a8f x3 : 000000000007a120
[ 4235.298732] x2 : dead000000000100 x1 : 00000000000f3fe1 x0 : dead000000000122
[ 4235.298738] Call trace:
[ 4235.298741] evl_get_timer_overruns+0xec/0x1b0
[ 4235.298746] timerfd_oob_read+0x1dc/0x254
[ 4235.298753] EVL_read+0x50/0xd0
[ 4235.298757] handle_pipelined_syscall+0x314/0x350
[ 4235.298762] __pipeline_syscall+0xe8/0x2a0
[ 4235.298766] pipeline_syscall+0x64/0x110
[ 4235.298770] el0_svc_common.constprop.0+0x48/0x12c
[ 4235.298777] do_el0_svc+0x28/0x90
[ 4235.298781] el0_svc+0x4c/0xd0
[ 4235.298787] el0t_64_sync_handler+0xa4/0x130
[ 4235.298791] el0t_64_sync+0x1a0/0x1a4
[ 4235.298798] Code: 8b150042 f9002662 362ffc80 a9410262 (f9000440)
^[[7mESC^[[27m[7mReally kill this window [y/n]^[[7mESC^[[27m[27m^[[7mESC^[[27m[K^[[7m^M^[[27m \b ^[[7m^MESC^[[27m[?1l^[[7mESC^[[27m>^[[7mESC^[[27m[55;1H
^[[7mESC^[[27m[?1049l^[[7mESC^[[27m[23;0;0t[screen is terminating]
^[[7mESC^[[27m[?2004h^[[7mESC^[[27m]0;sri@sri-linux: ~^[[7m^GESC^[[27m[01;32msri@sri-linux^[[7mESC^[[27m[00m:^[[7mESC^[[27m[01;34m~^[[7mESC^[[27m[00m$ less cm4-^[[7m^G^[[27m
cm4-ip cm4-uart.log
^[[7mESC^[[27m]0;sri@sri-linux: ~^[[7m^GESC^[[27m[01;32msri@sri-linux^[[7mESC^[[27m[00m:^[[7mESC^[[27m[01;34m~^[[7mESC^[[27m[00m$ less cm4-uart.log
^[[7mESC^[[27m[?2004l^[[7m^M^[[27m"cm4-uart.log" may be a binary file. See it anyway?
^[[7mESC^[[27m[?1049h^[[7mESC^[[27m[22;0;0t^[[7mESC^[[27m[?1h^[[7mESC^[[27m=^[[7m^M^[[27mScript started on 2022-12-08 12:26:35-08:00 [TERM="xterm-256color" TTY="/dev/pts/3" COLUMNS="203" LINES="55"]
^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[?2004h^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m]0;sri@sri-linux: ~^[[7mESC^[[27m[7m^GESC^[[7mESC^[[27m[27m[01;32msri@sri-linux^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[00m:^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[01;34m~^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[00m$ cm4-uart
^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[?2004l^[[7mESC^[[27m[7m^M^[[7mESC^[[27m[27m[sudo] password for sri:
^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[!p^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[?3;4l^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[4l^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m>^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[?1049h^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[22;0;0t^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[4l^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[?1h^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m=^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[0m^[[7mESC^[[27m[ \b7mESC^[[7mESC^[[27m[27m(B^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[1;55r^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[H^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[2J^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[H^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[2J
kas-rpi4 login: kas
Password:
Last login: Thu Dec 8 12:14:07 PST 2022 from 10.8.235.68 on pts/0
^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[H^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[55;203H^[[7mESC^[[27m[7mESC^[[7mESC^[[27m[27m[5;1HOS Version: 1.0.0.00159
kas-rpi4:~$ [ 1450.758188] audit: type=1334 audit(1670531261.301:308): prog-id=162 op=LOAD
[ 1450.767983] audit: type=1334 audit(1670531261.309:309): prog-id=163 op=LOAD
[ 1450.787833] audit: type=1334 audit(1670531261.309:310): prog-id=0 op=UNLOAD
[ 1450.810821] audit: type=1334 audit(1670531261.309:311): prog-id=0 op=UNLOAD
[ 1450.828759] audit: type=1334 audit(1670531261.325:312): prog-id=164 op=LOAD
[ 1450.855127] audit: type=1334 audit(1670531261.325:313): prog-id=165 op=LOAD
[ 1450.877848] audit: type=1334 audit(1670531261.325:314): prog-id=0 op=UNLOAD
[ 1450.893381] audit: type=1334 audit(1670531261.325:315): prog-id=0 op=UNLOAD
[ 1450.907996] audit: type=1334 audit(1670531261.345:316): prog-id=166 op=LOAD
[ 1450.924586] audit: type=1334 audit(1670531261.345:317): prog-id=167 op=LOAD
[ 1456.345933] systemd-journald[122]: Time jumped backwards, rotating.
[ 1459.454508] kauditd_printk_skb: 14 callbacks suppressed
[ 1459.454518] audit: t
^[[7m(END)^[[27m^[[K^[[K^[[H^[M[ 4235.298665] pc : evl_get_timer_overruns+0xec/0x1b0
^[[H^[M[ 4235.298660] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
^[[H^[M[ 4235.298658] IRQ stage: EVL
^[[H^[M[ 4235.298656] Hardware name: Raspberry Pi Compute Module 4 Rev 1.0 (DT)
^[[H^[M[ 4235.298650] CPU: 3 PID: 11723 Comm: EcatCyclic-tid_ Tainted: G C 5.15.35-xeno-v5.15.77-evl3-rebase #1
^[[H^[Ml sch_fq_codel fuse drm drm_panel_orientation_quirks backlight configfs ipv6
^[[H^[M vc4 cec drm_kms_helper i2c_gpio snd_soc_core i2c_algo_bit snd_pcm_dmaengine snd_pcm snd_timer snd syscopyarea sysfillrect sysimgblt fb_sys_fops ax88179_178a uio_pdrv_genirq uio nvmem_rmem cfg80211 rfkil^[[H^[M[ 4235.298593] Modules linked in: cdc_mbim cdc_wdm cdc_ncm cdc_ether bcm2835_v4l2(C) bcm2835_mmal_vchiq(C) videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common raspberrypi_hwmon videodev mc^[[H^[M[ 4235.298590] ---------------------------------
^[[H^[M[ 4235.298588] <idle>-0 2*.~1. 4234536655us : evl_flush_wait: wq=&timerfd->readers
^[[H^[M[ 4235.298582] <idle>-0 2*.~1. 4234536654us : evl_timer_expire: timer=timerfd_handler
^[[H^[M[ 4235.298576] EcatCycl-11723 3d.~2. 4234536650us : evl_switched_oob: state=0x4000 info=0x0
^[[H^[M[ 4235.298569] EcatAcyc-11722 3*.~2. 4234536648us : evl_switch_context: { ROOT/3[0] prio=-1, state=0x1010 } => { EcatCyclic-tid_11723[11723] prio=90 }
^[[H^[M[ 4235.298562] EcatAcyc-11722 3*.~2. 4234536646us : evl_pick_thread: { next=EcatCyclic-tid_11723[11723] }
^[[H^[M[ 4235.298555] EcatAcyc-11722 3*.~2. 4234536643us : evl_schedule: flags=0x10000000, local_flags=0x0
^[[H^[M[ 4235.298549] EcatAcyc-11722 3*..2. 4234536641us : evl_release_thread: name=EcatCyclic-tid_11723 pid=11723 mask=0x80 info=0x0
^[[H^[M[ 4235.298542] EcatCycl-11723 3..... 4234536633us : evl_switch_oob: state=0x4080 info=0x0
^[[H^[M[ 4235.298535] EcatCycl-11723 3..... 4234536630us : evl_inband_sysentry: syscall=oob_read
^[[H^[M[ 4235.298529] T5Server-367 0*..1. 4234536207us : evl_timer_shot: [proxy-timer/0] at 4234.538204 (delay: 1997 us, 107855 cycles)
^[[H^[M[ 4235.298521] T5Server-367 0*..1. 4234536206us : evl_timer_start: timer=[proxy-timer/0] value=4234527549803 interval=0
^[[H^[M[ 4235.298514] <...>-141 0*Nh.. 4234536178us : evl_timer_shot: [proxy-timer/0] at 4234.538655 (delay: 2477 us, 133773 cycles)
^[[H^[M[ 4235.298507] <...>-141 0*Nh.. 4234536176us : evl_timer_start: timer=[proxy-timer/0] value=4234528000148 interval=0
^[[H^[M[ 4235.298500] <...>-141 0D.~.. 4234536170us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 355484 us, 19196142 cycles)
^[[H^[M[ 4235.298493] <...>-141 0D.~.. 4234536168us : evl_timer_expire: timer=[proxy-timer/0]
^[[H^[M[ 4235.298486] <idle>-0 2*.~1. 4234536156us : evl_timer_shot: timerfd_handler at 4234.536654 (delay: 497 us, 26875 cycles)
^[[H^[M[ 4235.298480] <idle>-0 2*.~1. 4234536155us : evl_flush_wait: wq=&timerfd->readers
^[[H^[M[ 4235.298474] <idle>-0 2*.~1. 4234536155us : evl_timer_expire: timer=timerfd_handler
^[[H^[M[ 4235.298467] systemd--11715 0*Nh.. 4234535956us : evl_timer_shot: [proxy-timer/0] at 4234.536165 (delay: 209 us, 11294 cycles)
^[[H^[M[ 4235.298460] systemd--11715 0*Nh.. 4234535954us : evl_timer_start: timer=[proxy-timer/0] value=4234525510473 interval=0
^[[H^[M[ 4235.298453] systemd--11715 0D.~.. 4234535942us : evl_timer_shot: timerfd_handler at 4234.891654 (delay: 355712 us, 19208488 cycles)
^[[H^[M[ 4235.298445] systemd--11715 0D.~.. 4234535939us : evl_timer_expire: timer=[proxy-timer/0]
^[[H^[M[ 4235.298439] <idle>-0 2*.~1. 4234535666us : evl_timer_shot: timerfd_handler at 4234.536155 (delay: 489 us, 26423 cycles)
^[[H^[M[ 4235.298432] <idle>-0 2*.~1. 4234535663us : evl_flush_wait: wq=&timerfd->readers
^[[H^[M[ 4235.298425] <idle>-0 2*.~1. 4234535658us : evl_timer_expire: timer=timerfd_handler
^[[H^[M[ 4235.298419] EcatCycl-11723 3..... 4234535224us : evl_switched_inband: state=0x4080 info=0x100
^[[H^[M[ 4235.298413] EcatCycl-11723 3*..2. 4234535222us : evl_switch_tail: { current=ROOT/3[0] }
^[[H^[M[ 4235.298406] EcatAcyc-11722 3d.h1. 4234535217us : evl_inband_wakeup: pid=11723 comm=EcatCyclic-tid_
^[[H^[M[ 4235.298399] EcatAcyc-11722 3*.~2. 4234535213us : evl_switch_tail: { current=ROOT/3[0] }
^[[H^[M[ 4235.298392] EcatCycl-11723 3*.~2. 4234535212us : evl_switch_context: { EcatCyclic-tid_11723[11723] prio=90, state=0x4080 } => { ROOT/3[0] prio=-1 }
^[[H^[M[ 4235.298385] EcatCycl-11723 3*.~2. 4234535211us : evl_pick_thread: { next=ROOT/3[0] }
^[[H^[M[ 4235.298379] EcatCycl-11723 3*.~2. 4234535210us : evl_schedule: flags=0x10000000, local_flags=0x0
^[[H^[M[ 4235.298372] EcatCycl-11723 3d.~2. 4234535208us : evl_switch_inband: cause=in-band syscall
^[[H^[M[ 4235.298366] EcatCycl-11723 3d.~2. 4234535192us : evl_inband_sysexit: result=0
^[[H^[M[ 4235.298359] EcatCycl-11723 3*.~2. 4234535190us : evl_pick_thread: { next=EcatCyclic-tid_11723[11723] }
^[[H^[M[ 4235.298353] EcatCycl-11723 3d.~2. 4234535189us : evl_schedule: flags=0x10000000, local_flags=0x0
^[[H^[M[ 4235.298346] EcatCycl-11723 3*.~2. 4234535188us : evl_thread_set_current_prio: thread=00000000762154bd pid=11723 prio=90
^[[H^[M[ 4235.298339] EcatCycl-11723 3d.~2. 4234535186us : evl_thread_setsched: thread=EcatCyclic-tid_11723 policy=fifo param={ priority=90 }
^[[H^[M[ 4235.298330] EcatCycl-11723 3d.~2. 4234535182us : evl_switched_oob: state=0x6000 info=0x0
^[[H^[M[ 4235.298323] EcatAcyc-11722 3*.~2. 4234535178us : evl_switch_context: { ROOT/3[0] prio=-1, state=0x1010 } => { EcatCyclic-tid_11723[11723] prio=0 }
^[[H^[M[ 4235.298316] EcatAcyc-11722 3*.~2. 4234535177us : evl_pick_thread: { next=EcatCyclic-tid_11723[11723] }
^[[H^[M[ 4235.298309] EcatAcyc-11722 3*.~2. 4234535173us : evl_schedule: flags=0x10000000, local_flags=0x0
^[[H^[M[ 4235.298302] EcatAcyc-11722 3*..2. 4234535170us : evl_release_thread: name=EcatCyclic-tid_11723 pid=11723 mask=0x80 info=0x0
^[[H^[M[ 4235.298295] EcatCycl-11723 3..... 4234535165us : evl_switch_oob: state=0x6080 info=0x0
^[[H^[M[ 4235.298289] EcatCycl-11723 3..... 4234535163us : evl_inband_sysentry: syscall=oob_ioctl
^[[55;1H^[[K:^[[K^[[K^[[?1l^[>^[[?1049l^[[23;0;0t^[[?2004h^[]0;sri@sri-linux: ~\a^[[01;32msri@sri-linux^[[00m:^[[01;34m~^[[00m$ mv cm4-uart.log timer-crashed-evel-\b^[[K\b^[[K\b^[[Kl-ev3en\b^[[K\b^[[K\b^[[Kent.log
^[[?2004l^[[?2004h^[]0;sri@sri-linux: ~\a^[[01;32msri@sri-linux^[[00m:^[[01;34m~^[[00m$ scrip \b^[[Kt \b^[[K\b^[[K\b^[[K\b^[[K\b^[[K\b^[[K\b^[[K\a\aexit
^[[?2004lexit
Script done on 2022-12-08 15:22:58-08:00 [COMMAND_EXIT_CODE="0"]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Xenomai crash in timer IRQ
2022-12-09 17:05 ` Sri Subramanian
@ 2022-12-19 21:51 ` Sri Subramanian
0 siblings, 0 replies; 13+ messages in thread
From: Sri Subramanian @ 2022-12-19 21:51 UTC (permalink / raw)
To: Philippe Gerum; +Cc: xenomai
Hi Philippe,
I've had no luck reproducing this condition with a simple test. Is
there any useful information in the trace I attached previously? Would
you like me to trace specific events or subsystems?
Thanks
Sri
On Fri, Dec 9, 2022 at 9:05 AM Sri Subramanian
<sridhar.subramanian@gmail.com> wrote:
>
> Hi Philippe,
>
> I was able to reproduce one crash with a trace on evl events! If there
> are other specific events you want me to include, please let me know.
> I can only get it to hang with no output (after an overnight run) if I
> include all events.
>
> To answer your questions, we don't have any custom drivers. I'll check
> the application for stale timers. I had no luck with reproducing the
> crash with a simple test but will continue to pursue it.
>
> Sri
>
> On Wed, Dec 7, 2022 at 11:58 PM Philippe Gerum <rpm@xenomai.org> wrote:
> >
> >
> > Sri Subramanian <sridhar.subramanian@gmail.com> writes:
> >
> > > Hi Philippe,
> > >
> > > With next-evl-rebase, I am not able to reproduce the crash. Instead
> > > the system reboots (from the watchdog) or hangs (if I disable the
> > > watchdog).
> > >
> > > I'll try going back to the original setup and try other experiments --
> > > maybe try to reproduce this with a test program or attach a debugger
> > > to get more data.
> > >
> >
> > Ack, at this point it should be possible to reproduce this bug with a
> > simple test case if you think the core is doing something wrong when the
> > kernel wallclock is set.
> >
> > If you have custom drivers, you may want to check the lifetime of any
> > standalone evl_timer structs it might define for correctness, to make
> > sure there is no stale timers being reprogrammed (unlikely, but worth a
> > check anyway).
> >
> > --
> > Philippe.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Xenomai crash in timer IRQ
2022-12-08 7:53 ` Philippe Gerum
2022-12-09 17:05 ` Sri Subramanian
@ 2023-05-31 20:40 ` Sri Subramanian
2023-06-03 10:38 ` Philippe Gerum
1 sibling, 1 reply; 13+ messages in thread
From: Sri Subramanian @ 2023-05-31 20:40 UTC (permalink / raw)
To: Philippe Gerum; +Cc: xenomai
[-- Attachment #1: Type: text/plain, Size: 1881 bytes --]
Hi Philippe,
Sorry for the long pause in this thread.
However, I am able to reproduce the kernel crash stack (off
timerfd_oob_read) with a simple test program which does the following:
In a loop:
create timer fd
create thread which
sets up periodic triggers off the monotonic timer
closes the timer after a few periods.
join w/thread
If the main thread and the timer thread are in the same cpu, I can run
it with hectic in the background for 2 hrs at least. However, if the
threads are on different cpus, the kernel crashes within minutes. I am
attaching the crash and the test program.
Environment:
HW: Raspberry Pi (CM4)
SW: Xenomai 5.15.109 tag, Libevl r44
Test steps:
# hectic -s 200 -q&
# ./timer-crash1 -a 0 -t 1 -q&
BTW: this thread's history implies changes to the realtime clock, but
really, this doesn't have anything to do with it.
Thanks,
Sri Subramanian
On Wed, Dec 7, 2022 at 11:58 PM Philippe Gerum <rpm@xenomai.org> wrote:
>
>
> Sri Subramanian <sridhar.subramanian@gmail.com> writes:
>
> > Hi Philippe,
> >
> > With next-evl-rebase, I am not able to reproduce the crash. Instead
> > the system reboots (from the watchdog) or hangs (if I disable the
> > watchdog).
> >
> > I'll try going back to the original setup and try other experiments --
> > maybe try to reproduce this with a test program or attach a debugger
> > to get more data.
> >
>
> Ack, at this point it should be possible to reproduce this bug with a
> simple test case if you think the core is doing something wrong when the
> kernel wallclock is set.
>
> If you have custom drivers, you may want to check the lifetime of any
> standalone evl_timer structs it might define for correctness, to make
> sure there is no stale timers being reprogrammed (unlikely, but worth a
> check anyway).
>
> --
> Philippe.
[-- Attachment #2: timer-crash1.cpp --]
[-- Type: text/plain, Size: 5535 bytes --]
// To build: g++ -o timer-crash1 timer-crash1.cpp -lpthread -levl
// To run: sudo ./timer-crash1 -p 20 -a 1 -t 2
// Options:
// -p <priority> [default=20, main thread @ <priority>-1, timer-thread @ <priority>]
// -a <affinity> [default=none, main thread affinity]
// -t <affinity> [default=none, timer thread affinity]
// -q [default=false, no printing of timer-thread creation counts]
#include <pthread.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <sched.h>
#include <unistd.h>
#include <stdint.h>
#include <time.h>
#include <errno.h>
#include <evl/evl.h>
#include <evl/thread.h>
#include <evl/timer.h>
int8_t Affinity = -1; // Main thread affinity
int8_t Thread_Affinity = -1; // Timer thread affinity
int Thread_Priority = 20; // Timer thread priority, default = 20
int Timer_Fd = -1;
bool Quiet = false;
static void HandleError(const char *msg, int error = errno)
{
if (error != 0) {
errno = error;
perror(msg);
exit(error);
}
}
static void HandleEvlError(const char *msg, int error)
{
if (error < 0) {
errno = -error;
perror(msg);
exit(-error);
}
}
static void SetAffinity(int8_t cpu)
{
if(cpu < 0)
return;
cpu_set_t cpuset;
CPU_ZERO(&cpuset);
CPU_SET(cpu, &cpuset);
HandleError("pthread_set_affinity_np",
pthread_setaffinity_np(pthread_self(), sizeof(cpuset), &cpuset));
}
static void CreateTimer()
{
Timer_Fd = evl_new_timer(EVL_CLOCK_MONOTONIC);
if (Timer_Fd < 0) {
perror("evl_new_timer");
exit(errno);
}
}
static void DeleteTimer()
{
close(Timer_Fd);
}
static void timespec_add_ns(struct timespec *__restrict r,
const struct timespec *__restrict t,
long ns)
{
long s, rem;
s = ns / 1E9;
rem = ns - s * 1E9;
r->tv_sec = t->tv_sec + s;
r->tv_nsec = t->tv_nsec + rem;
if (r->tv_nsec >= 1E9) {
r->tv_sec++;
r->tv_nsec -= 1E9;
}
}
static void SetTimer()
{
struct itimerspec value, ovalue;
struct timespec now;
HandleEvlError("evl_read_clock", evl_read_clock(EVL_CLOCK_MONOTONIC, &now));
timespec_add_ns(&value.it_value, &now, 1E6);
value.it_interval.tv_sec = 0;
value.it_interval.tv_nsec = 25 * 1E3;
HandleEvlError("evl_set_timer", (evl_set_timer(Timer_Fd, &value, &ovalue)));
}
static void WaitTimer()
{
uint64_t ticks;
HandleEvlError("oob_read", oob_read(Timer_Fd, &ticks, sizeof(ticks)));
}
static void InitThread()
{
sched_param schedParam;
schedParam.sched_priority = Thread_Priority;
if (pthread_setschedparam(pthread_self(), SCHED_FIFO, &schedParam) != 0)
HandleError("pthread_setschedparam", errno);
HandleEvlError("evl_attach_self", evl_attach_self("timer-crash1-child-%lu", (uint64_t)getpid()));
SetAffinity(Thread_Affinity);
}
static void *ThreadFn(void *)
{
InitThread();
SetTimer();
for (int i = 0; i < 8; i++) {
WaitTimer();
}
DeleteTimer();
(void) evl_detach_self();
return nullptr;
}
static pthread_t StartThread()
{
pthread_attr_t attr;
HandleError("pthread_attr_init", pthread_attr_init(&attr));
HandleError("pthread_attr_setinheritsched",
pthread_attr_setinheritsched(&attr, PTHREAD_INHERIT_SCHED));
pthread_t thread_id;
HandleError("pthread_create", pthread_create(&thread_id, &attr, &ThreadFn, nullptr));
HandleError("pthread_attr_destroy", pthread_attr_destroy(&attr));
return thread_id;
}
static void JoinThread(pthread_t id)
{
void *thread_return;
HandleError("pthread_join", pthread_join(id, &thread_return));
}
static void ParseArgs(int argc, char **argv)
{
opterr = 0;
char *affinity_str = nullptr;
char *thread_affinity_str = nullptr;
char *prio_str = nullptr;
int c;
while ((c = getopt(argc, argv, "a:p:t:q")) != -1) {
switch (c)
{
case 'a':
affinity_str = optarg;
break;
case 'p':
prio_str = optarg;
break;
case 't':
thread_affinity_str = optarg;
break;
case 'q':
Quiet = true;
break;
default:
fprintf(stderr, "Options: [-q] [-a <affinity>] [-t <thread-affinity>] [-p <prio>]\n");
exit(1);
}
}
if (affinity_str != nullptr) {
Affinity = atoi(affinity_str);
if (Affinity < 0 || Affinity > 3) {
fprintf(stderr, "Bad affinity [%s]\n", affinity_str);
exit(1);
}
}
if (thread_affinity_str != nullptr) {
Thread_Affinity = atoi(thread_affinity_str);
if (Thread_Affinity < 0 || Thread_Affinity > 3) {
fprintf(stderr, "Bad thread affinity [%s]\n", thread_affinity_str);
exit(1);
}
}
if (prio_str != nullptr) {
Thread_Priority = atoi(prio_str);
if (Thread_Priority < 0 || Thread_Priority > 99) {
fprintf(stderr, "Bad priority [%s]\n", prio_str);
exit(1);
}
}
}
static void InitMainThread()
{
HandleEvlError("evl_init", evl_init());
SetAffinity(Affinity);
const auto curpid = getpid();
sched_param sp;
if (sched_getparam(curpid, &sp) != 0)
HandleError("sched_getparam", errno);
sp.sched_priority = Thread_Priority - 1;
if (sched_setscheduler(curpid, SCHED_FIFO, &sp) != 0)
HandleError("sched_setscheduler", errno);
HandleEvlError("evl_attach_self", evl_attach_self("timer-crash1-main-%lu", (uint64_t)getpid()));
}
int main(int argc, char **argv)
{
ParseArgs(argc, argv);
InitMainThread();
for(int i=0; ; ++i) {
const int n = 0x400;
if(!Quiet && !(i & (n-1)))
evl_printf("\r%d", i);
CreateTimer();
pthread_t id = StartThread();
JoinThread(id);
}
}
[-- Attachment #3: xenomai-timer-crash.txt --]
[-- Type: text/plain, Size: 8721 bytes --]
[ 407.186303] Internal error: Oops: 96000005 [#11] PREEMPT SMP
[ 407.186305] Modules linked in: ofpart cy_hal_generic spidev cy_spimem mtd lm75 at24 rtc_pcf85063 regmap_i2c bcm2835_v4l2(C) bcm2835_mmal_vchiq(C) videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 raspberrypi_hwmon videobuf2_common spi_bcm2835 videodev mc vc4 snd_soc_hdmi_codec cec lan743x drm_kms_helper snd_soc_core snd_pcm_dmaengine snd_pcm snd_timer snd syscopyarea sysfillrect sysimgblt fb_sys_fops i2c_gpio i2c_algo_bit nvmem_rmem uio_pdrv_genirq uio cfg80211 rfkill sch_fq_codel fuse drm drm_panel_orientation_quirks backlight configfs ipv6
[ 407.186344] CPU: 1 PID: 1104 Comm: timer-crash1-ch Tainted: G D C 5.15.109-mutex-interitance-fixes #1
[ 407.186347] Hardware name: Raspberry Pi Compute Module 4 Rev 1.1 (DT)
[ 407.186349] IRQ stage: Linux
[ 407.186350] pstate: 800003c5 (Nzcv DAIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 407.186353] pc : _raw_spin_lock_irq+0x68/0xa0
[ 407.186356] lr : _raw_spin_lock_irq+0x18/0xa0
[ 407.186360] sp : ffffffc00b2c1450
[ 407.186361] x29: ffffffc00b2c1450 x28: ffffff81110144c0 x27: ffffff81110144c0
[ 407.186365] x26: ffffff810283b088 x25: ffffffc008070428 x24: ffffffc008945788
[ 407.186369] x23: 00000000000003c0 x22: 000000000000000b x21: ffffffc00b2c15a7
[ 407.186373] x20: ffffff81110144c0 x19: 0000000000000000 x18: 0000000000000038
[ 407.186377] x17: 3030303030303030 x16: 3030303030303020 x15: ffffffffffff4848
[ 407.186382] x14: 0000000000000000 x13: 2164656465656e20 x12: 736920746f6f6265
[ 407.186386] x11: 722074756220746c x10: 7561662065766973 x9 : 6f6f626572207475
[ 407.186390] x8 : 6220746c75616620 x7 : 000000000105197f x6 : 000000000105197f
[ 407.186394] x5 : 0000000000000009 x4 : 0000000000000000 x3 : 0000000000000001
[ 407.186397] x2 : 0000000000000001 x1 : 0000000000000000 x0 : 0000000000000001
[ 407.186401] Call trace:
[ 407.186402] _raw_spin_lock_irq+0x68/0xa0
[ 407.186405] exit_signals+0xfc/0x1b0
[ 407.186408] do_exit+0xb4/0x9b0
[ 407.186410] make_task_dead+0x50/0x80
[ 407.186413] die+0x1f0/0x290
[ 407.186417] die_kernel_fault+0x64/0x74
[ 407.186420] __do_kernel_fault+0x90/0x18c
[ 407.186422] do_page_fault+0x218/0x4a0
[ 407.186426] do_translation_fault+0x58/0x80
[ 407.186430] do_mem_abort+0x44/0x134
[ 407.186432] el1_abort+0x40/0x60
[ 407.186436] el1h_64_sync_handler+0xa4/0xd0
[ 407.186440] el1h_64_sync+0x78/0x7c
[ 407.186442] _raw_spin_lock_irq+0x68/0xa0
[ 407.186445] exit_signals+0xfc/0x1b0
[ 407.186448] do_exit+0xb4/0x9b0
[ 407.186450] make_task_dead+0x50/0x80
[ 407.186453] die+0x1f0/0x290
[ 407.186457] die_kernel_fault+0x64/0x74
[ 407.186459] __do_kernel_fault+0x90/0x18c
[ 407.186462] do_page_fault+0x218/0x4a0
[ 407.186466] do_translation_fault+0x58/0x80
[ 407.186469] do_mem_abort+0x44/0x134
[ 407.186472] el1_abort+0x40/0x60
[ 407.186475] el1h_64_sync_handler+0xa4/0xd0
[ 407.186479] el1h_64_sync+0x78/0x7c
[ 407.186481] _raw_spin_lock_irq+0x68/0xa0
[ 407.186484] exit_signals+0xfc/0x1b0
[ 407.186487] do_exit+0xb4/0x9b0
[ 407.186489] make_task_dead+0x50/0x80
[ 407.186492] die+0x1f0/0x290
[ 407.186495] die_kernel_fault+0x64/0x74
[ 407.186498] __do_kernel_fault+0x90/0x18c
[ 407.186501] do_page_fault+0x218/0x4a0
[ 407.186504] do_translation_fault+0x58/0x80
[ 407.186508] do_mem_abort+0x44/0x134
[ 407.186511] el1_abort+0x40/0x60
[ 407.186514] el1h_64_sync_handler+0xa4/0xd0
[ 407.186518] el1h_64_sync+0x78/0x7c
[ 407.186520] _raw_spin_lock_irq+0x68/0xa0
[ 407.186523] exit_signals+0xfc/0x1b0
[ 407.186526] do_exit+0xb4/0x9b0
[ 407.186528] make_task_dead+0x50/0x80
[ 407.186531] die+0x1f0/0x290
[ 407.186534] die_kernel_fault+0x64/0x74
[ 407.186537] __do_kernel_fault+0x90/0x18c
[ 407.186540] do_page_fault+0x218/0x4a0
[ 407.186544] do_translation_fault+0x58/0x80
[ 407.186547] do_mem_abort+0x44/0x134
[ 407.186550] el1_abort+0x40/0x60
[ 407.186553] el1h_64_sync_handler+0xa4/0xd0
[ 407.186557] el1h_64_sync+0x78/0x7c
[ 407.186559] _raw_spin_lock_irq+0x68/0xa0
[ 407.186562] exit_signals+0xfc/0x1b0
[ 407.186565] do_exit+0xb4/0x9b0
[ 407.186567] make_task_dead+0x50/0x80
[ 407.186569] die+0x1f0/0x290
[ 407.186573] die_kernel_fault+0x64/0x74
[ 407.186576] __do_kernel_fault+0x90/0x18c
[ 407.186578] do_page_fault+0x218/0x4a0
[ 407.186582] do_translation_fault+0x58/0x80
[ 407.186586] do_mem_abort+0x44/0x134
[ 407.186589] el1_abort+0x40/0x60
[ 407.186592] el1h_64_sync_handler+0xa4/0xd0
[ 407.186596] el1h_64_sync+0x78/0x7c
[ 407.186598] _raw_spin_lock_irq+0x68/0xa0
[ 407.186601] exit_signals+0xfc/0x1b0
[ 407.186603] do_exit+0xb4/0x9b0
[ 407.186606] make_task_dead+0x50/0x80
[ 407.186608] die+0x1f0/0x290
[ 407.186612] die_kernel_fault+0x64/0x74
[ 407.186614] __do_kernel_fault+0x90/0x18c
[ 407.186617] do_page_fault+0x218/0x4a0
[ 407.186621] do_translation_fault+0x58/0x80
[ 407.186624] do_mem_abort+0x44/0x134
[ 407.186627] el1_abort+0x40/0x60
[ 407.186630] el1h_64_sync_handler+0xa4/0xd0
[ 407.186634] el1h_64_sync+0x78/0x7c
[ 407.186637] _raw_spin_lock_irq+0x68/0xa0
[ 407.186640] exit_signals+0xfc/0x1b0
[ 407.186642] do_exit+0xb4/0x9b0
[ 407.186645] make_task_dead+0x50/0x80
[ 407.186647] die+0x1f0/0x290
[ 407.186651] die_kernel_fault+0x64/0x74
[ 407.186654] __do_kernel_fault+0x90/0x18c
[ 407.186656] do_page_fault+0x218/0x4a0
[ 407.186660] do_translation_fault+0x58/0x80
[ 407.186664] do_mem_abort+0x44/0x134
[ 407.186666] el1_abort+0x40/0x60
[ 407.186670] el1h_64_sync_handler+0xa4/0xd0
[ 407.186673] el1h_64_sync+0x78/0x7c
[ 407.186675] _raw_spin_lock_irq+0x68/0xa0
[ 407.186679] exit_signals+0xfc/0x1b0
[ 407.186681] do_exit+0xb4/0x9b0
[ 407.186683] make_task_dead+0x50/0x80
[ 407.186686] die+0x1f0/0x290
[ 407.186689] die_kernel_fault+0x64/0x74
[ 407.186692] __do_kernel_fault+0x90/0x18c
[ 407.186695] do_page_fault+0x218/0x4a0
[ 407.186698] do_translation_fault+0x58/0x80
[ 407.186702] do_mem_abort+0x44/0x134
[ 407.186705] el1_abort+0x40/0x60
[ 407.186708] el1h_64_sync_handler+0xa4/0xd0
[ 407.186712] el1h_64_sync+0x78/0x7c
[ 407.186714] _raw_spin_lock_irq+0x68/0xa0
[ 407.186717] exit_signals+0xfc/0x1b0
[ 407.186719] do_exit+0xb4/0x9b0
[ 407.186722] make_task_dead+0x50/0x80
[ 407.186724] die+0x1f0/0x290
[ 407.186728] bug_handler+0x4c/0x6c
[ 407.186732] call_break_hook+0x68/0x7c
[ 407.186734] brk_handler+0x1c/0x60
[ 407.186736] do_debug_exception+0xbc/0x1a0
[ 407.186739] el1_dbg+0x70/0x90
[ 407.186743] el1h_64_sync_handler+0x94/0xd0
[ 407.186746] el1h_64_sync+0x78/0x7c
[ 407.186749] do_task_dead+0x48/0x50
[ 407.186752] do_exit+0x600/0x9b0
[ 407.186755] make_task_dead+0x50/0x80
[ 407.186757] die+0x1f0/0x290
[ 407.186761] die_kernel_fault+0x64/0x74
[ 407.186764] __do_kernel_fault+0x90/0x18c
[ 407.186766] do_bad_area+0xe4/0x124
[ 407.186769] do_translation_fault+0x44/0x80
[ 407.186773] do_mem_abort+0x44/0x134
[ 407.186776] el1_abort+0x40/0x60
[ 407.186779] el1h_64_sync_handler+0xa4/0xd0
[ 407.186783] el1h_64_sync+0x78/0x7c
[ 407.186785] evl_get_timer_overruns+0xec/0x1b0
[ 407.186788] timerfd_oob_read+0x230/0x270
[ 407.186792] EVL_read+0x50/0xe0
[ 407.186794] handle_pipelined_syscall+0x238/0x280
[ 407.186796] __pipeline_syscall+0xe8/0x2b0
[ 407.186799] pipeline_syscall+0x70/0x170
[ 407.186802] el0_svc_common.constprop.0+0x48/0x12c
[ 407.186805] do_el0_svc+0x28/0xa0
[ 407.186808] el0_svc+0x4c/0xd0
[ 407.186812] el0t_64_sync_handler+0xa4/0x130
[ 407.186816] el0t_64_sync+0x1a0/0x1a4
[ 407.186819] Code: d65f03c0 d2800001 52800022 f9800271 (885ffe60)
[ 407.949950] ---[ end trace 666bbd81997c9761 ]---
[ 407.949952] note: timer-crash1-ch[1104] exited with preempt_count 1
[ 407.949954] Fixing recursive fault but reboot is needed!
[ 407.949956] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[ 407.949957] Mem abort info:
[ 407.949958] ESR = 0x0000000096000005
[ 407.949959] EC = 0x25: DABT (current EL), IL = 32 bits
[ 407.949961] SET = 0, FnV = 0
[ 407.949962] EA = 0, S1PTW = 0
[ 407.949963] FSC = 0x05: level 1 translation fault
[ 407.949965] Data abort info:
[ 407.949966] ISV = 0, ISS = 0x00000005
[ 407.949967] CM = 0, WnR = 0
[ 407.949968] user pgtable: 4k pages, 39-bit VAs, pgdp=00000001040e0000
[ 407.949970] [0000000000000000] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000
[ 407.949974] Internal error: Oops: 96000005 [#12] PREEMPT SMP
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Xenomai crash in timer IRQ
2023-05-31 20:40 ` Sri Subramanian
@ 2023-06-03 10:38 ` Philippe Gerum
2023-06-03 13:55 ` Philippe Gerum
0 siblings, 1 reply; 13+ messages in thread
From: Philippe Gerum @ 2023-06-03 10:38 UTC (permalink / raw)
To: Sri Subramanian; +Cc: xenomai
Sri Subramanian <sridhar.subramanian@gmail.com> writes:
> Hi Philippe,
>
> Sorry for the long pause in this thread.
>
> However, I am able to reproduce the kernel crash stack (off
> timerfd_oob_read) with a simple test program which does the following:
>
> In a loop:
> create timer fd
> create thread which
> sets up periodic triggers off the monotonic timer
> closes the timer after a few periods.
> join w/thread
>
> If the main thread and the timer thread are in the same cpu, I can run
> it with hectic in the background for 2 hrs at least. However, if the
> threads are on different cpus, the kernel crashes within minutes. I am
> attaching the crash and the test program.
>
> Environment:
> HW: Raspberry Pi (CM4)
> SW: Xenomai 5.15.109 tag, Libevl r44
>
> Test steps:
> # hectic -s 200 -q&
> # ./timer-crash1 -a 0 -t 1 -q&
>
> BTW: this thread's history implies changes to the realtime clock, but
> really, this doesn't have anything to do with it.
Ouch, confirmed. Something is wrong, this is a generic issue. Working on
it.
root@homelab-qemu-x86_64:~# ./timerfd_crash -a 1 -t 2
0[ 347.430316] ------------[ cut here ]------------
[ 347.430319] WARNING: CPU: 2 PID: 3241 at kernel/evl/timer.c:430 evl_get_timer_overruns+0x1ac/0x1c0
[ 347.430332] Modules linked in:
[ 347.430334] CPU: 2 PID: 3241 Comm: timer-crash1-ch Not tainted 6.1.29+ #178
[ 347.430338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014
[ 347.430340] IRQ stage: EVL
[ 347.430342] RIP: 0010:evl_get_timer_overruns+0x1ac/0x1c0
[ 347.430348] Code: eb d5 e8 c7 d7 f5 ff 49 89 c4 e9 9f fe ff ff 4c 89 f6 48 89 df e8 44 fd ff ff 48 8b 4b 48 e9 ba fe ff ff 0f 0b e9 13 ff ff ff <0f> 0b e9 2e ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f
[ 347.430350] RSP: 0018:ffffc900016c7dc0 EFLAGS: 00010002
[ 347.430353] RAX: 0000000000000005 RBX: ffff88810429b400 RCX: 0000000000000007
[ 347.430355] RDX: 00000000000025f9 RSI: 00000000000061a8 RDI: ffff888237ac87d0
[ 347.430357] RBP: ffff888237ac87d0 R08: 00000000ffffffff R09: 0000000000000001
[ 347.430358] R10: 0000000000000082 R11: 0000000000000000 R12: 00000050e066c458
[ 347.430360] R13: 0000000000000001 R14: ffff88810429b4b0 R15: ffff88810429b400
[ 347.430368] FS: 00007fa304e746c0(0000) GS:ffff888237b00000(0000) knlGS:0000000000000000
[ 347.430370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 347.430372] CR2: 0000563c64c7d018 CR3: 00000001050ea000 CR4: 00000000000406e0
[ 347.430374] Call Trace:
[ 347.430376] <TASK>
[ 347.430378] timerfd_oob_read+0x190/0x210
[ 347.430385] EVL_read+0x46/0x90
[ 347.430388] do_oob_syscall+0x11a/0x360
[ 347.430392] handle_oob_syscall+0xa5/0xf0
[ 347.430395] pipeline_syscall+0x93/0x110
[ 347.430399] syscall_enter_from_user_mode+0x57/0x120
[ 347.430406] do_syscall_64+0x1d/0xa0
[ 347.430410] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 347.430414] RIP: 0033:0x7fa305064308
[ 347.430416] Code: Unable to access opcode bytes at 0x7fa3050642de.
[ 347.430418] RSP: 002b:00007fa304e73a30 EFLAGS: 00000246 ORIG_RAX: 00000000000
--
Philippe.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Xenomai crash in timer IRQ
2023-06-03 10:38 ` Philippe Gerum
@ 2023-06-03 13:55 ` Philippe Gerum
2023-06-03 14:31 ` Philippe Gerum
2023-06-05 4:47 ` Sri Subramanian
0 siblings, 2 replies; 13+ messages in thread
From: Philippe Gerum @ 2023-06-03 13:55 UTC (permalink / raw)
To: Sri Subramanian; +Cc: xenomai
Philippe Gerum <rpm@xenomai.org> writes:
> Sri Subramanian <sridhar.subramanian@gmail.com> writes:
>
>> Hi Philippe,
>>
>> Sorry for the long pause in this thread.
>>
>> However, I am able to reproduce the kernel crash stack (off
>> timerfd_oob_read) with a simple test program which does the following:
>>
>> In a loop:
>> create timer fd
>> create thread which
>> sets up periodic triggers off the monotonic timer
>> closes the timer after a few periods.
>> join w/thread
>>
>> If the main thread and the timer thread are in the same cpu, I can run
>> it with hectic in the background for 2 hrs at least. However, if the
>> threads are on different cpus, the kernel crashes within minutes. I am
>> attaching the crash and the test program.
>>
>> Environment:
>> HW: Raspberry Pi (CM4)
>> SW: Xenomai 5.15.109 tag, Libevl r44
>>
>> Test steps:
>> # hectic -s 200 -q&
>> # ./timer-crash1 -a 0 -t 1 -q&
>>
>> BTW: this thread's history implies changes to the realtime clock, but
>> really, this doesn't have anything to do with it.
>
> Ouch, confirmed. Something is wrong, this is a generic issue. Working on
> it.
>
> root@homelab-qemu-x86_64:~# ./timerfd_crash -a 1 -t 2
> 0[ 347.430316] ------------[ cut here ]------------
> [ 347.430319] WARNING: CPU: 2 PID: 3241 at kernel/evl/timer.c:430 evl_get_timer_overruns+0x1ac/0x1c0
> [ 347.430332] Modules linked in:
> [ 347.430334] CPU: 2 PID: 3241 Comm: timer-crash1-ch Not tainted 6.1.29+ #178
> [ 347.430338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014
> [ 347.430340] IRQ stage: EVL
> [ 347.430342] RIP: 0010:evl_get_timer_overruns+0x1ac/0x1c0
> [ 347.430348] Code: eb d5 e8 c7 d7 f5 ff 49 89 c4 e9 9f fe ff ff 4c 89 f6 48 89 df e8 44 fd ff ff 48 8b 4b 48 e9 ba fe ff ff 0f 0b e9 13 ff ff ff <0f> 0b e9 2e ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f
> [ 347.430350] RSP: 0018:ffffc900016c7dc0 EFLAGS: 00010002
> [ 347.430353] RAX: 0000000000000005 RBX: ffff88810429b400 RCX: 0000000000000007
> [ 347.430355] RDX: 00000000000025f9 RSI: 00000000000061a8 RDI: ffff888237ac87d0
> [ 347.430357] RBP: ffff888237ac87d0 R08: 00000000ffffffff R09: 0000000000000001
> [ 347.430358] R10: 0000000000000082 R11: 0000000000000000 R12: 00000050e066c458
> [ 347.430360] R13: 0000000000000001 R14: ffff88810429b4b0 R15: ffff88810429b400
> [ 347.430368] FS: 00007fa304e746c0(0000) GS:ffff888237b00000(0000) knlGS:0000000000000000
> [ 347.430370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 347.430372] CR2: 0000563c64c7d018 CR3: 00000001050ea000 CR4: 00000000000406e0
> [ 347.430374] Call Trace:
> [ 347.430376] <TASK>
> [ 347.430378] timerfd_oob_read+0x190/0x210
> [ 347.430385] EVL_read+0x46/0x90
> [ 347.430388] do_oob_syscall+0x11a/0x360
> [ 347.430392] handle_oob_syscall+0xa5/0xf0
> [ 347.430395] pipeline_syscall+0x93/0x110
> [ 347.430399] syscall_enter_from_user_mode+0x57/0x120
> [ 347.430406] do_syscall_64+0x1d/0xa0
> [ 347.430410] entry_SYSCALL_64_after_hwframe+0x63/0xcd
> [ 347.430414] RIP: 0033:0x7fa305064308
> [ 347.430416] Code: Unable to access opcode bytes at 0x7fa3050642de.
> [ 347.430418] RSP: 002b:00007fa304e73a30 EFLAGS: 00000246 ORIG_RAX: 00000000000
Please try this patch, this should help. To sum up, the conditions for
the bug to trigger are as follows:
- the timer and the thread consuming its ticks should run on distinct
CPUs,
- an overrun condition must exist (the interval set to 25µs in the test
case made this more likely fortunately),
This is an architecture-independent issue, caused by a SMP race.
Thanks for the neat test case, this definitely helped in spotting the
issue quickly.
diff --git a/include/evl/timer.h b/include/evl/timer.h
index 0f5616b26e385..c5ce4a0fff406 100644
--- a/include/evl/timer.h
+++ b/include/evl/timer.h
@@ -179,7 +179,7 @@ struct evl_timer {
int status;
ktime_t interval; /* 0 == oneshot */
ktime_t start_date;
- u64 pexpect_ticks; /* periodic release date */
+ u64 consumed_ticks; /* advanced by evl_get_timer_overruns() */
u64 periodic_ticks;
#ifdef CONFIG_SMP
struct evl_rq *rq;
@@ -263,7 +263,7 @@ static inline
ktime_t evl_get_timer_next_date(struct evl_timer *timer)
{
return ktime_add_ns(timer->start_date,
- timer->pexpect_ticks * ktime_to_ns(timer->interval));
+ timer->consumed_ticks * ktime_to_ns(timer->interval));
}
void __evl_init_timer(struct evl_timer *timer,
diff --git a/kernel/evl/clock.c b/kernel/evl/clock.c
index ca5422f2de3fd..e72db15cc6ca8 100644
--- a/kernel/evl/clock.c
+++ b/kernel/evl/clock.c
@@ -85,7 +85,7 @@ static void adjust_timer(struct evl_clock *clock,
*/
div = ktime_divns(-diff, ktime_to_ns(period));
timer->periodic_ticks -= div;
- timer->pexpect_ticks -= div;
+ timer->consumed_ticks -= div;
evl_update_timer_date(timer);
}
diff --git a/kernel/evl/timer.c b/kernel/evl/timer.c
index c609676eae5ba..a53a53eb3e6f9 100644
--- a/kernel/evl/timer.c
+++ b/kernel/evl/timer.c
@@ -160,7 +160,7 @@ void evl_start_timer(struct evl_timer *timer,
if (!timeout_infinite(interval)) {
timer->interval = interval;
timer->start_date = value;
- timer->pexpect_ticks = 0;
+ timer->consumed_ticks = 0;
timer->periodic_ticks = 0;
timer->status |= EVL_TIMER_PERIODIC;
}
@@ -415,21 +415,59 @@ unsigned long evl_get_timer_overruns(struct evl_timer *timer)
struct evl_tqueue *tq;
ktime_t now, delta;
- now = evl_read_clock(timer->clock);
+ if (EVL_WARN_ON_ONCE(CORE, !evl_timer_is_periodic(timer)))
+ return 0;
+
base = lock_timer_base(timer, &flags);
+ now = evl_read_clock(timer->clock);
+ /*
+ * Measure the lateness with respect to the expected expiry
+ * date in the timeline for this timer. This ideal expiry date
+ * would be the start date of the timer, plus the interval
+ * duration times the number of ticks which were consumed so
+ * far. Any delay longer than the periodic interval denotes an
+ * overrun condition since we missed at least one tick.
+ */
delta = ktime_sub(now, evl_get_timer_next_date(timer));
if (likely(delta < timer->interval))
goto done;
overruns = ktime_divns(delta, ktime_to_ns(timer->interval));
- timer->pexpect_ticks += overruns;
+
+ /*
+ * Compensate for the current lateness as the timer progresses
+ * in the timeline, consuming all missed ticks in one gulp to
+ * catch up.
+ */
+ timer->consumed_ticks += overruns;
+
+ /*
+ * We are about to re-enqueue the timer for its next expiry
+ * date. However, we refrain from doing so if either:
+ *
+ * - the timer was deactivated, in which case there is no
+ * point in planning for further expiry.
+ *
+ * - the timer is still running but not queued, which means
+ * that do_clock_tick() dequeued it then released the base
+ * lock on a remote CPU for running its handler. Since the
+ * timer is running and we hold that lock at the moment,
+ * do_clock_tick() is certainly going to either advance the
+ * expiry as required or shut the timer down after we drop the
+ * base lock. IOW, we may leave reprogramming to the core tick
+ * handler.
+ */
if (!evl_timer_is_running(timer))
goto done;
- EVL_WARN_ON_ONCE(CORE, (timer->status &
- (EVL_TIMER_DEQUEUED|EVL_TIMER_PERIODIC))
- != EVL_TIMER_PERIODIC);
+ if (timer->status & EVL_TIMER_DEQUEUED)
+ goto done;
+
+ /*
+ * Re-enqueue the periodic timer so that it ticks at the next
+ * interval boundary.
+ */
tq = &base->q;
evl_dequeue_timer(timer, tq);
while (evl_tdate(timer) < now) {
@@ -439,7 +477,7 @@ unsigned long evl_get_timer_overruns(struct evl_timer *timer)
program_timer(timer, tq);
done:
- timer->pexpect_ticks++;
+ timer->consumed_ticks++;
unlock_timer_base(base, flags);
--
Philippe.
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: Xenomai crash in timer IRQ
2023-06-03 13:55 ` Philippe Gerum
@ 2023-06-03 14:31 ` Philippe Gerum
2023-06-05 4:47 ` Sri Subramanian
1 sibling, 0 replies; 13+ messages in thread
From: Philippe Gerum @ 2023-06-03 14:31 UTC (permalink / raw)
To: Sri Subramanian; +Cc: xenomai
Philippe Gerum <rpm@xenomai.org> writes:
> Philippe Gerum <rpm@xenomai.org> writes:
>
>> Sri Subramanian <sridhar.subramanian@gmail.com> writes:
>>
>>> Hi Philippe,
>>>
>>> Sorry for the long pause in this thread.
>>>
>>> However, I am able to reproduce the kernel crash stack (off
>>> timerfd_oob_read) with a simple test program which does the following:
>>>
>>> In a loop:
>>> create timer fd
>>> create thread which
>>> sets up periodic triggers off the monotonic timer
>>> closes the timer after a few periods.
>>> join w/thread
>>>
>>> If the main thread and the timer thread are in the same cpu, I can run
>>> it with hectic in the background for 2 hrs at least. However, if the
>>> threads are on different cpus, the kernel crashes within minutes. I am
>>> attaching the crash and the test program.
>>>
>>> Environment:
>>> HW: Raspberry Pi (CM4)
>>> SW: Xenomai 5.15.109 tag, Libevl r44
>>>
>>> Test steps:
>>> # hectic -s 200 -q&
>>> # ./timer-crash1 -a 0 -t 1 -q&
>>>
>>> BTW: this thread's history implies changes to the realtime clock, but
>>> really, this doesn't have anything to do with it.
>>
>> Ouch, confirmed. Something is wrong, this is a generic issue. Working on
>> it.
>>
>> root@homelab-qemu-x86_64:~# ./timerfd_crash -a 1 -t 2
>> 0[ 347.430316] ------------[ cut here ]------------
>> [ 347.430319] WARNING: CPU: 2 PID: 3241 at kernel/evl/timer.c:430 evl_get_timer_overruns+0x1ac/0x1c0
>> [ 347.430332] Modules linked in:
>> [ 347.430334] CPU: 2 PID: 3241 Comm: timer-crash1-ch Not tainted 6.1.29+ #178
>> [ 347.430338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014
>> [ 347.430340] IRQ stage: EVL
>> [ 347.430342] RIP: 0010:evl_get_timer_overruns+0x1ac/0x1c0
>> [ 347.430348] Code: eb d5 e8 c7 d7 f5 ff 49 89 c4 e9 9f fe ff ff 4c 89 f6 48 89 df e8 44 fd ff ff 48 8b 4b 48 e9 ba fe ff ff 0f 0b e9 13 ff ff ff <0f> 0b e9 2e ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f
>> [ 347.430350] RSP: 0018:ffffc900016c7dc0 EFLAGS: 00010002
>> [ 347.430353] RAX: 0000000000000005 RBX: ffff88810429b400 RCX: 0000000000000007
>> [ 347.430355] RDX: 00000000000025f9 RSI: 00000000000061a8 RDI: ffff888237ac87d0
>> [ 347.430357] RBP: ffff888237ac87d0 R08: 00000000ffffffff R09: 0000000000000001
>> [ 347.430358] R10: 0000000000000082 R11: 0000000000000000 R12: 00000050e066c458
>> [ 347.430360] R13: 0000000000000001 R14: ffff88810429b4b0 R15: ffff88810429b400
>> [ 347.430368] FS: 00007fa304e746c0(0000) GS:ffff888237b00000(0000) knlGS:0000000000000000
>> [ 347.430370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [ 347.430372] CR2: 0000563c64c7d018 CR3: 00000001050ea000 CR4: 00000000000406e0
>> [ 347.430374] Call Trace:
>> [ 347.430376] <TASK>
>> [ 347.430378] timerfd_oob_read+0x190/0x210
>> [ 347.430385] EVL_read+0x46/0x90
>> [ 347.430388] do_oob_syscall+0x11a/0x360
>> [ 347.430392] handle_oob_syscall+0xa5/0xf0
>> [ 347.430395] pipeline_syscall+0x93/0x110
>> [ 347.430399] syscall_enter_from_user_mode+0x57/0x120
>> [ 347.430406] do_syscall_64+0x1d/0xa0
>> [ 347.430410] entry_SYSCALL_64_after_hwframe+0x63/0xcd
>> [ 347.430414] RIP: 0033:0x7fa305064308
>> [ 347.430416] Code: Unable to access opcode bytes at 0x7fa3050642de.
>> [ 347.430418] RSP: 002b:00007fa304e73a30 EFLAGS: 00000246 ORIG_RAX: 00000000000
>
> Please try this patch, this should help. To sum up, the conditions for
> the bug to trigger are as follows:
>
> - the timer and the thread consuming its ticks should run on distinct
> CPUs,
>
> - an overrun condition must exist (the interval set to 25µs in the test
> case made this more likely fortunately),
>
> This is an architecture-independent issue, caused by a SMP race.
>
> Thanks for the neat test case, this definitely helped in spotting the
> issue quickly.
>
> diff --git a/include/evl/timer.h b/include/evl/timer.h
> index 0f5616b26e385..c5ce4a0fff406 100644
> --- a/include/evl/timer.h
> +++ b/include/evl/timer.h
> @@ -179,7 +179,7 @@ struct evl_timer {
> int status;
> ktime_t interval; /* 0 == oneshot */
> ktime_t start_date;
> - u64 pexpect_ticks; /* periodic release date */
> + u64 consumed_ticks; /* advanced by evl_get_timer_overruns() */
> u64 periodic_ticks;
> #ifdef CONFIG_SMP
> struct evl_rq *rq;
> @@ -263,7 +263,7 @@ static inline
> ktime_t evl_get_timer_next_date(struct evl_timer *timer)
> {
> return ktime_add_ns(timer->start_date,
> - timer->pexpect_ticks * ktime_to_ns(timer->interval));
> + timer->consumed_ticks * ktime_to_ns(timer->interval));
> }
>
> void __evl_init_timer(struct evl_timer *timer,
> diff --git a/kernel/evl/clock.c b/kernel/evl/clock.c
> index ca5422f2de3fd..e72db15cc6ca8 100644
> --- a/kernel/evl/clock.c
> +++ b/kernel/evl/clock.c
> @@ -85,7 +85,7 @@ static void adjust_timer(struct evl_clock *clock,
> */
> div = ktime_divns(-diff, ktime_to_ns(period));
> timer->periodic_ticks -= div;
> - timer->pexpect_ticks -= div;
> + timer->consumed_ticks -= div;
> evl_update_timer_date(timer);
> }
>
> diff --git a/kernel/evl/timer.c b/kernel/evl/timer.c
> index c609676eae5ba..a53a53eb3e6f9 100644
> --- a/kernel/evl/timer.c
> +++ b/kernel/evl/timer.c
> @@ -160,7 +160,7 @@ void evl_start_timer(struct evl_timer *timer,
> if (!timeout_infinite(interval)) {
> timer->interval = interval;
> timer->start_date = value;
> - timer->pexpect_ticks = 0;
> + timer->consumed_ticks = 0;
> timer->periodic_ticks = 0;
> timer->status |= EVL_TIMER_PERIODIC;
> }
> @@ -415,21 +415,59 @@ unsigned long evl_get_timer_overruns(struct evl_timer *timer)
> struct evl_tqueue *tq;
> ktime_t now, delta;
>
> - now = evl_read_clock(timer->clock);
> + if (EVL_WARN_ON_ONCE(CORE, !evl_timer_is_periodic(timer)))
> + return 0;
> +
> base = lock_timer_base(timer, &flags);
> + now = evl_read_clock(timer->clock);
>
> + /*
> + * Measure the lateness with respect to the expected expiry
> + * date in the timeline for this timer. This ideal expiry date
> + * would be the start date of the timer, plus the interval
> + * duration times the number of ticks which were consumed so
> + * far. Any delay longer than the periodic interval denotes an
> + * overrun condition since we missed at least one tick.
> + */
> delta = ktime_sub(now, evl_get_timer_next_date(timer));
> if (likely(delta < timer->interval))
> goto done;
>
> overruns = ktime_divns(delta, ktime_to_ns(timer->interval));
> - timer->pexpect_ticks += overruns;
> +
> + /*
> + * Compensate for the current lateness as the timer progresses
> + * in the timeline, consuming all missed ticks in one gulp to
> + * catch up.
> + */
> + timer->consumed_ticks += overruns;
> +
> + /*
> + * We are about to re-enqueue the timer for its next expiry
> + * date. However, we refrain from doing so if either:
> + *
> + * - the timer was deactivated, in which case there is no
> + * point in planning for further expiry.
> + *
> + * - the timer is still running but not queued, which means
> + * that do_clock_tick() dequeued it then released the base
> + * lock on a remote CPU for running its handler. Since the
> + * timer is running and we hold that lock at the moment,
> + * do_clock_tick() is certainly going to either advance the
> + * expiry as required or shut the timer down after we drop the
> + * base lock. IOW, we may leave reprogramming to the core tick
> + * handler.
> + */
> if (!evl_timer_is_running(timer))
> goto done;
>
> - EVL_WARN_ON_ONCE(CORE, (timer->status &
> - (EVL_TIMER_DEQUEUED|EVL_TIMER_PERIODIC))
> - != EVL_TIMER_PERIODIC);
> + if (timer->status & EVL_TIMER_DEQUEUED)
> + goto done;
> +
> + /*
> + * Re-enqueue the periodic timer so that it ticks at the next
> + * interval boundary.
> + */
> tq = &base->q;
> evl_dequeue_timer(timer, tq);
> while (evl_tdate(timer) < now) {
> @@ -439,7 +477,7 @@ unsigned long evl_get_timer_overruns(struct evl_timer *timer)
>
> program_timer(timer, tq);
> done:
> - timer->pexpect_ticks++;
> + timer->consumed_ticks++;
>
> unlock_timer_base(base, flags);
Full patch with the appropriate commit log.
commit 1a2f7029b1f8e408d0ba47c3fd1517e077013423 (HEAD -> rebase/v6.1.y-evl)
Author: Philippe Gerum <rpm@xenomai.org>
Date: Sat Jun 3 16:14:46 2023 +0200
evl/timer: fix SMP race in overrun handling
evl_get_timer_overruns() might attempt to dequeue an unqueued timer
when handling an overrun, if the caller and the timer run on distinct
CPUs.
Fix this by detecting the pattern illustrated below, in which case
evl_get_timer_overruns() may leave the requeuing operation to the core
clock tick handler.
T: a thread waiting for wakeup events on a periodic timeline.
t: the periodic timer.
CPU0 CPU1
---- ----
T: <wait_for_next_period>
do_clock_tick()
lock(t->base)
dequeue(t)
unlock(t->base)
t->handler()
lock(t->base)
enqueue(t)
unlock(t->base)
remote-wakeup(T)
<OVERRUN>
do_clock_tick()
lock(t->base)
dequeue(t)
unlock(t->base)
T: evl_get_timer_overruns()
lock(t->base)
dequeue(t)
<!> double removal
See https://lore.kernel.org/xenomai/874jnoyghn.fsf@xenomai.org/T/
Signed-off-by: Philippe Gerum <rpm@xenomai.org>
diff --git a/include/evl/timer.h b/include/evl/timer.h
index 0f5616b26e385..c5ce4a0fff406 100644
--- a/include/evl/timer.h
+++ b/include/evl/timer.h
@@ -179,7 +179,7 @@ struct evl_timer {
int status;
ktime_t interval; /* 0 == oneshot */
ktime_t start_date;
- u64 pexpect_ticks; /* periodic release date */
+ u64 consumed_ticks; /* advanced by evl_get_timer_overruns() */
u64 periodic_ticks;
#ifdef CONFIG_SMP
struct evl_rq *rq;
@@ -263,7 +263,7 @@ static inline
ktime_t evl_get_timer_next_date(struct evl_timer *timer)
{
return ktime_add_ns(timer->start_date,
- timer->pexpect_ticks * ktime_to_ns(timer->interval));
+ timer->consumed_ticks * ktime_to_ns(timer->interval));
}
void __evl_init_timer(struct evl_timer *timer,
diff --git a/kernel/evl/clock.c b/kernel/evl/clock.c
index ca5422f2de3fd..e72db15cc6ca8 100644
--- a/kernel/evl/clock.c
+++ b/kernel/evl/clock.c
@@ -85,7 +85,7 @@ static void adjust_timer(struct evl_clock *clock,
*/
div = ktime_divns(-diff, ktime_to_ns(period));
timer->periodic_ticks -= div;
- timer->pexpect_ticks -= div;
+ timer->consumed_ticks -= div;
evl_update_timer_date(timer);
}
diff --git a/kernel/evl/timer.c b/kernel/evl/timer.c
index c609676eae5ba..a53a53eb3e6f9 100644
--- a/kernel/evl/timer.c
+++ b/kernel/evl/timer.c
@@ -160,7 +160,7 @@ void evl_start_timer(struct evl_timer *timer,
if (!timeout_infinite(interval)) {
timer->interval = interval;
timer->start_date = value;
- timer->pexpect_ticks = 0;
+ timer->consumed_ticks = 0;
timer->periodic_ticks = 0;
timer->status |= EVL_TIMER_PERIODIC;
}
@@ -415,21 +415,59 @@ unsigned long evl_get_timer_overruns(struct evl_timer *timer)
struct evl_tqueue *tq;
ktime_t now, delta;
- now = evl_read_clock(timer->clock);
+ if (EVL_WARN_ON_ONCE(CORE, !evl_timer_is_periodic(timer)))
+ return 0;
+
base = lock_timer_base(timer, &flags);
+ now = evl_read_clock(timer->clock);
+ /*
+ * Measure the lateness with respect to the expected expiry
+ * date in the timeline for this timer. This ideal expiry date
+ * would be the start date of the timer, plus the interval
+ * duration times the number of ticks which were consumed so
+ * far. Any delay longer than the periodic interval denotes an
+ * overrun condition since we missed at least one tick.
+ */
delta = ktime_sub(now, evl_get_timer_next_date(timer));
if (likely(delta < timer->interval))
goto done;
overruns = ktime_divns(delta, ktime_to_ns(timer->interval));
- timer->pexpect_ticks += overruns;
+
+ /*
+ * Compensate for the current lateness as the timer progresses
+ * in the timeline, consuming all missed ticks in one gulp to
+ * catch up.
+ */
+ timer->consumed_ticks += overruns;
+
+ /*
+ * We are about to re-enqueue the timer for its next expiry
+ * date. However, we refrain from doing so if either:
+ *
+ * - the timer was deactivated, in which case there is no
+ * point in planning for further expiry.
+ *
+ * - the timer is still running but not queued, which means
+ * that do_clock_tick() dequeued it then released the base
+ * lock on a remote CPU for running its handler. Since the
+ * timer is running and we hold that lock at the moment,
+ * do_clock_tick() is certainly going to either advance the
+ * expiry as required or shut the timer down after we drop the
+ * base lock. IOW, we may leave reprogramming to the core tick
+ * handler.
+ */
if (!evl_timer_is_running(timer))
goto done;
- EVL_WARN_ON_ONCE(CORE, (timer->status &
- (EVL_TIMER_DEQUEUED|EVL_TIMER_PERIODIC))
- != EVL_TIMER_PERIODIC);
+ if (timer->status & EVL_TIMER_DEQUEUED)
+ goto done;
+
+ /*
+ * Re-enqueue the periodic timer so that it ticks at the next
+ * interval boundary.
+ */
tq = &base->q;
evl_dequeue_timer(timer, tq);
while (evl_tdate(timer) < now) {
@@ -439,7 +477,7 @@ unsigned long evl_get_timer_overruns(struct evl_timer *timer)
program_timer(timer, tq);
done:
- timer->pexpect_ticks++;
+ timer->consumed_ticks++;
unlock_timer_base(base, flags);
--
Philippe.
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: Xenomai crash in timer IRQ
2023-06-03 13:55 ` Philippe Gerum
2023-06-03 14:31 ` Philippe Gerum
@ 2023-06-05 4:47 ` Sri Subramanian
2023-06-06 15:38 ` Sri Subramanian
1 sibling, 1 reply; 13+ messages in thread
From: Sri Subramanian @ 2023-06-05 4:47 UTC (permalink / raw)
To: Philippe Gerum; +Cc: xenomai
Hello Philippe,
Thanks for your quick fix! I can confirm it works for the test program
in my environment.
I'll try the fix in the product and let you know.
Thanks again!
Sri
On Sat, Jun 3, 2023 at 7:14 AM Philippe Gerum <rpm@xenomai.org> wrote:
>
>
> Philippe Gerum <rpm@xenomai.org> writes:
>
> > Sri Subramanian <sridhar.subramanian@gmail.com> writes:
> >
> >> Hi Philippe,
> >>
> >> Sorry for the long pause in this thread.
> >>
> >> However, I am able to reproduce the kernel crash stack (off
> >> timerfd_oob_read) with a simple test program which does the following:
> >>
> >> In a loop:
> >> create timer fd
> >> create thread which
> >> sets up periodic triggers off the monotonic timer
> >> closes the timer after a few periods.
> >> join w/thread
> >>
> >> If the main thread and the timer thread are in the same cpu, I can run
> >> it with hectic in the background for 2 hrs at least. However, if the
> >> threads are on different cpus, the kernel crashes within minutes. I am
> >> attaching the crash and the test program.
> >>
> >> Environment:
> >> HW: Raspberry Pi (CM4)
> >> SW: Xenomai 5.15.109 tag, Libevl r44
> >>
> >> Test steps:
> >> # hectic -s 200 -q&
> >> # ./timer-crash1 -a 0 -t 1 -q&
> >>
> >> BTW: this thread's history implies changes to the realtime clock, but
> >> really, this doesn't have anything to do with it.
> >
> > Ouch, confirmed. Something is wrong, this is a generic issue. Working on
> > it.
> >
> > root@homelab-qemu-x86_64:~# ./timerfd_crash -a 1 -t 2
> > 0[ 347.430316] ------------[ cut here ]------------
> > [ 347.430319] WARNING: CPU: 2 PID: 3241 at kernel/evl/timer.c:430 evl_get_timer_overruns+0x1ac/0x1c0
> > [ 347.430332] Modules linked in:
> > [ 347.430334] CPU: 2 PID: 3241 Comm: timer-crash1-ch Not tainted 6.1.29+ #178
> > [ 347.430338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014
> > [ 347.430340] IRQ stage: EVL
> > [ 347.430342] RIP: 0010:evl_get_timer_overruns+0x1ac/0x1c0
> > [ 347.430348] Code: eb d5 e8 c7 d7 f5 ff 49 89 c4 e9 9f fe ff ff 4c 89 f6 48 89 df e8 44 fd ff ff 48 8b 4b 48 e9 ba fe ff ff 0f 0b e9 13 ff ff ff <0f> 0b e9 2e ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f
> > [ 347.430350] RSP: 0018:ffffc900016c7dc0 EFLAGS: 00010002
> > [ 347.430353] RAX: 0000000000000005 RBX: ffff88810429b400 RCX: 0000000000000007
> > [ 347.430355] RDX: 00000000000025f9 RSI: 00000000000061a8 RDI: ffff888237ac87d0
> > [ 347.430357] RBP: ffff888237ac87d0 R08: 00000000ffffffff R09: 0000000000000001
> > [ 347.430358] R10: 0000000000000082 R11: 0000000000000000 R12: 00000050e066c458
> > [ 347.430360] R13: 0000000000000001 R14: ffff88810429b4b0 R15: ffff88810429b400
> > [ 347.430368] FS: 00007fa304e746c0(0000) GS:ffff888237b00000(0000) knlGS:0000000000000000
> > [ 347.430370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 347.430372] CR2: 0000563c64c7d018 CR3: 00000001050ea000 CR4: 00000000000406e0
> > [ 347.430374] Call Trace:
> > [ 347.430376] <TASK>
> > [ 347.430378] timerfd_oob_read+0x190/0x210
> > [ 347.430385] EVL_read+0x46/0x90
> > [ 347.430388] do_oob_syscall+0x11a/0x360
> > [ 347.430392] handle_oob_syscall+0xa5/0xf0
> > [ 347.430395] pipeline_syscall+0x93/0x110
> > [ 347.430399] syscall_enter_from_user_mode+0x57/0x120
> > [ 347.430406] do_syscall_64+0x1d/0xa0
> > [ 347.430410] entry_SYSCALL_64_after_hwframe+0x63/0xcd
> > [ 347.430414] RIP: 0033:0x7fa305064308
> > [ 347.430416] Code: Unable to access opcode bytes at 0x7fa3050642de.
> > [ 347.430418] RSP: 002b:00007fa304e73a30 EFLAGS: 00000246 ORIG_RAX: 00000000000
>
> Please try this patch, this should help. To sum up, the conditions for
> the bug to trigger are as follows:
>
> - the timer and the thread consuming its ticks should run on distinct
> CPUs,
>
> - an overrun condition must exist (the interval set to 25µs in the test
> case made this more likely fortunately),
>
> This is an architecture-independent issue, caused by a SMP race.
>
> Thanks for the neat test case, this definitely helped in spotting the
> issue quickly.
>
> diff --git a/include/evl/timer.h b/include/evl/timer.h
> index 0f5616b26e385..c5ce4a0fff406 100644
> --- a/include/evl/timer.h
> +++ b/include/evl/timer.h
> @@ -179,7 +179,7 @@ struct evl_timer {
> int status;
> ktime_t interval; /* 0 == oneshot */
> ktime_t start_date;
> - u64 pexpect_ticks; /* periodic release date */
> + u64 consumed_ticks; /* advanced by evl_get_timer_overruns() */
> u64 periodic_ticks;
> #ifdef CONFIG_SMP
> struct evl_rq *rq;
> @@ -263,7 +263,7 @@ static inline
> ktime_t evl_get_timer_next_date(struct evl_timer *timer)
> {
> return ktime_add_ns(timer->start_date,
> - timer->pexpect_ticks * ktime_to_ns(timer->interval));
> + timer->consumed_ticks * ktime_to_ns(timer->interval));
> }
>
> void __evl_init_timer(struct evl_timer *timer,
> diff --git a/kernel/evl/clock.c b/kernel/evl/clock.c
> index ca5422f2de3fd..e72db15cc6ca8 100644
> --- a/kernel/evl/clock.c
> +++ b/kernel/evl/clock.c
> @@ -85,7 +85,7 @@ static void adjust_timer(struct evl_clock *clock,
> */
> div = ktime_divns(-diff, ktime_to_ns(period));
> timer->periodic_ticks -= div;
> - timer->pexpect_ticks -= div;
> + timer->consumed_ticks -= div;
> evl_update_timer_date(timer);
> }
>
> diff --git a/kernel/evl/timer.c b/kernel/evl/timer.c
> index c609676eae5ba..a53a53eb3e6f9 100644
> --- a/kernel/evl/timer.c
> +++ b/kernel/evl/timer.c
> @@ -160,7 +160,7 @@ void evl_start_timer(struct evl_timer *timer,
> if (!timeout_infinite(interval)) {
> timer->interval = interval;
> timer->start_date = value;
> - timer->pexpect_ticks = 0;
> + timer->consumed_ticks = 0;
> timer->periodic_ticks = 0;
> timer->status |= EVL_TIMER_PERIODIC;
> }
> @@ -415,21 +415,59 @@ unsigned long evl_get_timer_overruns(struct evl_timer *timer)
> struct evl_tqueue *tq;
> ktime_t now, delta;
>
> - now = evl_read_clock(timer->clock);
> + if (EVL_WARN_ON_ONCE(CORE, !evl_timer_is_periodic(timer)))
> + return 0;
> +
> base = lock_timer_base(timer, &flags);
> + now = evl_read_clock(timer->clock);
>
> + /*
> + * Measure the lateness with respect to the expected expiry
> + * date in the timeline for this timer. This ideal expiry date
> + * would be the start date of the timer, plus the interval
> + * duration times the number of ticks which were consumed so
> + * far. Any delay longer than the periodic interval denotes an
> + * overrun condition since we missed at least one tick.
> + */
> delta = ktime_sub(now, evl_get_timer_next_date(timer));
> if (likely(delta < timer->interval))
> goto done;
>
> overruns = ktime_divns(delta, ktime_to_ns(timer->interval));
> - timer->pexpect_ticks += overruns;
> +
> + /*
> + * Compensate for the current lateness as the timer progresses
> + * in the timeline, consuming all missed ticks in one gulp to
> + * catch up.
> + */
> + timer->consumed_ticks += overruns;
> +
> + /*
> + * We are about to re-enqueue the timer for its next expiry
> + * date. However, we refrain from doing so if either:
> + *
> + * - the timer was deactivated, in which case there is no
> + * point in planning for further expiry.
> + *
> + * - the timer is still running but not queued, which means
> + * that do_clock_tick() dequeued it then released the base
> + * lock on a remote CPU for running its handler. Since the
> + * timer is running and we hold that lock at the moment,
> + * do_clock_tick() is certainly going to either advance the
> + * expiry as required or shut the timer down after we drop the
> + * base lock. IOW, we may leave reprogramming to the core tick
> + * handler.
> + */
> if (!evl_timer_is_running(timer))
> goto done;
>
> - EVL_WARN_ON_ONCE(CORE, (timer->status &
> - (EVL_TIMER_DEQUEUED|EVL_TIMER_PERIODIC))
> - != EVL_TIMER_PERIODIC);
> + if (timer->status & EVL_TIMER_DEQUEUED)
> + goto done;
> +
> + /*
> + * Re-enqueue the periodic timer so that it ticks at the next
> + * interval boundary.
> + */
> tq = &base->q;
> evl_dequeue_timer(timer, tq);
> while (evl_tdate(timer) < now) {
> @@ -439,7 +477,7 @@ unsigned long evl_get_timer_overruns(struct evl_timer *timer)
>
> program_timer(timer, tq);
> done:
> - timer->pexpect_ticks++;
> + timer->consumed_ticks++;
>
> unlock_timer_base(base, flags);
>
> --
> Philippe.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Xenomai crash in timer IRQ
2023-06-05 4:47 ` Sri Subramanian
@ 2023-06-06 15:38 ` Sri Subramanian
0 siblings, 0 replies; 13+ messages in thread
From: Sri Subramanian @ 2023-06-06 15:38 UTC (permalink / raw)
To: Philippe Gerum; +Cc: xenomai
Hello Phillipe,
I've tested the patch on the product, and can confirm that the patched
kernel works well and doesn't crash.
Thanks,
Sri
On Sun, Jun 4, 2023 at 9:47 PM Sri Subramanian
<sridhar.subramanian@gmail.com> wrote:
>
> Hello Philippe,
>
> Thanks for your quick fix! I can confirm it works for the test program
> in my environment.
>
> I'll try the fix in the product and let you know.
>
> Thanks again!
> Sri
>
> On Sat, Jun 3, 2023 at 7:14 AM Philippe Gerum <rpm@xenomai.org> wrote:
> >
> >
> > Philippe Gerum <rpm@xenomai.org> writes:
> >
> > > Sri Subramanian <sridhar.subramanian@gmail.com> writes:
> > >
> > >> Hi Philippe,
> > >>
> > >> Sorry for the long pause in this thread.
> > >>
> > >> However, I am able to reproduce the kernel crash stack (off
> > >> timerfd_oob_read) with a simple test program which does the following:
> > >>
> > >> In a loop:
> > >> create timer fd
> > >> create thread which
> > >> sets up periodic triggers off the monotonic timer
> > >> closes the timer after a few periods.
> > >> join w/thread
> > >>
> > >> If the main thread and the timer thread are in the same cpu, I can run
> > >> it with hectic in the background for 2 hrs at least. However, if the
> > >> threads are on different cpus, the kernel crashes within minutes. I am
> > >> attaching the crash and the test program.
> > >>
> > >> Environment:
> > >> HW: Raspberry Pi (CM4)
> > >> SW: Xenomai 5.15.109 tag, Libevl r44
> > >>
> > >> Test steps:
> > >> # hectic -s 200 -q&
> > >> # ./timer-crash1 -a 0 -t 1 -q&
> > >>
> > >> BTW: this thread's history implies changes to the realtime clock, but
> > >> really, this doesn't have anything to do with it.
> > >
> > > Ouch, confirmed. Something is wrong, this is a generic issue. Working on
> > > it.
> > >
> > > root@homelab-qemu-x86_64:~# ./timerfd_crash -a 1 -t 2
> > > 0[ 347.430316] ------------[ cut here ]------------
> > > [ 347.430319] WARNING: CPU: 2 PID: 3241 at kernel/evl/timer.c:430 evl_get_timer_overruns+0x1ac/0x1c0
> > > [ 347.430332] Modules linked in:
> > > [ 347.430334] CPU: 2 PID: 3241 Comm: timer-crash1-ch Not tainted 6.1.29+ #178
> > > [ 347.430338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014
> > > [ 347.430340] IRQ stage: EVL
> > > [ 347.430342] RIP: 0010:evl_get_timer_overruns+0x1ac/0x1c0
> > > [ 347.430348] Code: eb d5 e8 c7 d7 f5 ff 49 89 c4 e9 9f fe ff ff 4c 89 f6 48 89 df e8 44 fd ff ff 48 8b 4b 48 e9 ba fe ff ff 0f 0b e9 13 ff ff ff <0f> 0b e9 2e ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f
> > > [ 347.430350] RSP: 0018:ffffc900016c7dc0 EFLAGS: 00010002
> > > [ 347.430353] RAX: 0000000000000005 RBX: ffff88810429b400 RCX: 0000000000000007
> > > [ 347.430355] RDX: 00000000000025f9 RSI: 00000000000061a8 RDI: ffff888237ac87d0
> > > [ 347.430357] RBP: ffff888237ac87d0 R08: 00000000ffffffff R09: 0000000000000001
> > > [ 347.430358] R10: 0000000000000082 R11: 0000000000000000 R12: 00000050e066c458
> > > [ 347.430360] R13: 0000000000000001 R14: ffff88810429b4b0 R15: ffff88810429b400
> > > [ 347.430368] FS: 00007fa304e746c0(0000) GS:ffff888237b00000(0000) knlGS:0000000000000000
> > > [ 347.430370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [ 347.430372] CR2: 0000563c64c7d018 CR3: 00000001050ea000 CR4: 00000000000406e0
> > > [ 347.430374] Call Trace:
> > > [ 347.430376] <TASK>
> > > [ 347.430378] timerfd_oob_read+0x190/0x210
> > > [ 347.430385] EVL_read+0x46/0x90
> > > [ 347.430388] do_oob_syscall+0x11a/0x360
> > > [ 347.430392] handle_oob_syscall+0xa5/0xf0
> > > [ 347.430395] pipeline_syscall+0x93/0x110
> > > [ 347.430399] syscall_enter_from_user_mode+0x57/0x120
> > > [ 347.430406] do_syscall_64+0x1d/0xa0
> > > [ 347.430410] entry_SYSCALL_64_after_hwframe+0x63/0xcd
> > > [ 347.430414] RIP: 0033:0x7fa305064308
> > > [ 347.430416] Code: Unable to access opcode bytes at 0x7fa3050642de.
> > > [ 347.430418] RSP: 002b:00007fa304e73a30 EFLAGS: 00000246 ORIG_RAX: 00000000000
> >
> > Please try this patch, this should help. To sum up, the conditions for
> > the bug to trigger are as follows:
> >
> > - the timer and the thread consuming its ticks should run on distinct
> > CPUs,
> >
> > - an overrun condition must exist (the interval set to 25µs in the test
> > case made this more likely fortunately),
> >
> > This is an architecture-independent issue, caused by a SMP race.
> >
> > Thanks for the neat test case, this definitely helped in spotting the
> > issue quickly.
> >
> > diff --git a/include/evl/timer.h b/include/evl/timer.h
> > index 0f5616b26e385..c5ce4a0fff406 100644
> > --- a/include/evl/timer.h
> > +++ b/include/evl/timer.h
> > @@ -179,7 +179,7 @@ struct evl_timer {
> > int status;
> > ktime_t interval; /* 0 == oneshot */
> > ktime_t start_date;
> > - u64 pexpect_ticks; /* periodic release date */
> > + u64 consumed_ticks; /* advanced by evl_get_timer_overruns() */
> > u64 periodic_ticks;
> > #ifdef CONFIG_SMP
> > struct evl_rq *rq;
> > @@ -263,7 +263,7 @@ static inline
> > ktime_t evl_get_timer_next_date(struct evl_timer *timer)
> > {
> > return ktime_add_ns(timer->start_date,
> > - timer->pexpect_ticks * ktime_to_ns(timer->interval));
> > + timer->consumed_ticks * ktime_to_ns(timer->interval));
> > }
> >
> > void __evl_init_timer(struct evl_timer *timer,
> > diff --git a/kernel/evl/clock.c b/kernel/evl/clock.c
> > index ca5422f2de3fd..e72db15cc6ca8 100644
> > --- a/kernel/evl/clock.c
> > +++ b/kernel/evl/clock.c
> > @@ -85,7 +85,7 @@ static void adjust_timer(struct evl_clock *clock,
> > */
> > div = ktime_divns(-diff, ktime_to_ns(period));
> > timer->periodic_ticks -= div;
> > - timer->pexpect_ticks -= div;
> > + timer->consumed_ticks -= div;
> > evl_update_timer_date(timer);
> > }
> >
> > diff --git a/kernel/evl/timer.c b/kernel/evl/timer.c
> > index c609676eae5ba..a53a53eb3e6f9 100644
> > --- a/kernel/evl/timer.c
> > +++ b/kernel/evl/timer.c
> > @@ -160,7 +160,7 @@ void evl_start_timer(struct evl_timer *timer,
> > if (!timeout_infinite(interval)) {
> > timer->interval = interval;
> > timer->start_date = value;
> > - timer->pexpect_ticks = 0;
> > + timer->consumed_ticks = 0;
> > timer->periodic_ticks = 0;
> > timer->status |= EVL_TIMER_PERIODIC;
> > }
> > @@ -415,21 +415,59 @@ unsigned long evl_get_timer_overruns(struct evl_timer *timer)
> > struct evl_tqueue *tq;
> > ktime_t now, delta;
> >
> > - now = evl_read_clock(timer->clock);
> > + if (EVL_WARN_ON_ONCE(CORE, !evl_timer_is_periodic(timer)))
> > + return 0;
> > +
> > base = lock_timer_base(timer, &flags);
> > + now = evl_read_clock(timer->clock);
> >
> > + /*
> > + * Measure the lateness with respect to the expected expiry
> > + * date in the timeline for this timer. This ideal expiry date
> > + * would be the start date of the timer, plus the interval
> > + * duration times the number of ticks which were consumed so
> > + * far. Any delay longer than the periodic interval denotes an
> > + * overrun condition since we missed at least one tick.
> > + */
> > delta = ktime_sub(now, evl_get_timer_next_date(timer));
> > if (likely(delta < timer->interval))
> > goto done;
> >
> > overruns = ktime_divns(delta, ktime_to_ns(timer->interval));
> > - timer->pexpect_ticks += overruns;
> > +
> > + /*
> > + * Compensate for the current lateness as the timer progresses
> > + * in the timeline, consuming all missed ticks in one gulp to
> > + * catch up.
> > + */
> > + timer->consumed_ticks += overruns;
> > +
> > + /*
> > + * We are about to re-enqueue the timer for its next expiry
> > + * date. However, we refrain from doing so if either:
> > + *
> > + * - the timer was deactivated, in which case there is no
> > + * point in planning for further expiry.
> > + *
> > + * - the timer is still running but not queued, which means
> > + * that do_clock_tick() dequeued it then released the base
> > + * lock on a remote CPU for running its handler. Since the
> > + * timer is running and we hold that lock at the moment,
> > + * do_clock_tick() is certainly going to either advance the
> > + * expiry as required or shut the timer down after we drop the
> > + * base lock. IOW, we may leave reprogramming to the core tick
> > + * handler.
> > + */
> > if (!evl_timer_is_running(timer))
> > goto done;
> >
> > - EVL_WARN_ON_ONCE(CORE, (timer->status &
> > - (EVL_TIMER_DEQUEUED|EVL_TIMER_PERIODIC))
> > - != EVL_TIMER_PERIODIC);
> > + if (timer->status & EVL_TIMER_DEQUEUED)
> > + goto done;
> > +
> > + /*
> > + * Re-enqueue the periodic timer so that it ticks at the next
> > + * interval boundary.
> > + */
> > tq = &base->q;
> > evl_dequeue_timer(timer, tq);
> > while (evl_tdate(timer) < now) {
> > @@ -439,7 +477,7 @@ unsigned long evl_get_timer_overruns(struct evl_timer *timer)
> >
> > program_timer(timer, tq);
> > done:
> > - timer->pexpect_ticks++;
> > + timer->consumed_ticks++;
> >
> > unlock_timer_base(base, flags);
> >
> > --
> > Philippe.
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2023-06-06 15:38 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-12-02 20:45 Xenomai crash in timer IRQ Sri Subramanian
2022-12-03 16:06 ` Philippe Gerum
2022-12-03 16:33 ` Philippe Gerum
2022-12-07 21:00 ` Sri Subramanian
2022-12-08 7:53 ` Philippe Gerum
2022-12-09 17:05 ` Sri Subramanian
2022-12-19 21:51 ` Sri Subramanian
2023-05-31 20:40 ` Sri Subramanian
2023-06-03 10:38 ` Philippe Gerum
2023-06-03 13:55 ` Philippe Gerum
2023-06-03 14:31 ` Philippe Gerum
2023-06-05 4:47 ` Sri Subramanian
2023-06-06 15:38 ` Sri Subramanian
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.