* gdb test failure debug status update
@ 2021-04-28 3:19 Chen, Hongzhan
2021-04-28 14:18 ` Philippe Gerum
0 siblings, 1 reply; 16+ messages in thread
From: Chen, Hongzhan @ 2021-04-28 3:19 UTC (permalink / raw)
To: xenomai
According to my validation, gdb test fail on dovetail 5.10 branch but pass on v5.9-evl4 tag with same for-upstream/dovetail
xenomai code base.
After further debug , the issue is more clear for me. Gdb test failure because low priority thread smokey userspace is still
executed after "cobalt_shadow_relaxed: state=0x4488c0 info=0x200" like log [1] on dovetail-5.10 branch.
The weird thing is that its following first ftrace log happen at 62235.848583 after cobalt_shadow_relaxed in log [1].
It is almost 3ms happened after cobalt_shadow_relaxed. The low priority smoke thread user space is executed during this
3ms period so that test fail.
But in success case with v5.9-evl4 like in log [2], the time interval between cobalt_shadow_relaxed and the following first ftrace log
is only about 1us. It seems that low priority smokey userspace do not have chance to execute in this 1us because gdb test is successful.
My question is why there is even no interrupt happened during that about 3ms period in failure case? Tick seems in abnormal behavior.
Please comment if you have any ideas to further debug it.
PS: All my tests run on same up Xtream board.
1:
smokey-2024 [000] ..s1 62235.845648: softirq_exit: vec=6 [action=TASKLET]
smokey-2024 [000] d.s2 62235.845648: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x312
smokey-2024 [000] d.s1 62235.845648: irq_disable: caller=__do_softirq+0x280/0x428 parent=0x0
smokey-2024 [000] d.s2 62235.845648: rcu_dyntick: --= 4000000000000002 4000000000000000 0x312
smokey-2024 [000] *..2 62235.845649: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x312
smokey-2024 [000] *..1 62235.845649: irq_enable: caller=sync_current_irq_stage+0x259/0x280 parent=0x0
smokey-2024 [000] *..2 62235.845649: rcu_dyntick: --= 4000000000000002 4000000000000000 0x312
smokey-2024 [000] ...1 62235.845650: cobalt_shadow_relaxed: state=0x4488c0 info=0x200
smokey-2024 [000] *..1 62235.848583: irq_disable: caller=irqentry_enter+0x40/0xe0 parent=0x0
smokey-2024 [000] *.~1 62235.848583: irq_pipeline_entry: irq=4354
smokey-2024 [000] *.~1 62235.848584: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
smokey-2024 [000] *.~1 62235.848584: local_timer_entry: vector=236
smokey-2024 [000] *.~1 62235.848585: cobalt_timer_expire: timer=000000002aae459f
smokey-2024 [000] *.~1 62235.848585: local_timer_exit: vector=236
smokey-2024 [000] *.~1 62235.848585: irq_handler_exit: irq=4354 ret=handled
smokey-2024 [000] *.~1 62235.848585: irq_pipeline_exit: irq=4354
smokey-2024 [000] *.~1 62235.848586: irq_enable: caller=arch_pipeline_entry+0x4e/0xc0 parent=0x0
smokey-2024 [000] *..2 62235.848586: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x312
smokey-2024 [000] *..1 62235.848586: irq_disable: caller=sync_current_irq_stage+0x246/0x280 parent=0x0
smokey-2024 [000] *..2 62235.848586: rcu_dyntick: --= 4000000000000002 4000000000000000 0x312
smokey-2024 [000] d.h1 62235.848587: irq_handler_entry: irq=124 name=proxy tick
smokey-2024 [000] d.h2 62235.848587: lock_acquire: 000000004fef83e9 hrtimer_bases.lock
smokey-2024 [000] d.h2 62235.848588: lock_acquire: 00000000ed2aa56a read tk_core.seq.seqcount
smokey-2024 [000] d.h2 62235.848588: lock_release: 00000000ed2aa56a tk_core.seq.seqcount
smokey-2024 [000] d.h2 62235.848588: hrtimer_cancel: hrtimer=000000009dcbf7ab
smokey-2024 [000] d.h2 62235.848588: lock_release: 000000004fef83e9 hrtimer_bases.lock
smokey-2024 [000] d.h1 62235.848589: hrtimer_expire_entry: hrtimer=000000009dcbf7ab function=tick_sched_timer now=62232458752330
smokey-2024 [000] d.h1 62235.848589: lock_acquire: 00000000ed2aa56a read tk_core.seq.seqcount
smokey-2024 [000] d.h1 62235.848589: lock_release: 00000000ed2aa56a tk_core.seq.seqcount
smokey-2024 [000] d.h2 62235.848590: lock_acquire: 000000002286d635 jiffies_lock
smokey-2024 [000] d.h2 62235.848590: lock_acquire: 000000007174a29a (null)
smokey-2024 [000] d.h2 62235.848590: lock_release: 000000007174a29a (null)
[2] :
smokey-703 [000] d.s2 5674.404286: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x26e
smokey-703 [000] d.s1 5674.404286: irq_disable: caller=inband_irq_save+0x20/0x30 parent=0x0
smokey-703 [000] d.s2 5674.404286: rcu_dyntick: --= 4000000000000002 4000000000000000 0x26e
smokey-703 [000] d.s1 5674.404287: rcu_batch_end: rcu_sched CBs-invoked=8 idle=....
smokey-703 [000] d.s2 5674.404287: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x26e
smokey-703 [000] d.s1 5674.404287: irq_enable: caller=rcu_core+0x3b6/0xb60 parent=0x0
smokey-703 [000] d.s2 5674.404287: rcu_dyntick: --= 4000000000000002 4000000000000000 0x26e
smokey-703 [000] ..s1 5674.404288: rcu_utilization: End RCU core
smokey-703 [000] ..s1 5674.404288: softirq_exit: vec=9 [action=RCU]
smokey-703 [000] d.s2 5674.404288: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x26e
smokey-703 [000] d.s1 5674.404288: irq_disable: caller=__do_softirq+0x2bd/0x471 parent=0x0
smokey-703 [000] d.s2 5674.404289: rcu_dyntick: --= 4000000000000002 4000000000000000 0x26e
smokey-703 [000] *..2 5674.404289: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x26e
smokey-703 [000] *..1 5674.404289: irq_enable: caller=sync_current_irq_stage+0x259/0x280 parent=0x0
smokey-703 [000] *..2 5674.404290: rcu_dyntick: --= 4000000000000002 4000000000000000 0x26e
smokey-703 [000] ...1 5674.404290: cobalt_shadow_relaxed: state=0x4488c0 info=0x200
smokey-703 [000] *..2 5674.404291: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x26e
smokey-703 [000] *..1 5674.404291: irq_disable: caller=arch_pipeline_entry+0x13a/0x170 parent=0x0
smokey-703 [000] *..2 5674.404292: rcu_dyntick: --= 4000000000000002 4000000000000000 0x26e
smokey-703 [000] d..2 5674.404292: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x26e
smokey-703 [000] d..1 5674.404292: irq_enable: caller=exit_to_user_mode_prepare+0x118/0x1e0 parent=0x0
smokey-703 [000] d..2 5674.404292: rcu_dyntick: --= 4000000000000002 4000000000000000 0x26e
smokey-703 [000] d..2 5674.404293: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x26e
smokey-703 [000] d..1 5674.404293: irq_disable: caller=_raw_spin_lock_irq+0x12/0x50 parent=0x0
smokey-703 [000] d..2 5674.404293: rcu_dyntick: --= 4000000000000002 4000000000000000 0x26e
smokey-703 [000] d..2 5674.404294: lock_acquire: 000000007f29b2fb &sighand->siglock
smokey-703 [000] d..2 5674.404294: kmem_cache_free: call_site=__dequeue_signal+0xf7/0x150 ptr=000000003929a99e
smokey-703 [000] d..3 5674.404295: lock_acquire: 000000002505fb17 &p->pi_lock
smokey-703 [000] d..3 5674.404295: lock_release: 000000002505fb17 &p->pi_lock
smokey-703 [000] d..2 5674.404296: lock_release: 000000007f29b2fb &sighand->siglock
smokey-703 [000] d..3 5674.404296: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x26e
smokey-703 [000] d..2 5674.404296: irq_enable: caller=_raw_spin_unlock_irq+0x1f/0x30 parent=0x0
smokey-703 [000] d..3 5674.404296: rcu_dyntick: --= 4000000000000002 4000000000000000 0x26e
smokey-703 [000] ...2 5674.404297: lock_acquire: 00000000e714a308 read tasklist_lock
smokey-703 [000] ...3 5674.404297: lock_acquire: 00000000909a50fe read rcu_read_lock
Regards
Hongzhan Chen
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: gdb test failure debug status update 2021-04-28 3:19 gdb test failure debug status update Chen, Hongzhan @ 2021-04-28 14:18 ` Philippe Gerum 2021-04-28 14:30 ` Philippe Gerum 0 siblings, 1 reply; 16+ messages in thread From: Philippe Gerum @ 2021-04-28 14:18 UTC (permalink / raw) To: Chen, Hongzhan; +Cc: xenomai Chen, Hongzhan via Xenomai <xenomai@xenomai.org> writes: > According to my validation, gdb test fail on dovetail 5.10 branch but pass on v5.9-evl4 tag with same for-upstream/dovetail > xenomai code base. > > After further debug , the issue is more clear for me. Gdb test failure because low priority thread smokey userspace is still > executed after "cobalt_shadow_relaxed: state=0x4488c0 info=0x200" like log [1] on dovetail-5.10 branch. > The weird thing is that its following first ftrace log happen at 62235.848583 after cobalt_shadow_relaxed in log [1]. > It is almost 3ms happened after cobalt_shadow_relaxed. The low priority smoke thread user space is executed during this > 3ms period so that test fail. > > But in success case with v5.9-evl4 like in log [2], the time interval between cobalt_shadow_relaxed and the following first ftrace log > is only about 1us. It seems that low priority smokey userspace do not have chance to execute in this 1us because gdb test is successful. > > My question is why there is even no interrupt happened during that about 3ms period in failure case? Tick seems in abnormal behavior. > Please comment if you have any ideas to further debug it. > > PS: All my tests run on same up Xtream board. <snip> Let's put aside the tick issue for now, there may be a valid reason for this delay with dynticks enabled. The issue at stake may be related to the way a return to kernel space is forced on a @user task (Dovetail has an integrated service for triggering this called dovetail_request_ucall()). The logic for doing so is as follows: 1. @user hits a breakpoint, which is an exception Dovetail-wise 2. @user gets XNDBGSTOP set into its flags because Cobalt notices it is being debugged via a breakpoint trap, then relaxed as a result of taking a exception in general, so that we may traverse the common trap handling code safely. 3. since XNDBGSTOP is a blocking bit Cobalt-wise, it should prevent @user from being picked for scheduling by the real-time core, next time a Cobalt considers rescheduling that is. However, since @user is currently relaxed, it can still run under the supervision of the common Linux scheduler. This is what the log[1] show. 4. the common/in-band kernel code stops @user due to the ptrace stop condition caused by the breakpoint, waiting for a continuation event to happen. Therefore, upon PTRACE_CONT (i.e. gdb continue), we need to force @user to call back into kernel context (handle_ptrace_cont -> dovetail_request_ucall), then ask for a switch to primary mode from there, which should eventually happen when @user is about to leave the kernel (on x86, this now happens from a generic kernel entry/exit code in kernel/entry/*). As a result, handle_taskexit_event() runs, figures out that @user is pending a switch to primary mode. As it switches to primary mode, @user would be blocked by Cobalt from running further, because XNDBGSTOP is set into its internal state. So, I would check a few things for starters: - is dovetail_request_ucall() working properly. - is XNCONTHI properly set into the local Cobalt flags of @user when handle_user_return() is entered. - is this path taken as expected once dovetail_request_ucall() has run for @user: exit_to_user_mode_prepare -> do_retuser -> inband_retuser_notify (kernel/entry/common.c)? It may be a good idea to enable all cobalt tracepoints, add one to handle_user_return() too. -- Philippe. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: gdb test failure debug status update 2021-04-28 14:18 ` Philippe Gerum @ 2021-04-28 14:30 ` Philippe Gerum 2021-04-29 6:31 ` Chen, Hongzhan 0 siblings, 1 reply; 16+ messages in thread From: Philippe Gerum @ 2021-04-28 14:30 UTC (permalink / raw) To: Philippe Gerum; +Cc: Chen, Hongzhan, xenomai Philippe Gerum via Xenomai <xenomai@xenomai.org> writes: > Chen, Hongzhan via Xenomai <xenomai@xenomai.org> writes: > >> According to my validation, gdb test fail on dovetail 5.10 branch but pass on v5.9-evl4 tag with same for-upstream/dovetail >> xenomai code base. >> >> After further debug , the issue is more clear for me. Gdb test failure because low priority thread smokey userspace is still >> executed after "cobalt_shadow_relaxed: state=0x4488c0 info=0x200" like log [1] on dovetail-5.10 branch. >> The weird thing is that its following first ftrace log happen at 62235.848583 after cobalt_shadow_relaxed in log [1]. >> It is almost 3ms happened after cobalt_shadow_relaxed. The low priority smoke thread user space is executed during this >> 3ms period so that test fail. >> >> But in success case with v5.9-evl4 like in log [2], the time interval between cobalt_shadow_relaxed and the following first ftrace log >> is only about 1us. It seems that low priority smokey userspace do not have chance to execute in this 1us because gdb test is successful. >> >> My question is why there is even no interrupt happened during that about 3ms period in failure case? Tick seems in abnormal behavior. >> Please comment if you have any ideas to further debug it. >> >> PS: All my tests run on same up Xtream board. > > <snip> > > Let's put aside the tick issue for now, there may be a valid reason for > this delay with dynticks enabled. > > The issue at stake may be related to the way a return to kernel space is > forced on a @user task (Dovetail has an integrated service for > triggering this called dovetail_request_ucall()). > > The logic for doing so is as follows: > > 1. @user hits a breakpoint, which is an exception Dovetail-wise > > 2. @user gets XNDBGSTOP set into its flags because Cobalt notices it is > being debugged via a breakpoint trap, then relaxed as a result of taking > a exception in general, so that we may traverse the common trap handling code > safely. > > 3. since XNDBGSTOP is a blocking bit Cobalt-wise, it should prevent > @user from being picked for scheduling by the real-time core, next time > a Cobalt considers rescheduling that is. However, since @user is > currently relaxed, it can still run under the supervision of the common > Linux scheduler. This is what the log[1] show. > > 4. the common/in-band kernel code stops @user due to the ptrace stop > condition caused by the breakpoint, waiting for a continuation event to > happen. > > Therefore, upon PTRACE_CONT (i.e. gdb continue), we need to force @user > to call back into kernel context (handle_ptrace_cont -> > dovetail_request_ucall), then ask for a switch to primary mode from > there, which should eventually happen when @user is about to leave the > kernel (on x86, this now happens from a generic kernel entry/exit code > in kernel/entry/*). As a result, handle_taskexit_event() runs, figures Not quite: s,handle_taskexit_event,handle_user_return, -- Philippe. ^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: gdb test failure debug status update 2021-04-28 14:30 ` Philippe Gerum @ 2021-04-29 6:31 ` Chen, Hongzhan 2021-04-30 5:25 ` Chen, Hongzhan 0 siblings, 1 reply; 16+ messages in thread From: Chen, Hongzhan @ 2021-04-29 6:31 UTC (permalink / raw) To: Philippe Gerum; +Cc: xenomai > >-----Original Message----- >From: Philippe Gerum <rpm@xenomai.org> >Sent: Wednesday, April 28, 2021 10:30 PM >To: Philippe Gerum <rpm@xenomai.org> >Cc: Chen, Hongzhan <hongzhan.chen@intel.com>; xenomai@xenomai.org >Subject: Re: gdb test failure debug status update > > >Philippe Gerum via Xenomai <xenomai@xenomai.org> writes: > >> Chen, Hongzhan via Xenomai <xenomai@xenomai.org> writes: >> >>> According to my validation, gdb test fail on dovetail 5.10 branch but pass on v5.9-evl4 tag with same for-upstream/dovetail >>> xenomai code base. >>> >>> After further debug , the issue is more clear for me. Gdb test failure because low priority thread smokey userspace is still >>> executed after "cobalt_shadow_relaxed: state=0x4488c0 info=0x200" like log [1] on dovetail-5.10 branch. >>> The weird thing is that its following first ftrace log happen at 62235.848583 after cobalt_shadow_relaxed in log [1]. >>> It is almost 3ms happened after cobalt_shadow_relaxed. The low priority smoke thread user space is executed during this >>> 3ms period so that test fail. >>> >>> But in success case with v5.9-evl4 like in log [2], the time interval between cobalt_shadow_relaxed and the following first ftrace log >>> is only about 1us. It seems that low priority smokey userspace do not have chance to execute in this 1us because gdb test is successful. >>> >>> My question is why there is even no interrupt happened during that about 3ms period in failure case? Tick seems in abnormal behavior. >>> Please comment if you have any ideas to further debug it. >>> >>> PS: All my tests run on same up Xtream board. >> >> <snip> >> >> Let's put aside the tick issue for now, there may be a valid reason for >> this delay with dynticks enabled. >> >> The issue at stake may be related to the way a return to kernel space is >> forced on a @user task (Dovetail has an integrated service for >> triggering this called dovetail_request_ucall()). >> >> The logic for doing so is as follows: >> >> 1. @user hits a breakpoint, which is an exception Dovetail-wise >> >> 2. @user gets XNDBGSTOP set into its flags because Cobalt notices it is >> being debugged via a breakpoint trap, then relaxed as a result of taking >> a exception in general, so that we may traverse the common trap handling code >> safely. >> >> 3. since XNDBGSTOP is a blocking bit Cobalt-wise, it should prevent >> @user from being picked for scheduling by the real-time core, next time >> a Cobalt considers rescheduling that is. However, since @user is >> currently relaxed, it can still run under the supervision of the common >> Linux scheduler. This is what the log[1] show. >> >> 4. the common/in-band kernel code stops @user due to the ptrace stop >> condition caused by the breakpoint, waiting for a continuation event to >> happen. >> >> Therefore, upon PTRACE_CONT (i.e. gdb continue), we need to force @user >> to call back into kernel context (handle_ptrace_cont -> >> dovetail_request_ucall), then ask for a switch to primary mode from >> there, which should eventually happen when @user is about to leave the >> kernel (on x86, this now happens from a generic kernel entry/exit code >> in kernel/entry/*). As a result, handle_taskexit_event() runs, figures > >Not quite: >s,handle_taskexit_event,handle_user_return, There is trace_cobalt_shadow_unmap in handle_taskexit_event. I added trace_cobalt_handle_user_return and trace_cobalt_handle_ptrace_cont like patch [3] in log [4]. 1. In the log [4], the first cobalt_handle_ptrace_cont is followed by cobalt_handle_user_return that clear info flag XNCONTHI for smokey-2499 because cobalt_handle_ptrace_cont called dovetail_request_ucall . the TIF_RETUSER bit was cleared for smokey-2499. After then 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000 clear state flag XNDBGSTOP. After that, thread smokey-2499 run without these bit set till "hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000". 2. There is second cobalt_handle_ptrace_cont called for smokey-2499 but at that time dovetail_request_ucall would not be called because XNSSTEP Is not set like "smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0" 3. hi-thread-2504 breakpoint at its first breakpoint_target so it trigger hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3 It call dovetail_request_ucall(current); for itself and also call xnthread_set_info(thread, XNCONTHI); That is why we can see this bit set in hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200 4. finally ,gdb call sys_tkill in gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13) and trigger that handle_oob_mayday is called so that xnthread_relax for smokey-2499 Is called in "smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined." It seems that there never call dovetail_request_ucall for process smokey-2499 in log[4] so that final xhthread_relax would have chance to return to user? The second handle_ptrace_cont happen too much earlier because XNSSTEP still have not been set so that system never have change to call dovetail_request_ucall for smokey-2499. In this case , do we need to do hack in 3 , when handle_oob_trap_entry is called for hi-thread-2504 we call xnthread_set_info(thread, XNCONTHI); and dovetail_request_ucall for all threads that have same cobalt_process? Is this solution OK for fixing this case? [3]: index 07dcc7a63..cd76961f5 100644 --- a/kernel/cobalt/dovetail/kevents.c +++ b/kernel/cobalt/dovetail/kevents.c @@ -331,6 +331,8 @@ static int handle_user_return(struct task_struct *task) if (thread == NULL) return KEVENT_PROPAGATE; + trace_cobalt_handle_user_return(thread); + if (xnthread_test_info(thread, XNCONTHI)) { xnlock_get_irqsave(&nklock, s); xnthread_clear_info(thread, XNCONTHI); @@ -468,6 +470,8 @@ int handle_ptrace_resume(struct task_struct *tracee) if (thread == NULL) return KEVENT_PROPAGATE; + trace_cobalt_handle_ptrace_resume(thread); + if (xnthread_test_state(thread, XNSSTEP)) { xnlock_get_irqsave(&nklock, s); @@ -485,6 +489,8 @@ static void handle_ptrace_cont(void) struct xnthread *curr = xnthread_current(); spl_t s; + trace_cobalt_handle_ptrace_cont(curr); + xnlock_get_irqsave(&nklock, s); if (xnthread_test_state(curr, XNSSTEP)) { @@ -604,6 +610,7 @@ void resume_oob_task(struct task_struct *p) /* inband, oob stage stalled */ if (unlikely(xnthread_test_info(thread, XNCONTHI))) { xnthread_clear_info(thread, XNCONTHI); dovetail_clear_ucall(); + trace_cobalt_resume_oob_task(thread); } /* Unregister as debugged thread in case we postponed this. */ diff --git a/kernel/cobalt/trace/cobalt-core.h b/kernel/cobalt/trace/cobalt-core.h index 78f66a106..88a4067fd 100644 --- a/kernel/cobalt/trace/cobalt-core.h +++ b/kernel/cobalt/trace/cobalt-core.h @@ -459,6 +459,27 @@ DEFINE_EVENT(curr_thread_event, cobalt_thread_set_mode, TP_ARGS(thread) ); +DEFINE_EVENT(thread_event, cobalt_handle_ptrace_cont, + TP_PROTO(struct xnthread *thread), [4]: gdb-2495 [000] d..3 61513.834825: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] ...1 61513.834827: cobalt_handle_ptrace_cont: pid=2499 state=0x4488c0 info=0x200 smokey-2499 [000] d..1 61513.834828: cobalt_handle_user_return: pid=2499 state=0x4488c0 info=0x200 smokey-2499 [000] d..1 61513.834828: cobalt_shadow_gohard: state=0x4488c0 info=0x0 smokey-2499 [000] d..3 61513.834830: sched_stat_wait: comm=gdb pid=2495 delay=8017 [ns] smokey-2499 [000] d..3 61513.834831: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] *..3 61513.834831: cobalt_thread_resume: name=smokey pid=2499 mask=0x80 gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1 gdb-2495 [000] *..3 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000 gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1 gdb-2495 [000] *.~3 61513.834833: cobalt_schedule: status=0x10000000 gdb-2495 [000] *.~3 61513.834833: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] *.~3 61513.834833: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1 smokey-2499 [000] d.~3 61513.834835: cobalt_shadow_hardened: state=0x48040 info=0x0 smokey-2499 [000] d.~3 61513.834836: cobalt_head_sysentry: syscall=sem_init smokey-2499 [000] d.~3 61513.834839: cobalt_psem_init: sem=0x4e(anon) flags=0x0() value=0 smokey-2499 [000] d.~3 61513.834839: cobalt_head_sysexit: result=0 smokey-2499 [000] d.~3 61513.834848: cobalt_shadow_gorelax: reason=syscall smokey-2499 [000] d.~3 61513.834848: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey smokey-2499 [000] *.~3 61513.834848: cobalt_thread_suspend: pid=2499 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 smokey-2499 [000] *.~3 61513.834849: cobalt_schedule: status=0x10000000 smokey-2499 [000] *.~3 61513.834849: cobalt_trace_pid: pid=2499, prio=1 smokey-2499 [000] *.~3 61513.834849: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 gdb-2495 [000] *.~3 61513.834850: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] d.h3 61513.834851: irq_handler_entry: irq=120 name=in-band work gdb-2495 [000] d.h3 61513.834852: cobalt_lostage_wakeup: pid=2499 comm=smokey gdb-2495 [000] d.h5 61513.834852: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dNh6 61513.834853: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dNh3 61513.834853: irq_handler_exit: irq=120 ret=handled gdb-2495 [000] dN.3 61513.834854: sched_stat_runtime: comm=gdb pid=2495 runtime=23565 [ns] vruntime=300514015543 [ns] gdb-2495 [000] d..3 61513.834855: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] ...1 61513.834856: cobalt_shadow_relaxed: state=0x480c0 info=0x0 smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min(policy: 1) smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min -> 0x1 smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max(policy: 1) smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max -> 0x63 smokey-2499 [000] ..~1 61513.834862: cobalt_head_sysentry: syscall=thread_getschedparam_ex smokey-2499 [000] ...1 61513.834862: cobalt_root_sysentry: syscall=thread_getschedparam_ex smokey-2499 [000] ...1 61513.834863: cobalt_pthread_getschedparam: pth=000000008d36be1c policy=fifo param={ priority=1 } smokey-2499 [000] ...1 61513.834864: cobalt_root_sysexit: result=0 smokey-2499 [000] ...1 61513.834872: sys_mmap(addr: 0, len: 11000, prot: 0, flags: 20022, fd: ffffffff, off: 0) smokey-2499 [000] ...1 61513.834877: sys_mmap -> 0x7ffff67b9000 smokey-2499 [000] ...1 61513.834878: sys_mprotect(start: 7ffff67ba000, len: 10000, prot: 3) smokey-2499 [000] ...1 61513.834906: sys_mprotect -> 0x0 smokey-2499 [000] ...1 61513.834909: sys_clone(clone_flags: 3d0f00, newsp: 7ffff67c8fb0, parent_tidptr: 7ffff67c99d0, child_tidptr: 7ffff67c99d0, tls: 7ffff67c9700) smokey-2499 [000] ...1 61513.834923: task_newtask: pid=2504 comm=smokey clone_flags=3d0f00 oom_score_adj=0 smokey-2499 [000] ...1 61513.834923: sched_process_fork: comm=smokey pid=2499 child_comm=smokey child_pid=2504 smokey-2499 [000] d..3 61513.834925: sched_wakeup_new: comm=smokey pid=2504 prio=98 target_cpu=000 smokey-2499 [000] d..3 61513.834928: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0 smokey-2499 [000] d..3 61513.834931: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=t ==> next_comm=smokey next_pid=2504 next_prio=98 hi-thread-2504 [000] ...1 61513.834932: sys_clone -> 0x0 hi-thread-2504 [000] d..3 61513.834934: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=2 hi-thread-2504 [000] d..3 61513.834935: sched_stat_wait: comm=gdb pid=2495 delay=82002 [ns] hi-thread-2504 [000] d..3 61513.834936: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=t ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] d..2 61513.834937: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000 gdb-2495 [000] ...1 61513.834940: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1) gdb-2495 [000] ...1 61513.834943: sys_read -> 0x1 gdb-2495 [000] ...1 61513.834943: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1) gdb-2495 [000] ...1 61513.834944: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.834944: sys_write(fd: 10, buf: 5585fe0354b1, count: 1) gdb-2495 [000] ...1 61513.834946: sys_write -> 0x1 gdb-2495 [000] ...1 61513.834947: sys_rt_sigreturn() gdb-2495 [000] ...1 61513.834953: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1) gdb-2495 [000] ...1 61513.834953: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.834955: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0) gdb-2495 [000] ...1 61513.834956: sys_poll -> 0x1 gdb-2495 [000] ...1 61513.834957: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) gdb-2495 [000] ...1 61513.834958: sys_read -> 0x1 gdb-2495 [000] ...1 61513.834959: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) gdb-2495 [000] ...1 61513.834960: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8) gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask -> 0x0 gdb-2495 [000] ...1 61513.834962: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) gdb-2495 [000] ...1 61513.834962: sched_process_wait: comm=gdb pid=0 prio=120 gdb-2495 [000] ...1 61513.834964: sys_wait4 -> 0x9c3 gdb-2495 [000] ...1 61513.834965: sys_ptrace(request: 4201, pid: 9c3, addr: 0, data: 7ffe8b2a7d08) gdb-2495 [000] d..3 61513.834966: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.834967: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.834967: sys_wait4(upid: 9c8, stat_addr: 7ffe8b2a7cfc, options: 40000000, ru: 0) gdb-2495 [000] ...1 61513.834968: sched_process_wait: comm=gdb pid=2504 prio=120 gdb-2495 [000] ...1 61513.834969: sys_wait4 -> 0x9c8 gdb-2495 [000] ...1 61513.834972: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578) gdb-2495 [000] d..3 61513.834972: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.834975: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.834976: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a78d8) gdb-2495 [000] d..3 61513.834977: sched_wait_task: comm=smokey pid=2504 prio=98 gdb-2495 [000] ...1 61513.834977: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.834979: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.834989: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.834990: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff67c9700) gdb-2495 [000] ...1 61513.834992: sys_pread64 -> 0x900 gdb-2495 [000] ...1 61513.834993: sys_close(fd: 12) gdb-2495 [000] ...1 61513.834994: sys_close -> 0x0 gdb-2495 [000] ...1 61513.835010: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578) gdb-2495 [000] d..3 61513.835011: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.835012: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835013: sys_ptrace(request: 3, pid: 9c3, addr: a8, data: 7ffe8b2a78d8) gdb-2495 [000] d..3 61513.835013: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.835014: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835015: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.835022: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.835022: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff758f740) gdb-2495 [000] ...1 61513.835025: sys_pread64 -> 0x900 gdb-2495 [000] ...1 61513.835025: sys_close(fd: 12) gdb-2495 [000] ...1 61513.835026: sys_close -> 0x0 gdb-2495 [000] ...1 61513.835031: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) gdb-2495 [000] ...1 61513.835032: sched_process_wait: comm=gdb pid=0 prio=120 gdb-2495 [000] ...1 61513.835033: sys_wait4 -> 0x0 gdb-2495 [000] ...1 61513.835036: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0) gdb-2495 [000] d..3 61513.835037: sched_wait_task: comm=smokey pid=2504 prio=98 gdb-2495 [000] ...1 61513.835039: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835043: sys_ptrace(request: 6, pid: 9c8, addr: 388, data: 0) gdb-2495 [000] d..3 61513.835044: sched_wait_task: comm=smokey pid=2504 prio=98 gdb-2495 [000] ...1 61513.835044: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835045: sys_ptrace(request: 7, pid: 9c8, addr: 1, data: 0) gdb-2495 [000] d..3 61513.835046: sched_wait_task: comm=smokey pid=2504 prio=98 gdb-2495 [000] d..3 61513.835047: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000 gdb-2495 [000] dN.4 61513.835048: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000 gdb-2495 [000] .N.1 61513.835048: sys_ptrace -> 0x0 gdb-2495 [000] dN.3 61513.835050: sched_stat_runtime: comm=gdb pid=2495 runtime=112561 [ns] vruntime=300514128104 [ns] gdb-2495 [000] d..3 61513.835051: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=98 hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list(head: 7ffff67c99e0, len: 18) hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list -> 0x0 hi-thread-2504 [000] ...1 61513.835056: sys_futex(uaddr: 7ffff67c9d18, op: 80, val: 2, utime: 0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700) hi-thread-2504 [000] d..3 61513.835059: sched_stat_wait: comm=gdb pid=2495 delay=10883 [ns] hi-thread-2504 [000] d..3 61513.835060: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] ...1 61513.835063: sys_ptrace(request: c, pid: 9c3, addr: 0, data: 7ffe8b2a6ef0) gdb-2495 [000] d..3 61513.835064: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.835065: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835068: sys_ptrace(request: 7, pid: 9c3, addr: 1, data: 0) gdb-2495 [000] d..3 61513.835069: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] d..3 61513.835069: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dN.4 61513.835070: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] .N.1 61513.835071: sys_ptrace -> 0x0 gdb-2495 [000] dN.3 61513.835071: sched_stat_runtime: comm=gdb pid=2495 runtime=12210 [ns] vruntime=300514140314 [ns] gdb-2495 [000] d..3 61513.835072: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0 smokey-2499 [000] ...1 61513.835074: sys_clone -> 0x9c8 smokey-2499 [000] ...1 61513.835075: sys_sched_setaffinity(pid: 9c8, len: 80, user_mask_ptr: 55555563e7c0) smokey-2499 [000] ...1 61513.835077: sys_sched_setaffinity -> 0x0 smokey-2499 [000] ...1 61513.835077: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 0, val3: 7ffff67c9700) smokey-2499 [000] d..3 61513.835078: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000 smokey-2499 [000] d..4 61513.835079: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000 smokey-2499 [000] ...1 61513.835080: sys_futex -> 0x1 smokey-2499 [000] ...1 61513.835086: sys_futex(uaddr: 7fffffffe828, op: 189, val: 0, utime: 7fffffffe780, uaddr2: 0, val3: ffffffff) smokey-2499 [000] d..3 61513.835089: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=smokey next_pid=2504 next_prio=98 hi-thread-2504 [000] ...1 61513.835090: sys_futex -> 0x0 hi-thread-2504 [000] ...1 61513.835090: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700) hi-thread-2504 [000] ...1 61513.835091: sys_futex -> 0x0 hi-thread-2504 [000] ...1 61513.835092: sys_sched_setscheduler(pid: 9c8, policy: 1, param: 7ffff67c8e70) hi-thread-2504 [000] ...1 61513.835095: sys_sched_setscheduler -> 0x0 hi-thread-2504 [000] ..~1 61513.835096: cobalt_head_sysentry: syscall=thread_create hi-thread-2504 [000] ...1 61513.835097: cobalt_root_sysentry: syscall=thread_create hi-thread-2504 [000] ...1 61513.835098: cobalt_pthread_create: pth=00000000ba35a01d policy=fifo param={ priority=2 } hi-thread-2504 [000] ...1 61513.835099: cobalt_timer_migrate: timer=000000004d21df54 cpu=0 hi-thread-2504 [000] ...1 61513.835100: cobalt_timer_migrate: timer=00000000c1301853 cpu=0 hi-thread-2504 [000] ...1 61513.835101: cobalt_thread_set_current_prio: thread=00000000e9fda3c4 pid=-1 prio=2 hi-thread-2504 [000] ...1 61513.835102: cobalt_thread_init: thread=00000000e9fda3c4 name=smokey flags=0x48000 class=rt prio=2 hi-thread-2504 [000] ...1 61513.835104: cobalt_shadow_map: thread=00000000e9fda3c4 pid=2504 prio=2 hi-thread-2504 [000] D..1 61513.835104: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_start: pid=2504 state=0x480d0 info=0x0 hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_resume: name=smokey pid=2504 mask=0x10 hi-thread-2504 [000] D..1 61513.835106: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] ...1 61513.835106: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] ...1 61513.835107: cobalt_shadow_gohard: state=0x480c0 info=0x0 hi-thread-2504 [000] d..3 61513.835108: sched_stat_wait: comm=gdb pid=2495 delay=37231 [ns] hi-thread-2504 [000] d..3 61513.835109: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] *..3 61513.835109: cobalt_thread_resume: name=smokey pid=2504 mask=0x80 gdb-2495 [000] *..3 61513.835110: cobalt_trace_pid: pid=2504, prio=2 gdb-2495 [000] *.~3 61513.835110: cobalt_schedule: status=0x10000000 gdb-2495 [000] *.~3 61513.835110: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] *.~3 61513.835111: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2 hi-thread-2504 [000] d.~3 61513.835112: cobalt_shadow_hardened: state=0x48040 info=0x0 hi-thread-2504 [000] d.~3 61513.835112: cobalt_root_sysexit: result=0 hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysentry: syscall=get_current hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysexit: result=0 hi-thread-2504 [000] d.~3 61513.835116: cobalt_shadow_gorelax: reason=syscall hi-thread-2504 [000] d.~3 61513.835117: cobalt_lostage_request: request=wakeup pid=2504 comm=smokey hi-thread-2504 [000] *.~3 61513.835117: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] *.~3 61513.835117: cobalt_schedule: status=0x10000000 hi-thread-2504 [000] *.~3 61513.835118: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] *.~3 61513.835118: cobalt_switch_context: prev_name=smokey prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 gdb-2495 [000] *.~3 61513.835119: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] d.h3 61513.835120: irq_handler_entry: irq=120 name=in-band work gdb-2495 [000] d.h3 61513.835120: cobalt_lostage_wakeup: pid=2504 comm=smokey gdb-2495 [000] d.h5 61513.835120: sched_waking: comm=smokey pid=2504 prio=97 target_cpu=000 gdb-2495 [000] dNh6 61513.835121: sched_wakeup: comm=smokey pid=2504 prio=97 target_cpu=000 gdb-2495 [000] dNh3 61513.835122: irq_handler_exit: irq=120 ret=handled gdb-2495 [000] dN.3 61513.835122: sched_stat_runtime: comm=gdb pid=2495 runtime=13441 [ns] vruntime=300514153755 [ns] gdb-2495 [000] d..3 61513.835123: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=97 hi-thread-2504 [000] ...1 61513.835124: cobalt_shadow_relaxed: state=0x480c0 info=0x0 hi-thread-2504 [000] ...1 61513.835125: sys_futex(uaddr: 7fffffffe828, op: 81, val: 1, utime: 0, uaddr2: 7ffff67c8e74, val3: 7fffffffe828) hi-thread-2504 [000] d..3 61513.835126: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 hi-thread-2504 [000] d..4 61513.835126: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 hi-thread-2504 [000] ...1 61513.835127: sys_futex -> 0x1 hi-thread-2504 [000] ..~1 61513.835128: cobalt_head_sysentry: syscall=migrate hi-thread-2504 [000] ...1 61513.835128: cobalt_root_sysentry: syscall=migrate hi-thread-2504 [000] ...1 61513.835128: cobalt_shadow_gohard: state=0x480c0 info=0x0 hi-thread-2504 [000] d..3 61513.835130: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] *..3 61513.835130: cobalt_thread_resume: name=smokey pid=2504 mask=0x80 smokey-2499 [000] *..3 61513.835130: cobalt_trace_pid: pid=2504, prio=2 smokey-2499 [000] *.~3 61513.835131: cobalt_schedule: status=0x10000000 smokey-2499 [000] *.~3 61513.835131: cobalt_trace_pid: pid=2499, prio=-1 smokey-2499 [000] *.~3 61513.835131: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2 hi-thread-2504 [000] d.~3 61513.835132: cobalt_shadow_hardened: state=0x48040 info=0x0 hi-thread-2504 [000] d.~3 61513.835132: cobalt_root_sysexit: result=1 hi-thread-2504 [000] d.~3 61513.835133: cobalt_head_sysentry: syscall=thread_setname hi-thread-2504 [000] d.~3 61513.835134: cobalt_pthread_setname: pth=00000000ba35a01d name=hi-thread hi-thread-2504 [000] d.~3 61513.835135: cobalt_head_sysexit: result=0 hi-thread-2504 [000] d.~3 61513.835136: cobalt_head_sysentry: syscall=clock_nanosleep hi-thread-2504 [000] d.~3 61513.835137: cobalt_clock_nanosleep: clock_id=0 flags=0x0() rqt=(0.001000000) hi-thread-2504 [000] *.~3 61513.835138: cobalt_thread_suspend: pid=2504 mask=0x4 timeout=1000001 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] *.~3 61513.835138: cobalt_timer_start: timer=000000004d21df54(smokey) value=1000001 interval=0 mode=rel hi-thread-2504 [000] *.~3 61513.835138: cobalt_schedule: status=0x10000000 hi-thread-2504 [000] *.~3 61513.835139: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] *.~3 61513.835139: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x48044 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 smokey-2499 [000] *.~3 61513.835140: cobalt_trace_pid: pid=2499, prio=-1 smokey-2499 [000] ...1 61513.835141: sys_futex -> 0x0 smokey-2499 [000] ..~1 61513.835144: cobalt_head_sysentry: syscall=migrate smokey-2499 [000] ...1 61513.835144: cobalt_root_sysentry: syscall=migrate smokey-2499 [000] ...1 61513.835144: cobalt_shadow_gohard: state=0x480c0 info=0x0 smokey-2499 [000] d..3 61513.835145: sched_stat_wait: comm=gdb pid=2495 delay=23982 [ns] smokey-2499 [000] d..3 61513.835146: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] *..3 61513.835147: cobalt_thread_resume: name=smokey pid=2499 mask=0x80 gdb-2495 [000] *..3 61513.835147: cobalt_trace_pid: pid=2499, prio=1 gdb-2495 [000] *.~3 61513.835147: cobalt_schedule: status=0x10000000 gdb-2495 [000] *.~3 61513.835147: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] *.~3 61513.835148: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1 smokey-2499 [000] d.~3 61513.835148: cobalt_shadow_hardened: state=0x48040 info=0x0 smokey-2499 [000] d.~3 61513.835149: cobalt_root_sysexit: result=1 smokey-2499 [000] *.~3 61513.835219: irq_pipeline_entry: irq=4354 smokey-2499 [000] *.~3 61513.835219: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt smokey-2499 [000] *.~3 61513.835220: cobalt_timer_expire: timer=00000000d13b8aa9 smokey-2499 [000] *.~3 61513.835220: cobalt_tick_shot: next tick at 61513.836136 (delay: 915 us) smokey-2499 [000] *.~3 61513.835221: irq_handler_exit: irq=4354 ret=handled smokey-2499 [000] *.~3 61513.835221: irq_pipeline_exit: irq=4354 smokey-2499 [000] *.~3 61513.836137: irq_pipeline_entry: irq=4354 smokey-2499 [000] *.~3 61513.836137: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt smokey-2499 [000] *.~3 61513.836138: cobalt_timer_expire: timer=000000004d21df54 smokey-2499 [000] *.~3 61513.836138: cobalt_thread_resume: name=hi-thread pid=2504 mask=0x4 smokey-2499 [000] *.~3 61513.836138: cobalt_trace_pid: pid=2504, prio=2 smokey-2499 [000] *.~3 61513.836139: cobalt_timer_stop: timer=000000004d21df54 smokey-2499 [000] *.~3 61513.836139: irq_handler_exit: irq=4354 ret=handled smokey-2499 [000] *.~3 61513.836139: irq_pipeline_exit: irq=4354 smokey-2499 [000] *.~3 61513.836140: cobalt_schedule: status=0x10000000 smokey-2499 [000] *.~3 61513.836140: cobalt_trace_pid: pid=2499, prio=1 smokey-2499 [000] *.~3 61513.836140: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x48048 ==> next_name=hi-thread next_pid=2504 next_prio=2 hi-thread-2504 [000] *.~3 61513.836141: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] d.~3 61513.836141: cobalt_head_sysexit: result=0 hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3 hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] d.~3 61513.836143: cobalt_shadow_gorelax: reason=fault hi-thread-2504 [000] d.~3 61513.836143: cobalt_lostage_request: request=wakeup pid=2504 comm=hi-thread hi-thread-2504 [000] *.~3 61513.836144: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] *.~3 61513.836144: cobalt_schedule: status=0x10000000 hi-thread-2504 [000] *.~3 61513.836144: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] *.~3 61513.836145: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 gdb-2495 [000] *.~3 61513.836146: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] d.h3 61513.836146: irq_handler_entry: irq=120 name=in-band work gdb-2495 [000] d.h3 61513.836147: cobalt_lostage_wakeup: pid=2504 comm=hi-thread gdb-2495 [000] d.h5 61513.836147: sched_waking: comm=hi-thread pid=2504 prio=97 target_cpu=000 gdb-2495 [000] dNh6 61513.836147: sched_wakeup: comm=hi-thread pid=2504 prio=97 target_cpu=000 gdb-2495 [000] dNh3 61513.836148: irq_handler_exit: irq=120 ret=handled gdb-2495 [000] dNh3 61513.836148: irq_handler_entry: irq=124 name=proxy tick gdb-2495 [000] dNh3 61513.836151: softirq_raise: vec=1 [action=TIMER] gdb-2495 [000] dNh3 61513.836152: softirq_raise: vec=9 [action=RCU] gdb-2495 [000] dNh4 61513.836152: sched_stat_runtime: comm=gdb pid=2495 runtime=1007474 [ns] vruntime=300515161229 [ns] gdb-2495 [000] *Nh3 61513.836154: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2395843 interval=0 mode=rel gdb-2495 [000] *Nh3 61513.836155: cobalt_tick_shot: next tick at 61513.838550 (delay: 2395 us) gdb-2495 [000] dNh3 61513.836155: irq_handler_exit: irq=124 ret=handled gdb-2495 [000] .Ns3 61513.836156: softirq_entry: vec=1 [action=TIMER] gdb-2495 [000] .Ns3 61513.836157: softirq_exit: vec=1 [action=TIMER] gdb-2495 [000] .Ns3 61513.836157: softirq_entry: vec=9 [action=RCU] gdb-2495 [000] dNs6 61513.836158: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000 gdb-2495 [000] dNs7 61513.836159: sched_stat_runtime: comm=gdb pid=2495 runtime=6881 [ns] vruntime=300515168110 [ns] gdb-2495 [000] dNs7 61513.836160: sched_stat_blocked: comm=rcu_sched pid=11 delay=4188739 [ns] gdb-2495 [000] dNs7 61513.836161: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000 gdb-2495 [000] .Ns3 61513.836164: softirq_exit: vec=9 [action=RCU] gdb-2495 [000] d..3 61513.836165: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=hi-thread next_pid=2504 next_prio=97 hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200 hi-thread-2504 [000] *..2 61513.836169: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] *..2 61513.836170: cobalt_thread_suspend: pid=2504 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] d..2 61513.836170: signal_generate: sig=5 errno=0 code=128 comm=hi-thread pid=2504 grp=0 res=0 hi-thread-2504 [000] d..3 61513.836173: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0 hi-thread-2504 [000] d..3 61513.836175: sched_stat_wait: comm=rcu_sched pid=11 delay=14490 [ns] hi-thread-2504 [000] d..3 61513.836175: sched_switch: prev_comm=hi-thread prev_pid=2504 prev_prio=97 prev_state=t ==> next_comm=rcu_sched next_pid=11 next_prio=120 rcu_sched-11 [000] d..3 61513.836179: sched_stat_runtime: comm=rcu_sched pid=11 runtime=4960 [ns] vruntime=300512173070 [ns] rcu_sched-11 [000] d..3 61513.836179: sched_stat_wait: comm=gdb pid=2495 delay=19450 [ns] rcu_sched-11 [000] d..3 61513.836180: sched_switch: prev_comm=rcu_sched prev_pid=11 prev_prio=120 prev_state=I ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] ...1 61513.836187: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cb0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836197: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836198: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7ae0) gdb-2495 [000] ...1 61513.836199: sys_newfstat -> 0x0 gdb-2495 [000] ...1 61513.836200: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) gdb-2495 [000] ...1 61513.836214: sys_read -> 0x3f1 gdb-2495 [000] ...1 61513.836217: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836217: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836221: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8) gdb-2495 [000] ...1 61513.836222: sys_rt_sigprocmask -> 0x0 gdb-2495 [000] d..2 61513.836223: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000 gdb-2495 [000] ...1 61513.836225: sys_read(fd: f, buf: 7ffe8b2a7667, count: 1) gdb-2495 [000] ...1 61513.836226: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.836227: sys_write(fd: 10, buf: 5585fe0354b1, count: 1) gdb-2495 [000] ...1 61513.836229: sys_write -> 0x1 gdb-2495 [000] ...1 61513.836229: sys_rt_sigreturn() gdb-2495 [000] ...1 61513.836233: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1) gdb-2495 [000] ...1 61513.836234: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.836235: sys_write(fd: 1, buf: 5585fe7a5290, count: 27) gdb-2495 [000] d..4 61513.836236: sched_waking: comm=smokey pid=2493 prio=120 target_cpu=000 gdb-2495 [000] d..5 61513.836237: sched_stat_runtime: comm=gdb pid=2495 runtime=58399 [ns] vruntime=300515226509 [ns] gdb-2495 [000] d..5 61513.836238: sched_stat_sleep: comm=smokey pid=2493 delay=1717179 [ns] gdb-2495 [000] dN.5 61513.836239: sched_wakeup: comm=smokey pid=2493 prio=120 target_cpu=000 gdb-2495 [000] .N.1 61513.836240: sys_write -> 0x27 gdb-2495 [000] dN.3 61513.836241: sched_stat_wait: comm=smokey pid=2493 delay=0 [ns] gdb-2495 [000] d..3 61513.836241: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2493 next_prio=120 smokey-2493 [000] ...1 61513.836244: sys_read -> 0x1 smokey-2493 [000] ..~1 61513.836246: cobalt_head_sysentry: syscall=read ... smokey-2493 [000] d..3 61513.836344: sched_stat_runtime: comm=smokey pid=2493 runtime=106636 [ns] vruntime=300512333145 [ns] smokey-2493 [000] d..3 61513.836344: sched_stat_wait: comm=gdb pid=2495 delay=106636 [ns] smokey-2493 [000] d..3 61513.836345: sched_switch: prev_comm=smokey prev_pid=2493 prev_prio=120 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] ...1 61513.836347: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0) gdb-2495 [000] ...1 61513.836348: sys_poll -> 0x1 gdb-2495 [000] ...1 61513.836350: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) gdb-2495 [000] ...1 61513.836351: sys_read -> 0x1 gdb-2495 [000] ...1 61513.836352: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) gdb-2495 [000] ...1 61513.836352: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.836353: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8) gdb-2495 [000] ...1 61513.836354: sys_rt_sigprocmask -> 0x0 gdb-2495 [000] ...1 61513.836355: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) gdb-2495 [000] ...1 61513.836355: sched_process_wait: comm=gdb pid=0 prio=120 gdb-2495 [000] ...1 61513.836357: sys_wait4 -> 0x9c8 gdb-2495 [000] ...1 61513.836360: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0) gdb-2495 [000] d..3 61513.836361: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836363: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836367: sys_ptrace(request: 4202, pid: 9c8, addr: 0, data: 7ffe8b2a7cb0) gdb-2495 [000] d..3 61513.836368: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836369: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836370: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6f10) gdb-2495 [000] d..3 61513.836370: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836371: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836372: sys_ptrace(request: d, pid: 9c8, addr: 0, data: 7ffe8b2a6f10) gdb-2495 [000] d..3 61513.836373: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836374: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836376: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) gdb-2495 [000] ...1 61513.836376: sched_process_wait: comm=gdb pid=0 prio=120 gdb-2495 [000] ...1 61513.836377: sys_wait4 -> 0x0 gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8) gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask -> 0x0 gdb-2495 [000] ...1 61513.836382: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cd0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836399: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836400: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7b50) gdb-2495 [000] ...1 61513.836401: sys_newfstat -> 0x0 gdb-2495 [000] ...1 61513.836402: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) gdb-2495 [000] ...1 61513.836409: sys_read -> 0x140 gdb-2495 [000] ...1 61513.836410: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400) gdb-2495 [000] ...1 61513.836410: sys_read -> 0x0 gdb-2495 [000] ...1 61513.836415: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836416: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836419: sys_read(fd: f, buf: 7ffe8b2a7f17, count: 1) gdb-2495 [000] ...1 61513.836420: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.836421: sys_write(fd: 10, buf: 5585fe0354b1, count: 1) gdb-2495 [000] ...1 61513.836422: sys_write -> 0x1 gdb-2495 [000] ...1 61513.836425: sys_ptrace(request: 1, pid: 9c8, addr: 7ffff777f370, data: 7ffe8b2a7878) gdb-2495 [000] d..3 61513.836426: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836428: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836430: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a7bd8) gdb-2495 [000] d..3 61513.836430: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836431: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836433: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a70e0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836446: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836446: sys_pread64(fd: 12, buf: 7ffe8b2a7430, count: 900, pos: 7ffff67c9700) gdb-2495 [000] ...1 61513.836449: sys_pread64 -> 0x900 gdb-2495 [000] ...1 61513.836450: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836450: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836479: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836488: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836489: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950) gdb-2495 [000] ...1 61513.836490: sys_newfstat -> 0x0 gdb-2495 [000] ...1 61513.836490: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400) gdb-2495 [000] ...1 61513.836498: sys_read -> 0x14d gdb-2495 [000] ...1 61513.836499: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) gdb-2495 [000] ...1 61513.836499: sys_read -> 0x0 gdb-2495 [000] ...1 61513.836503: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836504: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836507: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836514: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836515: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950) gdb-2495 [000] ...1 61513.836515: sys_newfstat -> 0x0 gdb-2495 [000] ...1 61513.836516: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) gdb-2495 [000] ...1 61513.836521: sys_read -> 0x13b gdb-2495 [000] ...1 61513.836522: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400) gdb-2495 [000] ...1 61513.836522: sys_read -> 0x0 gdb-2495 [000] ...1 61513.836525: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836525: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13) gdb-2495 [000] *..3 61513.836532: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000 gdb-2495 [000] *..3 61513.836532: cobalt_trace_pid: pid=2499, prio=1 gdb-2495 [000] *..3 61513.836533: cobalt_thread_unblock: pid=2499 state=0x48848 info=0x0 gdb-2495 [000] *.~3 61513.836586: cobalt_schedule: status=0x10000000 gdb-2495 [000] *.~3 61513.836586: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] *.~3 61513.836587: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1 smokey-2499 [000] *.~3 61513.836589: cobalt_trace_pid: pid=2499, prio=1 smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined smokey-2499 [000] *.~3 61513.836589: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey smokey-2499 [000] *.~3 61513.836590: cobalt_thread_suspend: pid=2499 mask=0x400080 timeout=0 timeout_mode=0 wchan=0000000000000000 smokey-2499 [000] *.~3 61513.836590: cobalt_schedule: status=0x10000000 smokey-2499 [000] *.~3 61513.836590: cobalt_trace_pid: pid=2499, prio=1 smokey-2499 [000] *.~3 61513.836591: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x4488c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 gdb-2495 [000] *.~3 61513.836592: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] d..5 61513.836592: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dN.6 61513.836594: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dN.3 61513.836594: signal_generate: sig=19 errno=0 code=-6 comm=smokey pid=2499 grp=0 res=0 gdb-2495 [000] dNh3 61513.836595: irq_handler_entry: irq=120 name=in-band work gdb-2495 [000] dNh3 61513.836596: cobalt_lostage_wakeup: pid=2499 comm=smokey gdb-2495 [000] *N~4 61513.838552: irq_pipeline_entry: irq=4354 gdb-2495 [000] *N~4 61513.838553: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt gdb-2495 [000] *N~4 61513.838554: cobalt_timer_expire: timer=00000000d13b8aa9 gdb-2495 [000] *N~4 61513.838555: irq_handler_exit: irq=4354 ret=handled gdb-2495 [000] *N~4 61513.838555: irq_pipeline_exit: irq=4354 gdb-2495 [000] dNh8 61513.981277: sched_waking: comm=systemd-journal pid=163 prio=120 target_cpu=000 gdb-2495 [000] dNh9 61513.981278: sched_stat_runtime: comm=gdb pid=2495 runtime=144934239 [ns] vruntime=300660160748 [ns] gdb-2495 [000] dNh9 61513.981280: sched_stat_sleep: comm=systemd-journal pid=163 delay=111416392895 [ns] gdb-2495 [000] dNh9 61513.981280: sched_wakeup: comm=systemd-journal pid=163 prio=120 target_cpu=000 gdb-2495 [000] dNh3 61513.981281: irq_handler_exit: irq=120 ret=handled gdb-2495 [000] dNh3 61513.981282: irq_handler_entry: irq=124 name=proxy tick gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=1 [action=TIMER] gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=9 [action=RCU] gdb-2495 [000] dNh4 61513.981286: sched_stat_runtime: comm=gdb pid=2495 runtime=8375 [ns] vruntime=300660169123 [ns] gdb-2495 [000] dNh5 61513.981288: sched_waking: comm=cobalt_printf pid=2494 prio=120 target_cpu=000 gdb-2495 [000] dNh6 61513.981289: sched_stat_runtime: comm=gdb pid=2495 runtime=2987 [ns] vruntime=300660172110 [ns] gdb-2495 [000] dNh6 61513.981291: sched_stat_sleep: comm=cobalt_printf pid=2494 delay=238660319 [ns] gdb-2495 [000] dNh6 61513.981291: sched_wakeup: comm=cobalt_printf pid=2494 prio=120 target_cpu=000 gdb-2495 [000] dNh5 61513.981292: sched_waking: comm=cobalt_printf pid=2503 prio=120 target_cpu=000 gdb-2495 [000] dNh6 61513.981292: sched_stat_runtime: comm=gdb pid=2495 runtime=3069 [ns] vruntime=300660175179 [ns] gdb-2495 [000] dNh6 61513.981294: sched_stat_sleep: comm=cobalt_printf pid=2503 delay=146611442 [ns] gdb-2495 [000] dNh6 61513.981294: sched_wakeup: comm=cobalt_printf pid=2503 prio=120 target_cpu=000 gdb-2495 [000] *Nh3 61513.981295: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=588476 interval=0 mode=rel gdb-2495 [000] *Nh3 61513.981296: cobalt_tick_shot: next tick at 61513.981884 (delay: 588 us) gdb-2495 [000] dNh3 61513.981296: irq_handler_exit: irq=124 ret=handled gdb-2495 [000] .Ns3 61513.981297: softirq_entry: vec=1 [action=TIMER] gdb-2495 [000] dNs5 61513.981298: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000 gdb-2495 [000] dNs6 61513.981298: sched_stat_runtime: comm=gdb pid=2495 runtime=6218 [ns] vruntime=300660181397 [ns] gdb-2495 [000] dNs6 61513.981299: sched_stat_blocked: comm=rcu_sched pid=11 delay=145119923 [ns] gdb-2495 [000] dNs6 61513.981299: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000 gdb-2495 [000] dNs7 61513.981302: sched_waking: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000 gdb-2495 [000] dNs8 61513.981302: sched_stat_runtime: comm=gdb pid=2495 runtime=3884 [ns] vruntime=300660185281 [ns] gdb-2495 [000] dNs8 61513.981303: sched_stat_blocked: comm=kworker/0:1 pid=1932 delay=206048652 [ns] gdb-2495 [000] dNs8 61513.981304: sched_wakeup: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000 gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981312: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] .Ns3 61513.981314: softirq_exit: vec=1 [action=TIMER] gdb-2495 [000] .Ns3 61513.981314: softirq_entry: vec=9 [action=RCU] gdb-2495 [000] .Ns3 61513.981315: softirq_exit: vec=9 [action=RCU] gdb-2495 [000] dNs5 61513.981316: sched_waking: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000 gdb-2495 [000] dNs6 61513.981316: sched_stat_runtime: comm=gdb pid=2495 runtime=13841 [ns] vruntime=300660199122 [ns] gdb-2495 [000] dNs6 61513.981317: sched_stat_sleep: comm=ksoftirqd/0 pid=10 delay=149349034 [ns] gdb-2495 [000] dNs6 61513.981317: sched_wakeup: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000 gdb-2495 [000] .N.1 61513.981318: sys_tkill -> 0x0 gdb-2495 [000] d..3 61513.981320: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] *.~3 61513.981888: irq_pipeline_entry: irq=4354 smokey-2499 [000] *.~3 61513.981888: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt smokey-2499 [000] *.~3 61513.981889: cobalt_timer_expire: timer=00000000d13b8aa9 smokey-2499 [000] *.~3 61513.981889: irq_handler_exit: irq=4354 ret=handled smokey-2499 [000] *.~3 61513.981890: irq_pipeline_exit: irq=4354 smokey-2499 [000] d.h1 61514.106435: irq_handler_entry: irq=124 name=proxy tick smokey-2499 [000] *.h1 61514.106441: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2109534 interval=0 mode=rel smokey-2499 [000] *.h1 61514.106441: cobalt_tick_shot: next tick at 61514.108550 (delay: 2109 us) smokey-2499 [000] d.h1 61514.106442: irq_handler_exit: irq=124 ret=handled smokey-2499 [000] ...1 61514.106442: cobalt_shadow_relaxed: state=0x4488c0 info=0x200 smokey-2499 [000] *.~1 61514.108552: irq_pipeline_entry: irq=4354 Regards Hongzhan Chen > >-- >Philippe. ^ permalink raw reply related [flat|nested] 16+ messages in thread
* RE: gdb test failure debug status update 2021-04-29 6:31 ` Chen, Hongzhan @ 2021-04-30 5:25 ` Chen, Hongzhan 2021-04-30 5:51 ` Chen, Hongzhan 2021-04-30 7:34 ` Philippe Gerum 0 siblings, 2 replies; 16+ messages in thread From: Chen, Hongzhan @ 2021-04-30 5:25 UTC (permalink / raw) To: Chen, Hongzhan, Philippe Gerum, xenomai The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax. That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode because kernel exit code is never called in this case? Regards Hongzhan Chen -----Original Message----- From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai Sent: Thursday, April 29, 2021 2:31 PM To: Philippe Gerum <rpm@xenomai.org> Cc: xenomai@xenomai.org Subject: RE: gdb test failure debug status update > >-----Original Message----- >From: Philippe Gerum <rpm@xenomai.org> >Sent: Wednesday, April 28, 2021 10:30 PM >To: Philippe Gerum <rpm@xenomai.org> >Cc: Chen, Hongzhan <hongzhan.chen@intel.com>; xenomai@xenomai.org >Subject: Re: gdb test failure debug status update > > >Philippe Gerum via Xenomai <xenomai@xenomai.org> writes: > >> Chen, Hongzhan via Xenomai <xenomai@xenomai.org> writes: >> >>> According to my validation, gdb test fail on dovetail 5.10 branch but pass on v5.9-evl4 tag with same for-upstream/dovetail >>> xenomai code base. >>> >>> After further debug , the issue is more clear for me. Gdb test failure because low priority thread smokey userspace is still >>> executed after "cobalt_shadow_relaxed: state=0x4488c0 info=0x200" like log [1] on dovetail-5.10 branch. >>> The weird thing is that its following first ftrace log happen at 62235.848583 after cobalt_shadow_relaxed in log [1]. >>> It is almost 3ms happened after cobalt_shadow_relaxed. The low priority smoke thread user space is executed during this >>> 3ms period so that test fail. >>> >>> But in success case with v5.9-evl4 like in log [2], the time interval between cobalt_shadow_relaxed and the following first ftrace log >>> is only about 1us. It seems that low priority smokey userspace do not have chance to execute in this 1us because gdb test is successful. >>> >>> My question is why there is even no interrupt happened during that about 3ms period in failure case? Tick seems in abnormal behavior. >>> Please comment if you have any ideas to further debug it. >>> >>> PS: All my tests run on same up Xtream board. >> >> <snip> >> >> Let's put aside the tick issue for now, there may be a valid reason for >> this delay with dynticks enabled. >> >> The issue at stake may be related to the way a return to kernel space is >> forced on a @user task (Dovetail has an integrated service for >> triggering this called dovetail_request_ucall()). >> >> The logic for doing so is as follows: >> >> 1. @user hits a breakpoint, which is an exception Dovetail-wise >> >> 2. @user gets XNDBGSTOP set into its flags because Cobalt notices it is >> being debugged via a breakpoint trap, then relaxed as a result of taking >> a exception in general, so that we may traverse the common trap handling code >> safely. >> >> 3. since XNDBGSTOP is a blocking bit Cobalt-wise, it should prevent >> @user from being picked for scheduling by the real-time core, next time >> a Cobalt considers rescheduling that is. However, since @user is >> currently relaxed, it can still run under the supervision of the common >> Linux scheduler. This is what the log[1] show. >> >> 4. the common/in-band kernel code stops @user due to the ptrace stop >> condition caused by the breakpoint, waiting for a continuation event to >> happen. >> >> Therefore, upon PTRACE_CONT (i.e. gdb continue), we need to force @user >> to call back into kernel context (handle_ptrace_cont -> >> dovetail_request_ucall), then ask for a switch to primary mode from >> there, which should eventually happen when @user is about to leave the >> kernel (on x86, this now happens from a generic kernel entry/exit code >> in kernel/entry/*). As a result, handle_taskexit_event() runs, figures > >Not quite: >s,handle_taskexit_event,handle_user_return, There is trace_cobalt_shadow_unmap in handle_taskexit_event. I added trace_cobalt_handle_user_return and trace_cobalt_handle_ptrace_cont like patch [3] in log [4]. 1. In the log [4], the first cobalt_handle_ptrace_cont is followed by cobalt_handle_user_return that clear info flag XNCONTHI for smokey-2499 because cobalt_handle_ptrace_cont called dovetail_request_ucall . the TIF_RETUSER bit was cleared for smokey-2499. After then 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000 clear state flag XNDBGSTOP. After that, thread smokey-2499 run without these bit set till "hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000". 2. There is second cobalt_handle_ptrace_cont called for smokey-2499 but at that time dovetail_request_ucall would not be called because XNSSTEP Is not set like "smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0" 3. hi-thread-2504 breakpoint at its first breakpoint_target so it trigger hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3 It call dovetail_request_ucall(current); for itself and also call xnthread_set_info(thread, XNCONTHI); That is why we can see this bit set in hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200 4. finally ,gdb call sys_tkill in gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13) and trigger that handle_oob_mayday is called so that xnthread_relax for smokey-2499 Is called in "smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined." It seems that there never call dovetail_request_ucall for process smokey-2499 in log[4] so that final xhthread_relax would have chance to return to user? The second handle_ptrace_cont happen too much earlier because XNSSTEP still have not been set so that system never have change to call dovetail_request_ucall for smokey-2499. In this case , do we need to do hack in 3 , when handle_oob_trap_entry is called for hi-thread-2504 we call xnthread_set_info(thread, XNCONTHI); and dovetail_request_ucall for all threads that have same cobalt_process? Is this solution OK for fixing this case? [3]: index 07dcc7a63..cd76961f5 100644 --- a/kernel/cobalt/dovetail/kevents.c +++ b/kernel/cobalt/dovetail/kevents.c @@ -331,6 +331,8 @@ static int handle_user_return(struct task_struct *task) if (thread == NULL) return KEVENT_PROPAGATE; + trace_cobalt_handle_user_return(thread); + if (xnthread_test_info(thread, XNCONTHI)) { xnlock_get_irqsave(&nklock, s); xnthread_clear_info(thread, XNCONTHI); @@ -468,6 +470,8 @@ int handle_ptrace_resume(struct task_struct *tracee) if (thread == NULL) return KEVENT_PROPAGATE; + trace_cobalt_handle_ptrace_resume(thread); + if (xnthread_test_state(thread, XNSSTEP)) { xnlock_get_irqsave(&nklock, s); @@ -485,6 +489,8 @@ static void handle_ptrace_cont(void) struct xnthread *curr = xnthread_current(); spl_t s; + trace_cobalt_handle_ptrace_cont(curr); + xnlock_get_irqsave(&nklock, s); if (xnthread_test_state(curr, XNSSTEP)) { @@ -604,6 +610,7 @@ void resume_oob_task(struct task_struct *p) /* inband, oob stage stalled */ if (unlikely(xnthread_test_info(thread, XNCONTHI))) { xnthread_clear_info(thread, XNCONTHI); dovetail_clear_ucall(); + trace_cobalt_resume_oob_task(thread); } /* Unregister as debugged thread in case we postponed this. */ diff --git a/kernel/cobalt/trace/cobalt-core.h b/kernel/cobalt/trace/cobalt-core.h index 78f66a106..88a4067fd 100644 --- a/kernel/cobalt/trace/cobalt-core.h +++ b/kernel/cobalt/trace/cobalt-core.h @@ -459,6 +459,27 @@ DEFINE_EVENT(curr_thread_event, cobalt_thread_set_mode, TP_ARGS(thread) ); +DEFINE_EVENT(thread_event, cobalt_handle_ptrace_cont, + TP_PROTO(struct xnthread *thread), [4]: gdb-2495 [000] d..3 61513.834825: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] ...1 61513.834827: cobalt_handle_ptrace_cont: pid=2499 state=0x4488c0 info=0x200 smokey-2499 [000] d..1 61513.834828: cobalt_handle_user_return: pid=2499 state=0x4488c0 info=0x200 smokey-2499 [000] d..1 61513.834828: cobalt_shadow_gohard: state=0x4488c0 info=0x0 smokey-2499 [000] d..3 61513.834830: sched_stat_wait: comm=gdb pid=2495 delay=8017 [ns] smokey-2499 [000] d..3 61513.834831: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] *..3 61513.834831: cobalt_thread_resume: name=smokey pid=2499 mask=0x80 gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1 gdb-2495 [000] *..3 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000 gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1 gdb-2495 [000] *.~3 61513.834833: cobalt_schedule: status=0x10000000 gdb-2495 [000] *.~3 61513.834833: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] *.~3 61513.834833: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1 smokey-2499 [000] d.~3 61513.834835: cobalt_shadow_hardened: state=0x48040 info=0x0 smokey-2499 [000] d.~3 61513.834836: cobalt_head_sysentry: syscall=sem_init smokey-2499 [000] d.~3 61513.834839: cobalt_psem_init: sem=0x4e(anon) flags=0x0() value=0 smokey-2499 [000] d.~3 61513.834839: cobalt_head_sysexit: result=0 smokey-2499 [000] d.~3 61513.834848: cobalt_shadow_gorelax: reason=syscall smokey-2499 [000] d.~3 61513.834848: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey smokey-2499 [000] *.~3 61513.834848: cobalt_thread_suspend: pid=2499 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 smokey-2499 [000] *.~3 61513.834849: cobalt_schedule: status=0x10000000 smokey-2499 [000] *.~3 61513.834849: cobalt_trace_pid: pid=2499, prio=1 smokey-2499 [000] *.~3 61513.834849: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 gdb-2495 [000] *.~3 61513.834850: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] d.h3 61513.834851: irq_handler_entry: irq=120 name=in-band work gdb-2495 [000] d.h3 61513.834852: cobalt_lostage_wakeup: pid=2499 comm=smokey gdb-2495 [000] d.h5 61513.834852: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dNh6 61513.834853: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dNh3 61513.834853: irq_handler_exit: irq=120 ret=handled gdb-2495 [000] dN.3 61513.834854: sched_stat_runtime: comm=gdb pid=2495 runtime=23565 [ns] vruntime=300514015543 [ns] gdb-2495 [000] d..3 61513.834855: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] ...1 61513.834856: cobalt_shadow_relaxed: state=0x480c0 info=0x0 smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min(policy: 1) smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min -> 0x1 smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max(policy: 1) smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max -> 0x63 smokey-2499 [000] ..~1 61513.834862: cobalt_head_sysentry: syscall=thread_getschedparam_ex smokey-2499 [000] ...1 61513.834862: cobalt_root_sysentry: syscall=thread_getschedparam_ex smokey-2499 [000] ...1 61513.834863: cobalt_pthread_getschedparam: pth=000000008d36be1c policy=fifo param={ priority=1 } smokey-2499 [000] ...1 61513.834864: cobalt_root_sysexit: result=0 smokey-2499 [000] ...1 61513.834872: sys_mmap(addr: 0, len: 11000, prot: 0, flags: 20022, fd: ffffffff, off: 0) smokey-2499 [000] ...1 61513.834877: sys_mmap -> 0x7ffff67b9000 smokey-2499 [000] ...1 61513.834878: sys_mprotect(start: 7ffff67ba000, len: 10000, prot: 3) smokey-2499 [000] ...1 61513.834906: sys_mprotect -> 0x0 smokey-2499 [000] ...1 61513.834909: sys_clone(clone_flags: 3d0f00, newsp: 7ffff67c8fb0, parent_tidptr: 7ffff67c99d0, child_tidptr: 7ffff67c99d0, tls: 7ffff67c9700) smokey-2499 [000] ...1 61513.834923: task_newtask: pid=2504 comm=smokey clone_flags=3d0f00 oom_score_adj=0 smokey-2499 [000] ...1 61513.834923: sched_process_fork: comm=smokey pid=2499 child_comm=smokey child_pid=2504 smokey-2499 [000] d..3 61513.834925: sched_wakeup_new: comm=smokey pid=2504 prio=98 target_cpu=000 smokey-2499 [000] d..3 61513.834928: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0 smokey-2499 [000] d..3 61513.834931: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=t ==> next_comm=smokey next_pid=2504 next_prio=98 hi-thread-2504 [000] ...1 61513.834932: sys_clone -> 0x0 hi-thread-2504 [000] d..3 61513.834934: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=2 hi-thread-2504 [000] d..3 61513.834935: sched_stat_wait: comm=gdb pid=2495 delay=82002 [ns] hi-thread-2504 [000] d..3 61513.834936: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=t ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] d..2 61513.834937: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000 gdb-2495 [000] ...1 61513.834940: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1) gdb-2495 [000] ...1 61513.834943: sys_read -> 0x1 gdb-2495 [000] ...1 61513.834943: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1) gdb-2495 [000] ...1 61513.834944: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.834944: sys_write(fd: 10, buf: 5585fe0354b1, count: 1) gdb-2495 [000] ...1 61513.834946: sys_write -> 0x1 gdb-2495 [000] ...1 61513.834947: sys_rt_sigreturn() gdb-2495 [000] ...1 61513.834953: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1) gdb-2495 [000] ...1 61513.834953: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.834955: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0) gdb-2495 [000] ...1 61513.834956: sys_poll -> 0x1 gdb-2495 [000] ...1 61513.834957: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) gdb-2495 [000] ...1 61513.834958: sys_read -> 0x1 gdb-2495 [000] ...1 61513.834959: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) gdb-2495 [000] ...1 61513.834960: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8) gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask -> 0x0 gdb-2495 [000] ...1 61513.834962: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) gdb-2495 [000] ...1 61513.834962: sched_process_wait: comm=gdb pid=0 prio=120 gdb-2495 [000] ...1 61513.834964: sys_wait4 -> 0x9c3 gdb-2495 [000] ...1 61513.834965: sys_ptrace(request: 4201, pid: 9c3, addr: 0, data: 7ffe8b2a7d08) gdb-2495 [000] d..3 61513.834966: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.834967: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.834967: sys_wait4(upid: 9c8, stat_addr: 7ffe8b2a7cfc, options: 40000000, ru: 0) gdb-2495 [000] ...1 61513.834968: sched_process_wait: comm=gdb pid=2504 prio=120 gdb-2495 [000] ...1 61513.834969: sys_wait4 -> 0x9c8 gdb-2495 [000] ...1 61513.834972: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578) gdb-2495 [000] d..3 61513.834972: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.834975: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.834976: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a78d8) gdb-2495 [000] d..3 61513.834977: sched_wait_task: comm=smokey pid=2504 prio=98 gdb-2495 [000] ...1 61513.834977: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.834979: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.834989: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.834990: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff67c9700) gdb-2495 [000] ...1 61513.834992: sys_pread64 -> 0x900 gdb-2495 [000] ...1 61513.834993: sys_close(fd: 12) gdb-2495 [000] ...1 61513.834994: sys_close -> 0x0 gdb-2495 [000] ...1 61513.835010: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578) gdb-2495 [000] d..3 61513.835011: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.835012: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835013: sys_ptrace(request: 3, pid: 9c3, addr: a8, data: 7ffe8b2a78d8) gdb-2495 [000] d..3 61513.835013: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.835014: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835015: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.835022: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.835022: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff758f740) gdb-2495 [000] ...1 61513.835025: sys_pread64 -> 0x900 gdb-2495 [000] ...1 61513.835025: sys_close(fd: 12) gdb-2495 [000] ...1 61513.835026: sys_close -> 0x0 gdb-2495 [000] ...1 61513.835031: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) gdb-2495 [000] ...1 61513.835032: sched_process_wait: comm=gdb pid=0 prio=120 gdb-2495 [000] ...1 61513.835033: sys_wait4 -> 0x0 gdb-2495 [000] ...1 61513.835036: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0) gdb-2495 [000] d..3 61513.835037: sched_wait_task: comm=smokey pid=2504 prio=98 gdb-2495 [000] ...1 61513.835039: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835043: sys_ptrace(request: 6, pid: 9c8, addr: 388, data: 0) gdb-2495 [000] d..3 61513.835044: sched_wait_task: comm=smokey pid=2504 prio=98 gdb-2495 [000] ...1 61513.835044: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835045: sys_ptrace(request: 7, pid: 9c8, addr: 1, data: 0) gdb-2495 [000] d..3 61513.835046: sched_wait_task: comm=smokey pid=2504 prio=98 gdb-2495 [000] d..3 61513.835047: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000 gdb-2495 [000] dN.4 61513.835048: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000 gdb-2495 [000] .N.1 61513.835048: sys_ptrace -> 0x0 gdb-2495 [000] dN.3 61513.835050: sched_stat_runtime: comm=gdb pid=2495 runtime=112561 [ns] vruntime=300514128104 [ns] gdb-2495 [000] d..3 61513.835051: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=98 hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list(head: 7ffff67c99e0, len: 18) hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list -> 0x0 hi-thread-2504 [000] ...1 61513.835056: sys_futex(uaddr: 7ffff67c9d18, op: 80, val: 2, utime: 0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700) hi-thread-2504 [000] d..3 61513.835059: sched_stat_wait: comm=gdb pid=2495 delay=10883 [ns] hi-thread-2504 [000] d..3 61513.835060: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] ...1 61513.835063: sys_ptrace(request: c, pid: 9c3, addr: 0, data: 7ffe8b2a6ef0) gdb-2495 [000] d..3 61513.835064: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.835065: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835068: sys_ptrace(request: 7, pid: 9c3, addr: 1, data: 0) gdb-2495 [000] d..3 61513.835069: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] d..3 61513.835069: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dN.4 61513.835070: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] .N.1 61513.835071: sys_ptrace -> 0x0 gdb-2495 [000] dN.3 61513.835071: sched_stat_runtime: comm=gdb pid=2495 runtime=12210 [ns] vruntime=300514140314 [ns] gdb-2495 [000] d..3 61513.835072: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0 smokey-2499 [000] ...1 61513.835074: sys_clone -> 0x9c8 smokey-2499 [000] ...1 61513.835075: sys_sched_setaffinity(pid: 9c8, len: 80, user_mask_ptr: 55555563e7c0) smokey-2499 [000] ...1 61513.835077: sys_sched_setaffinity -> 0x0 smokey-2499 [000] ...1 61513.835077: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 0, val3: 7ffff67c9700) smokey-2499 [000] d..3 61513.835078: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000 smokey-2499 [000] d..4 61513.835079: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000 smokey-2499 [000] ...1 61513.835080: sys_futex -> 0x1 smokey-2499 [000] ...1 61513.835086: sys_futex(uaddr: 7fffffffe828, op: 189, val: 0, utime: 7fffffffe780, uaddr2: 0, val3: ffffffff) smokey-2499 [000] d..3 61513.835089: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=smokey next_pid=2504 next_prio=98 hi-thread-2504 [000] ...1 61513.835090: sys_futex -> 0x0 hi-thread-2504 [000] ...1 61513.835090: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700) hi-thread-2504 [000] ...1 61513.835091: sys_futex -> 0x0 hi-thread-2504 [000] ...1 61513.835092: sys_sched_setscheduler(pid: 9c8, policy: 1, param: 7ffff67c8e70) hi-thread-2504 [000] ...1 61513.835095: sys_sched_setscheduler -> 0x0 hi-thread-2504 [000] ..~1 61513.835096: cobalt_head_sysentry: syscall=thread_create hi-thread-2504 [000] ...1 61513.835097: cobalt_root_sysentry: syscall=thread_create hi-thread-2504 [000] ...1 61513.835098: cobalt_pthread_create: pth=00000000ba35a01d policy=fifo param={ priority=2 } hi-thread-2504 [000] ...1 61513.835099: cobalt_timer_migrate: timer=000000004d21df54 cpu=0 hi-thread-2504 [000] ...1 61513.835100: cobalt_timer_migrate: timer=00000000c1301853 cpu=0 hi-thread-2504 [000] ...1 61513.835101: cobalt_thread_set_current_prio: thread=00000000e9fda3c4 pid=-1 prio=2 hi-thread-2504 [000] ...1 61513.835102: cobalt_thread_init: thread=00000000e9fda3c4 name=smokey flags=0x48000 class=rt prio=2 hi-thread-2504 [000] ...1 61513.835104: cobalt_shadow_map: thread=00000000e9fda3c4 pid=2504 prio=2 hi-thread-2504 [000] D..1 61513.835104: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_start: pid=2504 state=0x480d0 info=0x0 hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_resume: name=smokey pid=2504 mask=0x10 hi-thread-2504 [000] D..1 61513.835106: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] ...1 61513.835106: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] ...1 61513.835107: cobalt_shadow_gohard: state=0x480c0 info=0x0 hi-thread-2504 [000] d..3 61513.835108: sched_stat_wait: comm=gdb pid=2495 delay=37231 [ns] hi-thread-2504 [000] d..3 61513.835109: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] *..3 61513.835109: cobalt_thread_resume: name=smokey pid=2504 mask=0x80 gdb-2495 [000] *..3 61513.835110: cobalt_trace_pid: pid=2504, prio=2 gdb-2495 [000] *.~3 61513.835110: cobalt_schedule: status=0x10000000 gdb-2495 [000] *.~3 61513.835110: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] *.~3 61513.835111: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2 hi-thread-2504 [000] d.~3 61513.835112: cobalt_shadow_hardened: state=0x48040 info=0x0 hi-thread-2504 [000] d.~3 61513.835112: cobalt_root_sysexit: result=0 hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysentry: syscall=get_current hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysexit: result=0 hi-thread-2504 [000] d.~3 61513.835116: cobalt_shadow_gorelax: reason=syscall hi-thread-2504 [000] d.~3 61513.835117: cobalt_lostage_request: request=wakeup pid=2504 comm=smokey hi-thread-2504 [000] *.~3 61513.835117: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] *.~3 61513.835117: cobalt_schedule: status=0x10000000 hi-thread-2504 [000] *.~3 61513.835118: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] *.~3 61513.835118: cobalt_switch_context: prev_name=smokey prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 gdb-2495 [000] *.~3 61513.835119: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] d.h3 61513.835120: irq_handler_entry: irq=120 name=in-band work gdb-2495 [000] d.h3 61513.835120: cobalt_lostage_wakeup: pid=2504 comm=smokey gdb-2495 [000] d.h5 61513.835120: sched_waking: comm=smokey pid=2504 prio=97 target_cpu=000 gdb-2495 [000] dNh6 61513.835121: sched_wakeup: comm=smokey pid=2504 prio=97 target_cpu=000 gdb-2495 [000] dNh3 61513.835122: irq_handler_exit: irq=120 ret=handled gdb-2495 [000] dN.3 61513.835122: sched_stat_runtime: comm=gdb pid=2495 runtime=13441 [ns] vruntime=300514153755 [ns] gdb-2495 [000] d..3 61513.835123: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=97 hi-thread-2504 [000] ...1 61513.835124: cobalt_shadow_relaxed: state=0x480c0 info=0x0 hi-thread-2504 [000] ...1 61513.835125: sys_futex(uaddr: 7fffffffe828, op: 81, val: 1, utime: 0, uaddr2: 7ffff67c8e74, val3: 7fffffffe828) hi-thread-2504 [000] d..3 61513.835126: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 hi-thread-2504 [000] d..4 61513.835126: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 hi-thread-2504 [000] ...1 61513.835127: sys_futex -> 0x1 hi-thread-2504 [000] ..~1 61513.835128: cobalt_head_sysentry: syscall=migrate hi-thread-2504 [000] ...1 61513.835128: cobalt_root_sysentry: syscall=migrate hi-thread-2504 [000] ...1 61513.835128: cobalt_shadow_gohard: state=0x480c0 info=0x0 hi-thread-2504 [000] d..3 61513.835130: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] *..3 61513.835130: cobalt_thread_resume: name=smokey pid=2504 mask=0x80 smokey-2499 [000] *..3 61513.835130: cobalt_trace_pid: pid=2504, prio=2 smokey-2499 [000] *.~3 61513.835131: cobalt_schedule: status=0x10000000 smokey-2499 [000] *.~3 61513.835131: cobalt_trace_pid: pid=2499, prio=-1 smokey-2499 [000] *.~3 61513.835131: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2 hi-thread-2504 [000] d.~3 61513.835132: cobalt_shadow_hardened: state=0x48040 info=0x0 hi-thread-2504 [000] d.~3 61513.835132: cobalt_root_sysexit: result=1 hi-thread-2504 [000] d.~3 61513.835133: cobalt_head_sysentry: syscall=thread_setname hi-thread-2504 [000] d.~3 61513.835134: cobalt_pthread_setname: pth=00000000ba35a01d name=hi-thread hi-thread-2504 [000] d.~3 61513.835135: cobalt_head_sysexit: result=0 hi-thread-2504 [000] d.~3 61513.835136: cobalt_head_sysentry: syscall=clock_nanosleep hi-thread-2504 [000] d.~3 61513.835137: cobalt_clock_nanosleep: clock_id=0 flags=0x0() rqt=(0.001000000) hi-thread-2504 [000] *.~3 61513.835138: cobalt_thread_suspend: pid=2504 mask=0x4 timeout=1000001 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] *.~3 61513.835138: cobalt_timer_start: timer=000000004d21df54(smokey) value=1000001 interval=0 mode=rel hi-thread-2504 [000] *.~3 61513.835138: cobalt_schedule: status=0x10000000 hi-thread-2504 [000] *.~3 61513.835139: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] *.~3 61513.835139: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x48044 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 smokey-2499 [000] *.~3 61513.835140: cobalt_trace_pid: pid=2499, prio=-1 smokey-2499 [000] ...1 61513.835141: sys_futex -> 0x0 smokey-2499 [000] ..~1 61513.835144: cobalt_head_sysentry: syscall=migrate smokey-2499 [000] ...1 61513.835144: cobalt_root_sysentry: syscall=migrate smokey-2499 [000] ...1 61513.835144: cobalt_shadow_gohard: state=0x480c0 info=0x0 smokey-2499 [000] d..3 61513.835145: sched_stat_wait: comm=gdb pid=2495 delay=23982 [ns] smokey-2499 [000] d..3 61513.835146: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] *..3 61513.835147: cobalt_thread_resume: name=smokey pid=2499 mask=0x80 gdb-2495 [000] *..3 61513.835147: cobalt_trace_pid: pid=2499, prio=1 gdb-2495 [000] *.~3 61513.835147: cobalt_schedule: status=0x10000000 gdb-2495 [000] *.~3 61513.835147: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] *.~3 61513.835148: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1 smokey-2499 [000] d.~3 61513.835148: cobalt_shadow_hardened: state=0x48040 info=0x0 smokey-2499 [000] d.~3 61513.835149: cobalt_root_sysexit: result=1 smokey-2499 [000] *.~3 61513.835219: irq_pipeline_entry: irq=4354 smokey-2499 [000] *.~3 61513.835219: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt smokey-2499 [000] *.~3 61513.835220: cobalt_timer_expire: timer=00000000d13b8aa9 smokey-2499 [000] *.~3 61513.835220: cobalt_tick_shot: next tick at 61513.836136 (delay: 915 us) smokey-2499 [000] *.~3 61513.835221: irq_handler_exit: irq=4354 ret=handled smokey-2499 [000] *.~3 61513.835221: irq_pipeline_exit: irq=4354 smokey-2499 [000] *.~3 61513.836137: irq_pipeline_entry: irq=4354 smokey-2499 [000] *.~3 61513.836137: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt smokey-2499 [000] *.~3 61513.836138: cobalt_timer_expire: timer=000000004d21df54 smokey-2499 [000] *.~3 61513.836138: cobalt_thread_resume: name=hi-thread pid=2504 mask=0x4 smokey-2499 [000] *.~3 61513.836138: cobalt_trace_pid: pid=2504, prio=2 smokey-2499 [000] *.~3 61513.836139: cobalt_timer_stop: timer=000000004d21df54 smokey-2499 [000] *.~3 61513.836139: irq_handler_exit: irq=4354 ret=handled smokey-2499 [000] *.~3 61513.836139: irq_pipeline_exit: irq=4354 smokey-2499 [000] *.~3 61513.836140: cobalt_schedule: status=0x10000000 smokey-2499 [000] *.~3 61513.836140: cobalt_trace_pid: pid=2499, prio=1 smokey-2499 [000] *.~3 61513.836140: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x48048 ==> next_name=hi-thread next_pid=2504 next_prio=2 hi-thread-2504 [000] *.~3 61513.836141: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] d.~3 61513.836141: cobalt_head_sysexit: result=0 hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3 hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] d.~3 61513.836143: cobalt_shadow_gorelax: reason=fault hi-thread-2504 [000] d.~3 61513.836143: cobalt_lostage_request: request=wakeup pid=2504 comm=hi-thread hi-thread-2504 [000] *.~3 61513.836144: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] *.~3 61513.836144: cobalt_schedule: status=0x10000000 hi-thread-2504 [000] *.~3 61513.836144: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] *.~3 61513.836145: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 gdb-2495 [000] *.~3 61513.836146: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] d.h3 61513.836146: irq_handler_entry: irq=120 name=in-band work gdb-2495 [000] d.h3 61513.836147: cobalt_lostage_wakeup: pid=2504 comm=hi-thread gdb-2495 [000] d.h5 61513.836147: sched_waking: comm=hi-thread pid=2504 prio=97 target_cpu=000 gdb-2495 [000] dNh6 61513.836147: sched_wakeup: comm=hi-thread pid=2504 prio=97 target_cpu=000 gdb-2495 [000] dNh3 61513.836148: irq_handler_exit: irq=120 ret=handled gdb-2495 [000] dNh3 61513.836148: irq_handler_entry: irq=124 name=proxy tick gdb-2495 [000] dNh3 61513.836151: softirq_raise: vec=1 [action=TIMER] gdb-2495 [000] dNh3 61513.836152: softirq_raise: vec=9 [action=RCU] gdb-2495 [000] dNh4 61513.836152: sched_stat_runtime: comm=gdb pid=2495 runtime=1007474 [ns] vruntime=300515161229 [ns] gdb-2495 [000] *Nh3 61513.836154: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2395843 interval=0 mode=rel gdb-2495 [000] *Nh3 61513.836155: cobalt_tick_shot: next tick at 61513.838550 (delay: 2395 us) gdb-2495 [000] dNh3 61513.836155: irq_handler_exit: irq=124 ret=handled gdb-2495 [000] .Ns3 61513.836156: softirq_entry: vec=1 [action=TIMER] gdb-2495 [000] .Ns3 61513.836157: softirq_exit: vec=1 [action=TIMER] gdb-2495 [000] .Ns3 61513.836157: softirq_entry: vec=9 [action=RCU] gdb-2495 [000] dNs6 61513.836158: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000 gdb-2495 [000] dNs7 61513.836159: sched_stat_runtime: comm=gdb pid=2495 runtime=6881 [ns] vruntime=300515168110 [ns] gdb-2495 [000] dNs7 61513.836160: sched_stat_blocked: comm=rcu_sched pid=11 delay=4188739 [ns] gdb-2495 [000] dNs7 61513.836161: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000 gdb-2495 [000] .Ns3 61513.836164: softirq_exit: vec=9 [action=RCU] gdb-2495 [000] d..3 61513.836165: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=hi-thread next_pid=2504 next_prio=97 hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200 hi-thread-2504 [000] *..2 61513.836169: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] *..2 61513.836170: cobalt_thread_suspend: pid=2504 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] d..2 61513.836170: signal_generate: sig=5 errno=0 code=128 comm=hi-thread pid=2504 grp=0 res=0 hi-thread-2504 [000] d..3 61513.836173: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0 hi-thread-2504 [000] d..3 61513.836175: sched_stat_wait: comm=rcu_sched pid=11 delay=14490 [ns] hi-thread-2504 [000] d..3 61513.836175: sched_switch: prev_comm=hi-thread prev_pid=2504 prev_prio=97 prev_state=t ==> next_comm=rcu_sched next_pid=11 next_prio=120 rcu_sched-11 [000] d..3 61513.836179: sched_stat_runtime: comm=rcu_sched pid=11 runtime=4960 [ns] vruntime=300512173070 [ns] rcu_sched-11 [000] d..3 61513.836179: sched_stat_wait: comm=gdb pid=2495 delay=19450 [ns] rcu_sched-11 [000] d..3 61513.836180: sched_switch: prev_comm=rcu_sched prev_pid=11 prev_prio=120 prev_state=I ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] ...1 61513.836187: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cb0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836197: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836198: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7ae0) gdb-2495 [000] ...1 61513.836199: sys_newfstat -> 0x0 gdb-2495 [000] ...1 61513.836200: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) gdb-2495 [000] ...1 61513.836214: sys_read -> 0x3f1 gdb-2495 [000] ...1 61513.836217: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836217: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836221: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8) gdb-2495 [000] ...1 61513.836222: sys_rt_sigprocmask -> 0x0 gdb-2495 [000] d..2 61513.836223: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000 gdb-2495 [000] ...1 61513.836225: sys_read(fd: f, buf: 7ffe8b2a7667, count: 1) gdb-2495 [000] ...1 61513.836226: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.836227: sys_write(fd: 10, buf: 5585fe0354b1, count: 1) gdb-2495 [000] ...1 61513.836229: sys_write -> 0x1 gdb-2495 [000] ...1 61513.836229: sys_rt_sigreturn() gdb-2495 [000] ...1 61513.836233: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1) gdb-2495 [000] ...1 61513.836234: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.836235: sys_write(fd: 1, buf: 5585fe7a5290, count: 27) gdb-2495 [000] d..4 61513.836236: sched_waking: comm=smokey pid=2493 prio=120 target_cpu=000 gdb-2495 [000] d..5 61513.836237: sched_stat_runtime: comm=gdb pid=2495 runtime=58399 [ns] vruntime=300515226509 [ns] gdb-2495 [000] d..5 61513.836238: sched_stat_sleep: comm=smokey pid=2493 delay=1717179 [ns] gdb-2495 [000] dN.5 61513.836239: sched_wakeup: comm=smokey pid=2493 prio=120 target_cpu=000 gdb-2495 [000] .N.1 61513.836240: sys_write -> 0x27 gdb-2495 [000] dN.3 61513.836241: sched_stat_wait: comm=smokey pid=2493 delay=0 [ns] gdb-2495 [000] d..3 61513.836241: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2493 next_prio=120 smokey-2493 [000] ...1 61513.836244: sys_read -> 0x1 smokey-2493 [000] ..~1 61513.836246: cobalt_head_sysentry: syscall=read ... smokey-2493 [000] d..3 61513.836344: sched_stat_runtime: comm=smokey pid=2493 runtime=106636 [ns] vruntime=300512333145 [ns] smokey-2493 [000] d..3 61513.836344: sched_stat_wait: comm=gdb pid=2495 delay=106636 [ns] smokey-2493 [000] d..3 61513.836345: sched_switch: prev_comm=smokey prev_pid=2493 prev_prio=120 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] ...1 61513.836347: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0) gdb-2495 [000] ...1 61513.836348: sys_poll -> 0x1 gdb-2495 [000] ...1 61513.836350: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) gdb-2495 [000] ...1 61513.836351: sys_read -> 0x1 gdb-2495 [000] ...1 61513.836352: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) gdb-2495 [000] ...1 61513.836352: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.836353: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8) gdb-2495 [000] ...1 61513.836354: sys_rt_sigprocmask -> 0x0 gdb-2495 [000] ...1 61513.836355: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) gdb-2495 [000] ...1 61513.836355: sched_process_wait: comm=gdb pid=0 prio=120 gdb-2495 [000] ...1 61513.836357: sys_wait4 -> 0x9c8 gdb-2495 [000] ...1 61513.836360: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0) gdb-2495 [000] d..3 61513.836361: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836363: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836367: sys_ptrace(request: 4202, pid: 9c8, addr: 0, data: 7ffe8b2a7cb0) gdb-2495 [000] d..3 61513.836368: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836369: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836370: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6f10) gdb-2495 [000] d..3 61513.836370: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836371: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836372: sys_ptrace(request: d, pid: 9c8, addr: 0, data: 7ffe8b2a6f10) gdb-2495 [000] d..3 61513.836373: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836374: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836376: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) gdb-2495 [000] ...1 61513.836376: sched_process_wait: comm=gdb pid=0 prio=120 gdb-2495 [000] ...1 61513.836377: sys_wait4 -> 0x0 gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8) gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask -> 0x0 gdb-2495 [000] ...1 61513.836382: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cd0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836399: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836400: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7b50) gdb-2495 [000] ...1 61513.836401: sys_newfstat -> 0x0 gdb-2495 [000] ...1 61513.836402: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) gdb-2495 [000] ...1 61513.836409: sys_read -> 0x140 gdb-2495 [000] ...1 61513.836410: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400) gdb-2495 [000] ...1 61513.836410: sys_read -> 0x0 gdb-2495 [000] ...1 61513.836415: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836416: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836419: sys_read(fd: f, buf: 7ffe8b2a7f17, count: 1) gdb-2495 [000] ...1 61513.836420: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.836421: sys_write(fd: 10, buf: 5585fe0354b1, count: 1) gdb-2495 [000] ...1 61513.836422: sys_write -> 0x1 gdb-2495 [000] ...1 61513.836425: sys_ptrace(request: 1, pid: 9c8, addr: 7ffff777f370, data: 7ffe8b2a7878) gdb-2495 [000] d..3 61513.836426: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836428: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836430: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a7bd8) gdb-2495 [000] d..3 61513.836430: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836431: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836433: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a70e0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836446: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836446: sys_pread64(fd: 12, buf: 7ffe8b2a7430, count: 900, pos: 7ffff67c9700) gdb-2495 [000] ...1 61513.836449: sys_pread64 -> 0x900 gdb-2495 [000] ...1 61513.836450: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836450: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836479: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836488: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836489: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950) gdb-2495 [000] ...1 61513.836490: sys_newfstat -> 0x0 gdb-2495 [000] ...1 61513.836490: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400) gdb-2495 [000] ...1 61513.836498: sys_read -> 0x14d gdb-2495 [000] ...1 61513.836499: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) gdb-2495 [000] ...1 61513.836499: sys_read -> 0x0 gdb-2495 [000] ...1 61513.836503: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836504: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836507: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836514: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836515: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950) gdb-2495 [000] ...1 61513.836515: sys_newfstat -> 0x0 gdb-2495 [000] ...1 61513.836516: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) gdb-2495 [000] ...1 61513.836521: sys_read -> 0x13b gdb-2495 [000] ...1 61513.836522: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400) gdb-2495 [000] ...1 61513.836522: sys_read -> 0x0 gdb-2495 [000] ...1 61513.836525: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836525: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13) gdb-2495 [000] *..3 61513.836532: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000 gdb-2495 [000] *..3 61513.836532: cobalt_trace_pid: pid=2499, prio=1 gdb-2495 [000] *..3 61513.836533: cobalt_thread_unblock: pid=2499 state=0x48848 info=0x0 gdb-2495 [000] *.~3 61513.836586: cobalt_schedule: status=0x10000000 gdb-2495 [000] *.~3 61513.836586: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] *.~3 61513.836587: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1 smokey-2499 [000] *.~3 61513.836589: cobalt_trace_pid: pid=2499, prio=1 smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined smokey-2499 [000] *.~3 61513.836589: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey smokey-2499 [000] *.~3 61513.836590: cobalt_thread_suspend: pid=2499 mask=0x400080 timeout=0 timeout_mode=0 wchan=0000000000000000 smokey-2499 [000] *.~3 61513.836590: cobalt_schedule: status=0x10000000 smokey-2499 [000] *.~3 61513.836590: cobalt_trace_pid: pid=2499, prio=1 smokey-2499 [000] *.~3 61513.836591: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x4488c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 gdb-2495 [000] *.~3 61513.836592: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] d..5 61513.836592: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dN.6 61513.836594: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dN.3 61513.836594: signal_generate: sig=19 errno=0 code=-6 comm=smokey pid=2499 grp=0 res=0 gdb-2495 [000] dNh3 61513.836595: irq_handler_entry: irq=120 name=in-band work gdb-2495 [000] dNh3 61513.836596: cobalt_lostage_wakeup: pid=2499 comm=smokey gdb-2495 [000] *N~4 61513.838552: irq_pipeline_entry: irq=4354 gdb-2495 [000] *N~4 61513.838553: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt gdb-2495 [000] *N~4 61513.838554: cobalt_timer_expire: timer=00000000d13b8aa9 gdb-2495 [000] *N~4 61513.838555: irq_handler_exit: irq=4354 ret=handled gdb-2495 [000] *N~4 61513.838555: irq_pipeline_exit: irq=4354 gdb-2495 [000] dNh8 61513.981277: sched_waking: comm=systemd-journal pid=163 prio=120 target_cpu=000 gdb-2495 [000] dNh9 61513.981278: sched_stat_runtime: comm=gdb pid=2495 runtime=144934239 [ns] vruntime=300660160748 [ns] gdb-2495 [000] dNh9 61513.981280: sched_stat_sleep: comm=systemd-journal pid=163 delay=111416392895 [ns] gdb-2495 [000] dNh9 61513.981280: sched_wakeup: comm=systemd-journal pid=163 prio=120 target_cpu=000 gdb-2495 [000] dNh3 61513.981281: irq_handler_exit: irq=120 ret=handled gdb-2495 [000] dNh3 61513.981282: irq_handler_entry: irq=124 name=proxy tick gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=1 [action=TIMER] gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=9 [action=RCU] gdb-2495 [000] dNh4 61513.981286: sched_stat_runtime: comm=gdb pid=2495 runtime=8375 [ns] vruntime=300660169123 [ns] gdb-2495 [000] dNh5 61513.981288: sched_waking: comm=cobalt_printf pid=2494 prio=120 target_cpu=000 gdb-2495 [000] dNh6 61513.981289: sched_stat_runtime: comm=gdb pid=2495 runtime=2987 [ns] vruntime=300660172110 [ns] gdb-2495 [000] dNh6 61513.981291: sched_stat_sleep: comm=cobalt_printf pid=2494 delay=238660319 [ns] gdb-2495 [000] dNh6 61513.981291: sched_wakeup: comm=cobalt_printf pid=2494 prio=120 target_cpu=000 gdb-2495 [000] dNh5 61513.981292: sched_waking: comm=cobalt_printf pid=2503 prio=120 target_cpu=000 gdb-2495 [000] dNh6 61513.981292: sched_stat_runtime: comm=gdb pid=2495 runtime=3069 [ns] vruntime=300660175179 [ns] gdb-2495 [000] dNh6 61513.981294: sched_stat_sleep: comm=cobalt_printf pid=2503 delay=146611442 [ns] gdb-2495 [000] dNh6 61513.981294: sched_wakeup: comm=cobalt_printf pid=2503 prio=120 target_cpu=000 gdb-2495 [000] *Nh3 61513.981295: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=588476 interval=0 mode=rel gdb-2495 [000] *Nh3 61513.981296: cobalt_tick_shot: next tick at 61513.981884 (delay: 588 us) gdb-2495 [000] dNh3 61513.981296: irq_handler_exit: irq=124 ret=handled gdb-2495 [000] .Ns3 61513.981297: softirq_entry: vec=1 [action=TIMER] gdb-2495 [000] dNs5 61513.981298: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000 gdb-2495 [000] dNs6 61513.981298: sched_stat_runtime: comm=gdb pid=2495 runtime=6218 [ns] vruntime=300660181397 [ns] gdb-2495 [000] dNs6 61513.981299: sched_stat_blocked: comm=rcu_sched pid=11 delay=145119923 [ns] gdb-2495 [000] dNs6 61513.981299: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000 gdb-2495 [000] dNs7 61513.981302: sched_waking: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000 gdb-2495 [000] dNs8 61513.981302: sched_stat_runtime: comm=gdb pid=2495 runtime=3884 [ns] vruntime=300660185281 [ns] gdb-2495 [000] dNs8 61513.981303: sched_stat_blocked: comm=kworker/0:1 pid=1932 delay=206048652 [ns] gdb-2495 [000] dNs8 61513.981304: sched_wakeup: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000 gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981312: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] .Ns3 61513.981314: softirq_exit: vec=1 [action=TIMER] gdb-2495 [000] .Ns3 61513.981314: softirq_entry: vec=9 [action=RCU] gdb-2495 [000] .Ns3 61513.981315: softirq_exit: vec=9 [action=RCU] gdb-2495 [000] dNs5 61513.981316: sched_waking: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000 gdb-2495 [000] dNs6 61513.981316: sched_stat_runtime: comm=gdb pid=2495 runtime=13841 [ns] vruntime=300660199122 [ns] gdb-2495 [000] dNs6 61513.981317: sched_stat_sleep: comm=ksoftirqd/0 pid=10 delay=149349034 [ns] gdb-2495 [000] dNs6 61513.981317: sched_wakeup: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000 gdb-2495 [000] .N.1 61513.981318: sys_tkill -> 0x0 gdb-2495 [000] d..3 61513.981320: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] *.~3 61513.981888: irq_pipeline_entry: irq=4354 smokey-2499 [000] *.~3 61513.981888: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt smokey-2499 [000] *.~3 61513.981889: cobalt_timer_expire: timer=00000000d13b8aa9 smokey-2499 [000] *.~3 61513.981889: irq_handler_exit: irq=4354 ret=handled smokey-2499 [000] *.~3 61513.981890: irq_pipeline_exit: irq=4354 smokey-2499 [000] d.h1 61514.106435: irq_handler_entry: irq=124 name=proxy tick smokey-2499 [000] *.h1 61514.106441: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2109534 interval=0 mode=rel smokey-2499 [000] *.h1 61514.106441: cobalt_tick_shot: next tick at 61514.108550 (delay: 2109 us) smokey-2499 [000] d.h1 61514.106442: irq_handler_exit: irq=124 ret=handled smokey-2499 [000] ...1 61514.106442: cobalt_shadow_relaxed: state=0x4488c0 info=0x200 smokey-2499 [000] *.~1 61514.108552: irq_pipeline_entry: irq=4354 Regards Hongzhan Chen > >-- >Philippe. ^ permalink raw reply related [flat|nested] 16+ messages in thread
* RE: gdb test failure debug status update 2021-04-30 5:25 ` Chen, Hongzhan @ 2021-04-30 5:51 ` Chen, Hongzhan 2021-04-30 7:36 ` Philippe Gerum 2021-04-30 7:34 ` Philippe Gerum 1 sibling, 1 reply; 16+ messages in thread From: Chen, Hongzhan @ 2021-04-30 5:51 UTC (permalink / raw) To: Philippe Gerum, xenomai After applied following patch , the gdb test failure issue seems fixed. Basically , it enforce going back to primary mode after check that XNCONTHI is set for current thread. It seems ugly. Please suggest different solution. @@ -355,9 +357,36 @@ static int handle_user_return(struct task_struct *task) void handle_oob_mayday(struct pt_regs *regs) { + struct xnthread *thread; + spl_t s; + int err; + XENO_BUG_ON(COBALT, !xnthread_test_state(xnthread_current(), XNUSER)); + thread = xnthread_current(); + + trace_cobalt_handle_oob_mayday(thread); + xnthread_relax(0, 0); + + if (xnthread_test_info(thread, XNCONTHI)) { + xnlock_get_irqsave(&nklock, s); + xnthread_clear_info(thread, XNCONTHI); + xnlock_put_irqrestore(&nklock, s); + + err = xnthread_harden(); + + /* + * XNCONTHI may or may not have been re-applied if + * harden bailed out due to pending signals. Make sure + * it is set in that case. + */ + if (err == -ERESTARTSYS) { + xnlock_get_irqsave(&nklock, s); + xnthread_set_info(thread, XNCONTHI); + xnlock_put_irqrestore(&nklock, s); + } + } } Regards Hongzhan Chen -----Original Message----- From: Chen, Hongzhan <hongzhan.chen@intel.com> Sent: Friday, April 30, 2021 1:26 PM To: Chen, Hongzhan <hongzhan.chen@intel.com>; Philippe Gerum <rpm@xenomai.org>; xenomai@xenomai.org Subject: RE: gdb test failure debug status update The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax. That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode because kernel exit code is never called in this case? Regards Hongzhan Chen -----Original Message----- From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai Sent: Thursday, April 29, 2021 2:31 PM To: Philippe Gerum <rpm@xenomai.org> Cc: xenomai@xenomai.org Subject: RE: gdb test failure debug status update > >-----Original Message----- >From: Philippe Gerum <rpm@xenomai.org> >Sent: Wednesday, April 28, 2021 10:30 PM >To: Philippe Gerum <rpm@xenomai.org> >Cc: Chen, Hongzhan <hongzhan.chen@intel.com>; xenomai@xenomai.org >Subject: Re: gdb test failure debug status update > > >Philippe Gerum via Xenomai <xenomai@xenomai.org> writes: > >> Chen, Hongzhan via Xenomai <xenomai@xenomai.org> writes: >> >>> According to my validation, gdb test fail on dovetail 5.10 branch but pass on v5.9-evl4 tag with same for-upstream/dovetail >>> xenomai code base. >>> >>> After further debug , the issue is more clear for me. Gdb test failure because low priority thread smokey userspace is still >>> executed after "cobalt_shadow_relaxed: state=0x4488c0 info=0x200" like log [1] on dovetail-5.10 branch. >>> The weird thing is that its following first ftrace log happen at 62235.848583 after cobalt_shadow_relaxed in log [1]. >>> It is almost 3ms happened after cobalt_shadow_relaxed. The low priority smoke thread user space is executed during this >>> 3ms period so that test fail. >>> >>> But in success case with v5.9-evl4 like in log [2], the time interval between cobalt_shadow_relaxed and the following first ftrace log >>> is only about 1us. It seems that low priority smokey userspace do not have chance to execute in this 1us because gdb test is successful. >>> >>> My question is why there is even no interrupt happened during that about 3ms period in failure case? Tick seems in abnormal behavior. >>> Please comment if you have any ideas to further debug it. >>> >>> PS: All my tests run on same up Xtream board. >> >> <snip> >> >> Let's put aside the tick issue for now, there may be a valid reason for >> this delay with dynticks enabled. >> >> The issue at stake may be related to the way a return to kernel space is >> forced on a @user task (Dovetail has an integrated service for >> triggering this called dovetail_request_ucall()). >> >> The logic for doing so is as follows: >> >> 1. @user hits a breakpoint, which is an exception Dovetail-wise >> >> 2. @user gets XNDBGSTOP set into its flags because Cobalt notices it is >> being debugged via a breakpoint trap, then relaxed as a result of taking >> a exception in general, so that we may traverse the common trap handling code >> safely. >> >> 3. since XNDBGSTOP is a blocking bit Cobalt-wise, it should prevent >> @user from being picked for scheduling by the real-time core, next time >> a Cobalt considers rescheduling that is. However, since @user is >> currently relaxed, it can still run under the supervision of the common >> Linux scheduler. This is what the log[1] show. >> >> 4. the common/in-band kernel code stops @user due to the ptrace stop >> condition caused by the breakpoint, waiting for a continuation event to >> happen. >> >> Therefore, upon PTRACE_CONT (i.e. gdb continue), we need to force @user >> to call back into kernel context (handle_ptrace_cont -> >> dovetail_request_ucall), then ask for a switch to primary mode from >> there, which should eventually happen when @user is about to leave the >> kernel (on x86, this now happens from a generic kernel entry/exit code >> in kernel/entry/*). As a result, handle_taskexit_event() runs, figures > >Not quite: >s,handle_taskexit_event,handle_user_return, There is trace_cobalt_shadow_unmap in handle_taskexit_event. I added trace_cobalt_handle_user_return and trace_cobalt_handle_ptrace_cont like patch [3] in log [4]. 1. In the log [4], the first cobalt_handle_ptrace_cont is followed by cobalt_handle_user_return that clear info flag XNCONTHI for smokey-2499 because cobalt_handle_ptrace_cont called dovetail_request_ucall . the TIF_RETUSER bit was cleared for smokey-2499. After then 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000 clear state flag XNDBGSTOP. After that, thread smokey-2499 run without these bit set till "hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000". 2. There is second cobalt_handle_ptrace_cont called for smokey-2499 but at that time dovetail_request_ucall would not be called because XNSSTEP Is not set like "smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0" 3. hi-thread-2504 breakpoint at its first breakpoint_target so it trigger hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3 It call dovetail_request_ucall(current); for itself and also call xnthread_set_info(thread, XNCONTHI); That is why we can see this bit set in hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200 4. finally ,gdb call sys_tkill in gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13) and trigger that handle_oob_mayday is called so that xnthread_relax for smokey-2499 Is called in "smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined." It seems that there never call dovetail_request_ucall for process smokey-2499 in log[4] so that final xhthread_relax would have chance to return to user? The second handle_ptrace_cont happen too much earlier because XNSSTEP still have not been set so that system never have change to call dovetail_request_ucall for smokey-2499. In this case , do we need to do hack in 3 , when handle_oob_trap_entry is called for hi-thread-2504 we call xnthread_set_info(thread, XNCONTHI); and dovetail_request_ucall for all threads that have same cobalt_process? Is this solution OK for fixing this case? [3]: index 07dcc7a63..cd76961f5 100644 --- a/kernel/cobalt/dovetail/kevents.c +++ b/kernel/cobalt/dovetail/kevents.c @@ -331,6 +331,8 @@ static int handle_user_return(struct task_struct *task) if (thread == NULL) return KEVENT_PROPAGATE; + trace_cobalt_handle_user_return(thread); + if (xnthread_test_info(thread, XNCONTHI)) { xnlock_get_irqsave(&nklock, s); xnthread_clear_info(thread, XNCONTHI); @@ -468,6 +470,8 @@ int handle_ptrace_resume(struct task_struct *tracee) if (thread == NULL) return KEVENT_PROPAGATE; + trace_cobalt_handle_ptrace_resume(thread); + if (xnthread_test_state(thread, XNSSTEP)) { xnlock_get_irqsave(&nklock, s); @@ -485,6 +489,8 @@ static void handle_ptrace_cont(void) struct xnthread *curr = xnthread_current(); spl_t s; + trace_cobalt_handle_ptrace_cont(curr); + xnlock_get_irqsave(&nklock, s); if (xnthread_test_state(curr, XNSSTEP)) { @@ -604,6 +610,7 @@ void resume_oob_task(struct task_struct *p) /* inband, oob stage stalled */ if (unlikely(xnthread_test_info(thread, XNCONTHI))) { xnthread_clear_info(thread, XNCONTHI); dovetail_clear_ucall(); + trace_cobalt_resume_oob_task(thread); } /* Unregister as debugged thread in case we postponed this. */ diff --git a/kernel/cobalt/trace/cobalt-core.h b/kernel/cobalt/trace/cobalt-core.h index 78f66a106..88a4067fd 100644 --- a/kernel/cobalt/trace/cobalt-core.h +++ b/kernel/cobalt/trace/cobalt-core.h @@ -459,6 +459,27 @@ DEFINE_EVENT(curr_thread_event, cobalt_thread_set_mode, TP_ARGS(thread) ); +DEFINE_EVENT(thread_event, cobalt_handle_ptrace_cont, + TP_PROTO(struct xnthread *thread), [4]: gdb-2495 [000] d..3 61513.834825: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] ...1 61513.834827: cobalt_handle_ptrace_cont: pid=2499 state=0x4488c0 info=0x200 smokey-2499 [000] d..1 61513.834828: cobalt_handle_user_return: pid=2499 state=0x4488c0 info=0x200 smokey-2499 [000] d..1 61513.834828: cobalt_shadow_gohard: state=0x4488c0 info=0x0 smokey-2499 [000] d..3 61513.834830: sched_stat_wait: comm=gdb pid=2495 delay=8017 [ns] smokey-2499 [000] d..3 61513.834831: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] *..3 61513.834831: cobalt_thread_resume: name=smokey pid=2499 mask=0x80 gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1 gdb-2495 [000] *..3 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000 gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1 gdb-2495 [000] *.~3 61513.834833: cobalt_schedule: status=0x10000000 gdb-2495 [000] *.~3 61513.834833: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] *.~3 61513.834833: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1 smokey-2499 [000] d.~3 61513.834835: cobalt_shadow_hardened: state=0x48040 info=0x0 smokey-2499 [000] d.~3 61513.834836: cobalt_head_sysentry: syscall=sem_init smokey-2499 [000] d.~3 61513.834839: cobalt_psem_init: sem=0x4e(anon) flags=0x0() value=0 smokey-2499 [000] d.~3 61513.834839: cobalt_head_sysexit: result=0 smokey-2499 [000] d.~3 61513.834848: cobalt_shadow_gorelax: reason=syscall smokey-2499 [000] d.~3 61513.834848: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey smokey-2499 [000] *.~3 61513.834848: cobalt_thread_suspend: pid=2499 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 smokey-2499 [000] *.~3 61513.834849: cobalt_schedule: status=0x10000000 smokey-2499 [000] *.~3 61513.834849: cobalt_trace_pid: pid=2499, prio=1 smokey-2499 [000] *.~3 61513.834849: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 gdb-2495 [000] *.~3 61513.834850: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] d.h3 61513.834851: irq_handler_entry: irq=120 name=in-band work gdb-2495 [000] d.h3 61513.834852: cobalt_lostage_wakeup: pid=2499 comm=smokey gdb-2495 [000] d.h5 61513.834852: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dNh6 61513.834853: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dNh3 61513.834853: irq_handler_exit: irq=120 ret=handled gdb-2495 [000] dN.3 61513.834854: sched_stat_runtime: comm=gdb pid=2495 runtime=23565 [ns] vruntime=300514015543 [ns] gdb-2495 [000] d..3 61513.834855: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] ...1 61513.834856: cobalt_shadow_relaxed: state=0x480c0 info=0x0 smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min(policy: 1) smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min -> 0x1 smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max(policy: 1) smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max -> 0x63 smokey-2499 [000] ..~1 61513.834862: cobalt_head_sysentry: syscall=thread_getschedparam_ex smokey-2499 [000] ...1 61513.834862: cobalt_root_sysentry: syscall=thread_getschedparam_ex smokey-2499 [000] ...1 61513.834863: cobalt_pthread_getschedparam: pth=000000008d36be1c policy=fifo param={ priority=1 } smokey-2499 [000] ...1 61513.834864: cobalt_root_sysexit: result=0 smokey-2499 [000] ...1 61513.834872: sys_mmap(addr: 0, len: 11000, prot: 0, flags: 20022, fd: ffffffff, off: 0) smokey-2499 [000] ...1 61513.834877: sys_mmap -> 0x7ffff67b9000 smokey-2499 [000] ...1 61513.834878: sys_mprotect(start: 7ffff67ba000, len: 10000, prot: 3) smokey-2499 [000] ...1 61513.834906: sys_mprotect -> 0x0 smokey-2499 [000] ...1 61513.834909: sys_clone(clone_flags: 3d0f00, newsp: 7ffff67c8fb0, parent_tidptr: 7ffff67c99d0, child_tidptr: 7ffff67c99d0, tls: 7ffff67c9700) smokey-2499 [000] ...1 61513.834923: task_newtask: pid=2504 comm=smokey clone_flags=3d0f00 oom_score_adj=0 smokey-2499 [000] ...1 61513.834923: sched_process_fork: comm=smokey pid=2499 child_comm=smokey child_pid=2504 smokey-2499 [000] d..3 61513.834925: sched_wakeup_new: comm=smokey pid=2504 prio=98 target_cpu=000 smokey-2499 [000] d..3 61513.834928: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0 smokey-2499 [000] d..3 61513.834931: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=t ==> next_comm=smokey next_pid=2504 next_prio=98 hi-thread-2504 [000] ...1 61513.834932: sys_clone -> 0x0 hi-thread-2504 [000] d..3 61513.834934: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=2 hi-thread-2504 [000] d..3 61513.834935: sched_stat_wait: comm=gdb pid=2495 delay=82002 [ns] hi-thread-2504 [000] d..3 61513.834936: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=t ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] d..2 61513.834937: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000 gdb-2495 [000] ...1 61513.834940: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1) gdb-2495 [000] ...1 61513.834943: sys_read -> 0x1 gdb-2495 [000] ...1 61513.834943: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1) gdb-2495 [000] ...1 61513.834944: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.834944: sys_write(fd: 10, buf: 5585fe0354b1, count: 1) gdb-2495 [000] ...1 61513.834946: sys_write -> 0x1 gdb-2495 [000] ...1 61513.834947: sys_rt_sigreturn() gdb-2495 [000] ...1 61513.834953: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1) gdb-2495 [000] ...1 61513.834953: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.834955: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0) gdb-2495 [000] ...1 61513.834956: sys_poll -> 0x1 gdb-2495 [000] ...1 61513.834957: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) gdb-2495 [000] ...1 61513.834958: sys_read -> 0x1 gdb-2495 [000] ...1 61513.834959: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) gdb-2495 [000] ...1 61513.834960: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8) gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask -> 0x0 gdb-2495 [000] ...1 61513.834962: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) gdb-2495 [000] ...1 61513.834962: sched_process_wait: comm=gdb pid=0 prio=120 gdb-2495 [000] ...1 61513.834964: sys_wait4 -> 0x9c3 gdb-2495 [000] ...1 61513.834965: sys_ptrace(request: 4201, pid: 9c3, addr: 0, data: 7ffe8b2a7d08) gdb-2495 [000] d..3 61513.834966: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.834967: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.834967: sys_wait4(upid: 9c8, stat_addr: 7ffe8b2a7cfc, options: 40000000, ru: 0) gdb-2495 [000] ...1 61513.834968: sched_process_wait: comm=gdb pid=2504 prio=120 gdb-2495 [000] ...1 61513.834969: sys_wait4 -> 0x9c8 gdb-2495 [000] ...1 61513.834972: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578) gdb-2495 [000] d..3 61513.834972: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.834975: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.834976: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a78d8) gdb-2495 [000] d..3 61513.834977: sched_wait_task: comm=smokey pid=2504 prio=98 gdb-2495 [000] ...1 61513.834977: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.834979: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.834989: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.834990: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff67c9700) gdb-2495 [000] ...1 61513.834992: sys_pread64 -> 0x900 gdb-2495 [000] ...1 61513.834993: sys_close(fd: 12) gdb-2495 [000] ...1 61513.834994: sys_close -> 0x0 gdb-2495 [000] ...1 61513.835010: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578) gdb-2495 [000] d..3 61513.835011: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.835012: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835013: sys_ptrace(request: 3, pid: 9c3, addr: a8, data: 7ffe8b2a78d8) gdb-2495 [000] d..3 61513.835013: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.835014: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835015: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.835022: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.835022: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff758f740) gdb-2495 [000] ...1 61513.835025: sys_pread64 -> 0x900 gdb-2495 [000] ...1 61513.835025: sys_close(fd: 12) gdb-2495 [000] ...1 61513.835026: sys_close -> 0x0 gdb-2495 [000] ...1 61513.835031: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) gdb-2495 [000] ...1 61513.835032: sched_process_wait: comm=gdb pid=0 prio=120 gdb-2495 [000] ...1 61513.835033: sys_wait4 -> 0x0 gdb-2495 [000] ...1 61513.835036: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0) gdb-2495 [000] d..3 61513.835037: sched_wait_task: comm=smokey pid=2504 prio=98 gdb-2495 [000] ...1 61513.835039: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835043: sys_ptrace(request: 6, pid: 9c8, addr: 388, data: 0) gdb-2495 [000] d..3 61513.835044: sched_wait_task: comm=smokey pid=2504 prio=98 gdb-2495 [000] ...1 61513.835044: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835045: sys_ptrace(request: 7, pid: 9c8, addr: 1, data: 0) gdb-2495 [000] d..3 61513.835046: sched_wait_task: comm=smokey pid=2504 prio=98 gdb-2495 [000] d..3 61513.835047: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000 gdb-2495 [000] dN.4 61513.835048: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000 gdb-2495 [000] .N.1 61513.835048: sys_ptrace -> 0x0 gdb-2495 [000] dN.3 61513.835050: sched_stat_runtime: comm=gdb pid=2495 runtime=112561 [ns] vruntime=300514128104 [ns] gdb-2495 [000] d..3 61513.835051: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=98 hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list(head: 7ffff67c99e0, len: 18) hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list -> 0x0 hi-thread-2504 [000] ...1 61513.835056: sys_futex(uaddr: 7ffff67c9d18, op: 80, val: 2, utime: 0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700) hi-thread-2504 [000] d..3 61513.835059: sched_stat_wait: comm=gdb pid=2495 delay=10883 [ns] hi-thread-2504 [000] d..3 61513.835060: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] ...1 61513.835063: sys_ptrace(request: c, pid: 9c3, addr: 0, data: 7ffe8b2a6ef0) gdb-2495 [000] d..3 61513.835064: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.835065: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835068: sys_ptrace(request: 7, pid: 9c3, addr: 1, data: 0) gdb-2495 [000] d..3 61513.835069: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] d..3 61513.835069: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dN.4 61513.835070: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] .N.1 61513.835071: sys_ptrace -> 0x0 gdb-2495 [000] dN.3 61513.835071: sched_stat_runtime: comm=gdb pid=2495 runtime=12210 [ns] vruntime=300514140314 [ns] gdb-2495 [000] d..3 61513.835072: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0 smokey-2499 [000] ...1 61513.835074: sys_clone -> 0x9c8 smokey-2499 [000] ...1 61513.835075: sys_sched_setaffinity(pid: 9c8, len: 80, user_mask_ptr: 55555563e7c0) smokey-2499 [000] ...1 61513.835077: sys_sched_setaffinity -> 0x0 smokey-2499 [000] ...1 61513.835077: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 0, val3: 7ffff67c9700) smokey-2499 [000] d..3 61513.835078: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000 smokey-2499 [000] d..4 61513.835079: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000 smokey-2499 [000] ...1 61513.835080: sys_futex -> 0x1 smokey-2499 [000] ...1 61513.835086: sys_futex(uaddr: 7fffffffe828, op: 189, val: 0, utime: 7fffffffe780, uaddr2: 0, val3: ffffffff) smokey-2499 [000] d..3 61513.835089: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=smokey next_pid=2504 next_prio=98 hi-thread-2504 [000] ...1 61513.835090: sys_futex -> 0x0 hi-thread-2504 [000] ...1 61513.835090: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700) hi-thread-2504 [000] ...1 61513.835091: sys_futex -> 0x0 hi-thread-2504 [000] ...1 61513.835092: sys_sched_setscheduler(pid: 9c8, policy: 1, param: 7ffff67c8e70) hi-thread-2504 [000] ...1 61513.835095: sys_sched_setscheduler -> 0x0 hi-thread-2504 [000] ..~1 61513.835096: cobalt_head_sysentry: syscall=thread_create hi-thread-2504 [000] ...1 61513.835097: cobalt_root_sysentry: syscall=thread_create hi-thread-2504 [000] ...1 61513.835098: cobalt_pthread_create: pth=00000000ba35a01d policy=fifo param={ priority=2 } hi-thread-2504 [000] ...1 61513.835099: cobalt_timer_migrate: timer=000000004d21df54 cpu=0 hi-thread-2504 [000] ...1 61513.835100: cobalt_timer_migrate: timer=00000000c1301853 cpu=0 hi-thread-2504 [000] ...1 61513.835101: cobalt_thread_set_current_prio: thread=00000000e9fda3c4 pid=-1 prio=2 hi-thread-2504 [000] ...1 61513.835102: cobalt_thread_init: thread=00000000e9fda3c4 name=smokey flags=0x48000 class=rt prio=2 hi-thread-2504 [000] ...1 61513.835104: cobalt_shadow_map: thread=00000000e9fda3c4 pid=2504 prio=2 hi-thread-2504 [000] D..1 61513.835104: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_start: pid=2504 state=0x480d0 info=0x0 hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_resume: name=smokey pid=2504 mask=0x10 hi-thread-2504 [000] D..1 61513.835106: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] ...1 61513.835106: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] ...1 61513.835107: cobalt_shadow_gohard: state=0x480c0 info=0x0 hi-thread-2504 [000] d..3 61513.835108: sched_stat_wait: comm=gdb pid=2495 delay=37231 [ns] hi-thread-2504 [000] d..3 61513.835109: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] *..3 61513.835109: cobalt_thread_resume: name=smokey pid=2504 mask=0x80 gdb-2495 [000] *..3 61513.835110: cobalt_trace_pid: pid=2504, prio=2 gdb-2495 [000] *.~3 61513.835110: cobalt_schedule: status=0x10000000 gdb-2495 [000] *.~3 61513.835110: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] *.~3 61513.835111: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2 hi-thread-2504 [000] d.~3 61513.835112: cobalt_shadow_hardened: state=0x48040 info=0x0 hi-thread-2504 [000] d.~3 61513.835112: cobalt_root_sysexit: result=0 hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysentry: syscall=get_current hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysexit: result=0 hi-thread-2504 [000] d.~3 61513.835116: cobalt_shadow_gorelax: reason=syscall hi-thread-2504 [000] d.~3 61513.835117: cobalt_lostage_request: request=wakeup pid=2504 comm=smokey hi-thread-2504 [000] *.~3 61513.835117: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] *.~3 61513.835117: cobalt_schedule: status=0x10000000 hi-thread-2504 [000] *.~3 61513.835118: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] *.~3 61513.835118: cobalt_switch_context: prev_name=smokey prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 gdb-2495 [000] *.~3 61513.835119: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] d.h3 61513.835120: irq_handler_entry: irq=120 name=in-band work gdb-2495 [000] d.h3 61513.835120: cobalt_lostage_wakeup: pid=2504 comm=smokey gdb-2495 [000] d.h5 61513.835120: sched_waking: comm=smokey pid=2504 prio=97 target_cpu=000 gdb-2495 [000] dNh6 61513.835121: sched_wakeup: comm=smokey pid=2504 prio=97 target_cpu=000 gdb-2495 [000] dNh3 61513.835122: irq_handler_exit: irq=120 ret=handled gdb-2495 [000] dN.3 61513.835122: sched_stat_runtime: comm=gdb pid=2495 runtime=13441 [ns] vruntime=300514153755 [ns] gdb-2495 [000] d..3 61513.835123: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=97 hi-thread-2504 [000] ...1 61513.835124: cobalt_shadow_relaxed: state=0x480c0 info=0x0 hi-thread-2504 [000] ...1 61513.835125: sys_futex(uaddr: 7fffffffe828, op: 81, val: 1, utime: 0, uaddr2: 7ffff67c8e74, val3: 7fffffffe828) hi-thread-2504 [000] d..3 61513.835126: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 hi-thread-2504 [000] d..4 61513.835126: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 hi-thread-2504 [000] ...1 61513.835127: sys_futex -> 0x1 hi-thread-2504 [000] ..~1 61513.835128: cobalt_head_sysentry: syscall=migrate hi-thread-2504 [000] ...1 61513.835128: cobalt_root_sysentry: syscall=migrate hi-thread-2504 [000] ...1 61513.835128: cobalt_shadow_gohard: state=0x480c0 info=0x0 hi-thread-2504 [000] d..3 61513.835130: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] *..3 61513.835130: cobalt_thread_resume: name=smokey pid=2504 mask=0x80 smokey-2499 [000] *..3 61513.835130: cobalt_trace_pid: pid=2504, prio=2 smokey-2499 [000] *.~3 61513.835131: cobalt_schedule: status=0x10000000 smokey-2499 [000] *.~3 61513.835131: cobalt_trace_pid: pid=2499, prio=-1 smokey-2499 [000] *.~3 61513.835131: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2 hi-thread-2504 [000] d.~3 61513.835132: cobalt_shadow_hardened: state=0x48040 info=0x0 hi-thread-2504 [000] d.~3 61513.835132: cobalt_root_sysexit: result=1 hi-thread-2504 [000] d.~3 61513.835133: cobalt_head_sysentry: syscall=thread_setname hi-thread-2504 [000] d.~3 61513.835134: cobalt_pthread_setname: pth=00000000ba35a01d name=hi-thread hi-thread-2504 [000] d.~3 61513.835135: cobalt_head_sysexit: result=0 hi-thread-2504 [000] d.~3 61513.835136: cobalt_head_sysentry: syscall=clock_nanosleep hi-thread-2504 [000] d.~3 61513.835137: cobalt_clock_nanosleep: clock_id=0 flags=0x0() rqt=(0.001000000) hi-thread-2504 [000] *.~3 61513.835138: cobalt_thread_suspend: pid=2504 mask=0x4 timeout=1000001 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] *.~3 61513.835138: cobalt_timer_start: timer=000000004d21df54(smokey) value=1000001 interval=0 mode=rel hi-thread-2504 [000] *.~3 61513.835138: cobalt_schedule: status=0x10000000 hi-thread-2504 [000] *.~3 61513.835139: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] *.~3 61513.835139: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x48044 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 smokey-2499 [000] *.~3 61513.835140: cobalt_trace_pid: pid=2499, prio=-1 smokey-2499 [000] ...1 61513.835141: sys_futex -> 0x0 smokey-2499 [000] ..~1 61513.835144: cobalt_head_sysentry: syscall=migrate smokey-2499 [000] ...1 61513.835144: cobalt_root_sysentry: syscall=migrate smokey-2499 [000] ...1 61513.835144: cobalt_shadow_gohard: state=0x480c0 info=0x0 smokey-2499 [000] d..3 61513.835145: sched_stat_wait: comm=gdb pid=2495 delay=23982 [ns] smokey-2499 [000] d..3 61513.835146: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] *..3 61513.835147: cobalt_thread_resume: name=smokey pid=2499 mask=0x80 gdb-2495 [000] *..3 61513.835147: cobalt_trace_pid: pid=2499, prio=1 gdb-2495 [000] *.~3 61513.835147: cobalt_schedule: status=0x10000000 gdb-2495 [000] *.~3 61513.835147: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] *.~3 61513.835148: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1 smokey-2499 [000] d.~3 61513.835148: cobalt_shadow_hardened: state=0x48040 info=0x0 smokey-2499 [000] d.~3 61513.835149: cobalt_root_sysexit: result=1 smokey-2499 [000] *.~3 61513.835219: irq_pipeline_entry: irq=4354 smokey-2499 [000] *.~3 61513.835219: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt smokey-2499 [000] *.~3 61513.835220: cobalt_timer_expire: timer=00000000d13b8aa9 smokey-2499 [000] *.~3 61513.835220: cobalt_tick_shot: next tick at 61513.836136 (delay: 915 us) smokey-2499 [000] *.~3 61513.835221: irq_handler_exit: irq=4354 ret=handled smokey-2499 [000] *.~3 61513.835221: irq_pipeline_exit: irq=4354 smokey-2499 [000] *.~3 61513.836137: irq_pipeline_entry: irq=4354 smokey-2499 [000] *.~3 61513.836137: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt smokey-2499 [000] *.~3 61513.836138: cobalt_timer_expire: timer=000000004d21df54 smokey-2499 [000] *.~3 61513.836138: cobalt_thread_resume: name=hi-thread pid=2504 mask=0x4 smokey-2499 [000] *.~3 61513.836138: cobalt_trace_pid: pid=2504, prio=2 smokey-2499 [000] *.~3 61513.836139: cobalt_timer_stop: timer=000000004d21df54 smokey-2499 [000] *.~3 61513.836139: irq_handler_exit: irq=4354 ret=handled smokey-2499 [000] *.~3 61513.836139: irq_pipeline_exit: irq=4354 smokey-2499 [000] *.~3 61513.836140: cobalt_schedule: status=0x10000000 smokey-2499 [000] *.~3 61513.836140: cobalt_trace_pid: pid=2499, prio=1 smokey-2499 [000] *.~3 61513.836140: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x48048 ==> next_name=hi-thread next_pid=2504 next_prio=2 hi-thread-2504 [000] *.~3 61513.836141: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] d.~3 61513.836141: cobalt_head_sysexit: result=0 hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3 hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] d.~3 61513.836143: cobalt_shadow_gorelax: reason=fault hi-thread-2504 [000] d.~3 61513.836143: cobalt_lostage_request: request=wakeup pid=2504 comm=hi-thread hi-thread-2504 [000] *.~3 61513.836144: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] *.~3 61513.836144: cobalt_schedule: status=0x10000000 hi-thread-2504 [000] *.~3 61513.836144: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] *.~3 61513.836145: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 gdb-2495 [000] *.~3 61513.836146: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] d.h3 61513.836146: irq_handler_entry: irq=120 name=in-band work gdb-2495 [000] d.h3 61513.836147: cobalt_lostage_wakeup: pid=2504 comm=hi-thread gdb-2495 [000] d.h5 61513.836147: sched_waking: comm=hi-thread pid=2504 prio=97 target_cpu=000 gdb-2495 [000] dNh6 61513.836147: sched_wakeup: comm=hi-thread pid=2504 prio=97 target_cpu=000 gdb-2495 [000] dNh3 61513.836148: irq_handler_exit: irq=120 ret=handled gdb-2495 [000] dNh3 61513.836148: irq_handler_entry: irq=124 name=proxy tick gdb-2495 [000] dNh3 61513.836151: softirq_raise: vec=1 [action=TIMER] gdb-2495 [000] dNh3 61513.836152: softirq_raise: vec=9 [action=RCU] gdb-2495 [000] dNh4 61513.836152: sched_stat_runtime: comm=gdb pid=2495 runtime=1007474 [ns] vruntime=300515161229 [ns] gdb-2495 [000] *Nh3 61513.836154: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2395843 interval=0 mode=rel gdb-2495 [000] *Nh3 61513.836155: cobalt_tick_shot: next tick at 61513.838550 (delay: 2395 us) gdb-2495 [000] dNh3 61513.836155: irq_handler_exit: irq=124 ret=handled gdb-2495 [000] .Ns3 61513.836156: softirq_entry: vec=1 [action=TIMER] gdb-2495 [000] .Ns3 61513.836157: softirq_exit: vec=1 [action=TIMER] gdb-2495 [000] .Ns3 61513.836157: softirq_entry: vec=9 [action=RCU] gdb-2495 [000] dNs6 61513.836158: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000 gdb-2495 [000] dNs7 61513.836159: sched_stat_runtime: comm=gdb pid=2495 runtime=6881 [ns] vruntime=300515168110 [ns] gdb-2495 [000] dNs7 61513.836160: sched_stat_blocked: comm=rcu_sched pid=11 delay=4188739 [ns] gdb-2495 [000] dNs7 61513.836161: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000 gdb-2495 [000] .Ns3 61513.836164: softirq_exit: vec=9 [action=RCU] gdb-2495 [000] d..3 61513.836165: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=hi-thread next_pid=2504 next_prio=97 hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200 hi-thread-2504 [000] *..2 61513.836169: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] *..2 61513.836170: cobalt_thread_suspend: pid=2504 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] d..2 61513.836170: signal_generate: sig=5 errno=0 code=128 comm=hi-thread pid=2504 grp=0 res=0 hi-thread-2504 [000] d..3 61513.836173: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0 hi-thread-2504 [000] d..3 61513.836175: sched_stat_wait: comm=rcu_sched pid=11 delay=14490 [ns] hi-thread-2504 [000] d..3 61513.836175: sched_switch: prev_comm=hi-thread prev_pid=2504 prev_prio=97 prev_state=t ==> next_comm=rcu_sched next_pid=11 next_prio=120 rcu_sched-11 [000] d..3 61513.836179: sched_stat_runtime: comm=rcu_sched pid=11 runtime=4960 [ns] vruntime=300512173070 [ns] rcu_sched-11 [000] d..3 61513.836179: sched_stat_wait: comm=gdb pid=2495 delay=19450 [ns] rcu_sched-11 [000] d..3 61513.836180: sched_switch: prev_comm=rcu_sched prev_pid=11 prev_prio=120 prev_state=I ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] ...1 61513.836187: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cb0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836197: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836198: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7ae0) gdb-2495 [000] ...1 61513.836199: sys_newfstat -> 0x0 gdb-2495 [000] ...1 61513.836200: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) gdb-2495 [000] ...1 61513.836214: sys_read -> 0x3f1 gdb-2495 [000] ...1 61513.836217: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836217: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836221: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8) gdb-2495 [000] ...1 61513.836222: sys_rt_sigprocmask -> 0x0 gdb-2495 [000] d..2 61513.836223: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000 gdb-2495 [000] ...1 61513.836225: sys_read(fd: f, buf: 7ffe8b2a7667, count: 1) gdb-2495 [000] ...1 61513.836226: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.836227: sys_write(fd: 10, buf: 5585fe0354b1, count: 1) gdb-2495 [000] ...1 61513.836229: sys_write -> 0x1 gdb-2495 [000] ...1 61513.836229: sys_rt_sigreturn() gdb-2495 [000] ...1 61513.836233: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1) gdb-2495 [000] ...1 61513.836234: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.836235: sys_write(fd: 1, buf: 5585fe7a5290, count: 27) gdb-2495 [000] d..4 61513.836236: sched_waking: comm=smokey pid=2493 prio=120 target_cpu=000 gdb-2495 [000] d..5 61513.836237: sched_stat_runtime: comm=gdb pid=2495 runtime=58399 [ns] vruntime=300515226509 [ns] gdb-2495 [000] d..5 61513.836238: sched_stat_sleep: comm=smokey pid=2493 delay=1717179 [ns] gdb-2495 [000] dN.5 61513.836239: sched_wakeup: comm=smokey pid=2493 prio=120 target_cpu=000 gdb-2495 [000] .N.1 61513.836240: sys_write -> 0x27 gdb-2495 [000] dN.3 61513.836241: sched_stat_wait: comm=smokey pid=2493 delay=0 [ns] gdb-2495 [000] d..3 61513.836241: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2493 next_prio=120 smokey-2493 [000] ...1 61513.836244: sys_read -> 0x1 smokey-2493 [000] ..~1 61513.836246: cobalt_head_sysentry: syscall=read ... smokey-2493 [000] d..3 61513.836344: sched_stat_runtime: comm=smokey pid=2493 runtime=106636 [ns] vruntime=300512333145 [ns] smokey-2493 [000] d..3 61513.836344: sched_stat_wait: comm=gdb pid=2495 delay=106636 [ns] smokey-2493 [000] d..3 61513.836345: sched_switch: prev_comm=smokey prev_pid=2493 prev_prio=120 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] ...1 61513.836347: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0) gdb-2495 [000] ...1 61513.836348: sys_poll -> 0x1 gdb-2495 [000] ...1 61513.836350: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) gdb-2495 [000] ...1 61513.836351: sys_read -> 0x1 gdb-2495 [000] ...1 61513.836352: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) gdb-2495 [000] ...1 61513.836352: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.836353: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8) gdb-2495 [000] ...1 61513.836354: sys_rt_sigprocmask -> 0x0 gdb-2495 [000] ...1 61513.836355: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) gdb-2495 [000] ...1 61513.836355: sched_process_wait: comm=gdb pid=0 prio=120 gdb-2495 [000] ...1 61513.836357: sys_wait4 -> 0x9c8 gdb-2495 [000] ...1 61513.836360: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0) gdb-2495 [000] d..3 61513.836361: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836363: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836367: sys_ptrace(request: 4202, pid: 9c8, addr: 0, data: 7ffe8b2a7cb0) gdb-2495 [000] d..3 61513.836368: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836369: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836370: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6f10) gdb-2495 [000] d..3 61513.836370: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836371: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836372: sys_ptrace(request: d, pid: 9c8, addr: 0, data: 7ffe8b2a6f10) gdb-2495 [000] d..3 61513.836373: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836374: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836376: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) gdb-2495 [000] ...1 61513.836376: sched_process_wait: comm=gdb pid=0 prio=120 gdb-2495 [000] ...1 61513.836377: sys_wait4 -> 0x0 gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8) gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask -> 0x0 gdb-2495 [000] ...1 61513.836382: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cd0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836399: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836400: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7b50) gdb-2495 [000] ...1 61513.836401: sys_newfstat -> 0x0 gdb-2495 [000] ...1 61513.836402: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) gdb-2495 [000] ...1 61513.836409: sys_read -> 0x140 gdb-2495 [000] ...1 61513.836410: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400) gdb-2495 [000] ...1 61513.836410: sys_read -> 0x0 gdb-2495 [000] ...1 61513.836415: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836416: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836419: sys_read(fd: f, buf: 7ffe8b2a7f17, count: 1) gdb-2495 [000] ...1 61513.836420: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.836421: sys_write(fd: 10, buf: 5585fe0354b1, count: 1) gdb-2495 [000] ...1 61513.836422: sys_write -> 0x1 gdb-2495 [000] ...1 61513.836425: sys_ptrace(request: 1, pid: 9c8, addr: 7ffff777f370, data: 7ffe8b2a7878) gdb-2495 [000] d..3 61513.836426: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836428: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836430: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a7bd8) gdb-2495 [000] d..3 61513.836430: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836431: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836433: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a70e0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836446: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836446: sys_pread64(fd: 12, buf: 7ffe8b2a7430, count: 900, pos: 7ffff67c9700) gdb-2495 [000] ...1 61513.836449: sys_pread64 -> 0x900 gdb-2495 [000] ...1 61513.836450: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836450: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836479: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836488: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836489: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950) gdb-2495 [000] ...1 61513.836490: sys_newfstat -> 0x0 gdb-2495 [000] ...1 61513.836490: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400) gdb-2495 [000] ...1 61513.836498: sys_read -> 0x14d gdb-2495 [000] ...1 61513.836499: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) gdb-2495 [000] ...1 61513.836499: sys_read -> 0x0 gdb-2495 [000] ...1 61513.836503: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836504: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836507: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836514: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836515: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950) gdb-2495 [000] ...1 61513.836515: sys_newfstat -> 0x0 gdb-2495 [000] ...1 61513.836516: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) gdb-2495 [000] ...1 61513.836521: sys_read -> 0x13b gdb-2495 [000] ...1 61513.836522: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400) gdb-2495 [000] ...1 61513.836522: sys_read -> 0x0 gdb-2495 [000] ...1 61513.836525: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836525: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13) gdb-2495 [000] *..3 61513.836532: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000 gdb-2495 [000] *..3 61513.836532: cobalt_trace_pid: pid=2499, prio=1 gdb-2495 [000] *..3 61513.836533: cobalt_thread_unblock: pid=2499 state=0x48848 info=0x0 gdb-2495 [000] *.~3 61513.836586: cobalt_schedule: status=0x10000000 gdb-2495 [000] *.~3 61513.836586: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] *.~3 61513.836587: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1 smokey-2499 [000] *.~3 61513.836589: cobalt_trace_pid: pid=2499, prio=1 smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined smokey-2499 [000] *.~3 61513.836589: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey smokey-2499 [000] *.~3 61513.836590: cobalt_thread_suspend: pid=2499 mask=0x400080 timeout=0 timeout_mode=0 wchan=0000000000000000 smokey-2499 [000] *.~3 61513.836590: cobalt_schedule: status=0x10000000 smokey-2499 [000] *.~3 61513.836590: cobalt_trace_pid: pid=2499, prio=1 smokey-2499 [000] *.~3 61513.836591: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x4488c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 gdb-2495 [000] *.~3 61513.836592: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] d..5 61513.836592: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dN.6 61513.836594: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dN.3 61513.836594: signal_generate: sig=19 errno=0 code=-6 comm=smokey pid=2499 grp=0 res=0 gdb-2495 [000] dNh3 61513.836595: irq_handler_entry: irq=120 name=in-band work gdb-2495 [000] dNh3 61513.836596: cobalt_lostage_wakeup: pid=2499 comm=smokey gdb-2495 [000] *N~4 61513.838552: irq_pipeline_entry: irq=4354 gdb-2495 [000] *N~4 61513.838553: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt gdb-2495 [000] *N~4 61513.838554: cobalt_timer_expire: timer=00000000d13b8aa9 gdb-2495 [000] *N~4 61513.838555: irq_handler_exit: irq=4354 ret=handled gdb-2495 [000] *N~4 61513.838555: irq_pipeline_exit: irq=4354 gdb-2495 [000] dNh8 61513.981277: sched_waking: comm=systemd-journal pid=163 prio=120 target_cpu=000 gdb-2495 [000] dNh9 61513.981278: sched_stat_runtime: comm=gdb pid=2495 runtime=144934239 [ns] vruntime=300660160748 [ns] gdb-2495 [000] dNh9 61513.981280: sched_stat_sleep: comm=systemd-journal pid=163 delay=111416392895 [ns] gdb-2495 [000] dNh9 61513.981280: sched_wakeup: comm=systemd-journal pid=163 prio=120 target_cpu=000 gdb-2495 [000] dNh3 61513.981281: irq_handler_exit: irq=120 ret=handled gdb-2495 [000] dNh3 61513.981282: irq_handler_entry: irq=124 name=proxy tick gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=1 [action=TIMER] gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=9 [action=RCU] gdb-2495 [000] dNh4 61513.981286: sched_stat_runtime: comm=gdb pid=2495 runtime=8375 [ns] vruntime=300660169123 [ns] gdb-2495 [000] dNh5 61513.981288: sched_waking: comm=cobalt_printf pid=2494 prio=120 target_cpu=000 gdb-2495 [000] dNh6 61513.981289: sched_stat_runtime: comm=gdb pid=2495 runtime=2987 [ns] vruntime=300660172110 [ns] gdb-2495 [000] dNh6 61513.981291: sched_stat_sleep: comm=cobalt_printf pid=2494 delay=238660319 [ns] gdb-2495 [000] dNh6 61513.981291: sched_wakeup: comm=cobalt_printf pid=2494 prio=120 target_cpu=000 gdb-2495 [000] dNh5 61513.981292: sched_waking: comm=cobalt_printf pid=2503 prio=120 target_cpu=000 gdb-2495 [000] dNh6 61513.981292: sched_stat_runtime: comm=gdb pid=2495 runtime=3069 [ns] vruntime=300660175179 [ns] gdb-2495 [000] dNh6 61513.981294: sched_stat_sleep: comm=cobalt_printf pid=2503 delay=146611442 [ns] gdb-2495 [000] dNh6 61513.981294: sched_wakeup: comm=cobalt_printf pid=2503 prio=120 target_cpu=000 gdb-2495 [000] *Nh3 61513.981295: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=588476 interval=0 mode=rel gdb-2495 [000] *Nh3 61513.981296: cobalt_tick_shot: next tick at 61513.981884 (delay: 588 us) gdb-2495 [000] dNh3 61513.981296: irq_handler_exit: irq=124 ret=handled gdb-2495 [000] .Ns3 61513.981297: softirq_entry: vec=1 [action=TIMER] gdb-2495 [000] dNs5 61513.981298: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000 gdb-2495 [000] dNs6 61513.981298: sched_stat_runtime: comm=gdb pid=2495 runtime=6218 [ns] vruntime=300660181397 [ns] gdb-2495 [000] dNs6 61513.981299: sched_stat_blocked: comm=rcu_sched pid=11 delay=145119923 [ns] gdb-2495 [000] dNs6 61513.981299: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000 gdb-2495 [000] dNs7 61513.981302: sched_waking: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000 gdb-2495 [000] dNs8 61513.981302: sched_stat_runtime: comm=gdb pid=2495 runtime=3884 [ns] vruntime=300660185281 [ns] gdb-2495 [000] dNs8 61513.981303: sched_stat_blocked: comm=kworker/0:1 pid=1932 delay=206048652 [ns] gdb-2495 [000] dNs8 61513.981304: sched_wakeup: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000 gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981312: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] .Ns3 61513.981314: softirq_exit: vec=1 [action=TIMER] gdb-2495 [000] .Ns3 61513.981314: softirq_entry: vec=9 [action=RCU] gdb-2495 [000] .Ns3 61513.981315: softirq_exit: vec=9 [action=RCU] gdb-2495 [000] dNs5 61513.981316: sched_waking: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000 gdb-2495 [000] dNs6 61513.981316: sched_stat_runtime: comm=gdb pid=2495 runtime=13841 [ns] vruntime=300660199122 [ns] gdb-2495 [000] dNs6 61513.981317: sched_stat_sleep: comm=ksoftirqd/0 pid=10 delay=149349034 [ns] gdb-2495 [000] dNs6 61513.981317: sched_wakeup: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000 gdb-2495 [000] .N.1 61513.981318: sys_tkill -> 0x0 gdb-2495 [000] d..3 61513.981320: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] *.~3 61513.981888: irq_pipeline_entry: irq=4354 smokey-2499 [000] *.~3 61513.981888: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt smokey-2499 [000] *.~3 61513.981889: cobalt_timer_expire: timer=00000000d13b8aa9 smokey-2499 [000] *.~3 61513.981889: irq_handler_exit: irq=4354 ret=handled smokey-2499 [000] *.~3 61513.981890: irq_pipeline_exit: irq=4354 smokey-2499 [000] d.h1 61514.106435: irq_handler_entry: irq=124 name=proxy tick smokey-2499 [000] *.h1 61514.106441: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2109534 interval=0 mode=rel smokey-2499 [000] *.h1 61514.106441: cobalt_tick_shot: next tick at 61514.108550 (delay: 2109 us) smokey-2499 [000] d.h1 61514.106442: irq_handler_exit: irq=124 ret=handled smokey-2499 [000] ...1 61514.106442: cobalt_shadow_relaxed: state=0x4488c0 info=0x200 smokey-2499 [000] *.~1 61514.108552: irq_pipeline_entry: irq=4354 Regards Hongzhan Chen > >-- >Philippe. ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: gdb test failure debug status update 2021-04-30 5:51 ` Chen, Hongzhan @ 2021-04-30 7:36 ` Philippe Gerum 0 siblings, 0 replies; 16+ messages in thread From: Philippe Gerum @ 2021-04-30 7:36 UTC (permalink / raw) To: Chen, Hongzhan; +Cc: xenomai Chen, Hongzhan <hongzhan.chen@intel.com> writes: > After applied following patch , the gdb test failure issue seems fixed. Basically , it enforce going back to > primary mode after check that XNCONTHI is set for current thread. It seems ugly. Please suggest different > solution. Agreed, this is ugly. This code is switching the current task back to oob mode right after it forced it to transition from oob to in-band mode. I'm not sure why that works in the end, we need to explain the mayday issue first in order to figure out. > > @@ -355,9 +357,36 @@ static int handle_user_return(struct task_struct *task) > > void handle_oob_mayday(struct pt_regs *regs) > { > + struct xnthread *thread; > + spl_t s; > + int err; > + > XENO_BUG_ON(COBALT, !xnthread_test_state(xnthread_current(), XNUSER)); > > + thread = xnthread_current(); > + > + trace_cobalt_handle_oob_mayday(thread); > + > xnthread_relax(0, 0); > + > + if (xnthread_test_info(thread, XNCONTHI)) { > + xnlock_get_irqsave(&nklock, s); > + xnthread_clear_info(thread, XNCONTHI); > + xnlock_put_irqrestore(&nklock, s); > + > + err = xnthread_harden(); > + > + /* > + * XNCONTHI may or may not have been re-applied if > + * harden bailed out due to pending signals. Make sure > + * it is set in that case. > + */ > + if (err == -ERESTARTSYS) { > + xnlock_get_irqsave(&nklock, s); > + xnthread_set_info(thread, XNCONTHI); > + xnlock_put_irqrestore(&nklock, s); > + } > + } > } > > > Regards > > Hongzhan Chen > > -----Original Message----- > From: Chen, Hongzhan <hongzhan.chen@intel.com> > Sent: Friday, April 30, 2021 1:26 PM > To: Chen, Hongzhan <hongzhan.chen@intel.com>; Philippe Gerum <rpm@xenomai.org>; xenomai@xenomai.org > Subject: RE: gdb test failure debug status update > > The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish > ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax. > That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in > arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from > xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to > call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode > because kernel exit code is never called in this case? > > Regards > > Hongzhan Chen > > -----Original Message----- > From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai > Sent: Thursday, April 29, 2021 2:31 PM > To: Philippe Gerum <rpm@xenomai.org> > Cc: xenomai@xenomai.org > Subject: RE: gdb test failure debug status update > > >> >>-----Original Message----- >>From: Philippe Gerum <rpm@xenomai.org> >>Sent: Wednesday, April 28, 2021 10:30 PM >>To: Philippe Gerum <rpm@xenomai.org> >>Cc: Chen, Hongzhan <hongzhan.chen@intel.com>; xenomai@xenomai.org >>Subject: Re: gdb test failure debug status update >> >> >>Philippe Gerum via Xenomai <xenomai@xenomai.org> writes: >> >>> Chen, Hongzhan via Xenomai <xenomai@xenomai.org> writes: >>> >>>> According to my validation, gdb test fail on dovetail 5.10 branch but pass on v5.9-evl4 tag with same for-upstream/dovetail >>>> xenomai code base. >>>> >>>> After further debug , the issue is more clear for me. Gdb test failure because low priority thread smokey userspace is still >>>> executed after "cobalt_shadow_relaxed: state=0x4488c0 info=0x200" like log [1] on dovetail-5.10 branch. >>>> The weird thing is that its following first ftrace log happen at 62235.848583 after cobalt_shadow_relaxed in log [1]. >>>> It is almost 3ms happened after cobalt_shadow_relaxed. The low priority smoke thread user space is executed during this >>>> 3ms period so that test fail. >>>> >>>> But in success case with v5.9-evl4 like in log [2], the time interval between cobalt_shadow_relaxed and the following first ftrace log >>>> is only about 1us. It seems that low priority smokey userspace do not have chance to execute in this 1us because gdb test is successful. >>>> >>>> My question is why there is even no interrupt happened during that about 3ms period in failure case? Tick seems in abnormal behavior. >>>> Please comment if you have any ideas to further debug it. >>>> >>>> PS: All my tests run on same up Xtream board. >>> >>> <snip> >>> >>> Let's put aside the tick issue for now, there may be a valid reason for >>> this delay with dynticks enabled. >>> >>> The issue at stake may be related to the way a return to kernel space is >>> forced on a @user task (Dovetail has an integrated service for >>> triggering this called dovetail_request_ucall()). >>> >>> The logic for doing so is as follows: >>> >>> 1. @user hits a breakpoint, which is an exception Dovetail-wise >>> >>> 2. @user gets XNDBGSTOP set into its flags because Cobalt notices it is >>> being debugged via a breakpoint trap, then relaxed as a result of taking >>> a exception in general, so that we may traverse the common trap handling code >>> safely. >>> >>> 3. since XNDBGSTOP is a blocking bit Cobalt-wise, it should prevent >>> @user from being picked for scheduling by the real-time core, next time >>> a Cobalt considers rescheduling that is. However, since @user is >>> currently relaxed, it can still run under the supervision of the common >>> Linux scheduler. This is what the log[1] show. >>> >>> 4. the common/in-band kernel code stops @user due to the ptrace stop >>> condition caused by the breakpoint, waiting for a continuation event to >>> happen. >>> >>> Therefore, upon PTRACE_CONT (i.e. gdb continue), we need to force @user >>> to call back into kernel context (handle_ptrace_cont -> >>> dovetail_request_ucall), then ask for a switch to primary mode from >>> there, which should eventually happen when @user is about to leave the >>> kernel (on x86, this now happens from a generic kernel entry/exit code >>> in kernel/entry/*). As a result, handle_taskexit_event() runs, figures >> >>Not quite: >>s,handle_taskexit_event,handle_user_return, > > There is trace_cobalt_shadow_unmap in handle_taskexit_event. I added > trace_cobalt_handle_user_return and trace_cobalt_handle_ptrace_cont > like patch [3] in log [4]. > > 1. In the log [4], the first cobalt_handle_ptrace_cont is followed by cobalt_handle_user_return > that clear info flag XNCONTHI for smokey-2499 because cobalt_handle_ptrace_cont called dovetail_request_ucall . > the TIF_RETUSER bit was cleared for smokey-2499. > After then 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000 > clear state flag XNDBGSTOP. After that, thread smokey-2499 run without these bit set till > "hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000". > > 2. There is second cobalt_handle_ptrace_cont called for smokey-2499 but at that time dovetail_request_ucall > would not be called because XNSSTEP Is not set like "smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0" > > 3. hi-thread-2504 breakpoint at its first breakpoint_target so it trigger hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3 > It call dovetail_request_ucall(current); for itself and also call xnthread_set_info(thread, XNCONTHI); That is why we can see this bit set in > hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200 > > 4. finally ,gdb call sys_tkill in gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13) and trigger that handle_oob_mayday is called so that xnthread_relax for > smokey-2499 Is called in "smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined." > > > It seems that there never call dovetail_request_ucall for process smokey-2499 in log[4] so that final xhthread_relax would have chance to return to user? > The second handle_ptrace_cont happen too much earlier because XNSSTEP still have not been set so that system never have change to call dovetail_request_ucall > for smokey-2499. > In this case , do we need to do hack in 3 , when handle_oob_trap_entry is called for hi-thread-2504 we call xnthread_set_info(thread, XNCONTHI); > and dovetail_request_ucall for all threads that have same cobalt_process? Is this solution OK for fixing this case? > > > [3]: > index 07dcc7a63..cd76961f5 100644 > --- a/kernel/cobalt/dovetail/kevents.c > +++ b/kernel/cobalt/dovetail/kevents.c > @@ -331,6 +331,8 @@ static int handle_user_return(struct task_struct *task) > if (thread == NULL) > return KEVENT_PROPAGATE; > > + trace_cobalt_handle_user_return(thread); > + > if (xnthread_test_info(thread, XNCONTHI)) { > xnlock_get_irqsave(&nklock, s); > xnthread_clear_info(thread, XNCONTHI); > @@ -468,6 +470,8 @@ int handle_ptrace_resume(struct task_struct *tracee) > if (thread == NULL) > return KEVENT_PROPAGATE; > > + trace_cobalt_handle_ptrace_resume(thread); > + > if (xnthread_test_state(thread, XNSSTEP)) { > xnlock_get_irqsave(&nklock, s); > > @@ -485,6 +489,8 @@ static void handle_ptrace_cont(void) > struct xnthread *curr = xnthread_current(); > spl_t s; > > + trace_cobalt_handle_ptrace_cont(curr); > + > xnlock_get_irqsave(&nklock, s); > > if (xnthread_test_state(curr, XNSSTEP)) { > @@ -604,6 +610,7 @@ void resume_oob_task(struct task_struct *p) /* inband, oob stage stalled */ > if (unlikely(xnthread_test_info(thread, XNCONTHI))) { > xnthread_clear_info(thread, XNCONTHI); > dovetail_clear_ucall(); > + trace_cobalt_resume_oob_task(thread); > } > > /* Unregister as debugged thread in case we postponed this. */ > diff --git a/kernel/cobalt/trace/cobalt-core.h b/kernel/cobalt/trace/cobalt-core.h > index 78f66a106..88a4067fd 100644 > --- a/kernel/cobalt/trace/cobalt-core.h > +++ b/kernel/cobalt/trace/cobalt-core.h > @@ -459,6 +459,27 @@ DEFINE_EVENT(curr_thread_event, cobalt_thread_set_mode, > TP_ARGS(thread) > ); > > +DEFINE_EVENT(thread_event, cobalt_handle_ptrace_cont, > + TP_PROTO(struct xnthread *thread), > > [4]: > gdb-2495 [000] d..3 61513.834825: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 > smokey-2499 [000] ...1 61513.834827: cobalt_handle_ptrace_cont: pid=2499 state=0x4488c0 info=0x200 > smokey-2499 [000] d..1 61513.834828: cobalt_handle_user_return: pid=2499 state=0x4488c0 info=0x200 > smokey-2499 [000] d..1 61513.834828: cobalt_shadow_gohard: state=0x4488c0 info=0x0 > smokey-2499 [000] d..3 61513.834830: sched_stat_wait: comm=gdb pid=2495 delay=8017 [ns] > smokey-2499 [000] d..3 61513.834831: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 > gdb-2495 [000] *..3 61513.834831: cobalt_thread_resume: name=smokey pid=2499 mask=0x80 > gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1 > gdb-2495 [000] *..3 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000 > gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1 > gdb-2495 [000] *.~3 61513.834833: cobalt_schedule: status=0x10000000 > gdb-2495 [000] *.~3 61513.834833: cobalt_trace_pid: pid=2495, prio=-1 > gdb-2495 [000] *.~3 61513.834833: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1 > smokey-2499 [000] d.~3 61513.834835: cobalt_shadow_hardened: state=0x48040 info=0x0 > smokey-2499 [000] d.~3 61513.834836: cobalt_head_sysentry: syscall=sem_init > smokey-2499 [000] d.~3 61513.834839: cobalt_psem_init: sem=0x4e(anon) flags=0x0() value=0 > smokey-2499 [000] d.~3 61513.834839: cobalt_head_sysexit: result=0 > smokey-2499 [000] d.~3 61513.834848: cobalt_shadow_gorelax: reason=syscall > smokey-2499 [000] d.~3 61513.834848: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey > smokey-2499 [000] *.~3 61513.834848: cobalt_thread_suspend: pid=2499 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 > smokey-2499 [000] *.~3 61513.834849: cobalt_schedule: status=0x10000000 > smokey-2499 [000] *.~3 61513.834849: cobalt_trace_pid: pid=2499, prio=1 > smokey-2499 [000] *.~3 61513.834849: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 > gdb-2495 [000] *.~3 61513.834850: cobalt_trace_pid: pid=2495, prio=-1 > gdb-2495 [000] d.h3 61513.834851: irq_handler_entry: irq=120 name=in-band work > gdb-2495 [000] d.h3 61513.834852: cobalt_lostage_wakeup: pid=2499 comm=smokey > gdb-2495 [000] d.h5 61513.834852: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 > gdb-2495 [000] dNh6 61513.834853: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 > gdb-2495 [000] dNh3 61513.834853: irq_handler_exit: irq=120 ret=handled > gdb-2495 [000] dN.3 61513.834854: sched_stat_runtime: comm=gdb pid=2495 runtime=23565 [ns] vruntime=300514015543 [ns] > gdb-2495 [000] d..3 61513.834855: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 > smokey-2499 [000] ...1 61513.834856: cobalt_shadow_relaxed: state=0x480c0 info=0x0 > smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min(policy: 1) > smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min -> 0x1 > smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max(policy: 1) > smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max -> 0x63 > smokey-2499 [000] ..~1 61513.834862: cobalt_head_sysentry: syscall=thread_getschedparam_ex > smokey-2499 [000] ...1 61513.834862: cobalt_root_sysentry: syscall=thread_getschedparam_ex > smokey-2499 [000] ...1 61513.834863: cobalt_pthread_getschedparam: pth=000000008d36be1c policy=fifo param={ priority=1 } > smokey-2499 [000] ...1 61513.834864: cobalt_root_sysexit: result=0 > smokey-2499 [000] ...1 61513.834872: sys_mmap(addr: 0, len: 11000, prot: 0, flags: 20022, fd: ffffffff, off: 0) > smokey-2499 [000] ...1 61513.834877: sys_mmap -> 0x7ffff67b9000 > smokey-2499 [000] ...1 61513.834878: sys_mprotect(start: 7ffff67ba000, len: 10000, prot: 3) > smokey-2499 [000] ...1 61513.834906: sys_mprotect -> 0x0 > smokey-2499 [000] ...1 61513.834909: sys_clone(clone_flags: 3d0f00, newsp: 7ffff67c8fb0, parent_tidptr: 7ffff67c99d0, child_tidptr: 7ffff67c99d0, tls: 7ffff67c9700) > smokey-2499 [000] ...1 61513.834923: task_newtask: pid=2504 comm=smokey clone_flags=3d0f00 oom_score_adj=0 > smokey-2499 [000] ...1 61513.834923: sched_process_fork: comm=smokey pid=2499 child_comm=smokey child_pid=2504 > smokey-2499 [000] d..3 61513.834925: sched_wakeup_new: comm=smokey pid=2504 prio=98 target_cpu=000 > smokey-2499 [000] d..3 61513.834928: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0 > smokey-2499 [000] d..3 61513.834931: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=t ==> next_comm=smokey next_pid=2504 next_prio=98 > hi-thread-2504 [000] ...1 61513.834932: sys_clone -> 0x0 > hi-thread-2504 [000] d..3 61513.834934: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=2 > hi-thread-2504 [000] d..3 61513.834935: sched_stat_wait: comm=gdb pid=2495 delay=82002 [ns] > hi-thread-2504 [000] d..3 61513.834936: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=t ==> next_comm=gdb next_pid=2495 next_prio=120 > gdb-2495 [000] d..2 61513.834937: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000 > gdb-2495 [000] ...1 61513.834940: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1) > gdb-2495 [000] ...1 61513.834943: sys_read -> 0x1 > gdb-2495 [000] ...1 61513.834943: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1) > gdb-2495 [000] ...1 61513.834944: sys_read -> 0xfffffffffffffff5 > gdb-2495 [000] ...1 61513.834944: sys_write(fd: 10, buf: 5585fe0354b1, count: 1) > gdb-2495 [000] ...1 61513.834946: sys_write -> 0x1 > gdb-2495 [000] ...1 61513.834947: sys_rt_sigreturn() > gdb-2495 [000] ...1 61513.834953: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1) > gdb-2495 [000] ...1 61513.834953: sys_read -> 0xfffffffffffffff5 > gdb-2495 [000] ...1 61513.834955: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0) > gdb-2495 [000] ...1 61513.834956: sys_poll -> 0x1 > gdb-2495 [000] ...1 61513.834957: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) > gdb-2495 [000] ...1 61513.834958: sys_read -> 0x1 > gdb-2495 [000] ...1 61513.834959: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) > gdb-2495 [000] ...1 61513.834960: sys_read -> 0xfffffffffffffff5 > gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8) > gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask -> 0x0 > gdb-2495 [000] ...1 61513.834962: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) > gdb-2495 [000] ...1 61513.834962: sched_process_wait: comm=gdb pid=0 prio=120 > gdb-2495 [000] ...1 61513.834964: sys_wait4 -> 0x9c3 > gdb-2495 [000] ...1 61513.834965: sys_ptrace(request: 4201, pid: 9c3, addr: 0, data: 7ffe8b2a7d08) > gdb-2495 [000] d..3 61513.834966: sched_wait_task: comm=smokey pid=2499 prio=98 > gdb-2495 [000] ...1 61513.834967: sys_ptrace -> 0x0 > gdb-2495 [000] ...1 61513.834967: sys_wait4(upid: 9c8, stat_addr: 7ffe8b2a7cfc, options: 40000000, ru: 0) > gdb-2495 [000] ...1 61513.834968: sched_process_wait: comm=gdb pid=2504 prio=120 > gdb-2495 [000] ...1 61513.834969: sys_wait4 -> 0x9c8 > gdb-2495 [000] ...1 61513.834972: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578) > gdb-2495 [000] d..3 61513.834972: sched_wait_task: comm=smokey pid=2499 prio=98 > gdb-2495 [000] ...1 61513.834975: sys_ptrace -> 0x0 > gdb-2495 [000] ...1 61513.834976: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a78d8) > gdb-2495 [000] d..3 61513.834977: sched_wait_task: comm=smokey pid=2504 prio=98 > gdb-2495 [000] ...1 61513.834977: sys_ptrace -> 0x0 > gdb-2495 [000] ...1 61513.834979: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0) > gdb-2495 [000] ...1 61513.834989: sys_openat -> 0x12 > gdb-2495 [000] ...1 61513.834990: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff67c9700) > gdb-2495 [000] ...1 61513.834992: sys_pread64 -> 0x900 > gdb-2495 [000] ...1 61513.834993: sys_close(fd: 12) > gdb-2495 [000] ...1 61513.834994: sys_close -> 0x0 > gdb-2495 [000] ...1 61513.835010: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578) > gdb-2495 [000] d..3 61513.835011: sched_wait_task: comm=smokey pid=2499 prio=98 > gdb-2495 [000] ...1 61513.835012: sys_ptrace -> 0x0 > gdb-2495 [000] ...1 61513.835013: sys_ptrace(request: 3, pid: 9c3, addr: a8, data: 7ffe8b2a78d8) > gdb-2495 [000] d..3 61513.835013: sched_wait_task: comm=smokey pid=2499 prio=98 > gdb-2495 [000] ...1 61513.835014: sys_ptrace -> 0x0 > gdb-2495 [000] ...1 61513.835015: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0) > gdb-2495 [000] ...1 61513.835022: sys_openat -> 0x12 > gdb-2495 [000] ...1 61513.835022: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff758f740) > gdb-2495 [000] ...1 61513.835025: sys_pread64 -> 0x900 > gdb-2495 [000] ...1 61513.835025: sys_close(fd: 12) > gdb-2495 [000] ...1 61513.835026: sys_close -> 0x0 > gdb-2495 [000] ...1 61513.835031: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) > gdb-2495 [000] ...1 61513.835032: sched_process_wait: comm=gdb pid=0 prio=120 > gdb-2495 [000] ...1 61513.835033: sys_wait4 -> 0x0 > gdb-2495 [000] ...1 61513.835036: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0) > gdb-2495 [000] d..3 61513.835037: sched_wait_task: comm=smokey pid=2504 prio=98 > gdb-2495 [000] ...1 61513.835039: sys_ptrace -> 0x0 > gdb-2495 [000] ...1 61513.835043: sys_ptrace(request: 6, pid: 9c8, addr: 388, data: 0) > gdb-2495 [000] d..3 61513.835044: sched_wait_task: comm=smokey pid=2504 prio=98 > gdb-2495 [000] ...1 61513.835044: sys_ptrace -> 0x0 > gdb-2495 [000] ...1 61513.835045: sys_ptrace(request: 7, pid: 9c8, addr: 1, data: 0) > gdb-2495 [000] d..3 61513.835046: sched_wait_task: comm=smokey pid=2504 prio=98 > gdb-2495 [000] d..3 61513.835047: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000 > gdb-2495 [000] dN.4 61513.835048: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000 > gdb-2495 [000] .N.1 61513.835048: sys_ptrace -> 0x0 > gdb-2495 [000] dN.3 61513.835050: sched_stat_runtime: comm=gdb pid=2495 runtime=112561 [ns] vruntime=300514128104 [ns] > gdb-2495 [000] d..3 61513.835051: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=98 > hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list(head: 7ffff67c99e0, len: 18) > hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list -> 0x0 > hi-thread-2504 [000] ...1 61513.835056: sys_futex(uaddr: 7ffff67c9d18, op: 80, val: 2, utime: 0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700) > hi-thread-2504 [000] d..3 61513.835059: sched_stat_wait: comm=gdb pid=2495 delay=10883 [ns] > hi-thread-2504 [000] d..3 61513.835060: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 > gdb-2495 [000] ...1 61513.835063: sys_ptrace(request: c, pid: 9c3, addr: 0, data: 7ffe8b2a6ef0) > gdb-2495 [000] d..3 61513.835064: sched_wait_task: comm=smokey pid=2499 prio=98 > gdb-2495 [000] ...1 61513.835065: sys_ptrace -> 0x0 > gdb-2495 [000] ...1 61513.835068: sys_ptrace(request: 7, pid: 9c3, addr: 1, data: 0) > gdb-2495 [000] d..3 61513.835069: sched_wait_task: comm=smokey pid=2499 prio=98 > gdb-2495 [000] d..3 61513.835069: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 > gdb-2495 [000] dN.4 61513.835070: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 > gdb-2495 [000] .N.1 61513.835071: sys_ptrace -> 0x0 > gdb-2495 [000] dN.3 61513.835071: sched_stat_runtime: comm=gdb pid=2495 runtime=12210 [ns] vruntime=300514140314 [ns] > gdb-2495 [000] d..3 61513.835072: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 > smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0 > smokey-2499 [000] ...1 61513.835074: sys_clone -> 0x9c8 > smokey-2499 [000] ...1 61513.835075: sys_sched_setaffinity(pid: 9c8, len: 80, user_mask_ptr: 55555563e7c0) > smokey-2499 [000] ...1 61513.835077: sys_sched_setaffinity -> 0x0 > smokey-2499 [000] ...1 61513.835077: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 0, val3: 7ffff67c9700) > smokey-2499 [000] d..3 61513.835078: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000 > smokey-2499 [000] d..4 61513.835079: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000 > smokey-2499 [000] ...1 61513.835080: sys_futex -> 0x1 > smokey-2499 [000] ...1 61513.835086: sys_futex(uaddr: 7fffffffe828, op: 189, val: 0, utime: 7fffffffe780, uaddr2: 0, val3: ffffffff) > smokey-2499 [000] d..3 61513.835089: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=smokey next_pid=2504 next_prio=98 > hi-thread-2504 [000] ...1 61513.835090: sys_futex -> 0x0 > hi-thread-2504 [000] ...1 61513.835090: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700) > hi-thread-2504 [000] ...1 61513.835091: sys_futex -> 0x0 > hi-thread-2504 [000] ...1 61513.835092: sys_sched_setscheduler(pid: 9c8, policy: 1, param: 7ffff67c8e70) > hi-thread-2504 [000] ...1 61513.835095: sys_sched_setscheduler -> 0x0 > hi-thread-2504 [000] ..~1 61513.835096: cobalt_head_sysentry: syscall=thread_create > hi-thread-2504 [000] ...1 61513.835097: cobalt_root_sysentry: syscall=thread_create > hi-thread-2504 [000] ...1 61513.835098: cobalt_pthread_create: pth=00000000ba35a01d policy=fifo param={ priority=2 } > hi-thread-2504 [000] ...1 61513.835099: cobalt_timer_migrate: timer=000000004d21df54 cpu=0 > hi-thread-2504 [000] ...1 61513.835100: cobalt_timer_migrate: timer=00000000c1301853 cpu=0 > hi-thread-2504 [000] ...1 61513.835101: cobalt_thread_set_current_prio: thread=00000000e9fda3c4 pid=-1 prio=2 > hi-thread-2504 [000] ...1 61513.835102: cobalt_thread_init: thread=00000000e9fda3c4 name=smokey flags=0x48000 class=rt prio=2 > hi-thread-2504 [000] ...1 61513.835104: cobalt_shadow_map: thread=00000000e9fda3c4 pid=2504 prio=2 > hi-thread-2504 [000] D..1 61513.835104: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 > hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_start: pid=2504 state=0x480d0 info=0x0 > hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_resume: name=smokey pid=2504 mask=0x10 > hi-thread-2504 [000] D..1 61513.835106: cobalt_trace_pid: pid=2504, prio=2 > hi-thread-2504 [000] ...1 61513.835106: cobalt_trace_pid: pid=2504, prio=2 > hi-thread-2504 [000] ...1 61513.835107: cobalt_shadow_gohard: state=0x480c0 info=0x0 > hi-thread-2504 [000] d..3 61513.835108: sched_stat_wait: comm=gdb pid=2495 delay=37231 [ns] > hi-thread-2504 [000] d..3 61513.835109: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 > gdb-2495 [000] *..3 61513.835109: cobalt_thread_resume: name=smokey pid=2504 mask=0x80 > gdb-2495 [000] *..3 61513.835110: cobalt_trace_pid: pid=2504, prio=2 > gdb-2495 [000] *.~3 61513.835110: cobalt_schedule: status=0x10000000 > gdb-2495 [000] *.~3 61513.835110: cobalt_trace_pid: pid=2495, prio=-1 > gdb-2495 [000] *.~3 61513.835111: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2 > hi-thread-2504 [000] d.~3 61513.835112: cobalt_shadow_hardened: state=0x48040 info=0x0 > hi-thread-2504 [000] d.~3 61513.835112: cobalt_root_sysexit: result=0 > hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysentry: syscall=get_current > hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysexit: result=0 > hi-thread-2504 [000] d.~3 61513.835116: cobalt_shadow_gorelax: reason=syscall > hi-thread-2504 [000] d.~3 61513.835117: cobalt_lostage_request: request=wakeup pid=2504 comm=smokey > hi-thread-2504 [000] *.~3 61513.835117: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 > hi-thread-2504 [000] *.~3 61513.835117: cobalt_schedule: status=0x10000000 > hi-thread-2504 [000] *.~3 61513.835118: cobalt_trace_pid: pid=2504, prio=2 > hi-thread-2504 [000] *.~3 61513.835118: cobalt_switch_context: prev_name=smokey prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 > gdb-2495 [000] *.~3 61513.835119: cobalt_trace_pid: pid=2495, prio=-1 > gdb-2495 [000] d.h3 61513.835120: irq_handler_entry: irq=120 name=in-band work > gdb-2495 [000] d.h3 61513.835120: cobalt_lostage_wakeup: pid=2504 comm=smokey > gdb-2495 [000] d.h5 61513.835120: sched_waking: comm=smokey pid=2504 prio=97 target_cpu=000 > gdb-2495 [000] dNh6 61513.835121: sched_wakeup: comm=smokey pid=2504 prio=97 target_cpu=000 > gdb-2495 [000] dNh3 61513.835122: irq_handler_exit: irq=120 ret=handled > gdb-2495 [000] dN.3 61513.835122: sched_stat_runtime: comm=gdb pid=2495 runtime=13441 [ns] vruntime=300514153755 [ns] > gdb-2495 [000] d..3 61513.835123: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=97 > hi-thread-2504 [000] ...1 61513.835124: cobalt_shadow_relaxed: state=0x480c0 info=0x0 > hi-thread-2504 [000] ...1 61513.835125: sys_futex(uaddr: 7fffffffe828, op: 81, val: 1, utime: 0, uaddr2: 7ffff67c8e74, val3: 7fffffffe828) > hi-thread-2504 [000] d..3 61513.835126: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 > hi-thread-2504 [000] d..4 61513.835126: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 > hi-thread-2504 [000] ...1 61513.835127: sys_futex -> 0x1 > hi-thread-2504 [000] ..~1 61513.835128: cobalt_head_sysentry: syscall=migrate > hi-thread-2504 [000] ...1 61513.835128: cobalt_root_sysentry: syscall=migrate > hi-thread-2504 [000] ...1 61513.835128: cobalt_shadow_gohard: state=0x480c0 info=0x0 > hi-thread-2504 [000] d..3 61513.835130: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=smokey next_pid=2499 next_prio=98 > smokey-2499 [000] *..3 61513.835130: cobalt_thread_resume: name=smokey pid=2504 mask=0x80 > smokey-2499 [000] *..3 61513.835130: cobalt_trace_pid: pid=2504, prio=2 > smokey-2499 [000] *.~3 61513.835131: cobalt_schedule: status=0x10000000 > smokey-2499 [000] *.~3 61513.835131: cobalt_trace_pid: pid=2499, prio=-1 > smokey-2499 [000] *.~3 61513.835131: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2 > hi-thread-2504 [000] d.~3 61513.835132: cobalt_shadow_hardened: state=0x48040 info=0x0 > hi-thread-2504 [000] d.~3 61513.835132: cobalt_root_sysexit: result=1 > hi-thread-2504 [000] d.~3 61513.835133: cobalt_head_sysentry: syscall=thread_setname > hi-thread-2504 [000] d.~3 61513.835134: cobalt_pthread_setname: pth=00000000ba35a01d name=hi-thread > hi-thread-2504 [000] d.~3 61513.835135: cobalt_head_sysexit: result=0 > hi-thread-2504 [000] d.~3 61513.835136: cobalt_head_sysentry: syscall=clock_nanosleep > hi-thread-2504 [000] d.~3 61513.835137: cobalt_clock_nanosleep: clock_id=0 flags=0x0() rqt=(0.001000000) > hi-thread-2504 [000] *.~3 61513.835138: cobalt_thread_suspend: pid=2504 mask=0x4 timeout=1000001 timeout_mode=0 wchan=0000000000000000 > hi-thread-2504 [000] *.~3 61513.835138: cobalt_timer_start: timer=000000004d21df54(smokey) value=1000001 interval=0 mode=rel > hi-thread-2504 [000] *.~3 61513.835138: cobalt_schedule: status=0x10000000 > hi-thread-2504 [000] *.~3 61513.835139: cobalt_trace_pid: pid=2504, prio=2 > hi-thread-2504 [000] *.~3 61513.835139: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x48044 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 > smokey-2499 [000] *.~3 61513.835140: cobalt_trace_pid: pid=2499, prio=-1 > smokey-2499 [000] ...1 61513.835141: sys_futex -> 0x0 > smokey-2499 [000] ..~1 61513.835144: cobalt_head_sysentry: syscall=migrate > smokey-2499 [000] ...1 61513.835144: cobalt_root_sysentry: syscall=migrate > smokey-2499 [000] ...1 61513.835144: cobalt_shadow_gohard: state=0x480c0 info=0x0 > smokey-2499 [000] d..3 61513.835145: sched_stat_wait: comm=gdb pid=2495 delay=23982 [ns] > smokey-2499 [000] d..3 61513.835146: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 > gdb-2495 [000] *..3 61513.835147: cobalt_thread_resume: name=smokey pid=2499 mask=0x80 > gdb-2495 [000] *..3 61513.835147: cobalt_trace_pid: pid=2499, prio=1 > gdb-2495 [000] *.~3 61513.835147: cobalt_schedule: status=0x10000000 > gdb-2495 [000] *.~3 61513.835147: cobalt_trace_pid: pid=2495, prio=-1 > gdb-2495 [000] *.~3 61513.835148: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1 > smokey-2499 [000] d.~3 61513.835148: cobalt_shadow_hardened: state=0x48040 info=0x0 > smokey-2499 [000] d.~3 61513.835149: cobalt_root_sysexit: result=1 > smokey-2499 [000] *.~3 61513.835219: irq_pipeline_entry: irq=4354 > smokey-2499 [000] *.~3 61513.835219: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt > smokey-2499 [000] *.~3 61513.835220: cobalt_timer_expire: timer=00000000d13b8aa9 > smokey-2499 [000] *.~3 61513.835220: cobalt_tick_shot: next tick at 61513.836136 (delay: 915 us) > smokey-2499 [000] *.~3 61513.835221: irq_handler_exit: irq=4354 ret=handled > smokey-2499 [000] *.~3 61513.835221: irq_pipeline_exit: irq=4354 > smokey-2499 [000] *.~3 61513.836137: irq_pipeline_entry: irq=4354 > smokey-2499 [000] *.~3 61513.836137: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt > smokey-2499 [000] *.~3 61513.836138: cobalt_timer_expire: timer=000000004d21df54 > smokey-2499 [000] *.~3 61513.836138: cobalt_thread_resume: name=hi-thread pid=2504 mask=0x4 > smokey-2499 [000] *.~3 61513.836138: cobalt_trace_pid: pid=2504, prio=2 > smokey-2499 [000] *.~3 61513.836139: cobalt_timer_stop: timer=000000004d21df54 > smokey-2499 [000] *.~3 61513.836139: irq_handler_exit: irq=4354 ret=handled > smokey-2499 [000] *.~3 61513.836139: irq_pipeline_exit: irq=4354 > smokey-2499 [000] *.~3 61513.836140: cobalt_schedule: status=0x10000000 > smokey-2499 [000] *.~3 61513.836140: cobalt_trace_pid: pid=2499, prio=1 > smokey-2499 [000] *.~3 61513.836140: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x48048 ==> next_name=hi-thread next_pid=2504 next_prio=2 > hi-thread-2504 [000] *.~3 61513.836141: cobalt_trace_pid: pid=2504, prio=2 > hi-thread-2504 [000] d.~3 61513.836141: cobalt_head_sysexit: result=0 > hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3 > hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000 > hi-thread-2504 [000] d.~3 61513.836143: cobalt_shadow_gorelax: reason=fault > hi-thread-2504 [000] d.~3 61513.836143: cobalt_lostage_request: request=wakeup pid=2504 comm=hi-thread > hi-thread-2504 [000] *.~3 61513.836144: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 > hi-thread-2504 [000] *.~3 61513.836144: cobalt_schedule: status=0x10000000 > hi-thread-2504 [000] *.~3 61513.836144: cobalt_trace_pid: pid=2504, prio=2 > hi-thread-2504 [000] *.~3 61513.836145: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 > gdb-2495 [000] *.~3 61513.836146: cobalt_trace_pid: pid=2495, prio=-1 > gdb-2495 [000] d.h3 61513.836146: irq_handler_entry: irq=120 name=in-band work > gdb-2495 [000] d.h3 61513.836147: cobalt_lostage_wakeup: pid=2504 comm=hi-thread > gdb-2495 [000] d.h5 61513.836147: sched_waking: comm=hi-thread pid=2504 prio=97 target_cpu=000 > gdb-2495 [000] dNh6 61513.836147: sched_wakeup: comm=hi-thread pid=2504 prio=97 target_cpu=000 > gdb-2495 [000] dNh3 61513.836148: irq_handler_exit: irq=120 ret=handled > gdb-2495 [000] dNh3 61513.836148: irq_handler_entry: irq=124 name=proxy tick > gdb-2495 [000] dNh3 61513.836151: softirq_raise: vec=1 [action=TIMER] > gdb-2495 [000] dNh3 61513.836152: softirq_raise: vec=9 [action=RCU] > gdb-2495 [000] dNh4 61513.836152: sched_stat_runtime: comm=gdb pid=2495 runtime=1007474 [ns] vruntime=300515161229 [ns] > gdb-2495 [000] *Nh3 61513.836154: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2395843 interval=0 mode=rel > gdb-2495 [000] *Nh3 61513.836155: cobalt_tick_shot: next tick at 61513.838550 (delay: 2395 us) > gdb-2495 [000] dNh3 61513.836155: irq_handler_exit: irq=124 ret=handled > gdb-2495 [000] .Ns3 61513.836156: softirq_entry: vec=1 [action=TIMER] > gdb-2495 [000] .Ns3 61513.836157: softirq_exit: vec=1 [action=TIMER] > gdb-2495 [000] .Ns3 61513.836157: softirq_entry: vec=9 [action=RCU] > gdb-2495 [000] dNs6 61513.836158: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000 > gdb-2495 [000] dNs7 61513.836159: sched_stat_runtime: comm=gdb pid=2495 runtime=6881 [ns] vruntime=300515168110 [ns] > gdb-2495 [000] dNs7 61513.836160: sched_stat_blocked: comm=rcu_sched pid=11 delay=4188739 [ns] > gdb-2495 [000] dNs7 61513.836161: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000 > gdb-2495 [000] .Ns3 61513.836164: softirq_exit: vec=9 [action=RCU] > gdb-2495 [000] d..3 61513.836165: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=hi-thread next_pid=2504 next_prio=97 > hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200 > hi-thread-2504 [000] *..2 61513.836169: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000 > hi-thread-2504 [000] *..2 61513.836170: cobalt_thread_suspend: pid=2504 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000 > hi-thread-2504 [000] d..2 61513.836170: signal_generate: sig=5 errno=0 code=128 comm=hi-thread pid=2504 grp=0 res=0 > hi-thread-2504 [000] d..3 61513.836173: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0 > hi-thread-2504 [000] d..3 61513.836175: sched_stat_wait: comm=rcu_sched pid=11 delay=14490 [ns] > hi-thread-2504 [000] d..3 61513.836175: sched_switch: prev_comm=hi-thread prev_pid=2504 prev_prio=97 prev_state=t ==> next_comm=rcu_sched next_pid=11 next_prio=120 > rcu_sched-11 [000] d..3 61513.836179: sched_stat_runtime: comm=rcu_sched pid=11 runtime=4960 [ns] vruntime=300512173070 [ns] > rcu_sched-11 [000] d..3 61513.836179: sched_stat_wait: comm=gdb pid=2495 delay=19450 [ns] > rcu_sched-11 [000] d..3 61513.836180: sched_switch: prev_comm=rcu_sched prev_pid=11 prev_prio=120 prev_state=I ==> next_comm=gdb next_pid=2495 next_prio=120 > gdb-2495 [000] ...1 61513.836187: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cb0, flags: 80000, mode: 0) > gdb-2495 [000] ...1 61513.836197: sys_openat -> 0x12 > gdb-2495 [000] ...1 61513.836198: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7ae0) > gdb-2495 [000] ...1 61513.836199: sys_newfstat -> 0x0 > gdb-2495 [000] ...1 61513.836200: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) > gdb-2495 [000] ...1 61513.836214: sys_read -> 0x3f1 > gdb-2495 [000] ...1 61513.836217: sys_close(fd: 12) > gdb-2495 [000] ...1 61513.836217: sys_close -> 0x0 > gdb-2495 [000] ...1 61513.836221: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8) > gdb-2495 [000] ...1 61513.836222: sys_rt_sigprocmask -> 0x0 > gdb-2495 [000] d..2 61513.836223: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000 > gdb-2495 [000] ...1 61513.836225: sys_read(fd: f, buf: 7ffe8b2a7667, count: 1) > gdb-2495 [000] ...1 61513.836226: sys_read -> 0xfffffffffffffff5 > gdb-2495 [000] ...1 61513.836227: sys_write(fd: 10, buf: 5585fe0354b1, count: 1) > gdb-2495 [000] ...1 61513.836229: sys_write -> 0x1 > gdb-2495 [000] ...1 61513.836229: sys_rt_sigreturn() > gdb-2495 [000] ...1 61513.836233: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1) > gdb-2495 [000] ...1 61513.836234: sys_read -> 0xfffffffffffffff5 > gdb-2495 [000] ...1 61513.836235: sys_write(fd: 1, buf: 5585fe7a5290, count: 27) > gdb-2495 [000] d..4 61513.836236: sched_waking: comm=smokey pid=2493 prio=120 target_cpu=000 > gdb-2495 [000] d..5 61513.836237: sched_stat_runtime: comm=gdb pid=2495 runtime=58399 [ns] vruntime=300515226509 [ns] > gdb-2495 [000] d..5 61513.836238: sched_stat_sleep: comm=smokey pid=2493 delay=1717179 [ns] > gdb-2495 [000] dN.5 61513.836239: sched_wakeup: comm=smokey pid=2493 prio=120 target_cpu=000 > gdb-2495 [000] .N.1 61513.836240: sys_write -> 0x27 > gdb-2495 [000] dN.3 61513.836241: sched_stat_wait: comm=smokey pid=2493 delay=0 [ns] > gdb-2495 [000] d..3 61513.836241: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2493 next_prio=120 > smokey-2493 [000] ...1 61513.836244: sys_read -> 0x1 > smokey-2493 [000] ..~1 61513.836246: cobalt_head_sysentry: syscall=read > ... > smokey-2493 [000] d..3 61513.836344: sched_stat_runtime: comm=smokey pid=2493 runtime=106636 [ns] vruntime=300512333145 [ns] > smokey-2493 [000] d..3 61513.836344: sched_stat_wait: comm=gdb pid=2495 delay=106636 [ns] > smokey-2493 [000] d..3 61513.836345: sched_switch: prev_comm=smokey prev_pid=2493 prev_prio=120 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 > gdb-2495 [000] ...1 61513.836347: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0) > gdb-2495 [000] ...1 61513.836348: sys_poll -> 0x1 > gdb-2495 [000] ...1 61513.836350: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) > gdb-2495 [000] ...1 61513.836351: sys_read -> 0x1 > gdb-2495 [000] ...1 61513.836352: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) > gdb-2495 [000] ...1 61513.836352: sys_read -> 0xfffffffffffffff5 > gdb-2495 [000] ...1 61513.836353: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8) > gdb-2495 [000] ...1 61513.836354: sys_rt_sigprocmask -> 0x0 > gdb-2495 [000] ...1 61513.836355: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) > gdb-2495 [000] ...1 61513.836355: sched_process_wait: comm=gdb pid=0 prio=120 > gdb-2495 [000] ...1 61513.836357: sys_wait4 -> 0x9c8 > gdb-2495 [000] ...1 61513.836360: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0) > gdb-2495 [000] d..3 61513.836361: sched_wait_task: comm=hi-thread pid=2504 prio=97 > gdb-2495 [000] ...1 61513.836363: sys_ptrace -> 0x0 > gdb-2495 [000] ...1 61513.836367: sys_ptrace(request: 4202, pid: 9c8, addr: 0, data: 7ffe8b2a7cb0) > gdb-2495 [000] d..3 61513.836368: sched_wait_task: comm=hi-thread pid=2504 prio=97 > gdb-2495 [000] ...1 61513.836369: sys_ptrace -> 0x0 > gdb-2495 [000] ...1 61513.836370: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6f10) > gdb-2495 [000] d..3 61513.836370: sched_wait_task: comm=hi-thread pid=2504 prio=97 > gdb-2495 [000] ...1 61513.836371: sys_ptrace -> 0x0 > gdb-2495 [000] ...1 61513.836372: sys_ptrace(request: d, pid: 9c8, addr: 0, data: 7ffe8b2a6f10) > gdb-2495 [000] d..3 61513.836373: sched_wait_task: comm=hi-thread pid=2504 prio=97 > gdb-2495 [000] ...1 61513.836374: sys_ptrace -> 0x0 > gdb-2495 [000] ...1 61513.836376: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) > gdb-2495 [000] ...1 61513.836376: sched_process_wait: comm=gdb pid=0 prio=120 > gdb-2495 [000] ...1 61513.836377: sys_wait4 -> 0x0 > gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8) > gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask -> 0x0 > gdb-2495 [000] ...1 61513.836382: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cd0, flags: 80000, mode: 0) > gdb-2495 [000] ...1 61513.836399: sys_openat -> 0x12 > gdb-2495 [000] ...1 61513.836400: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7b50) > gdb-2495 [000] ...1 61513.836401: sys_newfstat -> 0x0 > gdb-2495 [000] ...1 61513.836402: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) > gdb-2495 [000] ...1 61513.836409: sys_read -> 0x140 > gdb-2495 [000] ...1 61513.836410: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400) > gdb-2495 [000] ...1 61513.836410: sys_read -> 0x0 > gdb-2495 [000] ...1 61513.836415: sys_close(fd: 12) > gdb-2495 [000] ...1 61513.836416: sys_close -> 0x0 > gdb-2495 [000] ...1 61513.836419: sys_read(fd: f, buf: 7ffe8b2a7f17, count: 1) > gdb-2495 [000] ...1 61513.836420: sys_read -> 0xfffffffffffffff5 > gdb-2495 [000] ...1 61513.836421: sys_write(fd: 10, buf: 5585fe0354b1, count: 1) > gdb-2495 [000] ...1 61513.836422: sys_write -> 0x1 > gdb-2495 [000] ...1 61513.836425: sys_ptrace(request: 1, pid: 9c8, addr: 7ffff777f370, data: 7ffe8b2a7878) > gdb-2495 [000] d..3 61513.836426: sched_wait_task: comm=hi-thread pid=2504 prio=97 > gdb-2495 [000] ...1 61513.836428: sys_ptrace -> 0x0 > gdb-2495 [000] ...1 61513.836430: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a7bd8) > gdb-2495 [000] d..3 61513.836430: sched_wait_task: comm=hi-thread pid=2504 prio=97 > gdb-2495 [000] ...1 61513.836431: sys_ptrace -> 0x0 > gdb-2495 [000] ...1 61513.836433: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a70e0, flags: 80000, mode: 0) > gdb-2495 [000] ...1 61513.836446: sys_openat -> 0x12 > gdb-2495 [000] ...1 61513.836446: sys_pread64(fd: 12, buf: 7ffe8b2a7430, count: 900, pos: 7ffff67c9700) > gdb-2495 [000] ...1 61513.836449: sys_pread64 -> 0x900 > gdb-2495 [000] ...1 61513.836450: sys_close(fd: 12) > gdb-2495 [000] ...1 61513.836450: sys_close -> 0x0 > gdb-2495 [000] ...1 61513.836479: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0) > gdb-2495 [000] ...1 61513.836488: sys_openat -> 0x12 > gdb-2495 [000] ...1 61513.836489: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950) > gdb-2495 [000] ...1 61513.836490: sys_newfstat -> 0x0 > gdb-2495 [000] ...1 61513.836490: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400) > gdb-2495 [000] ...1 61513.836498: sys_read -> 0x14d > gdb-2495 [000] ...1 61513.836499: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) > gdb-2495 [000] ...1 61513.836499: sys_read -> 0x0 > gdb-2495 [000] ...1 61513.836503: sys_close(fd: 12) > gdb-2495 [000] ...1 61513.836504: sys_close -> 0x0 > gdb-2495 [000] ...1 61513.836507: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0) > gdb-2495 [000] ...1 61513.836514: sys_openat -> 0x12 > gdb-2495 [000] ...1 61513.836515: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950) > gdb-2495 [000] ...1 61513.836515: sys_newfstat -> 0x0 > gdb-2495 [000] ...1 61513.836516: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) > gdb-2495 [000] ...1 61513.836521: sys_read -> 0x13b > gdb-2495 [000] ...1 61513.836522: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400) > gdb-2495 [000] ...1 61513.836522: sys_read -> 0x0 > gdb-2495 [000] ...1 61513.836525: sys_close(fd: 12) > gdb-2495 [000] ...1 61513.836525: sys_close -> 0x0 > gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13) > gdb-2495 [000] *..3 61513.836532: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000 > gdb-2495 [000] *..3 61513.836532: cobalt_trace_pid: pid=2499, prio=1 > gdb-2495 [000] *..3 61513.836533: cobalt_thread_unblock: pid=2499 state=0x48848 info=0x0 > gdb-2495 [000] *.~3 61513.836586: cobalt_schedule: status=0x10000000 > gdb-2495 [000] *.~3 61513.836586: cobalt_trace_pid: pid=2495, prio=-1 > gdb-2495 [000] *.~3 61513.836587: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1 > smokey-2499 [000] *.~3 61513.836589: cobalt_trace_pid: pid=2499, prio=1 > smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined > smokey-2499 [000] *.~3 61513.836589: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey > smokey-2499 [000] *.~3 61513.836590: cobalt_thread_suspend: pid=2499 mask=0x400080 timeout=0 timeout_mode=0 wchan=0000000000000000 > smokey-2499 [000] *.~3 61513.836590: cobalt_schedule: status=0x10000000 > smokey-2499 [000] *.~3 61513.836590: cobalt_trace_pid: pid=2499, prio=1 > smokey-2499 [000] *.~3 61513.836591: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x4488c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 > gdb-2495 [000] *.~3 61513.836592: cobalt_trace_pid: pid=2495, prio=-1 > gdb-2495 [000] d..5 61513.836592: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 > gdb-2495 [000] dN.6 61513.836594: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 > gdb-2495 [000] dN.3 61513.836594: signal_generate: sig=19 errno=0 code=-6 comm=smokey pid=2499 grp=0 res=0 > gdb-2495 [000] dNh3 61513.836595: irq_handler_entry: irq=120 name=in-band work > gdb-2495 [000] dNh3 61513.836596: cobalt_lostage_wakeup: pid=2499 comm=smokey > gdb-2495 [000] *N~4 61513.838552: irq_pipeline_entry: irq=4354 > gdb-2495 [000] *N~4 61513.838553: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt > gdb-2495 [000] *N~4 61513.838554: cobalt_timer_expire: timer=00000000d13b8aa9 > gdb-2495 [000] *N~4 61513.838555: irq_handler_exit: irq=4354 ret=handled > gdb-2495 [000] *N~4 61513.838555: irq_pipeline_exit: irq=4354 > gdb-2495 [000] dNh8 61513.981277: sched_waking: comm=systemd-journal pid=163 prio=120 target_cpu=000 > gdb-2495 [000] dNh9 61513.981278: sched_stat_runtime: comm=gdb pid=2495 runtime=144934239 [ns] vruntime=300660160748 [ns] > gdb-2495 [000] dNh9 61513.981280: sched_stat_sleep: comm=systemd-journal pid=163 delay=111416392895 [ns] > gdb-2495 [000] dNh9 61513.981280: sched_wakeup: comm=systemd-journal pid=163 prio=120 target_cpu=000 > gdb-2495 [000] dNh3 61513.981281: irq_handler_exit: irq=120 ret=handled > gdb-2495 [000] dNh3 61513.981282: irq_handler_entry: irq=124 name=proxy tick > gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=1 [action=TIMER] > gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=9 [action=RCU] > gdb-2495 [000] dNh4 61513.981286: sched_stat_runtime: comm=gdb pid=2495 runtime=8375 [ns] vruntime=300660169123 [ns] > gdb-2495 [000] dNh5 61513.981288: sched_waking: comm=cobalt_printf pid=2494 prio=120 target_cpu=000 > gdb-2495 [000] dNh6 61513.981289: sched_stat_runtime: comm=gdb pid=2495 runtime=2987 [ns] vruntime=300660172110 [ns] > gdb-2495 [000] dNh6 61513.981291: sched_stat_sleep: comm=cobalt_printf pid=2494 delay=238660319 [ns] > gdb-2495 [000] dNh6 61513.981291: sched_wakeup: comm=cobalt_printf pid=2494 prio=120 target_cpu=000 > gdb-2495 [000] dNh5 61513.981292: sched_waking: comm=cobalt_printf pid=2503 prio=120 target_cpu=000 > gdb-2495 [000] dNh6 61513.981292: sched_stat_runtime: comm=gdb pid=2495 runtime=3069 [ns] vruntime=300660175179 [ns] > gdb-2495 [000] dNh6 61513.981294: sched_stat_sleep: comm=cobalt_printf pid=2503 delay=146611442 [ns] > gdb-2495 [000] dNh6 61513.981294: sched_wakeup: comm=cobalt_printf pid=2503 prio=120 target_cpu=000 > gdb-2495 [000] *Nh3 61513.981295: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=588476 interval=0 mode=rel > gdb-2495 [000] *Nh3 61513.981296: cobalt_tick_shot: next tick at 61513.981884 (delay: 588 us) > gdb-2495 [000] dNh3 61513.981296: irq_handler_exit: irq=124 ret=handled > gdb-2495 [000] .Ns3 61513.981297: softirq_entry: vec=1 [action=TIMER] > gdb-2495 [000] dNs5 61513.981298: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000 > gdb-2495 [000] dNs6 61513.981298: sched_stat_runtime: comm=gdb pid=2495 runtime=6218 [ns] vruntime=300660181397 [ns] > gdb-2495 [000] dNs6 61513.981299: sched_stat_blocked: comm=rcu_sched pid=11 delay=145119923 [ns] > gdb-2495 [000] dNs6 61513.981299: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000 > gdb-2495 [000] dNs7 61513.981302: sched_waking: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000 > gdb-2495 [000] dNs8 61513.981302: sched_stat_runtime: comm=gdb pid=2495 runtime=3884 [ns] vruntime=300660185281 [ns] > gdb-2495 [000] dNs8 61513.981303: sched_stat_blocked: comm=kworker/0:1 pid=1932 delay=206048652 [ns] > gdb-2495 [000] dNs8 61513.981304: sched_wakeup: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000 > gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX] > gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX] > gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX] > gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX] > gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX] > gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX] > gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX] > gdb-2495 [000] dNs5 61513.981312: softirq_raise: vec=2 [action=NET_TX] > gdb-2495 [000] .Ns3 61513.981314: softirq_exit: vec=1 [action=TIMER] > gdb-2495 [000] .Ns3 61513.981314: softirq_entry: vec=9 [action=RCU] > gdb-2495 [000] .Ns3 61513.981315: softirq_exit: vec=9 [action=RCU] > gdb-2495 [000] dNs5 61513.981316: sched_waking: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000 > gdb-2495 [000] dNs6 61513.981316: sched_stat_runtime: comm=gdb pid=2495 runtime=13841 [ns] vruntime=300660199122 [ns] > gdb-2495 [000] dNs6 61513.981317: sched_stat_sleep: comm=ksoftirqd/0 pid=10 delay=149349034 [ns] > gdb-2495 [000] dNs6 61513.981317: sched_wakeup: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000 > gdb-2495 [000] .N.1 61513.981318: sys_tkill -> 0x0 > gdb-2495 [000] d..3 61513.981320: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 > smokey-2499 [000] *.~3 61513.981888: irq_pipeline_entry: irq=4354 > smokey-2499 [000] *.~3 61513.981888: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt > smokey-2499 [000] *.~3 61513.981889: cobalt_timer_expire: timer=00000000d13b8aa9 > smokey-2499 [000] *.~3 61513.981889: irq_handler_exit: irq=4354 ret=handled > smokey-2499 [000] *.~3 61513.981890: irq_pipeline_exit: irq=4354 > smokey-2499 [000] d.h1 61514.106435: irq_handler_entry: irq=124 name=proxy tick > smokey-2499 [000] *.h1 61514.106441: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2109534 interval=0 mode=rel > smokey-2499 [000] *.h1 61514.106441: cobalt_tick_shot: next tick at 61514.108550 (delay: 2109 us) > smokey-2499 [000] d.h1 61514.106442: irq_handler_exit: irq=124 ret=handled > smokey-2499 [000] ...1 61514.106442: cobalt_shadow_relaxed: state=0x4488c0 info=0x200 > smokey-2499 [000] *.~1 61514.108552: irq_pipeline_entry: irq=4354 > > Regards > > Hongzhan Chen > >> >>-- >>Philippe. -- Philippe. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: gdb test failure debug status update 2021-04-30 5:25 ` Chen, Hongzhan 2021-04-30 5:51 ` Chen, Hongzhan @ 2021-04-30 7:34 ` Philippe Gerum 2021-04-30 8:00 ` Philippe Gerum 1 sibling, 1 reply; 16+ messages in thread From: Philippe Gerum @ 2021-04-30 7:34 UTC (permalink / raw) To: Chen, Hongzhan; +Cc: xenomai Chen, Hongzhan <hongzhan.chen@intel.com> writes: > The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish > ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax. > That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in > arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from > xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to > call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode > because kernel exit code is never called in this case? > The IRQ frame is indeed kept from unwinding until the preempted task context returns from irq_exit_pipeline(), which branches to the Cobalt rescheduling procedure. From the Dovetail interface POV, irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow the companion core to perform post-IRQ chores, such as running its own rescheduling procedure. IOW, if task @foo is preempted by an IRQ, then suspended in oob context as a result of what the interrupt handler just did for it (e.g. raising XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will cause the @foo context to switch away, effectively preventing handle_irq_pipelined_finish() to return, until @foo resumes execution eventually. The underlying issue I'd like to understand, is why do we have a mayday condition raised in the context of this test in the first place? That may be the root of the issue, the mayday and synchronous-bp handling code may not cope that well. So, question to address first: who ends up calling dovetail_call_mayday(), detecting that the underlying task went south? And why that? -- Philippe. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: gdb test failure debug status update 2021-04-30 7:34 ` Philippe Gerum @ 2021-04-30 8:00 ` Philippe Gerum 2021-04-30 8:07 ` Chen, Hongzhan 0 siblings, 1 reply; 16+ messages in thread From: Philippe Gerum @ 2021-04-30 8:00 UTC (permalink / raw) To: Chen, Hongzhan; +Cc: xenomai Philippe Gerum <rpm@xenomai.org> writes: > Chen, Hongzhan <hongzhan.chen@intel.com> writes: > >> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish >> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax. >> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in >> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from >> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to >> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode >> because kernel exit code is never called in this case? >> > > The IRQ frame is indeed kept from unwinding until the preempted task > context returns from irq_exit_pipeline(), which branches to the Cobalt > rescheduling procedure. From the Dovetail interface POV, > irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow > the companion core to perform post-IRQ chores, such as running its own > rescheduling procedure. > > IOW, if task @foo is preempted by an IRQ, then suspended in oob context > as a result of what the interrupt handler just did for it (e.g. raising > XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then > handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will > cause the @foo context to switch away, effectively preventing > handle_irq_pipelined_finish() to return, until @foo resumes execution > eventually. > > The underlying issue I'd like to understand, is why do we have a mayday > condition raised in the context of this test in the first place? That > may be the root of the issue, the mayday and synchronous-bp handling > code may not cope that well. > > So, question to address first: who ends up calling > dovetail_call_mayday(), detecting that the underlying task went south? > And why that? Possibly a caller of __xnthread_kick() is causing that, e.g. handle_sigwake_event -> __xnthread_kick. -- Philippe. ^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: gdb test failure debug status update 2021-04-30 8:00 ` Philippe Gerum @ 2021-04-30 8:07 ` Chen, Hongzhan [not found] ` <DM5PR11MB18529649C47BF241930A2217F25E9@DM5PR11MB1852.namprd11.prod.outlook.com> 0 siblings, 1 reply; 16+ messages in thread From: Chen, Hongzhan @ 2021-04-30 8:07 UTC (permalink / raw) To: Philippe Gerum; +Cc: xenomai >-----Original Message----- >From: Philippe Gerum <rpm@xenomai.org> >Sent: Friday, April 30, 2021 4:01 PM >To: Chen, Hongzhan <hongzhan.chen@intel.com> >Cc: xenomai@xenomai.org >Subject: Re: gdb test failure debug status update > > >Philippe Gerum <rpm@xenomai.org> writes: > >> Chen, Hongzhan <hongzhan.chen@intel.com> writes: >> >>> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish >>> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax. >>> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in >>> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from >>> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to >>> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode >>> because kernel exit code is never called in this case? >>> >> >> The IRQ frame is indeed kept from unwinding until the preempted task >> context returns from irq_exit_pipeline(), which branches to the Cobalt >> rescheduling procedure. From the Dovetail interface POV, >> irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow >> the companion core to perform post-IRQ chores, such as running its own >> rescheduling procedure. >> >> IOW, if task @foo is preempted by an IRQ, then suspended in oob context >> as a result of what the interrupt handler just did for it (e.g. raising >> XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then >> handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will >> cause the @foo context to switch away, effectively preventing >> handle_irq_pipelined_finish() to return, until @foo resumes execution > eventually. > > The underlying issue I'd like to understand, is why do we have a mayday > condition raised in the context of this test in the first place? That > may be the root of the issue, the mayday and synchronous-bp handling > code may not cope that well. > > So, question to address first: who ends up calling > dovetail_call_mayday(), detecting that the underlying task went south? > And why that? > >Possibly a caller of __xnthread_kick() is causing that, e.g. >handle_sigwake_event -> __xnthread_kick. Yes, that is right. Like in line gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13) of log [4] , Sys_tkill would finally call __xnthread_kick to set MAYDAY through handle_sigwake_event by gdb and then Smokey thread is to handle mayday. I also get ftrace log for ipipe-5.4-xenomai, it has same caller. > >-- >Philippe. > [4]: gdb-2495 [000] d..3 61513.834825: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] ...1 61513.834827: cobalt_handle_ptrace_cont: pid=2499 state=0x4488c0 info=0x200 smokey-2499 [000] d..1 61513.834828: cobalt_handle_user_return: pid=2499 state=0x4488c0 info=0x200 smokey-2499 [000] d..1 61513.834828: cobalt_shadow_gohard: state=0x4488c0 info=0x0 smokey-2499 [000] d..3 61513.834830: sched_stat_wait: comm=gdb pid=2495 delay=8017 [ns] smokey-2499 [000] d..3 61513.834831: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] *..3 61513.834831: cobalt_thread_resume: name=smokey pid=2499 mask=0x80 gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1 gdb-2495 [000] *..3 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000 gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1 gdb-2495 [000] *.~3 61513.834833: cobalt_schedule: status=0x10000000 gdb-2495 [000] *.~3 61513.834833: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] *.~3 61513.834833: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1 smokey-2499 [000] d.~3 61513.834835: cobalt_shadow_hardened: state=0x48040 info=0x0 smokey-2499 [000] d.~3 61513.834836: cobalt_head_sysentry: syscall=sem_init smokey-2499 [000] d.~3 61513.834839: cobalt_psem_init: sem=0x4e(anon) flags=0x0() value=0 smokey-2499 [000] d.~3 61513.834839: cobalt_head_sysexit: result=0 smokey-2499 [000] d.~3 61513.834848: cobalt_shadow_gorelax: reason=syscall smokey-2499 [000] d.~3 61513.834848: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey smokey-2499 [000] *.~3 61513.834848: cobalt_thread_suspend: pid=2499 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 smokey-2499 [000] *.~3 61513.834849: cobalt_schedule: status=0x10000000 smokey-2499 [000] *.~3 61513.834849: cobalt_trace_pid: pid=2499, prio=1 smokey-2499 [000] *.~3 61513.834849: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 gdb-2495 [000] *.~3 61513.834850: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] d.h3 61513.834851: irq_handler_entry: irq=120 name=in-band work gdb-2495 [000] d.h3 61513.834852: cobalt_lostage_wakeup: pid=2499 comm=smokey gdb-2495 [000] d.h5 61513.834852: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dNh6 61513.834853: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dNh3 61513.834853: irq_handler_exit: irq=120 ret=handled gdb-2495 [000] dN.3 61513.834854: sched_stat_runtime: comm=gdb pid=2495 runtime=23565 [ns] vruntime=300514015543 [ns] gdb-2495 [000] d..3 61513.834855: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] ...1 61513.834856: cobalt_shadow_relaxed: state=0x480c0 info=0x0 smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min(policy: 1) smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min -> 0x1 smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max(policy: 1) smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max -> 0x63 smokey-2499 [000] ..~1 61513.834862: cobalt_head_sysentry: syscall=thread_getschedparam_ex smokey-2499 [000] ...1 61513.834862: cobalt_root_sysentry: syscall=thread_getschedparam_ex smokey-2499 [000] ...1 61513.834863: cobalt_pthread_getschedparam: pth=000000008d36be1c policy=fifo param={ priority=1 } smokey-2499 [000] ...1 61513.834864: cobalt_root_sysexit: result=0 smokey-2499 [000] ...1 61513.834872: sys_mmap(addr: 0, len: 11000, prot: 0, flags: 20022, fd: ffffffff, off: 0) smokey-2499 [000] ...1 61513.834877: sys_mmap -> 0x7ffff67b9000 smokey-2499 [000] ...1 61513.834878: sys_mprotect(start: 7ffff67ba000, len: 10000, prot: 3) smokey-2499 [000] ...1 61513.834906: sys_mprotect -> 0x0 smokey-2499 [000] ...1 61513.834909: sys_clone(clone_flags: 3d0f00, newsp: 7ffff67c8fb0, parent_tidptr: 7ffff67c99d0, child_tidptr: 7ffff67c99d0, tls: 7ffff67c9700) smokey-2499 [000] ...1 61513.834923: task_newtask: pid=2504 comm=smokey clone_flags=3d0f00 oom_score_adj=0 smokey-2499 [000] ...1 61513.834923: sched_process_fork: comm=smokey pid=2499 child_comm=smokey child_pid=2504 smokey-2499 [000] d..3 61513.834925: sched_wakeup_new: comm=smokey pid=2504 prio=98 target_cpu=000 smokey-2499 [000] d..3 61513.834928: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0 smokey-2499 [000] d..3 61513.834931: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=t ==> next_comm=smokey next_pid=2504 next_prio=98 hi-thread-2504 [000] ...1 61513.834932: sys_clone -> 0x0 hi-thread-2504 [000] d..3 61513.834934: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=2 hi-thread-2504 [000] d..3 61513.834935: sched_stat_wait: comm=gdb pid=2495 delay=82002 [ns] hi-thread-2504 [000] d..3 61513.834936: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=t ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] d..2 61513.834937: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000 gdb-2495 [000] ...1 61513.834940: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1) gdb-2495 [000] ...1 61513.834943: sys_read -> 0x1 gdb-2495 [000] ...1 61513.834943: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1) gdb-2495 [000] ...1 61513.834944: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.834944: sys_write(fd: 10, buf: 5585fe0354b1, count: 1) gdb-2495 [000] ...1 61513.834946: sys_write -> 0x1 gdb-2495 [000] ...1 61513.834947: sys_rt_sigreturn() gdb-2495 [000] ...1 61513.834953: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1) gdb-2495 [000] ...1 61513.834953: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.834955: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0) gdb-2495 [000] ...1 61513.834956: sys_poll -> 0x1 gdb-2495 [000] ...1 61513.834957: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) gdb-2495 [000] ...1 61513.834958: sys_read -> 0x1 gdb-2495 [000] ...1 61513.834959: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) gdb-2495 [000] ...1 61513.834960: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8) gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask -> 0x0 gdb-2495 [000] ...1 61513.834962: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) gdb-2495 [000] ...1 61513.834962: sched_process_wait: comm=gdb pid=0 prio=120 gdb-2495 [000] ...1 61513.834964: sys_wait4 -> 0x9c3 gdb-2495 [000] ...1 61513.834965: sys_ptrace(request: 4201, pid: 9c3, addr: 0, data: 7ffe8b2a7d08) gdb-2495 [000] d..3 61513.834966: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.834967: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.834967: sys_wait4(upid: 9c8, stat_addr: 7ffe8b2a7cfc, options: 40000000, ru: 0) gdb-2495 [000] ...1 61513.834968: sched_process_wait: comm=gdb pid=2504 prio=120 gdb-2495 [000] ...1 61513.834969: sys_wait4 -> 0x9c8 gdb-2495 [000] ...1 61513.834972: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578) gdb-2495 [000] d..3 61513.834972: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.834975: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.834976: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a78d8) gdb-2495 [000] d..3 61513.834977: sched_wait_task: comm=smokey pid=2504 prio=98 gdb-2495 [000] ...1 61513.834977: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.834979: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.834989: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.834990: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff67c9700) gdb-2495 [000] ...1 61513.834992: sys_pread64 -> 0x900 gdb-2495 [000] ...1 61513.834993: sys_close(fd: 12) gdb-2495 [000] ...1 61513.834994: sys_close -> 0x0 gdb-2495 [000] ...1 61513.835010: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578) gdb-2495 [000] d..3 61513.835011: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.835012: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835013: sys_ptrace(request: 3, pid: 9c3, addr: a8, data: 7ffe8b2a78d8) gdb-2495 [000] d..3 61513.835013: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.835014: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835015: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.835022: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.835022: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff758f740) gdb-2495 [000] ...1 61513.835025: sys_pread64 -> 0x900 gdb-2495 [000] ...1 61513.835025: sys_close(fd: 12) gdb-2495 [000] ...1 61513.835026: sys_close -> 0x0 gdb-2495 [000] ...1 61513.835031: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) gdb-2495 [000] ...1 61513.835032: sched_process_wait: comm=gdb pid=0 prio=120 gdb-2495 [000] ...1 61513.835033: sys_wait4 -> 0x0 gdb-2495 [000] ...1 61513.835036: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0) gdb-2495 [000] d..3 61513.835037: sched_wait_task: comm=smokey pid=2504 prio=98 gdb-2495 [000] ...1 61513.835039: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835043: sys_ptrace(request: 6, pid: 9c8, addr: 388, data: 0) gdb-2495 [000] d..3 61513.835044: sched_wait_task: comm=smokey pid=2504 prio=98 gdb-2495 [000] ...1 61513.835044: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835045: sys_ptrace(request: 7, pid: 9c8, addr: 1, data: 0) gdb-2495 [000] d..3 61513.835046: sched_wait_task: comm=smokey pid=2504 prio=98 gdb-2495 [000] d..3 61513.835047: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000 gdb-2495 [000] dN.4 61513.835048: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000 gdb-2495 [000] .N.1 61513.835048: sys_ptrace -> 0x0 gdb-2495 [000] dN.3 61513.835050: sched_stat_runtime: comm=gdb pid=2495 runtime=112561 [ns] vruntime=300514128104 [ns] gdb-2495 [000] d..3 61513.835051: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=98 hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list(head: 7ffff67c99e0, len: 18) hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list -> 0x0 hi-thread-2504 [000] ...1 61513.835056: sys_futex(uaddr: 7ffff67c9d18, op: 80, val: 2, utime: 0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700) hi-thread-2504 [000] d..3 61513.835059: sched_stat_wait: comm=gdb pid=2495 delay=10883 [ns] hi-thread-2504 [000] d..3 61513.835060: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] ...1 61513.835063: sys_ptrace(request: c, pid: 9c3, addr: 0, data: 7ffe8b2a6ef0) gdb-2495 [000] d..3 61513.835064: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] ...1 61513.835065: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.835068: sys_ptrace(request: 7, pid: 9c3, addr: 1, data: 0) gdb-2495 [000] d..3 61513.835069: sched_wait_task: comm=smokey pid=2499 prio=98 gdb-2495 [000] d..3 61513.835069: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dN.4 61513.835070: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] .N.1 61513.835071: sys_ptrace -> 0x0 gdb-2495 [000] dN.3 61513.835071: sched_stat_runtime: comm=gdb pid=2495 runtime=12210 [ns] vruntime=300514140314 [ns] gdb-2495 [000] d..3 61513.835072: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0 smokey-2499 [000] ...1 61513.835074: sys_clone -> 0x9c8 smokey-2499 [000] ...1 61513.835075: sys_sched_setaffinity(pid: 9c8, len: 80, user_mask_ptr: 55555563e7c0) smokey-2499 [000] ...1 61513.835077: sys_sched_setaffinity -> 0x0 smokey-2499 [000] ...1 61513.835077: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 0, val3: 7ffff67c9700) smokey-2499 [000] d..3 61513.835078: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000 smokey-2499 [000] d..4 61513.835079: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000 smokey-2499 [000] ...1 61513.835080: sys_futex -> 0x1 smokey-2499 [000] ...1 61513.835086: sys_futex(uaddr: 7fffffffe828, op: 189, val: 0, utime: 7fffffffe780, uaddr2: 0, val3: ffffffff) smokey-2499 [000] d..3 61513.835089: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=smokey next_pid=2504 next_prio=98 hi-thread-2504 [000] ...1 61513.835090: sys_futex -> 0x0 hi-thread-2504 [000] ...1 61513.835090: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700) hi-thread-2504 [000] ...1 61513.835091: sys_futex -> 0x0 hi-thread-2504 [000] ...1 61513.835092: sys_sched_setscheduler(pid: 9c8, policy: 1, param: 7ffff67c8e70) hi-thread-2504 [000] ...1 61513.835095: sys_sched_setscheduler -> 0x0 hi-thread-2504 [000] ..~1 61513.835096: cobalt_head_sysentry: syscall=thread_create hi-thread-2504 [000] ...1 61513.835097: cobalt_root_sysentry: syscall=thread_create hi-thread-2504 [000] ...1 61513.835098: cobalt_pthread_create: pth=00000000ba35a01d policy=fifo param={ priority=2 } hi-thread-2504 [000] ...1 61513.835099: cobalt_timer_migrate: timer=000000004d21df54 cpu=0 hi-thread-2504 [000] ...1 61513.835100: cobalt_timer_migrate: timer=00000000c1301853 cpu=0 hi-thread-2504 [000] ...1 61513.835101: cobalt_thread_set_current_prio: thread=00000000e9fda3c4 pid=-1 prio=2 hi-thread-2504 [000] ...1 61513.835102: cobalt_thread_init: thread=00000000e9fda3c4 name=smokey flags=0x48000 class=rt prio=2 hi-thread-2504 [000] ...1 61513.835104: cobalt_shadow_map: thread=00000000e9fda3c4 pid=2504 prio=2 hi-thread-2504 [000] D..1 61513.835104: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_start: pid=2504 state=0x480d0 info=0x0 hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_resume: name=smokey pid=2504 mask=0x10 hi-thread-2504 [000] D..1 61513.835106: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] ...1 61513.835106: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] ...1 61513.835107: cobalt_shadow_gohard: state=0x480c0 info=0x0 hi-thread-2504 [000] d..3 61513.835108: sched_stat_wait: comm=gdb pid=2495 delay=37231 [ns] hi-thread-2504 [000] d..3 61513.835109: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] *..3 61513.835109: cobalt_thread_resume: name=smokey pid=2504 mask=0x80 gdb-2495 [000] *..3 61513.835110: cobalt_trace_pid: pid=2504, prio=2 gdb-2495 [000] *.~3 61513.835110: cobalt_schedule: status=0x10000000 gdb-2495 [000] *.~3 61513.835110: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] *.~3 61513.835111: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2 hi-thread-2504 [000] d.~3 61513.835112: cobalt_shadow_hardened: state=0x48040 info=0x0 hi-thread-2504 [000] d.~3 61513.835112: cobalt_root_sysexit: result=0 hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysentry: syscall=get_current hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysexit: result=0 hi-thread-2504 [000] d.~3 61513.835116: cobalt_shadow_gorelax: reason=syscall hi-thread-2504 [000] d.~3 61513.835117: cobalt_lostage_request: request=wakeup pid=2504 comm=smokey hi-thread-2504 [000] *.~3 61513.835117: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] *.~3 61513.835117: cobalt_schedule: status=0x10000000 hi-thread-2504 [000] *.~3 61513.835118: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] *.~3 61513.835118: cobalt_switch_context: prev_name=smokey prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 gdb-2495 [000] *.~3 61513.835119: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] d.h3 61513.835120: irq_handler_entry: irq=120 name=in-band work gdb-2495 [000] d.h3 61513.835120: cobalt_lostage_wakeup: pid=2504 comm=smokey gdb-2495 [000] d.h5 61513.835120: sched_waking: comm=smokey pid=2504 prio=97 target_cpu=000 gdb-2495 [000] dNh6 61513.835121: sched_wakeup: comm=smokey pid=2504 prio=97 target_cpu=000 gdb-2495 [000] dNh3 61513.835122: irq_handler_exit: irq=120 ret=handled gdb-2495 [000] dN.3 61513.835122: sched_stat_runtime: comm=gdb pid=2495 runtime=13441 [ns] vruntime=300514153755 [ns] gdb-2495 [000] d..3 61513.835123: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=97 hi-thread-2504 [000] ...1 61513.835124: cobalt_shadow_relaxed: state=0x480c0 info=0x0 hi-thread-2504 [000] ...1 61513.835125: sys_futex(uaddr: 7fffffffe828, op: 81, val: 1, utime: 0, uaddr2: 7ffff67c8e74, val3: 7fffffffe828) hi-thread-2504 [000] d..3 61513.835126: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 hi-thread-2504 [000] d..4 61513.835126: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 hi-thread-2504 [000] ...1 61513.835127: sys_futex -> 0x1 hi-thread-2504 [000] ..~1 61513.835128: cobalt_head_sysentry: syscall=migrate hi-thread-2504 [000] ...1 61513.835128: cobalt_root_sysentry: syscall=migrate hi-thread-2504 [000] ...1 61513.835128: cobalt_shadow_gohard: state=0x480c0 info=0x0 hi-thread-2504 [000] d..3 61513.835130: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] *..3 61513.835130: cobalt_thread_resume: name=smokey pid=2504 mask=0x80 smokey-2499 [000] *..3 61513.835130: cobalt_trace_pid: pid=2504, prio=2 smokey-2499 [000] *.~3 61513.835131: cobalt_schedule: status=0x10000000 smokey-2499 [000] *.~3 61513.835131: cobalt_trace_pid: pid=2499, prio=-1 smokey-2499 [000] *.~3 61513.835131: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2 hi-thread-2504 [000] d.~3 61513.835132: cobalt_shadow_hardened: state=0x48040 info=0x0 hi-thread-2504 [000] d.~3 61513.835132: cobalt_root_sysexit: result=1 hi-thread-2504 [000] d.~3 61513.835133: cobalt_head_sysentry: syscall=thread_setname hi-thread-2504 [000] d.~3 61513.835134: cobalt_pthread_setname: pth=00000000ba35a01d name=hi-thread hi-thread-2504 [000] d.~3 61513.835135: cobalt_head_sysexit: result=0 hi-thread-2504 [000] d.~3 61513.835136: cobalt_head_sysentry: syscall=clock_nanosleep hi-thread-2504 [000] d.~3 61513.835137: cobalt_clock_nanosleep: clock_id=0 flags=0x0() rqt=(0.001000000) hi-thread-2504 [000] *.~3 61513.835138: cobalt_thread_suspend: pid=2504 mask=0x4 timeout=1000001 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] *.~3 61513.835138: cobalt_timer_start: timer=000000004d21df54(smokey) value=1000001 interval=0 mode=rel hi-thread-2504 [000] *.~3 61513.835138: cobalt_schedule: status=0x10000000 hi-thread-2504 [000] *.~3 61513.835139: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] *.~3 61513.835139: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x48044 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 smokey-2499 [000] *.~3 61513.835140: cobalt_trace_pid: pid=2499, prio=-1 smokey-2499 [000] ...1 61513.835141: sys_futex -> 0x0 smokey-2499 [000] ..~1 61513.835144: cobalt_head_sysentry: syscall=migrate smokey-2499 [000] ...1 61513.835144: cobalt_root_sysentry: syscall=migrate smokey-2499 [000] ...1 61513.835144: cobalt_shadow_gohard: state=0x480c0 info=0x0 smokey-2499 [000] d..3 61513.835145: sched_stat_wait: comm=gdb pid=2495 delay=23982 [ns] smokey-2499 [000] d..3 61513.835146: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] *..3 61513.835147: cobalt_thread_resume: name=smokey pid=2499 mask=0x80 gdb-2495 [000] *..3 61513.835147: cobalt_trace_pid: pid=2499, prio=1 gdb-2495 [000] *.~3 61513.835147: cobalt_schedule: status=0x10000000 gdb-2495 [000] *.~3 61513.835147: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] *.~3 61513.835148: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1 smokey-2499 [000] d.~3 61513.835148: cobalt_shadow_hardened: state=0x48040 info=0x0 smokey-2499 [000] d.~3 61513.835149: cobalt_root_sysexit: result=1 smokey-2499 [000] *.~3 61513.835219: irq_pipeline_entry: irq=4354 smokey-2499 [000] *.~3 61513.835219: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt smokey-2499 [000] *.~3 61513.835220: cobalt_timer_expire: timer=00000000d13b8aa9 smokey-2499 [000] *.~3 61513.835220: cobalt_tick_shot: next tick at 61513.836136 (delay: 915 us) smokey-2499 [000] *.~3 61513.835221: irq_handler_exit: irq=4354 ret=handled smokey-2499 [000] *.~3 61513.835221: irq_pipeline_exit: irq=4354 smokey-2499 [000] *.~3 61513.836137: irq_pipeline_entry: irq=4354 smokey-2499 [000] *.~3 61513.836137: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt smokey-2499 [000] *.~3 61513.836138: cobalt_timer_expire: timer=000000004d21df54 smokey-2499 [000] *.~3 61513.836138: cobalt_thread_resume: name=hi-thread pid=2504 mask=0x4 smokey-2499 [000] *.~3 61513.836138: cobalt_trace_pid: pid=2504, prio=2 smokey-2499 [000] *.~3 61513.836139: cobalt_timer_stop: timer=000000004d21df54 smokey-2499 [000] *.~3 61513.836139: irq_handler_exit: irq=4354 ret=handled smokey-2499 [000] *.~3 61513.836139: irq_pipeline_exit: irq=4354 smokey-2499 [000] *.~3 61513.836140: cobalt_schedule: status=0x10000000 smokey-2499 [000] *.~3 61513.836140: cobalt_trace_pid: pid=2499, prio=1 smokey-2499 [000] *.~3 61513.836140: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x48048 ==> next_name=hi-thread next_pid=2504 next_prio=2 hi-thread-2504 [000] *.~3 61513.836141: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] d.~3 61513.836141: cobalt_head_sysexit: result=0 hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3 hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] d.~3 61513.836143: cobalt_shadow_gorelax: reason=fault hi-thread-2504 [000] d.~3 61513.836143: cobalt_lostage_request: request=wakeup pid=2504 comm=hi-thread hi-thread-2504 [000] *.~3 61513.836144: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] *.~3 61513.836144: cobalt_schedule: status=0x10000000 hi-thread-2504 [000] *.~3 61513.836144: cobalt_trace_pid: pid=2504, prio=2 hi-thread-2504 [000] *.~3 61513.836145: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 gdb-2495 [000] *.~3 61513.836146: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] d.h3 61513.836146: irq_handler_entry: irq=120 name=in-band work gdb-2495 [000] d.h3 61513.836147: cobalt_lostage_wakeup: pid=2504 comm=hi-thread gdb-2495 [000] d.h5 61513.836147: sched_waking: comm=hi-thread pid=2504 prio=97 target_cpu=000 gdb-2495 [000] dNh6 61513.836147: sched_wakeup: comm=hi-thread pid=2504 prio=97 target_cpu=000 gdb-2495 [000] dNh3 61513.836148: irq_handler_exit: irq=120 ret=handled gdb-2495 [000] dNh3 61513.836148: irq_handler_entry: irq=124 name=proxy tick gdb-2495 [000] dNh3 61513.836151: softirq_raise: vec=1 [action=TIMER] gdb-2495 [000] dNh3 61513.836152: softirq_raise: vec=9 [action=RCU] gdb-2495 [000] dNh4 61513.836152: sched_stat_runtime: comm=gdb pid=2495 runtime=1007474 [ns] vruntime=300515161229 [ns] gdb-2495 [000] *Nh3 61513.836154: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2395843 interval=0 mode=rel gdb-2495 [000] *Nh3 61513.836155: cobalt_tick_shot: next tick at 61513.838550 (delay: 2395 us) gdb-2495 [000] dNh3 61513.836155: irq_handler_exit: irq=124 ret=handled gdb-2495 [000] .Ns3 61513.836156: softirq_entry: vec=1 [action=TIMER] gdb-2495 [000] .Ns3 61513.836157: softirq_exit: vec=1 [action=TIMER] gdb-2495 [000] .Ns3 61513.836157: softirq_entry: vec=9 [action=RCU] gdb-2495 [000] dNs6 61513.836158: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000 gdb-2495 [000] dNs7 61513.836159: sched_stat_runtime: comm=gdb pid=2495 runtime=6881 [ns] vruntime=300515168110 [ns] gdb-2495 [000] dNs7 61513.836160: sched_stat_blocked: comm=rcu_sched pid=11 delay=4188739 [ns] gdb-2495 [000] dNs7 61513.836161: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000 gdb-2495 [000] .Ns3 61513.836164: softirq_exit: vec=9 [action=RCU] gdb-2495 [000] d..3 61513.836165: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=hi-thread next_pid=2504 next_prio=97 hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200 hi-thread-2504 [000] *..2 61513.836169: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] *..2 61513.836170: cobalt_thread_suspend: pid=2504 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000 hi-thread-2504 [000] d..2 61513.836170: signal_generate: sig=5 errno=0 code=128 comm=hi-thread pid=2504 grp=0 res=0 hi-thread-2504 [000] d..3 61513.836173: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0 hi-thread-2504 [000] d..3 61513.836175: sched_stat_wait: comm=rcu_sched pid=11 delay=14490 [ns] hi-thread-2504 [000] d..3 61513.836175: sched_switch: prev_comm=hi-thread prev_pid=2504 prev_prio=97 prev_state=t ==> next_comm=rcu_sched next_pid=11 next_prio=120 rcu_sched-11 [000] d..3 61513.836179: sched_stat_runtime: comm=rcu_sched pid=11 runtime=4960 [ns] vruntime=300512173070 [ns] rcu_sched-11 [000] d..3 61513.836179: sched_stat_wait: comm=gdb pid=2495 delay=19450 [ns] rcu_sched-11 [000] d..3 61513.836180: sched_switch: prev_comm=rcu_sched prev_pid=11 prev_prio=120 prev_state=I ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] ...1 61513.836187: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cb0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836197: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836198: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7ae0) gdb-2495 [000] ...1 61513.836199: sys_newfstat -> 0x0 gdb-2495 [000] ...1 61513.836200: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) gdb-2495 [000] ...1 61513.836214: sys_read -> 0x3f1 gdb-2495 [000] ...1 61513.836217: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836217: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836221: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8) gdb-2495 [000] ...1 61513.836222: sys_rt_sigprocmask -> 0x0 gdb-2495 [000] d..2 61513.836223: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000 gdb-2495 [000] ...1 61513.836225: sys_read(fd: f, buf: 7ffe8b2a7667, count: 1) gdb-2495 [000] ...1 61513.836226: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.836227: sys_write(fd: 10, buf: 5585fe0354b1, count: 1) gdb-2495 [000] ...1 61513.836229: sys_write -> 0x1 gdb-2495 [000] ...1 61513.836229: sys_rt_sigreturn() gdb-2495 [000] ...1 61513.836233: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1) gdb-2495 [000] ...1 61513.836234: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.836235: sys_write(fd: 1, buf: 5585fe7a5290, count: 27) gdb-2495 [000] d..4 61513.836236: sched_waking: comm=smokey pid=2493 prio=120 target_cpu=000 gdb-2495 [000] d..5 61513.836237: sched_stat_runtime: comm=gdb pid=2495 runtime=58399 [ns] vruntime=300515226509 [ns] gdb-2495 [000] d..5 61513.836238: sched_stat_sleep: comm=smokey pid=2493 delay=1717179 [ns] gdb-2495 [000] dN.5 61513.836239: sched_wakeup: comm=smokey pid=2493 prio=120 target_cpu=000 gdb-2495 [000] .N.1 61513.836240: sys_write -> 0x27 gdb-2495 [000] dN.3 61513.836241: sched_stat_wait: comm=smokey pid=2493 delay=0 [ns] gdb-2495 [000] d..3 61513.836241: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2493 next_prio=120 smokey-2493 [000] ...1 61513.836244: sys_read -> 0x1 smokey-2493 [000] ..~1 61513.836246: cobalt_head_sysentry: syscall=read ... smokey-2493 [000] d..3 61513.836344: sched_stat_runtime: comm=smokey pid=2493 runtime=106636 [ns] vruntime=300512333145 [ns] smokey-2493 [000] d..3 61513.836344: sched_stat_wait: comm=gdb pid=2495 delay=106636 [ns] smokey-2493 [000] d..3 61513.836345: sched_switch: prev_comm=smokey prev_pid=2493 prev_prio=120 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120 gdb-2495 [000] ...1 61513.836347: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0) gdb-2495 [000] ...1 61513.836348: sys_poll -> 0x1 gdb-2495 [000] ...1 61513.836350: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) gdb-2495 [000] ...1 61513.836351: sys_read -> 0x1 gdb-2495 [000] ...1 61513.836352: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1) gdb-2495 [000] ...1 61513.836352: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.836353: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8) gdb-2495 [000] ...1 61513.836354: sys_rt_sigprocmask -> 0x0 gdb-2495 [000] ...1 61513.836355: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) gdb-2495 [000] ...1 61513.836355: sched_process_wait: comm=gdb pid=0 prio=120 gdb-2495 [000] ...1 61513.836357: sys_wait4 -> 0x9c8 gdb-2495 [000] ...1 61513.836360: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0) gdb-2495 [000] d..3 61513.836361: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836363: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836367: sys_ptrace(request: 4202, pid: 9c8, addr: 0, data: 7ffe8b2a7cb0) gdb-2495 [000] d..3 61513.836368: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836369: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836370: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6f10) gdb-2495 [000] d..3 61513.836370: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836371: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836372: sys_ptrace(request: d, pid: 9c8, addr: 0, data: 7ffe8b2a6f10) gdb-2495 [000] d..3 61513.836373: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836374: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836376: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0) gdb-2495 [000] ...1 61513.836376: sched_process_wait: comm=gdb pid=0 prio=120 gdb-2495 [000] ...1 61513.836377: sys_wait4 -> 0x0 gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8) gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask -> 0x0 gdb-2495 [000] ...1 61513.836382: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cd0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836399: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836400: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7b50) gdb-2495 [000] ...1 61513.836401: sys_newfstat -> 0x0 gdb-2495 [000] ...1 61513.836402: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) gdb-2495 [000] ...1 61513.836409: sys_read -> 0x140 gdb-2495 [000] ...1 61513.836410: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400) gdb-2495 [000] ...1 61513.836410: sys_read -> 0x0 gdb-2495 [000] ...1 61513.836415: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836416: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836419: sys_read(fd: f, buf: 7ffe8b2a7f17, count: 1) gdb-2495 [000] ...1 61513.836420: sys_read -> 0xfffffffffffffff5 gdb-2495 [000] ...1 61513.836421: sys_write(fd: 10, buf: 5585fe0354b1, count: 1) gdb-2495 [000] ...1 61513.836422: sys_write -> 0x1 gdb-2495 [000] ...1 61513.836425: sys_ptrace(request: 1, pid: 9c8, addr: 7ffff777f370, data: 7ffe8b2a7878) gdb-2495 [000] d..3 61513.836426: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836428: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836430: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a7bd8) gdb-2495 [000] d..3 61513.836430: sched_wait_task: comm=hi-thread pid=2504 prio=97 gdb-2495 [000] ...1 61513.836431: sys_ptrace -> 0x0 gdb-2495 [000] ...1 61513.836433: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a70e0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836446: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836446: sys_pread64(fd: 12, buf: 7ffe8b2a7430, count: 900, pos: 7ffff67c9700) gdb-2495 [000] ...1 61513.836449: sys_pread64 -> 0x900 gdb-2495 [000] ...1 61513.836450: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836450: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836479: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836488: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836489: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950) gdb-2495 [000] ...1 61513.836490: sys_newfstat -> 0x0 gdb-2495 [000] ...1 61513.836490: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400) gdb-2495 [000] ...1 61513.836498: sys_read -> 0x14d gdb-2495 [000] ...1 61513.836499: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) gdb-2495 [000] ...1 61513.836499: sys_read -> 0x0 gdb-2495 [000] ...1 61513.836503: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836504: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836507: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0) gdb-2495 [000] ...1 61513.836514: sys_openat -> 0x12 gdb-2495 [000] ...1 61513.836515: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950) gdb-2495 [000] ...1 61513.836515: sys_newfstat -> 0x0 gdb-2495 [000] ...1 61513.836516: sys_read(fd: 12, buf: 5585fe9960e0, count: 400) gdb-2495 [000] ...1 61513.836521: sys_read -> 0x13b gdb-2495 [000] ...1 61513.836522: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400) gdb-2495 [000] ...1 61513.836522: sys_read -> 0x0 gdb-2495 [000] ...1 61513.836525: sys_close(fd: 12) gdb-2495 [000] ...1 61513.836525: sys_close -> 0x0 gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13) gdb-2495 [000] *..3 61513.836532: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000 gdb-2495 [000] *..3 61513.836532: cobalt_trace_pid: pid=2499, prio=1 gdb-2495 [000] *..3 61513.836533: cobalt_thread_unblock: pid=2499 state=0x48848 info=0x0 gdb-2495 [000] *.~3 61513.836586: cobalt_schedule: status=0x10000000 gdb-2495 [000] *.~3 61513.836586: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] *.~3 61513.836587: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1 smokey-2499 [000] *.~3 61513.836589: cobalt_trace_pid: pid=2499, prio=1 smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined smokey-2499 [000] *.~3 61513.836589: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey smokey-2499 [000] *.~3 61513.836590: cobalt_thread_suspend: pid=2499 mask=0x400080 timeout=0 timeout_mode=0 wchan=0000000000000000 smokey-2499 [000] *.~3 61513.836590: cobalt_schedule: status=0x10000000 smokey-2499 [000] *.~3 61513.836590: cobalt_trace_pid: pid=2499, prio=1 smokey-2499 [000] *.~3 61513.836591: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x4488c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1 gdb-2495 [000] *.~3 61513.836592: cobalt_trace_pid: pid=2495, prio=-1 gdb-2495 [000] d..5 61513.836592: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dN.6 61513.836594: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000 gdb-2495 [000] dN.3 61513.836594: signal_generate: sig=19 errno=0 code=-6 comm=smokey pid=2499 grp=0 res=0 gdb-2495 [000] dNh3 61513.836595: irq_handler_entry: irq=120 name=in-band work gdb-2495 [000] dNh3 61513.836596: cobalt_lostage_wakeup: pid=2499 comm=smokey gdb-2495 [000] *N~4 61513.838552: irq_pipeline_entry: irq=4354 gdb-2495 [000] *N~4 61513.838553: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt gdb-2495 [000] *N~4 61513.838554: cobalt_timer_expire: timer=00000000d13b8aa9 gdb-2495 [000] *N~4 61513.838555: irq_handler_exit: irq=4354 ret=handled gdb-2495 [000] *N~4 61513.838555: irq_pipeline_exit: irq=4354 gdb-2495 [000] dNh8 61513.981277: sched_waking: comm=systemd-journal pid=163 prio=120 target_cpu=000 gdb-2495 [000] dNh9 61513.981278: sched_stat_runtime: comm=gdb pid=2495 runtime=144934239 [ns] vruntime=300660160748 [ns] gdb-2495 [000] dNh9 61513.981280: sched_stat_sleep: comm=systemd-journal pid=163 delay=111416392895 [ns] gdb-2495 [000] dNh9 61513.981280: sched_wakeup: comm=systemd-journal pid=163 prio=120 target_cpu=000 gdb-2495 [000] dNh3 61513.981281: irq_handler_exit: irq=120 ret=handled gdb-2495 [000] dNh3 61513.981282: irq_handler_entry: irq=124 name=proxy tick gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=1 [action=TIMER] gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=9 [action=RCU] gdb-2495 [000] dNh4 61513.981286: sched_stat_runtime: comm=gdb pid=2495 runtime=8375 [ns] vruntime=300660169123 [ns] gdb-2495 [000] dNh5 61513.981288: sched_waking: comm=cobalt_printf pid=2494 prio=120 target_cpu=000 gdb-2495 [000] dNh6 61513.981289: sched_stat_runtime: comm=gdb pid=2495 runtime=2987 [ns] vruntime=300660172110 [ns] gdb-2495 [000] dNh6 61513.981291: sched_stat_sleep: comm=cobalt_printf pid=2494 delay=238660319 [ns] gdb-2495 [000] dNh6 61513.981291: sched_wakeup: comm=cobalt_printf pid=2494 prio=120 target_cpu=000 gdb-2495 [000] dNh5 61513.981292: sched_waking: comm=cobalt_printf pid=2503 prio=120 target_cpu=000 gdb-2495 [000] dNh6 61513.981292: sched_stat_runtime: comm=gdb pid=2495 runtime=3069 [ns] vruntime=300660175179 [ns] gdb-2495 [000] dNh6 61513.981294: sched_stat_sleep: comm=cobalt_printf pid=2503 delay=146611442 [ns] gdb-2495 [000] dNh6 61513.981294: sched_wakeup: comm=cobalt_printf pid=2503 prio=120 target_cpu=000 gdb-2495 [000] *Nh3 61513.981295: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=588476 interval=0 mode=rel gdb-2495 [000] *Nh3 61513.981296: cobalt_tick_shot: next tick at 61513.981884 (delay: 588 us) gdb-2495 [000] dNh3 61513.981296: irq_handler_exit: irq=124 ret=handled gdb-2495 [000] .Ns3 61513.981297: softirq_entry: vec=1 [action=TIMER] gdb-2495 [000] dNs5 61513.981298: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000 gdb-2495 [000] dNs6 61513.981298: sched_stat_runtime: comm=gdb pid=2495 runtime=6218 [ns] vruntime=300660181397 [ns] gdb-2495 [000] dNs6 61513.981299: sched_stat_blocked: comm=rcu_sched pid=11 delay=145119923 [ns] gdb-2495 [000] dNs6 61513.981299: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000 gdb-2495 [000] dNs7 61513.981302: sched_waking: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000 gdb-2495 [000] dNs8 61513.981302: sched_stat_runtime: comm=gdb pid=2495 runtime=3884 [ns] vruntime=300660185281 [ns] gdb-2495 [000] dNs8 61513.981303: sched_stat_blocked: comm=kworker/0:1 pid=1932 delay=206048652 [ns] gdb-2495 [000] dNs8 61513.981304: sched_wakeup: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000 gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] dNs5 61513.981312: softirq_raise: vec=2 [action=NET_TX] gdb-2495 [000] .Ns3 61513.981314: softirq_exit: vec=1 [action=TIMER] gdb-2495 [000] .Ns3 61513.981314: softirq_entry: vec=9 [action=RCU] gdb-2495 [000] .Ns3 61513.981315: softirq_exit: vec=9 [action=RCU] gdb-2495 [000] dNs5 61513.981316: sched_waking: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000 gdb-2495 [000] dNs6 61513.981316: sched_stat_runtime: comm=gdb pid=2495 runtime=13841 [ns] vruntime=300660199122 [ns] gdb-2495 [000] dNs6 61513.981317: sched_stat_sleep: comm=ksoftirqd/0 pid=10 delay=149349034 [ns] gdb-2495 [000] dNs6 61513.981317: sched_wakeup: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000 gdb-2495 [000] .N.1 61513.981318: sys_tkill -> 0x0 gdb-2495 [000] d..3 61513.981320: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98 smokey-2499 [000] *.~3 61513.981888: irq_pipeline_entry: irq=4354 smokey-2499 [000] *.~3 61513.981888: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt smokey-2499 [000] *.~3 61513.981889: cobalt_timer_expire: timer=00000000d13b8aa9 smokey-2499 [000] *.~3 61513.981889: irq_handler_exit: irq=4354 ret=handled smokey-2499 [000] *.~3 61513.981890: irq_pipeline_exit: irq=4354 smokey-2499 [000] d.h1 61514.106435: irq_handler_entry: irq=124 name=proxy tick smokey-2499 [000] *.h1 61514.106441: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2109534 interval=0 mode=rel smokey-2499 [000] *.h1 61514.106441: cobalt_tick_shot: next tick at 61514.108550 (delay: 2109 us) smokey-2499 [000] d.h1 61514.106442: irq_handler_exit: irq=124 ret=handled smokey-2499 [000] ...1 61514.106442: cobalt_shadow_relaxed: state=0x4488c0 info=0x200 smokey-2499 [000] *.~1 61514.108552: irq_pipeline_entry: irq=4354 ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <DM5PR11MB18529649C47BF241930A2217F25E9@DM5PR11MB1852.namprd11.prod.outlook.com>]
[parent not found: <8735v82jmd.fsf@xenomai.org>]
* RE: gdb test failure debug status update [not found] ` <8735v82jmd.fsf@xenomai.org> @ 2021-05-06 2:00 ` Chen, Hongzhan 2021-05-07 1:10 ` Chen, Hongzhan 0 siblings, 1 reply; 16+ messages in thread From: Chen, Hongzhan @ 2021-05-06 2:00 UTC (permalink / raw) To: Philippe Gerum, xenomai >-----Original Message----- >From: Philippe Gerum <rpm@xenomai.org> >Sent: Friday, April 30, 2021 5:43 PM >To: Chen, Hongzhan <hongzhan.chen@intel.com> >Subject: Re: gdb test failure debug status update > > >Chen, Hongzhan <hongzhan.chen@intel.com> writes: > >>>-----Original Message----- >>>From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai >>>Sent: Friday, April 30, 2021 4:07 PM >>>To: Philippe Gerum <rpm@xenomai.org> >>>Cc: xenomai@xenomai.org >>>Subject: RE: gdb test failure debug status update >>> >>> >>> >>>>-----Original Message----- >>>>From: Philippe Gerum <rpm@xenomai.org> >>>>Sent: Friday, April 30, 2021 4:01 PM >>>>To: Chen, Hongzhan <hongzhan.chen@intel.com> >>>>Cc: xenomai@xenomai.org >>>>Subject: Re: gdb test failure debug status update >>>> >>>> >>>>Philippe Gerum <rpm@xenomai.org> writes: >>>> >>>>> Chen, Hongzhan <hongzhan.chen@intel.com> writes: >>>>> >>>>>> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish >>>>>> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax. >>>>>> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in >>>>>> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from >>>>>> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to >>>>>> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode >>>>>> because kernel exit code is never called in this case? >>>>>> >>>>> >>>>> The IRQ frame is indeed kept from unwinding until the preempted task >>>>> context returns from irq_exit_pipeline(), which branches to the Cobalt >>>>> rescheduling procedure. From the Dovetail interface POV, >>>>> irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow >>>>> the companion core to perform post-IRQ chores, such as running its own >>>>> rescheduling procedure. >>>>> >>>>> IOW, if task @foo is preempted by an IRQ, then suspended in oob context >>>>> as a result of what the interrupt handler just did for it (e.g. raising >>>>> XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then >>>>> handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will >>>>> cause the @foo context to switch away, effectively preventing >>>>> handle_irq_pipelined_finish() to return, until @foo resumes execution >>>> eventually. >> >> ln handle_irq_pipelined_finish, irq_exit_pipeline would at first be executed and it >> handle dovetail_call_mayday in the end. But issue happen after run dovetail_call_mayday >> because it call final xnthread_relax before gdb test failue. >> > >Can you add WARN_ON(1) to dovetail_call_mayday() and report about the >output? TIA, > >-- >Philippe. > Please check following output. [ 27.260261] ------------[ cut here ]------------ [ 27.260262] WARNING: CPU: 0 PID: 400 at kernel/dovetail.c:93 dovetail_call_mayday+0x6/0x20 [ 27.260263] Modules linked in: [ 27.260265] CPU: 0 PID: 400 Comm: smokey Tainted: G W 5.10.25+ #588 [ 27.260266] Hardware name: AAEON UP-WHL01/UP-WHL01, BIOS UPW1AM18D 06/23/2020 [ 27.260266] IRQ stage: Xenomai [ 27.260267] RIP: 0010:dovetail_call_mayday+0x6/0x20 [ 27.260269] Code: 00 00 00 00 0f 1f 44 00 00 31 c0 c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 53 <0f> 0b 9c 5b fa f0 80 67 03 fb 48 89 f7 e8 48 04 02 00 53 9d 5b c3 [ 27.260270] RSP: 0000:ffffc900014c3f30 EFLAGS: 00010006 [ 27.260271] RAX: 0000000014000004 RBX: ffffc900014c3f58 RCX: ffff888264e1cf40 [ 27.260272] RDX: ffff888264e1cf40 RSI: ffffc900014c3f58 RDI: ffff88810376e300 [ 27.260273] RBP: ffff888264e1cf40 R08: 0000000000003904 R09: ffff88810b408000 [ 27.260274] R10: ffffc900014c3d48 R11: ffffffff8274a5e8 R12: ffffffff81c00c3a [ 27.260274] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 27.260275] FS: 00007ffff758f740(0000) GS:ffff888264e00000(0000) knlGS:0000000000000000 [ 27.260276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 27.260277] CR2: 00007ff55f9909d8 CR3: 00000001156ea006 CR4: 00000000001706f0 [ 27.260278] Call Trace: [ 27.260279] handle_irq_pipelined_finish+0x154/0x190 [ 27.260279] asm_sysvec_apic_timer_interrupt+0x12/0x20 [ 27.260280] RIP: 0033:0x55555540b4d5 [ 27.260281] Code: Unable to access opcode bytes at RIP 0x55555540b4ab. [ 27.260282] RSP: 002b:00007fffffffe940 EFLAGS: 00000206 [ 27.260283] RAX: 0000000000043028 RBX: 000055555562a140 RCX: 0000000000000000 [ 27.260284] RDX: 000055555541fa4f RSI: 000000000000009f RDI: 000055555541fc68 [ 27.260285] RBP: 00007fffffffe950 R08: 0000000000000000 R09: 00000000ffffffff [ 27.260285] R10: 0000000000000000 R11: 00007ffff799c820 R12: 0000000000000002 [ 27.260287] R13: 000055555562c9f0 R14: 00007fffffffeb04 R15: 00007ffff758f740 [ 27.260287] irq event stamp: 2000 [ 27.260288] hardirqs last enabled at (1999): [<ffffffff8105face>] flush_tlb_mm_range+0x10e/0x140 [ 27.260289] hardirqs last disabled at (2000): [<ffffffff81bd2264>] exc_int3+0x44/0x1a0 [ 27.260290] softirqs last enabled at (784): [<ffffffff81e0030a>] __do_softirq+0x30a/0x42d [ 27.260291] softirqs last disabled at (777): [<ffffffff81c00faf>] asm_call_irq_on_stack+0xf/0x20 [ 27.260292] ---[ end trace ea56d34072e888ed ]--- Regards Hongzhan Chen ^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: gdb test failure debug status update 2021-05-06 2:00 ` Chen, Hongzhan @ 2021-05-07 1:10 ` Chen, Hongzhan 2021-05-09 17:46 ` Philippe Gerum 0 siblings, 1 reply; 16+ messages in thread From: Chen, Hongzhan @ 2021-05-07 1:10 UTC (permalink / raw) To: Philippe Gerum, xenomai >> >>>>-----Original Message----- >>>>From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai >>>>Sent: Friday, April 30, 2021 4:07 PM >>>>To: Philippe Gerum <rpm@xenomai.org> >>>>Cc: xenomai@xenomai.org >>>>Subject: RE: gdb test failure debug status update >>>> >>>> >>>> >>>>>-----Original Message----- >>>>>From: Philippe Gerum <rpm@xenomai.org> >>>>>Sent: Friday, April 30, 2021 4:01 PM >>>>>To: Chen, Hongzhan <hongzhan.chen@intel.com> >>>>>Cc: xenomai@xenomai.org >>>>>Subject: Re: gdb test failure debug status update >>>>> >>>>> >>>>>Philippe Gerum <rpm@xenomai.org> writes: >>>>> >>>>>> Chen, Hongzhan <hongzhan.chen@intel.com> writes: >>>>>> >>>>>>> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish >>>>>>> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax. >>>>>>> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in >>>>>>> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from >>>>>>> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to >>>>>>> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode >>>>>>> because kernel exit code is never called in this case? >>>>>>> >>>>>> >>>>>> The IRQ frame is indeed kept from unwinding until the preempted task >>>>>> context returns from irq_exit_pipeline(), which branches to the Cobalt >>>>>> rescheduling procedure. From the Dovetail interface POV, >>>>>> irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow >>>>>> the companion core to perform post-IRQ chores, such as running its own >>>>>> rescheduling procedure. >>>>>> >>>>>> IOW, if task @foo is preempted by an IRQ, then suspended in oob context >>>>>> as a result of what the interrupt handler just did for it (e.g. raising >>>>>> XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then >>>>>> handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will >>>>>> cause the @foo context to switch away, effectively preventing >>>>>> handle_irq_pipelined_finish() to return, until @foo resumes execution >>>>> eventually. >>> >>> ln handle_irq_pipelined_finish, irq_exit_pipeline would at first be executed and it >>> handle dovetail_call_mayday in the end. But issue happen after run dovetail_call_mayday >>> because it call final xnthread_relax before gdb test failue. >>> >> >>Can you add WARN_ON(1) to dovetail_call_mayday() and report about the >>output? TIA, >> >>-- >>Philippe. >> > >Please check following output. Hi Philippe, Please let me know if you have new patch or other thing to let me try. Regards Hongzhan Chen > >[ 27.260261] ------------[ cut here ]------------ >[ 27.260262] WARNING: CPU: 0 PID: 400 at kernel/dovetail.c:93 dovetail_call_mayday+0x6/0x20 >[ 27.260263] Modules linked in: >[ 27.260265] CPU: 0 PID: 400 Comm: smokey Tainted: G W 5.10.25+ #588 >[ 27.260266] Hardware name: AAEON UP-WHL01/UP-WHL01, BIOS UPW1AM18D 06/23/2020 >[ 27.260266] IRQ stage: Xenomai >[ 27.260267] RIP: 0010:dovetail_call_mayday+0x6/0x20 >[ 27.260269] Code: 00 00 00 00 0f 1f 44 00 00 31 c0 c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 53 <0f> 0b 9c 5b fa f0 80 67 03 fb 48 89 f7 e8 48 04 02 00 53 9d 5b c3 >[ 27.260270] RSP: 0000:ffffc900014c3f30 EFLAGS: 00010006 >[ 27.260271] RAX: 0000000014000004 RBX: ffffc900014c3f58 RCX: ffff888264e1cf40 >[ 27.260272] RDX: ffff888264e1cf40 RSI: ffffc900014c3f58 RDI: ffff88810376e300 >[ 27.260273] RBP: ffff888264e1cf40 R08: 0000000000003904 R09: ffff88810b408000 >[ 27.260274] R10: ffffc900014c3d48 R11: ffffffff8274a5e8 R12: ffffffff81c00c3a >[ 27.260274] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 >[ 27.260275] FS: 00007ffff758f740(0000) GS:ffff888264e00000(0000) knlGS:0000000000000000 >[ 27.260276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >[ 27.260277] CR2: 00007ff55f9909d8 CR3: 00000001156ea006 CR4: 00000000001706f0 >[ 27.260278] Call Trace: >[ 27.260279] handle_irq_pipelined_finish+0x154/0x190 >[ 27.260279] asm_sysvec_apic_timer_interrupt+0x12/0x20 >[ 27.260280] RIP: 0033:0x55555540b4d5 >[ 27.260281] Code: Unable to access opcode bytes at RIP 0x55555540b4ab. >[ 27.260282] RSP: 002b:00007fffffffe940 EFLAGS: 00000206 >[ 27.260283] RAX: 0000000000043028 RBX: 000055555562a140 RCX: 0000000000000000 >[ 27.260284] RDX: 000055555541fa4f RSI: 000000000000009f RDI: 000055555541fc68 >[ 27.260285] RBP: 00007fffffffe950 R08: 0000000000000000 R09: 00000000ffffffff >[ 27.260285] R10: 0000000000000000 R11: 00007ffff799c820 R12: 0000000000000002 >[ 27.260287] R13: 000055555562c9f0 R14: 00007fffffffeb04 R15: 00007ffff758f740 >[ 27.260287] irq event stamp: 2000 >[ 27.260288] hardirqs last enabled at (1999): [<ffffffff8105face>] flush_tlb_mm_range+0x10e/0x140 >[ 27.260289] hardirqs last disabled at (2000): [<ffffffff81bd2264>] exc_int3+0x44/0x1a0 >[ 27.260290] softirqs last enabled at (784): [<ffffffff81e0030a>] __do_softirq+0x30a/0x42d >[ 27.260291] softirqs last disabled at (777): [<ffffffff81c00faf>] asm_call_irq_on_stack+0xf/0x20 >[ 27.260292] ---[ end trace ea56d34072e888ed ]--- > >Regards > >Hongzhan Chen ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: gdb test failure debug status update 2021-05-07 1:10 ` Chen, Hongzhan @ 2021-05-09 17:46 ` Philippe Gerum 2021-05-09 17:49 ` Philippe Gerum 2021-05-15 15:55 ` Philippe Gerum 0 siblings, 2 replies; 16+ messages in thread From: Philippe Gerum @ 2021-05-09 17:46 UTC (permalink / raw) To: Chen, Hongzhan; +Cc: xenomai Chen, Hongzhan <hongzhan.chen@intel.com> writes: >>> >>>>>-----Original Message----- >>>>>From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai >>>>>Sent: Friday, April 30, 2021 4:07 PM >>>>>To: Philippe Gerum <rpm@xenomai.org> >>>>>Cc: xenomai@xenomai.org >>>>>Subject: RE: gdb test failure debug status update >>>>> >>>>> >>>>> >>>>>>-----Original Message----- >>>>>>From: Philippe Gerum <rpm@xenomai.org> >>>>>>Sent: Friday, April 30, 2021 4:01 PM >>>>>>To: Chen, Hongzhan <hongzhan.chen@intel.com> >>>>>>Cc: xenomai@xenomai.org >>>>>>Subject: Re: gdb test failure debug status update >>>>>> >>>>>> >>>>>>Philippe Gerum <rpm@xenomai.org> writes: >>>>>> >>>>>>> Chen, Hongzhan <hongzhan.chen@intel.com> writes: >>>>>>> >>>>>>>> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish >>>>>>>> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax. >>>>>>>> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in >>>>>>>> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from >>>>>>>> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to >>>>>>>> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode >>>>>>>> because kernel exit code is never called in this case? >>>>>>>> >>>>>>> >>>>>>> The IRQ frame is indeed kept from unwinding until the preempted task >>>>>>> context returns from irq_exit_pipeline(), which branches to the Cobalt >>>>>>> rescheduling procedure. From the Dovetail interface POV, >>>>>>> irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow >>>>>>> the companion core to perform post-IRQ chores, such as running its own >>>>>>> rescheduling procedure. >>>>>>> >>>>>>> IOW, if task @foo is preempted by an IRQ, then suspended in oob context >>>>>>> as a result of what the interrupt handler just did for it (e.g. raising >>>>>>> XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then >>>>>>> handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will >>>>>>> cause the @foo context to switch away, effectively preventing >>>>>>> handle_irq_pipelined_finish() to return, until @foo resumes execution >>>>>> eventually. >>>> >>>> ln handle_irq_pipelined_finish, irq_exit_pipeline would at first be executed and it >>>> handle dovetail_call_mayday in the end. But issue happen after run dovetail_call_mayday >>>> because it call final xnthread_relax before gdb test failue. >>>> >>> >>>Can you add WARN_ON(1) to dovetail_call_mayday() and report about the >>>output? TIA, >>> >>>-- >>>Philippe. >>> >> >>Please check following output. > > Hi Philippe, > > Please let me know if you have new patch or other thing to let me try. > I spent hours of this issue, and there may be a wrong basic assumption done in the smokey/gdb test. Specifically, handle_sigwake_event() un-stops the debuggee (lifting XNDBGSTOP), then sends a mayday notice to make sure that debuggee re-enters the kernel asap for leaving the oob stage. What might happen between these two events might not be as well-defined as this test expects (e.g. what if the debugger might be able to run more user code before the mayday trap is enforced?). I'll keep on debugging that stuff and let you know. -- Philippe. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: gdb test failure debug status update 2021-05-09 17:46 ` Philippe Gerum @ 2021-05-09 17:49 ` Philippe Gerum 2021-05-10 2:16 ` Chen, Hongzhan 2021-05-15 15:55 ` Philippe Gerum 1 sibling, 1 reply; 16+ messages in thread From: Philippe Gerum @ 2021-05-09 17:49 UTC (permalink / raw) To: Chen, Hongzhan; +Cc: xenomai Philippe Gerum <rpm@xenomai.org> writes: > Chen, Hongzhan <hongzhan.chen@intel.com> writes: > >>>> >>>>>>-----Original Message----- >>>>>>From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai >>>>>>Sent: Friday, April 30, 2021 4:07 PM >>>>>>To: Philippe Gerum <rpm@xenomai.org> >>>>>>Cc: xenomai@xenomai.org >>>>>>Subject: RE: gdb test failure debug status update >>>>>> >>>>>> >>>>>> >>>>>>>-----Original Message----- >>>>>>>From: Philippe Gerum <rpm@xenomai.org> >>>>>>>Sent: Friday, April 30, 2021 4:01 PM >>>>>>>To: Chen, Hongzhan <hongzhan.chen@intel.com> >>>>>>>Cc: xenomai@xenomai.org >>>>>>>Subject: Re: gdb test failure debug status update >>>>>>> >>>>>>> >>>>>>>Philippe Gerum <rpm@xenomai.org> writes: >>>>>>> >>>>>>>> Chen, Hongzhan <hongzhan.chen@intel.com> writes: >>>>>>>> >>>>>>>>> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish >>>>>>>>> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax. >>>>>>>>> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in >>>>>>>>> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from >>>>>>>>> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to >>>>>>>>> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode >>>>>>>>> because kernel exit code is never called in this case? >>>>>>>>> >>>>>>>> >>>>>>>> The IRQ frame is indeed kept from unwinding until the preempted task >>>>>>>> context returns from irq_exit_pipeline(), which branches to the Cobalt >>>>>>>> rescheduling procedure. From the Dovetail interface POV, >>>>>>>> irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow >>>>>>>> the companion core to perform post-IRQ chores, such as running its own >>>>>>>> rescheduling procedure. >>>>>>>> >>>>>>>> IOW, if task @foo is preempted by an IRQ, then suspended in oob context >>>>>>>> as a result of what the interrupt handler just did for it (e.g. raising >>>>>>>> XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then >>>>>>>> handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will >>>>>>>> cause the @foo context to switch away, effectively preventing >>>>>>>> handle_irq_pipelined_finish() to return, until @foo resumes execution >>>>>>> eventually. >>>>> >>>>> ln handle_irq_pipelined_finish, irq_exit_pipeline would at first be executed and it >>>>> handle dovetail_call_mayday in the end. But issue happen after run dovetail_call_mayday >>>>> because it call final xnthread_relax before gdb test failue. >>>>> >>>> >>>>Can you add WARN_ON(1) to dovetail_call_mayday() and report about the >>>>output? TIA, >>>> >>>>-- >>>>Philippe. >>>> >>> >>>Please check following output. >> >> Hi Philippe, >> >> Please let me know if you have new patch or other thing to let me try. >> > > I spent hours of this issue, and there may be a wrong basic assumption > done in the smokey/gdb test. Specifically, handle_sigwake_event() > un-stops the debuggee (lifting XNDBGSTOP), then sends a mayday notice to > make sure that debuggee re-enters the kernel asap for leaving the oob > stage. What might happen between these two events might not be as > well-defined as this test expects (e.g. what if the debugger might be I meant the _debuggee_ (obvously not the debugger in this case) > able to run more user code before the mayday trap is enforced?). > > I'll keep on debugging that stuff and let you know. -- Philippe. ^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: gdb test failure debug status update 2021-05-09 17:49 ` Philippe Gerum @ 2021-05-10 2:16 ` Chen, Hongzhan 0 siblings, 0 replies; 16+ messages in thread From: Chen, Hongzhan @ 2021-05-10 2:16 UTC (permalink / raw) To: Philippe Gerum; +Cc: xenomai >Philippe Gerum <rpm@xenomai.org> writes: > >> Chen, Hongzhan <hongzhan.chen@intel.com> writes: >> >>>>> >>>>>>>-----Original Message----- >>>>>>>From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai >>>>>>>Sent: Friday, April 30, 2021 4:07 PM >>>>>>>To: Philippe Gerum <rpm@xenomai.org> >>>>>>>Cc: xenomai@xenomai.org >>>>>>>Subject: RE: gdb test failure debug status update >>>>>>> >>>>>>> >>>>>>> >>>>>>>>-----Original Message----- >>>>>>>>From: Philippe Gerum <rpm@xenomai.org> >>>>>>>>Sent: Friday, April 30, 2021 4:01 PM >>>>>>>>To: Chen, Hongzhan <hongzhan.chen@intel.com> >>>>>>>>Cc: xenomai@xenomai.org >>>>>>>>Subject: Re: gdb test failure debug status update >>>>>>>> >>>>>>>> >>>>>>>>Philippe Gerum <rpm@xenomai.org> writes: >>>>>>>> >>>>>>>>> Chen, Hongzhan <hongzhan.chen@intel.com> writes: >>>>>>>>> >>>>>>>>>> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish >>>>>>>>>> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax. >>>>>>>>>> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in >>>>>>>>>> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from >>>>>>>>>> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to >>>>>>>>>> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode >>>>>>>>>> because kernel exit code is never called in this case? >>>>>>>>>> >>>>>>>>> >>>>>>>>> The IRQ frame is indeed kept from unwinding until the preempted task >>>>>>>>> context returns from irq_exit_pipeline(), which branches to the Cobalt >>>>>>>>> rescheduling procedure. From the Dovetail interface POV, >>>>>>>>> irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow >>>>>>>>> the companion core to perform post-IRQ chores, such as running its own >>>>>>>>> rescheduling procedure. >>>>>>>>> >>>>>>>>> IOW, if task @foo is preempted by an IRQ, then suspended in oob context >>>>>>>>> as a result of what the interrupt handler just did for it (e.g. raising >>>>>>>>> XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then >>>>>>>>> handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will >>>>>>>>> cause the @foo context to switch away, effectively preventing >>>>>>>>> handle_irq_pipelined_finish() to return, until @foo resumes execution >>>>>>>> eventually. >>>>>> >>>>>> ln handle_irq_pipelined_finish, irq_exit_pipeline would at first be executed and it >>>>>> handle dovetail_call_mayday in the end. But issue happen after run dovetail_call_mayday >>>>>> because it call final xnthread_relax before gdb test failue. >>>>>> >>>>> >>>>>Can you add WARN_ON(1) to dovetail_call_mayday() and report about the >>>>>output? TIA, >>>>> >>>>>-- >>>>>Philippe. >>>>> >>>> >>>>Please check following output. >>> >>> Hi Philippe, >>> >>> Please let me know if you have new patch or other thing to let me try. >>> >> >> I spent hours of this issue, and there may be a wrong basic assumption >> done in the smokey/gdb test. Specifically, handle_sigwake_event() >> un-stops the debuggee (lifting XNDBGSTOP), then sends a mayday notice to Current handle_sigwake_event call register_debugged_thread to suspend all threads including smokey thread that be subordinated to same process when hi-thread get sigwake event. Logically , we should relax all threads that be subordinated to same process after get mayday event because we already specifically handle it before and then ask xnthread schedule to schedule correct thread to run(which would schedule hi-thread at first finally and get correct behavior). Is this idea OK? Regards Hongzhan Chen >> make sure that debuggee re-enters the kernel asap for leaving the oob >> stage. What might happen between these two events might not be as >> well-defined as this test expects (e.g. what if the debugger might be > >I meant the _debuggee_ (obvously not the debugger in this case) > >> able to run more user code before the mayday trap is enforced?). >> >> I'll keep on debugging that stuff and let you know. > > >-- >Philippe. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: gdb test failure debug status update 2021-05-09 17:46 ` Philippe Gerum 2021-05-09 17:49 ` Philippe Gerum @ 2021-05-15 15:55 ` Philippe Gerum 1 sibling, 0 replies; 16+ messages in thread From: Philippe Gerum @ 2021-05-15 15:55 UTC (permalink / raw) To: Chen, Hongzhan; +Cc: xenomai Philippe Gerum <rpm@xenomai.org> writes: > Chen, Hongzhan <hongzhan.chen@intel.com> writes: > >>>> >>>>>>-----Original Message----- >>>>>>From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai >>>>>>Sent: Friday, April 30, 2021 4:07 PM >>>>>>To: Philippe Gerum <rpm@xenomai.org> >>>>>>Cc: xenomai@xenomai.org >>>>>>Subject: RE: gdb test failure debug status update >>>>>> >>>>>> >>>>>> >>>>>>>-----Original Message----- >>>>>>>From: Philippe Gerum <rpm@xenomai.org> >>>>>>>Sent: Friday, April 30, 2021 4:01 PM >>>>>>>To: Chen, Hongzhan <hongzhan.chen@intel.com> >>>>>>>Cc: xenomai@xenomai.org >>>>>>>Subject: Re: gdb test failure debug status update >>>>>>> >>>>>>> >>>>>>>Philippe Gerum <rpm@xenomai.org> writes: >>>>>>> >>>>>>>> Chen, Hongzhan <hongzhan.chen@intel.com> writes: >>>>>>>> >>>>>>>>> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish >>>>>>>>> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax. >>>>>>>>> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in >>>>>>>>> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from >>>>>>>>> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to >>>>>>>>> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode >>>>>>>>> because kernel exit code is never called in this case? >>>>>>>>> >>>>>>>> >>>>>>>> The IRQ frame is indeed kept from unwinding until the preempted task >>>>>>>> context returns from irq_exit_pipeline(), which branches to the Cobalt >>>>>>>> rescheduling procedure. From the Dovetail interface POV, >>>>>>>> irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow >>>>>>>> the companion core to perform post-IRQ chores, such as running its own >>>>>>>> rescheduling procedure. >>>>>>>> >>>>>>>> IOW, if task @foo is preempted by an IRQ, then suspended in oob context >>>>>>>> as a result of what the interrupt handler just did for it (e.g. raising >>>>>>>> XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then >>>>>>>> handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will >>>>>>>> cause the @foo context to switch away, effectively preventing >>>>>>>> handle_irq_pipelined_finish() to return, until @foo resumes execution >>>>>>> eventually. >>>>> >>>>> ln handle_irq_pipelined_finish, irq_exit_pipeline would at first be executed and it >>>>> handle dovetail_call_mayday in the end. But issue happen after run dovetail_call_mayday >>>>> because it call final xnthread_relax before gdb test failue. >>>>> >>>> >>>>Can you add WARN_ON(1) to dovetail_call_mayday() and report about the >>>>output? TIA, >>>> >>>>-- >>>>Philippe. >>>> >>> >>>Please check following output. >> >> Hi Philippe, >> >> Please let me know if you have new patch or other thing to let me try. >> > > I spent hours of this issue, and there may be a wrong basic assumption > done in the smokey/gdb test. Specifically, handle_sigwake_event() > un-stops the debuggee (lifting XNDBGSTOP), then sends a mayday notice to > make sure that debuggee re-enters the kernel asap for leaving the oob > stage. What might happen between these two events might not be as > well-defined as this test expects (e.g. what if the debugger might be > able to run more user code before the mayday trap is enforced?). > > I'll keep on debugging that stuff and let you know. This one was nailed down eventually. As you found out, the "retuser" event was missed and left pending. This could happen when a task resumes to user, unwinding an IRQ frame, while being demoted from oob to in-band context in the process. This is typically the case of the gdb test: [hi-pri-task] (...timed sleep...) [lo-spin-task] (...spinning...) TIMER-IRQENTRY (wakeup hi-pri-task) [hi-pri-task] (breakpoint) [gdb] tkill(lo-spin-task, SIGTRAP) handle_sigwake_event(lo-spin-task) notify_mayday(lo-spin-task) [lo-spin-task] handle_mayday_event(lo-spin-task) switch_inband() TIMER-IRQEXIT ** missing check for pending _TIF_WORK|_TIF_RETUSER ** Which explains why lo-spin-task could run un-preempted by hi-pri-task for a while, until the Cobalt core hits a rescheduling point eventually. Only x86 is affected, ARM and arm64 have a different way out of IRQ context which does not exhibit such issue. This bug is now fixed by [1] for v5.10-dovetail. As I was at it, I added a missing u-call request to the leave_oob helper. Thanks for your help in digging into this. [1] https://git.evlproject.org/linux-evl.git/commit/?h=dovetail/v5.10&id=cfcab38909d870d1ef484cd401fa00e52e86a8d0 -- Philippe. ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2021-05-15 15:55 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-28 3:19 gdb test failure debug status update Chen, Hongzhan
2021-04-28 14:18 ` Philippe Gerum
2021-04-28 14:30 ` Philippe Gerum
2021-04-29 6:31 ` Chen, Hongzhan
2021-04-30 5:25 ` Chen, Hongzhan
2021-04-30 5:51 ` Chen, Hongzhan
2021-04-30 7:36 ` Philippe Gerum
2021-04-30 7:34 ` Philippe Gerum
2021-04-30 8:00 ` Philippe Gerum
2021-04-30 8:07 ` Chen, Hongzhan
[not found] ` <DM5PR11MB18529649C47BF241930A2217F25E9@DM5PR11MB1852.namprd11.prod.outlook.com>
[not found] ` <8735v82jmd.fsf@xenomai.org>
2021-05-06 2:00 ` Chen, Hongzhan
2021-05-07 1:10 ` Chen, Hongzhan
2021-05-09 17:46 ` Philippe Gerum
2021-05-09 17:49 ` Philippe Gerum
2021-05-10 2:16 ` Chen, Hongzhan
2021-05-15 15:55 ` Philippe Gerum
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.