* gdb test failure debug status update
@ 2021-04-28 3:19 Chen, Hongzhan
2021-04-28 14:18 ` Philippe Gerum
0 siblings, 1 reply; 16+ messages in thread
From: Chen, Hongzhan @ 2021-04-28 3:19 UTC (permalink / raw)
To: xenomai
According to my validation, gdb test fail on dovetail 5.10 branch but pass on v5.9-evl4 tag with same for-upstream/dovetail
xenomai code base.
After further debug , the issue is more clear for me. Gdb test failure because low priority thread smokey userspace is still
executed after "cobalt_shadow_relaxed: state=0x4488c0 info=0x200" like log [1] on dovetail-5.10 branch.
The weird thing is that its following first ftrace log happen at 62235.848583 after cobalt_shadow_relaxed in log [1].
It is almost 3ms happened after cobalt_shadow_relaxed. The low priority smoke thread user space is executed during this
3ms period so that test fail.
But in success case with v5.9-evl4 like in log [2], the time interval between cobalt_shadow_relaxed and the following first ftrace log
is only about 1us. It seems that low priority smokey userspace do not have chance to execute in this 1us because gdb test is successful.
My question is why there is even no interrupt happened during that about 3ms period in failure case? Tick seems in abnormal behavior.
Please comment if you have any ideas to further debug it.
PS: All my tests run on same up Xtream board.
1:
smokey-2024 [000] ..s1 62235.845648: softirq_exit: vec=6 [action=TASKLET]
smokey-2024 [000] d.s2 62235.845648: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x312
smokey-2024 [000] d.s1 62235.845648: irq_disable: caller=__do_softirq+0x280/0x428 parent=0x0
smokey-2024 [000] d.s2 62235.845648: rcu_dyntick: --= 4000000000000002 4000000000000000 0x312
smokey-2024 [000] *..2 62235.845649: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x312
smokey-2024 [000] *..1 62235.845649: irq_enable: caller=sync_current_irq_stage+0x259/0x280 parent=0x0
smokey-2024 [000] *..2 62235.845649: rcu_dyntick: --= 4000000000000002 4000000000000000 0x312
smokey-2024 [000] ...1 62235.845650: cobalt_shadow_relaxed: state=0x4488c0 info=0x200
smokey-2024 [000] *..1 62235.848583: irq_disable: caller=irqentry_enter+0x40/0xe0 parent=0x0
smokey-2024 [000] *.~1 62235.848583: irq_pipeline_entry: irq=4354
smokey-2024 [000] *.~1 62235.848584: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
smokey-2024 [000] *.~1 62235.848584: local_timer_entry: vector=236
smokey-2024 [000] *.~1 62235.848585: cobalt_timer_expire: timer=000000002aae459f
smokey-2024 [000] *.~1 62235.848585: local_timer_exit: vector=236
smokey-2024 [000] *.~1 62235.848585: irq_handler_exit: irq=4354 ret=handled
smokey-2024 [000] *.~1 62235.848585: irq_pipeline_exit: irq=4354
smokey-2024 [000] *.~1 62235.848586: irq_enable: caller=arch_pipeline_entry+0x4e/0xc0 parent=0x0
smokey-2024 [000] *..2 62235.848586: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x312
smokey-2024 [000] *..1 62235.848586: irq_disable: caller=sync_current_irq_stage+0x246/0x280 parent=0x0
smokey-2024 [000] *..2 62235.848586: rcu_dyntick: --= 4000000000000002 4000000000000000 0x312
smokey-2024 [000] d.h1 62235.848587: irq_handler_entry: irq=124 name=proxy tick
smokey-2024 [000] d.h2 62235.848587: lock_acquire: 000000004fef83e9 hrtimer_bases.lock
smokey-2024 [000] d.h2 62235.848588: lock_acquire: 00000000ed2aa56a read tk_core.seq.seqcount
smokey-2024 [000] d.h2 62235.848588: lock_release: 00000000ed2aa56a tk_core.seq.seqcount
smokey-2024 [000] d.h2 62235.848588: hrtimer_cancel: hrtimer=000000009dcbf7ab
smokey-2024 [000] d.h2 62235.848588: lock_release: 000000004fef83e9 hrtimer_bases.lock
smokey-2024 [000] d.h1 62235.848589: hrtimer_expire_entry: hrtimer=000000009dcbf7ab function=tick_sched_timer now=62232458752330
smokey-2024 [000] d.h1 62235.848589: lock_acquire: 00000000ed2aa56a read tk_core.seq.seqcount
smokey-2024 [000] d.h1 62235.848589: lock_release: 00000000ed2aa56a tk_core.seq.seqcount
smokey-2024 [000] d.h2 62235.848590: lock_acquire: 000000002286d635 jiffies_lock
smokey-2024 [000] d.h2 62235.848590: lock_acquire: 000000007174a29a (null)
smokey-2024 [000] d.h2 62235.848590: lock_release: 000000007174a29a (null)
[2] :
smokey-703 [000] d.s2 5674.404286: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x26e
smokey-703 [000] d.s1 5674.404286: irq_disable: caller=inband_irq_save+0x20/0x30 parent=0x0
smokey-703 [000] d.s2 5674.404286: rcu_dyntick: --= 4000000000000002 4000000000000000 0x26e
smokey-703 [000] d.s1 5674.404287: rcu_batch_end: rcu_sched CBs-invoked=8 idle=....
smokey-703 [000] d.s2 5674.404287: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x26e
smokey-703 [000] d.s1 5674.404287: irq_enable: caller=rcu_core+0x3b6/0xb60 parent=0x0
smokey-703 [000] d.s2 5674.404287: rcu_dyntick: --= 4000000000000002 4000000000000000 0x26e
smokey-703 [000] ..s1 5674.404288: rcu_utilization: End RCU core
smokey-703 [000] ..s1 5674.404288: softirq_exit: vec=9 [action=RCU]
smokey-703 [000] d.s2 5674.404288: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x26e
smokey-703 [000] d.s1 5674.404288: irq_disable: caller=__do_softirq+0x2bd/0x471 parent=0x0
smokey-703 [000] d.s2 5674.404289: rcu_dyntick: --= 4000000000000002 4000000000000000 0x26e
smokey-703 [000] *..2 5674.404289: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x26e
smokey-703 [000] *..1 5674.404289: irq_enable: caller=sync_current_irq_stage+0x259/0x280 parent=0x0
smokey-703 [000] *..2 5674.404290: rcu_dyntick: --= 4000000000000002 4000000000000000 0x26e
smokey-703 [000] ...1 5674.404290: cobalt_shadow_relaxed: state=0x4488c0 info=0x200
smokey-703 [000] *..2 5674.404291: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x26e
smokey-703 [000] *..1 5674.404291: irq_disable: caller=arch_pipeline_entry+0x13a/0x170 parent=0x0
smokey-703 [000] *..2 5674.404292: rcu_dyntick: --= 4000000000000002 4000000000000000 0x26e
smokey-703 [000] d..2 5674.404292: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x26e
smokey-703 [000] d..1 5674.404292: irq_enable: caller=exit_to_user_mode_prepare+0x118/0x1e0 parent=0x0
smokey-703 [000] d..2 5674.404292: rcu_dyntick: --= 4000000000000002 4000000000000000 0x26e
smokey-703 [000] d..2 5674.404293: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x26e
smokey-703 [000] d..1 5674.404293: irq_disable: caller=_raw_spin_lock_irq+0x12/0x50 parent=0x0
smokey-703 [000] d..2 5674.404293: rcu_dyntick: --= 4000000000000002 4000000000000000 0x26e
smokey-703 [000] d..2 5674.404294: lock_acquire: 000000007f29b2fb &sighand->siglock
smokey-703 [000] d..2 5674.404294: kmem_cache_free: call_site=__dequeue_signal+0xf7/0x150 ptr=000000003929a99e
smokey-703 [000] d..3 5674.404295: lock_acquire: 000000002505fb17 &p->pi_lock
smokey-703 [000] d..3 5674.404295: lock_release: 000000002505fb17 &p->pi_lock
smokey-703 [000] d..2 5674.404296: lock_release: 000000007f29b2fb &sighand->siglock
smokey-703 [000] d..3 5674.404296: rcu_dyntick: ++= 4000000000000000 4000000000000002 0x26e
smokey-703 [000] d..2 5674.404296: irq_enable: caller=_raw_spin_unlock_irq+0x1f/0x30 parent=0x0
smokey-703 [000] d..3 5674.404296: rcu_dyntick: --= 4000000000000002 4000000000000000 0x26e
smokey-703 [000] ...2 5674.404297: lock_acquire: 00000000e714a308 read tasklist_lock
smokey-703 [000] ...3 5674.404297: lock_acquire: 00000000909a50fe read rcu_read_lock
Regards
Hongzhan Chen
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: gdb test failure debug status update
2021-04-28 3:19 gdb test failure debug status update Chen, Hongzhan
@ 2021-04-28 14:18 ` Philippe Gerum
2021-04-28 14:30 ` Philippe Gerum
0 siblings, 1 reply; 16+ messages in thread
From: Philippe Gerum @ 2021-04-28 14:18 UTC (permalink / raw)
To: Chen, Hongzhan; +Cc: xenomai
Chen, Hongzhan via Xenomai <xenomai@xenomai.org> writes:
> According to my validation, gdb test fail on dovetail 5.10 branch but pass on v5.9-evl4 tag with same for-upstream/dovetail
> xenomai code base.
>
> After further debug , the issue is more clear for me. Gdb test failure because low priority thread smokey userspace is still
> executed after "cobalt_shadow_relaxed: state=0x4488c0 info=0x200" like log [1] on dovetail-5.10 branch.
> The weird thing is that its following first ftrace log happen at 62235.848583 after cobalt_shadow_relaxed in log [1].
> It is almost 3ms happened after cobalt_shadow_relaxed. The low priority smoke thread user space is executed during this
> 3ms period so that test fail.
>
> But in success case with v5.9-evl4 like in log [2], the time interval between cobalt_shadow_relaxed and the following first ftrace log
> is only about 1us. It seems that low priority smokey userspace do not have chance to execute in this 1us because gdb test is successful.
>
> My question is why there is even no interrupt happened during that about 3ms period in failure case? Tick seems in abnormal behavior.
> Please comment if you have any ideas to further debug it.
>
> PS: All my tests run on same up Xtream board.
<snip>
Let's put aside the tick issue for now, there may be a valid reason for
this delay with dynticks enabled.
The issue at stake may be related to the way a return to kernel space is
forced on a @user task (Dovetail has an integrated service for
triggering this called dovetail_request_ucall()).
The logic for doing so is as follows:
1. @user hits a breakpoint, which is an exception Dovetail-wise
2. @user gets XNDBGSTOP set into its flags because Cobalt notices it is
being debugged via a breakpoint trap, then relaxed as a result of taking
a exception in general, so that we may traverse the common trap handling code
safely.
3. since XNDBGSTOP is a blocking bit Cobalt-wise, it should prevent
@user from being picked for scheduling by the real-time core, next time
a Cobalt considers rescheduling that is. However, since @user is
currently relaxed, it can still run under the supervision of the common
Linux scheduler. This is what the log[1] show.
4. the common/in-band kernel code stops @user due to the ptrace stop
condition caused by the breakpoint, waiting for a continuation event to
happen.
Therefore, upon PTRACE_CONT (i.e. gdb continue), we need to force @user
to call back into kernel context (handle_ptrace_cont ->
dovetail_request_ucall), then ask for a switch to primary mode from
there, which should eventually happen when @user is about to leave the
kernel (on x86, this now happens from a generic kernel entry/exit code
in kernel/entry/*). As a result, handle_taskexit_event() runs, figures
out that @user is pending a switch to primary mode. As it switches to
primary mode, @user would be blocked by Cobalt from running further,
because XNDBGSTOP is set into its internal state.
So, I would check a few things for starters:
- is dovetail_request_ucall() working properly.
- is XNCONTHI properly set into the local Cobalt flags of @user when
handle_user_return() is entered.
- is this path taken as expected once dovetail_request_ucall() has run
for @user:
exit_to_user_mode_prepare -> do_retuser -> inband_retuser_notify
(kernel/entry/common.c)?
It may be a good idea to enable all cobalt tracepoints, add one to
handle_user_return() too.
--
Philippe.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: gdb test failure debug status update
2021-04-28 14:18 ` Philippe Gerum
@ 2021-04-28 14:30 ` Philippe Gerum
2021-04-29 6:31 ` Chen, Hongzhan
0 siblings, 1 reply; 16+ messages in thread
From: Philippe Gerum @ 2021-04-28 14:30 UTC (permalink / raw)
To: Philippe Gerum; +Cc: Chen, Hongzhan, xenomai
Philippe Gerum via Xenomai <xenomai@xenomai.org> writes:
> Chen, Hongzhan via Xenomai <xenomai@xenomai.org> writes:
>
>> According to my validation, gdb test fail on dovetail 5.10 branch but pass on v5.9-evl4 tag with same for-upstream/dovetail
>> xenomai code base.
>>
>> After further debug , the issue is more clear for me. Gdb test failure because low priority thread smokey userspace is still
>> executed after "cobalt_shadow_relaxed: state=0x4488c0 info=0x200" like log [1] on dovetail-5.10 branch.
>> The weird thing is that its following first ftrace log happen at 62235.848583 after cobalt_shadow_relaxed in log [1].
>> It is almost 3ms happened after cobalt_shadow_relaxed. The low priority smoke thread user space is executed during this
>> 3ms period so that test fail.
>>
>> But in success case with v5.9-evl4 like in log [2], the time interval between cobalt_shadow_relaxed and the following first ftrace log
>> is only about 1us. It seems that low priority smokey userspace do not have chance to execute in this 1us because gdb test is successful.
>>
>> My question is why there is even no interrupt happened during that about 3ms period in failure case? Tick seems in abnormal behavior.
>> Please comment if you have any ideas to further debug it.
>>
>> PS: All my tests run on same up Xtream board.
>
> <snip>
>
> Let's put aside the tick issue for now, there may be a valid reason for
> this delay with dynticks enabled.
>
> The issue at stake may be related to the way a return to kernel space is
> forced on a @user task (Dovetail has an integrated service for
> triggering this called dovetail_request_ucall()).
>
> The logic for doing so is as follows:
>
> 1. @user hits a breakpoint, which is an exception Dovetail-wise
>
> 2. @user gets XNDBGSTOP set into its flags because Cobalt notices it is
> being debugged via a breakpoint trap, then relaxed as a result of taking
> a exception in general, so that we may traverse the common trap handling code
> safely.
>
> 3. since XNDBGSTOP is a blocking bit Cobalt-wise, it should prevent
> @user from being picked for scheduling by the real-time core, next time
> a Cobalt considers rescheduling that is. However, since @user is
> currently relaxed, it can still run under the supervision of the common
> Linux scheduler. This is what the log[1] show.
>
> 4. the common/in-band kernel code stops @user due to the ptrace stop
> condition caused by the breakpoint, waiting for a continuation event to
> happen.
>
> Therefore, upon PTRACE_CONT (i.e. gdb continue), we need to force @user
> to call back into kernel context (handle_ptrace_cont ->
> dovetail_request_ucall), then ask for a switch to primary mode from
> there, which should eventually happen when @user is about to leave the
> kernel (on x86, this now happens from a generic kernel entry/exit code
> in kernel/entry/*). As a result, handle_taskexit_event() runs, figures
Not quite:
s,handle_taskexit_event,handle_user_return,
--
Philippe.
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: gdb test failure debug status update
2021-04-28 14:30 ` Philippe Gerum
@ 2021-04-29 6:31 ` Chen, Hongzhan
2021-04-30 5:25 ` Chen, Hongzhan
0 siblings, 1 reply; 16+ messages in thread
From: Chen, Hongzhan @ 2021-04-29 6:31 UTC (permalink / raw)
To: Philippe Gerum; +Cc: xenomai
>
>-----Original Message-----
>From: Philippe Gerum <rpm@xenomai.org>
>Sent: Wednesday, April 28, 2021 10:30 PM
>To: Philippe Gerum <rpm@xenomai.org>
>Cc: Chen, Hongzhan <hongzhan.chen@intel.com>; xenomai@xenomai.org
>Subject: Re: gdb test failure debug status update
>
>
>Philippe Gerum via Xenomai <xenomai@xenomai.org> writes:
>
>> Chen, Hongzhan via Xenomai <xenomai@xenomai.org> writes:
>>
>>> According to my validation, gdb test fail on dovetail 5.10 branch but pass on v5.9-evl4 tag with same for-upstream/dovetail
>>> xenomai code base.
>>>
>>> After further debug , the issue is more clear for me. Gdb test failure because low priority thread smokey userspace is still
>>> executed after "cobalt_shadow_relaxed: state=0x4488c0 info=0x200" like log [1] on dovetail-5.10 branch.
>>> The weird thing is that its following first ftrace log happen at 62235.848583 after cobalt_shadow_relaxed in log [1].
>>> It is almost 3ms happened after cobalt_shadow_relaxed. The low priority smoke thread user space is executed during this
>>> 3ms period so that test fail.
>>>
>>> But in success case with v5.9-evl4 like in log [2], the time interval between cobalt_shadow_relaxed and the following first ftrace log
>>> is only about 1us. It seems that low priority smokey userspace do not have chance to execute in this 1us because gdb test is successful.
>>>
>>> My question is why there is even no interrupt happened during that about 3ms period in failure case? Tick seems in abnormal behavior.
>>> Please comment if you have any ideas to further debug it.
>>>
>>> PS: All my tests run on same up Xtream board.
>>
>> <snip>
>>
>> Let's put aside the tick issue for now, there may be a valid reason for
>> this delay with dynticks enabled.
>>
>> The issue at stake may be related to the way a return to kernel space is
>> forced on a @user task (Dovetail has an integrated service for
>> triggering this called dovetail_request_ucall()).
>>
>> The logic for doing so is as follows:
>>
>> 1. @user hits a breakpoint, which is an exception Dovetail-wise
>>
>> 2. @user gets XNDBGSTOP set into its flags because Cobalt notices it is
>> being debugged via a breakpoint trap, then relaxed as a result of taking
>> a exception in general, so that we may traverse the common trap handling code
>> safely.
>>
>> 3. since XNDBGSTOP is a blocking bit Cobalt-wise, it should prevent
>> @user from being picked for scheduling by the real-time core, next time
>> a Cobalt considers rescheduling that is. However, since @user is
>> currently relaxed, it can still run under the supervision of the common
>> Linux scheduler. This is what the log[1] show.
>>
>> 4. the common/in-band kernel code stops @user due to the ptrace stop
>> condition caused by the breakpoint, waiting for a continuation event to
>> happen.
>>
>> Therefore, upon PTRACE_CONT (i.e. gdb continue), we need to force @user
>> to call back into kernel context (handle_ptrace_cont ->
>> dovetail_request_ucall), then ask for a switch to primary mode from
>> there, which should eventually happen when @user is about to leave the
>> kernel (on x86, this now happens from a generic kernel entry/exit code
>> in kernel/entry/*). As a result, handle_taskexit_event() runs, figures
>
>Not quite:
>s,handle_taskexit_event,handle_user_return,
There is trace_cobalt_shadow_unmap in handle_taskexit_event. I added
trace_cobalt_handle_user_return and trace_cobalt_handle_ptrace_cont
like patch [3] in log [4].
1. In the log [4], the first cobalt_handle_ptrace_cont is followed by cobalt_handle_user_return
that clear info flag XNCONTHI for smokey-2499 because cobalt_handle_ptrace_cont called dovetail_request_ucall .
the TIF_RETUSER bit was cleared for smokey-2499.
After then 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000
clear state flag XNDBGSTOP. After that, thread smokey-2499 run without these bit set till
"hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000".
2. There is second cobalt_handle_ptrace_cont called for smokey-2499 but at that time dovetail_request_ucall
would not be called because XNSSTEP Is not set like "smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0"
3. hi-thread-2504 breakpoint at its first breakpoint_target so it trigger hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3
It call dovetail_request_ucall(current); for itself and also call xnthread_set_info(thread, XNCONTHI); That is why we can see this bit set in
hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200
4. finally ,gdb call sys_tkill in gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13) and trigger that handle_oob_mayday is called so that xnthread_relax for
smokey-2499 Is called in "smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined."
It seems that there never call dovetail_request_ucall for process smokey-2499 in log[4] so that final xhthread_relax would have chance to return to user?
The second handle_ptrace_cont happen too much earlier because XNSSTEP still have not been set so that system never have change to call dovetail_request_ucall
for smokey-2499.
In this case , do we need to do hack in 3 , when handle_oob_trap_entry is called for hi-thread-2504 we call xnthread_set_info(thread, XNCONTHI);
and dovetail_request_ucall for all threads that have same cobalt_process? Is this solution OK for fixing this case?
[3]:
index 07dcc7a63..cd76961f5 100644
--- a/kernel/cobalt/dovetail/kevents.c
+++ b/kernel/cobalt/dovetail/kevents.c
@@ -331,6 +331,8 @@ static int handle_user_return(struct task_struct *task)
if (thread == NULL)
return KEVENT_PROPAGATE;
+ trace_cobalt_handle_user_return(thread);
+
if (xnthread_test_info(thread, XNCONTHI)) {
xnlock_get_irqsave(&nklock, s);
xnthread_clear_info(thread, XNCONTHI);
@@ -468,6 +470,8 @@ int handle_ptrace_resume(struct task_struct *tracee)
if (thread == NULL)
return KEVENT_PROPAGATE;
+ trace_cobalt_handle_ptrace_resume(thread);
+
if (xnthread_test_state(thread, XNSSTEP)) {
xnlock_get_irqsave(&nklock, s);
@@ -485,6 +489,8 @@ static void handle_ptrace_cont(void)
struct xnthread *curr = xnthread_current();
spl_t s;
+ trace_cobalt_handle_ptrace_cont(curr);
+
xnlock_get_irqsave(&nklock, s);
if (xnthread_test_state(curr, XNSSTEP)) {
@@ -604,6 +610,7 @@ void resume_oob_task(struct task_struct *p) /* inband, oob stage stalled */
if (unlikely(xnthread_test_info(thread, XNCONTHI))) {
xnthread_clear_info(thread, XNCONTHI);
dovetail_clear_ucall();
+ trace_cobalt_resume_oob_task(thread);
}
/* Unregister as debugged thread in case we postponed this. */
diff --git a/kernel/cobalt/trace/cobalt-core.h b/kernel/cobalt/trace/cobalt-core.h
index 78f66a106..88a4067fd 100644
--- a/kernel/cobalt/trace/cobalt-core.h
+++ b/kernel/cobalt/trace/cobalt-core.h
@@ -459,6 +459,27 @@ DEFINE_EVENT(curr_thread_event, cobalt_thread_set_mode,
TP_ARGS(thread)
);
+DEFINE_EVENT(thread_event, cobalt_handle_ptrace_cont,
+ TP_PROTO(struct xnthread *thread),
[4]:
gdb-2495 [000] d..3 61513.834825: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] ...1 61513.834827: cobalt_handle_ptrace_cont: pid=2499 state=0x4488c0 info=0x200
smokey-2499 [000] d..1 61513.834828: cobalt_handle_user_return: pid=2499 state=0x4488c0 info=0x200
smokey-2499 [000] d..1 61513.834828: cobalt_shadow_gohard: state=0x4488c0 info=0x0
smokey-2499 [000] d..3 61513.834830: sched_stat_wait: comm=gdb pid=2495 delay=8017 [ns]
smokey-2499 [000] d..3 61513.834831: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] *..3 61513.834831: cobalt_thread_resume: name=smokey pid=2499 mask=0x80
gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1
gdb-2495 [000] *..3 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000
gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1
gdb-2495 [000] *.~3 61513.834833: cobalt_schedule: status=0x10000000
gdb-2495 [000] *.~3 61513.834833: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] *.~3 61513.834833: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1
smokey-2499 [000] d.~3 61513.834835: cobalt_shadow_hardened: state=0x48040 info=0x0
smokey-2499 [000] d.~3 61513.834836: cobalt_head_sysentry: syscall=sem_init
smokey-2499 [000] d.~3 61513.834839: cobalt_psem_init: sem=0x4e(anon) flags=0x0() value=0
smokey-2499 [000] d.~3 61513.834839: cobalt_head_sysexit: result=0
smokey-2499 [000] d.~3 61513.834848: cobalt_shadow_gorelax: reason=syscall
smokey-2499 [000] d.~3 61513.834848: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey
smokey-2499 [000] *.~3 61513.834848: cobalt_thread_suspend: pid=2499 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
smokey-2499 [000] *.~3 61513.834849: cobalt_schedule: status=0x10000000
smokey-2499 [000] *.~3 61513.834849: cobalt_trace_pid: pid=2499, prio=1
smokey-2499 [000] *.~3 61513.834849: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
gdb-2495 [000] *.~3 61513.834850: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] d.h3 61513.834851: irq_handler_entry: irq=120 name=in-band work
gdb-2495 [000] d.h3 61513.834852: cobalt_lostage_wakeup: pid=2499 comm=smokey
gdb-2495 [000] d.h5 61513.834852: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dNh6 61513.834853: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dNh3 61513.834853: irq_handler_exit: irq=120 ret=handled
gdb-2495 [000] dN.3 61513.834854: sched_stat_runtime: comm=gdb pid=2495 runtime=23565 [ns] vruntime=300514015543 [ns]
gdb-2495 [000] d..3 61513.834855: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] ...1 61513.834856: cobalt_shadow_relaxed: state=0x480c0 info=0x0
smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min(policy: 1)
smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min -> 0x1
smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max(policy: 1)
smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max -> 0x63
smokey-2499 [000] ..~1 61513.834862: cobalt_head_sysentry: syscall=thread_getschedparam_ex
smokey-2499 [000] ...1 61513.834862: cobalt_root_sysentry: syscall=thread_getschedparam_ex
smokey-2499 [000] ...1 61513.834863: cobalt_pthread_getschedparam: pth=000000008d36be1c policy=fifo param={ priority=1 }
smokey-2499 [000] ...1 61513.834864: cobalt_root_sysexit: result=0
smokey-2499 [000] ...1 61513.834872: sys_mmap(addr: 0, len: 11000, prot: 0, flags: 20022, fd: ffffffff, off: 0)
smokey-2499 [000] ...1 61513.834877: sys_mmap -> 0x7ffff67b9000
smokey-2499 [000] ...1 61513.834878: sys_mprotect(start: 7ffff67ba000, len: 10000, prot: 3)
smokey-2499 [000] ...1 61513.834906: sys_mprotect -> 0x0
smokey-2499 [000] ...1 61513.834909: sys_clone(clone_flags: 3d0f00, newsp: 7ffff67c8fb0, parent_tidptr: 7ffff67c99d0, child_tidptr: 7ffff67c99d0, tls: 7ffff67c9700)
smokey-2499 [000] ...1 61513.834923: task_newtask: pid=2504 comm=smokey clone_flags=3d0f00 oom_score_adj=0
smokey-2499 [000] ...1 61513.834923: sched_process_fork: comm=smokey pid=2499 child_comm=smokey child_pid=2504
smokey-2499 [000] d..3 61513.834925: sched_wakeup_new: comm=smokey pid=2504 prio=98 target_cpu=000
smokey-2499 [000] d..3 61513.834928: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0
smokey-2499 [000] d..3 61513.834931: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=t ==> next_comm=smokey next_pid=2504 next_prio=98
hi-thread-2504 [000] ...1 61513.834932: sys_clone -> 0x0
hi-thread-2504 [000] d..3 61513.834934: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=2
hi-thread-2504 [000] d..3 61513.834935: sched_stat_wait: comm=gdb pid=2495 delay=82002 [ns]
hi-thread-2504 [000] d..3 61513.834936: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=t ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] d..2 61513.834937: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000
gdb-2495 [000] ...1 61513.834940: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1)
gdb-2495 [000] ...1 61513.834943: sys_read -> 0x1
gdb-2495 [000] ...1 61513.834943: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1)
gdb-2495 [000] ...1 61513.834944: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.834944: sys_write(fd: 10, buf: 5585fe0354b1, count: 1)
gdb-2495 [000] ...1 61513.834946: sys_write -> 0x1
gdb-2495 [000] ...1 61513.834947: sys_rt_sigreturn()
gdb-2495 [000] ...1 61513.834953: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1)
gdb-2495 [000] ...1 61513.834953: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.834955: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0)
gdb-2495 [000] ...1 61513.834956: sys_poll -> 0x1
gdb-2495 [000] ...1 61513.834957: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
gdb-2495 [000] ...1 61513.834958: sys_read -> 0x1
gdb-2495 [000] ...1 61513.834959: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
gdb-2495 [000] ...1 61513.834960: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8)
gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask -> 0x0
gdb-2495 [000] ...1 61513.834962: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
gdb-2495 [000] ...1 61513.834962: sched_process_wait: comm=gdb pid=0 prio=120
gdb-2495 [000] ...1 61513.834964: sys_wait4 -> 0x9c3
gdb-2495 [000] ...1 61513.834965: sys_ptrace(request: 4201, pid: 9c3, addr: 0, data: 7ffe8b2a7d08)
gdb-2495 [000] d..3 61513.834966: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.834967: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.834967: sys_wait4(upid: 9c8, stat_addr: 7ffe8b2a7cfc, options: 40000000, ru: 0)
gdb-2495 [000] ...1 61513.834968: sched_process_wait: comm=gdb pid=2504 prio=120
gdb-2495 [000] ...1 61513.834969: sys_wait4 -> 0x9c8
gdb-2495 [000] ...1 61513.834972: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578)
gdb-2495 [000] d..3 61513.834972: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.834975: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.834976: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a78d8)
gdb-2495 [000] d..3 61513.834977: sched_wait_task: comm=smokey pid=2504 prio=98
gdb-2495 [000] ...1 61513.834977: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.834979: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.834989: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.834990: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff67c9700)
gdb-2495 [000] ...1 61513.834992: sys_pread64 -> 0x900
gdb-2495 [000] ...1 61513.834993: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.834994: sys_close -> 0x0
gdb-2495 [000] ...1 61513.835010: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578)
gdb-2495 [000] d..3 61513.835011: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.835012: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835013: sys_ptrace(request: 3, pid: 9c3, addr: a8, data: 7ffe8b2a78d8)
gdb-2495 [000] d..3 61513.835013: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.835014: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835015: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.835022: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.835022: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff758f740)
gdb-2495 [000] ...1 61513.835025: sys_pread64 -> 0x900
gdb-2495 [000] ...1 61513.835025: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.835026: sys_close -> 0x0
gdb-2495 [000] ...1 61513.835031: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
gdb-2495 [000] ...1 61513.835032: sched_process_wait: comm=gdb pid=0 prio=120
gdb-2495 [000] ...1 61513.835033: sys_wait4 -> 0x0
gdb-2495 [000] ...1 61513.835036: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0)
gdb-2495 [000] d..3 61513.835037: sched_wait_task: comm=smokey pid=2504 prio=98
gdb-2495 [000] ...1 61513.835039: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835043: sys_ptrace(request: 6, pid: 9c8, addr: 388, data: 0)
gdb-2495 [000] d..3 61513.835044: sched_wait_task: comm=smokey pid=2504 prio=98
gdb-2495 [000] ...1 61513.835044: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835045: sys_ptrace(request: 7, pid: 9c8, addr: 1, data: 0)
gdb-2495 [000] d..3 61513.835046: sched_wait_task: comm=smokey pid=2504 prio=98
gdb-2495 [000] d..3 61513.835047: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000
gdb-2495 [000] dN.4 61513.835048: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000
gdb-2495 [000] .N.1 61513.835048: sys_ptrace -> 0x0
gdb-2495 [000] dN.3 61513.835050: sched_stat_runtime: comm=gdb pid=2495 runtime=112561 [ns] vruntime=300514128104 [ns]
gdb-2495 [000] d..3 61513.835051: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=98
hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list(head: 7ffff67c99e0, len: 18)
hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list -> 0x0
hi-thread-2504 [000] ...1 61513.835056: sys_futex(uaddr: 7ffff67c9d18, op: 80, val: 2, utime: 0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700)
hi-thread-2504 [000] d..3 61513.835059: sched_stat_wait: comm=gdb pid=2495 delay=10883 [ns]
hi-thread-2504 [000] d..3 61513.835060: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] ...1 61513.835063: sys_ptrace(request: c, pid: 9c3, addr: 0, data: 7ffe8b2a6ef0)
gdb-2495 [000] d..3 61513.835064: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.835065: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835068: sys_ptrace(request: 7, pid: 9c3, addr: 1, data: 0)
gdb-2495 [000] d..3 61513.835069: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] d..3 61513.835069: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dN.4 61513.835070: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] .N.1 61513.835071: sys_ptrace -> 0x0
gdb-2495 [000] dN.3 61513.835071: sched_stat_runtime: comm=gdb pid=2495 runtime=12210 [ns] vruntime=300514140314 [ns]
gdb-2495 [000] d..3 61513.835072: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0
smokey-2499 [000] ...1 61513.835074: sys_clone -> 0x9c8
smokey-2499 [000] ...1 61513.835075: sys_sched_setaffinity(pid: 9c8, len: 80, user_mask_ptr: 55555563e7c0)
smokey-2499 [000] ...1 61513.835077: sys_sched_setaffinity -> 0x0
smokey-2499 [000] ...1 61513.835077: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 0, val3: 7ffff67c9700)
smokey-2499 [000] d..3 61513.835078: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000
smokey-2499 [000] d..4 61513.835079: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000
smokey-2499 [000] ...1 61513.835080: sys_futex -> 0x1
smokey-2499 [000] ...1 61513.835086: sys_futex(uaddr: 7fffffffe828, op: 189, val: 0, utime: 7fffffffe780, uaddr2: 0, val3: ffffffff)
smokey-2499 [000] d..3 61513.835089: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=smokey next_pid=2504 next_prio=98
hi-thread-2504 [000] ...1 61513.835090: sys_futex -> 0x0
hi-thread-2504 [000] ...1 61513.835090: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700)
hi-thread-2504 [000] ...1 61513.835091: sys_futex -> 0x0
hi-thread-2504 [000] ...1 61513.835092: sys_sched_setscheduler(pid: 9c8, policy: 1, param: 7ffff67c8e70)
hi-thread-2504 [000] ...1 61513.835095: sys_sched_setscheduler -> 0x0
hi-thread-2504 [000] ..~1 61513.835096: cobalt_head_sysentry: syscall=thread_create
hi-thread-2504 [000] ...1 61513.835097: cobalt_root_sysentry: syscall=thread_create
hi-thread-2504 [000] ...1 61513.835098: cobalt_pthread_create: pth=00000000ba35a01d policy=fifo param={ priority=2 }
hi-thread-2504 [000] ...1 61513.835099: cobalt_timer_migrate: timer=000000004d21df54 cpu=0
hi-thread-2504 [000] ...1 61513.835100: cobalt_timer_migrate: timer=00000000c1301853 cpu=0
hi-thread-2504 [000] ...1 61513.835101: cobalt_thread_set_current_prio: thread=00000000e9fda3c4 pid=-1 prio=2
hi-thread-2504 [000] ...1 61513.835102: cobalt_thread_init: thread=00000000e9fda3c4 name=smokey flags=0x48000 class=rt prio=2
hi-thread-2504 [000] ...1 61513.835104: cobalt_shadow_map: thread=00000000e9fda3c4 pid=2504 prio=2
hi-thread-2504 [000] D..1 61513.835104: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_start: pid=2504 state=0x480d0 info=0x0
hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_resume: name=smokey pid=2504 mask=0x10
hi-thread-2504 [000] D..1 61513.835106: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] ...1 61513.835106: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] ...1 61513.835107: cobalt_shadow_gohard: state=0x480c0 info=0x0
hi-thread-2504 [000] d..3 61513.835108: sched_stat_wait: comm=gdb pid=2495 delay=37231 [ns]
hi-thread-2504 [000] d..3 61513.835109: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] *..3 61513.835109: cobalt_thread_resume: name=smokey pid=2504 mask=0x80
gdb-2495 [000] *..3 61513.835110: cobalt_trace_pid: pid=2504, prio=2
gdb-2495 [000] *.~3 61513.835110: cobalt_schedule: status=0x10000000
gdb-2495 [000] *.~3 61513.835110: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] *.~3 61513.835111: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2
hi-thread-2504 [000] d.~3 61513.835112: cobalt_shadow_hardened: state=0x48040 info=0x0
hi-thread-2504 [000] d.~3 61513.835112: cobalt_root_sysexit: result=0
hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysentry: syscall=get_current
hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysexit: result=0
hi-thread-2504 [000] d.~3 61513.835116: cobalt_shadow_gorelax: reason=syscall
hi-thread-2504 [000] d.~3 61513.835117: cobalt_lostage_request: request=wakeup pid=2504 comm=smokey
hi-thread-2504 [000] *.~3 61513.835117: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] *.~3 61513.835117: cobalt_schedule: status=0x10000000
hi-thread-2504 [000] *.~3 61513.835118: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] *.~3 61513.835118: cobalt_switch_context: prev_name=smokey prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
gdb-2495 [000] *.~3 61513.835119: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] d.h3 61513.835120: irq_handler_entry: irq=120 name=in-band work
gdb-2495 [000] d.h3 61513.835120: cobalt_lostage_wakeup: pid=2504 comm=smokey
gdb-2495 [000] d.h5 61513.835120: sched_waking: comm=smokey pid=2504 prio=97 target_cpu=000
gdb-2495 [000] dNh6 61513.835121: sched_wakeup: comm=smokey pid=2504 prio=97 target_cpu=000
gdb-2495 [000] dNh3 61513.835122: irq_handler_exit: irq=120 ret=handled
gdb-2495 [000] dN.3 61513.835122: sched_stat_runtime: comm=gdb pid=2495 runtime=13441 [ns] vruntime=300514153755 [ns]
gdb-2495 [000] d..3 61513.835123: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=97
hi-thread-2504 [000] ...1 61513.835124: cobalt_shadow_relaxed: state=0x480c0 info=0x0
hi-thread-2504 [000] ...1 61513.835125: sys_futex(uaddr: 7fffffffe828, op: 81, val: 1, utime: 0, uaddr2: 7ffff67c8e74, val3: 7fffffffe828)
hi-thread-2504 [000] d..3 61513.835126: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
hi-thread-2504 [000] d..4 61513.835126: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
hi-thread-2504 [000] ...1 61513.835127: sys_futex -> 0x1
hi-thread-2504 [000] ..~1 61513.835128: cobalt_head_sysentry: syscall=migrate
hi-thread-2504 [000] ...1 61513.835128: cobalt_root_sysentry: syscall=migrate
hi-thread-2504 [000] ...1 61513.835128: cobalt_shadow_gohard: state=0x480c0 info=0x0
hi-thread-2504 [000] d..3 61513.835130: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] *..3 61513.835130: cobalt_thread_resume: name=smokey pid=2504 mask=0x80
smokey-2499 [000] *..3 61513.835130: cobalt_trace_pid: pid=2504, prio=2
smokey-2499 [000] *.~3 61513.835131: cobalt_schedule: status=0x10000000
smokey-2499 [000] *.~3 61513.835131: cobalt_trace_pid: pid=2499, prio=-1
smokey-2499 [000] *.~3 61513.835131: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2
hi-thread-2504 [000] d.~3 61513.835132: cobalt_shadow_hardened: state=0x48040 info=0x0
hi-thread-2504 [000] d.~3 61513.835132: cobalt_root_sysexit: result=1
hi-thread-2504 [000] d.~3 61513.835133: cobalt_head_sysentry: syscall=thread_setname
hi-thread-2504 [000] d.~3 61513.835134: cobalt_pthread_setname: pth=00000000ba35a01d name=hi-thread
hi-thread-2504 [000] d.~3 61513.835135: cobalt_head_sysexit: result=0
hi-thread-2504 [000] d.~3 61513.835136: cobalt_head_sysentry: syscall=clock_nanosleep
hi-thread-2504 [000] d.~3 61513.835137: cobalt_clock_nanosleep: clock_id=0 flags=0x0() rqt=(0.001000000)
hi-thread-2504 [000] *.~3 61513.835138: cobalt_thread_suspend: pid=2504 mask=0x4 timeout=1000001 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] *.~3 61513.835138: cobalt_timer_start: timer=000000004d21df54(smokey) value=1000001 interval=0 mode=rel
hi-thread-2504 [000] *.~3 61513.835138: cobalt_schedule: status=0x10000000
hi-thread-2504 [000] *.~3 61513.835139: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] *.~3 61513.835139: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x48044 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
smokey-2499 [000] *.~3 61513.835140: cobalt_trace_pid: pid=2499, prio=-1
smokey-2499 [000] ...1 61513.835141: sys_futex -> 0x0
smokey-2499 [000] ..~1 61513.835144: cobalt_head_sysentry: syscall=migrate
smokey-2499 [000] ...1 61513.835144: cobalt_root_sysentry: syscall=migrate
smokey-2499 [000] ...1 61513.835144: cobalt_shadow_gohard: state=0x480c0 info=0x0
smokey-2499 [000] d..3 61513.835145: sched_stat_wait: comm=gdb pid=2495 delay=23982 [ns]
smokey-2499 [000] d..3 61513.835146: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] *..3 61513.835147: cobalt_thread_resume: name=smokey pid=2499 mask=0x80
gdb-2495 [000] *..3 61513.835147: cobalt_trace_pid: pid=2499, prio=1
gdb-2495 [000] *.~3 61513.835147: cobalt_schedule: status=0x10000000
gdb-2495 [000] *.~3 61513.835147: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] *.~3 61513.835148: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1
smokey-2499 [000] d.~3 61513.835148: cobalt_shadow_hardened: state=0x48040 info=0x0
smokey-2499 [000] d.~3 61513.835149: cobalt_root_sysexit: result=1
smokey-2499 [000] *.~3 61513.835219: irq_pipeline_entry: irq=4354
smokey-2499 [000] *.~3 61513.835219: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
smokey-2499 [000] *.~3 61513.835220: cobalt_timer_expire: timer=00000000d13b8aa9
smokey-2499 [000] *.~3 61513.835220: cobalt_tick_shot: next tick at 61513.836136 (delay: 915 us)
smokey-2499 [000] *.~3 61513.835221: irq_handler_exit: irq=4354 ret=handled
smokey-2499 [000] *.~3 61513.835221: irq_pipeline_exit: irq=4354
smokey-2499 [000] *.~3 61513.836137: irq_pipeline_entry: irq=4354
smokey-2499 [000] *.~3 61513.836137: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
smokey-2499 [000] *.~3 61513.836138: cobalt_timer_expire: timer=000000004d21df54
smokey-2499 [000] *.~3 61513.836138: cobalt_thread_resume: name=hi-thread pid=2504 mask=0x4
smokey-2499 [000] *.~3 61513.836138: cobalt_trace_pid: pid=2504, prio=2
smokey-2499 [000] *.~3 61513.836139: cobalt_timer_stop: timer=000000004d21df54
smokey-2499 [000] *.~3 61513.836139: irq_handler_exit: irq=4354 ret=handled
smokey-2499 [000] *.~3 61513.836139: irq_pipeline_exit: irq=4354
smokey-2499 [000] *.~3 61513.836140: cobalt_schedule: status=0x10000000
smokey-2499 [000] *.~3 61513.836140: cobalt_trace_pid: pid=2499, prio=1
smokey-2499 [000] *.~3 61513.836140: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x48048 ==> next_name=hi-thread next_pid=2504 next_prio=2
hi-thread-2504 [000] *.~3 61513.836141: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] d.~3 61513.836141: cobalt_head_sysexit: result=0
hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3
hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] d.~3 61513.836143: cobalt_shadow_gorelax: reason=fault
hi-thread-2504 [000] d.~3 61513.836143: cobalt_lostage_request: request=wakeup pid=2504 comm=hi-thread
hi-thread-2504 [000] *.~3 61513.836144: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] *.~3 61513.836144: cobalt_schedule: status=0x10000000
hi-thread-2504 [000] *.~3 61513.836144: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] *.~3 61513.836145: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
gdb-2495 [000] *.~3 61513.836146: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] d.h3 61513.836146: irq_handler_entry: irq=120 name=in-band work
gdb-2495 [000] d.h3 61513.836147: cobalt_lostage_wakeup: pid=2504 comm=hi-thread
gdb-2495 [000] d.h5 61513.836147: sched_waking: comm=hi-thread pid=2504 prio=97 target_cpu=000
gdb-2495 [000] dNh6 61513.836147: sched_wakeup: comm=hi-thread pid=2504 prio=97 target_cpu=000
gdb-2495 [000] dNh3 61513.836148: irq_handler_exit: irq=120 ret=handled
gdb-2495 [000] dNh3 61513.836148: irq_handler_entry: irq=124 name=proxy tick
gdb-2495 [000] dNh3 61513.836151: softirq_raise: vec=1 [action=TIMER]
gdb-2495 [000] dNh3 61513.836152: softirq_raise: vec=9 [action=RCU]
gdb-2495 [000] dNh4 61513.836152: sched_stat_runtime: comm=gdb pid=2495 runtime=1007474 [ns] vruntime=300515161229 [ns]
gdb-2495 [000] *Nh3 61513.836154: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2395843 interval=0 mode=rel
gdb-2495 [000] *Nh3 61513.836155: cobalt_tick_shot: next tick at 61513.838550 (delay: 2395 us)
gdb-2495 [000] dNh3 61513.836155: irq_handler_exit: irq=124 ret=handled
gdb-2495 [000] .Ns3 61513.836156: softirq_entry: vec=1 [action=TIMER]
gdb-2495 [000] .Ns3 61513.836157: softirq_exit: vec=1 [action=TIMER]
gdb-2495 [000] .Ns3 61513.836157: softirq_entry: vec=9 [action=RCU]
gdb-2495 [000] dNs6 61513.836158: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000
gdb-2495 [000] dNs7 61513.836159: sched_stat_runtime: comm=gdb pid=2495 runtime=6881 [ns] vruntime=300515168110 [ns]
gdb-2495 [000] dNs7 61513.836160: sched_stat_blocked: comm=rcu_sched pid=11 delay=4188739 [ns]
gdb-2495 [000] dNs7 61513.836161: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000
gdb-2495 [000] .Ns3 61513.836164: softirq_exit: vec=9 [action=RCU]
gdb-2495 [000] d..3 61513.836165: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=hi-thread next_pid=2504 next_prio=97
hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200
hi-thread-2504 [000] *..2 61513.836169: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] *..2 61513.836170: cobalt_thread_suspend: pid=2504 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] d..2 61513.836170: signal_generate: sig=5 errno=0 code=128 comm=hi-thread pid=2504 grp=0 res=0
hi-thread-2504 [000] d..3 61513.836173: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0
hi-thread-2504 [000] d..3 61513.836175: sched_stat_wait: comm=rcu_sched pid=11 delay=14490 [ns]
hi-thread-2504 [000] d..3 61513.836175: sched_switch: prev_comm=hi-thread prev_pid=2504 prev_prio=97 prev_state=t ==> next_comm=rcu_sched next_pid=11 next_prio=120
rcu_sched-11 [000] d..3 61513.836179: sched_stat_runtime: comm=rcu_sched pid=11 runtime=4960 [ns] vruntime=300512173070 [ns]
rcu_sched-11 [000] d..3 61513.836179: sched_stat_wait: comm=gdb pid=2495 delay=19450 [ns]
rcu_sched-11 [000] d..3 61513.836180: sched_switch: prev_comm=rcu_sched prev_pid=11 prev_prio=120 prev_state=I ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] ...1 61513.836187: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cb0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836197: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836198: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7ae0)
gdb-2495 [000] ...1 61513.836199: sys_newfstat -> 0x0
gdb-2495 [000] ...1 61513.836200: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
gdb-2495 [000] ...1 61513.836214: sys_read -> 0x3f1
gdb-2495 [000] ...1 61513.836217: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836217: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836221: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8)
gdb-2495 [000] ...1 61513.836222: sys_rt_sigprocmask -> 0x0
gdb-2495 [000] d..2 61513.836223: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000
gdb-2495 [000] ...1 61513.836225: sys_read(fd: f, buf: 7ffe8b2a7667, count: 1)
gdb-2495 [000] ...1 61513.836226: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.836227: sys_write(fd: 10, buf: 5585fe0354b1, count: 1)
gdb-2495 [000] ...1 61513.836229: sys_write -> 0x1
gdb-2495 [000] ...1 61513.836229: sys_rt_sigreturn()
gdb-2495 [000] ...1 61513.836233: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1)
gdb-2495 [000] ...1 61513.836234: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.836235: sys_write(fd: 1, buf: 5585fe7a5290, count: 27)
gdb-2495 [000] d..4 61513.836236: sched_waking: comm=smokey pid=2493 prio=120 target_cpu=000
gdb-2495 [000] d..5 61513.836237: sched_stat_runtime: comm=gdb pid=2495 runtime=58399 [ns] vruntime=300515226509 [ns]
gdb-2495 [000] d..5 61513.836238: sched_stat_sleep: comm=smokey pid=2493 delay=1717179 [ns]
gdb-2495 [000] dN.5 61513.836239: sched_wakeup: comm=smokey pid=2493 prio=120 target_cpu=000
gdb-2495 [000] .N.1 61513.836240: sys_write -> 0x27
gdb-2495 [000] dN.3 61513.836241: sched_stat_wait: comm=smokey pid=2493 delay=0 [ns]
gdb-2495 [000] d..3 61513.836241: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2493 next_prio=120
smokey-2493 [000] ...1 61513.836244: sys_read -> 0x1
smokey-2493 [000] ..~1 61513.836246: cobalt_head_sysentry: syscall=read
...
smokey-2493 [000] d..3 61513.836344: sched_stat_runtime: comm=smokey pid=2493 runtime=106636 [ns] vruntime=300512333145 [ns]
smokey-2493 [000] d..3 61513.836344: sched_stat_wait: comm=gdb pid=2495 delay=106636 [ns]
smokey-2493 [000] d..3 61513.836345: sched_switch: prev_comm=smokey prev_pid=2493 prev_prio=120 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] ...1 61513.836347: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0)
gdb-2495 [000] ...1 61513.836348: sys_poll -> 0x1
gdb-2495 [000] ...1 61513.836350: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
gdb-2495 [000] ...1 61513.836351: sys_read -> 0x1
gdb-2495 [000] ...1 61513.836352: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
gdb-2495 [000] ...1 61513.836352: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.836353: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8)
gdb-2495 [000] ...1 61513.836354: sys_rt_sigprocmask -> 0x0
gdb-2495 [000] ...1 61513.836355: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
gdb-2495 [000] ...1 61513.836355: sched_process_wait: comm=gdb pid=0 prio=120
gdb-2495 [000] ...1 61513.836357: sys_wait4 -> 0x9c8
gdb-2495 [000] ...1 61513.836360: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0)
gdb-2495 [000] d..3 61513.836361: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836363: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836367: sys_ptrace(request: 4202, pid: 9c8, addr: 0, data: 7ffe8b2a7cb0)
gdb-2495 [000] d..3 61513.836368: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836369: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836370: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6f10)
gdb-2495 [000] d..3 61513.836370: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836371: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836372: sys_ptrace(request: d, pid: 9c8, addr: 0, data: 7ffe8b2a6f10)
gdb-2495 [000] d..3 61513.836373: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836374: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836376: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
gdb-2495 [000] ...1 61513.836376: sched_process_wait: comm=gdb pid=0 prio=120
gdb-2495 [000] ...1 61513.836377: sys_wait4 -> 0x0
gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8)
gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask -> 0x0
gdb-2495 [000] ...1 61513.836382: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cd0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836399: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836400: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7b50)
gdb-2495 [000] ...1 61513.836401: sys_newfstat -> 0x0
gdb-2495 [000] ...1 61513.836402: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
gdb-2495 [000] ...1 61513.836409: sys_read -> 0x140
gdb-2495 [000] ...1 61513.836410: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400)
gdb-2495 [000] ...1 61513.836410: sys_read -> 0x0
gdb-2495 [000] ...1 61513.836415: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836416: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836419: sys_read(fd: f, buf: 7ffe8b2a7f17, count: 1)
gdb-2495 [000] ...1 61513.836420: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.836421: sys_write(fd: 10, buf: 5585fe0354b1, count: 1)
gdb-2495 [000] ...1 61513.836422: sys_write -> 0x1
gdb-2495 [000] ...1 61513.836425: sys_ptrace(request: 1, pid: 9c8, addr: 7ffff777f370, data: 7ffe8b2a7878)
gdb-2495 [000] d..3 61513.836426: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836428: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836430: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a7bd8)
gdb-2495 [000] d..3 61513.836430: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836431: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836433: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a70e0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836446: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836446: sys_pread64(fd: 12, buf: 7ffe8b2a7430, count: 900, pos: 7ffff67c9700)
gdb-2495 [000] ...1 61513.836449: sys_pread64 -> 0x900
gdb-2495 [000] ...1 61513.836450: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836450: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836479: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836488: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836489: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950)
gdb-2495 [000] ...1 61513.836490: sys_newfstat -> 0x0
gdb-2495 [000] ...1 61513.836490: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400)
gdb-2495 [000] ...1 61513.836498: sys_read -> 0x14d
gdb-2495 [000] ...1 61513.836499: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
gdb-2495 [000] ...1 61513.836499: sys_read -> 0x0
gdb-2495 [000] ...1 61513.836503: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836504: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836507: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836514: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836515: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950)
gdb-2495 [000] ...1 61513.836515: sys_newfstat -> 0x0
gdb-2495 [000] ...1 61513.836516: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
gdb-2495 [000] ...1 61513.836521: sys_read -> 0x13b
gdb-2495 [000] ...1 61513.836522: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400)
gdb-2495 [000] ...1 61513.836522: sys_read -> 0x0
gdb-2495 [000] ...1 61513.836525: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836525: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13)
gdb-2495 [000] *..3 61513.836532: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000
gdb-2495 [000] *..3 61513.836532: cobalt_trace_pid: pid=2499, prio=1
gdb-2495 [000] *..3 61513.836533: cobalt_thread_unblock: pid=2499 state=0x48848 info=0x0
gdb-2495 [000] *.~3 61513.836586: cobalt_schedule: status=0x10000000
gdb-2495 [000] *.~3 61513.836586: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] *.~3 61513.836587: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1
smokey-2499 [000] *.~3 61513.836589: cobalt_trace_pid: pid=2499, prio=1
smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined
smokey-2499 [000] *.~3 61513.836589: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey
smokey-2499 [000] *.~3 61513.836590: cobalt_thread_suspend: pid=2499 mask=0x400080 timeout=0 timeout_mode=0 wchan=0000000000000000
smokey-2499 [000] *.~3 61513.836590: cobalt_schedule: status=0x10000000
smokey-2499 [000] *.~3 61513.836590: cobalt_trace_pid: pid=2499, prio=1
smokey-2499 [000] *.~3 61513.836591: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x4488c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
gdb-2495 [000] *.~3 61513.836592: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] d..5 61513.836592: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dN.6 61513.836594: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dN.3 61513.836594: signal_generate: sig=19 errno=0 code=-6 comm=smokey pid=2499 grp=0 res=0
gdb-2495 [000] dNh3 61513.836595: irq_handler_entry: irq=120 name=in-band work
gdb-2495 [000] dNh3 61513.836596: cobalt_lostage_wakeup: pid=2499 comm=smokey
gdb-2495 [000] *N~4 61513.838552: irq_pipeline_entry: irq=4354
gdb-2495 [000] *N~4 61513.838553: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
gdb-2495 [000] *N~4 61513.838554: cobalt_timer_expire: timer=00000000d13b8aa9
gdb-2495 [000] *N~4 61513.838555: irq_handler_exit: irq=4354 ret=handled
gdb-2495 [000] *N~4 61513.838555: irq_pipeline_exit: irq=4354
gdb-2495 [000] dNh8 61513.981277: sched_waking: comm=systemd-journal pid=163 prio=120 target_cpu=000
gdb-2495 [000] dNh9 61513.981278: sched_stat_runtime: comm=gdb pid=2495 runtime=144934239 [ns] vruntime=300660160748 [ns]
gdb-2495 [000] dNh9 61513.981280: sched_stat_sleep: comm=systemd-journal pid=163 delay=111416392895 [ns]
gdb-2495 [000] dNh9 61513.981280: sched_wakeup: comm=systemd-journal pid=163 prio=120 target_cpu=000
gdb-2495 [000] dNh3 61513.981281: irq_handler_exit: irq=120 ret=handled
gdb-2495 [000] dNh3 61513.981282: irq_handler_entry: irq=124 name=proxy tick
gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=1 [action=TIMER]
gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=9 [action=RCU]
gdb-2495 [000] dNh4 61513.981286: sched_stat_runtime: comm=gdb pid=2495 runtime=8375 [ns] vruntime=300660169123 [ns]
gdb-2495 [000] dNh5 61513.981288: sched_waking: comm=cobalt_printf pid=2494 prio=120 target_cpu=000
gdb-2495 [000] dNh6 61513.981289: sched_stat_runtime: comm=gdb pid=2495 runtime=2987 [ns] vruntime=300660172110 [ns]
gdb-2495 [000] dNh6 61513.981291: sched_stat_sleep: comm=cobalt_printf pid=2494 delay=238660319 [ns]
gdb-2495 [000] dNh6 61513.981291: sched_wakeup: comm=cobalt_printf pid=2494 prio=120 target_cpu=000
gdb-2495 [000] dNh5 61513.981292: sched_waking: comm=cobalt_printf pid=2503 prio=120 target_cpu=000
gdb-2495 [000] dNh6 61513.981292: sched_stat_runtime: comm=gdb pid=2495 runtime=3069 [ns] vruntime=300660175179 [ns]
gdb-2495 [000] dNh6 61513.981294: sched_stat_sleep: comm=cobalt_printf pid=2503 delay=146611442 [ns]
gdb-2495 [000] dNh6 61513.981294: sched_wakeup: comm=cobalt_printf pid=2503 prio=120 target_cpu=000
gdb-2495 [000] *Nh3 61513.981295: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=588476 interval=0 mode=rel
gdb-2495 [000] *Nh3 61513.981296: cobalt_tick_shot: next tick at 61513.981884 (delay: 588 us)
gdb-2495 [000] dNh3 61513.981296: irq_handler_exit: irq=124 ret=handled
gdb-2495 [000] .Ns3 61513.981297: softirq_entry: vec=1 [action=TIMER]
gdb-2495 [000] dNs5 61513.981298: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000
gdb-2495 [000] dNs6 61513.981298: sched_stat_runtime: comm=gdb pid=2495 runtime=6218 [ns] vruntime=300660181397 [ns]
gdb-2495 [000] dNs6 61513.981299: sched_stat_blocked: comm=rcu_sched pid=11 delay=145119923 [ns]
gdb-2495 [000] dNs6 61513.981299: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000
gdb-2495 [000] dNs7 61513.981302: sched_waking: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000
gdb-2495 [000] dNs8 61513.981302: sched_stat_runtime: comm=gdb pid=2495 runtime=3884 [ns] vruntime=300660185281 [ns]
gdb-2495 [000] dNs8 61513.981303: sched_stat_blocked: comm=kworker/0:1 pid=1932 delay=206048652 [ns]
gdb-2495 [000] dNs8 61513.981304: sched_wakeup: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000
gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981312: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] .Ns3 61513.981314: softirq_exit: vec=1 [action=TIMER]
gdb-2495 [000] .Ns3 61513.981314: softirq_entry: vec=9 [action=RCU]
gdb-2495 [000] .Ns3 61513.981315: softirq_exit: vec=9 [action=RCU]
gdb-2495 [000] dNs5 61513.981316: sched_waking: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000
gdb-2495 [000] dNs6 61513.981316: sched_stat_runtime: comm=gdb pid=2495 runtime=13841 [ns] vruntime=300660199122 [ns]
gdb-2495 [000] dNs6 61513.981317: sched_stat_sleep: comm=ksoftirqd/0 pid=10 delay=149349034 [ns]
gdb-2495 [000] dNs6 61513.981317: sched_wakeup: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000
gdb-2495 [000] .N.1 61513.981318: sys_tkill -> 0x0
gdb-2495 [000] d..3 61513.981320: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] *.~3 61513.981888: irq_pipeline_entry: irq=4354
smokey-2499 [000] *.~3 61513.981888: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
smokey-2499 [000] *.~3 61513.981889: cobalt_timer_expire: timer=00000000d13b8aa9
smokey-2499 [000] *.~3 61513.981889: irq_handler_exit: irq=4354 ret=handled
smokey-2499 [000] *.~3 61513.981890: irq_pipeline_exit: irq=4354
smokey-2499 [000] d.h1 61514.106435: irq_handler_entry: irq=124 name=proxy tick
smokey-2499 [000] *.h1 61514.106441: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2109534 interval=0 mode=rel
smokey-2499 [000] *.h1 61514.106441: cobalt_tick_shot: next tick at 61514.108550 (delay: 2109 us)
smokey-2499 [000] d.h1 61514.106442: irq_handler_exit: irq=124 ret=handled
smokey-2499 [000] ...1 61514.106442: cobalt_shadow_relaxed: state=0x4488c0 info=0x200
smokey-2499 [000] *.~1 61514.108552: irq_pipeline_entry: irq=4354
Regards
Hongzhan Chen
>
>--
>Philippe.
^ permalink raw reply related [flat|nested] 16+ messages in thread
* RE: gdb test failure debug status update
2021-04-29 6:31 ` Chen, Hongzhan
@ 2021-04-30 5:25 ` Chen, Hongzhan
2021-04-30 5:51 ` Chen, Hongzhan
2021-04-30 7:34 ` Philippe Gerum
0 siblings, 2 replies; 16+ messages in thread
From: Chen, Hongzhan @ 2021-04-30 5:25 UTC (permalink / raw)
To: Chen, Hongzhan, Philippe Gerum, xenomai
The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish
->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax.
That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in
arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from
xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to
call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode
because kernel exit code is never called in this case?
Regards
Hongzhan Chen
-----Original Message-----
From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai
Sent: Thursday, April 29, 2021 2:31 PM
To: Philippe Gerum <rpm@xenomai.org>
Cc: xenomai@xenomai.org
Subject: RE: gdb test failure debug status update
>
>-----Original Message-----
>From: Philippe Gerum <rpm@xenomai.org>
>Sent: Wednesday, April 28, 2021 10:30 PM
>To: Philippe Gerum <rpm@xenomai.org>
>Cc: Chen, Hongzhan <hongzhan.chen@intel.com>; xenomai@xenomai.org
>Subject: Re: gdb test failure debug status update
>
>
>Philippe Gerum via Xenomai <xenomai@xenomai.org> writes:
>
>> Chen, Hongzhan via Xenomai <xenomai@xenomai.org> writes:
>>
>>> According to my validation, gdb test fail on dovetail 5.10 branch but pass on v5.9-evl4 tag with same for-upstream/dovetail
>>> xenomai code base.
>>>
>>> After further debug , the issue is more clear for me. Gdb test failure because low priority thread smokey userspace is still
>>> executed after "cobalt_shadow_relaxed: state=0x4488c0 info=0x200" like log [1] on dovetail-5.10 branch.
>>> The weird thing is that its following first ftrace log happen at 62235.848583 after cobalt_shadow_relaxed in log [1].
>>> It is almost 3ms happened after cobalt_shadow_relaxed. The low priority smoke thread user space is executed during this
>>> 3ms period so that test fail.
>>>
>>> But in success case with v5.9-evl4 like in log [2], the time interval between cobalt_shadow_relaxed and the following first ftrace log
>>> is only about 1us. It seems that low priority smokey userspace do not have chance to execute in this 1us because gdb test is successful.
>>>
>>> My question is why there is even no interrupt happened during that about 3ms period in failure case? Tick seems in abnormal behavior.
>>> Please comment if you have any ideas to further debug it.
>>>
>>> PS: All my tests run on same up Xtream board.
>>
>> <snip>
>>
>> Let's put aside the tick issue for now, there may be a valid reason for
>> this delay with dynticks enabled.
>>
>> The issue at stake may be related to the way a return to kernel space is
>> forced on a @user task (Dovetail has an integrated service for
>> triggering this called dovetail_request_ucall()).
>>
>> The logic for doing so is as follows:
>>
>> 1. @user hits a breakpoint, which is an exception Dovetail-wise
>>
>> 2. @user gets XNDBGSTOP set into its flags because Cobalt notices it is
>> being debugged via a breakpoint trap, then relaxed as a result of taking
>> a exception in general, so that we may traverse the common trap handling code
>> safely.
>>
>> 3. since XNDBGSTOP is a blocking bit Cobalt-wise, it should prevent
>> @user from being picked for scheduling by the real-time core, next time
>> a Cobalt considers rescheduling that is. However, since @user is
>> currently relaxed, it can still run under the supervision of the common
>> Linux scheduler. This is what the log[1] show.
>>
>> 4. the common/in-band kernel code stops @user due to the ptrace stop
>> condition caused by the breakpoint, waiting for a continuation event to
>> happen.
>>
>> Therefore, upon PTRACE_CONT (i.e. gdb continue), we need to force @user
>> to call back into kernel context (handle_ptrace_cont ->
>> dovetail_request_ucall), then ask for a switch to primary mode from
>> there, which should eventually happen when @user is about to leave the
>> kernel (on x86, this now happens from a generic kernel entry/exit code
>> in kernel/entry/*). As a result, handle_taskexit_event() runs, figures
>
>Not quite:
>s,handle_taskexit_event,handle_user_return,
There is trace_cobalt_shadow_unmap in handle_taskexit_event. I added
trace_cobalt_handle_user_return and trace_cobalt_handle_ptrace_cont
like patch [3] in log [4].
1. In the log [4], the first cobalt_handle_ptrace_cont is followed by cobalt_handle_user_return
that clear info flag XNCONTHI for smokey-2499 because cobalt_handle_ptrace_cont called dovetail_request_ucall .
the TIF_RETUSER bit was cleared for smokey-2499.
After then 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000
clear state flag XNDBGSTOP. After that, thread smokey-2499 run without these bit set till
"hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000".
2. There is second cobalt_handle_ptrace_cont called for smokey-2499 but at that time dovetail_request_ucall
would not be called because XNSSTEP Is not set like "smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0"
3. hi-thread-2504 breakpoint at its first breakpoint_target so it trigger hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3
It call dovetail_request_ucall(current); for itself and also call xnthread_set_info(thread, XNCONTHI); That is why we can see this bit set in
hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200
4. finally ,gdb call sys_tkill in gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13) and trigger that handle_oob_mayday is called so that xnthread_relax for
smokey-2499 Is called in "smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined."
It seems that there never call dovetail_request_ucall for process smokey-2499 in log[4] so that final xhthread_relax would have chance to return to user?
The second handle_ptrace_cont happen too much earlier because XNSSTEP still have not been set so that system never have change to call dovetail_request_ucall
for smokey-2499.
In this case , do we need to do hack in 3 , when handle_oob_trap_entry is called for hi-thread-2504 we call xnthread_set_info(thread, XNCONTHI);
and dovetail_request_ucall for all threads that have same cobalt_process? Is this solution OK for fixing this case?
[3]:
index 07dcc7a63..cd76961f5 100644
--- a/kernel/cobalt/dovetail/kevents.c
+++ b/kernel/cobalt/dovetail/kevents.c
@@ -331,6 +331,8 @@ static int handle_user_return(struct task_struct *task)
if (thread == NULL)
return KEVENT_PROPAGATE;
+ trace_cobalt_handle_user_return(thread);
+
if (xnthread_test_info(thread, XNCONTHI)) {
xnlock_get_irqsave(&nklock, s);
xnthread_clear_info(thread, XNCONTHI);
@@ -468,6 +470,8 @@ int handle_ptrace_resume(struct task_struct *tracee)
if (thread == NULL)
return KEVENT_PROPAGATE;
+ trace_cobalt_handle_ptrace_resume(thread);
+
if (xnthread_test_state(thread, XNSSTEP)) {
xnlock_get_irqsave(&nklock, s);
@@ -485,6 +489,8 @@ static void handle_ptrace_cont(void)
struct xnthread *curr = xnthread_current();
spl_t s;
+ trace_cobalt_handle_ptrace_cont(curr);
+
xnlock_get_irqsave(&nklock, s);
if (xnthread_test_state(curr, XNSSTEP)) {
@@ -604,6 +610,7 @@ void resume_oob_task(struct task_struct *p) /* inband, oob stage stalled */
if (unlikely(xnthread_test_info(thread, XNCONTHI))) {
xnthread_clear_info(thread, XNCONTHI);
dovetail_clear_ucall();
+ trace_cobalt_resume_oob_task(thread);
}
/* Unregister as debugged thread in case we postponed this. */
diff --git a/kernel/cobalt/trace/cobalt-core.h b/kernel/cobalt/trace/cobalt-core.h
index 78f66a106..88a4067fd 100644
--- a/kernel/cobalt/trace/cobalt-core.h
+++ b/kernel/cobalt/trace/cobalt-core.h
@@ -459,6 +459,27 @@ DEFINE_EVENT(curr_thread_event, cobalt_thread_set_mode,
TP_ARGS(thread)
);
+DEFINE_EVENT(thread_event, cobalt_handle_ptrace_cont,
+ TP_PROTO(struct xnthread *thread),
[4]:
gdb-2495 [000] d..3 61513.834825: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] ...1 61513.834827: cobalt_handle_ptrace_cont: pid=2499 state=0x4488c0 info=0x200
smokey-2499 [000] d..1 61513.834828: cobalt_handle_user_return: pid=2499 state=0x4488c0 info=0x200
smokey-2499 [000] d..1 61513.834828: cobalt_shadow_gohard: state=0x4488c0 info=0x0
smokey-2499 [000] d..3 61513.834830: sched_stat_wait: comm=gdb pid=2495 delay=8017 [ns]
smokey-2499 [000] d..3 61513.834831: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] *..3 61513.834831: cobalt_thread_resume: name=smokey pid=2499 mask=0x80
gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1
gdb-2495 [000] *..3 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000
gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1
gdb-2495 [000] *.~3 61513.834833: cobalt_schedule: status=0x10000000
gdb-2495 [000] *.~3 61513.834833: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] *.~3 61513.834833: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1
smokey-2499 [000] d.~3 61513.834835: cobalt_shadow_hardened: state=0x48040 info=0x0
smokey-2499 [000] d.~3 61513.834836: cobalt_head_sysentry: syscall=sem_init
smokey-2499 [000] d.~3 61513.834839: cobalt_psem_init: sem=0x4e(anon) flags=0x0() value=0
smokey-2499 [000] d.~3 61513.834839: cobalt_head_sysexit: result=0
smokey-2499 [000] d.~3 61513.834848: cobalt_shadow_gorelax: reason=syscall
smokey-2499 [000] d.~3 61513.834848: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey
smokey-2499 [000] *.~3 61513.834848: cobalt_thread_suspend: pid=2499 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
smokey-2499 [000] *.~3 61513.834849: cobalt_schedule: status=0x10000000
smokey-2499 [000] *.~3 61513.834849: cobalt_trace_pid: pid=2499, prio=1
smokey-2499 [000] *.~3 61513.834849: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
gdb-2495 [000] *.~3 61513.834850: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] d.h3 61513.834851: irq_handler_entry: irq=120 name=in-band work
gdb-2495 [000] d.h3 61513.834852: cobalt_lostage_wakeup: pid=2499 comm=smokey
gdb-2495 [000] d.h5 61513.834852: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dNh6 61513.834853: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dNh3 61513.834853: irq_handler_exit: irq=120 ret=handled
gdb-2495 [000] dN.3 61513.834854: sched_stat_runtime: comm=gdb pid=2495 runtime=23565 [ns] vruntime=300514015543 [ns]
gdb-2495 [000] d..3 61513.834855: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] ...1 61513.834856: cobalt_shadow_relaxed: state=0x480c0 info=0x0
smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min(policy: 1)
smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min -> 0x1
smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max(policy: 1)
smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max -> 0x63
smokey-2499 [000] ..~1 61513.834862: cobalt_head_sysentry: syscall=thread_getschedparam_ex
smokey-2499 [000] ...1 61513.834862: cobalt_root_sysentry: syscall=thread_getschedparam_ex
smokey-2499 [000] ...1 61513.834863: cobalt_pthread_getschedparam: pth=000000008d36be1c policy=fifo param={ priority=1 }
smokey-2499 [000] ...1 61513.834864: cobalt_root_sysexit: result=0
smokey-2499 [000] ...1 61513.834872: sys_mmap(addr: 0, len: 11000, prot: 0, flags: 20022, fd: ffffffff, off: 0)
smokey-2499 [000] ...1 61513.834877: sys_mmap -> 0x7ffff67b9000
smokey-2499 [000] ...1 61513.834878: sys_mprotect(start: 7ffff67ba000, len: 10000, prot: 3)
smokey-2499 [000] ...1 61513.834906: sys_mprotect -> 0x0
smokey-2499 [000] ...1 61513.834909: sys_clone(clone_flags: 3d0f00, newsp: 7ffff67c8fb0, parent_tidptr: 7ffff67c99d0, child_tidptr: 7ffff67c99d0, tls: 7ffff67c9700)
smokey-2499 [000] ...1 61513.834923: task_newtask: pid=2504 comm=smokey clone_flags=3d0f00 oom_score_adj=0
smokey-2499 [000] ...1 61513.834923: sched_process_fork: comm=smokey pid=2499 child_comm=smokey child_pid=2504
smokey-2499 [000] d..3 61513.834925: sched_wakeup_new: comm=smokey pid=2504 prio=98 target_cpu=000
smokey-2499 [000] d..3 61513.834928: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0
smokey-2499 [000] d..3 61513.834931: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=t ==> next_comm=smokey next_pid=2504 next_prio=98
hi-thread-2504 [000] ...1 61513.834932: sys_clone -> 0x0
hi-thread-2504 [000] d..3 61513.834934: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=2
hi-thread-2504 [000] d..3 61513.834935: sched_stat_wait: comm=gdb pid=2495 delay=82002 [ns]
hi-thread-2504 [000] d..3 61513.834936: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=t ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] d..2 61513.834937: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000
gdb-2495 [000] ...1 61513.834940: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1)
gdb-2495 [000] ...1 61513.834943: sys_read -> 0x1
gdb-2495 [000] ...1 61513.834943: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1)
gdb-2495 [000] ...1 61513.834944: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.834944: sys_write(fd: 10, buf: 5585fe0354b1, count: 1)
gdb-2495 [000] ...1 61513.834946: sys_write -> 0x1
gdb-2495 [000] ...1 61513.834947: sys_rt_sigreturn()
gdb-2495 [000] ...1 61513.834953: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1)
gdb-2495 [000] ...1 61513.834953: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.834955: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0)
gdb-2495 [000] ...1 61513.834956: sys_poll -> 0x1
gdb-2495 [000] ...1 61513.834957: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
gdb-2495 [000] ...1 61513.834958: sys_read -> 0x1
gdb-2495 [000] ...1 61513.834959: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
gdb-2495 [000] ...1 61513.834960: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8)
gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask -> 0x0
gdb-2495 [000] ...1 61513.834962: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
gdb-2495 [000] ...1 61513.834962: sched_process_wait: comm=gdb pid=0 prio=120
gdb-2495 [000] ...1 61513.834964: sys_wait4 -> 0x9c3
gdb-2495 [000] ...1 61513.834965: sys_ptrace(request: 4201, pid: 9c3, addr: 0, data: 7ffe8b2a7d08)
gdb-2495 [000] d..3 61513.834966: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.834967: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.834967: sys_wait4(upid: 9c8, stat_addr: 7ffe8b2a7cfc, options: 40000000, ru: 0)
gdb-2495 [000] ...1 61513.834968: sched_process_wait: comm=gdb pid=2504 prio=120
gdb-2495 [000] ...1 61513.834969: sys_wait4 -> 0x9c8
gdb-2495 [000] ...1 61513.834972: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578)
gdb-2495 [000] d..3 61513.834972: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.834975: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.834976: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a78d8)
gdb-2495 [000] d..3 61513.834977: sched_wait_task: comm=smokey pid=2504 prio=98
gdb-2495 [000] ...1 61513.834977: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.834979: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.834989: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.834990: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff67c9700)
gdb-2495 [000] ...1 61513.834992: sys_pread64 -> 0x900
gdb-2495 [000] ...1 61513.834993: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.834994: sys_close -> 0x0
gdb-2495 [000] ...1 61513.835010: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578)
gdb-2495 [000] d..3 61513.835011: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.835012: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835013: sys_ptrace(request: 3, pid: 9c3, addr: a8, data: 7ffe8b2a78d8)
gdb-2495 [000] d..3 61513.835013: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.835014: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835015: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.835022: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.835022: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff758f740)
gdb-2495 [000] ...1 61513.835025: sys_pread64 -> 0x900
gdb-2495 [000] ...1 61513.835025: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.835026: sys_close -> 0x0
gdb-2495 [000] ...1 61513.835031: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
gdb-2495 [000] ...1 61513.835032: sched_process_wait: comm=gdb pid=0 prio=120
gdb-2495 [000] ...1 61513.835033: sys_wait4 -> 0x0
gdb-2495 [000] ...1 61513.835036: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0)
gdb-2495 [000] d..3 61513.835037: sched_wait_task: comm=smokey pid=2504 prio=98
gdb-2495 [000] ...1 61513.835039: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835043: sys_ptrace(request: 6, pid: 9c8, addr: 388, data: 0)
gdb-2495 [000] d..3 61513.835044: sched_wait_task: comm=smokey pid=2504 prio=98
gdb-2495 [000] ...1 61513.835044: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835045: sys_ptrace(request: 7, pid: 9c8, addr: 1, data: 0)
gdb-2495 [000] d..3 61513.835046: sched_wait_task: comm=smokey pid=2504 prio=98
gdb-2495 [000] d..3 61513.835047: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000
gdb-2495 [000] dN.4 61513.835048: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000
gdb-2495 [000] .N.1 61513.835048: sys_ptrace -> 0x0
gdb-2495 [000] dN.3 61513.835050: sched_stat_runtime: comm=gdb pid=2495 runtime=112561 [ns] vruntime=300514128104 [ns]
gdb-2495 [000] d..3 61513.835051: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=98
hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list(head: 7ffff67c99e0, len: 18)
hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list -> 0x0
hi-thread-2504 [000] ...1 61513.835056: sys_futex(uaddr: 7ffff67c9d18, op: 80, val: 2, utime: 0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700)
hi-thread-2504 [000] d..3 61513.835059: sched_stat_wait: comm=gdb pid=2495 delay=10883 [ns]
hi-thread-2504 [000] d..3 61513.835060: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] ...1 61513.835063: sys_ptrace(request: c, pid: 9c3, addr: 0, data: 7ffe8b2a6ef0)
gdb-2495 [000] d..3 61513.835064: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.835065: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835068: sys_ptrace(request: 7, pid: 9c3, addr: 1, data: 0)
gdb-2495 [000] d..3 61513.835069: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] d..3 61513.835069: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dN.4 61513.835070: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] .N.1 61513.835071: sys_ptrace -> 0x0
gdb-2495 [000] dN.3 61513.835071: sched_stat_runtime: comm=gdb pid=2495 runtime=12210 [ns] vruntime=300514140314 [ns]
gdb-2495 [000] d..3 61513.835072: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0
smokey-2499 [000] ...1 61513.835074: sys_clone -> 0x9c8
smokey-2499 [000] ...1 61513.835075: sys_sched_setaffinity(pid: 9c8, len: 80, user_mask_ptr: 55555563e7c0)
smokey-2499 [000] ...1 61513.835077: sys_sched_setaffinity -> 0x0
smokey-2499 [000] ...1 61513.835077: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 0, val3: 7ffff67c9700)
smokey-2499 [000] d..3 61513.835078: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000
smokey-2499 [000] d..4 61513.835079: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000
smokey-2499 [000] ...1 61513.835080: sys_futex -> 0x1
smokey-2499 [000] ...1 61513.835086: sys_futex(uaddr: 7fffffffe828, op: 189, val: 0, utime: 7fffffffe780, uaddr2: 0, val3: ffffffff)
smokey-2499 [000] d..3 61513.835089: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=smokey next_pid=2504 next_prio=98
hi-thread-2504 [000] ...1 61513.835090: sys_futex -> 0x0
hi-thread-2504 [000] ...1 61513.835090: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700)
hi-thread-2504 [000] ...1 61513.835091: sys_futex -> 0x0
hi-thread-2504 [000] ...1 61513.835092: sys_sched_setscheduler(pid: 9c8, policy: 1, param: 7ffff67c8e70)
hi-thread-2504 [000] ...1 61513.835095: sys_sched_setscheduler -> 0x0
hi-thread-2504 [000] ..~1 61513.835096: cobalt_head_sysentry: syscall=thread_create
hi-thread-2504 [000] ...1 61513.835097: cobalt_root_sysentry: syscall=thread_create
hi-thread-2504 [000] ...1 61513.835098: cobalt_pthread_create: pth=00000000ba35a01d policy=fifo param={ priority=2 }
hi-thread-2504 [000] ...1 61513.835099: cobalt_timer_migrate: timer=000000004d21df54 cpu=0
hi-thread-2504 [000] ...1 61513.835100: cobalt_timer_migrate: timer=00000000c1301853 cpu=0
hi-thread-2504 [000] ...1 61513.835101: cobalt_thread_set_current_prio: thread=00000000e9fda3c4 pid=-1 prio=2
hi-thread-2504 [000] ...1 61513.835102: cobalt_thread_init: thread=00000000e9fda3c4 name=smokey flags=0x48000 class=rt prio=2
hi-thread-2504 [000] ...1 61513.835104: cobalt_shadow_map: thread=00000000e9fda3c4 pid=2504 prio=2
hi-thread-2504 [000] D..1 61513.835104: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_start: pid=2504 state=0x480d0 info=0x0
hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_resume: name=smokey pid=2504 mask=0x10
hi-thread-2504 [000] D..1 61513.835106: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] ...1 61513.835106: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] ...1 61513.835107: cobalt_shadow_gohard: state=0x480c0 info=0x0
hi-thread-2504 [000] d..3 61513.835108: sched_stat_wait: comm=gdb pid=2495 delay=37231 [ns]
hi-thread-2504 [000] d..3 61513.835109: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] *..3 61513.835109: cobalt_thread_resume: name=smokey pid=2504 mask=0x80
gdb-2495 [000] *..3 61513.835110: cobalt_trace_pid: pid=2504, prio=2
gdb-2495 [000] *.~3 61513.835110: cobalt_schedule: status=0x10000000
gdb-2495 [000] *.~3 61513.835110: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] *.~3 61513.835111: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2
hi-thread-2504 [000] d.~3 61513.835112: cobalt_shadow_hardened: state=0x48040 info=0x0
hi-thread-2504 [000] d.~3 61513.835112: cobalt_root_sysexit: result=0
hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysentry: syscall=get_current
hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysexit: result=0
hi-thread-2504 [000] d.~3 61513.835116: cobalt_shadow_gorelax: reason=syscall
hi-thread-2504 [000] d.~3 61513.835117: cobalt_lostage_request: request=wakeup pid=2504 comm=smokey
hi-thread-2504 [000] *.~3 61513.835117: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] *.~3 61513.835117: cobalt_schedule: status=0x10000000
hi-thread-2504 [000] *.~3 61513.835118: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] *.~3 61513.835118: cobalt_switch_context: prev_name=smokey prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
gdb-2495 [000] *.~3 61513.835119: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] d.h3 61513.835120: irq_handler_entry: irq=120 name=in-band work
gdb-2495 [000] d.h3 61513.835120: cobalt_lostage_wakeup: pid=2504 comm=smokey
gdb-2495 [000] d.h5 61513.835120: sched_waking: comm=smokey pid=2504 prio=97 target_cpu=000
gdb-2495 [000] dNh6 61513.835121: sched_wakeup: comm=smokey pid=2504 prio=97 target_cpu=000
gdb-2495 [000] dNh3 61513.835122: irq_handler_exit: irq=120 ret=handled
gdb-2495 [000] dN.3 61513.835122: sched_stat_runtime: comm=gdb pid=2495 runtime=13441 [ns] vruntime=300514153755 [ns]
gdb-2495 [000] d..3 61513.835123: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=97
hi-thread-2504 [000] ...1 61513.835124: cobalt_shadow_relaxed: state=0x480c0 info=0x0
hi-thread-2504 [000] ...1 61513.835125: sys_futex(uaddr: 7fffffffe828, op: 81, val: 1, utime: 0, uaddr2: 7ffff67c8e74, val3: 7fffffffe828)
hi-thread-2504 [000] d..3 61513.835126: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
hi-thread-2504 [000] d..4 61513.835126: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
hi-thread-2504 [000] ...1 61513.835127: sys_futex -> 0x1
hi-thread-2504 [000] ..~1 61513.835128: cobalt_head_sysentry: syscall=migrate
hi-thread-2504 [000] ...1 61513.835128: cobalt_root_sysentry: syscall=migrate
hi-thread-2504 [000] ...1 61513.835128: cobalt_shadow_gohard: state=0x480c0 info=0x0
hi-thread-2504 [000] d..3 61513.835130: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] *..3 61513.835130: cobalt_thread_resume: name=smokey pid=2504 mask=0x80
smokey-2499 [000] *..3 61513.835130: cobalt_trace_pid: pid=2504, prio=2
smokey-2499 [000] *.~3 61513.835131: cobalt_schedule: status=0x10000000
smokey-2499 [000] *.~3 61513.835131: cobalt_trace_pid: pid=2499, prio=-1
smokey-2499 [000] *.~3 61513.835131: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2
hi-thread-2504 [000] d.~3 61513.835132: cobalt_shadow_hardened: state=0x48040 info=0x0
hi-thread-2504 [000] d.~3 61513.835132: cobalt_root_sysexit: result=1
hi-thread-2504 [000] d.~3 61513.835133: cobalt_head_sysentry: syscall=thread_setname
hi-thread-2504 [000] d.~3 61513.835134: cobalt_pthread_setname: pth=00000000ba35a01d name=hi-thread
hi-thread-2504 [000] d.~3 61513.835135: cobalt_head_sysexit: result=0
hi-thread-2504 [000] d.~3 61513.835136: cobalt_head_sysentry: syscall=clock_nanosleep
hi-thread-2504 [000] d.~3 61513.835137: cobalt_clock_nanosleep: clock_id=0 flags=0x0() rqt=(0.001000000)
hi-thread-2504 [000] *.~3 61513.835138: cobalt_thread_suspend: pid=2504 mask=0x4 timeout=1000001 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] *.~3 61513.835138: cobalt_timer_start: timer=000000004d21df54(smokey) value=1000001 interval=0 mode=rel
hi-thread-2504 [000] *.~3 61513.835138: cobalt_schedule: status=0x10000000
hi-thread-2504 [000] *.~3 61513.835139: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] *.~3 61513.835139: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x48044 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
smokey-2499 [000] *.~3 61513.835140: cobalt_trace_pid: pid=2499, prio=-1
smokey-2499 [000] ...1 61513.835141: sys_futex -> 0x0
smokey-2499 [000] ..~1 61513.835144: cobalt_head_sysentry: syscall=migrate
smokey-2499 [000] ...1 61513.835144: cobalt_root_sysentry: syscall=migrate
smokey-2499 [000] ...1 61513.835144: cobalt_shadow_gohard: state=0x480c0 info=0x0
smokey-2499 [000] d..3 61513.835145: sched_stat_wait: comm=gdb pid=2495 delay=23982 [ns]
smokey-2499 [000] d..3 61513.835146: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] *..3 61513.835147: cobalt_thread_resume: name=smokey pid=2499 mask=0x80
gdb-2495 [000] *..3 61513.835147: cobalt_trace_pid: pid=2499, prio=1
gdb-2495 [000] *.~3 61513.835147: cobalt_schedule: status=0x10000000
gdb-2495 [000] *.~3 61513.835147: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] *.~3 61513.835148: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1
smokey-2499 [000] d.~3 61513.835148: cobalt_shadow_hardened: state=0x48040 info=0x0
smokey-2499 [000] d.~3 61513.835149: cobalt_root_sysexit: result=1
smokey-2499 [000] *.~3 61513.835219: irq_pipeline_entry: irq=4354
smokey-2499 [000] *.~3 61513.835219: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
smokey-2499 [000] *.~3 61513.835220: cobalt_timer_expire: timer=00000000d13b8aa9
smokey-2499 [000] *.~3 61513.835220: cobalt_tick_shot: next tick at 61513.836136 (delay: 915 us)
smokey-2499 [000] *.~3 61513.835221: irq_handler_exit: irq=4354 ret=handled
smokey-2499 [000] *.~3 61513.835221: irq_pipeline_exit: irq=4354
smokey-2499 [000] *.~3 61513.836137: irq_pipeline_entry: irq=4354
smokey-2499 [000] *.~3 61513.836137: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
smokey-2499 [000] *.~3 61513.836138: cobalt_timer_expire: timer=000000004d21df54
smokey-2499 [000] *.~3 61513.836138: cobalt_thread_resume: name=hi-thread pid=2504 mask=0x4
smokey-2499 [000] *.~3 61513.836138: cobalt_trace_pid: pid=2504, prio=2
smokey-2499 [000] *.~3 61513.836139: cobalt_timer_stop: timer=000000004d21df54
smokey-2499 [000] *.~3 61513.836139: irq_handler_exit: irq=4354 ret=handled
smokey-2499 [000] *.~3 61513.836139: irq_pipeline_exit: irq=4354
smokey-2499 [000] *.~3 61513.836140: cobalt_schedule: status=0x10000000
smokey-2499 [000] *.~3 61513.836140: cobalt_trace_pid: pid=2499, prio=1
smokey-2499 [000] *.~3 61513.836140: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x48048 ==> next_name=hi-thread next_pid=2504 next_prio=2
hi-thread-2504 [000] *.~3 61513.836141: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] d.~3 61513.836141: cobalt_head_sysexit: result=0
hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3
hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] d.~3 61513.836143: cobalt_shadow_gorelax: reason=fault
hi-thread-2504 [000] d.~3 61513.836143: cobalt_lostage_request: request=wakeup pid=2504 comm=hi-thread
hi-thread-2504 [000] *.~3 61513.836144: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] *.~3 61513.836144: cobalt_schedule: status=0x10000000
hi-thread-2504 [000] *.~3 61513.836144: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] *.~3 61513.836145: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
gdb-2495 [000] *.~3 61513.836146: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] d.h3 61513.836146: irq_handler_entry: irq=120 name=in-band work
gdb-2495 [000] d.h3 61513.836147: cobalt_lostage_wakeup: pid=2504 comm=hi-thread
gdb-2495 [000] d.h5 61513.836147: sched_waking: comm=hi-thread pid=2504 prio=97 target_cpu=000
gdb-2495 [000] dNh6 61513.836147: sched_wakeup: comm=hi-thread pid=2504 prio=97 target_cpu=000
gdb-2495 [000] dNh3 61513.836148: irq_handler_exit: irq=120 ret=handled
gdb-2495 [000] dNh3 61513.836148: irq_handler_entry: irq=124 name=proxy tick
gdb-2495 [000] dNh3 61513.836151: softirq_raise: vec=1 [action=TIMER]
gdb-2495 [000] dNh3 61513.836152: softirq_raise: vec=9 [action=RCU]
gdb-2495 [000] dNh4 61513.836152: sched_stat_runtime: comm=gdb pid=2495 runtime=1007474 [ns] vruntime=300515161229 [ns]
gdb-2495 [000] *Nh3 61513.836154: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2395843 interval=0 mode=rel
gdb-2495 [000] *Nh3 61513.836155: cobalt_tick_shot: next tick at 61513.838550 (delay: 2395 us)
gdb-2495 [000] dNh3 61513.836155: irq_handler_exit: irq=124 ret=handled
gdb-2495 [000] .Ns3 61513.836156: softirq_entry: vec=1 [action=TIMER]
gdb-2495 [000] .Ns3 61513.836157: softirq_exit: vec=1 [action=TIMER]
gdb-2495 [000] .Ns3 61513.836157: softirq_entry: vec=9 [action=RCU]
gdb-2495 [000] dNs6 61513.836158: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000
gdb-2495 [000] dNs7 61513.836159: sched_stat_runtime: comm=gdb pid=2495 runtime=6881 [ns] vruntime=300515168110 [ns]
gdb-2495 [000] dNs7 61513.836160: sched_stat_blocked: comm=rcu_sched pid=11 delay=4188739 [ns]
gdb-2495 [000] dNs7 61513.836161: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000
gdb-2495 [000] .Ns3 61513.836164: softirq_exit: vec=9 [action=RCU]
gdb-2495 [000] d..3 61513.836165: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=hi-thread next_pid=2504 next_prio=97
hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200
hi-thread-2504 [000] *..2 61513.836169: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] *..2 61513.836170: cobalt_thread_suspend: pid=2504 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] d..2 61513.836170: signal_generate: sig=5 errno=0 code=128 comm=hi-thread pid=2504 grp=0 res=0
hi-thread-2504 [000] d..3 61513.836173: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0
hi-thread-2504 [000] d..3 61513.836175: sched_stat_wait: comm=rcu_sched pid=11 delay=14490 [ns]
hi-thread-2504 [000] d..3 61513.836175: sched_switch: prev_comm=hi-thread prev_pid=2504 prev_prio=97 prev_state=t ==> next_comm=rcu_sched next_pid=11 next_prio=120
rcu_sched-11 [000] d..3 61513.836179: sched_stat_runtime: comm=rcu_sched pid=11 runtime=4960 [ns] vruntime=300512173070 [ns]
rcu_sched-11 [000] d..3 61513.836179: sched_stat_wait: comm=gdb pid=2495 delay=19450 [ns]
rcu_sched-11 [000] d..3 61513.836180: sched_switch: prev_comm=rcu_sched prev_pid=11 prev_prio=120 prev_state=I ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] ...1 61513.836187: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cb0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836197: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836198: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7ae0)
gdb-2495 [000] ...1 61513.836199: sys_newfstat -> 0x0
gdb-2495 [000] ...1 61513.836200: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
gdb-2495 [000] ...1 61513.836214: sys_read -> 0x3f1
gdb-2495 [000] ...1 61513.836217: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836217: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836221: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8)
gdb-2495 [000] ...1 61513.836222: sys_rt_sigprocmask -> 0x0
gdb-2495 [000] d..2 61513.836223: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000
gdb-2495 [000] ...1 61513.836225: sys_read(fd: f, buf: 7ffe8b2a7667, count: 1)
gdb-2495 [000] ...1 61513.836226: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.836227: sys_write(fd: 10, buf: 5585fe0354b1, count: 1)
gdb-2495 [000] ...1 61513.836229: sys_write -> 0x1
gdb-2495 [000] ...1 61513.836229: sys_rt_sigreturn()
gdb-2495 [000] ...1 61513.836233: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1)
gdb-2495 [000] ...1 61513.836234: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.836235: sys_write(fd: 1, buf: 5585fe7a5290, count: 27)
gdb-2495 [000] d..4 61513.836236: sched_waking: comm=smokey pid=2493 prio=120 target_cpu=000
gdb-2495 [000] d..5 61513.836237: sched_stat_runtime: comm=gdb pid=2495 runtime=58399 [ns] vruntime=300515226509 [ns]
gdb-2495 [000] d..5 61513.836238: sched_stat_sleep: comm=smokey pid=2493 delay=1717179 [ns]
gdb-2495 [000] dN.5 61513.836239: sched_wakeup: comm=smokey pid=2493 prio=120 target_cpu=000
gdb-2495 [000] .N.1 61513.836240: sys_write -> 0x27
gdb-2495 [000] dN.3 61513.836241: sched_stat_wait: comm=smokey pid=2493 delay=0 [ns]
gdb-2495 [000] d..3 61513.836241: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2493 next_prio=120
smokey-2493 [000] ...1 61513.836244: sys_read -> 0x1
smokey-2493 [000] ..~1 61513.836246: cobalt_head_sysentry: syscall=read
...
smokey-2493 [000] d..3 61513.836344: sched_stat_runtime: comm=smokey pid=2493 runtime=106636 [ns] vruntime=300512333145 [ns]
smokey-2493 [000] d..3 61513.836344: sched_stat_wait: comm=gdb pid=2495 delay=106636 [ns]
smokey-2493 [000] d..3 61513.836345: sched_switch: prev_comm=smokey prev_pid=2493 prev_prio=120 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] ...1 61513.836347: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0)
gdb-2495 [000] ...1 61513.836348: sys_poll -> 0x1
gdb-2495 [000] ...1 61513.836350: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
gdb-2495 [000] ...1 61513.836351: sys_read -> 0x1
gdb-2495 [000] ...1 61513.836352: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
gdb-2495 [000] ...1 61513.836352: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.836353: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8)
gdb-2495 [000] ...1 61513.836354: sys_rt_sigprocmask -> 0x0
gdb-2495 [000] ...1 61513.836355: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
gdb-2495 [000] ...1 61513.836355: sched_process_wait: comm=gdb pid=0 prio=120
gdb-2495 [000] ...1 61513.836357: sys_wait4 -> 0x9c8
gdb-2495 [000] ...1 61513.836360: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0)
gdb-2495 [000] d..3 61513.836361: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836363: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836367: sys_ptrace(request: 4202, pid: 9c8, addr: 0, data: 7ffe8b2a7cb0)
gdb-2495 [000] d..3 61513.836368: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836369: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836370: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6f10)
gdb-2495 [000] d..3 61513.836370: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836371: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836372: sys_ptrace(request: d, pid: 9c8, addr: 0, data: 7ffe8b2a6f10)
gdb-2495 [000] d..3 61513.836373: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836374: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836376: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
gdb-2495 [000] ...1 61513.836376: sched_process_wait: comm=gdb pid=0 prio=120
gdb-2495 [000] ...1 61513.836377: sys_wait4 -> 0x0
gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8)
gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask -> 0x0
gdb-2495 [000] ...1 61513.836382: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cd0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836399: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836400: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7b50)
gdb-2495 [000] ...1 61513.836401: sys_newfstat -> 0x0
gdb-2495 [000] ...1 61513.836402: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
gdb-2495 [000] ...1 61513.836409: sys_read -> 0x140
gdb-2495 [000] ...1 61513.836410: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400)
gdb-2495 [000] ...1 61513.836410: sys_read -> 0x0
gdb-2495 [000] ...1 61513.836415: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836416: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836419: sys_read(fd: f, buf: 7ffe8b2a7f17, count: 1)
gdb-2495 [000] ...1 61513.836420: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.836421: sys_write(fd: 10, buf: 5585fe0354b1, count: 1)
gdb-2495 [000] ...1 61513.836422: sys_write -> 0x1
gdb-2495 [000] ...1 61513.836425: sys_ptrace(request: 1, pid: 9c8, addr: 7ffff777f370, data: 7ffe8b2a7878)
gdb-2495 [000] d..3 61513.836426: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836428: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836430: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a7bd8)
gdb-2495 [000] d..3 61513.836430: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836431: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836433: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a70e0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836446: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836446: sys_pread64(fd: 12, buf: 7ffe8b2a7430, count: 900, pos: 7ffff67c9700)
gdb-2495 [000] ...1 61513.836449: sys_pread64 -> 0x900
gdb-2495 [000] ...1 61513.836450: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836450: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836479: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836488: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836489: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950)
gdb-2495 [000] ...1 61513.836490: sys_newfstat -> 0x0
gdb-2495 [000] ...1 61513.836490: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400)
gdb-2495 [000] ...1 61513.836498: sys_read -> 0x14d
gdb-2495 [000] ...1 61513.836499: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
gdb-2495 [000] ...1 61513.836499: sys_read -> 0x0
gdb-2495 [000] ...1 61513.836503: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836504: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836507: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836514: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836515: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950)
gdb-2495 [000] ...1 61513.836515: sys_newfstat -> 0x0
gdb-2495 [000] ...1 61513.836516: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
gdb-2495 [000] ...1 61513.836521: sys_read -> 0x13b
gdb-2495 [000] ...1 61513.836522: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400)
gdb-2495 [000] ...1 61513.836522: sys_read -> 0x0
gdb-2495 [000] ...1 61513.836525: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836525: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13)
gdb-2495 [000] *..3 61513.836532: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000
gdb-2495 [000] *..3 61513.836532: cobalt_trace_pid: pid=2499, prio=1
gdb-2495 [000] *..3 61513.836533: cobalt_thread_unblock: pid=2499 state=0x48848 info=0x0
gdb-2495 [000] *.~3 61513.836586: cobalt_schedule: status=0x10000000
gdb-2495 [000] *.~3 61513.836586: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] *.~3 61513.836587: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1
smokey-2499 [000] *.~3 61513.836589: cobalt_trace_pid: pid=2499, prio=1
smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined
smokey-2499 [000] *.~3 61513.836589: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey
smokey-2499 [000] *.~3 61513.836590: cobalt_thread_suspend: pid=2499 mask=0x400080 timeout=0 timeout_mode=0 wchan=0000000000000000
smokey-2499 [000] *.~3 61513.836590: cobalt_schedule: status=0x10000000
smokey-2499 [000] *.~3 61513.836590: cobalt_trace_pid: pid=2499, prio=1
smokey-2499 [000] *.~3 61513.836591: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x4488c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
gdb-2495 [000] *.~3 61513.836592: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] d..5 61513.836592: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dN.6 61513.836594: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dN.3 61513.836594: signal_generate: sig=19 errno=0 code=-6 comm=smokey pid=2499 grp=0 res=0
gdb-2495 [000] dNh3 61513.836595: irq_handler_entry: irq=120 name=in-band work
gdb-2495 [000] dNh3 61513.836596: cobalt_lostage_wakeup: pid=2499 comm=smokey
gdb-2495 [000] *N~4 61513.838552: irq_pipeline_entry: irq=4354
gdb-2495 [000] *N~4 61513.838553: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
gdb-2495 [000] *N~4 61513.838554: cobalt_timer_expire: timer=00000000d13b8aa9
gdb-2495 [000] *N~4 61513.838555: irq_handler_exit: irq=4354 ret=handled
gdb-2495 [000] *N~4 61513.838555: irq_pipeline_exit: irq=4354
gdb-2495 [000] dNh8 61513.981277: sched_waking: comm=systemd-journal pid=163 prio=120 target_cpu=000
gdb-2495 [000] dNh9 61513.981278: sched_stat_runtime: comm=gdb pid=2495 runtime=144934239 [ns] vruntime=300660160748 [ns]
gdb-2495 [000] dNh9 61513.981280: sched_stat_sleep: comm=systemd-journal pid=163 delay=111416392895 [ns]
gdb-2495 [000] dNh9 61513.981280: sched_wakeup: comm=systemd-journal pid=163 prio=120 target_cpu=000
gdb-2495 [000] dNh3 61513.981281: irq_handler_exit: irq=120 ret=handled
gdb-2495 [000] dNh3 61513.981282: irq_handler_entry: irq=124 name=proxy tick
gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=1 [action=TIMER]
gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=9 [action=RCU]
gdb-2495 [000] dNh4 61513.981286: sched_stat_runtime: comm=gdb pid=2495 runtime=8375 [ns] vruntime=300660169123 [ns]
gdb-2495 [000] dNh5 61513.981288: sched_waking: comm=cobalt_printf pid=2494 prio=120 target_cpu=000
gdb-2495 [000] dNh6 61513.981289: sched_stat_runtime: comm=gdb pid=2495 runtime=2987 [ns] vruntime=300660172110 [ns]
gdb-2495 [000] dNh6 61513.981291: sched_stat_sleep: comm=cobalt_printf pid=2494 delay=238660319 [ns]
gdb-2495 [000] dNh6 61513.981291: sched_wakeup: comm=cobalt_printf pid=2494 prio=120 target_cpu=000
gdb-2495 [000] dNh5 61513.981292: sched_waking: comm=cobalt_printf pid=2503 prio=120 target_cpu=000
gdb-2495 [000] dNh6 61513.981292: sched_stat_runtime: comm=gdb pid=2495 runtime=3069 [ns] vruntime=300660175179 [ns]
gdb-2495 [000] dNh6 61513.981294: sched_stat_sleep: comm=cobalt_printf pid=2503 delay=146611442 [ns]
gdb-2495 [000] dNh6 61513.981294: sched_wakeup: comm=cobalt_printf pid=2503 prio=120 target_cpu=000
gdb-2495 [000] *Nh3 61513.981295: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=588476 interval=0 mode=rel
gdb-2495 [000] *Nh3 61513.981296: cobalt_tick_shot: next tick at 61513.981884 (delay: 588 us)
gdb-2495 [000] dNh3 61513.981296: irq_handler_exit: irq=124 ret=handled
gdb-2495 [000] .Ns3 61513.981297: softirq_entry: vec=1 [action=TIMER]
gdb-2495 [000] dNs5 61513.981298: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000
gdb-2495 [000] dNs6 61513.981298: sched_stat_runtime: comm=gdb pid=2495 runtime=6218 [ns] vruntime=300660181397 [ns]
gdb-2495 [000] dNs6 61513.981299: sched_stat_blocked: comm=rcu_sched pid=11 delay=145119923 [ns]
gdb-2495 [000] dNs6 61513.981299: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000
gdb-2495 [000] dNs7 61513.981302: sched_waking: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000
gdb-2495 [000] dNs8 61513.981302: sched_stat_runtime: comm=gdb pid=2495 runtime=3884 [ns] vruntime=300660185281 [ns]
gdb-2495 [000] dNs8 61513.981303: sched_stat_blocked: comm=kworker/0:1 pid=1932 delay=206048652 [ns]
gdb-2495 [000] dNs8 61513.981304: sched_wakeup: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000
gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981312: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] .Ns3 61513.981314: softirq_exit: vec=1 [action=TIMER]
gdb-2495 [000] .Ns3 61513.981314: softirq_entry: vec=9 [action=RCU]
gdb-2495 [000] .Ns3 61513.981315: softirq_exit: vec=9 [action=RCU]
gdb-2495 [000] dNs5 61513.981316: sched_waking: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000
gdb-2495 [000] dNs6 61513.981316: sched_stat_runtime: comm=gdb pid=2495 runtime=13841 [ns] vruntime=300660199122 [ns]
gdb-2495 [000] dNs6 61513.981317: sched_stat_sleep: comm=ksoftirqd/0 pid=10 delay=149349034 [ns]
gdb-2495 [000] dNs6 61513.981317: sched_wakeup: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000
gdb-2495 [000] .N.1 61513.981318: sys_tkill -> 0x0
gdb-2495 [000] d..3 61513.981320: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] *.~3 61513.981888: irq_pipeline_entry: irq=4354
smokey-2499 [000] *.~3 61513.981888: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
smokey-2499 [000] *.~3 61513.981889: cobalt_timer_expire: timer=00000000d13b8aa9
smokey-2499 [000] *.~3 61513.981889: irq_handler_exit: irq=4354 ret=handled
smokey-2499 [000] *.~3 61513.981890: irq_pipeline_exit: irq=4354
smokey-2499 [000] d.h1 61514.106435: irq_handler_entry: irq=124 name=proxy tick
smokey-2499 [000] *.h1 61514.106441: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2109534 interval=0 mode=rel
smokey-2499 [000] *.h1 61514.106441: cobalt_tick_shot: next tick at 61514.108550 (delay: 2109 us)
smokey-2499 [000] d.h1 61514.106442: irq_handler_exit: irq=124 ret=handled
smokey-2499 [000] ...1 61514.106442: cobalt_shadow_relaxed: state=0x4488c0 info=0x200
smokey-2499 [000] *.~1 61514.108552: irq_pipeline_entry: irq=4354
Regards
Hongzhan Chen
>
>--
>Philippe.
^ permalink raw reply related [flat|nested] 16+ messages in thread
* RE: gdb test failure debug status update
2021-04-30 5:25 ` Chen, Hongzhan
@ 2021-04-30 5:51 ` Chen, Hongzhan
2021-04-30 7:36 ` Philippe Gerum
2021-04-30 7:34 ` Philippe Gerum
1 sibling, 1 reply; 16+ messages in thread
From: Chen, Hongzhan @ 2021-04-30 5:51 UTC (permalink / raw)
To: Philippe Gerum, xenomai
After applied following patch , the gdb test failure issue seems fixed. Basically , it enforce going back to
primary mode after check that XNCONTHI is set for current thread. It seems ugly. Please suggest different
solution.
@@ -355,9 +357,36 @@ static int handle_user_return(struct task_struct *task)
void handle_oob_mayday(struct pt_regs *regs)
{
+ struct xnthread *thread;
+ spl_t s;
+ int err;
+
XENO_BUG_ON(COBALT, !xnthread_test_state(xnthread_current(), XNUSER));
+ thread = xnthread_current();
+
+ trace_cobalt_handle_oob_mayday(thread);
+
xnthread_relax(0, 0);
+
+ if (xnthread_test_info(thread, XNCONTHI)) {
+ xnlock_get_irqsave(&nklock, s);
+ xnthread_clear_info(thread, XNCONTHI);
+ xnlock_put_irqrestore(&nklock, s);
+
+ err = xnthread_harden();
+
+ /*
+ * XNCONTHI may or may not have been re-applied if
+ * harden bailed out due to pending signals. Make sure
+ * it is set in that case.
+ */
+ if (err == -ERESTARTSYS) {
+ xnlock_get_irqsave(&nklock, s);
+ xnthread_set_info(thread, XNCONTHI);
+ xnlock_put_irqrestore(&nklock, s);
+ }
+ }
}
Regards
Hongzhan Chen
-----Original Message-----
From: Chen, Hongzhan <hongzhan.chen@intel.com>
Sent: Friday, April 30, 2021 1:26 PM
To: Chen, Hongzhan <hongzhan.chen@intel.com>; Philippe Gerum <rpm@xenomai.org>; xenomai@xenomai.org
Subject: RE: gdb test failure debug status update
The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish
->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax.
That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in
arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from
xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to
call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode
because kernel exit code is never called in this case?
Regards
Hongzhan Chen
-----Original Message-----
From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai
Sent: Thursday, April 29, 2021 2:31 PM
To: Philippe Gerum <rpm@xenomai.org>
Cc: xenomai@xenomai.org
Subject: RE: gdb test failure debug status update
>
>-----Original Message-----
>From: Philippe Gerum <rpm@xenomai.org>
>Sent: Wednesday, April 28, 2021 10:30 PM
>To: Philippe Gerum <rpm@xenomai.org>
>Cc: Chen, Hongzhan <hongzhan.chen@intel.com>; xenomai@xenomai.org
>Subject: Re: gdb test failure debug status update
>
>
>Philippe Gerum via Xenomai <xenomai@xenomai.org> writes:
>
>> Chen, Hongzhan via Xenomai <xenomai@xenomai.org> writes:
>>
>>> According to my validation, gdb test fail on dovetail 5.10 branch but pass on v5.9-evl4 tag with same for-upstream/dovetail
>>> xenomai code base.
>>>
>>> After further debug , the issue is more clear for me. Gdb test failure because low priority thread smokey userspace is still
>>> executed after "cobalt_shadow_relaxed: state=0x4488c0 info=0x200" like log [1] on dovetail-5.10 branch.
>>> The weird thing is that its following first ftrace log happen at 62235.848583 after cobalt_shadow_relaxed in log [1].
>>> It is almost 3ms happened after cobalt_shadow_relaxed. The low priority smoke thread user space is executed during this
>>> 3ms period so that test fail.
>>>
>>> But in success case with v5.9-evl4 like in log [2], the time interval between cobalt_shadow_relaxed and the following first ftrace log
>>> is only about 1us. It seems that low priority smokey userspace do not have chance to execute in this 1us because gdb test is successful.
>>>
>>> My question is why there is even no interrupt happened during that about 3ms period in failure case? Tick seems in abnormal behavior.
>>> Please comment if you have any ideas to further debug it.
>>>
>>> PS: All my tests run on same up Xtream board.
>>
>> <snip>
>>
>> Let's put aside the tick issue for now, there may be a valid reason for
>> this delay with dynticks enabled.
>>
>> The issue at stake may be related to the way a return to kernel space is
>> forced on a @user task (Dovetail has an integrated service for
>> triggering this called dovetail_request_ucall()).
>>
>> The logic for doing so is as follows:
>>
>> 1. @user hits a breakpoint, which is an exception Dovetail-wise
>>
>> 2. @user gets XNDBGSTOP set into its flags because Cobalt notices it is
>> being debugged via a breakpoint trap, then relaxed as a result of taking
>> a exception in general, so that we may traverse the common trap handling code
>> safely.
>>
>> 3. since XNDBGSTOP is a blocking bit Cobalt-wise, it should prevent
>> @user from being picked for scheduling by the real-time core, next time
>> a Cobalt considers rescheduling that is. However, since @user is
>> currently relaxed, it can still run under the supervision of the common
>> Linux scheduler. This is what the log[1] show.
>>
>> 4. the common/in-band kernel code stops @user due to the ptrace stop
>> condition caused by the breakpoint, waiting for a continuation event to
>> happen.
>>
>> Therefore, upon PTRACE_CONT (i.e. gdb continue), we need to force @user
>> to call back into kernel context (handle_ptrace_cont ->
>> dovetail_request_ucall), then ask for a switch to primary mode from
>> there, which should eventually happen when @user is about to leave the
>> kernel (on x86, this now happens from a generic kernel entry/exit code
>> in kernel/entry/*). As a result, handle_taskexit_event() runs, figures
>
>Not quite:
>s,handle_taskexit_event,handle_user_return,
There is trace_cobalt_shadow_unmap in handle_taskexit_event. I added
trace_cobalt_handle_user_return and trace_cobalt_handle_ptrace_cont
like patch [3] in log [4].
1. In the log [4], the first cobalt_handle_ptrace_cont is followed by cobalt_handle_user_return
that clear info flag XNCONTHI for smokey-2499 because cobalt_handle_ptrace_cont called dovetail_request_ucall .
the TIF_RETUSER bit was cleared for smokey-2499.
After then 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000
clear state flag XNDBGSTOP. After that, thread smokey-2499 run without these bit set till
"hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000".
2. There is second cobalt_handle_ptrace_cont called for smokey-2499 but at that time dovetail_request_ucall
would not be called because XNSSTEP Is not set like "smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0"
3. hi-thread-2504 breakpoint at its first breakpoint_target so it trigger hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3
It call dovetail_request_ucall(current); for itself and also call xnthread_set_info(thread, XNCONTHI); That is why we can see this bit set in
hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200
4. finally ,gdb call sys_tkill in gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13) and trigger that handle_oob_mayday is called so that xnthread_relax for
smokey-2499 Is called in "smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined."
It seems that there never call dovetail_request_ucall for process smokey-2499 in log[4] so that final xhthread_relax would have chance to return to user?
The second handle_ptrace_cont happen too much earlier because XNSSTEP still have not been set so that system never have change to call dovetail_request_ucall
for smokey-2499.
In this case , do we need to do hack in 3 , when handle_oob_trap_entry is called for hi-thread-2504 we call xnthread_set_info(thread, XNCONTHI);
and dovetail_request_ucall for all threads that have same cobalt_process? Is this solution OK for fixing this case?
[3]:
index 07dcc7a63..cd76961f5 100644
--- a/kernel/cobalt/dovetail/kevents.c
+++ b/kernel/cobalt/dovetail/kevents.c
@@ -331,6 +331,8 @@ static int handle_user_return(struct task_struct *task)
if (thread == NULL)
return KEVENT_PROPAGATE;
+ trace_cobalt_handle_user_return(thread);
+
if (xnthread_test_info(thread, XNCONTHI)) {
xnlock_get_irqsave(&nklock, s);
xnthread_clear_info(thread, XNCONTHI);
@@ -468,6 +470,8 @@ int handle_ptrace_resume(struct task_struct *tracee)
if (thread == NULL)
return KEVENT_PROPAGATE;
+ trace_cobalt_handle_ptrace_resume(thread);
+
if (xnthread_test_state(thread, XNSSTEP)) {
xnlock_get_irqsave(&nklock, s);
@@ -485,6 +489,8 @@ static void handle_ptrace_cont(void)
struct xnthread *curr = xnthread_current();
spl_t s;
+ trace_cobalt_handle_ptrace_cont(curr);
+
xnlock_get_irqsave(&nklock, s);
if (xnthread_test_state(curr, XNSSTEP)) {
@@ -604,6 +610,7 @@ void resume_oob_task(struct task_struct *p) /* inband, oob stage stalled */
if (unlikely(xnthread_test_info(thread, XNCONTHI))) {
xnthread_clear_info(thread, XNCONTHI);
dovetail_clear_ucall();
+ trace_cobalt_resume_oob_task(thread);
}
/* Unregister as debugged thread in case we postponed this. */
diff --git a/kernel/cobalt/trace/cobalt-core.h b/kernel/cobalt/trace/cobalt-core.h
index 78f66a106..88a4067fd 100644
--- a/kernel/cobalt/trace/cobalt-core.h
+++ b/kernel/cobalt/trace/cobalt-core.h
@@ -459,6 +459,27 @@ DEFINE_EVENT(curr_thread_event, cobalt_thread_set_mode,
TP_ARGS(thread)
);
+DEFINE_EVENT(thread_event, cobalt_handle_ptrace_cont,
+ TP_PROTO(struct xnthread *thread),
[4]:
gdb-2495 [000] d..3 61513.834825: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] ...1 61513.834827: cobalt_handle_ptrace_cont: pid=2499 state=0x4488c0 info=0x200
smokey-2499 [000] d..1 61513.834828: cobalt_handle_user_return: pid=2499 state=0x4488c0 info=0x200
smokey-2499 [000] d..1 61513.834828: cobalt_shadow_gohard: state=0x4488c0 info=0x0
smokey-2499 [000] d..3 61513.834830: sched_stat_wait: comm=gdb pid=2495 delay=8017 [ns]
smokey-2499 [000] d..3 61513.834831: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] *..3 61513.834831: cobalt_thread_resume: name=smokey pid=2499 mask=0x80
gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1
gdb-2495 [000] *..3 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000
gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1
gdb-2495 [000] *.~3 61513.834833: cobalt_schedule: status=0x10000000
gdb-2495 [000] *.~3 61513.834833: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] *.~3 61513.834833: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1
smokey-2499 [000] d.~3 61513.834835: cobalt_shadow_hardened: state=0x48040 info=0x0
smokey-2499 [000] d.~3 61513.834836: cobalt_head_sysentry: syscall=sem_init
smokey-2499 [000] d.~3 61513.834839: cobalt_psem_init: sem=0x4e(anon) flags=0x0() value=0
smokey-2499 [000] d.~3 61513.834839: cobalt_head_sysexit: result=0
smokey-2499 [000] d.~3 61513.834848: cobalt_shadow_gorelax: reason=syscall
smokey-2499 [000] d.~3 61513.834848: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey
smokey-2499 [000] *.~3 61513.834848: cobalt_thread_suspend: pid=2499 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
smokey-2499 [000] *.~3 61513.834849: cobalt_schedule: status=0x10000000
smokey-2499 [000] *.~3 61513.834849: cobalt_trace_pid: pid=2499, prio=1
smokey-2499 [000] *.~3 61513.834849: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
gdb-2495 [000] *.~3 61513.834850: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] d.h3 61513.834851: irq_handler_entry: irq=120 name=in-band work
gdb-2495 [000] d.h3 61513.834852: cobalt_lostage_wakeup: pid=2499 comm=smokey
gdb-2495 [000] d.h5 61513.834852: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dNh6 61513.834853: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dNh3 61513.834853: irq_handler_exit: irq=120 ret=handled
gdb-2495 [000] dN.3 61513.834854: sched_stat_runtime: comm=gdb pid=2495 runtime=23565 [ns] vruntime=300514015543 [ns]
gdb-2495 [000] d..3 61513.834855: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] ...1 61513.834856: cobalt_shadow_relaxed: state=0x480c0 info=0x0
smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min(policy: 1)
smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min -> 0x1
smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max(policy: 1)
smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max -> 0x63
smokey-2499 [000] ..~1 61513.834862: cobalt_head_sysentry: syscall=thread_getschedparam_ex
smokey-2499 [000] ...1 61513.834862: cobalt_root_sysentry: syscall=thread_getschedparam_ex
smokey-2499 [000] ...1 61513.834863: cobalt_pthread_getschedparam: pth=000000008d36be1c policy=fifo param={ priority=1 }
smokey-2499 [000] ...1 61513.834864: cobalt_root_sysexit: result=0
smokey-2499 [000] ...1 61513.834872: sys_mmap(addr: 0, len: 11000, prot: 0, flags: 20022, fd: ffffffff, off: 0)
smokey-2499 [000] ...1 61513.834877: sys_mmap -> 0x7ffff67b9000
smokey-2499 [000] ...1 61513.834878: sys_mprotect(start: 7ffff67ba000, len: 10000, prot: 3)
smokey-2499 [000] ...1 61513.834906: sys_mprotect -> 0x0
smokey-2499 [000] ...1 61513.834909: sys_clone(clone_flags: 3d0f00, newsp: 7ffff67c8fb0, parent_tidptr: 7ffff67c99d0, child_tidptr: 7ffff67c99d0, tls: 7ffff67c9700)
smokey-2499 [000] ...1 61513.834923: task_newtask: pid=2504 comm=smokey clone_flags=3d0f00 oom_score_adj=0
smokey-2499 [000] ...1 61513.834923: sched_process_fork: comm=smokey pid=2499 child_comm=smokey child_pid=2504
smokey-2499 [000] d..3 61513.834925: sched_wakeup_new: comm=smokey pid=2504 prio=98 target_cpu=000
smokey-2499 [000] d..3 61513.834928: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0
smokey-2499 [000] d..3 61513.834931: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=t ==> next_comm=smokey next_pid=2504 next_prio=98
hi-thread-2504 [000] ...1 61513.834932: sys_clone -> 0x0
hi-thread-2504 [000] d..3 61513.834934: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=2
hi-thread-2504 [000] d..3 61513.834935: sched_stat_wait: comm=gdb pid=2495 delay=82002 [ns]
hi-thread-2504 [000] d..3 61513.834936: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=t ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] d..2 61513.834937: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000
gdb-2495 [000] ...1 61513.834940: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1)
gdb-2495 [000] ...1 61513.834943: sys_read -> 0x1
gdb-2495 [000] ...1 61513.834943: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1)
gdb-2495 [000] ...1 61513.834944: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.834944: sys_write(fd: 10, buf: 5585fe0354b1, count: 1)
gdb-2495 [000] ...1 61513.834946: sys_write -> 0x1
gdb-2495 [000] ...1 61513.834947: sys_rt_sigreturn()
gdb-2495 [000] ...1 61513.834953: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1)
gdb-2495 [000] ...1 61513.834953: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.834955: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0)
gdb-2495 [000] ...1 61513.834956: sys_poll -> 0x1
gdb-2495 [000] ...1 61513.834957: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
gdb-2495 [000] ...1 61513.834958: sys_read -> 0x1
gdb-2495 [000] ...1 61513.834959: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
gdb-2495 [000] ...1 61513.834960: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8)
gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask -> 0x0
gdb-2495 [000] ...1 61513.834962: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
gdb-2495 [000] ...1 61513.834962: sched_process_wait: comm=gdb pid=0 prio=120
gdb-2495 [000] ...1 61513.834964: sys_wait4 -> 0x9c3
gdb-2495 [000] ...1 61513.834965: sys_ptrace(request: 4201, pid: 9c3, addr: 0, data: 7ffe8b2a7d08)
gdb-2495 [000] d..3 61513.834966: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.834967: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.834967: sys_wait4(upid: 9c8, stat_addr: 7ffe8b2a7cfc, options: 40000000, ru: 0)
gdb-2495 [000] ...1 61513.834968: sched_process_wait: comm=gdb pid=2504 prio=120
gdb-2495 [000] ...1 61513.834969: sys_wait4 -> 0x9c8
gdb-2495 [000] ...1 61513.834972: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578)
gdb-2495 [000] d..3 61513.834972: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.834975: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.834976: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a78d8)
gdb-2495 [000] d..3 61513.834977: sched_wait_task: comm=smokey pid=2504 prio=98
gdb-2495 [000] ...1 61513.834977: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.834979: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.834989: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.834990: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff67c9700)
gdb-2495 [000] ...1 61513.834992: sys_pread64 -> 0x900
gdb-2495 [000] ...1 61513.834993: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.834994: sys_close -> 0x0
gdb-2495 [000] ...1 61513.835010: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578)
gdb-2495 [000] d..3 61513.835011: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.835012: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835013: sys_ptrace(request: 3, pid: 9c3, addr: a8, data: 7ffe8b2a78d8)
gdb-2495 [000] d..3 61513.835013: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.835014: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835015: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.835022: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.835022: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff758f740)
gdb-2495 [000] ...1 61513.835025: sys_pread64 -> 0x900
gdb-2495 [000] ...1 61513.835025: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.835026: sys_close -> 0x0
gdb-2495 [000] ...1 61513.835031: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
gdb-2495 [000] ...1 61513.835032: sched_process_wait: comm=gdb pid=0 prio=120
gdb-2495 [000] ...1 61513.835033: sys_wait4 -> 0x0
gdb-2495 [000] ...1 61513.835036: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0)
gdb-2495 [000] d..3 61513.835037: sched_wait_task: comm=smokey pid=2504 prio=98
gdb-2495 [000] ...1 61513.835039: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835043: sys_ptrace(request: 6, pid: 9c8, addr: 388, data: 0)
gdb-2495 [000] d..3 61513.835044: sched_wait_task: comm=smokey pid=2504 prio=98
gdb-2495 [000] ...1 61513.835044: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835045: sys_ptrace(request: 7, pid: 9c8, addr: 1, data: 0)
gdb-2495 [000] d..3 61513.835046: sched_wait_task: comm=smokey pid=2504 prio=98
gdb-2495 [000] d..3 61513.835047: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000
gdb-2495 [000] dN.4 61513.835048: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000
gdb-2495 [000] .N.1 61513.835048: sys_ptrace -> 0x0
gdb-2495 [000] dN.3 61513.835050: sched_stat_runtime: comm=gdb pid=2495 runtime=112561 [ns] vruntime=300514128104 [ns]
gdb-2495 [000] d..3 61513.835051: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=98
hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list(head: 7ffff67c99e0, len: 18)
hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list -> 0x0
hi-thread-2504 [000] ...1 61513.835056: sys_futex(uaddr: 7ffff67c9d18, op: 80, val: 2, utime: 0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700)
hi-thread-2504 [000] d..3 61513.835059: sched_stat_wait: comm=gdb pid=2495 delay=10883 [ns]
hi-thread-2504 [000] d..3 61513.835060: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] ...1 61513.835063: sys_ptrace(request: c, pid: 9c3, addr: 0, data: 7ffe8b2a6ef0)
gdb-2495 [000] d..3 61513.835064: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.835065: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835068: sys_ptrace(request: 7, pid: 9c3, addr: 1, data: 0)
gdb-2495 [000] d..3 61513.835069: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] d..3 61513.835069: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dN.4 61513.835070: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] .N.1 61513.835071: sys_ptrace -> 0x0
gdb-2495 [000] dN.3 61513.835071: sched_stat_runtime: comm=gdb pid=2495 runtime=12210 [ns] vruntime=300514140314 [ns]
gdb-2495 [000] d..3 61513.835072: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0
smokey-2499 [000] ...1 61513.835074: sys_clone -> 0x9c8
smokey-2499 [000] ...1 61513.835075: sys_sched_setaffinity(pid: 9c8, len: 80, user_mask_ptr: 55555563e7c0)
smokey-2499 [000] ...1 61513.835077: sys_sched_setaffinity -> 0x0
smokey-2499 [000] ...1 61513.835077: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 0, val3: 7ffff67c9700)
smokey-2499 [000] d..3 61513.835078: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000
smokey-2499 [000] d..4 61513.835079: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000
smokey-2499 [000] ...1 61513.835080: sys_futex -> 0x1
smokey-2499 [000] ...1 61513.835086: sys_futex(uaddr: 7fffffffe828, op: 189, val: 0, utime: 7fffffffe780, uaddr2: 0, val3: ffffffff)
smokey-2499 [000] d..3 61513.835089: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=smokey next_pid=2504 next_prio=98
hi-thread-2504 [000] ...1 61513.835090: sys_futex -> 0x0
hi-thread-2504 [000] ...1 61513.835090: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700)
hi-thread-2504 [000] ...1 61513.835091: sys_futex -> 0x0
hi-thread-2504 [000] ...1 61513.835092: sys_sched_setscheduler(pid: 9c8, policy: 1, param: 7ffff67c8e70)
hi-thread-2504 [000] ...1 61513.835095: sys_sched_setscheduler -> 0x0
hi-thread-2504 [000] ..~1 61513.835096: cobalt_head_sysentry: syscall=thread_create
hi-thread-2504 [000] ...1 61513.835097: cobalt_root_sysentry: syscall=thread_create
hi-thread-2504 [000] ...1 61513.835098: cobalt_pthread_create: pth=00000000ba35a01d policy=fifo param={ priority=2 }
hi-thread-2504 [000] ...1 61513.835099: cobalt_timer_migrate: timer=000000004d21df54 cpu=0
hi-thread-2504 [000] ...1 61513.835100: cobalt_timer_migrate: timer=00000000c1301853 cpu=0
hi-thread-2504 [000] ...1 61513.835101: cobalt_thread_set_current_prio: thread=00000000e9fda3c4 pid=-1 prio=2
hi-thread-2504 [000] ...1 61513.835102: cobalt_thread_init: thread=00000000e9fda3c4 name=smokey flags=0x48000 class=rt prio=2
hi-thread-2504 [000] ...1 61513.835104: cobalt_shadow_map: thread=00000000e9fda3c4 pid=2504 prio=2
hi-thread-2504 [000] D..1 61513.835104: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_start: pid=2504 state=0x480d0 info=0x0
hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_resume: name=smokey pid=2504 mask=0x10
hi-thread-2504 [000] D..1 61513.835106: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] ...1 61513.835106: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] ...1 61513.835107: cobalt_shadow_gohard: state=0x480c0 info=0x0
hi-thread-2504 [000] d..3 61513.835108: sched_stat_wait: comm=gdb pid=2495 delay=37231 [ns]
hi-thread-2504 [000] d..3 61513.835109: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] *..3 61513.835109: cobalt_thread_resume: name=smokey pid=2504 mask=0x80
gdb-2495 [000] *..3 61513.835110: cobalt_trace_pid: pid=2504, prio=2
gdb-2495 [000] *.~3 61513.835110: cobalt_schedule: status=0x10000000
gdb-2495 [000] *.~3 61513.835110: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] *.~3 61513.835111: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2
hi-thread-2504 [000] d.~3 61513.835112: cobalt_shadow_hardened: state=0x48040 info=0x0
hi-thread-2504 [000] d.~3 61513.835112: cobalt_root_sysexit: result=0
hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysentry: syscall=get_current
hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysexit: result=0
hi-thread-2504 [000] d.~3 61513.835116: cobalt_shadow_gorelax: reason=syscall
hi-thread-2504 [000] d.~3 61513.835117: cobalt_lostage_request: request=wakeup pid=2504 comm=smokey
hi-thread-2504 [000] *.~3 61513.835117: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] *.~3 61513.835117: cobalt_schedule: status=0x10000000
hi-thread-2504 [000] *.~3 61513.835118: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] *.~3 61513.835118: cobalt_switch_context: prev_name=smokey prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
gdb-2495 [000] *.~3 61513.835119: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] d.h3 61513.835120: irq_handler_entry: irq=120 name=in-band work
gdb-2495 [000] d.h3 61513.835120: cobalt_lostage_wakeup: pid=2504 comm=smokey
gdb-2495 [000] d.h5 61513.835120: sched_waking: comm=smokey pid=2504 prio=97 target_cpu=000
gdb-2495 [000] dNh6 61513.835121: sched_wakeup: comm=smokey pid=2504 prio=97 target_cpu=000
gdb-2495 [000] dNh3 61513.835122: irq_handler_exit: irq=120 ret=handled
gdb-2495 [000] dN.3 61513.835122: sched_stat_runtime: comm=gdb pid=2495 runtime=13441 [ns] vruntime=300514153755 [ns]
gdb-2495 [000] d..3 61513.835123: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=97
hi-thread-2504 [000] ...1 61513.835124: cobalt_shadow_relaxed: state=0x480c0 info=0x0
hi-thread-2504 [000] ...1 61513.835125: sys_futex(uaddr: 7fffffffe828, op: 81, val: 1, utime: 0, uaddr2: 7ffff67c8e74, val3: 7fffffffe828)
hi-thread-2504 [000] d..3 61513.835126: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
hi-thread-2504 [000] d..4 61513.835126: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
hi-thread-2504 [000] ...1 61513.835127: sys_futex -> 0x1
hi-thread-2504 [000] ..~1 61513.835128: cobalt_head_sysentry: syscall=migrate
hi-thread-2504 [000] ...1 61513.835128: cobalt_root_sysentry: syscall=migrate
hi-thread-2504 [000] ...1 61513.835128: cobalt_shadow_gohard: state=0x480c0 info=0x0
hi-thread-2504 [000] d..3 61513.835130: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] *..3 61513.835130: cobalt_thread_resume: name=smokey pid=2504 mask=0x80
smokey-2499 [000] *..3 61513.835130: cobalt_trace_pid: pid=2504, prio=2
smokey-2499 [000] *.~3 61513.835131: cobalt_schedule: status=0x10000000
smokey-2499 [000] *.~3 61513.835131: cobalt_trace_pid: pid=2499, prio=-1
smokey-2499 [000] *.~3 61513.835131: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2
hi-thread-2504 [000] d.~3 61513.835132: cobalt_shadow_hardened: state=0x48040 info=0x0
hi-thread-2504 [000] d.~3 61513.835132: cobalt_root_sysexit: result=1
hi-thread-2504 [000] d.~3 61513.835133: cobalt_head_sysentry: syscall=thread_setname
hi-thread-2504 [000] d.~3 61513.835134: cobalt_pthread_setname: pth=00000000ba35a01d name=hi-thread
hi-thread-2504 [000] d.~3 61513.835135: cobalt_head_sysexit: result=0
hi-thread-2504 [000] d.~3 61513.835136: cobalt_head_sysentry: syscall=clock_nanosleep
hi-thread-2504 [000] d.~3 61513.835137: cobalt_clock_nanosleep: clock_id=0 flags=0x0() rqt=(0.001000000)
hi-thread-2504 [000] *.~3 61513.835138: cobalt_thread_suspend: pid=2504 mask=0x4 timeout=1000001 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] *.~3 61513.835138: cobalt_timer_start: timer=000000004d21df54(smokey) value=1000001 interval=0 mode=rel
hi-thread-2504 [000] *.~3 61513.835138: cobalt_schedule: status=0x10000000
hi-thread-2504 [000] *.~3 61513.835139: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] *.~3 61513.835139: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x48044 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
smokey-2499 [000] *.~3 61513.835140: cobalt_trace_pid: pid=2499, prio=-1
smokey-2499 [000] ...1 61513.835141: sys_futex -> 0x0
smokey-2499 [000] ..~1 61513.835144: cobalt_head_sysentry: syscall=migrate
smokey-2499 [000] ...1 61513.835144: cobalt_root_sysentry: syscall=migrate
smokey-2499 [000] ...1 61513.835144: cobalt_shadow_gohard: state=0x480c0 info=0x0
smokey-2499 [000] d..3 61513.835145: sched_stat_wait: comm=gdb pid=2495 delay=23982 [ns]
smokey-2499 [000] d..3 61513.835146: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] *..3 61513.835147: cobalt_thread_resume: name=smokey pid=2499 mask=0x80
gdb-2495 [000] *..3 61513.835147: cobalt_trace_pid: pid=2499, prio=1
gdb-2495 [000] *.~3 61513.835147: cobalt_schedule: status=0x10000000
gdb-2495 [000] *.~3 61513.835147: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] *.~3 61513.835148: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1
smokey-2499 [000] d.~3 61513.835148: cobalt_shadow_hardened: state=0x48040 info=0x0
smokey-2499 [000] d.~3 61513.835149: cobalt_root_sysexit: result=1
smokey-2499 [000] *.~3 61513.835219: irq_pipeline_entry: irq=4354
smokey-2499 [000] *.~3 61513.835219: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
smokey-2499 [000] *.~3 61513.835220: cobalt_timer_expire: timer=00000000d13b8aa9
smokey-2499 [000] *.~3 61513.835220: cobalt_tick_shot: next tick at 61513.836136 (delay: 915 us)
smokey-2499 [000] *.~3 61513.835221: irq_handler_exit: irq=4354 ret=handled
smokey-2499 [000] *.~3 61513.835221: irq_pipeline_exit: irq=4354
smokey-2499 [000] *.~3 61513.836137: irq_pipeline_entry: irq=4354
smokey-2499 [000] *.~3 61513.836137: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
smokey-2499 [000] *.~3 61513.836138: cobalt_timer_expire: timer=000000004d21df54
smokey-2499 [000] *.~3 61513.836138: cobalt_thread_resume: name=hi-thread pid=2504 mask=0x4
smokey-2499 [000] *.~3 61513.836138: cobalt_trace_pid: pid=2504, prio=2
smokey-2499 [000] *.~3 61513.836139: cobalt_timer_stop: timer=000000004d21df54
smokey-2499 [000] *.~3 61513.836139: irq_handler_exit: irq=4354 ret=handled
smokey-2499 [000] *.~3 61513.836139: irq_pipeline_exit: irq=4354
smokey-2499 [000] *.~3 61513.836140: cobalt_schedule: status=0x10000000
smokey-2499 [000] *.~3 61513.836140: cobalt_trace_pid: pid=2499, prio=1
smokey-2499 [000] *.~3 61513.836140: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x48048 ==> next_name=hi-thread next_pid=2504 next_prio=2
hi-thread-2504 [000] *.~3 61513.836141: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] d.~3 61513.836141: cobalt_head_sysexit: result=0
hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3
hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] d.~3 61513.836143: cobalt_shadow_gorelax: reason=fault
hi-thread-2504 [000] d.~3 61513.836143: cobalt_lostage_request: request=wakeup pid=2504 comm=hi-thread
hi-thread-2504 [000] *.~3 61513.836144: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] *.~3 61513.836144: cobalt_schedule: status=0x10000000
hi-thread-2504 [000] *.~3 61513.836144: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] *.~3 61513.836145: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
gdb-2495 [000] *.~3 61513.836146: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] d.h3 61513.836146: irq_handler_entry: irq=120 name=in-band work
gdb-2495 [000] d.h3 61513.836147: cobalt_lostage_wakeup: pid=2504 comm=hi-thread
gdb-2495 [000] d.h5 61513.836147: sched_waking: comm=hi-thread pid=2504 prio=97 target_cpu=000
gdb-2495 [000] dNh6 61513.836147: sched_wakeup: comm=hi-thread pid=2504 prio=97 target_cpu=000
gdb-2495 [000] dNh3 61513.836148: irq_handler_exit: irq=120 ret=handled
gdb-2495 [000] dNh3 61513.836148: irq_handler_entry: irq=124 name=proxy tick
gdb-2495 [000] dNh3 61513.836151: softirq_raise: vec=1 [action=TIMER]
gdb-2495 [000] dNh3 61513.836152: softirq_raise: vec=9 [action=RCU]
gdb-2495 [000] dNh4 61513.836152: sched_stat_runtime: comm=gdb pid=2495 runtime=1007474 [ns] vruntime=300515161229 [ns]
gdb-2495 [000] *Nh3 61513.836154: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2395843 interval=0 mode=rel
gdb-2495 [000] *Nh3 61513.836155: cobalt_tick_shot: next tick at 61513.838550 (delay: 2395 us)
gdb-2495 [000] dNh3 61513.836155: irq_handler_exit: irq=124 ret=handled
gdb-2495 [000] .Ns3 61513.836156: softirq_entry: vec=1 [action=TIMER]
gdb-2495 [000] .Ns3 61513.836157: softirq_exit: vec=1 [action=TIMER]
gdb-2495 [000] .Ns3 61513.836157: softirq_entry: vec=9 [action=RCU]
gdb-2495 [000] dNs6 61513.836158: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000
gdb-2495 [000] dNs7 61513.836159: sched_stat_runtime: comm=gdb pid=2495 runtime=6881 [ns] vruntime=300515168110 [ns]
gdb-2495 [000] dNs7 61513.836160: sched_stat_blocked: comm=rcu_sched pid=11 delay=4188739 [ns]
gdb-2495 [000] dNs7 61513.836161: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000
gdb-2495 [000] .Ns3 61513.836164: softirq_exit: vec=9 [action=RCU]
gdb-2495 [000] d..3 61513.836165: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=hi-thread next_pid=2504 next_prio=97
hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200
hi-thread-2504 [000] *..2 61513.836169: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] *..2 61513.836170: cobalt_thread_suspend: pid=2504 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] d..2 61513.836170: signal_generate: sig=5 errno=0 code=128 comm=hi-thread pid=2504 grp=0 res=0
hi-thread-2504 [000] d..3 61513.836173: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0
hi-thread-2504 [000] d..3 61513.836175: sched_stat_wait: comm=rcu_sched pid=11 delay=14490 [ns]
hi-thread-2504 [000] d..3 61513.836175: sched_switch: prev_comm=hi-thread prev_pid=2504 prev_prio=97 prev_state=t ==> next_comm=rcu_sched next_pid=11 next_prio=120
rcu_sched-11 [000] d..3 61513.836179: sched_stat_runtime: comm=rcu_sched pid=11 runtime=4960 [ns] vruntime=300512173070 [ns]
rcu_sched-11 [000] d..3 61513.836179: sched_stat_wait: comm=gdb pid=2495 delay=19450 [ns]
rcu_sched-11 [000] d..3 61513.836180: sched_switch: prev_comm=rcu_sched prev_pid=11 prev_prio=120 prev_state=I ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] ...1 61513.836187: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cb0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836197: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836198: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7ae0)
gdb-2495 [000] ...1 61513.836199: sys_newfstat -> 0x0
gdb-2495 [000] ...1 61513.836200: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
gdb-2495 [000] ...1 61513.836214: sys_read -> 0x3f1
gdb-2495 [000] ...1 61513.836217: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836217: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836221: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8)
gdb-2495 [000] ...1 61513.836222: sys_rt_sigprocmask -> 0x0
gdb-2495 [000] d..2 61513.836223: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000
gdb-2495 [000] ...1 61513.836225: sys_read(fd: f, buf: 7ffe8b2a7667, count: 1)
gdb-2495 [000] ...1 61513.836226: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.836227: sys_write(fd: 10, buf: 5585fe0354b1, count: 1)
gdb-2495 [000] ...1 61513.836229: sys_write -> 0x1
gdb-2495 [000] ...1 61513.836229: sys_rt_sigreturn()
gdb-2495 [000] ...1 61513.836233: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1)
gdb-2495 [000] ...1 61513.836234: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.836235: sys_write(fd: 1, buf: 5585fe7a5290, count: 27)
gdb-2495 [000] d..4 61513.836236: sched_waking: comm=smokey pid=2493 prio=120 target_cpu=000
gdb-2495 [000] d..5 61513.836237: sched_stat_runtime: comm=gdb pid=2495 runtime=58399 [ns] vruntime=300515226509 [ns]
gdb-2495 [000] d..5 61513.836238: sched_stat_sleep: comm=smokey pid=2493 delay=1717179 [ns]
gdb-2495 [000] dN.5 61513.836239: sched_wakeup: comm=smokey pid=2493 prio=120 target_cpu=000
gdb-2495 [000] .N.1 61513.836240: sys_write -> 0x27
gdb-2495 [000] dN.3 61513.836241: sched_stat_wait: comm=smokey pid=2493 delay=0 [ns]
gdb-2495 [000] d..3 61513.836241: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2493 next_prio=120
smokey-2493 [000] ...1 61513.836244: sys_read -> 0x1
smokey-2493 [000] ..~1 61513.836246: cobalt_head_sysentry: syscall=read
...
smokey-2493 [000] d..3 61513.836344: sched_stat_runtime: comm=smokey pid=2493 runtime=106636 [ns] vruntime=300512333145 [ns]
smokey-2493 [000] d..3 61513.836344: sched_stat_wait: comm=gdb pid=2495 delay=106636 [ns]
smokey-2493 [000] d..3 61513.836345: sched_switch: prev_comm=smokey prev_pid=2493 prev_prio=120 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] ...1 61513.836347: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0)
gdb-2495 [000] ...1 61513.836348: sys_poll -> 0x1
gdb-2495 [000] ...1 61513.836350: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
gdb-2495 [000] ...1 61513.836351: sys_read -> 0x1
gdb-2495 [000] ...1 61513.836352: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
gdb-2495 [000] ...1 61513.836352: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.836353: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8)
gdb-2495 [000] ...1 61513.836354: sys_rt_sigprocmask -> 0x0
gdb-2495 [000] ...1 61513.836355: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
gdb-2495 [000] ...1 61513.836355: sched_process_wait: comm=gdb pid=0 prio=120
gdb-2495 [000] ...1 61513.836357: sys_wait4 -> 0x9c8
gdb-2495 [000] ...1 61513.836360: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0)
gdb-2495 [000] d..3 61513.836361: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836363: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836367: sys_ptrace(request: 4202, pid: 9c8, addr: 0, data: 7ffe8b2a7cb0)
gdb-2495 [000] d..3 61513.836368: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836369: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836370: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6f10)
gdb-2495 [000] d..3 61513.836370: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836371: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836372: sys_ptrace(request: d, pid: 9c8, addr: 0, data: 7ffe8b2a6f10)
gdb-2495 [000] d..3 61513.836373: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836374: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836376: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
gdb-2495 [000] ...1 61513.836376: sched_process_wait: comm=gdb pid=0 prio=120
gdb-2495 [000] ...1 61513.836377: sys_wait4 -> 0x0
gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8)
gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask -> 0x0
gdb-2495 [000] ...1 61513.836382: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cd0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836399: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836400: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7b50)
gdb-2495 [000] ...1 61513.836401: sys_newfstat -> 0x0
gdb-2495 [000] ...1 61513.836402: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
gdb-2495 [000] ...1 61513.836409: sys_read -> 0x140
gdb-2495 [000] ...1 61513.836410: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400)
gdb-2495 [000] ...1 61513.836410: sys_read -> 0x0
gdb-2495 [000] ...1 61513.836415: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836416: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836419: sys_read(fd: f, buf: 7ffe8b2a7f17, count: 1)
gdb-2495 [000] ...1 61513.836420: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.836421: sys_write(fd: 10, buf: 5585fe0354b1, count: 1)
gdb-2495 [000] ...1 61513.836422: sys_write -> 0x1
gdb-2495 [000] ...1 61513.836425: sys_ptrace(request: 1, pid: 9c8, addr: 7ffff777f370, data: 7ffe8b2a7878)
gdb-2495 [000] d..3 61513.836426: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836428: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836430: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a7bd8)
gdb-2495 [000] d..3 61513.836430: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836431: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836433: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a70e0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836446: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836446: sys_pread64(fd: 12, buf: 7ffe8b2a7430, count: 900, pos: 7ffff67c9700)
gdb-2495 [000] ...1 61513.836449: sys_pread64 -> 0x900
gdb-2495 [000] ...1 61513.836450: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836450: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836479: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836488: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836489: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950)
gdb-2495 [000] ...1 61513.836490: sys_newfstat -> 0x0
gdb-2495 [000] ...1 61513.836490: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400)
gdb-2495 [000] ...1 61513.836498: sys_read -> 0x14d
gdb-2495 [000] ...1 61513.836499: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
gdb-2495 [000] ...1 61513.836499: sys_read -> 0x0
gdb-2495 [000] ...1 61513.836503: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836504: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836507: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836514: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836515: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950)
gdb-2495 [000] ...1 61513.836515: sys_newfstat -> 0x0
gdb-2495 [000] ...1 61513.836516: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
gdb-2495 [000] ...1 61513.836521: sys_read -> 0x13b
gdb-2495 [000] ...1 61513.836522: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400)
gdb-2495 [000] ...1 61513.836522: sys_read -> 0x0
gdb-2495 [000] ...1 61513.836525: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836525: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13)
gdb-2495 [000] *..3 61513.836532: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000
gdb-2495 [000] *..3 61513.836532: cobalt_trace_pid: pid=2499, prio=1
gdb-2495 [000] *..3 61513.836533: cobalt_thread_unblock: pid=2499 state=0x48848 info=0x0
gdb-2495 [000] *.~3 61513.836586: cobalt_schedule: status=0x10000000
gdb-2495 [000] *.~3 61513.836586: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] *.~3 61513.836587: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1
smokey-2499 [000] *.~3 61513.836589: cobalt_trace_pid: pid=2499, prio=1
smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined
smokey-2499 [000] *.~3 61513.836589: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey
smokey-2499 [000] *.~3 61513.836590: cobalt_thread_suspend: pid=2499 mask=0x400080 timeout=0 timeout_mode=0 wchan=0000000000000000
smokey-2499 [000] *.~3 61513.836590: cobalt_schedule: status=0x10000000
smokey-2499 [000] *.~3 61513.836590: cobalt_trace_pid: pid=2499, prio=1
smokey-2499 [000] *.~3 61513.836591: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x4488c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
gdb-2495 [000] *.~3 61513.836592: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] d..5 61513.836592: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dN.6 61513.836594: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dN.3 61513.836594: signal_generate: sig=19 errno=0 code=-6 comm=smokey pid=2499 grp=0 res=0
gdb-2495 [000] dNh3 61513.836595: irq_handler_entry: irq=120 name=in-band work
gdb-2495 [000] dNh3 61513.836596: cobalt_lostage_wakeup: pid=2499 comm=smokey
gdb-2495 [000] *N~4 61513.838552: irq_pipeline_entry: irq=4354
gdb-2495 [000] *N~4 61513.838553: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
gdb-2495 [000] *N~4 61513.838554: cobalt_timer_expire: timer=00000000d13b8aa9
gdb-2495 [000] *N~4 61513.838555: irq_handler_exit: irq=4354 ret=handled
gdb-2495 [000] *N~4 61513.838555: irq_pipeline_exit: irq=4354
gdb-2495 [000] dNh8 61513.981277: sched_waking: comm=systemd-journal pid=163 prio=120 target_cpu=000
gdb-2495 [000] dNh9 61513.981278: sched_stat_runtime: comm=gdb pid=2495 runtime=144934239 [ns] vruntime=300660160748 [ns]
gdb-2495 [000] dNh9 61513.981280: sched_stat_sleep: comm=systemd-journal pid=163 delay=111416392895 [ns]
gdb-2495 [000] dNh9 61513.981280: sched_wakeup: comm=systemd-journal pid=163 prio=120 target_cpu=000
gdb-2495 [000] dNh3 61513.981281: irq_handler_exit: irq=120 ret=handled
gdb-2495 [000] dNh3 61513.981282: irq_handler_entry: irq=124 name=proxy tick
gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=1 [action=TIMER]
gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=9 [action=RCU]
gdb-2495 [000] dNh4 61513.981286: sched_stat_runtime: comm=gdb pid=2495 runtime=8375 [ns] vruntime=300660169123 [ns]
gdb-2495 [000] dNh5 61513.981288: sched_waking: comm=cobalt_printf pid=2494 prio=120 target_cpu=000
gdb-2495 [000] dNh6 61513.981289: sched_stat_runtime: comm=gdb pid=2495 runtime=2987 [ns] vruntime=300660172110 [ns]
gdb-2495 [000] dNh6 61513.981291: sched_stat_sleep: comm=cobalt_printf pid=2494 delay=238660319 [ns]
gdb-2495 [000] dNh6 61513.981291: sched_wakeup: comm=cobalt_printf pid=2494 prio=120 target_cpu=000
gdb-2495 [000] dNh5 61513.981292: sched_waking: comm=cobalt_printf pid=2503 prio=120 target_cpu=000
gdb-2495 [000] dNh6 61513.981292: sched_stat_runtime: comm=gdb pid=2495 runtime=3069 [ns] vruntime=300660175179 [ns]
gdb-2495 [000] dNh6 61513.981294: sched_stat_sleep: comm=cobalt_printf pid=2503 delay=146611442 [ns]
gdb-2495 [000] dNh6 61513.981294: sched_wakeup: comm=cobalt_printf pid=2503 prio=120 target_cpu=000
gdb-2495 [000] *Nh3 61513.981295: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=588476 interval=0 mode=rel
gdb-2495 [000] *Nh3 61513.981296: cobalt_tick_shot: next tick at 61513.981884 (delay: 588 us)
gdb-2495 [000] dNh3 61513.981296: irq_handler_exit: irq=124 ret=handled
gdb-2495 [000] .Ns3 61513.981297: softirq_entry: vec=1 [action=TIMER]
gdb-2495 [000] dNs5 61513.981298: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000
gdb-2495 [000] dNs6 61513.981298: sched_stat_runtime: comm=gdb pid=2495 runtime=6218 [ns] vruntime=300660181397 [ns]
gdb-2495 [000] dNs6 61513.981299: sched_stat_blocked: comm=rcu_sched pid=11 delay=145119923 [ns]
gdb-2495 [000] dNs6 61513.981299: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000
gdb-2495 [000] dNs7 61513.981302: sched_waking: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000
gdb-2495 [000] dNs8 61513.981302: sched_stat_runtime: comm=gdb pid=2495 runtime=3884 [ns] vruntime=300660185281 [ns]
gdb-2495 [000] dNs8 61513.981303: sched_stat_blocked: comm=kworker/0:1 pid=1932 delay=206048652 [ns]
gdb-2495 [000] dNs8 61513.981304: sched_wakeup: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000
gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981312: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] .Ns3 61513.981314: softirq_exit: vec=1 [action=TIMER]
gdb-2495 [000] .Ns3 61513.981314: softirq_entry: vec=9 [action=RCU]
gdb-2495 [000] .Ns3 61513.981315: softirq_exit: vec=9 [action=RCU]
gdb-2495 [000] dNs5 61513.981316: sched_waking: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000
gdb-2495 [000] dNs6 61513.981316: sched_stat_runtime: comm=gdb pid=2495 runtime=13841 [ns] vruntime=300660199122 [ns]
gdb-2495 [000] dNs6 61513.981317: sched_stat_sleep: comm=ksoftirqd/0 pid=10 delay=149349034 [ns]
gdb-2495 [000] dNs6 61513.981317: sched_wakeup: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000
gdb-2495 [000] .N.1 61513.981318: sys_tkill -> 0x0
gdb-2495 [000] d..3 61513.981320: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] *.~3 61513.981888: irq_pipeline_entry: irq=4354
smokey-2499 [000] *.~3 61513.981888: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
smokey-2499 [000] *.~3 61513.981889: cobalt_timer_expire: timer=00000000d13b8aa9
smokey-2499 [000] *.~3 61513.981889: irq_handler_exit: irq=4354 ret=handled
smokey-2499 [000] *.~3 61513.981890: irq_pipeline_exit: irq=4354
smokey-2499 [000] d.h1 61514.106435: irq_handler_entry: irq=124 name=proxy tick
smokey-2499 [000] *.h1 61514.106441: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2109534 interval=0 mode=rel
smokey-2499 [000] *.h1 61514.106441: cobalt_tick_shot: next tick at 61514.108550 (delay: 2109 us)
smokey-2499 [000] d.h1 61514.106442: irq_handler_exit: irq=124 ret=handled
smokey-2499 [000] ...1 61514.106442: cobalt_shadow_relaxed: state=0x4488c0 info=0x200
smokey-2499 [000] *.~1 61514.108552: irq_pipeline_entry: irq=4354
Regards
Hongzhan Chen
>
>--
>Philippe.
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: gdb test failure debug status update
2021-04-30 5:25 ` Chen, Hongzhan
2021-04-30 5:51 ` Chen, Hongzhan
@ 2021-04-30 7:34 ` Philippe Gerum
2021-04-30 8:00 ` Philippe Gerum
1 sibling, 1 reply; 16+ messages in thread
From: Philippe Gerum @ 2021-04-30 7:34 UTC (permalink / raw)
To: Chen, Hongzhan; +Cc: xenomai
Chen, Hongzhan <hongzhan.chen@intel.com> writes:
> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish
> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax.
> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in
> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from
> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to
> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode
> because kernel exit code is never called in this case?
>
The IRQ frame is indeed kept from unwinding until the preempted task
context returns from irq_exit_pipeline(), which branches to the Cobalt
rescheduling procedure. From the Dovetail interface POV,
irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow
the companion core to perform post-IRQ chores, such as running its own
rescheduling procedure.
IOW, if task @foo is preempted by an IRQ, then suspended in oob context
as a result of what the interrupt handler just did for it (e.g. raising
XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then
handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will
cause the @foo context to switch away, effectively preventing
handle_irq_pipelined_finish() to return, until @foo resumes execution
eventually.
The underlying issue I'd like to understand, is why do we have a mayday
condition raised in the context of this test in the first place? That
may be the root of the issue, the mayday and synchronous-bp handling
code may not cope that well.
So, question to address first: who ends up calling
dovetail_call_mayday(), detecting that the underlying task went south?
And why that?
--
Philippe.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: gdb test failure debug status update
2021-04-30 5:51 ` Chen, Hongzhan
@ 2021-04-30 7:36 ` Philippe Gerum
0 siblings, 0 replies; 16+ messages in thread
From: Philippe Gerum @ 2021-04-30 7:36 UTC (permalink / raw)
To: Chen, Hongzhan; +Cc: xenomai
Chen, Hongzhan <hongzhan.chen@intel.com> writes:
> After applied following patch , the gdb test failure issue seems fixed. Basically , it enforce going back to
> primary mode after check that XNCONTHI is set for current thread. It seems ugly. Please suggest different
> solution.
Agreed, this is ugly. This code is switching the current task back to
oob mode right after it forced it to transition from oob to in-band
mode. I'm not sure why that works in the end, we need to explain the
mayday issue first in order to figure out.
>
> @@ -355,9 +357,36 @@ static int handle_user_return(struct task_struct *task)
>
> void handle_oob_mayday(struct pt_regs *regs)
> {
> + struct xnthread *thread;
> + spl_t s;
> + int err;
> +
> XENO_BUG_ON(COBALT, !xnthread_test_state(xnthread_current(), XNUSER));
>
> + thread = xnthread_current();
> +
> + trace_cobalt_handle_oob_mayday(thread);
> +
> xnthread_relax(0, 0);
> +
> + if (xnthread_test_info(thread, XNCONTHI)) {
> + xnlock_get_irqsave(&nklock, s);
> + xnthread_clear_info(thread, XNCONTHI);
> + xnlock_put_irqrestore(&nklock, s);
> +
> + err = xnthread_harden();
> +
> + /*
> + * XNCONTHI may or may not have been re-applied if
> + * harden bailed out due to pending signals. Make sure
> + * it is set in that case.
> + */
> + if (err == -ERESTARTSYS) {
> + xnlock_get_irqsave(&nklock, s);
> + xnthread_set_info(thread, XNCONTHI);
> + xnlock_put_irqrestore(&nklock, s);
> + }
> + }
> }
>
>
> Regards
>
> Hongzhan Chen
>
> -----Original Message-----
> From: Chen, Hongzhan <hongzhan.chen@intel.com>
> Sent: Friday, April 30, 2021 1:26 PM
> To: Chen, Hongzhan <hongzhan.chen@intel.com>; Philippe Gerum <rpm@xenomai.org>; xenomai@xenomai.org
> Subject: RE: gdb test failure debug status update
>
> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish
> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax.
> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in
> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from
> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to
> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode
> because kernel exit code is never called in this case?
>
> Regards
>
> Hongzhan Chen
>
> -----Original Message-----
> From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai
> Sent: Thursday, April 29, 2021 2:31 PM
> To: Philippe Gerum <rpm@xenomai.org>
> Cc: xenomai@xenomai.org
> Subject: RE: gdb test failure debug status update
>
>
>>
>>-----Original Message-----
>>From: Philippe Gerum <rpm@xenomai.org>
>>Sent: Wednesday, April 28, 2021 10:30 PM
>>To: Philippe Gerum <rpm@xenomai.org>
>>Cc: Chen, Hongzhan <hongzhan.chen@intel.com>; xenomai@xenomai.org
>>Subject: Re: gdb test failure debug status update
>>
>>
>>Philippe Gerum via Xenomai <xenomai@xenomai.org> writes:
>>
>>> Chen, Hongzhan via Xenomai <xenomai@xenomai.org> writes:
>>>
>>>> According to my validation, gdb test fail on dovetail 5.10 branch but pass on v5.9-evl4 tag with same for-upstream/dovetail
>>>> xenomai code base.
>>>>
>>>> After further debug , the issue is more clear for me. Gdb test failure because low priority thread smokey userspace is still
>>>> executed after "cobalt_shadow_relaxed: state=0x4488c0 info=0x200" like log [1] on dovetail-5.10 branch.
>>>> The weird thing is that its following first ftrace log happen at 62235.848583 after cobalt_shadow_relaxed in log [1].
>>>> It is almost 3ms happened after cobalt_shadow_relaxed. The low priority smoke thread user space is executed during this
>>>> 3ms period so that test fail.
>>>>
>>>> But in success case with v5.9-evl4 like in log [2], the time interval between cobalt_shadow_relaxed and the following first ftrace log
>>>> is only about 1us. It seems that low priority smokey userspace do not have chance to execute in this 1us because gdb test is successful.
>>>>
>>>> My question is why there is even no interrupt happened during that about 3ms period in failure case? Tick seems in abnormal behavior.
>>>> Please comment if you have any ideas to further debug it.
>>>>
>>>> PS: All my tests run on same up Xtream board.
>>>
>>> <snip>
>>>
>>> Let's put aside the tick issue for now, there may be a valid reason for
>>> this delay with dynticks enabled.
>>>
>>> The issue at stake may be related to the way a return to kernel space is
>>> forced on a @user task (Dovetail has an integrated service for
>>> triggering this called dovetail_request_ucall()).
>>>
>>> The logic for doing so is as follows:
>>>
>>> 1. @user hits a breakpoint, which is an exception Dovetail-wise
>>>
>>> 2. @user gets XNDBGSTOP set into its flags because Cobalt notices it is
>>> being debugged via a breakpoint trap, then relaxed as a result of taking
>>> a exception in general, so that we may traverse the common trap handling code
>>> safely.
>>>
>>> 3. since XNDBGSTOP is a blocking bit Cobalt-wise, it should prevent
>>> @user from being picked for scheduling by the real-time core, next time
>>> a Cobalt considers rescheduling that is. However, since @user is
>>> currently relaxed, it can still run under the supervision of the common
>>> Linux scheduler. This is what the log[1] show.
>>>
>>> 4. the common/in-band kernel code stops @user due to the ptrace stop
>>> condition caused by the breakpoint, waiting for a continuation event to
>>> happen.
>>>
>>> Therefore, upon PTRACE_CONT (i.e. gdb continue), we need to force @user
>>> to call back into kernel context (handle_ptrace_cont ->
>>> dovetail_request_ucall), then ask for a switch to primary mode from
>>> there, which should eventually happen when @user is about to leave the
>>> kernel (on x86, this now happens from a generic kernel entry/exit code
>>> in kernel/entry/*). As a result, handle_taskexit_event() runs, figures
>>
>>Not quite:
>>s,handle_taskexit_event,handle_user_return,
>
> There is trace_cobalt_shadow_unmap in handle_taskexit_event. I added
> trace_cobalt_handle_user_return and trace_cobalt_handle_ptrace_cont
> like patch [3] in log [4].
>
> 1. In the log [4], the first cobalt_handle_ptrace_cont is followed by cobalt_handle_user_return
> that clear info flag XNCONTHI for smokey-2499 because cobalt_handle_ptrace_cont called dovetail_request_ucall .
> the TIF_RETUSER bit was cleared for smokey-2499.
> After then 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000
> clear state flag XNDBGSTOP. After that, thread smokey-2499 run without these bit set till
> "hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000".
>
> 2. There is second cobalt_handle_ptrace_cont called for smokey-2499 but at that time dovetail_request_ucall
> would not be called because XNSSTEP Is not set like "smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0"
>
> 3. hi-thread-2504 breakpoint at its first breakpoint_target so it trigger hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3
> It call dovetail_request_ucall(current); for itself and also call xnthread_set_info(thread, XNCONTHI); That is why we can see this bit set in
> hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200
>
> 4. finally ,gdb call sys_tkill in gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13) and trigger that handle_oob_mayday is called so that xnthread_relax for
> smokey-2499 Is called in "smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined."
>
>
> It seems that there never call dovetail_request_ucall for process smokey-2499 in log[4] so that final xhthread_relax would have chance to return to user?
> The second handle_ptrace_cont happen too much earlier because XNSSTEP still have not been set so that system never have change to call dovetail_request_ucall
> for smokey-2499.
> In this case , do we need to do hack in 3 , when handle_oob_trap_entry is called for hi-thread-2504 we call xnthread_set_info(thread, XNCONTHI);
> and dovetail_request_ucall for all threads that have same cobalt_process? Is this solution OK for fixing this case?
>
>
> [3]:
> index 07dcc7a63..cd76961f5 100644
> --- a/kernel/cobalt/dovetail/kevents.c
> +++ b/kernel/cobalt/dovetail/kevents.c
> @@ -331,6 +331,8 @@ static int handle_user_return(struct task_struct *task)
> if (thread == NULL)
> return KEVENT_PROPAGATE;
>
> + trace_cobalt_handle_user_return(thread);
> +
> if (xnthread_test_info(thread, XNCONTHI)) {
> xnlock_get_irqsave(&nklock, s);
> xnthread_clear_info(thread, XNCONTHI);
> @@ -468,6 +470,8 @@ int handle_ptrace_resume(struct task_struct *tracee)
> if (thread == NULL)
> return KEVENT_PROPAGATE;
>
> + trace_cobalt_handle_ptrace_resume(thread);
> +
> if (xnthread_test_state(thread, XNSSTEP)) {
> xnlock_get_irqsave(&nklock, s);
>
> @@ -485,6 +489,8 @@ static void handle_ptrace_cont(void)
> struct xnthread *curr = xnthread_current();
> spl_t s;
>
> + trace_cobalt_handle_ptrace_cont(curr);
> +
> xnlock_get_irqsave(&nklock, s);
>
> if (xnthread_test_state(curr, XNSSTEP)) {
> @@ -604,6 +610,7 @@ void resume_oob_task(struct task_struct *p) /* inband, oob stage stalled */
> if (unlikely(xnthread_test_info(thread, XNCONTHI))) {
> xnthread_clear_info(thread, XNCONTHI);
> dovetail_clear_ucall();
> + trace_cobalt_resume_oob_task(thread);
> }
>
> /* Unregister as debugged thread in case we postponed this. */
> diff --git a/kernel/cobalt/trace/cobalt-core.h b/kernel/cobalt/trace/cobalt-core.h
> index 78f66a106..88a4067fd 100644
> --- a/kernel/cobalt/trace/cobalt-core.h
> +++ b/kernel/cobalt/trace/cobalt-core.h
> @@ -459,6 +459,27 @@ DEFINE_EVENT(curr_thread_event, cobalt_thread_set_mode,
> TP_ARGS(thread)
> );
>
> +DEFINE_EVENT(thread_event, cobalt_handle_ptrace_cont,
> + TP_PROTO(struct xnthread *thread),
>
> [4]:
> gdb-2495 [000] d..3 61513.834825: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
> smokey-2499 [000] ...1 61513.834827: cobalt_handle_ptrace_cont: pid=2499 state=0x4488c0 info=0x200
> smokey-2499 [000] d..1 61513.834828: cobalt_handle_user_return: pid=2499 state=0x4488c0 info=0x200
> smokey-2499 [000] d..1 61513.834828: cobalt_shadow_gohard: state=0x4488c0 info=0x0
> smokey-2499 [000] d..3 61513.834830: sched_stat_wait: comm=gdb pid=2495 delay=8017 [ns]
> smokey-2499 [000] d..3 61513.834831: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
> gdb-2495 [000] *..3 61513.834831: cobalt_thread_resume: name=smokey pid=2499 mask=0x80
> gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1
> gdb-2495 [000] *..3 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000
> gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1
> gdb-2495 [000] *.~3 61513.834833: cobalt_schedule: status=0x10000000
> gdb-2495 [000] *.~3 61513.834833: cobalt_trace_pid: pid=2495, prio=-1
> gdb-2495 [000] *.~3 61513.834833: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1
> smokey-2499 [000] d.~3 61513.834835: cobalt_shadow_hardened: state=0x48040 info=0x0
> smokey-2499 [000] d.~3 61513.834836: cobalt_head_sysentry: syscall=sem_init
> smokey-2499 [000] d.~3 61513.834839: cobalt_psem_init: sem=0x4e(anon) flags=0x0() value=0
> smokey-2499 [000] d.~3 61513.834839: cobalt_head_sysexit: result=0
> smokey-2499 [000] d.~3 61513.834848: cobalt_shadow_gorelax: reason=syscall
> smokey-2499 [000] d.~3 61513.834848: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey
> smokey-2499 [000] *.~3 61513.834848: cobalt_thread_suspend: pid=2499 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
> smokey-2499 [000] *.~3 61513.834849: cobalt_schedule: status=0x10000000
> smokey-2499 [000] *.~3 61513.834849: cobalt_trace_pid: pid=2499, prio=1
> smokey-2499 [000] *.~3 61513.834849: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
> gdb-2495 [000] *.~3 61513.834850: cobalt_trace_pid: pid=2495, prio=-1
> gdb-2495 [000] d.h3 61513.834851: irq_handler_entry: irq=120 name=in-band work
> gdb-2495 [000] d.h3 61513.834852: cobalt_lostage_wakeup: pid=2499 comm=smokey
> gdb-2495 [000] d.h5 61513.834852: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
> gdb-2495 [000] dNh6 61513.834853: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
> gdb-2495 [000] dNh3 61513.834853: irq_handler_exit: irq=120 ret=handled
> gdb-2495 [000] dN.3 61513.834854: sched_stat_runtime: comm=gdb pid=2495 runtime=23565 [ns] vruntime=300514015543 [ns]
> gdb-2495 [000] d..3 61513.834855: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
> smokey-2499 [000] ...1 61513.834856: cobalt_shadow_relaxed: state=0x480c0 info=0x0
> smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min(policy: 1)
> smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min -> 0x1
> smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max(policy: 1)
> smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max -> 0x63
> smokey-2499 [000] ..~1 61513.834862: cobalt_head_sysentry: syscall=thread_getschedparam_ex
> smokey-2499 [000] ...1 61513.834862: cobalt_root_sysentry: syscall=thread_getschedparam_ex
> smokey-2499 [000] ...1 61513.834863: cobalt_pthread_getschedparam: pth=000000008d36be1c policy=fifo param={ priority=1 }
> smokey-2499 [000] ...1 61513.834864: cobalt_root_sysexit: result=0
> smokey-2499 [000] ...1 61513.834872: sys_mmap(addr: 0, len: 11000, prot: 0, flags: 20022, fd: ffffffff, off: 0)
> smokey-2499 [000] ...1 61513.834877: sys_mmap -> 0x7ffff67b9000
> smokey-2499 [000] ...1 61513.834878: sys_mprotect(start: 7ffff67ba000, len: 10000, prot: 3)
> smokey-2499 [000] ...1 61513.834906: sys_mprotect -> 0x0
> smokey-2499 [000] ...1 61513.834909: sys_clone(clone_flags: 3d0f00, newsp: 7ffff67c8fb0, parent_tidptr: 7ffff67c99d0, child_tidptr: 7ffff67c99d0, tls: 7ffff67c9700)
> smokey-2499 [000] ...1 61513.834923: task_newtask: pid=2504 comm=smokey clone_flags=3d0f00 oom_score_adj=0
> smokey-2499 [000] ...1 61513.834923: sched_process_fork: comm=smokey pid=2499 child_comm=smokey child_pid=2504
> smokey-2499 [000] d..3 61513.834925: sched_wakeup_new: comm=smokey pid=2504 prio=98 target_cpu=000
> smokey-2499 [000] d..3 61513.834928: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0
> smokey-2499 [000] d..3 61513.834931: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=t ==> next_comm=smokey next_pid=2504 next_prio=98
> hi-thread-2504 [000] ...1 61513.834932: sys_clone -> 0x0
> hi-thread-2504 [000] d..3 61513.834934: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=2
> hi-thread-2504 [000] d..3 61513.834935: sched_stat_wait: comm=gdb pid=2495 delay=82002 [ns]
> hi-thread-2504 [000] d..3 61513.834936: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=t ==> next_comm=gdb next_pid=2495 next_prio=120
> gdb-2495 [000] d..2 61513.834937: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000
> gdb-2495 [000] ...1 61513.834940: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1)
> gdb-2495 [000] ...1 61513.834943: sys_read -> 0x1
> gdb-2495 [000] ...1 61513.834943: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1)
> gdb-2495 [000] ...1 61513.834944: sys_read -> 0xfffffffffffffff5
> gdb-2495 [000] ...1 61513.834944: sys_write(fd: 10, buf: 5585fe0354b1, count: 1)
> gdb-2495 [000] ...1 61513.834946: sys_write -> 0x1
> gdb-2495 [000] ...1 61513.834947: sys_rt_sigreturn()
> gdb-2495 [000] ...1 61513.834953: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1)
> gdb-2495 [000] ...1 61513.834953: sys_read -> 0xfffffffffffffff5
> gdb-2495 [000] ...1 61513.834955: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0)
> gdb-2495 [000] ...1 61513.834956: sys_poll -> 0x1
> gdb-2495 [000] ...1 61513.834957: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
> gdb-2495 [000] ...1 61513.834958: sys_read -> 0x1
> gdb-2495 [000] ...1 61513.834959: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
> gdb-2495 [000] ...1 61513.834960: sys_read -> 0xfffffffffffffff5
> gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8)
> gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask -> 0x0
> gdb-2495 [000] ...1 61513.834962: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
> gdb-2495 [000] ...1 61513.834962: sched_process_wait: comm=gdb pid=0 prio=120
> gdb-2495 [000] ...1 61513.834964: sys_wait4 -> 0x9c3
> gdb-2495 [000] ...1 61513.834965: sys_ptrace(request: 4201, pid: 9c3, addr: 0, data: 7ffe8b2a7d08)
> gdb-2495 [000] d..3 61513.834966: sched_wait_task: comm=smokey pid=2499 prio=98
> gdb-2495 [000] ...1 61513.834967: sys_ptrace -> 0x0
> gdb-2495 [000] ...1 61513.834967: sys_wait4(upid: 9c8, stat_addr: 7ffe8b2a7cfc, options: 40000000, ru: 0)
> gdb-2495 [000] ...1 61513.834968: sched_process_wait: comm=gdb pid=2504 prio=120
> gdb-2495 [000] ...1 61513.834969: sys_wait4 -> 0x9c8
> gdb-2495 [000] ...1 61513.834972: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578)
> gdb-2495 [000] d..3 61513.834972: sched_wait_task: comm=smokey pid=2499 prio=98
> gdb-2495 [000] ...1 61513.834975: sys_ptrace -> 0x0
> gdb-2495 [000] ...1 61513.834976: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a78d8)
> gdb-2495 [000] d..3 61513.834977: sched_wait_task: comm=smokey pid=2504 prio=98
> gdb-2495 [000] ...1 61513.834977: sys_ptrace -> 0x0
> gdb-2495 [000] ...1 61513.834979: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0)
> gdb-2495 [000] ...1 61513.834989: sys_openat -> 0x12
> gdb-2495 [000] ...1 61513.834990: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff67c9700)
> gdb-2495 [000] ...1 61513.834992: sys_pread64 -> 0x900
> gdb-2495 [000] ...1 61513.834993: sys_close(fd: 12)
> gdb-2495 [000] ...1 61513.834994: sys_close -> 0x0
> gdb-2495 [000] ...1 61513.835010: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578)
> gdb-2495 [000] d..3 61513.835011: sched_wait_task: comm=smokey pid=2499 prio=98
> gdb-2495 [000] ...1 61513.835012: sys_ptrace -> 0x0
> gdb-2495 [000] ...1 61513.835013: sys_ptrace(request: 3, pid: 9c3, addr: a8, data: 7ffe8b2a78d8)
> gdb-2495 [000] d..3 61513.835013: sched_wait_task: comm=smokey pid=2499 prio=98
> gdb-2495 [000] ...1 61513.835014: sys_ptrace -> 0x0
> gdb-2495 [000] ...1 61513.835015: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0)
> gdb-2495 [000] ...1 61513.835022: sys_openat -> 0x12
> gdb-2495 [000] ...1 61513.835022: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff758f740)
> gdb-2495 [000] ...1 61513.835025: sys_pread64 -> 0x900
> gdb-2495 [000] ...1 61513.835025: sys_close(fd: 12)
> gdb-2495 [000] ...1 61513.835026: sys_close -> 0x0
> gdb-2495 [000] ...1 61513.835031: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
> gdb-2495 [000] ...1 61513.835032: sched_process_wait: comm=gdb pid=0 prio=120
> gdb-2495 [000] ...1 61513.835033: sys_wait4 -> 0x0
> gdb-2495 [000] ...1 61513.835036: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0)
> gdb-2495 [000] d..3 61513.835037: sched_wait_task: comm=smokey pid=2504 prio=98
> gdb-2495 [000] ...1 61513.835039: sys_ptrace -> 0x0
> gdb-2495 [000] ...1 61513.835043: sys_ptrace(request: 6, pid: 9c8, addr: 388, data: 0)
> gdb-2495 [000] d..3 61513.835044: sched_wait_task: comm=smokey pid=2504 prio=98
> gdb-2495 [000] ...1 61513.835044: sys_ptrace -> 0x0
> gdb-2495 [000] ...1 61513.835045: sys_ptrace(request: 7, pid: 9c8, addr: 1, data: 0)
> gdb-2495 [000] d..3 61513.835046: sched_wait_task: comm=smokey pid=2504 prio=98
> gdb-2495 [000] d..3 61513.835047: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000
> gdb-2495 [000] dN.4 61513.835048: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000
> gdb-2495 [000] .N.1 61513.835048: sys_ptrace -> 0x0
> gdb-2495 [000] dN.3 61513.835050: sched_stat_runtime: comm=gdb pid=2495 runtime=112561 [ns] vruntime=300514128104 [ns]
> gdb-2495 [000] d..3 61513.835051: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=98
> hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list(head: 7ffff67c99e0, len: 18)
> hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list -> 0x0
> hi-thread-2504 [000] ...1 61513.835056: sys_futex(uaddr: 7ffff67c9d18, op: 80, val: 2, utime: 0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700)
> hi-thread-2504 [000] d..3 61513.835059: sched_stat_wait: comm=gdb pid=2495 delay=10883 [ns]
> hi-thread-2504 [000] d..3 61513.835060: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
> gdb-2495 [000] ...1 61513.835063: sys_ptrace(request: c, pid: 9c3, addr: 0, data: 7ffe8b2a6ef0)
> gdb-2495 [000] d..3 61513.835064: sched_wait_task: comm=smokey pid=2499 prio=98
> gdb-2495 [000] ...1 61513.835065: sys_ptrace -> 0x0
> gdb-2495 [000] ...1 61513.835068: sys_ptrace(request: 7, pid: 9c3, addr: 1, data: 0)
> gdb-2495 [000] d..3 61513.835069: sched_wait_task: comm=smokey pid=2499 prio=98
> gdb-2495 [000] d..3 61513.835069: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
> gdb-2495 [000] dN.4 61513.835070: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
> gdb-2495 [000] .N.1 61513.835071: sys_ptrace -> 0x0
> gdb-2495 [000] dN.3 61513.835071: sched_stat_runtime: comm=gdb pid=2495 runtime=12210 [ns] vruntime=300514140314 [ns]
> gdb-2495 [000] d..3 61513.835072: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
> smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0
> smokey-2499 [000] ...1 61513.835074: sys_clone -> 0x9c8
> smokey-2499 [000] ...1 61513.835075: sys_sched_setaffinity(pid: 9c8, len: 80, user_mask_ptr: 55555563e7c0)
> smokey-2499 [000] ...1 61513.835077: sys_sched_setaffinity -> 0x0
> smokey-2499 [000] ...1 61513.835077: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 0, val3: 7ffff67c9700)
> smokey-2499 [000] d..3 61513.835078: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000
> smokey-2499 [000] d..4 61513.835079: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000
> smokey-2499 [000] ...1 61513.835080: sys_futex -> 0x1
> smokey-2499 [000] ...1 61513.835086: sys_futex(uaddr: 7fffffffe828, op: 189, val: 0, utime: 7fffffffe780, uaddr2: 0, val3: ffffffff)
> smokey-2499 [000] d..3 61513.835089: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=smokey next_pid=2504 next_prio=98
> hi-thread-2504 [000] ...1 61513.835090: sys_futex -> 0x0
> hi-thread-2504 [000] ...1 61513.835090: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700)
> hi-thread-2504 [000] ...1 61513.835091: sys_futex -> 0x0
> hi-thread-2504 [000] ...1 61513.835092: sys_sched_setscheduler(pid: 9c8, policy: 1, param: 7ffff67c8e70)
> hi-thread-2504 [000] ...1 61513.835095: sys_sched_setscheduler -> 0x0
> hi-thread-2504 [000] ..~1 61513.835096: cobalt_head_sysentry: syscall=thread_create
> hi-thread-2504 [000] ...1 61513.835097: cobalt_root_sysentry: syscall=thread_create
> hi-thread-2504 [000] ...1 61513.835098: cobalt_pthread_create: pth=00000000ba35a01d policy=fifo param={ priority=2 }
> hi-thread-2504 [000] ...1 61513.835099: cobalt_timer_migrate: timer=000000004d21df54 cpu=0
> hi-thread-2504 [000] ...1 61513.835100: cobalt_timer_migrate: timer=00000000c1301853 cpu=0
> hi-thread-2504 [000] ...1 61513.835101: cobalt_thread_set_current_prio: thread=00000000e9fda3c4 pid=-1 prio=2
> hi-thread-2504 [000] ...1 61513.835102: cobalt_thread_init: thread=00000000e9fda3c4 name=smokey flags=0x48000 class=rt prio=2
> hi-thread-2504 [000] ...1 61513.835104: cobalt_shadow_map: thread=00000000e9fda3c4 pid=2504 prio=2
> hi-thread-2504 [000] D..1 61513.835104: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
> hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_start: pid=2504 state=0x480d0 info=0x0
> hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_resume: name=smokey pid=2504 mask=0x10
> hi-thread-2504 [000] D..1 61513.835106: cobalt_trace_pid: pid=2504, prio=2
> hi-thread-2504 [000] ...1 61513.835106: cobalt_trace_pid: pid=2504, prio=2
> hi-thread-2504 [000] ...1 61513.835107: cobalt_shadow_gohard: state=0x480c0 info=0x0
> hi-thread-2504 [000] d..3 61513.835108: sched_stat_wait: comm=gdb pid=2495 delay=37231 [ns]
> hi-thread-2504 [000] d..3 61513.835109: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
> gdb-2495 [000] *..3 61513.835109: cobalt_thread_resume: name=smokey pid=2504 mask=0x80
> gdb-2495 [000] *..3 61513.835110: cobalt_trace_pid: pid=2504, prio=2
> gdb-2495 [000] *.~3 61513.835110: cobalt_schedule: status=0x10000000
> gdb-2495 [000] *.~3 61513.835110: cobalt_trace_pid: pid=2495, prio=-1
> gdb-2495 [000] *.~3 61513.835111: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2
> hi-thread-2504 [000] d.~3 61513.835112: cobalt_shadow_hardened: state=0x48040 info=0x0
> hi-thread-2504 [000] d.~3 61513.835112: cobalt_root_sysexit: result=0
> hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysentry: syscall=get_current
> hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysexit: result=0
> hi-thread-2504 [000] d.~3 61513.835116: cobalt_shadow_gorelax: reason=syscall
> hi-thread-2504 [000] d.~3 61513.835117: cobalt_lostage_request: request=wakeup pid=2504 comm=smokey
> hi-thread-2504 [000] *.~3 61513.835117: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
> hi-thread-2504 [000] *.~3 61513.835117: cobalt_schedule: status=0x10000000
> hi-thread-2504 [000] *.~3 61513.835118: cobalt_trace_pid: pid=2504, prio=2
> hi-thread-2504 [000] *.~3 61513.835118: cobalt_switch_context: prev_name=smokey prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
> gdb-2495 [000] *.~3 61513.835119: cobalt_trace_pid: pid=2495, prio=-1
> gdb-2495 [000] d.h3 61513.835120: irq_handler_entry: irq=120 name=in-band work
> gdb-2495 [000] d.h3 61513.835120: cobalt_lostage_wakeup: pid=2504 comm=smokey
> gdb-2495 [000] d.h5 61513.835120: sched_waking: comm=smokey pid=2504 prio=97 target_cpu=000
> gdb-2495 [000] dNh6 61513.835121: sched_wakeup: comm=smokey pid=2504 prio=97 target_cpu=000
> gdb-2495 [000] dNh3 61513.835122: irq_handler_exit: irq=120 ret=handled
> gdb-2495 [000] dN.3 61513.835122: sched_stat_runtime: comm=gdb pid=2495 runtime=13441 [ns] vruntime=300514153755 [ns]
> gdb-2495 [000] d..3 61513.835123: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=97
> hi-thread-2504 [000] ...1 61513.835124: cobalt_shadow_relaxed: state=0x480c0 info=0x0
> hi-thread-2504 [000] ...1 61513.835125: sys_futex(uaddr: 7fffffffe828, op: 81, val: 1, utime: 0, uaddr2: 7ffff67c8e74, val3: 7fffffffe828)
> hi-thread-2504 [000] d..3 61513.835126: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
> hi-thread-2504 [000] d..4 61513.835126: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
> hi-thread-2504 [000] ...1 61513.835127: sys_futex -> 0x1
> hi-thread-2504 [000] ..~1 61513.835128: cobalt_head_sysentry: syscall=migrate
> hi-thread-2504 [000] ...1 61513.835128: cobalt_root_sysentry: syscall=migrate
> hi-thread-2504 [000] ...1 61513.835128: cobalt_shadow_gohard: state=0x480c0 info=0x0
> hi-thread-2504 [000] d..3 61513.835130: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=smokey next_pid=2499 next_prio=98
> smokey-2499 [000] *..3 61513.835130: cobalt_thread_resume: name=smokey pid=2504 mask=0x80
> smokey-2499 [000] *..3 61513.835130: cobalt_trace_pid: pid=2504, prio=2
> smokey-2499 [000] *.~3 61513.835131: cobalt_schedule: status=0x10000000
> smokey-2499 [000] *.~3 61513.835131: cobalt_trace_pid: pid=2499, prio=-1
> smokey-2499 [000] *.~3 61513.835131: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2
> hi-thread-2504 [000] d.~3 61513.835132: cobalt_shadow_hardened: state=0x48040 info=0x0
> hi-thread-2504 [000] d.~3 61513.835132: cobalt_root_sysexit: result=1
> hi-thread-2504 [000] d.~3 61513.835133: cobalt_head_sysentry: syscall=thread_setname
> hi-thread-2504 [000] d.~3 61513.835134: cobalt_pthread_setname: pth=00000000ba35a01d name=hi-thread
> hi-thread-2504 [000] d.~3 61513.835135: cobalt_head_sysexit: result=0
> hi-thread-2504 [000] d.~3 61513.835136: cobalt_head_sysentry: syscall=clock_nanosleep
> hi-thread-2504 [000] d.~3 61513.835137: cobalt_clock_nanosleep: clock_id=0 flags=0x0() rqt=(0.001000000)
> hi-thread-2504 [000] *.~3 61513.835138: cobalt_thread_suspend: pid=2504 mask=0x4 timeout=1000001 timeout_mode=0 wchan=0000000000000000
> hi-thread-2504 [000] *.~3 61513.835138: cobalt_timer_start: timer=000000004d21df54(smokey) value=1000001 interval=0 mode=rel
> hi-thread-2504 [000] *.~3 61513.835138: cobalt_schedule: status=0x10000000
> hi-thread-2504 [000] *.~3 61513.835139: cobalt_trace_pid: pid=2504, prio=2
> hi-thread-2504 [000] *.~3 61513.835139: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x48044 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
> smokey-2499 [000] *.~3 61513.835140: cobalt_trace_pid: pid=2499, prio=-1
> smokey-2499 [000] ...1 61513.835141: sys_futex -> 0x0
> smokey-2499 [000] ..~1 61513.835144: cobalt_head_sysentry: syscall=migrate
> smokey-2499 [000] ...1 61513.835144: cobalt_root_sysentry: syscall=migrate
> smokey-2499 [000] ...1 61513.835144: cobalt_shadow_gohard: state=0x480c0 info=0x0
> smokey-2499 [000] d..3 61513.835145: sched_stat_wait: comm=gdb pid=2495 delay=23982 [ns]
> smokey-2499 [000] d..3 61513.835146: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
> gdb-2495 [000] *..3 61513.835147: cobalt_thread_resume: name=smokey pid=2499 mask=0x80
> gdb-2495 [000] *..3 61513.835147: cobalt_trace_pid: pid=2499, prio=1
> gdb-2495 [000] *.~3 61513.835147: cobalt_schedule: status=0x10000000
> gdb-2495 [000] *.~3 61513.835147: cobalt_trace_pid: pid=2495, prio=-1
> gdb-2495 [000] *.~3 61513.835148: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1
> smokey-2499 [000] d.~3 61513.835148: cobalt_shadow_hardened: state=0x48040 info=0x0
> smokey-2499 [000] d.~3 61513.835149: cobalt_root_sysexit: result=1
> smokey-2499 [000] *.~3 61513.835219: irq_pipeline_entry: irq=4354
> smokey-2499 [000] *.~3 61513.835219: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
> smokey-2499 [000] *.~3 61513.835220: cobalt_timer_expire: timer=00000000d13b8aa9
> smokey-2499 [000] *.~3 61513.835220: cobalt_tick_shot: next tick at 61513.836136 (delay: 915 us)
> smokey-2499 [000] *.~3 61513.835221: irq_handler_exit: irq=4354 ret=handled
> smokey-2499 [000] *.~3 61513.835221: irq_pipeline_exit: irq=4354
> smokey-2499 [000] *.~3 61513.836137: irq_pipeline_entry: irq=4354
> smokey-2499 [000] *.~3 61513.836137: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
> smokey-2499 [000] *.~3 61513.836138: cobalt_timer_expire: timer=000000004d21df54
> smokey-2499 [000] *.~3 61513.836138: cobalt_thread_resume: name=hi-thread pid=2504 mask=0x4
> smokey-2499 [000] *.~3 61513.836138: cobalt_trace_pid: pid=2504, prio=2
> smokey-2499 [000] *.~3 61513.836139: cobalt_timer_stop: timer=000000004d21df54
> smokey-2499 [000] *.~3 61513.836139: irq_handler_exit: irq=4354 ret=handled
> smokey-2499 [000] *.~3 61513.836139: irq_pipeline_exit: irq=4354
> smokey-2499 [000] *.~3 61513.836140: cobalt_schedule: status=0x10000000
> smokey-2499 [000] *.~3 61513.836140: cobalt_trace_pid: pid=2499, prio=1
> smokey-2499 [000] *.~3 61513.836140: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x48048 ==> next_name=hi-thread next_pid=2504 next_prio=2
> hi-thread-2504 [000] *.~3 61513.836141: cobalt_trace_pid: pid=2504, prio=2
> hi-thread-2504 [000] d.~3 61513.836141: cobalt_head_sysexit: result=0
> hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3
> hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000
> hi-thread-2504 [000] d.~3 61513.836143: cobalt_shadow_gorelax: reason=fault
> hi-thread-2504 [000] d.~3 61513.836143: cobalt_lostage_request: request=wakeup pid=2504 comm=hi-thread
> hi-thread-2504 [000] *.~3 61513.836144: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
> hi-thread-2504 [000] *.~3 61513.836144: cobalt_schedule: status=0x10000000
> hi-thread-2504 [000] *.~3 61513.836144: cobalt_trace_pid: pid=2504, prio=2
> hi-thread-2504 [000] *.~3 61513.836145: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
> gdb-2495 [000] *.~3 61513.836146: cobalt_trace_pid: pid=2495, prio=-1
> gdb-2495 [000] d.h3 61513.836146: irq_handler_entry: irq=120 name=in-band work
> gdb-2495 [000] d.h3 61513.836147: cobalt_lostage_wakeup: pid=2504 comm=hi-thread
> gdb-2495 [000] d.h5 61513.836147: sched_waking: comm=hi-thread pid=2504 prio=97 target_cpu=000
> gdb-2495 [000] dNh6 61513.836147: sched_wakeup: comm=hi-thread pid=2504 prio=97 target_cpu=000
> gdb-2495 [000] dNh3 61513.836148: irq_handler_exit: irq=120 ret=handled
> gdb-2495 [000] dNh3 61513.836148: irq_handler_entry: irq=124 name=proxy tick
> gdb-2495 [000] dNh3 61513.836151: softirq_raise: vec=1 [action=TIMER]
> gdb-2495 [000] dNh3 61513.836152: softirq_raise: vec=9 [action=RCU]
> gdb-2495 [000] dNh4 61513.836152: sched_stat_runtime: comm=gdb pid=2495 runtime=1007474 [ns] vruntime=300515161229 [ns]
> gdb-2495 [000] *Nh3 61513.836154: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2395843 interval=0 mode=rel
> gdb-2495 [000] *Nh3 61513.836155: cobalt_tick_shot: next tick at 61513.838550 (delay: 2395 us)
> gdb-2495 [000] dNh3 61513.836155: irq_handler_exit: irq=124 ret=handled
> gdb-2495 [000] .Ns3 61513.836156: softirq_entry: vec=1 [action=TIMER]
> gdb-2495 [000] .Ns3 61513.836157: softirq_exit: vec=1 [action=TIMER]
> gdb-2495 [000] .Ns3 61513.836157: softirq_entry: vec=9 [action=RCU]
> gdb-2495 [000] dNs6 61513.836158: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000
> gdb-2495 [000] dNs7 61513.836159: sched_stat_runtime: comm=gdb pid=2495 runtime=6881 [ns] vruntime=300515168110 [ns]
> gdb-2495 [000] dNs7 61513.836160: sched_stat_blocked: comm=rcu_sched pid=11 delay=4188739 [ns]
> gdb-2495 [000] dNs7 61513.836161: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000
> gdb-2495 [000] .Ns3 61513.836164: softirq_exit: vec=9 [action=RCU]
> gdb-2495 [000] d..3 61513.836165: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=hi-thread next_pid=2504 next_prio=97
> hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200
> hi-thread-2504 [000] *..2 61513.836169: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000
> hi-thread-2504 [000] *..2 61513.836170: cobalt_thread_suspend: pid=2504 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000
> hi-thread-2504 [000] d..2 61513.836170: signal_generate: sig=5 errno=0 code=128 comm=hi-thread pid=2504 grp=0 res=0
> hi-thread-2504 [000] d..3 61513.836173: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0
> hi-thread-2504 [000] d..3 61513.836175: sched_stat_wait: comm=rcu_sched pid=11 delay=14490 [ns]
> hi-thread-2504 [000] d..3 61513.836175: sched_switch: prev_comm=hi-thread prev_pid=2504 prev_prio=97 prev_state=t ==> next_comm=rcu_sched next_pid=11 next_prio=120
> rcu_sched-11 [000] d..3 61513.836179: sched_stat_runtime: comm=rcu_sched pid=11 runtime=4960 [ns] vruntime=300512173070 [ns]
> rcu_sched-11 [000] d..3 61513.836179: sched_stat_wait: comm=gdb pid=2495 delay=19450 [ns]
> rcu_sched-11 [000] d..3 61513.836180: sched_switch: prev_comm=rcu_sched prev_pid=11 prev_prio=120 prev_state=I ==> next_comm=gdb next_pid=2495 next_prio=120
> gdb-2495 [000] ...1 61513.836187: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cb0, flags: 80000, mode: 0)
> gdb-2495 [000] ...1 61513.836197: sys_openat -> 0x12
> gdb-2495 [000] ...1 61513.836198: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7ae0)
> gdb-2495 [000] ...1 61513.836199: sys_newfstat -> 0x0
> gdb-2495 [000] ...1 61513.836200: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
> gdb-2495 [000] ...1 61513.836214: sys_read -> 0x3f1
> gdb-2495 [000] ...1 61513.836217: sys_close(fd: 12)
> gdb-2495 [000] ...1 61513.836217: sys_close -> 0x0
> gdb-2495 [000] ...1 61513.836221: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8)
> gdb-2495 [000] ...1 61513.836222: sys_rt_sigprocmask -> 0x0
> gdb-2495 [000] d..2 61513.836223: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000
> gdb-2495 [000] ...1 61513.836225: sys_read(fd: f, buf: 7ffe8b2a7667, count: 1)
> gdb-2495 [000] ...1 61513.836226: sys_read -> 0xfffffffffffffff5
> gdb-2495 [000] ...1 61513.836227: sys_write(fd: 10, buf: 5585fe0354b1, count: 1)
> gdb-2495 [000] ...1 61513.836229: sys_write -> 0x1
> gdb-2495 [000] ...1 61513.836229: sys_rt_sigreturn()
> gdb-2495 [000] ...1 61513.836233: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1)
> gdb-2495 [000] ...1 61513.836234: sys_read -> 0xfffffffffffffff5
> gdb-2495 [000] ...1 61513.836235: sys_write(fd: 1, buf: 5585fe7a5290, count: 27)
> gdb-2495 [000] d..4 61513.836236: sched_waking: comm=smokey pid=2493 prio=120 target_cpu=000
> gdb-2495 [000] d..5 61513.836237: sched_stat_runtime: comm=gdb pid=2495 runtime=58399 [ns] vruntime=300515226509 [ns]
> gdb-2495 [000] d..5 61513.836238: sched_stat_sleep: comm=smokey pid=2493 delay=1717179 [ns]
> gdb-2495 [000] dN.5 61513.836239: sched_wakeup: comm=smokey pid=2493 prio=120 target_cpu=000
> gdb-2495 [000] .N.1 61513.836240: sys_write -> 0x27
> gdb-2495 [000] dN.3 61513.836241: sched_stat_wait: comm=smokey pid=2493 delay=0 [ns]
> gdb-2495 [000] d..3 61513.836241: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2493 next_prio=120
> smokey-2493 [000] ...1 61513.836244: sys_read -> 0x1
> smokey-2493 [000] ..~1 61513.836246: cobalt_head_sysentry: syscall=read
> ...
> smokey-2493 [000] d..3 61513.836344: sched_stat_runtime: comm=smokey pid=2493 runtime=106636 [ns] vruntime=300512333145 [ns]
> smokey-2493 [000] d..3 61513.836344: sched_stat_wait: comm=gdb pid=2495 delay=106636 [ns]
> smokey-2493 [000] d..3 61513.836345: sched_switch: prev_comm=smokey prev_pid=2493 prev_prio=120 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
> gdb-2495 [000] ...1 61513.836347: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0)
> gdb-2495 [000] ...1 61513.836348: sys_poll -> 0x1
> gdb-2495 [000] ...1 61513.836350: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
> gdb-2495 [000] ...1 61513.836351: sys_read -> 0x1
> gdb-2495 [000] ...1 61513.836352: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
> gdb-2495 [000] ...1 61513.836352: sys_read -> 0xfffffffffffffff5
> gdb-2495 [000] ...1 61513.836353: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8)
> gdb-2495 [000] ...1 61513.836354: sys_rt_sigprocmask -> 0x0
> gdb-2495 [000] ...1 61513.836355: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
> gdb-2495 [000] ...1 61513.836355: sched_process_wait: comm=gdb pid=0 prio=120
> gdb-2495 [000] ...1 61513.836357: sys_wait4 -> 0x9c8
> gdb-2495 [000] ...1 61513.836360: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0)
> gdb-2495 [000] d..3 61513.836361: sched_wait_task: comm=hi-thread pid=2504 prio=97
> gdb-2495 [000] ...1 61513.836363: sys_ptrace -> 0x0
> gdb-2495 [000] ...1 61513.836367: sys_ptrace(request: 4202, pid: 9c8, addr: 0, data: 7ffe8b2a7cb0)
> gdb-2495 [000] d..3 61513.836368: sched_wait_task: comm=hi-thread pid=2504 prio=97
> gdb-2495 [000] ...1 61513.836369: sys_ptrace -> 0x0
> gdb-2495 [000] ...1 61513.836370: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6f10)
> gdb-2495 [000] d..3 61513.836370: sched_wait_task: comm=hi-thread pid=2504 prio=97
> gdb-2495 [000] ...1 61513.836371: sys_ptrace -> 0x0
> gdb-2495 [000] ...1 61513.836372: sys_ptrace(request: d, pid: 9c8, addr: 0, data: 7ffe8b2a6f10)
> gdb-2495 [000] d..3 61513.836373: sched_wait_task: comm=hi-thread pid=2504 prio=97
> gdb-2495 [000] ...1 61513.836374: sys_ptrace -> 0x0
> gdb-2495 [000] ...1 61513.836376: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
> gdb-2495 [000] ...1 61513.836376: sched_process_wait: comm=gdb pid=0 prio=120
> gdb-2495 [000] ...1 61513.836377: sys_wait4 -> 0x0
> gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8)
> gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask -> 0x0
> gdb-2495 [000] ...1 61513.836382: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cd0, flags: 80000, mode: 0)
> gdb-2495 [000] ...1 61513.836399: sys_openat -> 0x12
> gdb-2495 [000] ...1 61513.836400: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7b50)
> gdb-2495 [000] ...1 61513.836401: sys_newfstat -> 0x0
> gdb-2495 [000] ...1 61513.836402: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
> gdb-2495 [000] ...1 61513.836409: sys_read -> 0x140
> gdb-2495 [000] ...1 61513.836410: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400)
> gdb-2495 [000] ...1 61513.836410: sys_read -> 0x0
> gdb-2495 [000] ...1 61513.836415: sys_close(fd: 12)
> gdb-2495 [000] ...1 61513.836416: sys_close -> 0x0
> gdb-2495 [000] ...1 61513.836419: sys_read(fd: f, buf: 7ffe8b2a7f17, count: 1)
> gdb-2495 [000] ...1 61513.836420: sys_read -> 0xfffffffffffffff5
> gdb-2495 [000] ...1 61513.836421: sys_write(fd: 10, buf: 5585fe0354b1, count: 1)
> gdb-2495 [000] ...1 61513.836422: sys_write -> 0x1
> gdb-2495 [000] ...1 61513.836425: sys_ptrace(request: 1, pid: 9c8, addr: 7ffff777f370, data: 7ffe8b2a7878)
> gdb-2495 [000] d..3 61513.836426: sched_wait_task: comm=hi-thread pid=2504 prio=97
> gdb-2495 [000] ...1 61513.836428: sys_ptrace -> 0x0
> gdb-2495 [000] ...1 61513.836430: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a7bd8)
> gdb-2495 [000] d..3 61513.836430: sched_wait_task: comm=hi-thread pid=2504 prio=97
> gdb-2495 [000] ...1 61513.836431: sys_ptrace -> 0x0
> gdb-2495 [000] ...1 61513.836433: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a70e0, flags: 80000, mode: 0)
> gdb-2495 [000] ...1 61513.836446: sys_openat -> 0x12
> gdb-2495 [000] ...1 61513.836446: sys_pread64(fd: 12, buf: 7ffe8b2a7430, count: 900, pos: 7ffff67c9700)
> gdb-2495 [000] ...1 61513.836449: sys_pread64 -> 0x900
> gdb-2495 [000] ...1 61513.836450: sys_close(fd: 12)
> gdb-2495 [000] ...1 61513.836450: sys_close -> 0x0
> gdb-2495 [000] ...1 61513.836479: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0)
> gdb-2495 [000] ...1 61513.836488: sys_openat -> 0x12
> gdb-2495 [000] ...1 61513.836489: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950)
> gdb-2495 [000] ...1 61513.836490: sys_newfstat -> 0x0
> gdb-2495 [000] ...1 61513.836490: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400)
> gdb-2495 [000] ...1 61513.836498: sys_read -> 0x14d
> gdb-2495 [000] ...1 61513.836499: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
> gdb-2495 [000] ...1 61513.836499: sys_read -> 0x0
> gdb-2495 [000] ...1 61513.836503: sys_close(fd: 12)
> gdb-2495 [000] ...1 61513.836504: sys_close -> 0x0
> gdb-2495 [000] ...1 61513.836507: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0)
> gdb-2495 [000] ...1 61513.836514: sys_openat -> 0x12
> gdb-2495 [000] ...1 61513.836515: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950)
> gdb-2495 [000] ...1 61513.836515: sys_newfstat -> 0x0
> gdb-2495 [000] ...1 61513.836516: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
> gdb-2495 [000] ...1 61513.836521: sys_read -> 0x13b
> gdb-2495 [000] ...1 61513.836522: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400)
> gdb-2495 [000] ...1 61513.836522: sys_read -> 0x0
> gdb-2495 [000] ...1 61513.836525: sys_close(fd: 12)
> gdb-2495 [000] ...1 61513.836525: sys_close -> 0x0
> gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13)
> gdb-2495 [000] *..3 61513.836532: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000
> gdb-2495 [000] *..3 61513.836532: cobalt_trace_pid: pid=2499, prio=1
> gdb-2495 [000] *..3 61513.836533: cobalt_thread_unblock: pid=2499 state=0x48848 info=0x0
> gdb-2495 [000] *.~3 61513.836586: cobalt_schedule: status=0x10000000
> gdb-2495 [000] *.~3 61513.836586: cobalt_trace_pid: pid=2495, prio=-1
> gdb-2495 [000] *.~3 61513.836587: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1
> smokey-2499 [000] *.~3 61513.836589: cobalt_trace_pid: pid=2499, prio=1
> smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined
> smokey-2499 [000] *.~3 61513.836589: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey
> smokey-2499 [000] *.~3 61513.836590: cobalt_thread_suspend: pid=2499 mask=0x400080 timeout=0 timeout_mode=0 wchan=0000000000000000
> smokey-2499 [000] *.~3 61513.836590: cobalt_schedule: status=0x10000000
> smokey-2499 [000] *.~3 61513.836590: cobalt_trace_pid: pid=2499, prio=1
> smokey-2499 [000] *.~3 61513.836591: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x4488c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
> gdb-2495 [000] *.~3 61513.836592: cobalt_trace_pid: pid=2495, prio=-1
> gdb-2495 [000] d..5 61513.836592: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
> gdb-2495 [000] dN.6 61513.836594: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
> gdb-2495 [000] dN.3 61513.836594: signal_generate: sig=19 errno=0 code=-6 comm=smokey pid=2499 grp=0 res=0
> gdb-2495 [000] dNh3 61513.836595: irq_handler_entry: irq=120 name=in-band work
> gdb-2495 [000] dNh3 61513.836596: cobalt_lostage_wakeup: pid=2499 comm=smokey
> gdb-2495 [000] *N~4 61513.838552: irq_pipeline_entry: irq=4354
> gdb-2495 [000] *N~4 61513.838553: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
> gdb-2495 [000] *N~4 61513.838554: cobalt_timer_expire: timer=00000000d13b8aa9
> gdb-2495 [000] *N~4 61513.838555: irq_handler_exit: irq=4354 ret=handled
> gdb-2495 [000] *N~4 61513.838555: irq_pipeline_exit: irq=4354
> gdb-2495 [000] dNh8 61513.981277: sched_waking: comm=systemd-journal pid=163 prio=120 target_cpu=000
> gdb-2495 [000] dNh9 61513.981278: sched_stat_runtime: comm=gdb pid=2495 runtime=144934239 [ns] vruntime=300660160748 [ns]
> gdb-2495 [000] dNh9 61513.981280: sched_stat_sleep: comm=systemd-journal pid=163 delay=111416392895 [ns]
> gdb-2495 [000] dNh9 61513.981280: sched_wakeup: comm=systemd-journal pid=163 prio=120 target_cpu=000
> gdb-2495 [000] dNh3 61513.981281: irq_handler_exit: irq=120 ret=handled
> gdb-2495 [000] dNh3 61513.981282: irq_handler_entry: irq=124 name=proxy tick
> gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=1 [action=TIMER]
> gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=9 [action=RCU]
> gdb-2495 [000] dNh4 61513.981286: sched_stat_runtime: comm=gdb pid=2495 runtime=8375 [ns] vruntime=300660169123 [ns]
> gdb-2495 [000] dNh5 61513.981288: sched_waking: comm=cobalt_printf pid=2494 prio=120 target_cpu=000
> gdb-2495 [000] dNh6 61513.981289: sched_stat_runtime: comm=gdb pid=2495 runtime=2987 [ns] vruntime=300660172110 [ns]
> gdb-2495 [000] dNh6 61513.981291: sched_stat_sleep: comm=cobalt_printf pid=2494 delay=238660319 [ns]
> gdb-2495 [000] dNh6 61513.981291: sched_wakeup: comm=cobalt_printf pid=2494 prio=120 target_cpu=000
> gdb-2495 [000] dNh5 61513.981292: sched_waking: comm=cobalt_printf pid=2503 prio=120 target_cpu=000
> gdb-2495 [000] dNh6 61513.981292: sched_stat_runtime: comm=gdb pid=2495 runtime=3069 [ns] vruntime=300660175179 [ns]
> gdb-2495 [000] dNh6 61513.981294: sched_stat_sleep: comm=cobalt_printf pid=2503 delay=146611442 [ns]
> gdb-2495 [000] dNh6 61513.981294: sched_wakeup: comm=cobalt_printf pid=2503 prio=120 target_cpu=000
> gdb-2495 [000] *Nh3 61513.981295: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=588476 interval=0 mode=rel
> gdb-2495 [000] *Nh3 61513.981296: cobalt_tick_shot: next tick at 61513.981884 (delay: 588 us)
> gdb-2495 [000] dNh3 61513.981296: irq_handler_exit: irq=124 ret=handled
> gdb-2495 [000] .Ns3 61513.981297: softirq_entry: vec=1 [action=TIMER]
> gdb-2495 [000] dNs5 61513.981298: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000
> gdb-2495 [000] dNs6 61513.981298: sched_stat_runtime: comm=gdb pid=2495 runtime=6218 [ns] vruntime=300660181397 [ns]
> gdb-2495 [000] dNs6 61513.981299: sched_stat_blocked: comm=rcu_sched pid=11 delay=145119923 [ns]
> gdb-2495 [000] dNs6 61513.981299: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000
> gdb-2495 [000] dNs7 61513.981302: sched_waking: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000
> gdb-2495 [000] dNs8 61513.981302: sched_stat_runtime: comm=gdb pid=2495 runtime=3884 [ns] vruntime=300660185281 [ns]
> gdb-2495 [000] dNs8 61513.981303: sched_stat_blocked: comm=kworker/0:1 pid=1932 delay=206048652 [ns]
> gdb-2495 [000] dNs8 61513.981304: sched_wakeup: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000
> gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX]
> gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX]
> gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX]
> gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX]
> gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX]
> gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX]
> gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX]
> gdb-2495 [000] dNs5 61513.981312: softirq_raise: vec=2 [action=NET_TX]
> gdb-2495 [000] .Ns3 61513.981314: softirq_exit: vec=1 [action=TIMER]
> gdb-2495 [000] .Ns3 61513.981314: softirq_entry: vec=9 [action=RCU]
> gdb-2495 [000] .Ns3 61513.981315: softirq_exit: vec=9 [action=RCU]
> gdb-2495 [000] dNs5 61513.981316: sched_waking: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000
> gdb-2495 [000] dNs6 61513.981316: sched_stat_runtime: comm=gdb pid=2495 runtime=13841 [ns] vruntime=300660199122 [ns]
> gdb-2495 [000] dNs6 61513.981317: sched_stat_sleep: comm=ksoftirqd/0 pid=10 delay=149349034 [ns]
> gdb-2495 [000] dNs6 61513.981317: sched_wakeup: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000
> gdb-2495 [000] .N.1 61513.981318: sys_tkill -> 0x0
> gdb-2495 [000] d..3 61513.981320: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
> smokey-2499 [000] *.~3 61513.981888: irq_pipeline_entry: irq=4354
> smokey-2499 [000] *.~3 61513.981888: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
> smokey-2499 [000] *.~3 61513.981889: cobalt_timer_expire: timer=00000000d13b8aa9
> smokey-2499 [000] *.~3 61513.981889: irq_handler_exit: irq=4354 ret=handled
> smokey-2499 [000] *.~3 61513.981890: irq_pipeline_exit: irq=4354
> smokey-2499 [000] d.h1 61514.106435: irq_handler_entry: irq=124 name=proxy tick
> smokey-2499 [000] *.h1 61514.106441: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2109534 interval=0 mode=rel
> smokey-2499 [000] *.h1 61514.106441: cobalt_tick_shot: next tick at 61514.108550 (delay: 2109 us)
> smokey-2499 [000] d.h1 61514.106442: irq_handler_exit: irq=124 ret=handled
> smokey-2499 [000] ...1 61514.106442: cobalt_shadow_relaxed: state=0x4488c0 info=0x200
> smokey-2499 [000] *.~1 61514.108552: irq_pipeline_entry: irq=4354
>
> Regards
>
> Hongzhan Chen
>
>>
>>--
>>Philippe.
--
Philippe.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: gdb test failure debug status update
2021-04-30 7:34 ` Philippe Gerum
@ 2021-04-30 8:00 ` Philippe Gerum
2021-04-30 8:07 ` Chen, Hongzhan
0 siblings, 1 reply; 16+ messages in thread
From: Philippe Gerum @ 2021-04-30 8:00 UTC (permalink / raw)
To: Chen, Hongzhan; +Cc: xenomai
Philippe Gerum <rpm@xenomai.org> writes:
> Chen, Hongzhan <hongzhan.chen@intel.com> writes:
>
>> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish
>> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax.
>> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in
>> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from
>> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to
>> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode
>> because kernel exit code is never called in this case?
>>
>
> The IRQ frame is indeed kept from unwinding until the preempted task
> context returns from irq_exit_pipeline(), which branches to the Cobalt
> rescheduling procedure. From the Dovetail interface POV,
> irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow
> the companion core to perform post-IRQ chores, such as running its own
> rescheduling procedure.
>
> IOW, if task @foo is preempted by an IRQ, then suspended in oob context
> as a result of what the interrupt handler just did for it (e.g. raising
> XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then
> handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will
> cause the @foo context to switch away, effectively preventing
> handle_irq_pipelined_finish() to return, until @foo resumes execution
> eventually.
>
> The underlying issue I'd like to understand, is why do we have a mayday
> condition raised in the context of this test in the first place? That
> may be the root of the issue, the mayday and synchronous-bp handling
> code may not cope that well.
>
> So, question to address first: who ends up calling
> dovetail_call_mayday(), detecting that the underlying task went south?
> And why that?
Possibly a caller of __xnthread_kick() is causing that, e.g.
handle_sigwake_event -> __xnthread_kick.
--
Philippe.
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: gdb test failure debug status update
2021-04-30 8:00 ` Philippe Gerum
@ 2021-04-30 8:07 ` Chen, Hongzhan
[not found] ` <DM5PR11MB18529649C47BF241930A2217F25E9@DM5PR11MB1852.namprd11.prod.outlook.com>
0 siblings, 1 reply; 16+ messages in thread
From: Chen, Hongzhan @ 2021-04-30 8:07 UTC (permalink / raw)
To: Philippe Gerum; +Cc: xenomai
>-----Original Message-----
>From: Philippe Gerum <rpm@xenomai.org>
>Sent: Friday, April 30, 2021 4:01 PM
>To: Chen, Hongzhan <hongzhan.chen@intel.com>
>Cc: xenomai@xenomai.org
>Subject: Re: gdb test failure debug status update
>
>
>Philippe Gerum <rpm@xenomai.org> writes:
>
>> Chen, Hongzhan <hongzhan.chen@intel.com> writes:
>>
>>> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish
>>> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax.
>>> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in
>>> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from
>>> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to
>>> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode
>>> because kernel exit code is never called in this case?
>>>
>>
>> The IRQ frame is indeed kept from unwinding until the preempted task
>> context returns from irq_exit_pipeline(), which branches to the Cobalt
>> rescheduling procedure. From the Dovetail interface POV,
>> irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow
>> the companion core to perform post-IRQ chores, such as running its own
>> rescheduling procedure.
>>
>> IOW, if task @foo is preempted by an IRQ, then suspended in oob context
>> as a result of what the interrupt handler just did for it (e.g. raising
>> XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then
>> handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will
>> cause the @foo context to switch away, effectively preventing
>> handle_irq_pipelined_finish() to return, until @foo resumes execution
> eventually.
>
> The underlying issue I'd like to understand, is why do we have a mayday
> condition raised in the context of this test in the first place? That
> may be the root of the issue, the mayday and synchronous-bp handling
> code may not cope that well.
>
> So, question to address first: who ends up calling
> dovetail_call_mayday(), detecting that the underlying task went south?
> And why that?
>
>Possibly a caller of __xnthread_kick() is causing that, e.g.
>handle_sigwake_event -> __xnthread_kick.
Yes, that is right. Like in line gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13) of log [4] ,
Sys_tkill would finally call __xnthread_kick to set MAYDAY through handle_sigwake_event by gdb and then
Smokey thread is to handle mayday. I also get ftrace log for ipipe-5.4-xenomai, it has same caller.
>
>--
>Philippe.
>
[4]:
gdb-2495 [000] d..3 61513.834825: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] ...1 61513.834827: cobalt_handle_ptrace_cont: pid=2499 state=0x4488c0 info=0x200
smokey-2499 [000] d..1 61513.834828: cobalt_handle_user_return: pid=2499 state=0x4488c0 info=0x200
smokey-2499 [000] d..1 61513.834828: cobalt_shadow_gohard: state=0x4488c0 info=0x0
smokey-2499 [000] d..3 61513.834830: sched_stat_wait: comm=gdb pid=2495 delay=8017 [ns]
smokey-2499 [000] d..3 61513.834831: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] *..3 61513.834831: cobalt_thread_resume: name=smokey pid=2499 mask=0x80
gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1
gdb-2495 [000] *..3 61513.834832: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000
gdb-2495 [000] *..3 61513.834832: cobalt_trace_pid: pid=2499, prio=1
gdb-2495 [000] *.~3 61513.834833: cobalt_schedule: status=0x10000000
gdb-2495 [000] *.~3 61513.834833: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] *.~3 61513.834833: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1
smokey-2499 [000] d.~3 61513.834835: cobalt_shadow_hardened: state=0x48040 info=0x0
smokey-2499 [000] d.~3 61513.834836: cobalt_head_sysentry: syscall=sem_init
smokey-2499 [000] d.~3 61513.834839: cobalt_psem_init: sem=0x4e(anon) flags=0x0() value=0
smokey-2499 [000] d.~3 61513.834839: cobalt_head_sysexit: result=0
smokey-2499 [000] d.~3 61513.834848: cobalt_shadow_gorelax: reason=syscall
smokey-2499 [000] d.~3 61513.834848: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey
smokey-2499 [000] *.~3 61513.834848: cobalt_thread_suspend: pid=2499 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
smokey-2499 [000] *.~3 61513.834849: cobalt_schedule: status=0x10000000
smokey-2499 [000] *.~3 61513.834849: cobalt_trace_pid: pid=2499, prio=1
smokey-2499 [000] *.~3 61513.834849: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
gdb-2495 [000] *.~3 61513.834850: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] d.h3 61513.834851: irq_handler_entry: irq=120 name=in-band work
gdb-2495 [000] d.h3 61513.834852: cobalt_lostage_wakeup: pid=2499 comm=smokey
gdb-2495 [000] d.h5 61513.834852: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dNh6 61513.834853: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dNh3 61513.834853: irq_handler_exit: irq=120 ret=handled
gdb-2495 [000] dN.3 61513.834854: sched_stat_runtime: comm=gdb pid=2495 runtime=23565 [ns] vruntime=300514015543 [ns]
gdb-2495 [000] d..3 61513.834855: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] ...1 61513.834856: cobalt_shadow_relaxed: state=0x480c0 info=0x0
smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min(policy: 1)
smokey-2499 [000] ...1 61513.834857: sys_sched_get_priority_min -> 0x1
smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max(policy: 1)
smokey-2499 [000] ...1 61513.834859: sys_sched_get_priority_max -> 0x63
smokey-2499 [000] ..~1 61513.834862: cobalt_head_sysentry: syscall=thread_getschedparam_ex
smokey-2499 [000] ...1 61513.834862: cobalt_root_sysentry: syscall=thread_getschedparam_ex
smokey-2499 [000] ...1 61513.834863: cobalt_pthread_getschedparam: pth=000000008d36be1c policy=fifo param={ priority=1 }
smokey-2499 [000] ...1 61513.834864: cobalt_root_sysexit: result=0
smokey-2499 [000] ...1 61513.834872: sys_mmap(addr: 0, len: 11000, prot: 0, flags: 20022, fd: ffffffff, off: 0)
smokey-2499 [000] ...1 61513.834877: sys_mmap -> 0x7ffff67b9000
smokey-2499 [000] ...1 61513.834878: sys_mprotect(start: 7ffff67ba000, len: 10000, prot: 3)
smokey-2499 [000] ...1 61513.834906: sys_mprotect -> 0x0
smokey-2499 [000] ...1 61513.834909: sys_clone(clone_flags: 3d0f00, newsp: 7ffff67c8fb0, parent_tidptr: 7ffff67c99d0, child_tidptr: 7ffff67c99d0, tls: 7ffff67c9700)
smokey-2499 [000] ...1 61513.834923: task_newtask: pid=2504 comm=smokey clone_flags=3d0f00 oom_score_adj=0
smokey-2499 [000] ...1 61513.834923: sched_process_fork: comm=smokey pid=2499 child_comm=smokey child_pid=2504
smokey-2499 [000] d..3 61513.834925: sched_wakeup_new: comm=smokey pid=2504 prio=98 target_cpu=000
smokey-2499 [000] d..3 61513.834928: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0
smokey-2499 [000] d..3 61513.834931: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=t ==> next_comm=smokey next_pid=2504 next_prio=98
hi-thread-2504 [000] ...1 61513.834932: sys_clone -> 0x0
hi-thread-2504 [000] d..3 61513.834934: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=2
hi-thread-2504 [000] d..3 61513.834935: sched_stat_wait: comm=gdb pid=2495 delay=82002 [ns]
hi-thread-2504 [000] d..3 61513.834936: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=t ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] d..2 61513.834937: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000
gdb-2495 [000] ...1 61513.834940: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1)
gdb-2495 [000] ...1 61513.834943: sys_read -> 0x1
gdb-2495 [000] ...1 61513.834943: sys_read(fd: f, buf: 7ffe8b2a71e7, count: 1)
gdb-2495 [000] ...1 61513.834944: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.834944: sys_write(fd: 10, buf: 5585fe0354b1, count: 1)
gdb-2495 [000] ...1 61513.834946: sys_write -> 0x1
gdb-2495 [000] ...1 61513.834947: sys_rt_sigreturn()
gdb-2495 [000] ...1 61513.834953: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1)
gdb-2495 [000] ...1 61513.834953: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.834955: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0)
gdb-2495 [000] ...1 61513.834956: sys_poll -> 0x1
gdb-2495 [000] ...1 61513.834957: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
gdb-2495 [000] ...1 61513.834958: sys_read -> 0x1
gdb-2495 [000] ...1 61513.834959: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
gdb-2495 [000] ...1 61513.834960: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8)
gdb-2495 [000] ...1 61513.834961: sys_rt_sigprocmask -> 0x0
gdb-2495 [000] ...1 61513.834962: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
gdb-2495 [000] ...1 61513.834962: sched_process_wait: comm=gdb pid=0 prio=120
gdb-2495 [000] ...1 61513.834964: sys_wait4 -> 0x9c3
gdb-2495 [000] ...1 61513.834965: sys_ptrace(request: 4201, pid: 9c3, addr: 0, data: 7ffe8b2a7d08)
gdb-2495 [000] d..3 61513.834966: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.834967: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.834967: sys_wait4(upid: 9c8, stat_addr: 7ffe8b2a7cfc, options: 40000000, ru: 0)
gdb-2495 [000] ...1 61513.834968: sched_process_wait: comm=gdb pid=2504 prio=120
gdb-2495 [000] ...1 61513.834969: sys_wait4 -> 0x9c8
gdb-2495 [000] ...1 61513.834972: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578)
gdb-2495 [000] d..3 61513.834972: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.834975: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.834976: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a78d8)
gdb-2495 [000] d..3 61513.834977: sched_wait_task: comm=smokey pid=2504 prio=98
gdb-2495 [000] ...1 61513.834977: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.834979: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.834989: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.834990: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff67c9700)
gdb-2495 [000] ...1 61513.834992: sys_pread64 -> 0x900
gdb-2495 [000] ...1 61513.834993: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.834994: sys_close -> 0x0
gdb-2495 [000] ...1 61513.835010: sys_ptrace(request: 1, pid: 9c3, addr: 7ffff777f370, data: 7ffe8b2a7578)
gdb-2495 [000] d..3 61513.835011: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.835012: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835013: sys_ptrace(request: 3, pid: 9c3, addr: a8, data: 7ffe8b2a78d8)
gdb-2495 [000] d..3 61513.835013: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.835014: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835015: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a6de0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.835022: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.835022: sys_pread64(fd: 12, buf: 7ffe8b2a7130, count: 900, pos: 7ffff758f740)
gdb-2495 [000] ...1 61513.835025: sys_pread64 -> 0x900
gdb-2495 [000] ...1 61513.835025: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.835026: sys_close -> 0x0
gdb-2495 [000] ...1 61513.835031: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
gdb-2495 [000] ...1 61513.835032: sched_process_wait: comm=gdb pid=0 prio=120
gdb-2495 [000] ...1 61513.835033: sys_wait4 -> 0x0
gdb-2495 [000] ...1 61513.835036: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0)
gdb-2495 [000] d..3 61513.835037: sched_wait_task: comm=smokey pid=2504 prio=98
gdb-2495 [000] ...1 61513.835039: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835043: sys_ptrace(request: 6, pid: 9c8, addr: 388, data: 0)
gdb-2495 [000] d..3 61513.835044: sched_wait_task: comm=smokey pid=2504 prio=98
gdb-2495 [000] ...1 61513.835044: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835045: sys_ptrace(request: 7, pid: 9c8, addr: 1, data: 0)
gdb-2495 [000] d..3 61513.835046: sched_wait_task: comm=smokey pid=2504 prio=98
gdb-2495 [000] d..3 61513.835047: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000
gdb-2495 [000] dN.4 61513.835048: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000
gdb-2495 [000] .N.1 61513.835048: sys_ptrace -> 0x0
gdb-2495 [000] dN.3 61513.835050: sched_stat_runtime: comm=gdb pid=2495 runtime=112561 [ns] vruntime=300514128104 [ns]
gdb-2495 [000] d..3 61513.835051: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=98
hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list(head: 7ffff67c99e0, len: 18)
hi-thread-2504 [000] ...1 61513.835055: sys_set_robust_list -> 0x0
hi-thread-2504 [000] ...1 61513.835056: sys_futex(uaddr: 7ffff67c9d18, op: 80, val: 2, utime: 0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700)
hi-thread-2504 [000] d..3 61513.835059: sched_stat_wait: comm=gdb pid=2495 delay=10883 [ns]
hi-thread-2504 [000] d..3 61513.835060: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] ...1 61513.835063: sys_ptrace(request: c, pid: 9c3, addr: 0, data: 7ffe8b2a6ef0)
gdb-2495 [000] d..3 61513.835064: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] ...1 61513.835065: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.835068: sys_ptrace(request: 7, pid: 9c3, addr: 1, data: 0)
gdb-2495 [000] d..3 61513.835069: sched_wait_task: comm=smokey pid=2499 prio=98
gdb-2495 [000] d..3 61513.835069: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dN.4 61513.835070: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] .N.1 61513.835071: sys_ptrace -> 0x0
gdb-2495 [000] dN.3 61513.835071: sched_stat_runtime: comm=gdb pid=2495 runtime=12210 [ns] vruntime=300514140314 [ns]
gdb-2495 [000] d..3 61513.835072: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] ...1 61513.835073: cobalt_handle_ptrace_cont: pid=2499 state=0x480c0 info=0x0
smokey-2499 [000] ...1 61513.835074: sys_clone -> 0x9c8
smokey-2499 [000] ...1 61513.835075: sys_sched_setaffinity(pid: 9c8, len: 80, user_mask_ptr: 55555563e7c0)
smokey-2499 [000] ...1 61513.835077: sys_sched_setaffinity -> 0x0
smokey-2499 [000] ...1 61513.835077: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 0, val3: 7ffff67c9700)
smokey-2499 [000] d..3 61513.835078: sched_waking: comm=smokey pid=2504 prio=98 target_cpu=000
smokey-2499 [000] d..4 61513.835079: sched_wakeup: comm=smokey pid=2504 prio=98 target_cpu=000
smokey-2499 [000] ...1 61513.835080: sys_futex -> 0x1
smokey-2499 [000] ...1 61513.835086: sys_futex(uaddr: 7fffffffe828, op: 189, val: 0, utime: 7fffffffe780, uaddr2: 0, val3: ffffffff)
smokey-2499 [000] d..3 61513.835089: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=smokey next_pid=2504 next_prio=98
hi-thread-2504 [000] ...1 61513.835090: sys_futex -> 0x0
hi-thread-2504 [000] ...1 61513.835090: sys_futex(uaddr: 7ffff67c9d18, op: 81, val: 1, utime: 7ffff67c99d0, uaddr2: 7ffff67c9700, val3: 7ffff67c9700)
hi-thread-2504 [000] ...1 61513.835091: sys_futex -> 0x0
hi-thread-2504 [000] ...1 61513.835092: sys_sched_setscheduler(pid: 9c8, policy: 1, param: 7ffff67c8e70)
hi-thread-2504 [000] ...1 61513.835095: sys_sched_setscheduler -> 0x0
hi-thread-2504 [000] ..~1 61513.835096: cobalt_head_sysentry: syscall=thread_create
hi-thread-2504 [000] ...1 61513.835097: cobalt_root_sysentry: syscall=thread_create
hi-thread-2504 [000] ...1 61513.835098: cobalt_pthread_create: pth=00000000ba35a01d policy=fifo param={ priority=2 }
hi-thread-2504 [000] ...1 61513.835099: cobalt_timer_migrate: timer=000000004d21df54 cpu=0
hi-thread-2504 [000] ...1 61513.835100: cobalt_timer_migrate: timer=00000000c1301853 cpu=0
hi-thread-2504 [000] ...1 61513.835101: cobalt_thread_set_current_prio: thread=00000000e9fda3c4 pid=-1 prio=2
hi-thread-2504 [000] ...1 61513.835102: cobalt_thread_init: thread=00000000e9fda3c4 name=smokey flags=0x48000 class=rt prio=2
hi-thread-2504 [000] ...1 61513.835104: cobalt_shadow_map: thread=00000000e9fda3c4 pid=2504 prio=2
hi-thread-2504 [000] D..1 61513.835104: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_start: pid=2504 state=0x480d0 info=0x0
hi-thread-2504 [000] D..1 61513.835105: cobalt_thread_resume: name=smokey pid=2504 mask=0x10
hi-thread-2504 [000] D..1 61513.835106: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] ...1 61513.835106: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] ...1 61513.835107: cobalt_shadow_gohard: state=0x480c0 info=0x0
hi-thread-2504 [000] d..3 61513.835108: sched_stat_wait: comm=gdb pid=2495 delay=37231 [ns]
hi-thread-2504 [000] d..3 61513.835109: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] *..3 61513.835109: cobalt_thread_resume: name=smokey pid=2504 mask=0x80
gdb-2495 [000] *..3 61513.835110: cobalt_trace_pid: pid=2504, prio=2
gdb-2495 [000] *.~3 61513.835110: cobalt_schedule: status=0x10000000
gdb-2495 [000] *.~3 61513.835110: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] *.~3 61513.835111: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2
hi-thread-2504 [000] d.~3 61513.835112: cobalt_shadow_hardened: state=0x48040 info=0x0
hi-thread-2504 [000] d.~3 61513.835112: cobalt_root_sysexit: result=0
hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysentry: syscall=get_current
hi-thread-2504 [000] d.~3 61513.835113: cobalt_head_sysexit: result=0
hi-thread-2504 [000] d.~3 61513.835116: cobalt_shadow_gorelax: reason=syscall
hi-thread-2504 [000] d.~3 61513.835117: cobalt_lostage_request: request=wakeup pid=2504 comm=smokey
hi-thread-2504 [000] *.~3 61513.835117: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] *.~3 61513.835117: cobalt_schedule: status=0x10000000
hi-thread-2504 [000] *.~3 61513.835118: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] *.~3 61513.835118: cobalt_switch_context: prev_name=smokey prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
gdb-2495 [000] *.~3 61513.835119: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] d.h3 61513.835120: irq_handler_entry: irq=120 name=in-band work
gdb-2495 [000] d.h3 61513.835120: cobalt_lostage_wakeup: pid=2504 comm=smokey
gdb-2495 [000] d.h5 61513.835120: sched_waking: comm=smokey pid=2504 prio=97 target_cpu=000
gdb-2495 [000] dNh6 61513.835121: sched_wakeup: comm=smokey pid=2504 prio=97 target_cpu=000
gdb-2495 [000] dNh3 61513.835122: irq_handler_exit: irq=120 ret=handled
gdb-2495 [000] dN.3 61513.835122: sched_stat_runtime: comm=gdb pid=2495 runtime=13441 [ns] vruntime=300514153755 [ns]
gdb-2495 [000] d..3 61513.835123: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2504 next_prio=97
hi-thread-2504 [000] ...1 61513.835124: cobalt_shadow_relaxed: state=0x480c0 info=0x0
hi-thread-2504 [000] ...1 61513.835125: sys_futex(uaddr: 7fffffffe828, op: 81, val: 1, utime: 0, uaddr2: 7ffff67c8e74, val3: 7fffffffe828)
hi-thread-2504 [000] d..3 61513.835126: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
hi-thread-2504 [000] d..4 61513.835126: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
hi-thread-2504 [000] ...1 61513.835127: sys_futex -> 0x1
hi-thread-2504 [000] ..~1 61513.835128: cobalt_head_sysentry: syscall=migrate
hi-thread-2504 [000] ...1 61513.835128: cobalt_root_sysentry: syscall=migrate
hi-thread-2504 [000] ...1 61513.835128: cobalt_shadow_gohard: state=0x480c0 info=0x0
hi-thread-2504 [000] d..3 61513.835130: sched_switch: prev_comm=smokey prev_pid=2504 prev_prio=97 prev_state=S ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] *..3 61513.835130: cobalt_thread_resume: name=smokey pid=2504 mask=0x80
smokey-2499 [000] *..3 61513.835130: cobalt_trace_pid: pid=2504, prio=2
smokey-2499 [000] *.~3 61513.835131: cobalt_schedule: status=0x10000000
smokey-2499 [000] *.~3 61513.835131: cobalt_trace_pid: pid=2499, prio=-1
smokey-2499 [000] *.~3 61513.835131: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2504 next_prio=2
hi-thread-2504 [000] d.~3 61513.835132: cobalt_shadow_hardened: state=0x48040 info=0x0
hi-thread-2504 [000] d.~3 61513.835132: cobalt_root_sysexit: result=1
hi-thread-2504 [000] d.~3 61513.835133: cobalt_head_sysentry: syscall=thread_setname
hi-thread-2504 [000] d.~3 61513.835134: cobalt_pthread_setname: pth=00000000ba35a01d name=hi-thread
hi-thread-2504 [000] d.~3 61513.835135: cobalt_head_sysexit: result=0
hi-thread-2504 [000] d.~3 61513.835136: cobalt_head_sysentry: syscall=clock_nanosleep
hi-thread-2504 [000] d.~3 61513.835137: cobalt_clock_nanosleep: clock_id=0 flags=0x0() rqt=(0.001000000)
hi-thread-2504 [000] *.~3 61513.835138: cobalt_thread_suspend: pid=2504 mask=0x4 timeout=1000001 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] *.~3 61513.835138: cobalt_timer_start: timer=000000004d21df54(smokey) value=1000001 interval=0 mode=rel
hi-thread-2504 [000] *.~3 61513.835138: cobalt_schedule: status=0x10000000
hi-thread-2504 [000] *.~3 61513.835139: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] *.~3 61513.835139: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x48044 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
smokey-2499 [000] *.~3 61513.835140: cobalt_trace_pid: pid=2499, prio=-1
smokey-2499 [000] ...1 61513.835141: sys_futex -> 0x0
smokey-2499 [000] ..~1 61513.835144: cobalt_head_sysentry: syscall=migrate
smokey-2499 [000] ...1 61513.835144: cobalt_root_sysentry: syscall=migrate
smokey-2499 [000] ...1 61513.835144: cobalt_shadow_gohard: state=0x480c0 info=0x0
smokey-2499 [000] d..3 61513.835145: sched_stat_wait: comm=gdb pid=2495 delay=23982 [ns]
smokey-2499 [000] d..3 61513.835146: sched_switch: prev_comm=smokey prev_pid=2499 prev_prio=98 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] *..3 61513.835147: cobalt_thread_resume: name=smokey pid=2499 mask=0x80
gdb-2495 [000] *..3 61513.835147: cobalt_trace_pid: pid=2499, prio=1
gdb-2495 [000] *.~3 61513.835147: cobalt_schedule: status=0x10000000
gdb-2495 [000] *.~3 61513.835147: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] *.~3 61513.835148: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1
smokey-2499 [000] d.~3 61513.835148: cobalt_shadow_hardened: state=0x48040 info=0x0
smokey-2499 [000] d.~3 61513.835149: cobalt_root_sysexit: result=1
smokey-2499 [000] *.~3 61513.835219: irq_pipeline_entry: irq=4354
smokey-2499 [000] *.~3 61513.835219: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
smokey-2499 [000] *.~3 61513.835220: cobalt_timer_expire: timer=00000000d13b8aa9
smokey-2499 [000] *.~3 61513.835220: cobalt_tick_shot: next tick at 61513.836136 (delay: 915 us)
smokey-2499 [000] *.~3 61513.835221: irq_handler_exit: irq=4354 ret=handled
smokey-2499 [000] *.~3 61513.835221: irq_pipeline_exit: irq=4354
smokey-2499 [000] *.~3 61513.836137: irq_pipeline_entry: irq=4354
smokey-2499 [000] *.~3 61513.836137: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
smokey-2499 [000] *.~3 61513.836138: cobalt_timer_expire: timer=000000004d21df54
smokey-2499 [000] *.~3 61513.836138: cobalt_thread_resume: name=hi-thread pid=2504 mask=0x4
smokey-2499 [000] *.~3 61513.836138: cobalt_trace_pid: pid=2504, prio=2
smokey-2499 [000] *.~3 61513.836139: cobalt_timer_stop: timer=000000004d21df54
smokey-2499 [000] *.~3 61513.836139: irq_handler_exit: irq=4354 ret=handled
smokey-2499 [000] *.~3 61513.836139: irq_pipeline_exit: irq=4354
smokey-2499 [000] *.~3 61513.836140: cobalt_schedule: status=0x10000000
smokey-2499 [000] *.~3 61513.836140: cobalt_trace_pid: pid=2499, prio=1
smokey-2499 [000] *.~3 61513.836140: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x48048 ==> next_name=hi-thread next_pid=2504 next_prio=2
hi-thread-2504 [000] *.~3 61513.836141: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] d.~3 61513.836141: cobalt_head_sysexit: result=0
hi-thread-2504 [000] *.~3 61513.836142: cobalt_thread_fault: ip=0x55555540b580 type=0x3
hi-thread-2504 [000] *.~3 61513.836143: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] d.~3 61513.836143: cobalt_shadow_gorelax: reason=fault
hi-thread-2504 [000] d.~3 61513.836143: cobalt_lostage_request: request=wakeup pid=2504 comm=hi-thread
hi-thread-2504 [000] *.~3 61513.836144: cobalt_thread_suspend: pid=2504 mask=0x80 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] *.~3 61513.836144: cobalt_schedule: status=0x10000000
hi-thread-2504 [000] *.~3 61513.836144: cobalt_trace_pid: pid=2504, prio=2
hi-thread-2504 [000] *.~3 61513.836145: cobalt_switch_context: prev_name=hi-thread prev_pid=2504 prev_prio=2 prev_state=0x480c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
gdb-2495 [000] *.~3 61513.836146: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] d.h3 61513.836146: irq_handler_entry: irq=120 name=in-band work
gdb-2495 [000] d.h3 61513.836147: cobalt_lostage_wakeup: pid=2504 comm=hi-thread
gdb-2495 [000] d.h5 61513.836147: sched_waking: comm=hi-thread pid=2504 prio=97 target_cpu=000
gdb-2495 [000] dNh6 61513.836147: sched_wakeup: comm=hi-thread pid=2504 prio=97 target_cpu=000
gdb-2495 [000] dNh3 61513.836148: irq_handler_exit: irq=120 ret=handled
gdb-2495 [000] dNh3 61513.836148: irq_handler_entry: irq=124 name=proxy tick
gdb-2495 [000] dNh3 61513.836151: softirq_raise: vec=1 [action=TIMER]
gdb-2495 [000] dNh3 61513.836152: softirq_raise: vec=9 [action=RCU]
gdb-2495 [000] dNh4 61513.836152: sched_stat_runtime: comm=gdb pid=2495 runtime=1007474 [ns] vruntime=300515161229 [ns]
gdb-2495 [000] *Nh3 61513.836154: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2395843 interval=0 mode=rel
gdb-2495 [000] *Nh3 61513.836155: cobalt_tick_shot: next tick at 61513.838550 (delay: 2395 us)
gdb-2495 [000] dNh3 61513.836155: irq_handler_exit: irq=124 ret=handled
gdb-2495 [000] .Ns3 61513.836156: softirq_entry: vec=1 [action=TIMER]
gdb-2495 [000] .Ns3 61513.836157: softirq_exit: vec=1 [action=TIMER]
gdb-2495 [000] .Ns3 61513.836157: softirq_entry: vec=9 [action=RCU]
gdb-2495 [000] dNs6 61513.836158: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000
gdb-2495 [000] dNs7 61513.836159: sched_stat_runtime: comm=gdb pid=2495 runtime=6881 [ns] vruntime=300515168110 [ns]
gdb-2495 [000] dNs7 61513.836160: sched_stat_blocked: comm=rcu_sched pid=11 delay=4188739 [ns]
gdb-2495 [000] dNs7 61513.836161: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000
gdb-2495 [000] .Ns3 61513.836164: softirq_exit: vec=9 [action=RCU]
gdb-2495 [000] d..3 61513.836165: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=hi-thread next_pid=2504 next_prio=97
hi-thread-2504 [000] ...1 61513.836167: cobalt_shadow_relaxed: state=0x480c0 info=0x200
hi-thread-2504 [000] *..2 61513.836169: cobalt_thread_suspend: pid=2499 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] *..2 61513.836170: cobalt_thread_suspend: pid=2504 mask=0x400000 timeout=0 timeout_mode=0 wchan=0000000000000000
hi-thread-2504 [000] d..2 61513.836170: signal_generate: sig=5 errno=0 code=128 comm=hi-thread pid=2504 grp=0 res=0
hi-thread-2504 [000] d..3 61513.836173: signal_generate: sig=17 errno=0 code=4 comm=gdb pid=2495 grp=1 res=0
hi-thread-2504 [000] d..3 61513.836175: sched_stat_wait: comm=rcu_sched pid=11 delay=14490 [ns]
hi-thread-2504 [000] d..3 61513.836175: sched_switch: prev_comm=hi-thread prev_pid=2504 prev_prio=97 prev_state=t ==> next_comm=rcu_sched next_pid=11 next_prio=120
rcu_sched-11 [000] d..3 61513.836179: sched_stat_runtime: comm=rcu_sched pid=11 runtime=4960 [ns] vruntime=300512173070 [ns]
rcu_sched-11 [000] d..3 61513.836179: sched_stat_wait: comm=gdb pid=2495 delay=19450 [ns]
rcu_sched-11 [000] d..3 61513.836180: sched_switch: prev_comm=rcu_sched prev_pid=11 prev_prio=120 prev_state=I ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] ...1 61513.836187: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cb0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836197: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836198: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7ae0)
gdb-2495 [000] ...1 61513.836199: sys_newfstat -> 0x0
gdb-2495 [000] ...1 61513.836200: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
gdb-2495 [000] ...1 61513.836214: sys_read -> 0x3f1
gdb-2495 [000] ...1 61513.836217: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836217: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836221: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8)
gdb-2495 [000] ...1 61513.836222: sys_rt_sigprocmask -> 0x0
gdb-2495 [000] d..2 61513.836223: signal_deliver: sig=17 errno=0 code=4 sa_handler=5585fdc414d0 sa_flags=14000000
gdb-2495 [000] ...1 61513.836225: sys_read(fd: f, buf: 7ffe8b2a7667, count: 1)
gdb-2495 [000] ...1 61513.836226: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.836227: sys_write(fd: 10, buf: 5585fe0354b1, count: 1)
gdb-2495 [000] ...1 61513.836229: sys_write -> 0x1
gdb-2495 [000] ...1 61513.836229: sys_rt_sigreturn()
gdb-2495 [000] ...1 61513.836233: sys_read(fd: 9, buf: 7ffe8b2a82e7, count: 1)
gdb-2495 [000] ...1 61513.836234: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.836235: sys_write(fd: 1, buf: 5585fe7a5290, count: 27)
gdb-2495 [000] d..4 61513.836236: sched_waking: comm=smokey pid=2493 prio=120 target_cpu=000
gdb-2495 [000] d..5 61513.836237: sched_stat_runtime: comm=gdb pid=2495 runtime=58399 [ns] vruntime=300515226509 [ns]
gdb-2495 [000] d..5 61513.836238: sched_stat_sleep: comm=smokey pid=2493 delay=1717179 [ns]
gdb-2495 [000] dN.5 61513.836239: sched_wakeup: comm=smokey pid=2493 prio=120 target_cpu=000
gdb-2495 [000] .N.1 61513.836240: sys_write -> 0x27
gdb-2495 [000] dN.3 61513.836241: sched_stat_wait: comm=smokey pid=2493 delay=0 [ns]
gdb-2495 [000] d..3 61513.836241: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2493 next_prio=120
smokey-2493 [000] ...1 61513.836244: sys_read -> 0x1
smokey-2493 [000] ..~1 61513.836246: cobalt_head_sysentry: syscall=read
...
smokey-2493 [000] d..3 61513.836344: sched_stat_runtime: comm=smokey pid=2493 runtime=106636 [ns] vruntime=300512333145 [ns]
smokey-2493 [000] d..3 61513.836344: sched_stat_wait: comm=gdb pid=2495 delay=106636 [ns]
smokey-2493 [000] d..3 61513.836345: sched_switch: prev_comm=smokey prev_pid=2493 prev_prio=120 prev_state=S ==> next_comm=gdb next_pid=2495 next_prio=120
gdb-2495 [000] ...1 61513.836347: sys_poll(ufds: 5585fe898350, nfds: 4, timeout_msecs: 0)
gdb-2495 [000] ...1 61513.836348: sys_poll -> 0x1
gdb-2495 [000] ...1 61513.836350: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
gdb-2495 [000] ...1 61513.836351: sys_read -> 0x1
gdb-2495 [000] ...1 61513.836352: sys_read(fd: f, buf: 7ffe8b2a7f27, count: 1)
gdb-2495 [000] ...1 61513.836352: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.836353: sys_rt_sigprocmask(how: 0, nset: 5585fe2d4a60, oset: 7ffe8b2a7e60, sigsetsize: 8)
gdb-2495 [000] ...1 61513.836354: sys_rt_sigprocmask -> 0x0
gdb-2495 [000] ...1 61513.836355: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
gdb-2495 [000] ...1 61513.836355: sched_process_wait: comm=gdb pid=0 prio=120
gdb-2495 [000] ...1 61513.836357: sys_wait4 -> 0x9c8
gdb-2495 [000] ...1 61513.836360: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6ef0)
gdb-2495 [000] d..3 61513.836361: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836363: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836367: sys_ptrace(request: 4202, pid: 9c8, addr: 0, data: 7ffe8b2a7cb0)
gdb-2495 [000] d..3 61513.836368: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836369: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836370: sys_ptrace(request: c, pid: 9c8, addr: 0, data: 7ffe8b2a6f10)
gdb-2495 [000] d..3 61513.836370: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836371: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836372: sys_ptrace(request: d, pid: 9c8, addr: 0, data: 7ffe8b2a6f10)
gdb-2495 [000] d..3 61513.836373: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836374: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836376: sys_wait4(upid: ffffffff, stat_addr: 7ffe8b2a7db4, options: 40000001, ru: 0)
gdb-2495 [000] ...1 61513.836376: sched_process_wait: comm=gdb pid=0 prio=120
gdb-2495 [000] ...1 61513.836377: sys_wait4 -> 0x0
gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask(how: 2, nset: 7ffe8b2a7e60, oset: 0, sigsetsize: 8)
gdb-2495 [000] ...1 61513.836379: sys_rt_sigprocmask -> 0x0
gdb-2495 [000] ...1 61513.836382: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7cd0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836399: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836400: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7b50)
gdb-2495 [000] ...1 61513.836401: sys_newfstat -> 0x0
gdb-2495 [000] ...1 61513.836402: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
gdb-2495 [000] ...1 61513.836409: sys_read -> 0x140
gdb-2495 [000] ...1 61513.836410: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400)
gdb-2495 [000] ...1 61513.836410: sys_read -> 0x0
gdb-2495 [000] ...1 61513.836415: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836416: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836419: sys_read(fd: f, buf: 7ffe8b2a7f17, count: 1)
gdb-2495 [000] ...1 61513.836420: sys_read -> 0xfffffffffffffff5
gdb-2495 [000] ...1 61513.836421: sys_write(fd: 10, buf: 5585fe0354b1, count: 1)
gdb-2495 [000] ...1 61513.836422: sys_write -> 0x1
gdb-2495 [000] ...1 61513.836425: sys_ptrace(request: 1, pid: 9c8, addr: 7ffff777f370, data: 7ffe8b2a7878)
gdb-2495 [000] d..3 61513.836426: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836428: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836430: sys_ptrace(request: 3, pid: 9c8, addr: a8, data: 7ffe8b2a7bd8)
gdb-2495 [000] d..3 61513.836430: sched_wait_task: comm=hi-thread pid=2504 prio=97
gdb-2495 [000] ...1 61513.836431: sys_ptrace -> 0x0
gdb-2495 [000] ...1 61513.836433: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a70e0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836446: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836446: sys_pread64(fd: 12, buf: 7ffe8b2a7430, count: 900, pos: 7ffff67c9700)
gdb-2495 [000] ...1 61513.836449: sys_pread64 -> 0x900
gdb-2495 [000] ...1 61513.836450: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836450: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836479: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836488: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836489: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950)
gdb-2495 [000] ...1 61513.836490: sys_newfstat -> 0x0
gdb-2495 [000] ...1 61513.836490: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400)
gdb-2495 [000] ...1 61513.836498: sys_read -> 0x14d
gdb-2495 [000] ...1 61513.836499: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
gdb-2495 [000] ...1 61513.836499: sys_read -> 0x0
gdb-2495 [000] ...1 61513.836503: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836504: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836507: sys_openat(dfd: ffffff9c, filename: 7ffe8b2a7ad0, flags: 80000, mode: 0)
gdb-2495 [000] ...1 61513.836514: sys_openat -> 0x12
gdb-2495 [000] ...1 61513.836515: sys_newfstat(fd: 12, statbuf: 7ffe8b2a7950)
gdb-2495 [000] ...1 61513.836515: sys_newfstat -> 0x0
gdb-2495 [000] ...1 61513.836516: sys_read(fd: 12, buf: 5585fe9960e0, count: 400)
gdb-2495 [000] ...1 61513.836521: sys_read -> 0x13b
gdb-2495 [000] ...1 61513.836522: sys_read(fd: 12, buf: 5585fea6ccc0, count: 400)
gdb-2495 [000] ...1 61513.836522: sys_read -> 0x0
gdb-2495 [000] ...1 61513.836525: sys_close(fd: 12)
gdb-2495 [000] ...1 61513.836525: sys_close -> 0x0
gdb-2495 [000] ...1 61513.836529: sys_tkill(pid: 9c3, sig: 13)
gdb-2495 [000] *..3 61513.836532: cobalt_thread_resume: name=smokey pid=2499 mask=0x400000
gdb-2495 [000] *..3 61513.836532: cobalt_trace_pid: pid=2499, prio=1
gdb-2495 [000] *..3 61513.836533: cobalt_thread_unblock: pid=2499 state=0x48848 info=0x0
gdb-2495 [000] *.~3 61513.836586: cobalt_schedule: status=0x10000000
gdb-2495 [000] *.~3 61513.836586: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] *.~3 61513.836587: cobalt_switch_context: prev_name=ROOT/0 prev_pid=0 prev_prio=-1 prev_state=0x18008 ==> next_name=smokey next_pid=2499 next_prio=1
smokey-2499 [000] *.~3 61513.836589: cobalt_trace_pid: pid=2499, prio=1
smokey-2499 [000] *.~3 61513.836589: cobalt_shadow_gorelax: reason=undefined
smokey-2499 [000] *.~3 61513.836589: cobalt_lostage_request: request=wakeup pid=2499 comm=smokey
smokey-2499 [000] *.~3 61513.836590: cobalt_thread_suspend: pid=2499 mask=0x400080 timeout=0 timeout_mode=0 wchan=0000000000000000
smokey-2499 [000] *.~3 61513.836590: cobalt_schedule: status=0x10000000
smokey-2499 [000] *.~3 61513.836590: cobalt_trace_pid: pid=2499, prio=1
smokey-2499 [000] *.~3 61513.836591: cobalt_switch_context: prev_name=smokey prev_pid=2499 prev_prio=1 prev_state=0x4488c0 ==> next_name=ROOT/0 next_pid=0 next_prio=-1
gdb-2495 [000] *.~3 61513.836592: cobalt_trace_pid: pid=2495, prio=-1
gdb-2495 [000] d..5 61513.836592: sched_waking: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dN.6 61513.836594: sched_wakeup: comm=smokey pid=2499 prio=98 target_cpu=000
gdb-2495 [000] dN.3 61513.836594: signal_generate: sig=19 errno=0 code=-6 comm=smokey pid=2499 grp=0 res=0
gdb-2495 [000] dNh3 61513.836595: irq_handler_entry: irq=120 name=in-band work
gdb-2495 [000] dNh3 61513.836596: cobalt_lostage_wakeup: pid=2499 comm=smokey
gdb-2495 [000] *N~4 61513.838552: irq_pipeline_entry: irq=4354
gdb-2495 [000] *N~4 61513.838553: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
gdb-2495 [000] *N~4 61513.838554: cobalt_timer_expire: timer=00000000d13b8aa9
gdb-2495 [000] *N~4 61513.838555: irq_handler_exit: irq=4354 ret=handled
gdb-2495 [000] *N~4 61513.838555: irq_pipeline_exit: irq=4354
gdb-2495 [000] dNh8 61513.981277: sched_waking: comm=systemd-journal pid=163 prio=120 target_cpu=000
gdb-2495 [000] dNh9 61513.981278: sched_stat_runtime: comm=gdb pid=2495 runtime=144934239 [ns] vruntime=300660160748 [ns]
gdb-2495 [000] dNh9 61513.981280: sched_stat_sleep: comm=systemd-journal pid=163 delay=111416392895 [ns]
gdb-2495 [000] dNh9 61513.981280: sched_wakeup: comm=systemd-journal pid=163 prio=120 target_cpu=000
gdb-2495 [000] dNh3 61513.981281: irq_handler_exit: irq=120 ret=handled
gdb-2495 [000] dNh3 61513.981282: irq_handler_entry: irq=124 name=proxy tick
gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=1 [action=TIMER]
gdb-2495 [000] dNh3 61513.981285: softirq_raise: vec=9 [action=RCU]
gdb-2495 [000] dNh4 61513.981286: sched_stat_runtime: comm=gdb pid=2495 runtime=8375 [ns] vruntime=300660169123 [ns]
gdb-2495 [000] dNh5 61513.981288: sched_waking: comm=cobalt_printf pid=2494 prio=120 target_cpu=000
gdb-2495 [000] dNh6 61513.981289: sched_stat_runtime: comm=gdb pid=2495 runtime=2987 [ns] vruntime=300660172110 [ns]
gdb-2495 [000] dNh6 61513.981291: sched_stat_sleep: comm=cobalt_printf pid=2494 delay=238660319 [ns]
gdb-2495 [000] dNh6 61513.981291: sched_wakeup: comm=cobalt_printf pid=2494 prio=120 target_cpu=000
gdb-2495 [000] dNh5 61513.981292: sched_waking: comm=cobalt_printf pid=2503 prio=120 target_cpu=000
gdb-2495 [000] dNh6 61513.981292: sched_stat_runtime: comm=gdb pid=2495 runtime=3069 [ns] vruntime=300660175179 [ns]
gdb-2495 [000] dNh6 61513.981294: sched_stat_sleep: comm=cobalt_printf pid=2503 delay=146611442 [ns]
gdb-2495 [000] dNh6 61513.981294: sched_wakeup: comm=cobalt_printf pid=2503 prio=120 target_cpu=000
gdb-2495 [000] *Nh3 61513.981295: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=588476 interval=0 mode=rel
gdb-2495 [000] *Nh3 61513.981296: cobalt_tick_shot: next tick at 61513.981884 (delay: 588 us)
gdb-2495 [000] dNh3 61513.981296: irq_handler_exit: irq=124 ret=handled
gdb-2495 [000] .Ns3 61513.981297: softirq_entry: vec=1 [action=TIMER]
gdb-2495 [000] dNs5 61513.981298: sched_waking: comm=rcu_sched pid=11 prio=120 target_cpu=000
gdb-2495 [000] dNs6 61513.981298: sched_stat_runtime: comm=gdb pid=2495 runtime=6218 [ns] vruntime=300660181397 [ns]
gdb-2495 [000] dNs6 61513.981299: sched_stat_blocked: comm=rcu_sched pid=11 delay=145119923 [ns]
gdb-2495 [000] dNs6 61513.981299: sched_wakeup: comm=rcu_sched pid=11 prio=120 target_cpu=000
gdb-2495 [000] dNs7 61513.981302: sched_waking: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000
gdb-2495 [000] dNs8 61513.981302: sched_stat_runtime: comm=gdb pid=2495 runtime=3884 [ns] vruntime=300660185281 [ns]
gdb-2495 [000] dNs8 61513.981303: sched_stat_blocked: comm=kworker/0:1 pid=1932 delay=206048652 [ns]
gdb-2495 [000] dNs8 61513.981304: sched_wakeup: comm=kworker/0:1 pid=1932 prio=120 target_cpu=000
gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981309: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981310: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981311: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] dNs5 61513.981312: softirq_raise: vec=2 [action=NET_TX]
gdb-2495 [000] .Ns3 61513.981314: softirq_exit: vec=1 [action=TIMER]
gdb-2495 [000] .Ns3 61513.981314: softirq_entry: vec=9 [action=RCU]
gdb-2495 [000] .Ns3 61513.981315: softirq_exit: vec=9 [action=RCU]
gdb-2495 [000] dNs5 61513.981316: sched_waking: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000
gdb-2495 [000] dNs6 61513.981316: sched_stat_runtime: comm=gdb pid=2495 runtime=13841 [ns] vruntime=300660199122 [ns]
gdb-2495 [000] dNs6 61513.981317: sched_stat_sleep: comm=ksoftirqd/0 pid=10 delay=149349034 [ns]
gdb-2495 [000] dNs6 61513.981317: sched_wakeup: comm=ksoftirqd/0 pid=10 prio=120 target_cpu=000
gdb-2495 [000] .N.1 61513.981318: sys_tkill -> 0x0
gdb-2495 [000] d..3 61513.981320: sched_switch: prev_comm=gdb prev_pid=2495 prev_prio=120 prev_state=R ==> next_comm=smokey next_pid=2499 next_prio=98
smokey-2499 [000] *.~3 61513.981888: irq_pipeline_entry: irq=4354
smokey-2499 [000] *.~3 61513.981888: irq_handler_entry: irq=4354 name=Out-of-band LAPIC timer interrupt
smokey-2499 [000] *.~3 61513.981889: cobalt_timer_expire: timer=00000000d13b8aa9
smokey-2499 [000] *.~3 61513.981889: irq_handler_exit: irq=4354 ret=handled
smokey-2499 [000] *.~3 61513.981890: irq_pipeline_exit: irq=4354
smokey-2499 [000] d.h1 61514.106435: irq_handler_entry: irq=124 name=proxy tick
smokey-2499 [000] *.h1 61514.106441: cobalt_timer_start: timer=00000000d13b8aa9([host-timer/0]) value=2109534 interval=0 mode=rel
smokey-2499 [000] *.h1 61514.106441: cobalt_tick_shot: next tick at 61514.108550 (delay: 2109 us)
smokey-2499 [000] d.h1 61514.106442: irq_handler_exit: irq=124 ret=handled
smokey-2499 [000] ...1 61514.106442: cobalt_shadow_relaxed: state=0x4488c0 info=0x200
smokey-2499 [000] *.~1 61514.108552: irq_pipeline_entry: irq=4354
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: gdb test failure debug status update
[not found] ` <8735v82jmd.fsf@xenomai.org>
@ 2021-05-06 2:00 ` Chen, Hongzhan
2021-05-07 1:10 ` Chen, Hongzhan
0 siblings, 1 reply; 16+ messages in thread
From: Chen, Hongzhan @ 2021-05-06 2:00 UTC (permalink / raw)
To: Philippe Gerum, xenomai
>-----Original Message-----
>From: Philippe Gerum <rpm@xenomai.org>
>Sent: Friday, April 30, 2021 5:43 PM
>To: Chen, Hongzhan <hongzhan.chen@intel.com>
>Subject: Re: gdb test failure debug status update
>
>
>Chen, Hongzhan <hongzhan.chen@intel.com> writes:
>
>>>-----Original Message-----
>>>From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai
>>>Sent: Friday, April 30, 2021 4:07 PM
>>>To: Philippe Gerum <rpm@xenomai.org>
>>>Cc: xenomai@xenomai.org
>>>Subject: RE: gdb test failure debug status update
>>>
>>>
>>>
>>>>-----Original Message-----
>>>>From: Philippe Gerum <rpm@xenomai.org>
>>>>Sent: Friday, April 30, 2021 4:01 PM
>>>>To: Chen, Hongzhan <hongzhan.chen@intel.com>
>>>>Cc: xenomai@xenomai.org
>>>>Subject: Re: gdb test failure debug status update
>>>>
>>>>
>>>>Philippe Gerum <rpm@xenomai.org> writes:
>>>>
>>>>> Chen, Hongzhan <hongzhan.chen@intel.com> writes:
>>>>>
>>>>>> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish
>>>>>> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax.
>>>>>> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in
>>>>>> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from
>>>>>> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to
>>>>>> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode
>>>>>> because kernel exit code is never called in this case?
>>>>>>
>>>>>
>>>>> The IRQ frame is indeed kept from unwinding until the preempted task
>>>>> context returns from irq_exit_pipeline(), which branches to the Cobalt
>>>>> rescheduling procedure. From the Dovetail interface POV,
>>>>> irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow
>>>>> the companion core to perform post-IRQ chores, such as running its own
>>>>> rescheduling procedure.
>>>>>
>>>>> IOW, if task @foo is preempted by an IRQ, then suspended in oob context
>>>>> as a result of what the interrupt handler just did for it (e.g. raising
>>>>> XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then
>>>>> handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will
>>>>> cause the @foo context to switch away, effectively preventing
>>>>> handle_irq_pipelined_finish() to return, until @foo resumes execution
>>>> eventually.
>>
>> ln handle_irq_pipelined_finish, irq_exit_pipeline would at first be executed and it
>> handle dovetail_call_mayday in the end. But issue happen after run dovetail_call_mayday
>> because it call final xnthread_relax before gdb test failue.
>>
>
>Can you add WARN_ON(1) to dovetail_call_mayday() and report about the
>output? TIA,
>
>--
>Philippe.
>
Please check following output.
[ 27.260261] ------------[ cut here ]------------
[ 27.260262] WARNING: CPU: 0 PID: 400 at kernel/dovetail.c:93 dovetail_call_mayday+0x6/0x20
[ 27.260263] Modules linked in:
[ 27.260265] CPU: 0 PID: 400 Comm: smokey Tainted: G W 5.10.25+ #588
[ 27.260266] Hardware name: AAEON UP-WHL01/UP-WHL01, BIOS UPW1AM18D 06/23/2020
[ 27.260266] IRQ stage: Xenomai
[ 27.260267] RIP: 0010:dovetail_call_mayday+0x6/0x20
[ 27.260269] Code: 00 00 00 00 0f 1f 44 00 00 31 c0 c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 53 <0f> 0b 9c 5b fa f0 80 67 03 fb 48 89 f7 e8 48 04 02 00 53 9d 5b c3
[ 27.260270] RSP: 0000:ffffc900014c3f30 EFLAGS: 00010006
[ 27.260271] RAX: 0000000014000004 RBX: ffffc900014c3f58 RCX: ffff888264e1cf40
[ 27.260272] RDX: ffff888264e1cf40 RSI: ffffc900014c3f58 RDI: ffff88810376e300
[ 27.260273] RBP: ffff888264e1cf40 R08: 0000000000003904 R09: ffff88810b408000
[ 27.260274] R10: ffffc900014c3d48 R11: ffffffff8274a5e8 R12: ffffffff81c00c3a
[ 27.260274] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 27.260275] FS: 00007ffff758f740(0000) GS:ffff888264e00000(0000) knlGS:0000000000000000
[ 27.260276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 27.260277] CR2: 00007ff55f9909d8 CR3: 00000001156ea006 CR4: 00000000001706f0
[ 27.260278] Call Trace:
[ 27.260279] handle_irq_pipelined_finish+0x154/0x190
[ 27.260279] asm_sysvec_apic_timer_interrupt+0x12/0x20
[ 27.260280] RIP: 0033:0x55555540b4d5
[ 27.260281] Code: Unable to access opcode bytes at RIP 0x55555540b4ab.
[ 27.260282] RSP: 002b:00007fffffffe940 EFLAGS: 00000206
[ 27.260283] RAX: 0000000000043028 RBX: 000055555562a140 RCX: 0000000000000000
[ 27.260284] RDX: 000055555541fa4f RSI: 000000000000009f RDI: 000055555541fc68
[ 27.260285] RBP: 00007fffffffe950 R08: 0000000000000000 R09: 00000000ffffffff
[ 27.260285] R10: 0000000000000000 R11: 00007ffff799c820 R12: 0000000000000002
[ 27.260287] R13: 000055555562c9f0 R14: 00007fffffffeb04 R15: 00007ffff758f740
[ 27.260287] irq event stamp: 2000
[ 27.260288] hardirqs last enabled at (1999): [<ffffffff8105face>] flush_tlb_mm_range+0x10e/0x140
[ 27.260289] hardirqs last disabled at (2000): [<ffffffff81bd2264>] exc_int3+0x44/0x1a0
[ 27.260290] softirqs last enabled at (784): [<ffffffff81e0030a>] __do_softirq+0x30a/0x42d
[ 27.260291] softirqs last disabled at (777): [<ffffffff81c00faf>] asm_call_irq_on_stack+0xf/0x20
[ 27.260292] ---[ end trace ea56d34072e888ed ]---
Regards
Hongzhan Chen
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: gdb test failure debug status update
2021-05-06 2:00 ` Chen, Hongzhan
@ 2021-05-07 1:10 ` Chen, Hongzhan
2021-05-09 17:46 ` Philippe Gerum
0 siblings, 1 reply; 16+ messages in thread
From: Chen, Hongzhan @ 2021-05-07 1:10 UTC (permalink / raw)
To: Philippe Gerum, xenomai
>>
>>>>-----Original Message-----
>>>>From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai
>>>>Sent: Friday, April 30, 2021 4:07 PM
>>>>To: Philippe Gerum <rpm@xenomai.org>
>>>>Cc: xenomai@xenomai.org
>>>>Subject: RE: gdb test failure debug status update
>>>>
>>>>
>>>>
>>>>>-----Original Message-----
>>>>>From: Philippe Gerum <rpm@xenomai.org>
>>>>>Sent: Friday, April 30, 2021 4:01 PM
>>>>>To: Chen, Hongzhan <hongzhan.chen@intel.com>
>>>>>Cc: xenomai@xenomai.org
>>>>>Subject: Re: gdb test failure debug status update
>>>>>
>>>>>
>>>>>Philippe Gerum <rpm@xenomai.org> writes:
>>>>>
>>>>>> Chen, Hongzhan <hongzhan.chen@intel.com> writes:
>>>>>>
>>>>>>> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish
>>>>>>> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax.
>>>>>>> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in
>>>>>>> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from
>>>>>>> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to
>>>>>>> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode
>>>>>>> because kernel exit code is never called in this case?
>>>>>>>
>>>>>>
>>>>>> The IRQ frame is indeed kept from unwinding until the preempted task
>>>>>> context returns from irq_exit_pipeline(), which branches to the Cobalt
>>>>>> rescheduling procedure. From the Dovetail interface POV,
>>>>>> irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow
>>>>>> the companion core to perform post-IRQ chores, such as running its own
>>>>>> rescheduling procedure.
>>>>>>
>>>>>> IOW, if task @foo is preempted by an IRQ, then suspended in oob context
>>>>>> as a result of what the interrupt handler just did for it (e.g. raising
>>>>>> XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then
>>>>>> handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will
>>>>>> cause the @foo context to switch away, effectively preventing
>>>>>> handle_irq_pipelined_finish() to return, until @foo resumes execution
>>>>> eventually.
>>>
>>> ln handle_irq_pipelined_finish, irq_exit_pipeline would at first be executed and it
>>> handle dovetail_call_mayday in the end. But issue happen after run dovetail_call_mayday
>>> because it call final xnthread_relax before gdb test failue.
>>>
>>
>>Can you add WARN_ON(1) to dovetail_call_mayday() and report about the
>>output? TIA,
>>
>>--
>>Philippe.
>>
>
>Please check following output.
Hi Philippe,
Please let me know if you have new patch or other thing to let me try.
Regards
Hongzhan Chen
>
>[ 27.260261] ------------[ cut here ]------------
>[ 27.260262] WARNING: CPU: 0 PID: 400 at kernel/dovetail.c:93 dovetail_call_mayday+0x6/0x20
>[ 27.260263] Modules linked in:
>[ 27.260265] CPU: 0 PID: 400 Comm: smokey Tainted: G W 5.10.25+ #588
>[ 27.260266] Hardware name: AAEON UP-WHL01/UP-WHL01, BIOS UPW1AM18D 06/23/2020
>[ 27.260266] IRQ stage: Xenomai
>[ 27.260267] RIP: 0010:dovetail_call_mayday+0x6/0x20
>[ 27.260269] Code: 00 00 00 00 0f 1f 44 00 00 31 c0 c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 53 <0f> 0b 9c 5b fa f0 80 67 03 fb 48 89 f7 e8 48 04 02 00 53 9d 5b c3
>[ 27.260270] RSP: 0000:ffffc900014c3f30 EFLAGS: 00010006
>[ 27.260271] RAX: 0000000014000004 RBX: ffffc900014c3f58 RCX: ffff888264e1cf40
>[ 27.260272] RDX: ffff888264e1cf40 RSI: ffffc900014c3f58 RDI: ffff88810376e300
>[ 27.260273] RBP: ffff888264e1cf40 R08: 0000000000003904 R09: ffff88810b408000
>[ 27.260274] R10: ffffc900014c3d48 R11: ffffffff8274a5e8 R12: ffffffff81c00c3a
>[ 27.260274] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>[ 27.260275] FS: 00007ffff758f740(0000) GS:ffff888264e00000(0000) knlGS:0000000000000000
>[ 27.260276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>[ 27.260277] CR2: 00007ff55f9909d8 CR3: 00000001156ea006 CR4: 00000000001706f0
>[ 27.260278] Call Trace:
>[ 27.260279] handle_irq_pipelined_finish+0x154/0x190
>[ 27.260279] asm_sysvec_apic_timer_interrupt+0x12/0x20
>[ 27.260280] RIP: 0033:0x55555540b4d5
>[ 27.260281] Code: Unable to access opcode bytes at RIP 0x55555540b4ab.
>[ 27.260282] RSP: 002b:00007fffffffe940 EFLAGS: 00000206
>[ 27.260283] RAX: 0000000000043028 RBX: 000055555562a140 RCX: 0000000000000000
>[ 27.260284] RDX: 000055555541fa4f RSI: 000000000000009f RDI: 000055555541fc68
>[ 27.260285] RBP: 00007fffffffe950 R08: 0000000000000000 R09: 00000000ffffffff
>[ 27.260285] R10: 0000000000000000 R11: 00007ffff799c820 R12: 0000000000000002
>[ 27.260287] R13: 000055555562c9f0 R14: 00007fffffffeb04 R15: 00007ffff758f740
>[ 27.260287] irq event stamp: 2000
>[ 27.260288] hardirqs last enabled at (1999): [<ffffffff8105face>] flush_tlb_mm_range+0x10e/0x140
>[ 27.260289] hardirqs last disabled at (2000): [<ffffffff81bd2264>] exc_int3+0x44/0x1a0
>[ 27.260290] softirqs last enabled at (784): [<ffffffff81e0030a>] __do_softirq+0x30a/0x42d
>[ 27.260291] softirqs last disabled at (777): [<ffffffff81c00faf>] asm_call_irq_on_stack+0xf/0x20
>[ 27.260292] ---[ end trace ea56d34072e888ed ]---
>
>Regards
>
>Hongzhan Chen
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: gdb test failure debug status update
2021-05-07 1:10 ` Chen, Hongzhan
@ 2021-05-09 17:46 ` Philippe Gerum
2021-05-09 17:49 ` Philippe Gerum
2021-05-15 15:55 ` Philippe Gerum
0 siblings, 2 replies; 16+ messages in thread
From: Philippe Gerum @ 2021-05-09 17:46 UTC (permalink / raw)
To: Chen, Hongzhan; +Cc: xenomai
Chen, Hongzhan <hongzhan.chen@intel.com> writes:
>>>
>>>>>-----Original Message-----
>>>>>From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai
>>>>>Sent: Friday, April 30, 2021 4:07 PM
>>>>>To: Philippe Gerum <rpm@xenomai.org>
>>>>>Cc: xenomai@xenomai.org
>>>>>Subject: RE: gdb test failure debug status update
>>>>>
>>>>>
>>>>>
>>>>>>-----Original Message-----
>>>>>>From: Philippe Gerum <rpm@xenomai.org>
>>>>>>Sent: Friday, April 30, 2021 4:01 PM
>>>>>>To: Chen, Hongzhan <hongzhan.chen@intel.com>
>>>>>>Cc: xenomai@xenomai.org
>>>>>>Subject: Re: gdb test failure debug status update
>>>>>>
>>>>>>
>>>>>>Philippe Gerum <rpm@xenomai.org> writes:
>>>>>>
>>>>>>> Chen, Hongzhan <hongzhan.chen@intel.com> writes:
>>>>>>>
>>>>>>>> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish
>>>>>>>> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax.
>>>>>>>> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in
>>>>>>>> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from
>>>>>>>> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to
>>>>>>>> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode
>>>>>>>> because kernel exit code is never called in this case?
>>>>>>>>
>>>>>>>
>>>>>>> The IRQ frame is indeed kept from unwinding until the preempted task
>>>>>>> context returns from irq_exit_pipeline(), which branches to the Cobalt
>>>>>>> rescheduling procedure. From the Dovetail interface POV,
>>>>>>> irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow
>>>>>>> the companion core to perform post-IRQ chores, such as running its own
>>>>>>> rescheduling procedure.
>>>>>>>
>>>>>>> IOW, if task @foo is preempted by an IRQ, then suspended in oob context
>>>>>>> as a result of what the interrupt handler just did for it (e.g. raising
>>>>>>> XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then
>>>>>>> handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will
>>>>>>> cause the @foo context to switch away, effectively preventing
>>>>>>> handle_irq_pipelined_finish() to return, until @foo resumes execution
>>>>>> eventually.
>>>>
>>>> ln handle_irq_pipelined_finish, irq_exit_pipeline would at first be executed and it
>>>> handle dovetail_call_mayday in the end. But issue happen after run dovetail_call_mayday
>>>> because it call final xnthread_relax before gdb test failue.
>>>>
>>>
>>>Can you add WARN_ON(1) to dovetail_call_mayday() and report about the
>>>output? TIA,
>>>
>>>--
>>>Philippe.
>>>
>>
>>Please check following output.
>
> Hi Philippe,
>
> Please let me know if you have new patch or other thing to let me try.
>
I spent hours of this issue, and there may be a wrong basic assumption
done in the smokey/gdb test. Specifically, handle_sigwake_event()
un-stops the debuggee (lifting XNDBGSTOP), then sends a mayday notice to
make sure that debuggee re-enters the kernel asap for leaving the oob
stage. What might happen between these two events might not be as
well-defined as this test expects (e.g. what if the debugger might be
able to run more user code before the mayday trap is enforced?).
I'll keep on debugging that stuff and let you know.
--
Philippe.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: gdb test failure debug status update
2021-05-09 17:46 ` Philippe Gerum
@ 2021-05-09 17:49 ` Philippe Gerum
2021-05-10 2:16 ` Chen, Hongzhan
2021-05-15 15:55 ` Philippe Gerum
1 sibling, 1 reply; 16+ messages in thread
From: Philippe Gerum @ 2021-05-09 17:49 UTC (permalink / raw)
To: Chen, Hongzhan; +Cc: xenomai
Philippe Gerum <rpm@xenomai.org> writes:
> Chen, Hongzhan <hongzhan.chen@intel.com> writes:
>
>>>>
>>>>>>-----Original Message-----
>>>>>>From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai
>>>>>>Sent: Friday, April 30, 2021 4:07 PM
>>>>>>To: Philippe Gerum <rpm@xenomai.org>
>>>>>>Cc: xenomai@xenomai.org
>>>>>>Subject: RE: gdb test failure debug status update
>>>>>>
>>>>>>
>>>>>>
>>>>>>>-----Original Message-----
>>>>>>>From: Philippe Gerum <rpm@xenomai.org>
>>>>>>>Sent: Friday, April 30, 2021 4:01 PM
>>>>>>>To: Chen, Hongzhan <hongzhan.chen@intel.com>
>>>>>>>Cc: xenomai@xenomai.org
>>>>>>>Subject: Re: gdb test failure debug status update
>>>>>>>
>>>>>>>
>>>>>>>Philippe Gerum <rpm@xenomai.org> writes:
>>>>>>>
>>>>>>>> Chen, Hongzhan <hongzhan.chen@intel.com> writes:
>>>>>>>>
>>>>>>>>> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish
>>>>>>>>> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax.
>>>>>>>>> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in
>>>>>>>>> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from
>>>>>>>>> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to
>>>>>>>>> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode
>>>>>>>>> because kernel exit code is never called in this case?
>>>>>>>>>
>>>>>>>>
>>>>>>>> The IRQ frame is indeed kept from unwinding until the preempted task
>>>>>>>> context returns from irq_exit_pipeline(), which branches to the Cobalt
>>>>>>>> rescheduling procedure. From the Dovetail interface POV,
>>>>>>>> irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow
>>>>>>>> the companion core to perform post-IRQ chores, such as running its own
>>>>>>>> rescheduling procedure.
>>>>>>>>
>>>>>>>> IOW, if task @foo is preempted by an IRQ, then suspended in oob context
>>>>>>>> as a result of what the interrupt handler just did for it (e.g. raising
>>>>>>>> XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then
>>>>>>>> handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will
>>>>>>>> cause the @foo context to switch away, effectively preventing
>>>>>>>> handle_irq_pipelined_finish() to return, until @foo resumes execution
>>>>>>> eventually.
>>>>>
>>>>> ln handle_irq_pipelined_finish, irq_exit_pipeline would at first be executed and it
>>>>> handle dovetail_call_mayday in the end. But issue happen after run dovetail_call_mayday
>>>>> because it call final xnthread_relax before gdb test failue.
>>>>>
>>>>
>>>>Can you add WARN_ON(1) to dovetail_call_mayday() and report about the
>>>>output? TIA,
>>>>
>>>>--
>>>>Philippe.
>>>>
>>>
>>>Please check following output.
>>
>> Hi Philippe,
>>
>> Please let me know if you have new patch or other thing to let me try.
>>
>
> I spent hours of this issue, and there may be a wrong basic assumption
> done in the smokey/gdb test. Specifically, handle_sigwake_event()
> un-stops the debuggee (lifting XNDBGSTOP), then sends a mayday notice to
> make sure that debuggee re-enters the kernel asap for leaving the oob
> stage. What might happen between these two events might not be as
> well-defined as this test expects (e.g. what if the debugger might be
I meant the _debuggee_ (obvously not the debugger in this case)
> able to run more user code before the mayday trap is enforced?).
>
> I'll keep on debugging that stuff and let you know.
--
Philippe.
^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: gdb test failure debug status update
2021-05-09 17:49 ` Philippe Gerum
@ 2021-05-10 2:16 ` Chen, Hongzhan
0 siblings, 0 replies; 16+ messages in thread
From: Chen, Hongzhan @ 2021-05-10 2:16 UTC (permalink / raw)
To: Philippe Gerum; +Cc: xenomai
>Philippe Gerum <rpm@xenomai.org> writes:
>
>> Chen, Hongzhan <hongzhan.chen@intel.com> writes:
>>
>>>>>
>>>>>>>-----Original Message-----
>>>>>>>From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai
>>>>>>>Sent: Friday, April 30, 2021 4:07 PM
>>>>>>>To: Philippe Gerum <rpm@xenomai.org>
>>>>>>>Cc: xenomai@xenomai.org
>>>>>>>Subject: RE: gdb test failure debug status update
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>-----Original Message-----
>>>>>>>>From: Philippe Gerum <rpm@xenomai.org>
>>>>>>>>Sent: Friday, April 30, 2021 4:01 PM
>>>>>>>>To: Chen, Hongzhan <hongzhan.chen@intel.com>
>>>>>>>>Cc: xenomai@xenomai.org
>>>>>>>>Subject: Re: gdb test failure debug status update
>>>>>>>>
>>>>>>>>
>>>>>>>>Philippe Gerum <rpm@xenomai.org> writes:
>>>>>>>>
>>>>>>>>> Chen, Hongzhan <hongzhan.chen@intel.com> writes:
>>>>>>>>>
>>>>>>>>>> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish
>>>>>>>>>> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax.
>>>>>>>>>> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in
>>>>>>>>>> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from
>>>>>>>>>> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to
>>>>>>>>>> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode
>>>>>>>>>> because kernel exit code is never called in this case?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The IRQ frame is indeed kept from unwinding until the preempted task
>>>>>>>>> context returns from irq_exit_pipeline(), which branches to the Cobalt
>>>>>>>>> rescheduling procedure. From the Dovetail interface POV,
>>>>>>>>> irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow
>>>>>>>>> the companion core to perform post-IRQ chores, such as running its own
>>>>>>>>> rescheduling procedure.
>>>>>>>>>
>>>>>>>>> IOW, if task @foo is preempted by an IRQ, then suspended in oob context
>>>>>>>>> as a result of what the interrupt handler just did for it (e.g. raising
>>>>>>>>> XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then
>>>>>>>>> handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will
>>>>>>>>> cause the @foo context to switch away, effectively preventing
>>>>>>>>> handle_irq_pipelined_finish() to return, until @foo resumes execution
>>>>>>>> eventually.
>>>>>>
>>>>>> ln handle_irq_pipelined_finish, irq_exit_pipeline would at first be executed and it
>>>>>> handle dovetail_call_mayday in the end. But issue happen after run dovetail_call_mayday
>>>>>> because it call final xnthread_relax before gdb test failue.
>>>>>>
>>>>>
>>>>>Can you add WARN_ON(1) to dovetail_call_mayday() and report about the
>>>>>output? TIA,
>>>>>
>>>>>--
>>>>>Philippe.
>>>>>
>>>>
>>>>Please check following output.
>>>
>>> Hi Philippe,
>>>
>>> Please let me know if you have new patch or other thing to let me try.
>>>
>>
>> I spent hours of this issue, and there may be a wrong basic assumption
>> done in the smokey/gdb test. Specifically, handle_sigwake_event()
>> un-stops the debuggee (lifting XNDBGSTOP), then sends a mayday notice to
Current handle_sigwake_event call register_debugged_thread to suspend
all threads including smokey thread that be subordinated to same process when hi-thread get sigwake
event. Logically , we should relax all threads that be subordinated to same process after
get mayday event because we already specifically handle it before and then ask xnthread schedule to
schedule correct thread to run(which would schedule hi-thread at first finally and get correct behavior). Is this idea OK?
Regards
Hongzhan Chen
>> make sure that debuggee re-enters the kernel asap for leaving the oob
>> stage. What might happen between these two events might not be as
>> well-defined as this test expects (e.g. what if the debugger might be
>
>I meant the _debuggee_ (obvously not the debugger in this case)
>
>> able to run more user code before the mayday trap is enforced?).
>>
>> I'll keep on debugging that stuff and let you know.
>
>
>--
>Philippe.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: gdb test failure debug status update
2021-05-09 17:46 ` Philippe Gerum
2021-05-09 17:49 ` Philippe Gerum
@ 2021-05-15 15:55 ` Philippe Gerum
1 sibling, 0 replies; 16+ messages in thread
From: Philippe Gerum @ 2021-05-15 15:55 UTC (permalink / raw)
To: Chen, Hongzhan; +Cc: xenomai
Philippe Gerum <rpm@xenomai.org> writes:
> Chen, Hongzhan <hongzhan.chen@intel.com> writes:
>
>>>>
>>>>>>-----Original Message-----
>>>>>>From: Xenomai <xenomai-bounces@xenomai.org> On Behalf Of Chen, Hongzhan via Xenomai
>>>>>>Sent: Friday, April 30, 2021 4:07 PM
>>>>>>To: Philippe Gerum <rpm@xenomai.org>
>>>>>>Cc: xenomai@xenomai.org
>>>>>>Subject: RE: gdb test failure debug status update
>>>>>>
>>>>>>
>>>>>>
>>>>>>>-----Original Message-----
>>>>>>>From: Philippe Gerum <rpm@xenomai.org>
>>>>>>>Sent: Friday, April 30, 2021 4:01 PM
>>>>>>>To: Chen, Hongzhan <hongzhan.chen@intel.com>
>>>>>>>Cc: xenomai@xenomai.org
>>>>>>>Subject: Re: gdb test failure debug status update
>>>>>>>
>>>>>>>
>>>>>>>Philippe Gerum <rpm@xenomai.org> writes:
>>>>>>>
>>>>>>>> Chen, Hongzhan <hongzhan.chen@intel.com> writes:
>>>>>>>>
>>>>>>>>> The final xnthread_relaxed call path is like this asm_sysv_apic_timer_interrupt ->handle_irq_pipelined_finish
>>>>>>>>> ->dovetail_call_mayday ->handle_oob_mayday>xnthread_relax.
>>>>>>>>> That means that handle_irq_pipelined_finish is called under OOB condition of arch_pipeline_entry in
>>>>>>>>> arch/x86/kernel/irq_pipeline.c. Does that means that kernel entry/exit code is never called after return from
>>>>>>>>> xnthread_relax to handle_irq_pipelined_finish then to asm_sysv_apic_timer_interrupt? Even I enforce to
>>>>>>>>> call dovetail_request_ucall before calling final xnthread_relax system would not try to switch back to primary mode
>>>>>>>>> because kernel exit code is never called in this case?
>>>>>>>>>
>>>>>>>>
>>>>>>>> The IRQ frame is indeed kept from unwinding until the preempted task
>>>>>>>> context returns from irq_exit_pipeline(), which branches to the Cobalt
>>>>>>>> rescheduling procedure. From the Dovetail interface POV,
>>>>>>>> irq_exit_pipeline() is called by handle_irq_pipelined_finish() to allow
>>>>>>>> the companion core to perform post-IRQ chores, such as running its own
>>>>>>>> rescheduling procedure.
>>>>>>>>
>>>>>>>> IOW, if task @foo is preempted by an IRQ, then suspended in oob context
>>>>>>>> as a result of what the interrupt handler just did for it (e.g. raising
>>>>>>>> XNDBGSTOP, XNRELAX, XNPEND, XNSUSP in its state), then
>>>>>>>> handle_irq_pipelined_finish()->irq_exit_pipeline()->xnsched_run() will
>>>>>>>> cause the @foo context to switch away, effectively preventing
>>>>>>>> handle_irq_pipelined_finish() to return, until @foo resumes execution
>>>>>>> eventually.
>>>>>
>>>>> ln handle_irq_pipelined_finish, irq_exit_pipeline would at first be executed and it
>>>>> handle dovetail_call_mayday in the end. But issue happen after run dovetail_call_mayday
>>>>> because it call final xnthread_relax before gdb test failue.
>>>>>
>>>>
>>>>Can you add WARN_ON(1) to dovetail_call_mayday() and report about the
>>>>output? TIA,
>>>>
>>>>--
>>>>Philippe.
>>>>
>>>
>>>Please check following output.
>>
>> Hi Philippe,
>>
>> Please let me know if you have new patch or other thing to let me try.
>>
>
> I spent hours of this issue, and there may be a wrong basic assumption
> done in the smokey/gdb test. Specifically, handle_sigwake_event()
> un-stops the debuggee (lifting XNDBGSTOP), then sends a mayday notice to
> make sure that debuggee re-enters the kernel asap for leaving the oob
> stage. What might happen between these two events might not be as
> well-defined as this test expects (e.g. what if the debugger might be
> able to run more user code before the mayday trap is enforced?).
>
> I'll keep on debugging that stuff and let you know.
This one was nailed down eventually. As you found out, the "retuser"
event was missed and left pending. This could happen when a task resumes
to user, unwinding an IRQ frame, while being demoted from oob to in-band
context in the process.
This is typically the case of the gdb test:
[hi-pri-task]
(...timed sleep...)
[lo-spin-task]
(...spinning...)
TIMER-IRQENTRY
(wakeup hi-pri-task)
[hi-pri-task]
(breakpoint)
[gdb]
tkill(lo-spin-task, SIGTRAP)
handle_sigwake_event(lo-spin-task)
notify_mayday(lo-spin-task)
[lo-spin-task]
handle_mayday_event(lo-spin-task)
switch_inband()
TIMER-IRQEXIT
** missing check for pending _TIF_WORK|_TIF_RETUSER **
Which explains why lo-spin-task could run un-preempted by hi-pri-task
for a while, until the Cobalt core hits a rescheduling point eventually.
Only x86 is affected, ARM and arm64 have a different way out of IRQ
context which does not exhibit such issue.
This bug is now fixed by [1] for v5.10-dovetail. As I was at it, I added
a missing u-call request to the leave_oob helper.
Thanks for your help in digging into this.
[1] https://git.evlproject.org/linux-evl.git/commit/?h=dovetail/v5.10&id=cfcab38909d870d1ef484cd401fa00e52e86a8d0
--
Philippe.
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2021-05-15 15:55 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-28 3:19 gdb test failure debug status update Chen, Hongzhan
2021-04-28 14:18 ` Philippe Gerum
2021-04-28 14:30 ` Philippe Gerum
2021-04-29 6:31 ` Chen, Hongzhan
2021-04-30 5:25 ` Chen, Hongzhan
2021-04-30 5:51 ` Chen, Hongzhan
2021-04-30 7:36 ` Philippe Gerum
2021-04-30 7:34 ` Philippe Gerum
2021-04-30 8:00 ` Philippe Gerum
2021-04-30 8:07 ` Chen, Hongzhan
[not found] ` <DM5PR11MB18529649C47BF241930A2217F25E9@DM5PR11MB1852.namprd11.prod.outlook.com>
[not found] ` <8735v82jmd.fsf@xenomai.org>
2021-05-06 2:00 ` Chen, Hongzhan
2021-05-07 1:10 ` Chen, Hongzhan
2021-05-09 17:46 ` Philippe Gerum
2021-05-09 17:49 ` Philippe Gerum
2021-05-10 2:16 ` Chen, Hongzhan
2021-05-15 15:55 ` Philippe Gerum
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.