* gdb test failure issue on xenomai 3.2 @ 2021-03-02 11:20 Chen, Hongzhan 2021-03-05 1:15 ` Chen, Hongzhan 0 siblings, 1 reply; 3+ messages in thread From: Chen, Hongzhan @ 2021-03-02 11:20 UTC (permalink / raw) To: Philippe Gerum, xenomai Hi Philippe, The gdb test fails on xenomai 3.2 because low priority smokey process still has chance to scheduled to execute it's user space's task. According to my debug , it seems it is related to handle_ptrace_resume of dovetail/kevents.c because in failed case it would try to resume hi-thread like stack [1] and then hi-thread call xnthread_harden like stack [2]. But on IPIPE-xenomai image it would resume smokey thread at first and then xnthread_harden would be called in smokey thread in the following. The problem is that I do know why in failed case it would xnthread_resume hi-thread at first like log[3] but with IPIPE-xenomai image it always xnthread_resume smokey non-rt thread like in log [4] so that I still do not know how to fix it. I still need more time to debug it out. Please comment. In addition , I found that there is comments in handle_ptstep_event of kernel/evl/thread.c. It said that "the ptracer might have switched focus" but in gdb test case breakpoint exactly switch to hi-thread from smokey process . Does that means we need to do some special handle for it ? But for me evl is quite so different from xenomai on handling INBAND_TASK_PTSTEP in handle_inband_event. [1]: 0xffffffff811c4020 xnthread_resume(): kernel/xenomai/thread.c, line 1039 0xffffffff811b51e0 handle_ptrace_resume(): ...ernel/xenomai/pipeline/kevents.c, line 474 0xffffffff81078bec inband_ptstep_notify(): ./include/linux/dovetail.h, line 121 0xffffffff81078bec ptrace_resume(): kernel/ptrace.c, line 830 0xffffffff81079f32 ptrace_request(): kernel/ptrace.c, line 1194 0xffffffff8102eaab arch_ptrace(): arch/x86/kernel/ptrace.c, line 818 0xffffffff8107949b __do_sys_ptrace(): kernel/ptrace.c, line 1279 0xffffffff8107949b __se_sys_ptrace(): kernel/ptrace.c, line 1244 0xffffffff8107949b __x64_sys_ptrace(): kernel/ptrace.c, line 1244 0xffffffff81b93549 do_syscall_64(): arch/x86/entry/common.c, line 53 0xffffffff81c0007c entry_SYSCALL_64_after_hwframe(): arch/x86/entry/entry_64.S, line 118 [2]: 0xffffffff811c6340 xnthread_harden(): kernel/xenomai/thread.c, line 1864 0xffffffff811b56fa handle_user_return(): ...ernel/xenomai/pipeline/kevents.c, line 339 0xffffffff811b56fa handle_inband_event(): ...ernel/xenomai/pipeline/kevents.c, line 514 0xffffffff8110791e do_retuser(): kernel/entry/common.c, line 238 0xffffffff8110791e exit_to_user_mode_prepare(): kernel/entry/common.c, line 262 0xffffffff81b97795 irqentry_exit_to_user_mode(): kernel/entry/common.c, line 356 0xffffffff81b941dc exc_int3(): arch/x86/kernel/traps.c, line 710 0xffffffff81c00aa1 asm_exc_int3(): .../arch/x86/include/asm/idtentry.h, line 592 [3]: gdb-1153 [003] d..3 17451.761941: sched_wait_task: comm=hi-thread pid=1160 prio=97 gdb-1153 [003] d..3 17451.761944: sched_wait_task: comm=hi-thread pid=1160 prio=97 gdb-1153 [003] d..3 17451.761947: sched_wait_task: comm=hi-thread pid=1160 prio=97 gdb-1153 [003] d..3 17451.761953: sched_wait_task: comm=hi-thread pid=1160 prio=97 gdb-1153 [003] d..3 17451.761958: sched_wait_task: comm=hi-thread pid=1160 prio=97 gdb-1153 [003] d..3 17451.761973: sched_wait_task: comm=hi-thread pid=1160 prio=97 gdb-1153 [003] D..1 17451.761976: cobalt_thread_resume: name=hi-thread pid=1160 mask=0x400000 gdb-1153 [003] D..1 17451.761977: cobalt_trace_pid: pid=1160, prio=2 gdb-1153 [003] d..3 17451.761978: sched_waking: comm=hi-thread pid=1160 prio=97 target_cpu=000 [4] gdb-839 [000] d..1 2420.981050: signal_deliver: sig=17 errno=0 code=4 sa_handler=55bb958074d0 sa_flags=14000000 gdb-839 [000] .... 2420.981069: sched_process_wait: comm=gdb pid=0 prio=120 gdb-839 [000] d..2 2420.981075: sched_wait_task: comm=hi-thread pid=846 prio=97 gdb-839 [000] d..2 2420.981080: sched_wait_task: comm=hi-thread pid=846 prio=97 gdb-839 [000] d..2 2420.981082: sched_wait_task: comm=hi-thread pid=846 prio=97 gdb-839 [000] d..2 2420.981084: sched_wait_task: comm=hi-thread pid=846 prio=97 gdb-839 [000] .... 2420.981087: sched_process_wait: comm=gdb pid=0 prio=120 gdb-839 [000] d..2 2420.981147: sched_wait_task: comm=hi-thread pid=846 prio=97 gdb-839 [000] d..2 2420.981150: sched_wait_task: comm=hi-thread pid=846 prio=97 gdb-839 [000] d..1 2420.981303: cobalt_thread_resume: name=smokey pid=841 mask=0x400000 gdb-839 [000] d..1 2420.981304: cobalt_thread_unblock: pid=841 state=0x48848 info=0x0 gdb-839 [000] d..1 2420.981305: cobalt_schedule_remote: status=0x10000000 gdb-839 [000] d..1 2420.981305: cobalt_schedule: status=0x10000000 Regards Hongzhan Chen ^ permalink raw reply [flat|nested] 3+ messages in thread
* RE: gdb test failure issue on xenomai 3.2 2021-03-02 11:20 gdb test failure issue on xenomai 3.2 Chen, Hongzhan @ 2021-03-05 1:15 ` Chen, Hongzhan 2021-03-05 10:56 ` Philippe Gerum 0 siblings, 1 reply; 3+ messages in thread From: Chen, Hongzhan @ 2021-03-05 1:15 UTC (permalink / raw) To: Philippe Gerum, xenomai Hi Philippe, I think I already found the root cause and almost fix it but still need time to validate it. Temporary patch like following: diff --git a/kernel/cobalt/dovetail/sched.c b/kernel/cobalt/dovetail/sched.c index de7c43b70..846b571b6 100644 --- a/kernel/cobalt/dovetail/sched.c +++ b/kernel/cobalt/dovetail/sched.c @@ -56,9 +56,22 @@ int pipeline_leave_inband(void) int pipeline_leave_oob_prepare(void) { - dovetail_leave_oob(); + int suspmask = XNRELAX; + struct xnthread *curr = xnthread_current(); - return XNRELAX; + dovetail_leave_oob(); + /* + * If current is being debugged, record that it should migrate + * back in case it resumes in userspace. If it resumes in + * kernel space, i.e. over a restarting syscall, the + * associated hardening will both clear XNCONTHI and disable + * the user return notifier again. + */ + if (xnthread_test_state(curr, XNSSTEP)) { + xnthread_set_info(curr, XNCONTHI); + suspmask |= XNDBGSTOP; + } + return suspmask; } void pipeline_leave_oob_finish(void) intel@intel-Z97X-UD5H:~/iotg/dovetail/xenomaioverdovetail/cobalt-dovetail/upstream/xenomai-rpm$ git diff kernel/cobalt/dovetail/kevents.c diff --git a/kernel/cobalt/dovetail/kevents.c b/kernel/cobalt/dovetail/kevents.c index 966a63ce0..9bceffaac 100644 --- a/kernel/cobalt/dovetail/kevents.c +++ b/kernel/cobalt/dovetail/kevents.c if (xnthread_test_info(thread, XNCONTHI)) { xnlock_get_irqsave(&nklock, s); xnthread_clear_info(thread, XNCONTHI); @@ -492,6 +494,8 @@ static void handle_ptrace_cont(void) unregister_debugged_thread(curr); xnthread_set_localinfo(curr, XNHICCUP); + + dovetail_request_ucall(current); } After validation is done , I would submit clean patch to review. Regards Hongzhan Chen -----Original Message----- From: Chen, Hongzhan Sent: Tuesday, March 2, 2021 7:21 PM To: Philippe Gerum <rpm@xenomai.org>; xenomai@xenomai.org Subject: gdb test failure issue on xenomai 3.2 Hi Philippe, The gdb test fails on xenomai 3.2 because low priority smokey process still has chance to scheduled to execute it's user space's task. According to my debug , it seems it is related to handle_ptrace_resume of dovetail/kevents.c because in failed case it would try to resume hi-thread like stack [1] and then hi-thread call xnthread_harden like stack [2]. But on IPIPE-xenomai image it would resume smokey thread at first and then xnthread_harden would be called in smokey thread in the following. The problem is that I do know why in failed case it would xnthread_resume hi-thread at first like log[3] but with IPIPE-xenomai image it always xnthread_resume smokey non-rt thread like in log [4] so that I still do not know how to fix it. I still need more time to debug it out. Please comment. In addition , I found that there is comments in handle_ptstep_event of kernel/evl/thread.c. It said that "the ptracer might have switched focus" but in gdb test case breakpoint exactly switch to hi-thread from smokey process . Does that means we need to do some special handle for it ? But for me evl is quite so different from xenomai on handling INBAND_TASK_PTSTEP in handle_inband_event. [1]: 0xffffffff811c4020 xnthread_resume(): kernel/xenomai/thread.c, line 1039 0xffffffff811b51e0 handle_ptrace_resume(): ...ernel/xenomai/pipeline/kevents.c, line 474 0xffffffff81078bec inband_ptstep_notify(): ./include/linux/dovetail.h, line 121 0xffffffff81078bec ptrace_resume(): kernel/ptrace.c, line 830 0xffffffff81079f32 ptrace_request(): kernel/ptrace.c, line 1194 0xffffffff8102eaab arch_ptrace(): arch/x86/kernel/ptrace.c, line 818 0xffffffff8107949b __do_sys_ptrace(): kernel/ptrace.c, line 1279 0xffffffff8107949b __se_sys_ptrace(): kernel/ptrace.c, line 1244 0xffffffff8107949b __x64_sys_ptrace(): kernel/ptrace.c, line 1244 0xffffffff81b93549 do_syscall_64(): arch/x86/entry/common.c, line 53 0xffffffff81c0007c entry_SYSCALL_64_after_hwframe(): arch/x86/entry/entry_64.S, line 118 [2]: 0xffffffff811c6340 xnthread_harden(): kernel/xenomai/thread.c, line 1864 0xffffffff811b56fa handle_user_return(): ...ernel/xenomai/pipeline/kevents.c, line 339 0xffffffff811b56fa handle_inband_event(): ...ernel/xenomai/pipeline/kevents.c, line 514 0xffffffff8110791e do_retuser(): kernel/entry/common.c, line 238 0xffffffff8110791e exit_to_user_mode_prepare(): kernel/entry/common.c, line 262 0xffffffff81b97795 irqentry_exit_to_user_mode(): kernel/entry/common.c, line 356 0xffffffff81b941dc exc_int3(): arch/x86/kernel/traps.c, line 710 0xffffffff81c00aa1 asm_exc_int3(): .../arch/x86/include/asm/idtentry.h, line 592 [3]: gdb-1153 [003] d..3 17451.761941: sched_wait_task: comm=hi-thread pid=1160 prio=97 gdb-1153 [003] d..3 17451.761944: sched_wait_task: comm=hi-thread pid=1160 prio=97 gdb-1153 [003] d..3 17451.761947: sched_wait_task: comm=hi-thread pid=1160 prio=97 gdb-1153 [003] d..3 17451.761953: sched_wait_task: comm=hi-thread pid=1160 prio=97 gdb-1153 [003] d..3 17451.761958: sched_wait_task: comm=hi-thread pid=1160 prio=97 gdb-1153 [003] d..3 17451.761973: sched_wait_task: comm=hi-thread pid=1160 prio=97 gdb-1153 [003] D..1 17451.761976: cobalt_thread_resume: name=hi-thread pid=1160 mask=0x400000 gdb-1153 [003] D..1 17451.761977: cobalt_trace_pid: pid=1160, prio=2 gdb-1153 [003] d..3 17451.761978: sched_waking: comm=hi-thread pid=1160 prio=97 target_cpu=000 [4] gdb-839 [000] d..1 2420.981050: signal_deliver: sig=17 errno=0 code=4 sa_handler=55bb958074d0 sa_flags=14000000 gdb-839 [000] .... 2420.981069: sched_process_wait: comm=gdb pid=0 prio=120 gdb-839 [000] d..2 2420.981075: sched_wait_task: comm=hi-thread pid=846 prio=97 gdb-839 [000] d..2 2420.981080: sched_wait_task: comm=hi-thread pid=846 prio=97 gdb-839 [000] d..2 2420.981082: sched_wait_task: comm=hi-thread pid=846 prio=97 gdb-839 [000] d..2 2420.981084: sched_wait_task: comm=hi-thread pid=846 prio=97 gdb-839 [000] .... 2420.981087: sched_process_wait: comm=gdb pid=0 prio=120 gdb-839 [000] d..2 2420.981147: sched_wait_task: comm=hi-thread pid=846 prio=97 gdb-839 [000] d..2 2420.981150: sched_wait_task: comm=hi-thread pid=846 prio=97 gdb-839 [000] d..1 2420.981303: cobalt_thread_resume: name=smokey pid=841 mask=0x400000 gdb-839 [000] d..1 2420.981304: cobalt_thread_unblock: pid=841 state=0x48848 info=0x0 gdb-839 [000] d..1 2420.981305: cobalt_schedule_remote: status=0x10000000 gdb-839 [000] d..1 2420.981305: cobalt_schedule: status=0x10000000 Regards Hongzhan Chen ^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: gdb test failure issue on xenomai 3.2 2021-03-05 1:15 ` Chen, Hongzhan @ 2021-03-05 10:56 ` Philippe Gerum 0 siblings, 0 replies; 3+ messages in thread From: Philippe Gerum @ 2021-03-05 10:56 UTC (permalink / raw) To: Chen, Hongzhan; +Cc: xenomai Hi Hongzhan, Sorry for the lag in replying. Chen, Hongzhan <hongzhan.chen@intel.com> writes: > Hi Philippe, > > I think I already found the root cause and almost fix it but still need time to validate it. > > Temporary patch like following: > > diff --git a/kernel/cobalt/dovetail/sched.c b/kernel/cobalt/dovetail/sched.c > index de7c43b70..846b571b6 100644 > --- a/kernel/cobalt/dovetail/sched.c > +++ b/kernel/cobalt/dovetail/sched.c > @@ -56,9 +56,22 @@ int pipeline_leave_inband(void) > > int pipeline_leave_oob_prepare(void) > { > - dovetail_leave_oob(); > + int suspmask = XNRELAX; > + struct xnthread *curr = xnthread_current(); > > - return XNRELAX; > + dovetail_leave_oob(); > + /* > + * If current is being debugged, record that it should migrate > + * back in case it resumes in userspace. If it resumes in > + * kernel space, i.e. over a restarting syscall, the > + * associated hardening will both clear XNCONTHI and disable > + * the user return notifier again. > + */ > + if (xnthread_test_state(curr, XNSSTEP)) { > + xnthread_set_info(curr, XNCONTHI); > + suspmask |= XNDBGSTOP; > + } > + return suspmask; > } Yes, that part was definitely missing. I guess I dropped it mistakenly due to the condition on the obsolete IPIPE_KEVT_USERINTRET check. > > void pipeline_leave_oob_finish(void) > intel@intel-Z97X-UD5H:~/iotg/dovetail/xenomaioverdovetail/cobalt-dovetail/upstream/xenomai-rpm$ git diff kernel/cobalt/dovetail/kevents.c > diff --git a/kernel/cobalt/dovetail/kevents.c b/kernel/cobalt/dovetail/kevents.c > index 966a63ce0..9bceffaac 100644 > --- a/kernel/cobalt/dovetail/kevents.c > +++ b/kernel/cobalt/dovetail/kevents.c > if (xnthread_test_info(thread, XNCONTHI)) { > xnlock_get_irqsave(&nklock, s); > xnthread_clear_info(thread, XNCONTHI); > @@ -492,6 +494,8 @@ static void handle_ptrace_cont(void) > unregister_debugged_thread(curr); > > xnthread_set_localinfo(curr, XNHICCUP); > + > + dovetail_request_ucall(current); > } > Looks good to me too. Once the ptrace core tells us that current is resuming from a stopped state, this is the right place to ask for switching back to oob mode. > After validation is done , I would submit clean patch to review. Thanks for looking at this one. This is quite tricky stuff. -- Philippe. ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-03-05 10:56 UTC | newest] Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-03-02 11:20 gdb test failure issue on xenomai 3.2 Chen, Hongzhan 2021-03-05 1:15 ` Chen, Hongzhan 2021-03-05 10:56 ` Philippe Gerum
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.