Hi Steven, FYI, here is another bisect result. The attached reproduce-* may help debug the issue. https://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git ftrace/core commit 42c269c88dc146982a54a8267f71abc99f12852a Author: Steven Rostedt (VMware) AuthorDate: Fri Mar 3 16:15:39 2017 -0500 Commit: Steven Rostedt (VMware) CommitDate: Fri Mar 24 20:51:49 2017 -0400 ftrace: Allow for function tracing to record init functions on boot up Adding a hook into free_reserve_area() that informs ftrace that boot up init text is being free, lets ftrace safely remove those init functions from its records, which keeps ftrace from trying to modify text that no longer exists. Note, this still does not allow for tracing .init text of modules, as modules require different work for freeing its init code. Link: http://lkml.kernel.org/r/1488502497.7212.24.camel(a)linux.intel.com Cc: linux-mm(a)kvack.org Cc: Vlastimil Babka Cc: Mel Gorman Cc: Peter Zijlstra Requested-by: Todd Brandt Signed-off-by: Steven Rostedt (VMware) dbeafd0d61 ftrace: Have function tracing start in early boot up 42c269c88d ftrace: Allow for function tracing to record init functions on boot up 2b87965a1e ftrace/x86: Do no run CPU sync when there is only one CPU online +-----------------------------------------------------------------------------------------+------------+------------+------------+ | | dbeafd0d61 | 42c269c88d | 2b87965a1e | +-----------------------------------------------------------------------------------------+------------+------------+------------+ | boot_successes | 914 | 301 | 308 | | boot_failures | 0 | 11 | 5 | | WARNING:at_kernel/trace/ftrace.c:#ftrace_bug | 0 | 8 | 5 | | kernel_BUG_at_arch/x86/kernel/ftrace.c | 0 | 8 | 5 | | invalid_opcode:#[##] | 0 | 8 | 5 | | Kernel_panic-not_syncing:Fatal_exception | 0 | 8 | 5 | | page_allocation_failure:order:#,mode:#(GFP_KERNEL|__GFP_NORETRY),nodemask=(null) | 0 | 1 | | | invoked_oom-killer:gfp_mask=0x | 0 | 2 | | | Mem-Info | 0 | 2 | | | Out_of_memory:Kill_process | 0 | 2 | | | page_allocation_failure:order:#,mode:#(GFP_HIGHUSER_MOVABLE|__GFP_ZERO),nodemask=(null) | 0 | 1 | | | page_allocation_failure:order:#,mode:#(GFP_HIGHUSER|__GFP_ACCOUNT),nodemask=(null) | 0 | 1 | | +-----------------------------------------------------------------------------------------+------------+------------+------------+ [main] 375 sockets created based on info from socket cachefile. [ 19.774381] ftrace: Failed on adding breakpoints (21794): [ 19.774381] ftrace: Failed on adding breakpoints (21794): [ 19.784712] ------------[ cut here ]------------ [ 19.784712] ------------[ cut here ]------------ [ 19.786756] WARNING: CPU: 0 PID: 4062 at kernel/trace/ftrace.c:2019 ftrace_bug+0x86/0x3f3 [ 19.786756] WARNING: CPU: 0 PID: 4062 at kernel/trace/ftrace.c:2019 ftrace_bug+0x86/0x3f3 [ 19.796219] Modules linked in: [ 19.796219] Modules linked in: [ 19.797642] CPU: 0 PID: 4062 Comm: trinity-main Not tainted 4.11.0-rc3-00005-g42c269c #2 [ 19.797642] CPU: 0 PID: 4062 Comm: trinity-main Not tainted 4.11.0-rc3-00005-g42c269c #2 [ 19.808709] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-20161025_171302-gandalf 04/01/2014 [ 19.808709] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-20161025_171302-gandalf 04/01/2014 [ 19.824447] Call Trace: [ 19.824447] Call Trace: [ 19.825205] dump_stack+0x19/0x1b [ 19.825205] dump_stack+0x19/0x1b [ 19.826117] __warn+0xc0/0xdb [ 19.826117] __warn+0xc0/0xdb [ 19.826952] ? early_idt_handler_common+0x52/0x52 [ 19.826952] ? early_idt_handler_common+0x52/0x52 [ 19.834097] warn_slowpath_null+0x1d/0x1f [ 19.834097] warn_slowpath_null+0x1d/0x1f [ 19.835232] ftrace_bug+0x86/0x3f3 [ 19.835232] ftrace_bug+0x86/0x3f3 [ 19.836313] ftrace_replace_code+0x213/0x33b [ 19.836313] ftrace_replace_code+0x213/0x33b [ 19.837496] ftrace_modify_all_code+0x8f/0x143 [ 19.837496] ftrace_modify_all_code+0x8f/0x143 [ 19.838696] arch_ftrace_update_code+0xf/0x17 [ 19.838696] arch_ftrace_update_code+0xf/0x17 [ 19.848741] ftrace_run_update_code+0x6a/0xc0 [ 19.848741] ftrace_run_update_code+0x6a/0xc0 [ 19.849961] ftrace_startup_enable+0x32/0x34 [ 19.849961] ftrace_startup_enable+0x32/0x34 [ 19.851114] ftrace_startup+0x1d0/0x1f2 [ 19.851114] ftrace_startup+0x1d0/0x1f2 [ 19.884646] register_ftrace_function+0x4f/0x65 [ 19.884646] register_ftrace_function+0x4f/0x65 [ 19.885889] perf_ftrace_event_register+0x48/0x1f0 [ 19.885889] perf_ftrace_event_register+0x48/0x1f0 [ 19.887173] perf_trace_init+0x22b/0x259 [ 19.887173] perf_trace_init+0x22b/0x259 [ 19.892452] perf_tp_event_init+0x29/0x3b [ 19.892452] perf_tp_event_init+0x29/0x3b [ 19.893865] perf_try_init_event+0x60/0x92 [ 19.893865] perf_try_init_event+0x60/0x92 [ 19.895302] perf_event_alloc+0x641/0xb16 [ 19.895302] perf_event_alloc+0x641/0xb16 [ 19.904253] SyS_perf_event_open+0x586/0xff4 [ 19.904253] SyS_perf_event_open+0x586/0xff4 [ 19.906719] entry_SYSCALL_64_fastpath+0x1f/0xc1 [ 19.906719] entry_SYSCALL_64_fastpath+0x1f/0xc1 [ 19.917213] RIP: 0033:0x7f2e7e8e46d9 [ 19.917213] RIP: 0033:0x7f2e7e8e46d9 [ 19.918614] RSP: 002b:00007fffaed60db8 EFLAGS: 00000246 ORIG_RAX: 000000000000012a [ 19.918614] RSP: 002b:00007fffaed60db8 EFLAGS: 00000246 ORIG_RAX: 000000000000012a [ 19.926216] RAX: ffffffffffffffda RBX: 00007f2e7ebac720 RCX: 00007f2e7e8e46d9 [ 19.926216] RAX: ffffffffffffffda RBX: 00007f2e7ebac720 RCX: 00007f2e7e8e46d9 [ 19.928858] RDX: ffffffffffffffff RSI: 0000000000000fde RDI: 000000000068b000 [ 19.928858] RDX: ffffffffffffffff RSI: 0000000000000fde RDI: 000000000068b000 [ 19.931256] RBP: 0000000000002011 R08: 0000000000000000 R09: 0000000000000004 [ 19.931256] RBP: 0000000000002011 R08: 0000000000000000 R09: 0000000000000004 [ 19.940870] R10: ffffffffffffffff R11: 0000000000000246 R12: 00007f2e7ebac778 [ 19.940870] R10: ffffffffffffffff R11: 0000000000000246 R12: 00007f2e7ebac778 [ 19.943283] R13: 0000000000002000 R14: 0000000000ab8c90 R15: 0000000000004361 [ 19.943283] R13: 0000000000002000 R14: 0000000000ab8c90 R15: 0000000000004361 [ 19.960500] ---[ end trace 22d2d4868fab0961 ]--- [ 19.960500] ---[ end trace 22d2d4868fab0961 ]--- # HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD git bisect start 7b4991c22de522177fe3185a11e1c1d97e46b073 c02ed2e75ef4c74e41e421acb4ef1494671585e8 -- git bisect bad 2756475d71dd70b2cc223c04266c5fa64f119040 # 20:07 B 19 1 1 1 Merge 'linux-review/Agustin-Vega-Frias/perf-qcom-Add-L3-cache-PMU-driver/20170326-033716' into devel-hourly-2017033111 git bisect bad 06b5190c36059ca4077dce9f80d7658dee173b62 # 20:30 B 9 1 0 0 Merge 'linux-review/Andreas-Klinger/IIO-bmp280-core-c-fix-error-in-humidity-calculation/20170327-212413' into devel-hourly-2017033111 git bisect good 7eb0bc94330ba9cece308dac3c1ded665107b39d # 21:06 G 185 0 0 0 Merge 'vfs/uaccess.parisc' into devel-hourly-2017033111 git bisect bad 3d74c51f8b6b14b0c8faceb15803df104e8f40a1 # 21:22 B 42 1 1 1 Merge 'slave-dma/topic/pl330' into devel-hourly-2017033111 git bisect good c87615ec9808d05a6becd38358a8d2633ab853a4 # 22:13 G 301 0 0 0 Merge 'arm-soc/next/arm64' into devel-hourly-2017033111 git bisect good 970f3b118d3253b1c9c65a4f8e960b6db233b0ad # 22:54 G 300 0 0 0 Merge 'linux-review/Chunyan-Zhang/Add-Spreadtrum-SP9860G-support/20170327-180921' into devel-hourly-2017033111 git bisect bad e3e2af47889e0bd60809e104828af23550a93c54 # 23:19 B 50 1 1 1 Merge 'trace/ftrace/core' into devel-hourly-2017033111 git bisect good 28af0b1ecafdc4cd3cac9b821cadf2b4fb4c0acc # 00:37 G 310 0 0 0 Merge 'linux-review/Abel-Vesa/arm-ftrace-fix-dynamic-ftrace-with-DEBUG_RODATA-and-FRAME_POINTER/20170327-043904' into devel-hourly-2017033111 git bisect good 3519cb309ee7b6cd0e11e143ea077e1f823bbb7e # 01:21 G 310 0 1 1 Merge 'arm-soc/mvebu/soc' into devel-hourly-2017033111 git bisect good 742a0737ce6ca52005003ea990e1e834b5608d16 # 02:05 G 306 0 0 0 Merge 'vfs/uaccess.microblaze' into devel-hourly-2017033111 git bisect good dbeafd0d6131d0f6ae8cd7551f5f4bf8c54aa49a # 02:43 G 310 0 2 2 ftrace: Have function tracing start in early boot up git bisect bad af0009fc16a45d091f896794e97a6457f9a7eddf # 03:01 B 46 1 0 0 tracing: Move trace_handle_return() out of line git bisect bad 42c269c88dc146982a54a8267f71abc99f12852a # 03:25 B 52 1 0 2 ftrace: Allow for function tracing to record init functions on boot up # first bad commit: [42c269c88dc146982a54a8267f71abc99f12852a] ftrace: Allow for function tracing to record init functions on boot up git bisect good dbeafd0d6131d0f6ae8cd7551f5f4bf8c54aa49a # 04:37 G 902 0 0 2 ftrace: Have function tracing start in early boot up # extra tests with CONFIG_DEBUG_INFO_REDUCED git bisect bad 42c269c88dc146982a54a8267f71abc99f12852a # 04:52 B 31 1 1 1 ftrace: Allow for function tracing to record init functions on boot up # extra tests on HEAD of linux-devel/devel-hourly-2017033111 git bisect bad 7b4991c22de522177fe3185a11e1c1d97e46b073 # 04:52 B 22 3 0 1 0day head guard for 'devel-hourly-2017033111' # extra tests on tree/branch trace/ftrace/core git bisect bad 2b87965a1e6b6051435315d3f04627a5dbad979c # 05:25 B 60 1 0 1 ftrace/x86: Do no run CPU sync when there is only one CPU online # extra tests with first bad commit reverted git bisect good 030af8cc860f2496e38d1b320db16450bf05909f # 06:11 G 304 0 0 0 Revert "ftrace: Allow for function tracing to record init functions on boot up" --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/lkp Intel Corporation