Hi Paul,

0day kernel testing robot got the below dmesg and the first bad commit is

git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master

commit 945fa9c631b04febe295a3a2a00c7e4a3cfb97db
Author:     Paul E. McKenney <paulmck@linux.vnet.ibm.com>
AuthorDate: Fri Mar 7 14:15:28 2014 -0800
Commit:     Paul E. McKenney <paulmck@linux.vnet.ibm.com>
CommitDate: Wed May 14 09:46:07 2014 -0700

    torture: Dump ftrace buffer when the RCU grace period stalls
    
    This commit adds a call to rcutorture_trace_dump() to dump the ftrace
    buffer when the RCU grace period stalls in order to help debug the
    stall.  Note that this is different than the RCU CPU stall warning,
    as it is rcutorture detecting the stall rather than the underlying RCU
    implementation.
    
    Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
    Reviewed-by: Josh Triplett <josh@joshtriplett.org>

===================================================
PARENT COMMIT NOT CLEAN. LOOK OUT FOR WRONG BISECT!
===================================================
Attached dmesg for the parent commit, too, to help confirm whether it is a noise error.
It does look like an unrelated warning.

+------------------------------------------------------+------------+------------+------------+
|                                                      | 499c632dce | 945fa9c631 | 0e980234c9 |
+------------------------------------------------------+------------+------------+------------+
| boot_successes                                       | 131        | 0          | 0          |
| boot_failures                                        | 9          | 20         | 40         |
| WARNING:at_kernel/events/core.c:perf_swevent_add()   | 9          |            |            |
| BUG:unable_to_handle_kernel                          | 0          | 20         | 20         |
| Oops                                                 | 0          | 20         | 40         |
| RIP:print_trace_line                                 | 0          | 20         | 40         |
| Kernel_panic-not_syncing:Fatal_exception             | 0          | 20         | 40         |
| backtrace:rcu_torture_stats                          | 0          | 20         | 40         |
| BUG:unable_to_handle_kernel_NULL_pointer_dereference | 0          | 0          | 20         |
+------------------------------------------------------+------------+------------+------------+

mount: mounting proc on /proc failed: No such device
[  121.930088] Dumping ftrace buffer:
[  121.930453] ---------------------------------
[  121.930865] BUG: unable to handle kernel NULL pointer dereference at           (null)
[  121.931644] IP: [<ffffffff8959b074>] print_trace_line+0x2b0/0x38a
[  121.932194] PGD 10876067 PUD 10892067 PMD 0 
[  121.932613] Oops: 0000 [#1] PREEMPT SMP 
[  121.933069] CPU: 0 PID: 47 Comm: rcu_torture_sta Not tainted 3.15.0-rc1-00026-g945fa9c #24
[  121.933837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
[  121.934716] task: ffff880012388000 ti: ffff88001238a000 task.ti: ffff88001238a000
[  121.935348] RIP: 0010:[<ffffffff8959b074>]  [<ffffffff8959b074>] print_trace_line+0x2b0/0x38a
[  121.936092] RSP: 0018:ffff88001238bca8  EFLAGS: 00010097
[  121.936547] RAX: 0000000000000000 RBX: ffffffff8d12e600 RCX: 0000000000000000
[  121.937159] RDX: 0000000000000001 RSI: ffffffff8ad13f33 RDI: 0000000000000000
[  121.937770] RBP: ffff88001238bcd8 R08: 0000000000000009 R09: 00000000fffffff4
[  121.938375] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000003bd1201
[  121.938983] R13: ffff880012a82014 R14: ffffffff8d12f6fc R15: 0000000000000000
[  121.939649] FS:  0000000000000000(0000) GS:ffff880013a00000(0000) knlGS:0000000000000000
[  121.940013] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  121.940013] CR2: 0000000000000000 CR3: 0000000010994000 CR4: 00000000000006b0
[  121.940013] Stack:
[  121.940013]  00000000018c2d94 0000000000000286 0000000000000001 0000000000000000
[  121.940013]  ffffffff8d12f6fc 0000000000000000 ffff88001238bd08 ffffffff8959d169
[  121.940013]  ffff88001099814c ffffffff8d118d8c 00000000ffffffff ffff88001238bec8
[  121.940013] Call Trace:
[  121.940013]  [<ffffffff8959d169>] ftrace_dump+0x180/0x21e
[  121.940013]  [<ffffffff89562d51>] rcutorture_trace_dump+0x5a/0x5c
[  121.940013]  [<ffffffff8956371d>] rcu_torture_printk+0x49d/0x4b8
[  121.940013]  [<ffffffff8956379d>] rcu_torture_stats_print+0x65/0x83
[  121.940013]  [<ffffffff895637bb>] ? rcu_torture_stats_print+0x83/0x83
[  121.940013]  [<ffffffff89563807>] rcu_torture_stats+0x4c/0x71
[  121.940013]  [<ffffffff8952745a>] kthread+0xf9/0x101
[  121.940013]  [<ffffffff89527361>] ? __kthread_parkme+0x86/0x86
[  121.940013]  [<ffffffff8a375c4c>] ret_from_fork+0x7c/0xb0
[  121.940013]  [<ffffffff89527361>] ? __kthread_parkme+0x86/0x86
[  121.940013] Code: e9 e2 00 00 00 41 f7 c4 00 10 00 00 74 7d f6 83 d8 00 00 00 02 74 74 8b bb 20 21 00 00 4c 8b bb f0 10 00 00 e8 6d b8 ff ff 89 c0 <49> 0f a3 07 19 c0 85 c0 75 56 48 63 93 20 21 00 00 48 8b 43 10 
[  121.940013] RIP  [<ffffffff8959b074>] print_trace_line+0x2b0/0x38a
[  121.940013]  RSP <ffff88001238bca8>
[  121.940013] CR2: 0000000000000000
[  121.940013] ---[ end trace 605ab88dfe93d9f3 ]---
[  121.940013] Kernel panic - not syncing: Fatal exception

git bisect start 0e980234c97f98be6619b9281d83777f725b94ff v3.2 --
git bisect good f741656d646f7a5b56ba86765205b954237e7956  # 14:01     20+     14  Merge tag 'stable/for-linus-3.8-rc7-tag-two' of git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen
git bisect good 9538e10086bd1301fe915683b1ba0a1de66d7483  # 14:23     20+      4  Merge git://www.linux-watchdog.org/linux-watchdog
git bisect good 83660b734bbbae4f15c1a9b14d0514d74f79dfd4  # 14:44     20+      4  Merge tag 'fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect good 1f8c538ed6a3323b06c2459e9ca36e0ae8bb0ebc  # 15:09     20+      0  Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux
git bisect good 3f583bc21977a608908b83d03ee2250426a5695c  # 15:23     20+      2  Merge tag 'iommu-updates-v3.15' of git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu
git bisect good 3e8072d48b2dd0898e99698018b2045f8cd49965  # 15:44     20+      1  Merge git://git.infradead.org/users/willy/linux-nvme
git bisect good 9352ad13391d8948fa919b97ed5ad68c96378f92  # 16:03     20+      2  torture: Add batch headers "--dryrun sched"
git bisect good 97dc0afed12b371f450d04cb406a8664437ec843  # 16:16     20+      3  torture: Make config-fragment filtering RCU-independent
git bisect good fff35c4e9fbb0982aef6230cbf713c2492925eca  # 16:33     20+      2  torture: Report diagnostics from qemu
git bisect good f189cc8c9451aa88bc718c57aa0228d2a7555956  # 16:49     20+      1  torture: Choose bzImage location based on architecture
git bisect  bad 945fa9c631b04febe295a3a2a00c7e4a3cfb97db  # 17:04      0-      6  torture: Dump ftrace buffer when the RCU grace period stalls
git bisect good 499c632dce47e95280f8b99540c5db294053c94c  # 17:27     20+      4  torture: Add tracing-enabled variant of TREE02
# first bad commit: [945fa9c631b04febe295a3a2a00c7e4a3cfb97db] torture: Dump ftrace buffer when the RCU grace period stalls
git bisect good 499c632dce47e95280f8b99540c5db294053c94c  # 17:34     60+      7  torture: Add tracing-enabled variant of TREE02
git bisect good 499c632dce47e95280f8b99540c5db294053c94c  # 17:40     60+      9  torture: Add tracing-enabled variant of TREE02
git bisect  bad fc486b03cae382601b366ab460b05e1a01bf69cd  # 17:56      0-     10  Merge tag 'stable/for-linus-3.17-b-rc4-arm-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip
git bisect  bad fcc8fd7117b3960d50439f2c4247f80d74974c95  # 18:07      2-      5  Add linux-next specific files for 20140912


This script may reproduce the error.

----------------------------------------------------------------------------
#!/bin/bash

kernel=$1
initrd=yocto-minimal-x86_64.cgz

wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/raw/master/initrd/$initrd

kvm=(
	qemu-system-x86_64
	-cpu kvm64
	-enable-kvm
	-kernel $kernel
	-initrd $initrd
	-m 320
	-smp 1
	-net nic,vlan=1,model=e1000
	-net user,vlan=1
	-boot order=nc
	-no-reboot
	-watchdog i6300esb
	-rtc base=localtime
	-serial stdio
	-display none
	-monitor null 
)

append=(
	hung_task_panic=1
	earlyprintk=ttyS0,115200
	debug
	apic=debug
	sysrq_always_enabled
	rcupdate.rcu_cpu_stall_timeout=100
	panic=-1
	softlockup_panic=1
	nmi_watchdog=panic
	oops=panic
	load_ramdisk=2
	prompt_ramdisk=0
	console=ttyS0,115200
	console=tty0
	vga=normal
	root=/dev/ram0
	rw
	drbd.minor_count=8
)

"${kvm[@]}" --append "${append[*]}"
----------------------------------------------------------------------------

Thanks,
Fengguang

_______________________________________________
LKP mailing list
LKP(a)linux.intel.com