Hi Paul,

0day kernel testing robot got the below dmesg and the first bad commit is

git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
commit 0e980234c97f98be6619b9281d83777f725b94ff
Author:     Paul E. McKenney <paulmck@linux.vnet.ibm.com>
AuthorDate: Wed Apr 16 10:07:09 2014 -0700
Commit:     Paul E. McKenney <paulmck@linux.vnet.ibm.com>
CommitDate: Wed May 14 09:46:10 2014 -0700

    percpu: Fix raw_cpu_inc_return()
    
    The definition for raw_cpu_add_return() uses the operation prefix
    "raw_add_return_", but the definitions in the various percpu.h files
    expect "raw_cpu_add_return_".  This commit therefore appropriately
    adjusts the definition of raw_cpu_add_return().
    
    Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
    Acked-by: Christoph Lameter <cl@linux.com>
    Reviewed-by: Josh Triplett <josh@joshtriplett.org>

===================================================
PARENT COMMIT NOT CLEAN. LOOK OUT FOR WRONG BISECT!
===================================================

Attached dmesg for the parent commit, too, to help confirm whether it is a noise error.
To me it looks like an unrelated warning.

+---------------------------------------------------------------+------------+------------+
|                                                               | afea227fd4 | 0e980234c9 |
+---------------------------------------------------------------+------------+------------+
| boot_successes                                                | 40         | 0          |
| boot_failures                                                 | 40         | 20         |
| WARNING:at_kernel/events/core.c:perf_swevent_add()            | 20         |            |
| WARNING:at_kernel/trace/ring_buffer.c:rb_reserve_next_event() | 20         | 0          |
| backtrace:ring_buffer_producer_thread                         | 20         |            |
| BUG:unable_to_handle_kernel_NULL_pointer_dereference          | 0          | 20         |
| Oops                                                          | 0          | 20         |
| RIP:print_trace_line                                          | 0          | 20         |
| Kernel_panic-not_syncing:Fatal_exception                      | 0          | 20         |
| backtrace:rcu_torture_stats                                   | 0          | 20         |
| backtrace:register_tracer                                     | 0          | 0          |
| backtrace:init_function_trace                                 | 0          | 0          |
| backtrace:kernel_init_freeable                                | 0          | 0          |
+---------------------------------------------------------------+------------+------------+

[  124.831322] rcu_bh: wait state: 1 ->state: 0x1
[  124.831755] Dumping ftrace buffer:
[  124.832087] ---------------------------------
[  124.832532] BUG: unable to handle kernel NULL pointer dereference at           (null)
[  124.833290] IP: [<ffffffff8d596377>] print_trace_line+0x2c3/0x39b
[  124.833916] PGD 12298067 PUD 122af067 PMD 0 
[  124.834367] Oops: 0000 [#1] PREEMPT SMP 
[  124.834783] CPU: 1 PID: 57 Comm: rcu_torture_sta Not tainted 3.15.0-rc1-00028-g0e98023 #18
[  124.835658] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[  124.836453] task: ffff880007ee6000 ti: ffff880007ee8000 task.ti: ffff880007ee8000
[  124.837480] RIP: 0010:[<ffffffff8d596377>]  [<ffffffff8d596377>] print_trace_line+0x2c3/0x39b
[  124.838538] RSP: 0018:ffff880007ee9c98  EFLAGS: 00010093
[  124.839033] RAX: 0000000000000000 RBX: ffffffff91104b50 RCX: 0000000000000000
[  124.839698] RDX: 0000000000000001 RSI: ffffffff8ecfaf6b RDI: 0000000000000000
[  124.840015] RBP: ffff880007ee9cc8 R08: 000000000000000a R09: 00000000fffffff4
[  124.840015] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000003bd1201
[  124.840015] R13: ffff8800123be014 R14: ffffffff91105c4c R15: 0000000000000000
[  124.840015] FS:  0000000000000000(0000) GS:ffff880012600000(0000) knlGS:0000000000000000
[  124.840015] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  124.840015] CR2: 0000000000000000 CR3: 00000000122a3000 CR4: 00000000000006a0
[  124.840015] Stack:
[  124.840015]  ffff880007ee9c01 0000000000000001 0000000000000286 ffffffff91105c4c
[  124.840015]  0000000000000000 0000000000000000 ffff880007ee9cf8 ffffffff8d598431
[  124.840015]  ffff880012d3414c ffffffff910ef5ac ffff880012d340ac ffff880012d3409e
[  124.840015] Call Trace:
[  124.840015]  [<ffffffff8d598431>] ftrace_dump+0x177/0x20b
[  124.840015]  [<ffffffff8d55e995>] rcutorture_trace_dump+0x57/0x59
[  124.840015]  [<ffffffff8d55f35b>] rcu_torture_printk+0x4af/0x4cd
[  124.840015]  [<ffffffff8d55f3fc>] ? rcu_torture_stats_print+0x83/0x83
[  124.840015]  [<ffffffff8d55f3de>] rcu_torture_stats_print+0x65/0x83
[  124.840015]  [<ffffffff8d55f44a>] rcu_torture_stats+0x4e/0x73
[  124.840015]  [<ffffffff8d52386a>] kthread+0xef/0xf7
[  124.840015]  [<ffffffff8d52377b>] ? __kthread_parkme+0x80/0x80
[  124.840015]  [<ffffffff8e35e98c>] ret_from_fork+0x7c/0xb0
[  124.840015]  [<ffffffff8d52377b>] ? __kthread_parkme+0x80/0x80
[  124.840015] Code: e9 e0 00 00 00 41 f7 c4 00 10 00 00 74 7c f6 83 d8 00 00 00 02 74 73 8b bb 20 21 00 00 4c 8b bb f0 10 00 00 e8 c4 b9 ff ff 89 c0 <49> 0f a3 07 19 c0 85 c0 75 55 8b bb 20 21 00 00 48 8b 43 10 48 
[  124.840015] RIP  [<ffffffff8d596377>] print_trace_line+0x2c3/0x39b
[  124.840015]  RSP <ffff880007ee9c98>
[  124.840015] CR2: 0000000000000000
[  124.840015] ---[ end trace a89a2d8b0a3f71b2 ]---
[  124.840015] Kernel panic - not syncing: Fatal exception

git bisect start v3.16 v3.15 --
git bisect  bad 5170a3b24a9141e2349a3420448743b7c68f2223  # 20:13      0-      5  Merge branch 'akpm' (patches from Andrew Morton)
git bisect  bad 1ad96bb0a20fa26b952b2250e89d14b6397bf618  # 20:18      0-     20  Merge tag 'gpio-v3.16-2' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-gpio
git bisect  bad 412dd3a6daf0cadce1b2d6a34fa3713f40255579  # 20:42      0-     20  Merge tag 'xfs-for-linus-3.16-rc1' of git://oss.sgi.com/xfs/xfs
git bisect  bad b1cce8032f6abe900b078d24f3c3938726528f97  # 20:59      1-     20  Merge branch 'for-next' of git://git.samba.org/sfrench/cifs-2.6
git bisect  bad da85d191f58a44e149a7c07dbae78b3042909798  # 21:08      0-      2  Merge branch 'for-3.16' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq
git bisect  bad f8409abdc592e13cefbe4e4a24a84b3d5741e85f  # 21:13      1-     20  Merge tag 'ext4_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4
git bisect  bad ed81e780a7dd5698a986f246fad6a1d8d0b6f9ce  # 21:17      0-     20  Merge branch 'x86-vdso-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect  bad 3f17ea6dea8ba5668873afa54628a91aaa3fb1c0  # 21:35      0-     20  Merge branch 'next' (accumulated 3.16 merge window patches) into master
git bisect good 49eb7b0750d9483c74e9c14ae6ea1e9d62481c3c  # 21:45     20+      0  Merge tag 'tty-3.16-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty into next
git bisect  bad 15b588303155b22edd559672905db8e59a44ef9a  # 21:58      0-     12  Merge tag 'fbdev-omap-3.16' of git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux into next
git bisect  bad 776edb59317ada867dfcddde40b55648beeb0078  # 22:04      0-     20  Merge branch 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip into next
git bisect good f456205265a61f1d649f8378eceaa163850cba4e  # 22:10     20+      0  Merge tag 'hwmon-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging into next
git bisect good e13cccfd86481bd4c0499577f44c570d334da79b  # 22:27     20+      0  Merge tag 'spi-v3.16' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi into next
git bisect good ff806d034ef8e9a95ff0b0532104dd65332e446b  # 22:40     20+      0  Merge branch 'for-v3.16' of git://git.linaro.org/people/mszyprowski/linux-dma-mapping into next
git bisect  bad 6348675c4e3612e001860354fea78258e041d9a1  # 22:55      0-     20  torture: Remove unused definition
git bisect good ab7d45053f99f44f81a221eb5c9fbe253ee94524  # 23:13     20+      1  torture: Increase stutter-end intensity
git bisect  bad d0d0606e2c13ad445a58b9d9547de617429cabf9  # 23:21      0-     20  rcutorture: Check for rcu_torture_fqs creation errors
git bisect good afea227fd4acf4f097a9e77bbc2f07d4856ebd01  # 23:33     20+     20  rcutorture: Export RCU grace-period kthread wait state to rcutorture
git bisect  bad 64e4b43ae050146fcfafe696e61efc306f73d449  # 23:41      0-     20  rcutorture: Make rcu_torture_reader() use cond_resched()
git bisect  bad ac1bea85781e9004da9b3e8a4b097c18492d857c  # 23:49      0-     20  sched,rcu: Make cond_resched() report RCU quiescent states
git bisect  bad 0e980234c97f98be6619b9281d83777f725b94ff  # 23:58      0-     20  percpu: Fix raw_cpu_inc_return()
# first bad commit: [0e980234c97f98be6619b9281d83777f725b94ff] percpu: Fix raw_cpu_inc_return()
git bisect good afea227fd4acf4f097a9e77bbc2f07d4856ebd01  # 00:00     60+     40  rcutorture: Export RCU grace-period kthread wait state to rcutorture
git bisect  bad 4cd8d82837097535d55ca63fee12f72c774b4a04  # 00:00      0-     11  0day head guard for 'devel-hourly-2014090117'
git bisect  bad 69e273c0b0a3c337a521d083374c918dc52c666f  # 00:02      0-     15  Linux 3.17-rc3
git bisect  bad d7cf2b3139909a354a71e2885c942e21a60ea062  # 00:09      0-      9  Add linux-next specific files for 20140829


This script may reproduce the error.

----------------------------------------------------------------------------
#!/bin/bash

kernel=$1
initrd=quantal-core-x86_64.cgz

wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/raw/master/initrd/$initrd

kvm=(
	qemu-system-x86_64
	-cpu kvm64
	-enable-kvm
	-kernel $kernel
	-initrd $initrd
	-m 320
	-smp 2
	-net nic,vlan=1,model=e1000
	-net user,vlan=1
	-boot order=nc
	-no-reboot
	-watchdog i6300esb
	-rtc base=localtime
	-serial stdio
	-display none
	-monitor null 
)

append=(
	hung_task_panic=1
	earlyprintk=ttyS0,115200
	debug
	apic=debug
	sysrq_always_enabled
	rcupdate.rcu_cpu_stall_timeout=100
	panic=-1
	softlockup_panic=1
	nmi_watchdog=panic
	oops=panic
	load_ramdisk=2
	prompt_ramdisk=0
	console=ttyS0,115200
	console=tty0
	vga=normal
	root=/dev/ram0
	rw
	drbd.minor_count=8
)

"${kvm[@]}" --append "${append[*]}"
----------------------------------------------------------------------------

Thanks,
Fengguang