Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master commit 6d25be5782e482eb93e3de0c94d0a517879377d0 Author: Thomas Gleixner AuthorDate: Wed Mar 13 17:55:48 2019 +0100 Commit: Ingo Molnar CommitDate: Tue Apr 16 16:55:15 2019 +0200 sched/core, workqueues: Distangle worker accounting from rq lock The worker accounting for CPU bound workers is plugged into the core scheduler code and the wakeup code. This is not a hard requirement and can be avoided by keeping track of the state in the workqueue code itself. Keep track of the sleeping state in the worker itself and call the notifier before entering the core scheduler. There might be false positives when the task is woken between that call and actually scheduling, but that's not really different from scheduling and being woken immediately after switching away. When nr_running is updated when the task is retunrning from schedule() then it is later compared when it is done from ttwu(). [ bigeasy: preempt_disable() around wq_worker_sleeping() by Daniel Bristot de Oliveira ] Signed-off-by: Thomas Gleixner Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Peter Zijlstra (Intel) Acked-by: Tejun Heo Cc: Daniel Bristot de Oliveira Cc: Lai Jiangshan Cc: Linus Torvalds Cc: Peter Zijlstra Link: http://lkml.kernel.org/r/ad2b29b5715f970bffc1a7026cabd6ff0b24076a.1532952814.git.bristot@redhat.com Signed-off-by: Ingo Molnar e2abb39811 sched/fair: Remove unneeded prototype of capacity_of() 6d25be5782 sched/core, workqueues: Distangle worker accounting from rq lock 9efcc4a129 afs: Fix unpinned address list during probing +--------------------------------------------------------------------------------+------------+------------+------------+ | | e2abb39811 | 6d25be5782 | 9efcc4a129 | +--------------------------------------------------------------------------------+------------+------------+------------+ | boot_successes | 997 | 806 | 27 | | boot_failures | 112 | 104 | 9 | | WARNING:at_kernel/rcu/rcutorture.c:#rcu_torture_fwd_prog | 61 | 52 | 2 | | RIP:rcu_torture_fwd_prog | 63 | 57 | 2 | | BUG:workqueue_lockup-pool | 12 | 16 | 1 | | BUG:kernel_hang_in_early-boot_stage,last_printk:early_console_in_setup_code | 14 | 13 | 6 | | BUG:kernel_timeout_in_early-boot_stage,last_printk:early_console_in_setup_code | 5 | | | | Initiating_system_reboot | 1 | | | | BUG:spinlock_recursion_on_CPU | 10 | 6 | | | RIP:rcu_torture_one_read | 2 | | | | INFO:rcu_preempt_self-detected_stall_on_CPU | 1 | 3 | | | RIP:iov_iter_copy_from_user_atomic | 1 | 3 | | | BUG:kernel_hang_in_boot_stage | 2 | | | | RIP:__schedule | 4 | 2 | | | BUG:kernel_hang_in_test_stage | 2 | | | | RIP:rcutorture_seq_diff | 3 | | | | BUG:spinlock_wrong_owner_on_CPU | 1 | | | | BUG:workqueue_leaked_lock_or_atomic:rcu_torture_sta | 1 | | | | BUG:sleeping_function_called_from_invalid_context_at_kernel/workqueue.c | 1 | | | | WARNING:at_kernel/workqueue.c:#worker_set_flags | 1 | | | | RIP:worker_set_flags | 1 | | | | BUG:scheduling_while_atomic | 1 | | | | BUG:unable_to_handle_kernel | 2 | 1 | | | Oops:#[##] | 2 | 1 | | | Kernel_panic-not_syncing:Fatal_exception | 6 | 1 | | | kernel_BUG_at_lib/list_debug.c | 4 | | | | invalid_opcode:#[##] | 4 | | | | RIP:__list_add_valid | 4 | | | | RIP:__rb_erase_color | 1 | | | | INFO:rcu_preempt_detected_stalls_on_CPUs/tasks | 1 | 7 | | | RIP:rcu_read_delay | 1 | 1 | | | RIP:kvm_async_pf_task_wait | 1 | | | | RIP:d_walk | 1 | | | | WARNING:at_kernel/workqueue.c:#wq_worker_sleeping | 0 | 8 | 1 | | RIP:wq_worker_sleeping | 0 | 8 | 1 | | RIP:kthread_data | 0 | 2 | | | RIP:to_kthread | 0 | 1 | | | RIP:wq_worker_running | 0 | 5 | 1 | | WARNING:at_kernel/workqueue.c:#worker_enter_idle | 0 | 3 | 1 | | RIP:worker_enter_idle | 0 | 3 | 1 | | BUG:kernel_timeout_in_boot_stage | 0 | 1 | | | BUG:soft_lockup-CPU##stuck_for#s![rcu_torture_fak:#] | 0 | 1 | | | RIP:preempt_count_equals | 0 | 2 | | | Kernel_panic-not_syncing:softlockup:hung_tasks | 0 | 2 | | | BUG:soft_lockup-CPU##stuck_for#s![rcu_torture_rea:#] | 0 | 1 | | | RIP:ftrace_stub | 0 | 1 | | | RIP:ftrace_likely_update | 0 | 3 | | | RIP:simple_write_end | 0 | 1 | | | RIP:delay_tsc | 0 | 2 | | | RIP:check_preemption_disabled | 0 | 1 | | +--------------------------------------------------------------------------------+------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot [ 5.924439] igt_debug 0x0000000000000200-0x0000000000000600: 1024: used [ 5.924883] igt_debug 0x0000000000000600-0x0000000000000a00: 1024: free [ 5.925327] igt_debug 0x0000000000000a00-0x0000000000000e00: 1024: used [ 5.925806] igt_debug 0x0000000000000e00-0x0000000000001000: 512: free [ 5.926286] igt_debug total: 4096, used 2048 free 2048 [ 52.816697] WARNING: CPU: 0 PID: 14 at kernel/workqueue.c:882 wq_worker_sleeping+0x72/0x136 [ 52.821963] Modules linked in: [ 52.821963] CPU: 0 PID: 14 Comm: kworker/0:1 Not tainted 5.1.0-rc3-00024-g6d25be5782e48 #1 [ 52.821963] Workqueue: rcu_gp wait_rcu_exp_gp [ 52.821963] RIP: 0010:wq_worker_sleeping+0x72/0x136 [ 52.821963] Code: 48 8b 58 40 45 85 ed 41 0f 95 c6 31 c9 31 d2 44 89 f6 e8 c5 3f 0d 00 48 ff 05 71 aa 59 03 45 85 ed 74 17 48 ff 05 6d aa 59 03 <0f> 0b 48 ff 05 6c aa 59 03 48 ff 05 6d aa 59 03 31 c9 31 d2 44 89 [ 52.821963] RSP: 0000:ffffa66380073b78 EFLAGS: 00010202 [ 52.821963] RAX: ffff97fe37b08de0 RBX: ffffffff9c2ae960 RCX: 0000000000000000 [ 52.821963] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff9cd4e028 [ 52.821963] RBP: ffffa66380073b98 R08: 0000000000000000 R09: 0000000000000000 [ 52.821963] R10: ffffa66380073c90 R11: 0000000000000000 R12: ffff97fe37b08de0 [ 52.821963] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000005000 [ 52.821963] FS: 0000000000000000(0000) GS:ffffffff9c250000(0000) knlGS:0000000000000000 [ 52.821963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 52.821963] CR2: 0000000000005000 CR3: 00000001b741a000 CR4: 00000000000006b0 [ 52.821963] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 52.821963] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 52.821963] Call Trace: [ 52.821963] schedule+0x7a/0x1c0 [ 52.821963] kvm_async_pf_task_wait+0x253/0x300 [ 52.821963] do_async_page_fault+0xbf/0x141 [ 52.821963] ? kvm_async_pf_task_wait+0x5/0x300 [ 52.821963] ? do_async_page_fault+0xbf/0x141 [ 52.821963] async_page_fault+0x1e/0x30 [ 52.821963] RIP: 0010:to_kthread+0x0/0x81 [ 52.821963] Code: 03 48 ff 05 44 b0 59 03 89 de 31 c9 31 d2 48 c7 c7 a8 ea d4 9c e8 9f ec 0c 00 48 ff 05 33 b0 59 03 5b 41 5c 41 5d 41 5e 5d c3 <55> 48 ff 05 a2 a8 59 03 48 89 e5 41 55 41 54 53 45 31 ed 49 89 fc [ 52.821963] RSP: 0000:ffffa66380073d68 EFLAGS: 00010206 [ 52.821963] RAX: 0000000000000000 RBX: ffff97fe37b52000 RCX: 0000000000000000 [ 52.821963] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff97fe37b52000 [ 52.821963] RBP: ffffa66380073d70 R08: 0000000000000000 R09: ffffffff9c2c0a18 [ 52.821963] R10: ffff97fe37b52060 R11: 0000000000000000 R12: 0000000000000000 [ 52.821963] R13: ffffffff9cd51418 R14: 0000000000002710 R15: 0000000000000000 [ 52.821963] ? kthread_data+0x15/0x22 [ 52.821963] wq_worker_running+0x15/0x53 [ 52.821963] schedule+0x1ab/0x1c0 [ 52.821963] schedule_timeout+0xd6/0x10a [ 52.821963] ? init_timer_key+0x41/0x41 [ 52.821963] rcu_exp_sel_wait_wake+0x58c/0xae2 [ 52.821963] wait_rcu_exp_gp+0x19/0x22 [ 52.821963] process_one_work+0x387/0x5eb [ 52.821963] ? worker_thread+0x4d2/0x5e3 [ 52.821963] worker_thread+0x401/0x5e3 [ 52.821963] ? rescuer_thread+0x492/0x492 [ 52.821963] kthread+0x19b/0x1aa [ 52.821963] ? kthread_flush_work+0x175/0x175 [ 52.821963] ret_from_fork+0x3a/0x50 [ 52.821963] random: get_random_bytes called from init_oops_id+0x2d/0x4c with crng_init=0 [ 52.821963] ---[ end trace 0be0a8f9c0f268e1 ]--- [ 64.886064] rcu-torture: rtc: (____ptrval____) ver: 324 tfle: 0 rta: 324 rtaf: 0 rtf: 315 rtmbe: 0 rtbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 101 barrier: 0/0:0 # HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD git bisect start v5.2 v5.1 -- git bisect bad 2c1212de6f9794a7becba5f219fa6ce8a8222c90 # 23:32 B 104 1 6 273 Merge tag 'spdx-5.2-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core git bisect bad 06cbd26d312edfe4a83ff541c23f8f866265eb24 # 00:18 B 26 1 0 0 Merge tag 'nfs-for-5.2-1' of git://git.linux-nfs.org/projects/anna/linux-nfs git bisect bad 9f2e3a53f7ec9ef55e9d01bc29a6285d291c151e # 05:24 B 269 1 20 20 Merge tag 'for-5.2-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux git bisect bad ba3934de557a2074b89d67e29e5ce119f042d057 # 08:23 B 131 1 11 11 Merge branch 'x86-platform-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect good 90489a72fba9529c85e051067ecb41183b8e982e # 16:37 G 900 0 74 74 Merge branch 'perf-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect bad 82ac4043cac5d75d8cda79bc8a095f8306f35c75 # 17:45 B 245 1 22 22 Merge branch 'x86-cache-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect bad a0e928ed7c603a47dca8643e58db224a799ff2c5 # 21:01 B 315 1 22 22 Merge branch 'timers-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect bad e00d4135751bfe786a9e26b5560b185ce3f9f963 # 23:36 B 193 1 16 16 Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect bad bee9853932e90ce94bce4276ec6b7b06bc48070b # 02:33 B 262 1 24 24 sched/core: Fix typo in comment git bisect good d8743230c9f4e92f370ecd2a90c680ddcede6ae5 # 08:53 G 902 0 97 97 sched/topology: Fix build_sched_groups() comment git bisect bad 6d25be5782e482eb93e3de0c94d0a517879377d0 # 10:00 B 62 1 12 12 sched/core, workqueues: Distangle worker accounting from rq lock git bisect good e2abb398115e9c33f3d1e25bf6d1d08badc58b13 # 23:19 G 900 0 89 89 sched/fair: Remove unneeded prototype of capacity_of() # first bad commit: [6d25be5782e482eb93e3de0c94d0a517879377d0] sched/core, workqueues: Distangle worker accounting from rq lock git bisect good e2abb398115e9c33f3d1e25bf6d1d08badc58b13 # 02:02 G 1000 0 21 110 sched/fair: Remove unneeded prototype of capacity_of() # extra tests with debug options git bisect good 6d25be5782e482eb93e3de0c94d0a517879377d0 # 09:37 G 900 0 147 147 sched/core, workqueues: Distangle worker accounting from rq lock # extra tests on head commit of linus/master git bisect bad 9efcc4a129363187c9bf15338692f107c5c9b6f0 # 10:25 B 30 1 7 7 afs: Fix unpinned address list during probing # bad: [9efcc4a129363187c9bf15338692f107c5c9b6f0] afs: Fix unpinned address list during probing # extra tests on linus/master # duplicated: [9efcc4a129363187c9bf15338692f107c5c9b6f0] afs: Fix unpinned address list during probing # extra tests on linux-next/master # 119: [89295c59c1f063b533d071ca49d0fa0c0783ca6f] Add linux-next specific files for 20200326 --- 0-DAY CI Kernel Test Service, Intel Corporation https://lists.01.org/hyperkitty/list/lkp@lists.01.org