* RCU stall warnings...
@ 2017-07-24 22:32 David Miller
2017-07-24 23:20 ` Paul E. McKenney
0 siblings, 1 reply; 6+ messages in thread
From: David Miller @ 2017-07-24 22:32 UTC (permalink / raw)
To: paulmck; +Cc: linux-kernel, sparclinux
Paul and other RCU experts,
Starting with 4.13-rc1 we're getting RCU stall dumps on sparc64, they
definitely didn't happen in 4.12
I tried to look for low hanging fruit in the kernel/rcu/ changes this
merge window, but nothing really stood out for me.
This only seems to trigger when the system is mostly idle.
Kconfig wise PREEMPT is disabled, and NO_HZ_COMMON and NO_HOZ_IDLE are
both enabled.
The dumps always look the same, see below. The idle values are even,
so those cpus should be in dynticks mode.
Perhaps I'm not doing something properly in our arch idle loop, but I
don't see anything obvious there.
[402131.682905] INFO: rcu_sched detected stalls on CPUs/tasks:
[402131.693768] 2-...: (4304 GPs behind) idle=990/0/0 softirq=80469/80469 fqs=0
[402131.708188] 3-...: (3286 GPs behind) idle=0bc/0/0 softirq=77239/77241 fqs=0
[402131.722595] 4-...: (5513 GPs behind) idle=2b4/0/0 softirq=74384/74386 fqs=0
[402131.737003] 5-...: (3596 GPs behind) idle=ad0/0/0 softirq=73109/73109 fqs=0
[402131.751411] 6-...: (3592 GPs behind) idle=e78/0/0 softirq=67679/67679 fqs=0
[402131.765835] 7-...: (3591 GPs behind) idle=9ac/0/0 softirq=65887/65889 fqs=0
[402131.780243] 9-...: (5 GPs behind) idle=724/0/0 softirq=88433/88436 fqs=0
[402131.794121] 11-...: (3286 GPs behind) idle=be4/0/0 softirq=81587/81589 fqs=0
[402131.808724] 12-...: (3596 GPs behind) idle=ed0/0/0 softirq=79542/79544 fqs=0
[402131.823307] 13-...: (3596 GPs behind) idle=9f8/0/0 softirq=73652/73655 fqs=0
[402131.837891] 14-...: (3593 GPs behind) idle=d3c/0/0 softirq=72326/72328 fqs=0
[402131.852476] 15-...: (3596 GPs behind) idle=6f4/0/0 softirq=69730/69732 fqs=0
[402131.867060] 20-...: (2566 GPs behind) idle=b80/0/0 softirq=79921/79925 fqs=0
[402131.881644] 21-...: (4701 GPs behind) idle=944/0/0 softirq=73890/73890 fqs=0
[402131.896228] 22-...: (2563 GPs behind) idle=b58/0/0 softirq=74554/74554 fqs=0
[402131.910812] 23-...: (3285 GPs behind) idle=b08/0/0 softirq=112972/112972 fqs=0
[402131.925749] 27-...: (3596 GPs behind) idle=5f4/0/0 softirq=79524/79524 fqs=0
[402131.940333] 28-...: (4070 GPs behind) idle=b30/0/0 softirq=71061/71063 fqs=0
[402131.954917] 29-...: (2430 GPs behind) idle=4b4/0/0 softirq=72431/72433 fqs=0
[402131.969501] 30-...: (389 GPs behind) idle=65c/0/0 softirq=68508/68510 fqs=0
[402131.983925] 31-...: (170 GPs behind) idle=174/0/0 softirq=72802/72804 fqs=0
[402131.998334] 33-...: (1627 GPs behind) idle=d2c/0/0 softirq=89821/89823 fqs=0
[402132.012918] 34-...: (391 GPs behind) idle=8b0/0/0 softirq=82561/82563 fqs=0
[402132.027326] 35-...: (169 GPs behind) idle=700/0/0 softirq=77960/77962 fqs=0
[402132.041734] 36-...: (3590 GPs behind) idle=cb4/0/0 softirq=79230/79232 fqs=0
[402132.056334] 37-...: (391 GPs behind) idle=264/0/0 softirq=71121/71123 fqs=0
[402132.070743] 38-...: (3594 GPs behind) idle=604/0/0 softirq=71422/71422 fqs=0
[402132.085327] 39-...: (3409 GPs behind) idle=eb4/0/0 softirq=67121/67121 fqs=0
[402132.099911] 42-...: (172 GPs behind) idle=d4c/0/0 softirq=121584/121588 fqs=0
[402132.114671] 43-...: (388 GPs behind) idle=3f0/0/0 softirq=80541/80541 fqs=0
[402132.129079] 44-...: (3605 GPs behind) idle=410/0/0 softirq=73502/73502 fqs=0
[402132.143663] 45-...: (3591 GPs behind) idle=078/0/0 softirq=74292/74292 fqs=0
[402132.158247] 46-...: (3594 GPs behind) idle=800/0/0 softirq=73200/73200 fqs=0
[402132.172832] 47-...: (3599 GPs behind) idle=460/0/0 softirq=68045/68047 fqs=0
[402132.187433] 50-...: (97 GPs behind) idle=180/0/0 softirq=81843/81843 fqs=0
[402132.201662] 51-...: (172 GPs behind) idle=5bc/0/0 softirq=76273/76277 fqs=0
[402132.216072] 52-...: (2915 GPs behind) idle=688/0/0 softirq=76044/76046 fqs=0
[402132.230656] 53-...: (2564 GPs behind) idle=6dc/0/0 softirq=75980/75982 fqs=0
[402132.245240] 54-...: (6357 GPs behind) idle=a40/0/0 softirq=72584/72584 fqs=0
[402132.259841] 55-...: (3286 GPs behind) idle=c34/0/0 softirq=70795/70795 fqs=0
[402132.274425] 57-...: (4 GPs behind) idle=d0c/0/0 softirq=90478/90480 fqs=0
[402132.288479] 59-...: (0 ticks this GP) idle=d2c/0/0 softirq=79767/79767 fqs=0
[402132.303065] 61-...: (5511 GPs behind) idle=4f0/0/0 softirq=78603/78603 fqs=0
[402132.317649] 62-...: (5509 GPs behind) idle=e88/0/0 softirq=73573/73575 fqs=0
[402132.332233] 63-...: (5518 GPs behind) idle=d94/0/0 softirq=71343/71343 fqs=0
[402132.346832] 64-...: (55 GPs behind) idle=a14/0/0 softirq=122673/122673 fqs=0
[402132.361417] 65-...: (134 GPs behind) idle=4a8/0/0 softirq=97047/97047 fqs=0
[402132.375825] 66-...: (2523 GPs behind) idle=418/0/0 softirq=81713/81713 fqs=0
[402132.390409] 67-...: (2470 GPs behind) idle=000/0/0 softirq=84375/84377 fqs=0
[402132.405001] 68-...: (2565 GPs behind) idle=fcc/0/0 softirq=72096/72096 fqs=0
[402132.419586] 69-...: (3655 GPs behind) idle=1e4/0/0 softirq=71565/71567 fqs=0
[402132.434170] 70-...: (3597 GPs behind) idle=89c/0/0 softirq=68341/68341 fqs=0
[402132.448754] 71-...: (3605 GPs behind) idle=f94/0/0 softirq=64666/64668 fqs=0
[402132.463333] 72-...: (49 GPs behind) idle=b40/0/0 softirq=112848/112848 fqs=0
[402132.477924] 73-...: (160 GPs behind) idle=db8/0/0 softirq=85959/85961 fqs=0
[402132.492340] 74-...: (157 GPs behind) idle=d28/0/0 softirq=81416/81416 fqs=0
[402132.506748] 75-...: (186 GPs behind) idle=34c/0/0 softirq=85910/85912 fqs=0
[402132.521158] 76-...: (4072 GPs behind) idle=764/0/0 softirq=75200/75202 fqs=0
[402132.535744] 77-...: (3695 GPs behind) idle=e28/0/0 softirq=76678/76678 fqs=0
[402132.550331] 78-...: (3691 GPs behind) idle=5b4/0/0 softirq=72681/72681 fqs=0
[402132.564925] 79-...: (11143 GPs behind) idle=4a8/0/0 softirq=73212/73212 fqs=0
[402132.579670] 80-...: (132 GPs behind) idle=044/0/0 softirq=117329/117329 fqs=0
[402132.594429] 81-...: (157 GPs behind) idle=cb4/0/0 softirq=87921/87923 fqs=0
[402132.608846] 82-...: (2432 GPs behind) idle=c3c/0/0 softirq=83531/83531 fqs=0
[402132.623432] 83-...: (389 GPs behind) idle=c94/0/0 softirq=79845/79847 fqs=0
[402132.637840] 84-...: (3588 GPs behind) idle=990/0/0 softirq=75763/75765 fqs=0
[402132.652424] 85-...: (3593 GPs behind) idle=be0/0/0 softirq=76481/76483 fqs=0
[402132.667020] 86-...: (5254 GPs behind) idle=4c0/0/0 softirq=73565/73567 fqs=0
[402132.681605] 87-...: (5507 GPs behind) idle=4a8/0/0 softirq=70595/70597 fqs=0
[402132.696182] 88-...: (162 GPs behind) idle=2ac/0/0 softirq=120362/120364 fqs=0
[402132.710942] 89-...: (155 GPs behind) idle=f6c/0/0 softirq=86298/86300 fqs=0
[402132.725367] 90-...: (161 GPs behind) idle=23c/0/0 softirq=81845/81845 fqs=0
[402132.739779] 91-...: (5236 GPs behind) idle=a84/0/0 softirq=76524/76524 fqs=0
[402132.754363] 92-...: (5073 GPs behind) idle=46c/0/0 softirq=78364/78366 fqs=0
[402132.768952] 93-...: (7496 GPs behind) idle=590/0/0 softirq=75926/75928 fqs=0
[402132.783536] 94-...: (3594 GPs behind) idle=fdc/0/0 softirq=73619/73619 fqs=0
[402132.798110] 95-...: (446 GPs behind) idle=4d8/0/0 softirq=67953/67953 fqs=0
[402132.812535] 96-...: (156 GPs behind) idle=4b0/0/0 softirq=151231/151231 fqs=0
[402132.827295] 97-...: (2066 GPs behind) idle=4a0/0/0 softirq=90382/90384 fqs=0
[402132.841879] 98-...: (3592 GPs behind) idle=e10/0/0 softirq=85418/85420 fqs=0
[402132.856454] 99-...: (90 GPs behind) idle=6c4/0/0 softirq=116865/116865 fqs=0
[402132.871038] 100-...: (162 GPs behind) idle=aa0/0/0 softirq=77088/77089 fqs=0
[402132.885627] 101-...: (157 GPs behind) idle=5e8/0/0 softirq=75034/75036 fqs=0
[402132.900211] 102-...: (3591 GPs behind) idle=a08/0/0 softirq=72174/72176 fqs=0
[402132.914971] 103-...: (3605 GPs behind) idle=f2c/0/0 softirq=68188/68190 fqs=0
[402132.929736] 104-...: (155 GPs behind) idle=0cc/0/0 softirq=107026/107026 fqs=0
[402132.944672] 105-...: (157 GPs behind) idle=c24/0/0 softirq=83549/83551 fqs=0
[402132.959256] 106-...: (1625 GPs behind) idle=6dc/0/0 softirq=80774/80774 fqs=0
[402132.974007] 107-...: (3589 GPs behind) idle=d4c/0/0 softirq=82554/82556 fqs=0
[402132.988768] 108-...: (3588 GPs behind) idle=cc8/0/0 softirq=76988/76990 fqs=0
[402133.003528] 109-...: (3597 GPs behind) idle=644/0/0 softirq=74768/74768 fqs=0
[402133.018288] 110-...: (5176 GPs behind) idle=96c/0/0 softirq=72869/72871 fqs=0
[402133.033048] 111-...: (3401 GPs behind) idle=458/0/0 softirq=70859/70859 fqs=0
[402133.047809] 112-...: (125 GPs behind) idle=308/0/0 softirq=108176/108176 fqs=0
[402133.062745] 113-...: (157 GPs behind) idle=e24/0/0 softirq=83982/83984 fqs=0
[402133.077321] 114-...: (725 GPs behind) idle=4b0/0/0 softirq=80102/80103 fqs=0
[402133.091922] 115-...: (3596 GPs behind) idle=744/0/0 softirq=79547/79547 fqs=0
[402133.106675] 116-...: (4977 GPs behind) idle=298/0/0 softirq=75316/75318 fqs=0
[402133.121435] 117-...: (3591 GPs behind) idle=00c/0/0 softirq=75114/75114 fqs=0
[402133.136188] 118-...: (3592 GPs behind) idle=d70/0/0 softirq=73200/73200 fqs=0
[402133.150946] 119-...: (5255 GPs behind) idle=d28/0/0 softirq=67479/67479 fqs=0
[402133.165708] 120-...: (138 GPs behind) idle=e08/0/0 softirq=112097/112099 fqs=0
[402133.180641] 121-...: (156 GPs behind) idle=4d0/0/0 softirq=89608/89608 fqs=0
[402133.195225] 122-...: (389 GPs behind) idle=648/0/0 softirq=81157/81159 fqs=0
[402133.209814] 123-...: (389 GPs behind) idle=3b0/0/0 softirq=81959/81961 fqs=0
[402133.224405] 124-...: (4102 GPs behind) idle=b50/0/0 softirq=75522/75524 fqs=0
[402133.239165] 125-...: (3591 GPs behind) idle=f64/0/0 softirq=74702/74702 fqs=0
[402133.253926] 126-...: (3593 GPs behind) idle=2a4/0/0 softirq=70870/70872 fqs=0
[402133.268686] 127-...: (4228 GPs behind) idle=508/0/0 softirq=72137/72139 fqs=0
[402133.283438] (detected by 48, t=2262 jiffies, g=53669, c=53668, q=18005)
[402133.297060] CPU[ 2]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/2:0]
[402133.317316] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402133.342191] CPU[ 3]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/3:0]
[402133.362453] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402133.387329] CPU[ 4]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/4:0]
[402133.407593] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402133.432470] CPU[ 5]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/5:0]
[402133.452747] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402133.477624] CPU[ 6]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/6:0]
[402133.497885] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402133.522763] CPU[ 7]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/7:0]
[402133.543040] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402133.567920] CPU[ 9]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/9:0]
[402133.588178] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402133.613055] CPU[ 11]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/11:0]
[402133.633509] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402133.658386] CPU[ 12]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/12:0]
[402133.678823] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402133.703709] CPU[ 13]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/13:0]
[402133.724146] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402133.749023] CPU[ 14]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/14:0]
[402133.769459] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402133.794336] CPU[ 15]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/15:0]
[402133.814773] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402133.839667] CPU[ 20]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/20:0]
[402133.860104] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402133.884981] CPU[ 21]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/21:0]
[402133.905414] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402133.930298] CPU[ 22]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/22:0]
[402133.950747] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402133.975617] CPU[ 23]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/23:0]
[402133.996053] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402134.020947] CPU[ 27]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/27:0]
[402134.041384] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402134.066269] CPU[ 28]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/28:0]
[402134.086705] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402134.111582] CPU[ 29]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/29:0]
[402134.132018] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402134.156896] CPU[ 30]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/30:0]
[402134.177333] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402134.202229] CPU[ 31]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/31:0]
[402134.222666] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402134.247548] CPU[ 33]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/33:0]
[402134.267978] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402134.292856] CPU[ 34]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/34:0]
[402134.313292] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402134.338186] CPU[ 35]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/35:0]
[402134.358625] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402134.383502] CPU[ 36]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/36:0]
[402134.403939] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402134.428817] CPU[ 37]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/37:0]
[402134.449265] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402134.474142] CPU[ 38]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/38:0]
[402134.494572] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402134.519457] CPU[ 39]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/39:0]
[402134.539894] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402134.564788] CPU[ 42]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/42:0]
[402134.585224] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402134.610095] CPU[ 43]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/43:0]
[402134.630548] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402134.655425] CPU[ 44]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/44:0]
[402134.675861] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402134.700744] CPU[ 45]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/45:0]
[402134.721180] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402134.746057] CPU[ 46]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/46:0]
[402134.766494] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402134.791379] CPU[ 47]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/47:0]
[402134.811816] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402134.836710] CPU[ 50]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/50:0]
[402134.857146] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402134.882023] CPU[ 51]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/51:0]
[402134.902460] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402134.927337] CPU[ 52]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/52:0]
[402134.947773] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402134.972667] CPU[ 53]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/53:0]
[402134.993104] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402135.017987] CPU[ 54]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/54:0]
[402135.038424] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402135.063301] CPU[ 55]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/55:0]
[402135.083737] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402135.108614] CPU[ 57]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/57:0]
[402135.129067] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402135.153945] CPU[ 59]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/59:0]
[402135.174381] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402135.199258] CPU[ 61]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/61:0]
[402135.219705] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402135.244582] CPU[ 62]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/62:0]
[402135.265019] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402135.289896] CPU[ 63]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/63:0]
[402135.310332] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402135.335231] CPU[ 64]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/64:0]
[402135.355653] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402135.380545] CPU[ 65]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/65:0]
[402135.400982] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402135.425859] CPU[ 66]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/66:0]
[402135.446295] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402135.471181] CPU[ 67]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/67:0]
[402135.491617] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402135.516495] CPU[ 68]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/68:0]
[402135.536931] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402135.561825] CPU[ 69]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/69:0]
[402135.582262] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402135.607140] CPU[ 70]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/70:0]
[402135.627595] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402135.652495] CPU[ 71]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/71:0]
[402135.672926] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402135.697815] CPU[ 72]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/72:0]
[402135.718231] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402135.743099] CPU[ 73]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/73:0]
[402135.763537] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402135.788420] CPU[ 74]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/74:0]
[402135.808853] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402135.833748] CPU[ 75]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/75:0]
[402135.854184] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402135.879062] CPU[ 76]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/76:0]
[402135.899499] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402135.924377] CPU[ 77]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/77:0]
[402135.944813] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402135.969707] CPU[ 78]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/78:0]
[402135.990144] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402136.015022] CPU[ 79]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/79:0]
[402136.035460] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402136.060338] CPU[ 80]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/80:0]
[402136.080783] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402136.105667] CPU[ 81]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/81:0]
[402136.126100] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402136.150977] CPU[ 82]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/82:0]
[402136.171413] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402136.196308] CPU[ 83]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/83:0]
[402136.216745] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402136.241623] CPU[ 84]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/84:0]
[402136.262059] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402136.286937] CPU[ 85]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/85:0]
[402136.307373] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402136.332267] CPU[ 86]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/86:0]
[402136.352704] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402136.377586] CPU[ 87]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/87:0]
[402136.398023] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402136.422900] CPU[ 88]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/88:0]
[402136.443337] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402136.468213] CPU[ 89]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/89:0]
[402136.488667] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402136.513544] CPU[ 90]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/90:0]
[402136.533981] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402136.558859] CPU[ 91]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/91:0]
[402136.579304] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402136.604182] CPU[ 92]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/92:0]
[402136.624619] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402136.649496] CPU[ 93]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/93:0]
[402136.669933] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402136.694827] CPU[ 94]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/94:0]
[402136.715264] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402136.740141] CPU[ 95]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/95:0]
[402136.760578] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402136.785459] CPU[ 96]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/96:0]
[402136.805895] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402136.830787] CPU[ 97]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/97:0]
[402136.851212] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402136.876094] CPU[ 98]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/98:0]
[402136.896547] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402136.921424] CPU[ 99]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/99:0]
[402136.941863] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402136.966741] CPU[100]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/100:0]
[402136.987359] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402137.012237] CPU[101]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/101:0]
[402137.032846] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402137.057723] CPU[102]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/102:0]
[402137.078336] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402137.103230] CPU[103]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/103:0]
[402137.123826] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402137.148728] CPU[104]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/104:0]
[402137.169327] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402137.194204] CPU[105]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/105:0]
[402137.214817] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402137.239703] CPU[106]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/106:0]
[402137.260316] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402137.285194] CPU[107]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/107:0]
[402137.305807] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402137.330685] CPU[108]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/108:0]
[402137.351297] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402137.376192] CPU[109]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/109:0]
[402137.396788] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402137.421682] CPU[110]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/110:0]
[402137.442297] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402137.467178] CPU[111]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/111:0]
[402137.487775] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402137.512669] CPU[112]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/112:0]
[402137.533279] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402137.558158] CPU[113]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/113:0]
[402137.578773] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402137.603650] CPU[114]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/114:0]
[402137.624265] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402137.649142] CPU[115]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/115:0]
[402137.669755] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402137.694643] CPU[116]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/116:0]
[402137.715256] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402137.740133] CPU[117]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/117:0]
[402137.760747] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402137.785624] CPU[118]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/118:0]
[402137.806236] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402137.831116] CPU[119]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/119:0]
[402137.851730] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402137.876610] CPU[120]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/120:0]
[402137.897223] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402137.922100] CPU[121]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/121:0]
[402137.942713] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402137.967596] CPU[122]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/122:0]
[402137.988209] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402138.013099] CPU[123]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/123:0]
[402138.033714] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402138.058584] CPU[124]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/124:0]
[402138.079197] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402138.104074] CPU[125]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/125:0]
[402138.124687] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402138.149583] CPU[126]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/126:0]
[402138.170183] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
[402138.195078] CPU[127]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/127:0]
[402138.215685] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
[402138.240512] rcu_sched kthread starved for 2757 jiffies! g53669 c53668 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
[402138.259779] rcu_sched S 0 9 2 0x06000000
[402138.270887] Call Trace:
[402138.275922] [00000000009af81c] schedule+0x1c/0xa0
[402138.285646] [00000000009b2ac0] schedule_timeout+0x1a0/0x340
[402138.297110] [00000000004c3dcc] rcu_gp_kthread+0x68c/0xa60
[402138.308222] [000000000048dd14] kthread+0xd4/0x120
[402138.317959] [0000000000406064] ret_from_fork+0x1c/0x2c
[402138.328537] [0000000000000000] (null)
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RCU stall warnings...
2017-07-24 22:32 RCU stall warnings David Miller
@ 2017-07-24 23:20 ` Paul E. McKenney
2017-07-24 23:34 ` David Miller
0 siblings, 1 reply; 6+ messages in thread
From: Paul E. McKenney @ 2017-07-24 23:20 UTC (permalink / raw)
To: David Miller; +Cc: linux-kernel, sparclinux
On Mon, Jul 24, 2017 at 03:32:48PM -0700, David Miller wrote:
>
> Paul and other RCU experts,
>
> Starting with 4.13-rc1 we're getting RCU stall dumps on sparc64, they
> definitely didn't happen in 4.12
>
> I tried to look for low hanging fruit in the kernel/rcu/ changes this
> merge window, but nothing really stood out for me.
>
> This only seems to trigger when the system is mostly idle.
>
> Kconfig wise PREEMPT is disabled, and NO_HZ_COMMON and NO_HOZ_IDLE are
> both enabled.
>
> The dumps always look the same, see below. The idle values are even,
> so those cpus should be in dynticks mode.
>
> Perhaps I'm not doing something properly in our arch idle loop, but I
> don't see anything obvious there.
It looks like the system isn't letting the rcu_sched grace-period kthread
run:
[402138.240512] rcu_sched kthread starved for 2757 jiffies! g53669 c53668 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
This kthread tried to wait for a few jiffies (the exact number depends
on HZ and the number of CPUs), but 2,757 jiffies have elapsed and it is
still waiting. This kthread is responsible for detecting idle CPUs and
reporting quiescent states on their behalf, so if this kthread doesn't
get a chance to run, then the stall warnings you are seeing are expected
behavior.
I am seeing someething like sort of like this in my rcutorture runs,
but only when I boot with nr_cpus quite a bit bigger than maxcpus, as in
something like nr_cpus=43 and maxcpus=8. This causes 8 CPUs to be brought
online at the usual time, and the other 35 come online some time later.
One difference from your situation is that I see the grace-period
kthread in ->state=0x401 (TASK_WAKING) instead of your ->state=0x1.
If I send extra wakeups to the grace-period kthread (which shouldn't be
needed), it does make progress, but then other kthreads fall into that
same half-woken state.
So now that I ahve shared the full extent of my ignorance on this topic,
any ideas? ;-)
Thanx, Paul
> [402131.682905] INFO: rcu_sched detected stalls on CPUs/tasks:
> [402131.693768] 2-...: (4304 GPs behind) idle=990/0/0 softirq=80469/80469 fqs=0
> [402131.708188] 3-...: (3286 GPs behind) idle=0bc/0/0 softirq=77239/77241 fqs=0
> [402131.722595] 4-...: (5513 GPs behind) idle=2b4/0/0 softirq=74384/74386 fqs=0
> [402131.737003] 5-...: (3596 GPs behind) idle=ad0/0/0 softirq=73109/73109 fqs=0
> [402131.751411] 6-...: (3592 GPs behind) idle=e78/0/0 softirq=67679/67679 fqs=0
> [402131.765835] 7-...: (3591 GPs behind) idle=9ac/0/0 softirq=65887/65889 fqs=0
> [402131.780243] 9-...: (5 GPs behind) idle=724/0/0 softirq=88433/88436 fqs=0
> [402131.794121] 11-...: (3286 GPs behind) idle=be4/0/0 softirq=81587/81589 fqs=0
> [402131.808724] 12-...: (3596 GPs behind) idle=ed0/0/0 softirq=79542/79544 fqs=0
> [402131.823307] 13-...: (3596 GPs behind) idle=9f8/0/0 softirq=73652/73655 fqs=0
> [402131.837891] 14-...: (3593 GPs behind) idle=d3c/0/0 softirq=72326/72328 fqs=0
> [402131.852476] 15-...: (3596 GPs behind) idle=6f4/0/0 softirq=69730/69732 fqs=0
> [402131.867060] 20-...: (2566 GPs behind) idle=b80/0/0 softirq=79921/79925 fqs=0
> [402131.881644] 21-...: (4701 GPs behind) idle=944/0/0 softirq=73890/73890 fqs=0
> [402131.896228] 22-...: (2563 GPs behind) idle=b58/0/0 softirq=74554/74554 fqs=0
> [402131.910812] 23-...: (3285 GPs behind) idle=b08/0/0 softirq=112972/112972 fqs=0
> [402131.925749] 27-...: (3596 GPs behind) idle=5f4/0/0 softirq=79524/79524 fqs=0
> [402131.940333] 28-...: (4070 GPs behind) idle=b30/0/0 softirq=71061/71063 fqs=0
> [402131.954917] 29-...: (2430 GPs behind) idle=4b4/0/0 softirq=72431/72433 fqs=0
> [402131.969501] 30-...: (389 GPs behind) idle=65c/0/0 softirq=68508/68510 fqs=0
> [402131.983925] 31-...: (170 GPs behind) idle=174/0/0 softirq=72802/72804 fqs=0
> [402131.998334] 33-...: (1627 GPs behind) idle=d2c/0/0 softirq=89821/89823 fqs=0
> [402132.012918] 34-...: (391 GPs behind) idle=8b0/0/0 softirq=82561/82563 fqs=0
> [402132.027326] 35-...: (169 GPs behind) idle=700/0/0 softirq=77960/77962 fqs=0
> [402132.041734] 36-...: (3590 GPs behind) idle=cb4/0/0 softirq=79230/79232 fqs=0
> [402132.056334] 37-...: (391 GPs behind) idle=264/0/0 softirq=71121/71123 fqs=0
> [402132.070743] 38-...: (3594 GPs behind) idle=604/0/0 softirq=71422/71422 fqs=0
> [402132.085327] 39-...: (3409 GPs behind) idle=eb4/0/0 softirq=67121/67121 fqs=0
> [402132.099911] 42-...: (172 GPs behind) idle=d4c/0/0 softirq=121584/121588 fqs=0
> [402132.114671] 43-...: (388 GPs behind) idle=3f0/0/0 softirq=80541/80541 fqs=0
> [402132.129079] 44-...: (3605 GPs behind) idle=410/0/0 softirq=73502/73502 fqs=0
> [402132.143663] 45-...: (3591 GPs behind) idle=078/0/0 softirq=74292/74292 fqs=0
> [402132.158247] 46-...: (3594 GPs behind) idle=800/0/0 softirq=73200/73200 fqs=0
> [402132.172832] 47-...: (3599 GPs behind) idle=460/0/0 softirq=68045/68047 fqs=0
> [402132.187433] 50-...: (97 GPs behind) idle=180/0/0 softirq=81843/81843 fqs=0
> [402132.201662] 51-...: (172 GPs behind) idle=5bc/0/0 softirq=76273/76277 fqs=0
> [402132.216072] 52-...: (2915 GPs behind) idle=688/0/0 softirq=76044/76046 fqs=0
> [402132.230656] 53-...: (2564 GPs behind) idle=6dc/0/0 softirq=75980/75982 fqs=0
> [402132.245240] 54-...: (6357 GPs behind) idle=a40/0/0 softirq=72584/72584 fqs=0
> [402132.259841] 55-...: (3286 GPs behind) idle=c34/0/0 softirq=70795/70795 fqs=0
> [402132.274425] 57-...: (4 GPs behind) idle=d0c/0/0 softirq=90478/90480 fqs=0
> [402132.288479] 59-...: (0 ticks this GP) idle=d2c/0/0 softirq=79767/79767 fqs=0
> [402132.303065] 61-...: (5511 GPs behind) idle=4f0/0/0 softirq=78603/78603 fqs=0
> [402132.317649] 62-...: (5509 GPs behind) idle=e88/0/0 softirq=73573/73575 fqs=0
> [402132.332233] 63-...: (5518 GPs behind) idle=d94/0/0 softirq=71343/71343 fqs=0
> [402132.346832] 64-...: (55 GPs behind) idle=a14/0/0 softirq=122673/122673 fqs=0
> [402132.361417] 65-...: (134 GPs behind) idle=4a8/0/0 softirq=97047/97047 fqs=0
> [402132.375825] 66-...: (2523 GPs behind) idle=418/0/0 softirq=81713/81713 fqs=0
> [402132.390409] 67-...: (2470 GPs behind) idle=000/0/0 softirq=84375/84377 fqs=0
> [402132.405001] 68-...: (2565 GPs behind) idle=fcc/0/0 softirq=72096/72096 fqs=0
> [402132.419586] 69-...: (3655 GPs behind) idle=1e4/0/0 softirq=71565/71567 fqs=0
> [402132.434170] 70-...: (3597 GPs behind) idle=89c/0/0 softirq=68341/68341 fqs=0
> [402132.448754] 71-...: (3605 GPs behind) idle=f94/0/0 softirq=64666/64668 fqs=0
> [402132.463333] 72-...: (49 GPs behind) idle=b40/0/0 softirq=112848/112848 fqs=0
> [402132.477924] 73-...: (160 GPs behind) idle=db8/0/0 softirq=85959/85961 fqs=0
> [402132.492340] 74-...: (157 GPs behind) idle=d28/0/0 softirq=81416/81416 fqs=0
> [402132.506748] 75-...: (186 GPs behind) idle=34c/0/0 softirq=85910/85912 fqs=0
> [402132.521158] 76-...: (4072 GPs behind) idle=764/0/0 softirq=75200/75202 fqs=0
> [402132.535744] 77-...: (3695 GPs behind) idle=e28/0/0 softirq=76678/76678 fqs=0
> [402132.550331] 78-...: (3691 GPs behind) idle=5b4/0/0 softirq=72681/72681 fqs=0
> [402132.564925] 79-...: (11143 GPs behind) idle=4a8/0/0 softirq=73212/73212 fqs=0
> [402132.579670] 80-...: (132 GPs behind) idle=044/0/0 softirq=117329/117329 fqs=0
> [402132.594429] 81-...: (157 GPs behind) idle=cb4/0/0 softirq=87921/87923 fqs=0
> [402132.608846] 82-...: (2432 GPs behind) idle=c3c/0/0 softirq=83531/83531 fqs=0
> [402132.623432] 83-...: (389 GPs behind) idle=c94/0/0 softirq=79845/79847 fqs=0
> [402132.637840] 84-...: (3588 GPs behind) idle=990/0/0 softirq=75763/75765 fqs=0
> [402132.652424] 85-...: (3593 GPs behind) idle=be0/0/0 softirq=76481/76483 fqs=0
> [402132.667020] 86-...: (5254 GPs behind) idle=4c0/0/0 softirq=73565/73567 fqs=0
> [402132.681605] 87-...: (5507 GPs behind) idle=4a8/0/0 softirq=70595/70597 fqs=0
> [402132.696182] 88-...: (162 GPs behind) idle=2ac/0/0 softirq=120362/120364 fqs=0
> [402132.710942] 89-...: (155 GPs behind) idle=f6c/0/0 softirq=86298/86300 fqs=0
> [402132.725367] 90-...: (161 GPs behind) idle=23c/0/0 softirq=81845/81845 fqs=0
> [402132.739779] 91-...: (5236 GPs behind) idle=a84/0/0 softirq=76524/76524 fqs=0
> [402132.754363] 92-...: (5073 GPs behind) idle=46c/0/0 softirq=78364/78366 fqs=0
> [402132.768952] 93-...: (7496 GPs behind) idle=590/0/0 softirq=75926/75928 fqs=0
> [402132.783536] 94-...: (3594 GPs behind) idle=fdc/0/0 softirq=73619/73619 fqs=0
> [402132.798110] 95-...: (446 GPs behind) idle=4d8/0/0 softirq=67953/67953 fqs=0
> [402132.812535] 96-...: (156 GPs behind) idle=4b0/0/0 softirq=151231/151231 fqs=0
> [402132.827295] 97-...: (2066 GPs behind) idle=4a0/0/0 softirq=90382/90384 fqs=0
> [402132.841879] 98-...: (3592 GPs behind) idle=e10/0/0 softirq=85418/85420 fqs=0
> [402132.856454] 99-...: (90 GPs behind) idle=6c4/0/0 softirq=116865/116865 fqs=0
> [402132.871038] 100-...: (162 GPs behind) idle=aa0/0/0 softirq=77088/77089 fqs=0
> [402132.885627] 101-...: (157 GPs behind) idle=5e8/0/0 softirq=75034/75036 fqs=0
> [402132.900211] 102-...: (3591 GPs behind) idle=a08/0/0 softirq=72174/72176 fqs=0
> [402132.914971] 103-...: (3605 GPs behind) idle=f2c/0/0 softirq=68188/68190 fqs=0
> [402132.929736] 104-...: (155 GPs behind) idle=0cc/0/0 softirq=107026/107026 fqs=0
> [402132.944672] 105-...: (157 GPs behind) idle=c24/0/0 softirq=83549/83551 fqs=0
> [402132.959256] 106-...: (1625 GPs behind) idle=6dc/0/0 softirq=80774/80774 fqs=0
> [402132.974007] 107-...: (3589 GPs behind) idle=d4c/0/0 softirq=82554/82556 fqs=0
> [402132.988768] 108-...: (3588 GPs behind) idle=cc8/0/0 softirq=76988/76990 fqs=0
> [402133.003528] 109-...: (3597 GPs behind) idle=644/0/0 softirq=74768/74768 fqs=0
> [402133.018288] 110-...: (5176 GPs behind) idle=96c/0/0 softirq=72869/72871 fqs=0
> [402133.033048] 111-...: (3401 GPs behind) idle=458/0/0 softirq=70859/70859 fqs=0
> [402133.047809] 112-...: (125 GPs behind) idle=308/0/0 softirq=108176/108176 fqs=0
> [402133.062745] 113-...: (157 GPs behind) idle=e24/0/0 softirq=83982/83984 fqs=0
> [402133.077321] 114-...: (725 GPs behind) idle=4b0/0/0 softirq=80102/80103 fqs=0
> [402133.091922] 115-...: (3596 GPs behind) idle=744/0/0 softirq=79547/79547 fqs=0
> [402133.106675] 116-...: (4977 GPs behind) idle=298/0/0 softirq=75316/75318 fqs=0
> [402133.121435] 117-...: (3591 GPs behind) idle=00c/0/0 softirq=75114/75114 fqs=0
> [402133.136188] 118-...: (3592 GPs behind) idle=d70/0/0 softirq=73200/73200 fqs=0
> [402133.150946] 119-...: (5255 GPs behind) idle=d28/0/0 softirq=67479/67479 fqs=0
> [402133.165708] 120-...: (138 GPs behind) idle=e08/0/0 softirq=112097/112099 fqs=0
> [402133.180641] 121-...: (156 GPs behind) idle=4d0/0/0 softirq=89608/89608 fqs=0
> [402133.195225] 122-...: (389 GPs behind) idle=648/0/0 softirq=81157/81159 fqs=0
> [402133.209814] 123-...: (389 GPs behind) idle=3b0/0/0 softirq=81959/81961 fqs=0
> [402133.224405] 124-...: (4102 GPs behind) idle=b50/0/0 softirq=75522/75524 fqs=0
> [402133.239165] 125-...: (3591 GPs behind) idle=f64/0/0 softirq=74702/74702 fqs=0
> [402133.253926] 126-...: (3593 GPs behind) idle=2a4/0/0 softirq=70870/70872 fqs=0
> [402133.268686] 127-...: (4228 GPs behind) idle=508/0/0 softirq=72137/72139 fqs=0
> [402133.283438] (detected by 48, t=2262 jiffies, g=53669, c=53668, q=18005)
> [402133.297060] CPU[ 2]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/2:0]
> [402133.317316] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.342191] CPU[ 3]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/3:0]
> [402133.362453] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402133.387329] CPU[ 4]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/4:0]
> [402133.407593] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.432470] CPU[ 5]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/5:0]
> [402133.452747] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402133.477624] CPU[ 6]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/6:0]
> [402133.497885] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402133.522763] CPU[ 7]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/7:0]
> [402133.543040] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402133.567920] CPU[ 9]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/9:0]
> [402133.588178] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.613055] CPU[ 11]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/11:0]
> [402133.633509] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402133.658386] CPU[ 12]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/12:0]
> [402133.678823] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.703709] CPU[ 13]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/13:0]
> [402133.724146] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.749023] CPU[ 14]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/14:0]
> [402133.769459] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.794336] CPU[ 15]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/15:0]
> [402133.814773] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.839667] CPU[ 20]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/20:0]
> [402133.860104] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402133.884981] CPU[ 21]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/21:0]
> [402133.905414] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.930298] CPU[ 22]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/22:0]
> [402133.950747] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402133.975617] CPU[ 23]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/23:0]
> [402133.996053] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.020947] CPU[ 27]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/27:0]
> [402134.041384] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.066269] CPU[ 28]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/28:0]
> [402134.086705] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.111582] CPU[ 29]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/29:0]
> [402134.132018] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.156896] CPU[ 30]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/30:0]
> [402134.177333] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402134.202229] CPU[ 31]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/31:0]
> [402134.222666] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402134.247548] CPU[ 33]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/33:0]
> [402134.267978] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.292856] CPU[ 34]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/34:0]
> [402134.313292] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402134.338186] CPU[ 35]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/35:0]
> [402134.358625] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.383502] CPU[ 36]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/36:0]
> [402134.403939] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.428817] CPU[ 37]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/37:0]
> [402134.449265] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.474142] CPU[ 38]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/38:0]
> [402134.494572] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.519457] CPU[ 39]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/39:0]
> [402134.539894] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402134.564788] CPU[ 42]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/42:0]
> [402134.585224] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402134.610095] CPU[ 43]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/43:0]
> [402134.630548] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.655425] CPU[ 44]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/44:0]
> [402134.675861] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.700744] CPU[ 45]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/45:0]
> [402134.721180] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402134.746057] CPU[ 46]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/46:0]
> [402134.766494] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.791379] CPU[ 47]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/47:0]
> [402134.811816] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402134.836710] CPU[ 50]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/50:0]
> [402134.857146] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.882023] CPU[ 51]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/51:0]
> [402134.902460] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402134.927337] CPU[ 52]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/52:0]
> [402134.947773] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402134.972667] CPU[ 53]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/53:0]
> [402134.993104] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.017987] CPU[ 54]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/54:0]
> [402135.038424] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.063301] CPU[ 55]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/55:0]
> [402135.083737] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.108614] CPU[ 57]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/57:0]
> [402135.129067] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.153945] CPU[ 59]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/59:0]
> [402135.174381] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.199258] CPU[ 61]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/61:0]
> [402135.219705] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.244582] CPU[ 62]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/62:0]
> [402135.265019] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.289896] CPU[ 63]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/63:0]
> [402135.310332] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.335231] CPU[ 64]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/64:0]
> [402135.355653] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.380545] CPU[ 65]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/65:0]
> [402135.400982] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.425859] CPU[ 66]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/66:0]
> [402135.446295] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.471181] CPU[ 67]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/67:0]
> [402135.491617] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.516495] CPU[ 68]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/68:0]
> [402135.536931] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.561825] CPU[ 69]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/69:0]
> [402135.582262] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.607140] CPU[ 70]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/70:0]
> [402135.627595] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.652495] CPU[ 71]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/71:0]
> [402135.672926] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.697815] CPU[ 72]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/72:0]
> [402135.718231] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.743099] CPU[ 73]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/73:0]
> [402135.763537] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.788420] CPU[ 74]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/74:0]
> [402135.808853] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.833748] CPU[ 75]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/75:0]
> [402135.854184] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.879062] CPU[ 76]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/76:0]
> [402135.899499] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402135.924377] CPU[ 77]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/77:0]
> [402135.944813] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402135.969707] CPU[ 78]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/78:0]
> [402135.990144] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.015022] CPU[ 79]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/79:0]
> [402136.035460] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.060338] CPU[ 80]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/80:0]
> [402136.080783] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.105667] CPU[ 81]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/81:0]
> [402136.126100] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.150977] CPU[ 82]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/82:0]
> [402136.171413] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.196308] CPU[ 83]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/83:0]
> [402136.216745] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.241623] CPU[ 84]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/84:0]
> [402136.262059] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.286937] CPU[ 85]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/85:0]
> [402136.307373] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.332267] CPU[ 86]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/86:0]
> [402136.352704] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.377586] CPU[ 87]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/87:0]
> [402136.398023] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.422900] CPU[ 88]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/88:0]
> [402136.443337] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.468213] CPU[ 89]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/89:0]
> [402136.488667] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.513544] CPU[ 90]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/90:0]
> [402136.533981] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.558859] CPU[ 91]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/91:0]
> [402136.579304] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.604182] CPU[ 92]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/92:0]
> [402136.624619] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.649496] CPU[ 93]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/93:0]
> [402136.669933] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.694827] CPU[ 94]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/94:0]
> [402136.715264] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.740141] CPU[ 95]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/95:0]
> [402136.760578] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.785459] CPU[ 96]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/96:0]
> [402136.805895] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.830787] CPU[ 97]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/97:0]
> [402136.851212] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.876094] CPU[ 98]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/98:0]
> [402136.896547] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402136.921424] CPU[ 99]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/99:0]
> [402136.941863] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402136.966741] CPU[100]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/100:0]
> [402136.987359] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.012237] CPU[101]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/101:0]
> [402137.032846] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.057723] CPU[102]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/102:0]
> [402137.078336] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.103230] CPU[103]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/103:0]
> [402137.123826] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.148728] CPU[104]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/104:0]
> [402137.169327] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.194204] CPU[105]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/105:0]
> [402137.214817] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.239703] CPU[106]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/106:0]
> [402137.260316] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.285194] CPU[107]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/107:0]
> [402137.305807] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.330685] CPU[108]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/108:0]
> [402137.351297] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.376192] CPU[109]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/109:0]
> [402137.396788] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.421682] CPU[110]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/110:0]
> [402137.442297] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.467178] CPU[111]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/111:0]
> [402137.487775] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.512669] CPU[112]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/112:0]
> [402137.533279] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.558158] CPU[113]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/113:0]
> [402137.578773] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.603650] CPU[114]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/114:0]
> [402137.624265] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.649142] CPU[115]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/115:0]
> [402137.669755] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.694643] CPU[116]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/116:0]
> [402137.715256] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.740133] CPU[117]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/117:0]
> [402137.760747] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.785624] CPU[118]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/118:0]
> [402137.806236] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.831116] CPU[119]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/119:0]
> [402137.851730] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.876610] CPU[120]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/120:0]
> [402137.897223] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402137.922100] CPU[121]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/121:0]
> [402137.942713] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402137.967596] CPU[122]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/122:0]
> [402137.988209] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402138.013099] CPU[123]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/123:0]
> [402138.033714] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402138.058584] CPU[124]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/124:0]
> [402138.079197] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402138.104074] CPU[125]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/125:0]
> [402138.124687] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402138.149583] CPU[126]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/126:0]
> [402138.170183] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x14/0x40]
> [402138.195078] CPU[127]: TSTATE[0000000080001603] TPC[000000000042c3d4] TNPC[000000000042c3d8] TASK[swapper/127:0]
> [402138.215685] TPC[arch_cpu_idle+0x74/0xa0] O7[arch_cpu_idle+0x60/0xa0] I7[do_idle+0xc4/0x1a0] RPC[cpu_startup_entry+0x1c/0x40]
> [402138.240512] rcu_sched kthread starved for 2757 jiffies! g53669 c53668 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
> [402138.259779] rcu_sched S 0 9 2 0x06000000
> [402138.270887] Call Trace:
> [402138.275922] [00000000009af81c] schedule+0x1c/0xa0
> [402138.285646] [00000000009b2ac0] schedule_timeout+0x1a0/0x340
> [402138.297110] [00000000004c3dcc] rcu_gp_kthread+0x68c/0xa60
> [402138.308222] [000000000048dd14] kthread+0xd4/0x120
> [402138.317959] [0000000000406064] ret_from_fork+0x1c/0x2c
> [402138.328537] [0000000000000000] (null)
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RCU stall warnings...
2017-07-24 23:20 ` Paul E. McKenney
@ 2017-07-24 23:34 ` David Miller
2017-07-24 23:49 ` Paul E. McKenney
2017-07-25 3:45 ` Stephen Rothwell
0 siblings, 2 replies; 6+ messages in thread
From: David Miller @ 2017-07-24 23:34 UTC (permalink / raw)
To: paulmck; +Cc: linux-kernel, sparclinux
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Date: Mon, 24 Jul 2017 16:20:33 -0700
> It looks like the system isn't letting the rcu_sched grace-period kthread
> run:
>
> [402138.240512] rcu_sched kthread starved for 2757 jiffies! g53669 c53668 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
>
> This kthread tried to wait for a few jiffies (the exact number depends
> on HZ and the number of CPUs), but 2,757 jiffies have elapsed and it is
> still waiting. This kthread is responsible for detecting idle CPUs and
> reporting quiescent states on their behalf, so if this kthread doesn't
> get a chance to run, then the stall warnings you are seeing are expected
> behavior.
>
> I am seeing someething like sort of like this in my rcutorture runs,
> but only when I boot with nr_cpus quite a bit bigger than maxcpus, as in
> something like nr_cpus=43 and maxcpus=8. This causes 8 CPUs to be brought
> online at the usual time, and the other 35 come online some time later.
> One difference from your situation is that I see the grace-period
> kthread in ->state=0x401 (TASK_WAKING) instead of your ->state=0x1.
> If I send extra wakeups to the grace-period kthread (which shouldn't be
> needed), it does make progress, but then other kthreads fall into that
> same half-woken state.
>
> So now that I ahve shared the full extent of my ignorance on this topic,
> any ideas? ;-)
Shoing my ignorance as well, after reading this, for some reason this
commit below sticks out to me. Maybe I should do a bisect and see if
it lands on this commit.
That would take a while as it's hard to forcibly set this thing off.
====================
commit f92c734f02cbf10e40569facff82059ae9b61920
Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Date: Mon Apr 10 15:40:35 2017 -0700
rcu: Prevent rcu_barrier() from starting needless grace periods
Currently rcu_barrier() uses call_rcu() to enqueue new callbacks
on each CPU with a non-empty callback list. This works, but means
that rcu_barrier() forces grace periods that are not otherwise needed.
The key point is that rcu_barrier() never needs to wait for a grace
period, but instead only for all pre-existing callbacks to be invoked.
This means that rcu_barrier()'s new callbacks should be placed in
the callback-list segment containing the last pre-existing callback.
This commit makes this change using the new rcu_segcblist_entrain()
function.
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RCU stall warnings...
2017-07-24 23:34 ` David Miller
@ 2017-07-24 23:49 ` Paul E. McKenney
2017-07-25 2:44 ` Paul E. McKenney
2017-07-25 3:45 ` Stephen Rothwell
1 sibling, 1 reply; 6+ messages in thread
From: Paul E. McKenney @ 2017-07-24 23:49 UTC (permalink / raw)
To: David Miller; +Cc: linux-kernel, sparclinux
On Mon, Jul 24, 2017 at 04:34:58PM -0700, David Miller wrote:
> From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> Date: Mon, 24 Jul 2017 16:20:33 -0700
>
> > It looks like the system isn't letting the rcu_sched grace-period kthread
> > run:
> >
> > [402138.240512] rcu_sched kthread starved for 2757 jiffies! g53669 c53668 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
> >
> > This kthread tried to wait for a few jiffies (the exact number depends
> > on HZ and the number of CPUs), but 2,757 jiffies have elapsed and it is
> > still waiting. This kthread is responsible for detecting idle CPUs and
> > reporting quiescent states on their behalf, so if this kthread doesn't
> > get a chance to run, then the stall warnings you are seeing are expected
> > behavior.
> >
> > I am seeing someething like sort of like this in my rcutorture runs,
> > but only when I boot with nr_cpus quite a bit bigger than maxcpus, as in
> > something like nr_cpus=43 and maxcpus=8. This causes 8 CPUs to be brought
> > online at the usual time, and the other 35 come online some time later.
> > One difference from your situation is that I see the grace-period
> > kthread in ->state=0x401 (TASK_WAKING) instead of your ->state=0x1.
> > If I send extra wakeups to the grace-period kthread (which shouldn't be
> > needed), it does make progress, but then other kthreads fall into that
> > same half-woken state.
> >
> > So now that I ahve shared the full extent of my ignorance on this topic,
> > any ideas? ;-)
>
> Shoing my ignorance as well, after reading this, for some reason this
> commit below sticks out to me. Maybe I should do a bisect and see if
> it lands on this commit.
I would be very surprised if this commit was the culprit, but then
again, I have been very surprised before.
> That would take a while as it's hard to forcibly set this thing off.
And my similar error can take awhile as well. But maybe I should try
forcing nr_cpus=43 and maxcpus=8 on older versions to see what happens.
A bisection would of course be quite helpful, depending of course on
the value of "a while". ;-)
Thanx, Paul
> ====================
> commit f92c734f02cbf10e40569facff82059ae9b61920
> Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Date: Mon Apr 10 15:40:35 2017 -0700
>
> rcu: Prevent rcu_barrier() from starting needless grace periods
>
> Currently rcu_barrier() uses call_rcu() to enqueue new callbacks
> on each CPU with a non-empty callback list. This works, but means
> that rcu_barrier() forces grace periods that are not otherwise needed.
> The key point is that rcu_barrier() never needs to wait for a grace
> period, but instead only for all pre-existing callbacks to be invoked.
> This means that rcu_barrier()'s new callbacks should be placed in
> the callback-list segment containing the last pre-existing callback.
>
> This commit makes this change using the new rcu_segcblist_entrain()
> function.
>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RCU stall warnings...
2017-07-24 23:49 ` Paul E. McKenney
@ 2017-07-25 2:44 ` Paul E. McKenney
0 siblings, 0 replies; 6+ messages in thread
From: Paul E. McKenney @ 2017-07-25 2:44 UTC (permalink / raw)
To: David Miller; +Cc: linux-kernel, sparclinux
On Mon, Jul 24, 2017 at 04:49:27PM -0700, Paul E. McKenney wrote:
> On Mon, Jul 24, 2017 at 04:34:58PM -0700, David Miller wrote:
> > From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> > Date: Mon, 24 Jul 2017 16:20:33 -0700
[ . . . ]
> > That would take a while as it's hard to forcibly set this thing off.
>
> And my similar error can take awhile as well. But maybe I should try
> forcing nr_cpus=43 and maxcpus=8 on older versions to see what happens.
>
> A bisection would of course be quite helpful, depending of course on
> the value of "a while". ;-)
And if "a while" is too long, one alternative is to enable event tracing
for timers, as in why didn't they wake the grace-period kthread?
For example, it is possible that that kthread set its status (which
was printed out in the "rcu_sched kthread starved" message) but for
whatever reason didn't make it to the swait_event_idle_timeout()
immediately afterwards;
rsp->gp_state = RCU_GP_WAIT_FQS;
ret = swait_event_idle_timeout(rsp->gp_wq,
rcu_gp_fqs_check_wake(rsp, &gf), j);
Or that the timer-based wakeup didn't happen for whatever reason. Or...
Thanx, Paul
> > ====================
> > commit f92c734f02cbf10e40569facff82059ae9b61920
> > Author: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Date: Mon Apr 10 15:40:35 2017 -0700
> >
> > rcu: Prevent rcu_barrier() from starting needless grace periods
> >
> > Currently rcu_barrier() uses call_rcu() to enqueue new callbacks
> > on each CPU with a non-empty callback list. This works, but means
> > that rcu_barrier() forces grace periods that are not otherwise needed.
> > The key point is that rcu_barrier() never needs to wait for a grace
> > period, but instead only for all pre-existing callbacks to be invoked.
> > This means that rcu_barrier()'s new callbacks should be placed in
> > the callback-list segment containing the last pre-existing callback.
> >
> > This commit makes this change using the new rcu_segcblist_entrain()
> > function.
> >
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RCU stall warnings...
2017-07-24 23:34 ` David Miller
2017-07-24 23:49 ` Paul E. McKenney
@ 2017-07-25 3:45 ` Stephen Rothwell
1 sibling, 0 replies; 6+ messages in thread
From: Stephen Rothwell @ 2017-07-25 3:45 UTC (permalink / raw)
To: David Miller; +Cc: paulmck, linux-kernel, sparclinux
Hi Dave,
On Mon, 24 Jul 2017 16:34:58 -0700 (PDT) David Miller <davem@davemloft.net> wrote:
>
> Shoing my ignorance as well, after reading this, for some reason this
> commit below sticks out to me. Maybe I should do a bisect and see if
> it lands on this commit.
>
> That would take a while as it's hard to forcibly set this thing off.
For a quick test, just try with the parent of that commit and (if that
is "good") then with that commit itself ...
--
Cheers,
Stephen Rothwell
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-07-25 3:45 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-07-24 22:32 RCU stall warnings David Miller
2017-07-24 23:20 ` Paul E. McKenney
2017-07-24 23:34 ` David Miller
2017-07-24 23:49 ` Paul E. McKenney
2017-07-25 2:44 ` Paul E. McKenney
2017-07-25 3:45 ` Stephen Rothwell
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).