rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17

[Ross Green <rgkernel@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 541 bytes --]

Continued testing with the latest linux-4.5-rc3 release.

Please find attached a copy of traces from dmesg:

There is a lot more debug and trace data so hopefully this will shed
some light on what might be happening here.

My testing remains run a series of simple benchmarks, let that run to
completion and then leave the system idle away with just a few daemons
running.

the self detected stalls in this instance turned up after a days run time.
There were  NO heavy artificial computational loads on the machine.



Regards,

Ross Green

[-- Attachment #2: dmesg-4.5-rc3 --]
[-- Type: application/octet-stream, Size: 29097 bytes --]

                                                                                                                           
# cat /proc/version                                                                                                         
Linux version 4.5.0-rc3-panda (rfg@red.dwarf) (gcc version 5.3.0 (Buildroot 2016.02-git-00183-g1682aee) ) #1 SMP PREEMPT Mon 6

$ 
$ 
$ 
$ uptime
 19:36:00 up 1 day, 25 min,  load average: 0.04, 0.08, 0.11
$ dmesg
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.5.0-rc3-panda (rfg@red.dwarf) (gcc version 5.3.0 (Buildroot 2016.02-git-00183-g1682aee) ) #1 SMP PREEMPT Mon Feb 8 17:35:31 AEDT 2016
[    0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] Machine model: TI OMAP4 PandaBoard-ES
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] OMAP4: Map 0xbfe00000 to fe600000 for dram barrier
[    0.000000] On node 0 totalpages: 261632
[    0.000000] free_area_init_node: node 0, pgdat c0966280, node_mem_map ef7f9000
[    0.000000]   Normal zone: 1536 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 196608 pages, LIFO batch:31
[    0.000000]   HighMem zone: 65024 pages, LIFO batch:15
[    0.000000] OMAP4460 ES1.1
[    0.000000] PERCPU: Embedded 12 pages/cpu @ef7ab000 s17984 r8192 d22976 u49152
[    0.000000] pcpu-alloc: s17984 r8192 d22976 u49152 alloc=12*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 260096
[    0.000000] Kernel command line: root=/dev/mmcblk0p2 rootwait console=ttyO2,115200
[    0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Memory: 1027232K/1046528K available (6879K kernel code, 252K rwdata, 2156K rodata, 304K init, 297K bss, 19296K reserved, 0K cma-reserved, 260096K highmem)
[    0.000000] Virtual kernel memory layout:
                   vector  : 0xffff0000 - 0xffff1000   (   4 kB)
                   fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
                   vmalloc : 0xf0800000 - 0xff800000   ( 240 MB)
                   lowmem  : 0xc0000000 - 0xf0000000   ( 768 MB)
                   pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
                   modules : 0xbf000000 - 0xbfe00000   (  14 MB)
                     .text : 0xc0008000 - 0xc08db0d4   (9037 kB)
                     .init : 0xc08dc000 - 0xc0928000   ( 304 kB)
                     .data : 0xc0928000 - 0xc0967280   ( 253 kB)
                      .bss : 0xc096a000 - 0xc09b47b8   ( 298 kB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000]  Build-time adjustment of leaf fanout to 32.
[    0.000000] NR_IRQS:16 nr_irqs:16 16
[    0.000000] L2C: platform modifies aux control register: 0x0e070000 -> 0x3e470000
[    0.000000] L2C: DT/platform modifies aux control register: 0x0e070000 -> 0x3e470000
[    0.000000] L2C-310 enabling early BRESP for Cortex-A9
[    0.000000] OMAP L2C310: ROM does not support power control setting
[    0.000000] L2C-310 dynamic clock gating disabled, standby mode disabled
[    0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
[    0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x4e470000
[    0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[    0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[    0.000000] ti_dt_clocks_register: failed to lookup clock node bandgap_fclk
[    0.000000] OMAP clockevent source: timer1 at 32768 Hz
[    0.000000] clocksource: 32k_counter: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 58327039986419 ns
[    0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65535999984741ns
[    0.000000] OMAP clocksource: 32k_counter at 32768 Hz
[    0.000854] Console: colour dummy device 80x30
[    0.000854] Calibrating delay loop... 1397.55 BogoMIPS (lpj=6987776)
[    0.089202] pid_max: default: 32768 minimum: 301
[    0.089263] Security Framework initialized
[    0.089324] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.089324] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.089935] CPU: Testing write buffer coherency: ok
[    0.090179] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.090240] Setting up static identity map for 0x80008280 - 0x800082d8
[    0.209197] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.209259] Brought up 2 CPUs
[    0.209289] SMP: Total of 2 processors activated (2795.11 BogoMIPS).
[    0.209289] CPU: All CPU(s) started in SVC mode.
[    0.210083] devtmpfs: initialized
[    0.226593] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[    0.227081] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.227233] xor: measuring software checksum speed
[    0.319183]    arm4regs  :   999.200 MB/sec
[    0.419189]    8regs     :   803.600 MB/sec
[    0.519165]    32regs    :   761.200 MB/sec
[    0.519195] xor: using function: arm4regs (999.200 MB/sec)
[    0.519439] pinctrl core: initialized pinctrl subsystem
[    0.521331] NET: Registered protocol family 16
[    0.522247] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.523956] omap_hwmod: l3_main_3 using broken dt data from ocp
[    0.525482] omap_hwmod: l3_main_2 using broken dt data from ocp
[    0.630645] OMAP GPIO hardware version 0.1
[    0.635864] omap-gpmc 50000000.gpmc: GPMC revision 6.0
[    0.635894] gpmc_mem_init: disabling cs 0 mapped at 0x0-0x1000000
[    0.636413] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[    0.636779] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[    0.637176] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[    0.648864] platform 4b501000.aes: Cannot lookup hwmod 'aes'
[    0.649139] platform 480a5000.des: Cannot lookup hwmod 'des'                                                                     
[    0.655120] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[    0.655151] hw-breakpoint: maximum watchpoint size is 4 bytes.
[    0.655761] omap4_sram_init:Unable to get sram pool needed to handle errata I688
[    0.656005] OMAP DMA hardware revision 0.0
[    0.889434] raid6: int32x1  gen()   103 MB/s
[    1.059570] raid6: int32x1  xor()    82 MB/s
[    1.229248] raid6: int32x2  gen()   134 MB/s
[    1.399230] raid6: int32x2  xor()    98 MB/s
[    1.569366] raid6: int32x4  gen()   145 MB/s
[    1.739349] raid6: int32x4  xor()    98 MB/s
[    1.909240] raid6: int32x8  gen()   160 MB/s
[    2.079406] raid6: int32x8  xor()    94 MB/s
[    2.079406] raid6: using algorithm int32x8 gen() 160 MB/s
[    2.079437] raid6: .... xor() 94 MB/s, rmw enabled
[    2.079437] raid6: using intx1 recovery algorithm
[    2.085418] omap-dma-engine 4a056000.dma-controller: OMAP DMA engine driver
[    2.159332] reg-fixed-voltage wl12xx_vmmc: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_wl12xx_gpio, deferring probe
[    2.163574] omap-iommu 4a066000.mmu: 4a066000.mmu registered
[    2.163696] omap-iommu 55082000.mmu: 55082000.mmu registered
[    2.165771] SCSI subsystem initialized
[    2.166046] usbcore: registered new interface driver usbfs
[    2.166168] usbcore: registered new interface driver hub
[    2.166320] usbcore: registered new device driver usb
[    2.167266] omap_i2c 48070000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c1_pins, deferring probe
[    2.167327] omap_i2c 48072000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c2_pins, deferring probe
[    2.167358] omap_i2c 48060000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c3_pins, deferring probe
[    2.167388] omap_i2c 48350000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c4_pins, deferring probe
[    2.167663] Linux video capture interface: v2.00
[    2.168395] Advanced Linux Sound Architecture Driver Initialized.
[    2.169830] clocksource: Switched to clocksource 32k_counter
[    2.183990] NET: Registered protocol family 2
[    2.184570] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[    2.184661] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[    2.184814] TCP: Hash tables configured (established 8192 bind 8192)
[    2.184875] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    2.184936] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    2.185119] NET: Registered protocol family 1
[    2.185516] RPC: Registered named UNIX socket transport module.
[    2.185516] RPC: Registered udp transport module.
[    2.185516] RPC: Registered tcp transport module.
[    2.185546] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    2.187530] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[    2.189666] futex hash table entries: 512 (order: 3, 32768 bytes)
[    2.208587] NFS: Registering the id_resolver key type
[    2.208618] Key type id_resolver registered
[    2.208618] Key type id_legacy registered
[    2.208648] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    2.208679] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    2.219909] ntfs: driver 2.1.32 [Flags: R/O].
[    2.220214] fuse init (API version 7.24)
[    2.223510] jitterentropy: Initialization failed with host not compliant with requirements: 2
[    2.223602] bounce: pool size: 64 pages
[    2.223632] io scheduler noop registered
[    2.223632] io scheduler deadline registered
[    2.223693] io scheduler cfq registered (default)
[    2.224548] pinctrl-single 4a100040.pinmux: 203 pins at pa fc100040 size 406
[    2.224792] pinctrl-single 4a31e040.pinmux: 28 pins at pa fc31e040 size 56
[    2.227752] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    2.229370] omap_uart 4806a000.serial: no wakeirq for uart0
[    2.229492] 4806a000.serial: ttyO0 at MMIO 0x4806a000 (irq = 226, base_baud = 3000000) is a OMAP UART0
[    2.230072] 4806c000.serial: ttyO1 at MMIO 0x4806c000 (irq = 227, base_baud = 3000000) is a OMAP UART1
[    2.230590] 48020000.serial: ttyO2 at MMIO 0x48020000 (irq = 228, base_baud = 3000000) is a OMAP UART2
[    3.127227] console [ttyO2] enabled
[    3.131378] 4806e000.serial: ttyO3 at MMIO 0x4806e000 (irq = 229, base_baud = 3000000) is a OMAP UART3
[    3.142120] [drm] Initialized drm 1.1.0 20060810
[    3.148651] 58000000.dss supply vdda_video not found, using dummy regulator
[    3.156036] OMAP DSS rev 4.0
[    3.159210] omapdss_dss 58000000.dss: bound 58001000.dispc (ops dispc_component_ops)
[    3.167541] omapdss_dss 58000000.dss: bound 58005000.encoder (ops dsi_component_ops)
[    3.176086] omapdss_dss 58000000.dss: bound 58006000.encoder (ops hdmi4_component_ops)
[    3.227966] loop: module loaded
[    3.234039] usbcore: registered new interface driver smsc95xx
[    3.240142] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    3.246978] ehci-omap: OMAP-EHCI Host Controller driver
[    3.269866] ehci-omap 4a064c00.ehci: EHCI Host Controller
[    3.275543] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1
[    3.283630] ehci-omap 4a064c00.ehci: irq 336, io mem 0x4a064c00
[    3.299865] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00
[    3.306243] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    3.313354] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.320953] usb usb1: Product: EHCI Host Controller
[    3.326049] usb usb1: Manufacturer: Linux 4.5.0-rc3-panda ehci_hcd
[    3.332519] usb usb1: SerialNumber: 4a064c00.ehci
[    3.338104] hub 1-0:1.0: USB hub found
[    3.342071] hub 1-0:1.0: 3 ports detected
[    3.346984] usbcore: registered new interface driver cdc_wdm
[    3.353057] usbcore: registered new interface driver usb-storage
[    3.359771] i2c /dev entries driver
[    3.363555] usbcore: registered new interface driver uvcvideo
[    3.369567] USB Video Class driver (1.1.1)
[    3.375091] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
[    3.382659] device-mapper: uevent: version 1.0.3
[    3.387817] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com
[    3.460845] hidraw: raw HID events driver (C) Jiri Kosina
[    3.466735] usbcore: registered new interface driver usbhid
[    3.472564] usbhid: USB HID core driver
[    3.477081] emif 4c000000.emif: emif_probe: device configured with addr = f08a8000 and IRQ247
[    3.486389] emif 4d000000.emif: emif_probe: device configured with addr = f08aa000 and IRQ248
[    3.497314] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[    3.505249] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517                                                         
[    3.512756] ip_tables: (C) 2000-2006 Netfilter Core Team
[    3.518432] Initializing XFRM netlink socket
[    3.522979] NET: Registered protocol family 17
[    3.527923] Key type dns_resolver registered
[    3.532623] twl: not initialized
[    3.536041] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[    3.544647] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[    3.553131] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[    3.561645] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[    3.570129] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[    3.578613] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[    3.587097] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1410000 Vs max 1316660
[    3.595581] omap2_set_init_voltage: unable to find boot up OPP for vdd_mpu
[    3.603149] omap2_set_init_voltage: unable to set vdd_mpu
[    3.608856] omap2_set_init_voltage: unable to find boot up OPP for vdd_core
[    3.616241] omap2_set_init_voltage: unable to set vdd_core
[    3.622100] omap2_set_init_voltage: unable to find boot up OPP for vdd_iva
[    3.629364] omap2_set_init_voltage: unable to set vdd_iva
[    3.635986] Power Management for TI OMAP4+ devices.
[    3.641174] OMAP4 PM: u-boot >= v2012.07 is required for full PM support
[    3.648345] ThumbEE CPU extension supported.
[    3.652984] Registering SWP/SWPB emulation handler
[    3.652984] SmartReflex Class3 initialized
[    3.662506] usb 1-1: new high-speed USB device number 2 using ehci-omap
[    3.687469] Btrfs loaded
[    3.692626] dmm 4e000000.dmm: initialized all PAT entries
[    3.718688] Skipping twl internal clock init and using bootloader value (unknown osc rate)
[    3.740264] twl 0-0048: PIH (irq 340) nested IRQs
[    3.746826] twl_rtc 48070000.i2c:twl@48:rtc: Enabling TWL-RTC
[    3.755584] twl_rtc 48070000.i2c:twl@48:rtc: rtc core: registered 48070000.i2c:twl@48 as rtc0
[    3.777282] twl6040 0-004b: clk32k is not handled
[    3.784545] omap_i2c 48070000.i2c: bus 0 rev0.11 at 400 kHz
[    3.791839] omap_i2c 48072000.i2c: bus 1 rev0.11 at 400 kHz
[    3.798645] omap_i2c 48060000.i2c: bus 2 rev0.11 at 100 kHz
[    3.805328] omap_i2c 48350000.i2c: bus 3 rev0.11 at 400 kHz
[    3.811676] omap_hsmmc 4809c000.mmc: omap_device: omap_device_enable() called from invalid state 1
[    3.849731] usb 1-1: New USB device found, idVendor=0424, idProduct=9514
[    3.856933] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    3.865173] hub 1-1:1.0: USB hub found
[    3.869964] hub 1-1:1.0: 5 ports detected
[    3.870361] omap_hsmmc 480d5000.mmc: omap_device: omap_device_enable() called from invalid state 1
[    4.020507] omap-abe-twl6040 sound: twl6040-legacy <-> 40132000.mcpdm mapping ok
[    4.033813] omap_hsmmc 480d5000.mmc: card claims to support voltages below defined range
[    4.043365] twl_rtc 48070000.i2c:twl@48:rtc: setting system clock to 2016-02-08 08:10:04 UTC (1454919004)
[    4.053649] sr_init: No PMIC hook to init smartreflex
[    4.059204] smartreflex smartreflex.0: omap_sr_probe: SmartReflex driver initialized
[    4.067810] smartreflex smartreflex.1: omap_sr_probe: SmartReflex driver initialized
[    4.076324] smartreflex smartreflex.2: omap_sr_probe: SmartReflex driver initialized
[    4.092987] mmc1: new SDIO card at address 0001
[    4.097869] mmc0: host does not support reading read-only switch, assuming write-enable
[    4.108367] VDAC: disabling
[    4.111511] mmc0: new high speed SDHC card at address e624
[    4.117645] VANA: disabling
[    4.121154] VUSB: disabling
[    4.124450] ALSA device list:
[    4.124450]   #0: PandaBoardES
[    4.130981] mmcblk0: mmc0:e624 SL32G 28.8 GiB 
[    4.165100]  mmcblk0: p1 p2 p3 p4
[    4.180999] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities
[    4.189727] usb 1-1.1: new high-speed USB device number 3 using ehci-omap
[    4.197662] EXT4-fs (mmcblk0p2): mounting ext2 file system using the ext4 subsystem
[    4.228881] EXT4-fs (mmcblk0p2): mounted filesystem without journal. Opts: (null)
[    4.236846] VFS: Mounted root (ext2 filesystem) readonly on device 179:2.
[    4.253448] devtmpfs: mounted
[    4.253448] Freeing unused kernel memory: 304K (c08dc000 - c0928000)
[    4.330383] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[    4.337707] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    4.348327] smsc95xx v1.0.4
[    4.458374] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-4a064c00.ehci-1.1, smsc95xx USB 2.0 Ethernet, 36:1e:61:c0:2a:6a
[    4.601470] EXT4-fs (mmcblk0p2): warning: mounting unchecked fs, running e2fsck is recommended
[    4.658142] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[    4.789886] random: dd urandom read with 29 bits of entropy available
[    7.270111] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[   17.209197] Bluetooth: Core ver 2.21
[   17.216766] NET: Registered protocol family 31
[   17.225830] Bluetooth: HCI device and connection manager initialized
[   17.236480] Bluetooth: HCI socket layer initialized
[   17.242248] Bluetooth: L2CAP socket layer initialized
[   17.247711] Bluetooth: SCO socket layer initialized
[   24.850402] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[   24.861816] NFSD: starting 90-second grace period (net c09607c0)
[   38.360900] random: nonblocking pool is initialized
[   39.164764] wlcore: loaded
[   52.577209] wlcore: firmware booted (Rev 6.3.10.0.133)
[   56.101531] wlan0: authenticate with 90:f6:52:81:ee:70
[   56.138732] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[   56.146728] wlan0: authenticated
[   56.162078] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[   56.172821] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=1)
[   56.188201] wlan0: associated
[   56.244812] wlcore: Association completed.
[  144.881042] NFSD: Unable to end grace period: -110
[  228.585998] Adding 2097148k swap on /dev/mmcblk0p3.  Priority:-1 extents:1 across:2097148k SS
[66515.022857] INFO: rcu_preempt self-detected stall on CPU
[66515.022888] INFO: rcu_preempt self-detected stall on CPU

[66515.022888]  1-...: (1 ticks this GP) idle=7cb/1/0 softirq=515093/515093 fqs=0 
[66515.022888] 
[66515.022888]  (t=8391 jiffies g=207416 c=207415 q=188)
[66515.022949] rcu_preempt kthread starved for 8391 jiffies! g207416 c207415 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
[66515.022949] rcu_preempt     S
[66515.022949]  c06b9c18 
[66515.022949]     0     7      2 0x00000000
[66515.022949] Backtrace: 

[66515.022979] [<c06b9a4c>] (__schedule) from [<c06b9fa8>] (schedule+0x54/0xc4)
[66515.023010]  r10:c092a518
[66515.023010]  r9:00000001
[66515.023010]  r8:0064e49b
[66515.023010]  r7:c092a100
[66515.023010]  r6:ef097ed4
[66515.023010]  r5:ef7b73c0
[66515.023010]  r4:ef096000

[66515.023040] [<c06b9f54>] (schedule) from [<c06bcda8>] (schedule_timeout+0x118/0x1ac)
[66515.023040]  r5:ef7b73c0
[66515.023040]  r4:ef7b73c0

[66515.023071] [<c06bcc90>] (schedule_timeout) from [<c0085f7c>] (rcu_gp_kthread+0x64c/0x99c)
[66515.023071]  r8:c092a100
[66515.023101]  r7:c093a210
[66515.023101]  r6:c093a21c
[66515.023101]  r5:c093a080
[66515.023101]  r4:00000000

[66515.023132] [<c0085930>] (rcu_gp_kthread) from [<c00503c4>] (kthread+0xe4/0xfc)
[66515.023132]  r7:c0085930

[66515.023162] [<c00502e0>] (kthread) from [<c000fb98>] (ret_from_fork+0x14/0x3c)
[66515.023162]  r7:00000000
[66515.023162]  r6:00000000
[66515.023162]  r5:c00502e0
[66515.023162]  r4:ef0407c0

[66515.023193] Task dump for CPU 0:
[66515.023193] swapper/0       R
[66515.023193]  running
[66515.023193]     0     0      0 0x00000000
[66515.023193] Backtrace: 

[66515.023193] Backtrace aborted due to bad frame pointer <c0929f54>
[66515.023193] Task dump for CPU 1:
[66515.023223] swapper/1       R
[66515.023223]  running
[66515.023223]     0     0      1 0x00000000
[66515.023223] Backtrace: 

[66515.023254] [<c0013f80>] (dump_backtrace) from [<c001417c>] (show_stack+0x18/0x1c)
[66515.023254]  r7:c092a518
[66515.023254]  r6:800c0193
[66515.023254]  r5:00000001
[66515.023254]  r4:ef077480

[66515.023254] [<c0014164>] (show_stack) from [<c005aaf8>] (sched_show_task+0xbc/0x110)
[66515.023315] [<c005aa3c>] (sched_show_task) from [<c005cf88>] (dump_cpu_task+0x40/0x48)
[66515.023315]  r5:c093a080
[66515.023315]  r4:00000001

[66515.023315] [<c005cf48>] (dump_cpu_task) from [<c0082754>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[66515.023345]  r5:c093a080
[66515.023345]  r4:00000002

[66515.023345] [<c00826b8>] (rcu_dump_cpu_stacks) from [<c0086bb8>] (rcu_check_callbacks+0x500/0x8e4)
[66515.023345]  r9:c093a080
[66515.023376]  r8:c092a628
[66515.023376]  r7:2ee95000
[66515.023376]  r6:c0926340
[66515.023376]  r5:c093a080
[66515.023376]  r4:ef7bb340
[66515.023406] [<c00866b8>] (rcu_check_callbacks) from [<c0089de8>] (update_process_times+0x40/0x6c)
[66515.023406]  r10:c009b09c
[66515.023406]  r9:00000001
[66515.023406]  r8:ef7b7c4c
[66515.023406]  r7:00003c7c
[66515.023437]  r6:6ebe2431
[66515.023437]  r5:00000000
[66515.023437]  r4:ef077480

[66515.023468] [<c0089da8>] (update_process_times) from [<c009b098>] (tick_sched_handle+0x50/0x54)
[66515.023468]  r5:ef09ff58
[66515.023468]  r4:ef7b7e40

[66515.023468] [<c009b048>] (tick_sched_handle) from [<c009b0ec>] (tick_sched_timer+0x50/0x94)
[66515.023498] [<c009b09c>] (tick_sched_timer) from [<c008a82c>] (__hrtimer_run_queues+0x110/0x1a4)
[66515.023498]  r7:00000000
[66515.023498]  r6:ef7b7c40
[66515.023498]  r5:ef7b7e40
[66515.023498]  r4:ef7b7c00

[66515.023529] [<c008a71c>] (__hrtimer_run_queues) from [<c008aec0>] (hrtimer_interrupt+0xac/0x1fc)
[66515.023529]  r10:ef7b7c78
[66515.023529]  r9:ef7b7c98
[66515.023529]  r8:ef7b7c14
[66515.023559]  r7:ef7b7cb8
[66515.023559]  r6:ffffffff
[66515.023559]  r5:00000003
[66515.023559]  r4:ef7b7c00

[66515.023590] [<c008ae14>] (hrtimer_interrupt) from [<c001720c>] (twd_handler+0x38/0x48)
[66515.023590]  r10:c092a4ec
[66515.023590]  r9:fa241100
[66515.023590]  r8:00000011
[66515.023590]  r7:ef0287c0
[66515.023590]  r6:c092a7a8
[66515.023590]  r5:ef005cc0
[66515.023620]  r4:00000001

[66515.023620] [<c00171d4>] (twd_handler) from [<c007cebc>] (handle_percpu_devid_irq+0x74/0x8c)
[66515.023651]  r5:ef005cc0
[66515.023651]  r4:ef7bd780

[66515.023651] [<c007ce48>] (handle_percpu_devid_irq) from [<c0078944>] (generic_handle_irq+0x2c/0x3c)
[66515.023681]  r9:fa241100
[66515.023681]  r8:ef008000
[66515.023681]  r7:00000001
[66515.023681]  r6:00000000
[66515.023681]  r5:00000000
[66515.023681]  r4:c092341c
[66515.023712] [<c0078918>] (generic_handle_irq) from [<c0078c3c>] (__handle_domain_irq+0x64/0xbc)
[66515.023712] [<c0078bd8>] (__handle_domain_irq) from [<c00094d4>] (gic_handle_irq+0x50/0x90)
[66515.023712]  r9:fa241100
[66515.023712]  r8:fa240100
[66515.023742]  r7:ef09ff58
[66515.023742]  r6:fa24010c
[66515.023742]  r5:c092a7a8
[66515.023742]  r4:c0947f48
[66515.023742] [<c0009484>] (gic_handle_irq) from [<c0014cd4>] (__irq_svc+0x54/0x90)
[66515.023773] Exception stack(0xef09ff58 to 0xef09ffa0)
[66515.023773] ff40:                                                       00000000 ef7b7388
[66515.023773] ff60: fe600000 00000000 ef09e000 c092a49c c0922364 ef09ffc8 c0967025 c0967025
[66515.023803] ff80: c092a4ec ef09ffb4 ef09ff94 ef09ffa8 c00289d4 c0010ca0 600c0113 ffffffff
[66515.023803]  r9:c0967025
[66515.023803]  r8:c0967025
[66515.023803]  r7:ef09ff8c
[66515.023803]  r6:ffffffff
[66515.023803]  r5:600c0113
[66515.023803]  r4:c0010ca0
[66515.023834] [<c0010c78>] (arch_cpu_idle) from [<c006f5b0>] (default_idle_call+0x28/0x34)
[66515.023864] [<c006f588>] (default_idle_call) from [<c006f6d4>] (cpu_startup_entry+0x118/0x188)
[66515.023864] [<c006f5bc>] (cpu_startup_entry) from [<c001627c>] (secondary_start_kernel+0x178/0x184)
[66515.023864]  r7:c096a288
[66515.023864]  r4:c093d140

[66515.023895] [<c0016104>] (secondary_start_kernel) from [<800095ac>] (0x800095ac)
[66515.023895]  r5:00000051
[66515.023895]  r4:af08406a


[66515.477172]  0-...: (1 GPs behind) idle=e19/1/0 softirq=475991/475991 fqs=36 
[66515.485565]   (t=8436 jiffies g=207416 c=207415 q=146)
[66515.491394] Task dump for CPU 0:
[66515.491394] swapper/0       R running      0     0      0 0x00000002
[66515.502075] Backtrace: 
[66515.502075] [<c0013f80>] (dump_backtrace) from [<c001417c>] (show_stack+0x18/0x1c)
[66515.512695]  r7:c092a518 r6:80010193 r5:00000000 r4:c092d180
[66515.512695] [<c0014164>] (show_stack) from [<c005aaf8>] (sched_show_task+0xbc/0x110)
[66515.512695] [<c005aa3c>] (sched_show_task) from [<c005cf88>] (dump_cpu_task+0x40/0x48)
[66515.512695]  r5:c093a080 r4:00000000
[66515.539093] [<c005cf48>] (dump_cpu_task) from [<c0082754>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[66515.539093]  r5:c093a080 r4:00000001
[66515.551635] [<c00826b8>] (rcu_dump_cpu_stacks) from [<c0086bb8>] (rcu_check_callbacks+0x500/0x8e4)
[66515.551635]  r9:c093a080 r8:c092a628 r7:2ee89000 r6:c0926340 r5:c093a080 r4:ef7af340
[66515.551635] [<c00866b8>] (rcu_check_callbacks) from [<c0089de8>] (update_process_times+0x40/0x6c)
[66515.578735]  r10:c009b09c r9:00000001 r8:ef7abc4c r7:00003c7c r6:6ebe2431 r5:00000000
[66515.578735]  r4:c092d180
[66515.589752] [<c0089da8>] (update_process_times) from [<c009b098>] (tick_sched_handle+0x50/0x54)
[66515.589752]  r5:c0929f10 r4:ef7abe40
[66515.602752] [<c009b048>] (tick_sched_handle) from [<c009b0ec>] (tick_sched_timer+0x50/0x94)
[66515.602752] [<c009b09c>] (tick_sched_timer) from [<c008a82c>] (__hrtimer_run_queues+0x110/0x1a4)
[66515.620880]  r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
[66515.620880] [<c008a71c>] (__hrtimer_run_queues) from [<c008aec0>] (hrtimer_interrupt+0xac/0x1fc)
[66515.636199]  r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8 r6:ffffffff r5:00000003
[66515.636199]  r4:ef7abc00
[66515.636199] [<c008ae14>] (hrtimer_interrupt) from [<c001720c>] (twd_handler+0x38/0x48)
[66515.655578]  r10:c092a4ec r9:fa241100 r8:00000011 r7:ef0287c0 r6:c092a7a8 r5:ef005cc0
[66515.655578]  r4:00000001
[66515.655578] [<c00171d4>] (twd_handler) from [<c007cebc>] (handle_percpu_devid_irq+0x74/0x8c)
[66515.666595]  r5:ef005cc0 r4:ef7b1780
[66515.679321] [<c007ce48>] (handle_percpu_devid_irq) from [<c0078944>] (generic_handle_irq+0x2c/0x3c)
[66515.679321]  r9:fa241100 r8:ef008000 r7:00000001 r6:00000000 r5:00000000 r4:c092341c
[66515.697113] [<c0078918>] (generic_handle_irq) from [<c0078c3c>] (__handle_domain_irq+0x64/0xbc)
[66515.706329] [<c0078bd8>] (__handle_domain_irq) from [<c00094d4>] (gic_handle_irq+0x50/0x90)
[66515.706329]  r9:fa241100 r8:fa240100 r7:c0929f10 r6:fa24010c r5:c092a7a8 r4:c0947f48
[66515.723388] [<c0009484>] (gic_handle_irq) from [<c0014cd4>] (__irq_svc+0x54/0x90)
[66515.723388] Exception stack(0xc0929f10 to 0xc0929f58)
[66515.723388] 9f00:                                     00000000 ef7ab388 fe600000 00000000
[66515.745300] 9f20: c0928000 c092a49c c0922364 c0929f80 c0967025 c0967025 c092a4ec c0929f6c
[66515.753967] 9f40: c0929f4c c0929f60 c00289d4 c0010ca0 60010113 ffffffff
[66515.753967]  r9:c0967025 r8:c0967025 r7:c0929f44 r6:ffffffff r5:60010113 r4:c0010ca0
[66515.753967] [<c0010c78>] (arch_cpu_idle) from [<c006f5b0>] (default_idle_call+0x28/0x34)
[66515.753967] [<c006f588>] (default_idle_call) from [<c006f6d4>] (cpu_startup_entry+0x118/0x188)
[66515.786895] [<c006f5bc>] (cpu_startup_entry) from [<c06b937c>] (rest_init+0x90/0x94)
[66515.786895]  r7:ffffffff r4:00000002
[66515.786895] [<c06b92ec>] (rest_init) from [<c08dcd0c>] (start_kernel+0x390/0x39c)
[66515.786895]  r5:c096a000 r4:00000000
[66515.810607] [<c08dc97c>] (start_kernel) from [<8000807c>] (0x8000807c)
$