* Re: Random stack corruption on v5.13 with dra76
[not found] <0f48c7e5-6acd-1143-35ef-3dea2255bec6@ideasonboard.com>
@ 2021-05-21 5:36 ` Tony Lindgren
2021-05-21 7:04 ` Tomi Valkeinen
0 siblings, 1 reply; 8+ messages in thread
From: Tony Lindgren @ 2021-05-21 5:36 UTC (permalink / raw)
To: Tomi Valkeinen
Cc: linux-arm-kernel, Nishanth Menon, Bajjuri, Praneeth, linux-omap
* Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> [210520 08:27]:
> Hi,
>
> I've noticed that the v5.13 rcs crash randomly (but quite often) on dra76 evm
> (I haven't tested other boards). Anyone else seen this problem?
I have not seen this so far and beagle-x15 is behaving for me.
Does it always happen on boot?
Regards,
Tony
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Random stack corruption on v5.13 with dra76
2021-05-21 5:36 ` Random stack corruption on v5.13 with dra76 Tony Lindgren
@ 2021-05-21 7:04 ` Tomi Valkeinen
2021-05-21 7:39 ` Tony Lindgren
0 siblings, 1 reply; 8+ messages in thread
From: Tomi Valkeinen @ 2021-05-21 7:04 UTC (permalink / raw)
To: Tony Lindgren
Cc: linux-arm-kernel, Nishanth Menon, Bajjuri, Praneeth, linux-omap
On 21/05/2021 08:36, Tony Lindgren wrote:
> * Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> [210520 08:27]:
>> Hi,
>>
>> I've noticed that the v5.13 rcs crash randomly (but quite often) on dra76 evm
>> (I haven't tested other boards). Anyone else seen this problem?
>
> I have not seen this so far and beagle-x15 is behaving for me.
>
> Does it always happen on boot?
No, but quite often. I can't really say how often, as it's annoyingly random.
I tried to bisect, but that proved to be difficult as sometimes I get multiple (5+)
successful boots before the crash.
I tested with x15, same issue (below). So... Something in my kernel config? Or compiler?
Looks like the crash happens always very soon after (or during) probing palmas.
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 5.13.0-rc2+ (tomba@deskari) (arm-none-linux-gnueabihf-gcc (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16)) 10.2.1 20201103
, GNU ld (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16)) 2.35.1.20201028) #2 SMP PREEMPT Fri May 21 09:55:16 EEST 2021
[ 0.000000] CPU: ARMv7 Processor [412fc0f2] revision 2 (ARMv7), cr=10c5387d
[ 0.000000] CPU: div instructions available: patching division code
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
[ 0.000000] OF: fdt: Machine model: TI AM5728 BeagleBoard-X15 rev C
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] Reserved memory: created CMA memory pool at 0x95800000, size 56 MiB
[ 0.000000] OF: reserved mem: initialized node ipu2-memory@95800000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created CMA memory pool at 0x99000000, size 64 MiB
[ 0.000000] OF: reserved mem: initialized node dsp1-memory@99000000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created CMA memory pool at 0x9d000000, size 32 MiB
[ 0.000000] OF: reserved mem: initialized node ipu1-memory@9d000000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created CMA memory pool at 0x9f000000, size 8 MiB
[ 0.000000] OF: reserved mem: initialized node dsp2-memory@9f000000, compatible id shared-dma-pool
[ 0.000000] cma: Reserved 32 MiB at 0xfd800000
[ 0.000000] OMAP4: Map 0xafe00000 to (ptrval) for dram barrier
[ 0.000000] Zone ranges:
[ 0.000000] Normal [mem 0x0000000080000000-0x00000000afdfffff]
[ 0.000000] HighMem [mem 0x00000000afe00000-0x00000000ffffefff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000080000000-0x00000000afdfffff]
[ 0.000000] node 0: [mem 0x00000000b0000000-0x00000000ffffefff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000ffffefff]
[ 0.000000] DRA752 ES2.0
[ 0.000000] percpu: Embedded 20 pages/cpu s52780 r8192 d20948 u81920
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 522051
[ 0.000000] Kernel command line: console=ttyS2,115200n8 root=/dev/nfs rw nfsroot=192.168.50.1:/nfs/rootfs,v3,tcp ip=192.168.50.15:::::eth0: no_console_suspend
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] Memory: 1850304K/2095100K available (11264K kernel code, 1189K rwdata, 3468K rodata, 1024K init, 10291K bss, 48188K reserved, 196608K cma-reserved, 1277948K hi
ghmem)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] ftrace: allocating 32491 entries in 64 pages
[ 0.000000] ftrace: allocated 64 pages with 1 groups
[ 0.000000] trace event string verifier disabled
[ 0.000000] Running RCU self tests
[ 0.000000] rcu: Preemptible hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU lockdep checking is enabled.
[ 0.000000] Trampoline variant of Tasks RCU enabled.
[ 0.000000] Rude variant of Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[ 0.000000] GIC: Using split EOI/Deactivate mode
[ 0.000000] random: get_random_bytes called from start_kernel+0x37c/0x5cc with crng_init=0
[ 0.000000] OMAP clocksource: 32k_counter at 32768 Hz
[ 0.000000] clocksource: 32k_counter: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 58327039986419 ns
[ 0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65535999984741ns
[ 0.001861] TI gptimer clockevent: always-on 32786 Hz at /ocp/interconnect@4ae00000/segment@10000/target-module@8000
[ 0.002716] TI gptimer percpu-dmtimer: 20000000 Hz at /ocp/interconnect@48000000/segment@0/target-module@34000
[ 0.003143] TI gptimer percpu-dmtimer: 20000000 Hz at /ocp/interconnect@48000000/segment@0/target-module@36000
[ 0.005676] Console: colour dummy device 80x30
[ 0.005706] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[ 0.005737] ... MAX_LOCKDEP_SUBCLASSES: 8
[ 0.005767] ... MAX_LOCK_DEPTH: 48
[ 0.005798] ... MAX_LOCKDEP_KEYS: 8192
[ 0.005828] ... CLASSHASH_SIZE: 4096
[ 0.005859] ... MAX_LOCKDEP_ENTRIES: 32768
[ 0.005859] ... MAX_LOCKDEP_CHAINS: 65536
[ 0.005889] ... CHAINHASH_SIZE: 32768
[ 0.005920] memory used by lock dependency info: 4061 kB
[ 0.005950] memory used for stack traces: 2112 kB
[ 0.005950] per task-struct memory footprint: 1536 bytes
[ 0.006103] Calibrating delay loop... 2000.48 BogoMIPS (lpj=10002432)
[ 0.052581] pid_max: default: 32768 minimum: 301
[ 0.053131] LSM: Security Framework initializing
[ 0.053466] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[ 0.053497] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[ 0.058013] CPU: Testing write buffer coherency: ok
[ 0.058258] CPU0: Spectre v2: using ICIALLU workaround
[ 0.059387] /cpus/cpu@0 missing clock-frequency property
[ 0.059509] /cpus/cpu@1 missing clock-frequency property
[ 0.059600] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.065643] Running RCU-tasks wait API self tests
[ 0.173889] Setting up static identity map for 0x80100000 - 0x80100078
[ 0.174896] rcu: Hierarchical SRCU implementation.
[ 0.185943] smp: Bringing up secondary CPUs ...
[ 0.245178] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.245178] CPU1: Spectre v2: using ICIALLU workaround
[ 0.246582] smp: Brought up 1 node, 2 CPUs
[ 0.246612] SMP: Total of 2 processors activated (4000.97 BogoMIPS).
[ 0.246643] CPU: All CPU(s) started in HYP mode.
[ 0.246673] CPU: Virtualization extensions available.
[ 0.250701] devtmpfs: initialized
[ 0.287078] Callback from call_rcu_tasks_rude() invoked.
[ 0.773742] VFP support v0.3: implementor 41 architecture 4 part 30 variant f rev 0
[ 0.788330] Callback from call_rcu_tasks() invoked.
[ 0.799255] DMA-API: preallocated 65536 debug entries
[ 0.799316] DMA-API: debugging enabled by kernel config
[ 0.799346] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.799438] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[ 0.804931] pinctrl core: initialized pinctrl subsystem
[ 0.816070] NET: Registered protocol family 16
[ 0.829681] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.838745] audit: initializing netlink subsys (disabled)
[ 0.846557] audit: type=2000 audit(0.830:1): state=initialized audit_enabled=0 res=1
[ 0.846710] cpuidle: using governor menu
[ 1.014648] platform encoder: Fixing up cyclic dependency with connector
[ 1.017974] No ATAGs?
[ 1.018554] hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
[ 1.018615] hw-breakpoint: maximum watchpoint size is 8 bytes.
[ 1.154907] Kprobes globally optimized
[ 1.193634] evm_5v0: supplied by main_12v0
[ 1.199981] iommu: Default domain type: Translated
[ 1.209228] SCSI subsystem initialized
[ 1.211456] usbcore: registered new interface driver usbfs
[ 1.212005] usbcore: registered new interface driver hub
[ 1.212402] usbcore: registered new device driver usb
[ 1.214843] pps_core: LinuxPPS API ver. 1 registered
[ 1.214874] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 1.215118] PTP clock support registered
[ 1.217590] Advanced Linux Sound Architecture Driver Initialized.
[ 1.225219] clocksource: Switched to clocksource 32k_counter
[ 1.903778] VFS: Disk quotas dquot_6.6.0
[ 1.904327] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[ 1.991577] NET: Registered protocol family 2
[ 1.992156] IP idents hash table entries: 16384 (order: 5, 131072 bytes, linear)
[ 1.996887] tcp_listen_portaddr_hash hash table entries: 512 (order: 2, 22528 bytes, linear)
[ 1.997100] TCP established hash table entries: 8192 (order: 3, 32768 bytes, linear)
[ 1.997344] TCP bind hash table entries: 8192 (order: 6, 327680 bytes, linear)
[ 1.998962] TCP: Hash tables configured (established 8192 bind 8192)
[ 1.999603] UDP hash table entries: 512 (order: 3, 49152 bytes, linear)
[ 1.999908] UDP-Lite hash table entries: 512 (order: 3, 49152 bytes, linear)
[ 2.001220] NET: Registered protocol family 1
[ 2.008972] RPC: Registered named UNIX socket transport module.
[ 2.009033] RPC: Registered udp transport module.
[ 2.009094] RPC: Registered tcp transport module.
[ 2.009124] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 2.011566] armv7-pmu pmu: hw perfevents: no interrupt-affinity property, guessing.
[ 2.013305] hw perfevents: enabled with armv7_cortex_a15 PMU driver, 7 counters available
[ 2.025726] Initialise system trusted keyrings
[ 2.026977] workingset: timestamp_bits=14 max_order=19 bucket_order=5
[ 2.145477] NFS: Registering the id_resolver key type
[ 2.145690] Key type id_resolver registered
[ 2.145751] Key type id_legacy registered
[ 2.146118] jffs2: version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.
[ 2.152099] Key type asymmetric registered
[ 2.152160] Asymmetric key parser 'x509' registered
[ 2.152709] bounce: pool size: 64 pages
[ 2.153076] io scheduler mq-deadline registered
[ 2.153137] io scheduler kyber registered
[ 2.167144] usbcore: registered new interface driver udlfb
[ 2.196990] Serial: 8250/16550 driver, 10 ports, IRQ sharing enabled
[ 2.331634] brd: module loaded
[ 2.424804] loop: module loaded
[ 2.433593] mtdoops: mtd device (mtddev=name/number) must be supplied
[ 2.446166] libphy: Fixed MDIO Bus: probed
[ 2.455200] random: fast init done
[ 2.457092] usbcore: registered new interface driver smsc95xx
[ 2.458343] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 2.458496] ehci-omap: OMAP-EHCI Host Controller driver
[ 2.459106] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 2.462524] i2c /dev entries driver
[ 2.468597] ledtrig-cpu: registered to indicate activity on CPUs
[ 2.481079] Initializing XFRM netlink socket
[ 2.481292] NET: Registered protocol family 17
[ 2.481475] NET: Registered protocol family 15
[ 2.481994] Key type dns_resolver registered
[ 2.482055] ThumbEE CPU extension supported.
[ 2.482147] Registering SWP/SWPB emulation handler
[ 2.482604] omap_voltage_late_init: Voltage driver support not added
[ 2.482666] sr_init_by_name: Unable to get voltage domain pointer for VDD mpu
[ 2.482696] sr_init_by_name: Unable to get voltage domain pointer for VDD core
[ 2.482757] Power Management for TI OMAP4+ devices.
[ 2.483612] SmartReflex Class3 initialized
[ 2.489929] Loading compiled-in X.509 certificates
[ 2.490997] kmemleak: Kernel memory leak detector initialized (mem pool available: 15866)
[ 2.495483] kmemleak: Automatic memory scanning thread started
[ 3.121429] pinctrl-single 4a003400.pinmux: 282 pins, size 1128
[ 3.350891] omap-dma-engine 4a056000.dma-controller: OMAP DMA engine driver (LinkedList1/2/3 supported)
[ 3.439331] l3init-clkctrl:0068:0: failed to disable
[ 3.504760] OMAP GPIO hardware version 0.1
[ 3.552490] wkupaon-clkctrl:0068:0: failed to disable
[ 3.593139] printk: console [ttyS2] disabled
[ 3.594207] 48020000.serial: ttyS2 at MMIO 0x48020000 (irq = 105, base_baud = 3000000) is a 8250
[ 4.670196] printk: console [ttyS2] enabled
[ 4.822723] omap_i2c 48060000.i2c: bus 2 rev0.12 at 400 kHz
[ 4.888458] palmas 0-0058: Irq flag is 0x00000008
[ 4.938812] palmas 0-0058: Muxing GPIO 2f, PWM 0, LED 0
[ 4.963867] smps12: supplied by regulator-dummy
[ 4.977294] smps3: supplied by regulator-dummy
[ 4.991668] smps45: supplied by regulator-dummy
[ 5.006256] smps6: supplied by regulator-dummy
[ 5.015319] Kernel panic - not syncing: corrupted stack end detected inside scheduler
[ 5.023284] CPU: 1 PID: 7 Comm: kworker/u4:0 Not tainted 5.13.0-rc2+ #2
[ 5.029937] Hardware name: Generic DRA74X (Flattened Device Tree)
[ 5.036102] Workqueue: events_unbound deferred_probe_work_func
[ 5.041992] Backtrace:
[ 5.044494] [<c0b99dc4>] (dump_backtrace) from [<c0b9a188>] (show_stack+0x20/0x24)
[ 5.052154] r7:00000080 r6:60000193 r5:00000000 r4:c11c18b8
[ 5.057861] [<c0b9a168>] (show_stack) from [<c0ba1024>] (dump_stack+0xd8/0x100)
[ 5.065246] [<c0ba0f4c>] (dump_stack) from [<c0b9aab0>] (panic+0x12c/0x368)
[ 5.072265] r7:00000001 r6:c0e4a92c r5:00000000 r4:c122b298
[ 5.077972] [<c0b9a984>] (panic) from [<c0babb70>] (__schedule+0xe78/0xf30)
[ 5.085021] r3:57ac6e9d r2:00000000 r1:c0e59e48 r0:c0e4a92c
[ 5.090728] r7:00000001
[ 5.093292] [<c0baacf8>] (__schedule) from [<c0babe34>] (preempt_schedule_common+0x3c/0xac)
[ 5.101715] r10:fa0700a4 r9:fa070000 r8:c0baca44 r7:00000000 r6:c0babed8 r5:c0babed8
[ 5.109619] r4:c2310000
[ 5.112182] [<c0babdf8>] (preempt_schedule_common) from [<c0babed8>] (preempt_schedule+0x34/0x38)
[ 5.121124] r7:00000000 r6:c23107a0 r5:eeb269c0 r4:40000113
[ 5.126831] [<c0babea4>] (preempt_schedule) from [<c0bb41b4>] (_raw_spin_unlock_irqrestore+0x6c/0x78)
[ 5.136138] [<c0bb4148>] (_raw_spin_unlock_irqrestore) from [<c0205a4c>] (del_timer_sync+0xe0/0x180)
[ 5.145355] r5:00000001 r4:eeb269c0
[ 5.148956] [<c020596c>] (del_timer_sync) from [<c0bb3308>] (schedule_timeout+0xb4/0x138)
[ 5.157226] r7:c1105d00 r6:c2310000 r5:c2310790 r4:ffff8d26
[ 5.162933] [<c0bb3254>] (schedule_timeout) from [<c0baca74>] (wait_for_completion_timeout+0xb8/0x138)
[ 5.172302] r7:c42b7050 r6:c42b7054 r5:c2310000 r4:00000064
[ 5.178009] [<c0bac9bc>] (wait_for_completion_timeout) from [<c0922bc8>] (omap_i2c_xfer_common+0x418/0x64c)
[ 5.187835] r8:00000001 r7:00000000 r6:c2310918 r5:00000001 r4:c42b7040
[ 5.194610] [<c09227b0>] (omap_i2c_xfer_common) from [<c0922e38>] (omap_i2c_xfer_irq+0x1c/0x20)
[ 5.203369] r10:c11091c0 r9:c1105d00 r8:c231090c r7:00000000 r6:00000002 r5:ffff8cc2
[ 5.211273] r4:c42b70a8
[ 5.213836] [<c0922e1c>] (omap_i2c_xfer_irq) from [<c091a73c>] (__i2c_transfer+0x184/0xb3c)
[ 5.222259] [<c091a5b8>] (__i2c_transfer) from [<c091b19c>] (i2c_transfer+0xa8/0x144)
[ 5.230163] r10:c45da8e8 r9:c120b463 r8:c1109210 r7:c1210fa0 r6:c231090c r5:00000002
[ 5.238037] r4:c42b70a8
[ 5.240600] [<c091b0f4>] (i2c_transfer) from [<c07c6998>] (regmap_i2c_read+0x6c/0xa8)
[ 5.248504] r6:c07c692c r5:00000000 r4:00000000
[ 5.253173] [<c07c692c>] (regmap_i2c_read) from [<c07c1324>] (_regmap_raw_read+0x138/0x4e0)
[ 5.261596] r6:c07c692c r5:00000001 r4:c5081800
[ 5.266235] [<c07c11ec>] (_regmap_raw_read) from [<c07c1720>] (_regmap_bus_read+0x54/0x80)
[ 5.274597] r10:c45da8e8 r9:00000003 r8:c4777270 r7:c2310a10 r6:c5063f41 r5:00000033
[ 5.282470] r4:c5081800
[ 5.285034] [<c07c16cc>] (_regmap_bus_read) from [<c07bfbf0>] (_regmap_read+0x74/0x284)
[ 5.293121] r7:c2310a10 r6:c5081800 r5:00000033 r4:c5081800
[ 5.298828] [<c07bfb7c>] (_regmap_read) from [<c07bfe4c>] (regmap_read+0x4c/0x70)
[ 5.306396] r10:c45da8e8 r9:00000003 r8:c4777270 r7:c45da800 r6:c2310a10 r5:00000033
[ 5.314270] r4:c5081800
[ 5.316833] [<c07bfe00>] (regmap_read) from [<c073ee30>] (regulator_get_voltage_sel_regmap+0x40/0x94)
[ 5.326141] r7:c45da800 r6:c0c54298 r5:00000124 r4:c45da800
[ 5.331848] [<c073edf0>] (regulator_get_voltage_sel_regmap) from [<c0732e44>] (regulator_attr_is_visible+0x18c/0x278)
[ 5.342559] r4:c45da8e8
[ 5.345123] [<c0732cb8>] (regulator_attr_is_visible) from [<c044c950>] (internal_create_group+0xe4/0x3e0)
[ 5.354766] r7:00000000 r6:c0c4dd98 r5:c11c9c00 r4:00000124
[ 5.360473] [<c044c86c>] (internal_create_group) from [<c044d27c>] (internal_create_groups.part.0+0x54/0x98)
[ 5.370391] r10:00000000 r9:00000000 r8:c11c9bec r7:c45da8e8 r6:c11c9bec r5:00000000
[ 5.378265] r4:c45da8e8
[ 5.380828] [<c044d228>] (internal_create_groups.part.0) from [<c044d2e4>] (sysfs_create_groups+0x24/0x30)
[ 5.390563] r9:c11c9ac8 r8:c1c2b3d8 r7:c1219db0 r6:c4761410 r5:00000000 r4:c45da8e8
[ 5.398376] [<c044d2c0>] (sysfs_create_groups) from [<c0795720>] (device_add+0x280/0x944)
[ 5.406616] [<c07954a0>] (device_add) from [<c073d5cc>] (regulator_register+0x694/0xc48)
[ 5.414794] r10:c3f13a74 r9:00000000 r8:00000001 r7:00000000 r6:c45da800 r5:c2310c10
[ 5.422668] r4:00000000
[ 5.425231] [<c073cf38>] (regulator_register) from [<c073fbd8>] (devm_regulator_register+0x48/0x84)
[ 5.434356] r10:c3f13a74 r9:c11ca7ac r8:c3f12040 r7:c2310c10 r6:c4761410 r5:c3f12600
[ 5.442260] r4:c4766cc0
[ 5.444824] [<c073fb90>] (devm_regulator_register) from [<c0743bb4>] (palmas_smps_registration+0x1b4/0x480)
[ 5.454650] r7:c11ca9bc r6:c11ca65c r5:00000006 r4:c3f12600
[ 5.460357] [<c0743a00>] (palmas_smps_registration) from [<c07441c8>] (palmas_regulators_probe+0x348/0x450)
[ 5.470184] r10:c11ca7ac r9:00000082 r8:c11ca944 r7:c473b040 r6:c3f12040 r5:c0743a00
[ 5.478057] r4:c474de80
[ 5.480621] [<c0743e80>] (palmas_regulators_probe) from [<c079ca2c>] (platform_probe+0x68/0xc8)
[ 5.489410] r10:c121a050 r9:c11ca5a8 r8:00000000 r7:c1c2b438 r6:c11ca5a8 r5:c4761410
[ 5.497283] r4:00000000
[ 5.499847] [<c079c9c4>] (platform_probe) from [<c079994c>] (really_probe+0x114/0x588)
[ 5.507843] r7:c1c2b438 r6:00000000 r5:c1c2b434 r4:c4761410
[ 5.513549] [<c0799838>] (really_probe) from [<c0799e48>] (driver_probe_device+0x88/0x1f0)
[ 5.521911] r10:eeb75228 r9:eeb7522c r8:c1c2b3d8 r7:c4761410 r6:c2310d7c r5:c11ca5a8
[ 5.529785] r4:c4761410
[ 5.532348] [<c0799dc0>] (driver_probe_device) from [<c079a5ac>] (__device_attach_driver+0xac/0x118)
[ 5.541564] r9:eeb7522c r8:c1c2b3d8 r7:c4761410 r6:c2310d7c r5:c11ca5a8 r4:00000001
[ 5.549377] [<c079a500>] (__device_attach_driver) from [<c079711c>] (bus_for_each_drv+0x90/0xe0)
[ 5.558227] r7:00000001 r6:c079a500 r5:c2310d7c r4:00000000
[ 5.563934] [<c079708c>] (bus_for_each_drv) from [<c079a1b0>] (__device_attach+0xe0/0x1ec)
[ 5.572265] r6:c47614a0 r5:c11d1b20 r4:c4761410
[ 5.576934] [<c079a0d0>] (__device_attach) from [<c079a634>] (device_initial_probe+0x1c/0x20)
[ 5.585540] r7:c1219db0 r6:c4761410 r5:c11d1b20 r4:c4761410
[ 5.591247] [<c079a618>] (device_initial_probe) from [<c0798694>] (bus_probe_device+0x98/0xa0)
[ 5.599914] [<c07985fc>] (bus_probe_device) from [<c079581c>] (device_add+0x37c/0x944)
[ 5.607910] r7:c1219db0 r6:c5080820 r5:00000000 r4:c4761410
[ 5.613616] [<c07954a0>] (device_add) from [<c0951cec>] (of_device_add+0x44/0x4c)
[ 5.621185] r10:c5080820 r9:00000001 r8:eeb75270 r7:00000000 r6:00000000 r5:c5080820
[ 5.629058] r4:c4761400
[ 5.631622] [<c0951ca8>] (of_device_add) from [<c09523a8>] (of_platform_device_create_pdata+0xa0/0xcc)
[ 5.641021] [<c0952308>] (of_platform_device_create_pdata) from [<c09525b4>] (of_platform_bus_create+0x1bc/0x34c)
[ 5.651367] r9:00000001 r8:00000000 r7:00000000 r6:00000000 r5:eeb7520c r4:00000000
[ 5.659149] [<c09523f8>] (of_platform_bus_create) from [<c0952914>] (of_platform_populate+0x90/0x124)
[ 5.668457] r10:c5080820 r9:00000001 r8:c5080820 r7:00000000 r6:00000000 r5:eeb74f2c
[ 5.676361] r4:eeb7520c
[ 5.678924] [<c0952884>] (of_platform_populate) from [<c0952a24>] (devm_of_platform_populate+0x4c/0x90)
[ 5.688385] r9:c0c687c0 r8:00000002 r7:c506c0c0 r6:00000000 r5:c474eec0 r4:c5080820
[ 5.696197] [<c09529d8>] (devm_of_platform_populate) from [<c07dc380>] (palmas_i2c_probe+0x478/0x608)
[ 5.705505] r7:c506c0c0 r6:00000000 r5:00000000 r4:c5004440
[ 5.711212] [<c07dbf08>] (palmas_i2c_probe) from [<c091a56c>] (i2c_device_probe+0x2e0/0x32c)
[ 5.719726] r10:c121a050 r9:c11d4ab0 r8:c07dbf08 r7:c5080804 r6:c5080800 r5:00000000
[ 5.727600] r4:c5080820
[ 5.730194] [<c091a28c>] (i2c_device_probe) from [<c079994c>] (really_probe+0x114/0x588)
[ 5.738342] r9:c11d4ab0 r8:00000000 r7:c1c2b438 r6:00000000 r5:c1c2b434 r4:c5080820
[ 5.746154] [<c0799838>] (really_probe) from [<c0799e48>] (driver_probe_device+0x88/0x1f0)
[ 5.754486] r10:eeb74f48 r9:eeb74f4c r8:c1c2b3d8 r7:c5080820 r6:c2311074 r5:c11d4ab0
[ 5.762359] r4:c5080820
[ 5.764953] [<c0799dc0>] (driver_probe_device) from [<c079a5ac>] (__device_attach_driver+0xac/0x118)
[ 5.774139] r9:eeb74f4c r8:c1c2b3d8 r7:c5080820 r6:c2311074 r5:c11d4ab0 r4:00000001
[ 5.781951] [<c079a500>] (__device_attach_driver) from [<c079711c>] (bus_for_each_drv+0x90/0xe0)
[ 5.790802] r7:00000001 r6:c079a500 r5:c2311074 r4:00000000
[ 5.796508] [<c079708c>] (bus_for_each_drv) from [<c079a1b0>] (__device_attach+0xe0/0x1ec)
[ 5.804870] r6:c50808b0 r5:c11e6880 r4:c5080820
[ 5.809509] [<c079a0d0>] (__device_attach) from [<c079a634>] (device_initial_probe+0x1c/0x20)
[ 5.818115] r7:c1219db0 r6:c5080820 r5:c11e6880 r4:c5080820
[ 5.823822] [<c079a618>] (device_initial_probe) from [<c0798694>] (bus_probe_device+0x98/0xa0)
[ 5.832519] [<c07985fc>] (bus_probe_device) from [<c079581c>] (device_add+0x37c/0x944)
[ 5.840484] r7:c1219db0 r6:c42b7150 r5:00000000 r4:c5080820
[ 5.846191] [<c07954a0>] (device_add) from [<c0795e08>] (device_register+0x24/0x28)
[ 5.853942] r10:c0ee18a0 r9:c5080820 r8:00000000 r7:c5080804 r6:c42b70a8 r5:c231117c
[ 5.861816] r4:c5080820
[ 5.864379] [<c0795de4>] (device_register) from [<c091b4e4>] (i2c_new_client_device+0x140/0x258)
[ 5.873229] r5:c231117c r4:c5080800
[ 5.876861] [<c091b3a4>] (i2c_new_client_device) from [<c091f71c>] (of_i2c_register_devices+0xdc/0x188)
[ 5.886322] r9:eeb74f2c r8:c1220128 r7:c42b7150 r6:c42b70a8 r5:eeb74cd4 r4:eeb74f90
[ 5.894134] [<c091f640>] (of_i2c_register_devices) from [<c091be30>] (i2c_register_adapter+0x21c/0x700)
[ 5.903594] r10:c09184d4 r9:c11e7a74 r8:c1c2eff8 r7:00000000 r6:c42b7150 r5:c11e7a74
[ 5.911499] r4:c42b70a8
[ 5.914062] [<c091bc14>] (i2c_register_adapter) from [<c091c374>] (__i2c_add_numbered_adapter+0x60/0xac)
[ 5.923614] r10:00000009 r9:00000009 r8:00000003 r7:c42b70a8 r6:c505fc10 r5:c42b70a8
[ 5.931518] r4:00000000
[ 5.934082] [<c091c314>] (__i2c_add_numbered_adapter) from [<c091c460>] (i2c_add_adapter+0xa0/0xe4)
[ 5.943206] r5:c42b70a8 r4:c42b7040
[ 5.946807] [<c091c3c0>] (i2c_add_adapter) from [<c091c530>] (i2c_add_numbered_adapter+0x2c/0x30)
[ 5.955749] r5:c505fc00 r4:c42b7040
[ 5.959381] [<c091c504>] (i2c_add_numbered_adapter) from [<c092236c>] (omap_i2c_probe+0x380/0x714)
[ 5.968414] [<c0921fec>] (omap_i2c_probe) from [<c079ca2c>] (platform_probe+0x68/0xc8)
[ 5.976379] r10:c121a050 r9:c11e7a20 r8:00000000 r7:c1c2b438 r6:c11e7a20 r5:c505fc10
[ 5.984283] r4:00000000
[ 5.986846] [<c079c9c4>] (platform_probe) from [<c079994c>] (really_probe+0x114/0x588)
[ 5.994842] r7:c1c2b438 r6:00000000 r5:c1c2b434 r4:c505fc10
[ 6.000549] [<c0799838>] (really_probe) from [<c0799e48>] (driver_probe_device+0x88/0x1f0)
[ 6.008880] r10:eeb74cf0 r9:eeb74cf4 r8:c1c2b3d8 r7:c505fc10 r6:c23113ac r5:c11e7a20
[ 6.016784] r4:c505fc10
[ 6.019348] [<c0799dc0>] (driver_probe_device) from [<c079a5ac>] (__device_attach_driver+0xac/0x118)
[ 6.028564] r9:eeb74cf4 r8:c1c2b3d8 r7:c505fc10 r6:c23113ac r5:c11e7a20 r4:00000001
[ 6.036346] [<c079a500>] (__device_attach_driver) from [<c079711c>] (bus_for_each_drv+0x90/0xe0)
[ 6.045227] r7:00000001 r6:c079a500 r5:c23113ac r4:00000000
[ 6.050933] [<c079708c>] (bus_for_each_drv) from [<c079a1b0>] (__device_attach+0xe0/0x1ec)
[ 6.059265] r6:c505fca0 r5:c11d1b20 r4:c505fc10
[ 6.063934] [<c079a0d0>] (__device_attach) from [<c079a634>] (device_initial_probe+0x1c/0x20)
[ 6.072509] r7:c1219db0 r6:c505fc10 r5:c11d1b20 r4:c505fc10
[ 6.078216] [<c079a618>] (device_initial_probe) from [<c0798694>] (bus_probe_device+0x98/0xa0)
[ 6.086914] [<c07985fc>] (bus_probe_device) from [<c079581c>] (device_add+0x37c/0x944)
[ 6.094909] r7:c1219db0 r6:c505e410 r5:00000000 r4:c505fc10
[ 6.100616] [<c07954a0>] (device_add) from [<c0951cec>] (of_device_add+0x44/0x4c)
[ 6.108154] r10:c1126648 r9:c11267e8 r8:eeb74d38 r7:00000000 r6:00000000 r5:c505e410
[ 6.116058] r4:c505fc00
[ 6.118621] [<c0951ca8>] (of_device_add) from [<c09523a8>] (of_platform_device_create_pdata+0xa0/0xcc)
[ 6.127990] [<c0952308>] (of_platform_device_create_pdata) from [<c09525b4>] (of_platform_bus_create+0x1bc/0x34c)
[ 6.138336] r9:c11267e8 r8:00000000 r7:c1126648 r6:00000000 r5:eeb74cd4 r4:00000000
[ 6.146148] [<c09523f8>] (of_platform_bus_create) from [<c0952914>] (of_platform_populate+0x90/0x124)
[ 6.155456] r10:c0c3d37c r9:00000001 r8:c505e410 r7:c1126648 r6:c0c3d3a4 r5:eeb74974
[ 6.163360] r4:eeb74cd4
[ 6.165924] [<c0952884>] (of_platform_populate) from [<c06d1098>] (sysc_probe+0x10f0/0x1634)
[ 6.174438] r9:00000001 r8:0000021f r7:c1c25298 r6:00000000 r5:c505e410 r4:c505a640
[ 6.182220] [<c06cffa8>] (sysc_probe) from [<c079ca2c>] (platform_probe+0x68/0xc8)
[ 6.189880] r10:c121a050 r9:c11c25ec r8:00000000 r7:c1c2b438 r6:c11c25ec r5:c505e410
[ 6.197753] r4:00000000
[ 6.200317] [<c079c9c4>] (platform_probe) from [<c079994c>] (really_probe+0x114/0x588)
[ 6.208312] r7:c1c2b438 r6:00000000 r5:c1c2b434 r4:c505e410
[ 6.214019] [<c0799838>] (really_probe) from [<c0799e48>] (driver_probe_device+0x88/0x1f0)
[ 6.222381] r10:eeb74990 r9:eeb74994 r8:c1c2b3d8 r7:c505e410 r6:c231168c r5:c11c25ec
[ 6.230255] r4:c505e410
[ 6.232818] [<c0799dc0>] (driver_probe_device) from [<c079a5ac>] (__device_attach_driver+0xac/0x118)
[ 6.242034] r9:eeb74994 r8:c1c2b3d8 r7:c505e410 r6:c231168c r5:c11c25ec r4:00000001
[ 6.249847] [<c079a500>] (__device_attach_driver) from [<c079711c>] (bus_for_each_drv+0x90/0xe0)
[ 6.258697] r7:00000001 r6:c079a500 r5:c231168c r4:00000000
[ 6.264404] [<c079708c>] (bus_for_each_drv) from [<c079a1b0>] (__device_attach+0xe0/0x1ec)
[ 6.272735] r6:c505e4a0 r5:c11d1b20 r4:c505e410
[ 6.277404] [<c079a0d0>] (__device_attach) from [<c079a634>] (device_initial_probe+0x1c/0x20)
[ 6.286010] r7:c1219db0 r6:c505e410 r5:c11d1b20 r4:c505e410
[ 6.291717] [<c079a618>] (device_initial_probe) from [<c0798694>] (bus_probe_device+0x98/0xa0)
[ 6.300384] [<c07985fc>] (bus_probe_device) from [<c079581c>] (device_add+0x37c/0x944)
[ 6.308380] r7:c1219db0 r6:c47a7010 r5:00000000 r4:c505e410
[ 6.314086] [<c07954a0>] (device_add) from [<c0951cec>] (of_device_add+0x44/0x4c)
[ 6.321655] r10:c11267a8 r9:c11267e8 r8:eeb749d8 r7:c1126804 r6:00000000 r5:c47a7010
[ 6.329528] r4:c505e400
[ 6.332092] [<c0951ca8>] (of_device_add) from [<c09523a8>] (of_platform_device_create_pdata+0xa0/0xcc)
[ 6.341491] [<c0952308>] (of_platform_device_create_pdata) from [<c09525b4>] (of_platform_bus_create+0x1bc/0x34c)
[ 6.351837] r9:c11267e8 r8:c1126804 r7:c1126648 r6:00000000 r5:eeb74974 r4:00000000
[ 6.359649] [<c09523f8>] (of_platform_bus_create) from [<c0952914>] (of_platform_populate+0x90/0x124)
[ 6.368927] r10:c121a050 r9:00000001 r8:c47a7010 r7:c1126648 r6:00000000 r5:eeb6dcf8
[ 6.376831] r4:eeb74974
[ 6.379394] [<c0952884>] (of_platform_populate) from [<c06cdd0c>] (simple_pm_bus_probe+0x54/0x80)
[ 6.388366] r9:c11c2510 r8:00000000 r7:c1c2b438 r6:c1126648 r5:eeb6dcf8 r4:c47a7010
[ 6.396148] [<c06cdcb8>] (simple_pm_bus_probe) from [<c079ca2c>] (platform_probe+0x68/0xc8)
[ 6.404571] r7:c1c2b438 r6:c11c2510 r5:c47a7010 r4:00000000
[ 6.410278] [<c079c9c4>] (platform_probe) from [<c079994c>] (really_probe+0x114/0x588)
[ 6.418273] r7:c1c2b438 r6:00000000 r5:c1c2b434 r4:c47a7010
[ 6.423980] [<c0799838>] (really_probe) from [<c0799e48>] (driver_probe_device+0x88/0x1f0)
[ 6.432312] r10:eeb6dd14 r9:eeb6dd18 r8:c1c2b3d8 r7:c47a7010 r6:c2311924 r5:c11c2510
[ 6.440216] r4:c47a7010
[ 6.442779] [<c0799dc0>] (driver_probe_device) from [<c079a5ac>] (__device_attach_driver+0xac/0x118)
[ 6.451995] r9:eeb6dd18 r8:c1c2b3d8 r7:c47a7010 r6:c2311924 r5:c11c2510 r4:00000001
[ 6.459777] [<c079a500>] (__device_attach_driver) from [<c079711c>] (bus_for_each_drv+0x90/0xe0)
[ 6.468658] r7:00000001 r6:c079a500 r5:c2311924 r4:00000000
[ 6.474365] [<c079708c>] (bus_for_each_drv) from [<c079a1b0>] (__device_attach+0xe0/0x1ec)
[ 6.482696] r6:c47a70a0 r5:c11d1b20 r4:c47a7010
[ 6.487365] [<c079a0d0>] (__device_attach) from [<c079a634>] (device_initial_probe+0x1c/0x20)
[ 6.495971] r7:c1219db0 r6:c47a7010 r5:c11d1b20 r4:c47a7010
[ 6.501678] [<c079a618>] (device_initial_probe) from [<c0798694>] (bus_probe_device+0x98/0xa0)
[ 6.510345] [<c07985fc>] (bus_probe_device) from [<c079581c>] (device_add+0x37c/0x944)
[ 6.518341] r7:c1219db0 r6:c47a5410 r5:00000000 r4:c47a7010
[ 6.524047] [<c07954a0>] (device_add) from [<c0951cec>] (of_device_add+0x44/0x4c)
[ 6.531585] r10:c1126648 r9:c1126798 r8:eeb6dd5c r7:c1126648 r6:00000000 r5:c47a5410
[ 6.539489] r4:c47a7000
[ 6.542053] [<c0951ca8>] (of_device_add) from [<c09523a8>] (of_platform_device_create_pdata+0xa0/0xcc)
[ 6.551452] [<c0952308>] (of_platform_device_create_pdata) from [<c09525b4>] (of_platform_bus_create+0x1bc/0x34c)
[ 6.561798] r9:c1126798 r8:c1126648 r7:c1126648 r6:00000000 r5:eeb6dcf8 r4:00000000
[ 6.569580] [<c09523f8>] (of_platform_bus_create) from [<c0952914>] (of_platform_populate+0x90/0x124)
[ 6.578887] r10:c121a050 r9:00000001 r8:c47a5410 r7:c1126648 r6:00000000 r5:eeb6da0c
[ 6.586791] r4:eeb6dcf8
[ 6.589355] [<c0952884>] (of_platform_populate) from [<c06cdd0c>] (simple_pm_bus_probe+0x54/0x80)
[ 6.598297] r9:c11c2510 r8:00000000 r7:c1c2b438 r6:c1126648 r5:eeb6da0c r4:c47a5410
[ 6.606109] [<c06cdcb8>] (simple_pm_bus_probe) from [<c079ca2c>] (platform_probe+0x68/0xc8)
[ 6.614532] r7:c1c2b438 r6:c11c2510 r5:c47a5410 r4:00000000
[ 6.620239] [<c079c9c4>] (platform_probe) from [<c079994c>] (really_probe+0x114/0x588)
[ 6.628234] r7:c1c2b438 r6:00000000 r5:c1c2b434 r4:c47a5410
[ 6.633941] [<c0799838>] (really_probe) from [<c0799e48>] (driver_probe_device+0x88/0x1f0)
[ 6.642272] r10:eeb6da28 r9:eeb6da2c r8:c1c2b3d8 r7:c47a5410 r6:c2311bbc r5:c11c2510
[ 6.650177] r4:c47a5410
[ 6.652740] [<c0799dc0>] (driver_probe_device) from [<c079a5ac>] (__device_attach_driver+0xac/0x118)
[ 6.661956] r9:eeb6da2c r8:c1c2b3d8 r7:c47a5410 r6:c2311bbc r5:c11c2510 r4:00000001
[ 6.669738] [<c079a500>] (__device_attach_driver) from [<c079711c>] (bus_for_each_drv+0x90/0xe0)
[ 6.678619] r7:00000001 r6:c079a500 r5:c2311bbc r4:00000000
[ 6.684326] [<c079708c>] (bus_for_each_drv) from [<c079a1b0>] (__device_attach+0xe0/0x1ec)
[ 6.692657] r6:c47a54a0 r5:c11d1b20 r4:c47a5410
[ 6.697326] [<c079a0d0>] (__device_attach) from [<c079a634>] (device_initial_probe+0x1c/0x20)
[ 6.705932] r7:c1219db0 r6:c47a5410 r5:c11d1b20 r4:c47a5410
[ 6.711639] [<c079a618>] (device_initial_probe) from [<c0798694>] (bus_probe_device+0x98/0xa0)
[ 6.720306] [<c07985fc>] (bus_probe_device) from [<c079581c>] (device_add+0x37c/0x944)
[ 6.728302] r7:c1219db0 r6:c397a010 r5:00000000 r4:c47a5410
[ 6.734008] [<c07954a0>] (device_add) from [<c0951cec>] (of_device_add+0x44/0x4c)
[ 6.741546] r10:c1126798 r9:c11267e8 r8:eeb6da70 r7:c1126648 r6:00000000 r5:c397a010
[ 6.749450] r4:c47a5400
[ 6.752014] [<c0951ca8>] (of_device_add) from [<c09523a8>] (of_platform_device_create_pdata+0xa0/0xcc)
[ 6.761413] [<c0952308>] (of_platform_device_create_pdata) from [<c09525b4>] (of_platform_bus_create+0x1bc/0x34c)
[ 6.771759] r9:c11267e8 r8:c1126648 r7:c1126648 r6:00000000 r5:eeb6da0c r4:00000000
[ 6.779541] [<c09523f8>] (of_platform_bus_create) from [<c0952914>] (of_platform_populate+0x90/0x124)
[ 6.788848] r10:c121a050 r9:00000001 r8:c397a010 r7:c1126648 r6:00000000 r5:eeb3a2fc
[ 6.796752] r4:eeb6da0c
[ 6.799316] [<c0952884>] (of_platform_populate) from [<c06cdd0c>] (simple_pm_bus_probe+0x54/0x80)
[ 6.808258] r9:c11c2510 r8:00000000 r7:c1c2b438 r6:c1126648 r5:eeb3a2fc r4:c397a010
[ 6.816070] [<c06cdcb8>] (simple_pm_bus_probe) from [<c079ca2c>] (platform_probe+0x68/0xc8)
[ 6.824493] r7:c1c2b438 r6:c11c2510 r5:c397a010 r4:00000000
[ 6.830200] [<c079c9c4>] (platform_probe) from [<c079994c>] (really_probe+0x114/0x588)
[ 6.838195] r7:c1c2b438 r6:00000000 r5:c1c2b434 r4:c397a010
[ 6.843902] [<c0799838>] (really_probe) from [<c0799e48>] (driver_probe_device+0x88/0x1f0)
[ 6.852233] r10:c0eb2124 r9:c397a010 r8:00000000 r7:c397a010 r6:c2311e54 r5:c11c2510
[ 6.860137] r4:c397a010
[ 6.862701] [<c0799dc0>] (driver_probe_device) from [<c079a5ac>] (__device_attach_driver+0xac/0x118)
[ 6.871887] r9:c397a010 r8:00000000 r7:c397a010 r6:c2311e54 r5:c11c2510 r4:00000001
[ 6.879699] [<c079a500>] (__device_attach_driver) from [<c079711c>] (bus_for_each_drv+0x90/0xe0)
[ 6.888549] r7:00000001 r6:c079a500 r5:c2311e54 r4:00000000
[ 6.894256] [<c079708c>] (bus_for_each_drv) from [<c079a1b0>] (__device_attach+0xe0/0x1ec)
[ 6.902618] r6:c397a0a0 r5:c11d1b20 r4:c397a010
[ 6.907257] [<c079a0d0>] (__device_attach) from [<c079a634>] (device_initial_probe+0x1c/0x20)
[ 6.915863] r7:c121a050 r6:c397a010 r5:c11d1b20 r4:c3941f74
[ 6.921569] [<c079a618>] (device_initial_probe) from [<c0798694>] (bus_probe_device+0x98/0xa0)
[ 6.930267] [<c07985fc>] (bus_probe_device) from [<c0799324>] (deferred_probe_work_func+0xac/0xfc)
[ 6.939300] r7:c121a050 r6:c11d1740 r5:c11d1654 r4:c3941f74
[ 6.945007] [<c0799278>] (deferred_probe_work_func) from [<c01642d4>] (process_one_work+0x270/0x83c)
[ 6.954223] r10:c1109210 r9:c120c3c0 r8:00000000 r7:c2020000 r6:c201c000 r5:c22de100
[ 6.962097] r4:c11d16f0 r3:c0799278
[ 6.965728] [<c0164064>] (process_one_work) from [<c0164a8c>] (worker_thread+0x1ec/0x524)
[ 6.973968] r10:c201c000 r9:c2310000 r8:00000088 r7:c201c038 r6:c22de114 r5:c201c000
[ 6.981872] r4:c22de100
[ 6.984436] [<c01648a0>] (worker_thread) from [<c016c508>] (kthread+0x178/0x1a4)
[ 6.991882] r10:c22f9e34 r9:c22de100 r8:c01648a0 r7:c2310000 r6:00000000 r5:c22bccc0
[ 6.999786] r4:c22de180 r3:00000000
[ 7.003387] [<c016c390>] (kthread) from [<c010013c>] (ret_from_fork+0x14/0x38)
[ 7.010681] Exception stack(0xc2311fb0 to 0xc2311ff8)
[ 7.015777] 1fa0: 00000000 00000000 00000000 00000000
[ 7.024017] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 7.032287] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 7.038940] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c016c390
[ 7.046844] r4:c22bccc0
[ 7.049530] CPU0: stopping
[ 7.052307] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.13.0-rc2+ #2
[ 7.058715] Hardware name: Generic DRA74X (Flattened Device Tree)
[ 7.064849] Backtrace:
[ 7.067352] [<c0b99dc4>] (dump_backtrace) from [<c0b9a188>] (show_stack+0x20/0x24)
[ 7.075012] r7:00000080 r6:60000193 r5:00000000 r4:c11c18b8
[ 7.080688] [<c0b9a168>] (show_stack) from [<c0ba1024>] (dump_stack+0xd8/0x100)
[ 7.088073] [<c0ba0f4c>] (dump_stack) from [<c0110e54>] (do_handle_IPI+0x384/0x3e0)
[ 7.095794] r7:c1109aec r6:c120c1e0 r5:00000000 r4:00000004
[ 7.101501] [<c0110ad0>] (do_handle_IPI) from [<c0110ed4>] (ipi_handler+0x24/0x30)
[ 7.109161] r9:c0110eb0 r8:c2006c80 r7:c1109aec r6:00000014 r5:c1109838 r4:c2188000
[ 7.116943] [<c0110eb0>] (ipi_handler) from [<c01d6880>] (handle_percpu_devid_irq+0xc4/0x1ac)
[ 7.125549] [<c01d67bc>] (handle_percpu_devid_irq) from [<c01cf1f4>] (__handle_domain_irq+0xbc/0x14c)
[ 7.134857] r9:c1100000 r8:c1092cd0 r7:00000000 r6:c1109838 r5:c1092cd0 r4:00000000
[ 7.142639] [<c01cf138>] (__handle_domain_irq) from [<c01016d4>] (gic_handle_irq+0x9c/0xc4)
[ 7.151062] r10:c1101eb8 r9:c1092cf4 r8:c11c1a44 r7:fa21200c r6:c1109aec r5:000003ff
[ 7.158966] r4:fa212000
[ 7.161529] [<c0101638>] (gic_handle_irq) from [<c0100b70>] (__irq_svc+0x70/0xb0)
[ 7.169067] Exception stack(0xc1101eb8 to 0xc1101f00)
[ 7.174163] 1ea0: 00000000 c0eadbd8
[ 7.182403] 1ec0: 00000000 fe600000 c1100000 00000000 c1109210 c1109264 c10932b0 00000000
[ 7.190643] 1ee0: 00000000 c1101f14 c1101ef4 c1101f08 c012a558 c010a0a4 60000113 ffffffff
[ 7.198883] r10:00000000 r9:c1100000 r8:c10932b0 r7:c1101eec r6:ffffffff r5:60000113
[ 7.206787] r4:c010a0a4 r3:c012a558
[ 7.210388] [<c010a078>] (arch_cpu_idle) from [<c0bb3c00>] (default_idle_call+0x58/0xa4)
[ 7.218536] [<c0bb3ba8>] (default_idle_call) from [<c0188638>] (do_idle+0x264/0x330)
[ 7.226348] [<c01883d4>] (do_idle) from [<c0188b18>] (cpu_startup_entry+0x28/0x30)
[ 7.234008] r10:10c5387d r9:c104ba7c r8:efdff840 r7:c11091c0 r6:00000000 r5:c120aeb8
[ 7.241882] r4:000000e1
[ 7.244445] [<c0188af0>] (cpu_startup_entry) from [<c0ba899c>] (rest_init+0x180/0x270)
[ 7.252441] [<c0ba881c>] (rest_init) from [<c1000aa8>] (arch_call_rest_init+0x18/0x1c)
[ 7.260437] r5:c104ba7c r4:c122a068
[ 7.264038] [<c1000a90>] (arch_call_rest_init) from [<c100100c>] (start_kernel+0x4e8/0x5cc)
[ 7.272460] [<c1000b24>] (start_kernel) from [<00000000>] (0x0)
[ 7.278442] r9:412fc0f2 r8:8fefd000 r7:00000000 r6:10c0387d r5:00000051 r4:c1000330
[ 7.286285] ---[ end Kernel panic - not syncing: corrupted stack end detected inside scheduler ]---
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Random stack corruption on v5.13 with dra76
2021-05-21 7:04 ` Tomi Valkeinen
@ 2021-05-21 7:39 ` Tony Lindgren
2021-05-21 8:45 ` Tomi Valkeinen
0 siblings, 1 reply; 8+ messages in thread
From: Tony Lindgren @ 2021-05-21 7:39 UTC (permalink / raw)
To: Tomi Valkeinen
Cc: linux-arm-kernel, Nishanth Menon, Bajjuri, Praneeth, linux-omap
* Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> [210521 07:05]:
> On 21/05/2021 08:36, Tony Lindgren wrote:
> > * Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> [210520 08:27]:
> > > Hi,
> > >
> > > I've noticed that the v5.13 rcs crash randomly (but quite often) on dra76 evm
> > > (I haven't tested other boards). Anyone else seen this problem?
> >
> > I have not seen this so far and beagle-x15 is behaving for me.
> >
> > Does it always happen on boot?
>
> No, but quite often. I can't really say how often, as it's annoyingly random.
> I tried to bisect, but that proved to be difficult as sometimes I get multiple (5+)
> successful boots before the crash.
>
> I tested with x15, same issue (below). So... Something in my kernel config? Or compiler?
> Looks like the crash happens always very soon after (or during) probing palmas.
After about 10 reboots with your .config I'm seeing it now too on
beagle-x15. So far no luck reproducing it with omap2plus_defconfig.
Regards,
Tony
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Random stack corruption on v5.13 with dra76
2021-05-21 7:39 ` Tony Lindgren
@ 2021-05-21 8:45 ` Tomi Valkeinen
2021-05-21 9:14 ` Tony Lindgren
0 siblings, 1 reply; 8+ messages in thread
From: Tomi Valkeinen @ 2021-05-21 8:45 UTC (permalink / raw)
To: Tony Lindgren
Cc: linux-arm-kernel, Nishanth Menon, Bajjuri, Praneeth, linux-omap
On 21/05/2021 10:39, Tony Lindgren wrote:
> * Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> [210521 07:05]:
>> On 21/05/2021 08:36, Tony Lindgren wrote:
>>> * Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> [210520 08:27]:
>>>> Hi,
>>>>
>>>> I've noticed that the v5.13 rcs crash randomly (but quite often) on dra76 evm
>>>> (I haven't tested other boards). Anyone else seen this problem?
>>>
>>> I have not seen this so far and beagle-x15 is behaving for me.
>>>
>>> Does it always happen on boot?
>>
>> No, but quite often. I can't really say how often, as it's annoyingly random.
>> I tried to bisect, but that proved to be difficult as sometimes I get multiple (5+)
>> successful boots before the crash.
>>
>> I tested with x15, same issue (below). So... Something in my kernel config? Or compiler?
>> Looks like the crash happens always very soon after (or during) probing palmas.
>
> After about 10 reboots with your .config I'm seeing it now too on
> beagle-x15. So far no luck reproducing it with omap2plus_defconfig.
I think I have an easy way to see if a kernel is good or bad, by
printing stack_not_used(current) in the first call to
omap_i2c_xfer_irq(). There's a huge drop between v5.12 and v5.13-rc1.
And interestingly, sometimes a simple printk seems to use hundreds of
bytes of stack (i.e. compare stack usage before and after the print).
But not always. So maybe the issue is somehow related to printk.
I'm bisecting.
Tomi
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Random stack corruption on v5.13 with dra76
2021-05-21 8:45 ` Tomi Valkeinen
@ 2021-05-21 9:14 ` Tony Lindgren
2021-05-21 10:30 ` Tomi Valkeinen
0 siblings, 1 reply; 8+ messages in thread
From: Tony Lindgren @ 2021-05-21 9:14 UTC (permalink / raw)
To: Tomi Valkeinen
Cc: linux-arm-kernel, Nishanth Menon, Bajjuri, Praneeth, linux-omap
* Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> [210521 08:45]:
> On 21/05/2021 10:39, Tony Lindgren wrote:
> > * Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> [210521 07:05]:
> > > On 21/05/2021 08:36, Tony Lindgren wrote:
> > > > * Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> [210520 08:27]:
> > > > > Hi,
> > > > >
> > > > > I've noticed that the v5.13 rcs crash randomly (but quite often) on dra76 evm
> > > > > (I haven't tested other boards). Anyone else seen this problem?
> > > >
> > > > I have not seen this so far and beagle-x15 is behaving for me.
> > > >
> > > > Does it always happen on boot?
> > >
> > > No, but quite often. I can't really say how often, as it's annoyingly random.
> > > I tried to bisect, but that proved to be difficult as sometimes I get multiple (5+)
> > > successful boots before the crash.
> > >
> > > I tested with x15, same issue (below). So... Something in my kernel config? Or compiler?
> > > Looks like the crash happens always very soon after (or during) probing palmas.
> >
> > After about 10 reboots with your .config I'm seeing it now too on
> > beagle-x15. So far no luck reproducing it with omap2plus_defconfig.
>
> I think I have an easy way to see if a kernel is good or bad, by printing
> stack_not_used(current) in the first call to omap_i2c_xfer_irq(). There's a
> huge drop between v5.12 and v5.13-rc1.
>
> And interestingly, sometimes a simple printk seems to use hundreds of bytes
> of stack (i.e. compare stack usage before and after the print). But not
> always. So maybe the issue is somehow related to printk.
>
> I'm bisecting.
OK sounds good to me.
Thanks,
Tony
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Random stack corruption on v5.13 with dra76
2021-05-21 9:14 ` Tony Lindgren
@ 2021-05-21 10:30 ` Tomi Valkeinen
2021-05-21 12:57 ` Tomi Valkeinen
0 siblings, 1 reply; 8+ messages in thread
From: Tomi Valkeinen @ 2021-05-21 10:30 UTC (permalink / raw)
To: Tony Lindgren
Cc: linux-arm-kernel, Nishanth Menon, Bajjuri, Praneeth, linux-omap
On 21/05/2021 12:14, Tony Lindgren wrote:
> * Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> [210521 08:45]:
>> On 21/05/2021 10:39, Tony Lindgren wrote:
>>> * Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> [210521 07:05]:
>>>> On 21/05/2021 08:36, Tony Lindgren wrote:
>>>>> * Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> [210520 08:27]:
>>>>>> Hi,
>>>>>>
>>>>>> I've noticed that the v5.13 rcs crash randomly (but quite often) on dra76 evm
>>>>>> (I haven't tested other boards). Anyone else seen this problem?
>>>>>
>>>>> I have not seen this so far and beagle-x15 is behaving for me.
>>>>>
>>>>> Does it always happen on boot?
>>>>
>>>> No, but quite often. I can't really say how often, as it's annoyingly random.
>>>> I tried to bisect, but that proved to be difficult as sometimes I get multiple (5+)
>>>> successful boots before the crash.
>>>>
>>>> I tested with x15, same issue (below). So... Something in my kernel config? Or compiler?
>>>> Looks like the crash happens always very soon after (or during) probing palmas.
>>>
>>> After about 10 reboots with your .config I'm seeing it now too on
>>> beagle-x15. So far no luck reproducing it with omap2plus_defconfig.
>>
>> I think I have an easy way to see if a kernel is good or bad, by printing
>> stack_not_used(current) in the first call to omap_i2c_xfer_irq(). There's a
>> huge drop between v5.12 and v5.13-rc1.
>>
>> And interestingly, sometimes a simple printk seems to use hundreds of bytes
>> of stack (i.e. compare stack usage before and after the print). But not
>> always. So maybe the issue is somehow related to printk.
>>
>> I'm bisecting.
>
> OK sounds good to me.
Well, I found the bad commit but unfortunately it doesn't exactly point
where the issue is.
f483a3e123410bd1c78af295bf65feffb6769a98 is the first bad commit
commit f483a3e123410bd1c78af295bf65feffb6769a98
Author: Tony Lindgren <tony@atomide.com>
Date: Wed Mar 10 14:03:48 2021 +0200
ARM: dts: Configure simple-pm-bus for dra7 l4_per1
We can now probe interconnects with device tree only configuration
using
simple-pm-bus and genpd.
Tested-by: Kishon Vijay Abraham I <kishon@ti.com>
Signed-off-by: Tony Lindgren <tony@atomide.com>
arch/arm/boot/dts/dra7-l4.dtsi | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
The difference is clear, though. With
9a75368b6426739e8b798592f084cb682d760568, which is the last good commit,
when I print the stack usage with stack_not_used() in three different
places in omap_i2c_xfer_irq(), I get always prints roughly like:
STACK FREE omap_i2c_xfer_irq: 2972, 2972, 2972
And these repeat exactly the same for each call to omap_i2c_xfer_irq (at
least during palmas probe).
With the bad commit the situation is different. The first call to
omap_i2c_xfer_irq prints:
STACK FREE omap_i2c_xfer_irq: 2024, 2024, 2024
so we're already using 1k more. But then, instead of the stack usage
staying the same, consecutive calls show increased stack usage. It
doesn't increase for each xfer call, but after about 10 calls, I'm
getting ~1800, ten calls more I see ~800, and going down to ~500.
However, with this bad commit, I don't see the empty stack going below
~500, so I don't get crashes. But going to a more recent commit, like
01d7136894410a71932096e0fb9f1d301b6ccf07, the situation is much worse.
The first print shows:
STACK FREE omap_i2c_xfer_irq: 1164, 1164, 1164
and it quickly goes to stack overflow.
Tomi
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Random stack corruption on v5.13 with dra76
2021-05-21 10:30 ` Tomi Valkeinen
@ 2021-05-21 12:57 ` Tomi Valkeinen
2021-05-21 13:06 ` Tony Lindgren
0 siblings, 1 reply; 8+ messages in thread
From: Tomi Valkeinen @ 2021-05-21 12:57 UTC (permalink / raw)
To: Tony Lindgren
Cc: linux-arm-kernel, Nishanth Menon, Bajjuri, Praneeth, linux-omap,
Laurent Pinchart
On 21/05/2021 13:30, Tomi Valkeinen wrote:
> On 21/05/2021 12:14, Tony Lindgren wrote:
>> * Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> [210521 08:45]:
>>> On 21/05/2021 10:39, Tony Lindgren wrote:
>>>> * Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> [210521 07:05]:
>>>>> On 21/05/2021 08:36, Tony Lindgren wrote:
>>>>>> * Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> [210520 08:27]:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I've noticed that the v5.13 rcs crash randomly (but quite often)
>>>>>>> on dra76 evm
>>>>>>> (I haven't tested other boards). Anyone else seen this problem?
>>>>>>
>>>>>> I have not seen this so far and beagle-x15 is behaving for me.
>>>>>>
>>>>>> Does it always happen on boot?
>>>>>
>>>>> No, but quite often. I can't really say how often, as it's
>>>>> annoyingly random.
>>>>> I tried to bisect, but that proved to be difficult as sometimes I
>>>>> get multiple (5+)
>>>>> successful boots before the crash.
>>>>>
>>>>> I tested with x15, same issue (below). So... Something in my kernel
>>>>> config? Or compiler?
>>>>> Looks like the crash happens always very soon after (or during)
>>>>> probing palmas.
>>>>
>>>> After about 10 reboots with your .config I'm seeing it now too on
>>>> beagle-x15. So far no luck reproducing it with omap2plus_defconfig.
>>>
>>> I think I have an easy way to see if a kernel is good or bad, by
>>> printing
>>> stack_not_used(current) in the first call to omap_i2c_xfer_irq().
>>> There's a
>>> huge drop between v5.12 and v5.13-rc1.
>>>
>>> And interestingly, sometimes a simple printk seems to use hundreds of
>>> bytes
>>> of stack (i.e. compare stack usage before and after the print). But not
>>> always. So maybe the issue is somehow related to printk.
>>>
>>> I'm bisecting.
>>
>> OK sounds good to me.
>
> Well, I found the bad commit but unfortunately it doesn't exactly point
> where the issue is.
>
> f483a3e123410bd1c78af295bf65feffb6769a98 is the first bad commit
> commit f483a3e123410bd1c78af295bf65feffb6769a98
> Author: Tony Lindgren <tony@atomide.com>
> Date: Wed Mar 10 14:03:48 2021 +0200
>
> ARM: dts: Configure simple-pm-bus for dra7 l4_per1
>
> We can now probe interconnects with device tree only configuration
> using
> simple-pm-bus and genpd.
>
> Tested-by: Kishon Vijay Abraham I <kishon@ti.com>
> Signed-off-by: Tony Lindgren <tony@atomide.com>
>
> arch/arm/boot/dts/dra7-l4.dtsi | 9 ++++++---
> 1 file changed, 6 insertions(+), 3 deletions(-)
>
>
> The difference is clear, though. With
> 9a75368b6426739e8b798592f084cb682d760568, which is the last good commit,
> when I print the stack usage with stack_not_used() in three different
> places in omap_i2c_xfer_irq(), I get always prints roughly like:
>
> STACK FREE omap_i2c_xfer_irq: 2972, 2972, 2972
>
> And these repeat exactly the same for each call to omap_i2c_xfer_irq (at
> least during palmas probe).
>
> With the bad commit the situation is different. The first call to
> omap_i2c_xfer_irq prints:
>
> STACK FREE omap_i2c_xfer_irq: 2024, 2024, 2024
>
> so we're already using 1k more. But then, instead of the stack usage
> staying the same, consecutive calls show increased stack usage. It
> doesn't increase for each xfer call, but after about 10 calls, I'm
> getting ~1800, ten calls more I see ~800, and going down to ~500.
>
> However, with this bad commit, I don't see the empty stack going below
> ~500, so I don't get crashes. But going to a more recent commit, like
> 01d7136894410a71932096e0fb9f1d301b6ccf07, the situation is much worse.
> The first print shows:
>
> STACK FREE omap_i2c_xfer_irq: 1164, 1164, 1164
>
> and it quickly goes to stack overflow.
Well... I guess there's no bug as such. The simple-pm-bus just uses a
lot more stack. Looking at the reports from CONFIG_STACK_TRACER, in the
good commit the biggest stack is in USB probing with depth of 4896. With
01d7136894410a71932096e0fb9f1d301b6ccf07, and a debug print in
omap_i2c_xfer_common, I can get to 7544.
I don't know what to do about this, so I'm giving up for the time being
and "fix" it for myself by increasing THREAD_SIZE_ORDER.
# cat /sys/kernel/debug/tracing/stack_trace
Depth Size Location (143 entries)
----- ---- --------
0) 7544 28 ftrace_graph_call+0x0/0xc
1) 7516 28 rcu_read_lock_sched_held+0x38/0x88
2) 7488 120 lock_acquire+0x1ac/0x560
3) 7368 16 _raw_spin_lock_irqsave+0x64/0x80
4) 7352 56 timekeeping_advance+0x3c/0x934
5) 7296 144 update_wall_time+0x1c/0x20
6) 7152 16 tick_do_update_jiffies64+0x1f0/0x304
7) 7136 104 tick_sched_timer+0xf0/0x10c
8) 7032 96 __hrtimer_run_queues+0x2d8/0x68c
9) 6936 112 hrtimer_interrupt+0x158/0x388
10) 6824 16 dmtimer_clockevent_interrupt+0x34/0x3c
11) 6808 64 __handle_irq_event_percpu+0x108/0x384
12) 6744 40 handle_irq_event+0x68/0xe0
13) 6704 32 handle_fasteoi_irq+0xbc/0x224
14) 6672 40 __handle_domain_irq+0x98/0x10c
15) 6632 48 gic_handle_irq+0x9c/0xc4
16) 6584 68 __irq_svc+0x70/0x98
17) 6516 60 console_unlock+0x444/0x688
18) 6456 196 vprintk_emit+0x138/0x344
19) 6260 68 vprintk_default+0x34/0x3c
20) 6192 48 vprintk_func+0xec/0x194
21) 6144 24 printk+0x3c/0x64
22) 6120 104 omap_i2c_xfer_common+0x698/0x6e4
23) 6016 16 omap_i2c_xfer_irq+0x1c/0x20
24) 6000 80 __i2c_transfer+0x184/0xa80
25) 5920 40 i2c_transfer+0xa8/0x144
26) 5880 64 regmap_i2c_read+0x6c/0xa8
27) 5816 72 _regmap_raw_read+0x138/0x474
28) 5744 40 _regmap_bus_read+0x54/0x80
29) 5704 56 _regmap_read+0x74/0x248
30) 5648 32 regmap_read+0x4c/0x70
31) 5616 32 regulator_get_voltage_sel_regmap+0x40/0x94
32) 5584 32 regulator_attr_is_visible+0x18c/0x278
33) 5552 88 internal_create_group+0xe4/0x3e0
34) 5464 40 internal_create_groups.part.0+0x54/0x98
35) 5424 16 sysfs_create_groups+0x24/0x30
36) 5408 96 device_add+0x280/0x944
37) 5312 104 regulator_register+0x694/0xc4c
38) 5208 32 devm_regulator_register+0x48/0x84
39) 5176 72 palmas_smps_registration+0x1b4/0x480
40) 5104 136 palmas_regulators_probe+0x348/0x450
41) 4968 32 platform_probe+0x68/0xc8
42) 4936 64 really_probe+0x114/0x588
43) 4872 48 driver_probe_device+0x88/0x1f0
44) 4824 32 __device_attach_driver+0xac/0x118
45) 4792 48 bus_for_each_drv+0x90/0xe0
46) 4744 48 __device_attach+0xe0/0x1ec
47) 4696 16 device_initial_probe+0x1c/0x20
48) 4680 32 bus_probe_device+0x98/0xa0
49) 4648 96 device_add+0x37c/0x944
50) 4552 16 of_device_add+0x44/0x4c
51) 4536 40 of_platform_device_create_pdata+0xa0/0xcc
52) 4496 112 of_platform_bus_create+0x1bc/0x34c
53) 4384 48 of_platform_populate+0x90/0x124
54) 4336 32 devm_of_platform_populate+0x4c/0x90
55) 4304 88 palmas_i2c_probe+0x478/0x608
56) 4216 40 i2c_device_probe+0x2a0/0x2f8
57) 4176 64 really_probe+0x114/0x588
58) 4112 48 driver_probe_device+0x88/0x1f0
59) 4064 32 __device_attach_driver+0xac/0x118
60) 4032 48 bus_for_each_drv+0x90/0xe0
61) 3984 48 __device_attach+0xe0/0x1ec
62) 3936 16 device_initial_probe+0x1c/0x20
63) 3920 32 bus_probe_device+0x98/0xa0
64) 3888 96 device_add+0x37c/0x944
65) 3792 24 device_register+0x24/0x28
66) 3768 48 i2c_new_client_device+0x140/0x264
67) 3720 112 of_i2c_register_devices+0xdc/0x188
68) 3608 64 i2c_register_adapter+0x220/0x700
69) 3544 32 __i2c_add_numbered_adapter+0x60/0xac
70) 3512 32 i2c_add_adapter+0xa0/0xe4
71) 3480 16 i2c_add_numbered_adapter+0x2c/0x30
72) 3464 80 omap_i2c_probe+0x404/0x6b8
73) 3384 32 platform_probe+0x68/0xc8
74) 3352 64 really_probe+0x114/0x588
75) 3288 48 driver_probe_device+0x88/0x1f0
76) 3240 32 __device_attach_driver+0xac/0x118
77) 3208 48 bus_for_each_drv+0x90/0xe0
78) 3160 48 __device_attach+0xe0/0x1ec
79) 3112 16 device_initial_probe+0x1c/0x20
80) 3096 32 bus_probe_device+0x98/0xa0
81) 3064 96 device_add+0x37c/0x944
82) 2968 16 of_device_add+0x44/0x4c
83) 2952 40 of_platform_device_create_pdata+0xa0/0xcc
84) 2912 112 of_platform_bus_create+0x1bc/0x34c
85) 2800 48 of_platform_populate+0x90/0x124
86) 2752 104 sysc_probe+0x10e4/0x1628
87) 2648 32 platform_probe+0x68/0xc8
88) 2616 64 really_probe+0x114/0x588
89) 2552 48 driver_probe_device+0x88/0x1f0
90) 2504 32 __device_attach_driver+0xac/0x118
91) 2472 48 bus_for_each_drv+0x90/0xe0
92) 2424 48 __device_attach+0xe0/0x1ec
93) 2376 16 device_initial_probe+0x1c/0x20
94) 2360 32 bus_probe_device+0x98/0xa0
95) 2328 96 device_add+0x37c/0x944
96) 2232 16 of_device_add+0x44/0x4c
97) 2216 40 of_platform_device_create_pdata+0xa0/0xcc
98) 2176 112 of_platform_bus_create+0x1bc/0x34c
99) 2064 48 of_platform_populate+0x90/0x124
100) 2016 32 simple_pm_bus_probe+0x54/0x80
101) 1984 32 platform_probe+0x68/0xc8
102) 1952 64 really_probe+0x114/0x588
103) 1888 48 driver_probe_device+0x88/0x1f0
104) 1840 32 __device_attach_driver+0xac/0x118
105) 1808 48 bus_for_each_drv+0x90/0xe0
106) 1760 48 __device_attach+0xe0/0x1ec
107) 1712 16 device_initial_probe+0x1c/0x20
108) 1696 32 bus_probe_device+0x98/0xa0
109) 1664 96 device_add+0x37c/0x944
110) 1568 16 of_device_add+0x44/0x4c
111) 1552 40 of_platform_device_create_pdata+0xa0/0xcc
112) 1512 112 of_platform_bus_create+0x1bc/0x34c
113) 1400 48 of_platform_populate+0x90/0x124
114) 1352 32 simple_pm_bus_probe+0x54/0x80
115) 1320 32 platform_probe+0x68/0xc8
116) 1288 64 really_probe+0x114/0x588
117) 1224 48 driver_probe_device+0x88/0x1f0
118) 1176 32 __device_attach_driver+0xac/0x118
119) 1144 48 bus_for_each_drv+0x90/0xe0
120) 1096 48 __device_attach+0xe0/0x1ec
121) 1048 16 device_initial_probe+0x1c/0x20
122) 1032 32 bus_probe_device+0x98/0xa0
123) 1000 96 device_add+0x37c/0x944
124) 904 16 of_device_add+0x44/0x4c
125) 888 40 of_platform_device_create_pdata+0xa0/0xcc
126) 848 112 of_platform_bus_create+0x1bc/0x34c
127) 736 48 of_platform_populate+0x90/0x124
128) 688 32 simple_pm_bus_probe+0x54/0x80
129) 656 32 platform_probe+0x68/0xc8
130) 624 64 really_probe+0x114/0x588
131) 560 48 driver_probe_device+0x88/0x1f0
132) 512 32 __device_attach_driver+0xac/0x118
133) 480 48 bus_for_each_drv+0x90/0xe0
134) 432 48 __device_attach+0xe0/0x1ec
135) 384 16 device_initial_probe+0x1c/0x20
136) 368 32 bus_probe_device+0x98/0xa0
137) 336 48 deferred_probe_work_func+0xac/0xfc
138) 288 104 process_one_work+0x270/0x7c4
139) 184 48 worker_thread+0x1ec/0x524
140) 136 48 kthread+0x160/0x178
141) 88 8 ret_from_fork+0x14/0x38
142) 80 80 0x0
Tomi
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Random stack corruption on v5.13 with dra76
2021-05-21 12:57 ` Tomi Valkeinen
@ 2021-05-21 13:06 ` Tony Lindgren
0 siblings, 0 replies; 8+ messages in thread
From: Tony Lindgren @ 2021-05-21 13:06 UTC (permalink / raw)
To: Tomi Valkeinen
Cc: linux-arm-kernel, Nishanth Menon, Bajjuri, Praneeth, linux-omap,
Laurent Pinchart
Hi,
* Tomi Valkeinen <tomi.valkeinen@ideasonboard.com> [210521 12:57]:
> Well... I guess there's no bug as such. The simple-pm-bus just uses a lot
> more stack. Looking at the reports from CONFIG_STACK_TRACER, in the good
> commit the biggest stack is in USB probing with depth of 4896. With
> 01d7136894410a71932096e0fb9f1d301b6ccf07, and a debug print in
> omap_i2c_xfer_common, I can get to 7544.
>
> I don't know what to do about this, so I'm giving up for the time being and
> "fix" it for myself by increasing THREAD_SIZE_ORDER.
>
> # cat /sys/kernel/debug/tracing/stack_trace
> Depth Size Location (143 entries)
> ----- ---- --------
> 0) 7544 28 ftrace_graph_call+0x0/0xc
> 1) 7516 28 rcu_read_lock_sched_held+0x38/0x88
> 2) 7488 120 lock_acquire+0x1ac/0x560
> 3) 7368 16 _raw_spin_lock_irqsave+0x64/0x80
> 4) 7352 56 timekeeping_advance+0x3c/0x934
> 5) 7296 144 update_wall_time+0x1c/0x20
> 6) 7152 16 tick_do_update_jiffies64+0x1f0/0x304
> 7) 7136 104 tick_sched_timer+0xf0/0x10c
> 8) 7032 96 __hrtimer_run_queues+0x2d8/0x68c
> 9) 6936 112 hrtimer_interrupt+0x158/0x388
> 10) 6824 16 dmtimer_clockevent_interrupt+0x34/0x3c
> 11) 6808 64 __handle_irq_event_percpu+0x108/0x384
> 12) 6744 40 handle_irq_event+0x68/0xe0
> 13) 6704 32 handle_fasteoi_irq+0xbc/0x224
> 14) 6672 40 __handle_domain_irq+0x98/0x10c
> 15) 6632 48 gic_handle_irq+0x9c/0xc4
> 16) 6584 68 __irq_svc+0x70/0x98
> 17) 6516 60 console_unlock+0x444/0x688
> 18) 6456 196 vprintk_emit+0x138/0x344
> 19) 6260 68 vprintk_default+0x34/0x3c
> 20) 6192 48 vprintk_func+0xec/0x194
> 21) 6144 24 printk+0x3c/0x64
> 22) 6120 104 omap_i2c_xfer_common+0x698/0x6e4
> 23) 6016 16 omap_i2c_xfer_irq+0x1c/0x20
> 24) 6000 80 __i2c_transfer+0x184/0xa80
> 25) 5920 40 i2c_transfer+0xa8/0x144
> 26) 5880 64 regmap_i2c_read+0x6c/0xa8
> 27) 5816 72 _regmap_raw_read+0x138/0x474
> 28) 5744 40 _regmap_bus_read+0x54/0x80
> 29) 5704 56 _regmap_read+0x74/0x248
> 30) 5648 32 regmap_read+0x4c/0x70
> 31) 5616 32 regulator_get_voltage_sel_regmap+0x40/0x94
> 32) 5584 32 regulator_attr_is_visible+0x18c/0x278
> 33) 5552 88 internal_create_group+0xe4/0x3e0
> 34) 5464 40 internal_create_groups.part.0+0x54/0x98
> 35) 5424 16 sysfs_create_groups+0x24/0x30
> 36) 5408 96 device_add+0x280/0x944
> 37) 5312 104 regulator_register+0x694/0xc4c
> 38) 5208 32 devm_regulator_register+0x48/0x84
> 39) 5176 72 palmas_smps_registration+0x1b4/0x480
> 40) 5104 136 palmas_regulators_probe+0x348/0x450
> 41) 4968 32 platform_probe+0x68/0xc8
> 42) 4936 64 really_probe+0x114/0x588
> 43) 4872 48 driver_probe_device+0x88/0x1f0
> 44) 4824 32 __device_attach_driver+0xac/0x118
> 45) 4792 48 bus_for_each_drv+0x90/0xe0
> 46) 4744 48 __device_attach+0xe0/0x1ec
> 47) 4696 16 device_initial_probe+0x1c/0x20
> 48) 4680 32 bus_probe_device+0x98/0xa0
> 49) 4648 96 device_add+0x37c/0x944
> 50) 4552 16 of_device_add+0x44/0x4c
> 51) 4536 40 of_platform_device_create_pdata+0xa0/0xcc
> 52) 4496 112 of_platform_bus_create+0x1bc/0x34c
> 53) 4384 48 of_platform_populate+0x90/0x124
> 54) 4336 32 devm_of_platform_populate+0x4c/0x90
> 55) 4304 88 palmas_i2c_probe+0x478/0x608
> 56) 4216 40 i2c_device_probe+0x2a0/0x2f8
> 57) 4176 64 really_probe+0x114/0x588
> 58) 4112 48 driver_probe_device+0x88/0x1f0
> 59) 4064 32 __device_attach_driver+0xac/0x118
> 60) 4032 48 bus_for_each_drv+0x90/0xe0
> 61) 3984 48 __device_attach+0xe0/0x1ec
> 62) 3936 16 device_initial_probe+0x1c/0x20
> 63) 3920 32 bus_probe_device+0x98/0xa0
> 64) 3888 96 device_add+0x37c/0x944
> 65) 3792 24 device_register+0x24/0x28
> 66) 3768 48 i2c_new_client_device+0x140/0x264
> 67) 3720 112 of_i2c_register_devices+0xdc/0x188
> 68) 3608 64 i2c_register_adapter+0x220/0x700
> 69) 3544 32 __i2c_add_numbered_adapter+0x60/0xac
> 70) 3512 32 i2c_add_adapter+0xa0/0xe4
> 71) 3480 16 i2c_add_numbered_adapter+0x2c/0x30
> 72) 3464 80 omap_i2c_probe+0x404/0x6b8
> 73) 3384 32 platform_probe+0x68/0xc8
> 74) 3352 64 really_probe+0x114/0x588
> 75) 3288 48 driver_probe_device+0x88/0x1f0
> 76) 3240 32 __device_attach_driver+0xac/0x118
> 77) 3208 48 bus_for_each_drv+0x90/0xe0
> 78) 3160 48 __device_attach+0xe0/0x1ec
> 79) 3112 16 device_initial_probe+0x1c/0x20
> 80) 3096 32 bus_probe_device+0x98/0xa0
> 81) 3064 96 device_add+0x37c/0x944
> 82) 2968 16 of_device_add+0x44/0x4c
> 83) 2952 40 of_platform_device_create_pdata+0xa0/0xcc
> 84) 2912 112 of_platform_bus_create+0x1bc/0x34c
> 85) 2800 48 of_platform_populate+0x90/0x124
> 86) 2752 104 sysc_probe+0x10e4/0x1628
> 87) 2648 32 platform_probe+0x68/0xc8
> 88) 2616 64 really_probe+0x114/0x588
> 89) 2552 48 driver_probe_device+0x88/0x1f0
> 90) 2504 32 __device_attach_driver+0xac/0x118
> 91) 2472 48 bus_for_each_drv+0x90/0xe0
> 92) 2424 48 __device_attach+0xe0/0x1ec
> 93) 2376 16 device_initial_probe+0x1c/0x20
> 94) 2360 32 bus_probe_device+0x98/0xa0
> 95) 2328 96 device_add+0x37c/0x944
> 96) 2232 16 of_device_add+0x44/0x4c
> 97) 2216 40 of_platform_device_create_pdata+0xa0/0xcc
> 98) 2176 112 of_platform_bus_create+0x1bc/0x34c
> 99) 2064 48 of_platform_populate+0x90/0x124
> 100) 2016 32 simple_pm_bus_probe+0x54/0x80
> 101) 1984 32 platform_probe+0x68/0xc8
> 102) 1952 64 really_probe+0x114/0x588
> 103) 1888 48 driver_probe_device+0x88/0x1f0
> 104) 1840 32 __device_attach_driver+0xac/0x118
> 105) 1808 48 bus_for_each_drv+0x90/0xe0
> 106) 1760 48 __device_attach+0xe0/0x1ec
> 107) 1712 16 device_initial_probe+0x1c/0x20
> 108) 1696 32 bus_probe_device+0x98/0xa0
> 109) 1664 96 device_add+0x37c/0x944
> 110) 1568 16 of_device_add+0x44/0x4c
> 111) 1552 40 of_platform_device_create_pdata+0xa0/0xcc
> 112) 1512 112 of_platform_bus_create+0x1bc/0x34c
> 113) 1400 48 of_platform_populate+0x90/0x124
> 114) 1352 32 simple_pm_bus_probe+0x54/0x80
> 115) 1320 32 platform_probe+0x68/0xc8
> 116) 1288 64 really_probe+0x114/0x588
> 117) 1224 48 driver_probe_device+0x88/0x1f0
> 118) 1176 32 __device_attach_driver+0xac/0x118
> 119) 1144 48 bus_for_each_drv+0x90/0xe0
> 120) 1096 48 __device_attach+0xe0/0x1ec
> 121) 1048 16 device_initial_probe+0x1c/0x20
> 122) 1032 32 bus_probe_device+0x98/0xa0
> 123) 1000 96 device_add+0x37c/0x944
> 124) 904 16 of_device_add+0x44/0x4c
> 125) 888 40 of_platform_device_create_pdata+0xa0/0xcc
> 126) 848 112 of_platform_bus_create+0x1bc/0x34c
> 127) 736 48 of_platform_populate+0x90/0x124
> 128) 688 32 simple_pm_bus_probe+0x54/0x80
> 129) 656 32 platform_probe+0x68/0xc8
> 130) 624 64 really_probe+0x114/0x588
> 131) 560 48 driver_probe_device+0x88/0x1f0
> 132) 512 32 __device_attach_driver+0xac/0x118
> 133) 480 48 bus_for_each_drv+0x90/0xe0
> 134) 432 48 __device_attach+0xe0/0x1ec
> 135) 384 16 device_initial_probe+0x1c/0x20
> 136) 368 32 bus_probe_device+0x98/0xa0
> 137) 336 48 deferred_probe_work_func+0xac/0xfc
> 138) 288 104 process_one_work+0x270/0x7c4
> 139) 184 48 worker_thread+0x1ec/0x524
> 140) 136 48 kthread+0x160/0x178
> 141) 88 8 ret_from_fork+0x14/0x38
> 142) 80 80 0x0
Heh that's a pretty crazy stack there :)
Tony
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-05-21 13:08 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <0f48c7e5-6acd-1143-35ef-3dea2255bec6@ideasonboard.com>
2021-05-21 5:36 ` Random stack corruption on v5.13 with dra76 Tony Lindgren
2021-05-21 7:04 ` Tomi Valkeinen
2021-05-21 7:39 ` Tony Lindgren
2021-05-21 8:45 ` Tomi Valkeinen
2021-05-21 9:14 ` Tony Lindgren
2021-05-21 10:30 ` Tomi Valkeinen
2021-05-21 12:57 ` Tomi Valkeinen
2021-05-21 13:06 ` Tony Lindgren
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).