* [3.4-rc3] Thread overran stack, or stack corrupted
@ 2012-04-17 17:21 Dave Jones
2012-04-17 20:20 ` Linus Torvalds
0 siblings, 1 reply; 26+ messages in thread
From: Dave Jones @ 2012-04-17 17:21 UTC (permalink / raw)
To: Linux Kernel; +Cc: Linus Torvalds
My syscall fuzzer started showing up some cases where it we seem to be
overrunning the stack. I added a WARN_ON when the stack is really low,
to see if there's a deep call trace, but it's not really telling me much ..
Dave
[ 5393.970003] trinity used greatest stack depth: 1048 bytes left
[ 5419.095374] trinity used greatest stack depth: 8 bytes left
[ 5419.095864] ------------[ cut here ]------------
[ 5419.096611] WARNING: at kernel/exit.c:892 do_exit+0xb77/0xb80()
[ 5419.097830] Hardware name: Precision WorkStation 490
[ 5419.098908] Modules linked in: scsi_transport_iscsi ipt_ULOG dccp_ipv6 tun hidp dccp_ipv4 dccp bnep can_raw sctp binfmt_misc l2tp_ppp l2tp_netlink l2tp_core can_bcm ip_queue rfcomm cmtp kernelcapi af_802154 phonet bluetooth rfkill can pppoe pppox ppp_generic slhc irda crc_ccitt rds af_key rose ax25 atm appletalk ipx p8022 psnap llc p8023 nfs fscache auth_rpcgss nfs_acl lockd ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables btrfs zlib_deflate libcrc32c dm_mirror dm_region_hash dm_log coretemp raid0 ppdev dcdbas usb_debug microcode snd_hda_codec_idt snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm serio_raw i2c_i801 pcspkr iTCO_wdt iTCO_vendor_support tg3 snd_timer i5000_edac snd edac_core soundcore snd_page_alloc i5k_amb shpchp parport_pc parport sunrpc firewire_ohci firewire_core crc_itu_t floppy nouveau ttm drm_kms_helper drm i2c_core mxm_wmi video wmi [last unloaded: scsi_wait_scan]
[ 5419.107431] Pid: 841, comm: trinity Tainted: G W 3.4.0-rc3+ #45
[ 5419.108688] Call Trace:
[ 5419.109803] [<ffffffff81065a9f>] warn_slowpath_common+0x7f/0xc0
[ 5419.111023] [<ffffffff81065afa>] warn_slowpath_null+0x1a/0x20
[ 5419.112463] [<ffffffff8106bba7>] do_exit+0xb77/0xb80
[ 5419.113525] [<ffffffff8106beff>] do_group_exit+0x4f/0xc0
[ 5419.114946] [<ffffffff8107eace>] get_signal_to_deliver+0x20e/0x880
[ 5419.116063] [<ffffffff8107bda0>] ? __send_signal+0x150/0x7f0
[ 5419.117469] [<ffffffff8108b820>] ? task_tgid_nr_ns+0x20/0x20
[ 5419.118805] [<ffffffff8101b315>] do_signal+0x65/0x5d0
[ 5419.120161] [<ffffffff816aed01>] ? _raw_spin_unlock_irq+0x41/0x70
[ 5419.121662] [<ffffffff8107f312>] ? set_current_blocked+0x52/0x60
[ 5419.122583] [<ffffffff813360de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 5419.123994] [<ffffffff816aed01>] ? _raw_spin_unlock_irq+0x41/0x70
[ 5419.125064] [<ffffffff8101b905>] do_notify_resume+0x65/0x80
[ 5419.126505] [<ffffffff813360de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 5419.127668] [<ffffffff816b74e2>] int_signal+0x12/0x17
[ 5419.129118] ---[ end trace bed9ff07ecc14c9d ]---
[ 5419.143061] BUG: unable to handle kernel NULL pointer dereference at 0000000000000048
[ 5419.143712] IP: [<ffffffff8106b230>] do_exit+0x200/0xb80
[ 5419.144004] PGD 202d44067 PUD 201876067 PMD 0
[ 5419.144004] Thread overran stack, or stack corrupted
[ 5419.144004] Oops: 0000 [#1] PREEMPT SMP
[ 5419.144004] CPU 0
[ 5419.144004] Modules linked in: scsi_transport_iscsi ipt_ULOG dccp_ipv6 tun hidp dccp_ipv4 dccp bnep can_raw sctp binfmt_misc l2tp_ppp l2tp_netlink l2tp_core can_bcm ip_queue rfcomm cmtp kernelcapi af_802154 phonet bluetooth rfkill can pppoe pppox ppp_generic slhc irda crc_ccitt rds af_key rose ax25 atm appletalk ipx p8022 psnap llc p8023 nfs fscache auth_rpcgss nfs_acl lockd ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables btrfs zlib_deflate libcrc32c dm_mirror dm_region_hash dm_log coretemp raid0 ppdev dcdbas usb_debug microcode snd_hda_codec_idt snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm serio_raw i2c_i801 pcspkr iTCO_wdt iTCO_vendor_support tg3 snd_timer i5000_edac snd edac_core soundcore snd_page_alloc i5k_amb shpchp parport_pc parport sunrpc firewire_ohci firewire_core crc_itu_t floppy nouveau ttm drm_kms_helper drm i2c_core mxm_wmi video wmi [last unloaded: scsi_wait_scan]
[ 5419.144004]
[ 5419.144004] Pid: 841, comm: trinity Tainted: G W 3.4.0-rc3+ #45 Dell Inc. Precision WorkStation 490 /0DT031
[ 5419.144004] RIP: 0010:[<ffffffff8106b230>] [<ffffffff8106b230>] do_exit+0x200/0xb80
[ 5419.144004] RSP: 0018:ffff88006f2dfcc8 EFLAGS: 00010246
[ 5419.144004] RAX: 0000000000000000 RBX: ffff880028820000 RCX: 0000000000000030
[ 5419.144004] RDX: 0000000000000030 RSI: 0000000000000001 RDI: 0000000000000000
[ 5419.144004] RBP: ffff88006f2dfd48 R08: 0000000000000000 R09: 0000000000000001
[ 5419.144004] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000008
[ 5419.144004] R13: 00007ffffffff000 R14: 0000000000000349 R15: ffff880028820000
[ 5419.144004] FS: 00007f8742d4f700(0000) GS:ffff880236600000(0000) knlGS:0000000000000000
[ 5419.144004] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 5419.144004] CR2: 0000000000000048 CR3: 00000001090fd000 CR4: 00000000000007f0
[ 5419.144004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5419.144004] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 5419.144004] Process trinity (pid: 841, threadinfo ffff88006f2de000, task ffff880028820000)
[ 5419.144004] Stack:
[ 5419.144004] 0000000000000000 0000000128820000 0000000000000000 00007fff00000000
[ 5419.144004] dead4ead00000000 ffffffffffffffff ffffffffffffffff ffffffff829bce50
[ 5419.144004] 0000000000000000 0000000000000000 ffffffff819c7001 0000000000000086
[ 5419.144004] Call Trace:
[ 5419.144004] [<ffffffff8106beff>] do_group_exit+0x4f/0xc0
[ 5419.144004] [<ffffffff8107eace>] get_signal_to_deliver+0x20e/0x880
[ 5419.144004] [<ffffffff8107bda0>] ? __send_signal+0x150/0x7f0
[ 5419.144004] [<ffffffff8108b820>] ? task_tgid_nr_ns+0x20/0x20
[ 5419.144004] [<ffffffff8101b315>] do_signal+0x65/0x5d0
[ 5419.144004] [<ffffffff816aed01>] ? _raw_spin_unlock_irq+0x41/0x70
[ 5419.144004] [<ffffffff8107f312>] ? set_current_blocked+0x52/0x60
[ 5419.144004] [<ffffffff813360de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 5419.144004] [<ffffffff816aed01>] ? _raw_spin_unlock_irq+0x41/0x70
[ 5419.194777] [<ffffffff8101b905>] do_notify_resume+0x65/0x80
[ 5419.194777] [<ffffffff813360de>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 5419.194777] [<ffffffff816b74e2>] int_signal+0x12/0x17
[ 5419.194777] Code: df e8 55 74 0e 00 be 01 00 00 00 48 89 df e8 d8 3e 08 00 44 8b 55 8c 45 85 d2 0f 85 49 05 00 00 48 8b 43 08 49 89 df 48 8b 40 08 <48> 8b 78 48 e8 37 37 07 00 48 89 df e8 7f fd 3a 00 48 89 df e8
[ 5419.194777] RIP [<ffffffff8106b230>] do_exit+0x200/0xb80
[ 5419.194777] RSP <ffff88006f2dfcc8>
[ 5419.194777] CR2: 0000000000000048
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-17 17:21 [3.4-rc3] Thread overran stack, or stack corrupted Dave Jones
@ 2012-04-17 20:20 ` Linus Torvalds
2012-04-17 20:32 ` Dave Jones
0 siblings, 1 reply; 26+ messages in thread
From: Linus Torvalds @ 2012-04-17 20:20 UTC (permalink / raw)
To: Dave Jones, Linux Kernel, Linus Torvalds
On Tue, Apr 17, 2012 at 10:21 AM, Dave Jones <davej@redhat.com> wrote:
> My syscall fuzzer started showing up some cases where it we seem to be
> overrunning the stack. I added a WARN_ON when the stack is really low,
> to see if there's a deep call trace, but it's not really telling me much ..
You seem to have added the WARN_ON() to check_stack_usage() itself.
That's not very useful, because it uses the *current* stack pointer.
Instead, how about just calling "show_trace()" with the actual lowest
stack pointer at that point? That should show you the stack as it was
when it was at its lowest, and that could actually be useful.
IOW, just something like
show_trace(NULL, NULL, (void *)end_of_stack(p) + lowest_to_date, NULL);
Or something kind of like that. Yes?
Linus
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-17 20:20 ` Linus Torvalds
@ 2012-04-17 20:32 ` Dave Jones
2012-04-18 1:36 ` Linus Torvalds
0 siblings, 1 reply; 26+ messages in thread
From: Dave Jones @ 2012-04-17 20:32 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Linux Kernel
On Tue, Apr 17, 2012 at 01:20:51PM -0700, Linus Torvalds wrote:
> On Tue, Apr 17, 2012 at 10:21 AM, Dave Jones <davej@redhat.com> wrote:
> > My syscall fuzzer started showing up some cases where it we seem to be
> > overrunning the stack. I added a WARN_ON when the stack is really low,
> > to see if there's a deep call trace, but it's not really telling me much ..
>
> You seem to have added the WARN_ON() to check_stack_usage() itself.
>
> That's not very useful, because it uses the *current* stack pointer.
> Instead, how about just calling "show_trace()" with the actual lowest
> stack pointer at that point? That should show you the stack as it was
> when it was at its lowest, and that could actually be useful.
>
> IOW, just something like
>
> show_trace(NULL, NULL, (void *)end_of_stack(p) + lowest_to_date, NULL);
>
> Or something kind of like that. Yes?
Ok, this builds. I'll run with this for a while, and see what falls out.
thanks,
Dave
--- linux/kernel/exit.c 2012-03-29 22:45:18.912241586 -0400
+++ linux/kernel/exit.c 2012-04-17 16:29:54.473445787 -0400
@@ -871,7 +871,7 @@
}
#ifdef CONFIG_DEBUG_STACK_USAGE
-static void check_stack_usage(void)
+static void check_stack_usage(struct task_struct *p)
{
static DEFINE_SPINLOCK(low_water_lock);
static int lowest_to_date = THREAD_SIZE;
@@ -888,11 +888,13 @@
"left\n",
current->comm, free);
lowest_to_date = free;
+ if (lowest_to_date < 512)
+ show_trace(NULL, NULL, (long unsigned int *)end_of_stack(p) + lowest_to_date, 0);
}
spin_unlock(&low_water_lock);
}
#else
-static inline void check_stack_usage(void) {}
+static inline void check_stack_usage(struct task_struct *p) {}
#endif
void do_exit(long code)
@@ -987,7 +989,7 @@
exit_shm(tsk);
exit_files(tsk);
exit_fs(tsk);
- check_stack_usage();
+ check_stack_usage(tsk);
exit_thread();
/*
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-17 20:32 ` Dave Jones
@ 2012-04-18 1:36 ` Linus Torvalds
2012-04-18 2:27 ` Steven Rostedt
2012-04-18 3:19 ` Dave Jones
0 siblings, 2 replies; 26+ messages in thread
From: Linus Torvalds @ 2012-04-18 1:36 UTC (permalink / raw)
To: Dave Jones, Linus Torvalds, Linux Kernel
On Tue, Apr 17, 2012 at 1:32 PM, Dave Jones <davej@redhat.com> wrote:
>
> Ok, this builds. I'll run with this for a while, and see what falls out.
So assuming that works and doesn't have some silly thinko in it, I
think it is a worthwhile addition to the whole stack debugging thing.
Right now, the message about "process xyz used most stack, x bytes
free" really is pretty useless. If it were to actually show "hey, this
was the deepest actual stack chain", that sounds quite interesting.
Of course, if the stack is largely used by some leaf function that
just has a big stack frame, that won't show up in the stack trace, but
that's presumably not the worst worry. And hopefully the caller of
that would still be pretty deep and show up without having been
overwritten.
Linus
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 1:36 ` Linus Torvalds
@ 2012-04-18 2:27 ` Steven Rostedt
2012-04-18 3:15 ` Dave Jones
2012-04-18 3:19 ` Dave Jones
1 sibling, 1 reply; 26+ messages in thread
From: Steven Rostedt @ 2012-04-18 2:27 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Dave Jones, Linux Kernel
On Tue, Apr 17, 2012 at 06:36:00PM -0700, Linus Torvalds wrote:
> On Tue, Apr 17, 2012 at 1:32 PM, Dave Jones <davej@redhat.com> wrote:
> >
> > Ok, this builds. I'll run with this for a while, and see what falls out.
>
> So assuming that works and doesn't have some silly thinko in it, I
> think it is a worthwhile addition to the whole stack debugging thing.
> Right now, the message about "process xyz used most stack, x bytes
> free" really is pretty useless. If it were to actually show "hey, this
> was the deepest actual stack chain", that sounds quite interesting.
>
> Of course, if the stack is largely used by some leaf function that
> just has a big stack frame, that won't show up in the stack trace, but
> that's presumably not the worst worry. And hopefully the caller of
> that would still be pretty deep and show up without having been
> overwritten.
Note we have something that checks the stack, even on leaf functions.
Enable CONFIG_STACK_TRACER
and then enable it with the following:
# echo 1 > /proc/sys/kernel/stack_tracer_enabled
# cat /sys/kernel/debug/tracing/stack_trace
Depth Size Location (40 entries)
----- ---- --------
0) 3056 208 select_task_rq_fair+0x30b/0x8b2
1) 2848 96 try_to_wake_up+0xc7/0x30e
2) 2752 16 default_wake_function+0x12/0x14
3) 2736 32 autoremove_wake_function+0x16/0x39
4) 2704 80 __wake_up_common+0x4e/0x84
5) 2624 64 __wake_up+0x39/0x4d
6) 2560 64 insert_work+0x8e/0x9b
7) 2496 48 __queue_work+0x2f/0x41
8) 2448 16 queue_work_on+0x48/0x4f
9) 2432 16 queue_work+0x1f/0x21
10) 2416 16 queue_delayed_work+0x13/0x28
11) 2400 32 ata_pio_queue_task+0x35/0x39
12) 2368 32 ata_sff_qc_issue+0x1e9/0x222
13) 2336 96 ata_qc_issue+0x25e/0x29c
14) 2240 80 __ata_scsi_queuecmd+0x193/0x1ef
15) 2160 80 ata_scsi_queuecmd+0x59/0x93
16) 2080 48 scsi_dispatch_cmd+0x1b1/0x233
17) 2032 96 scsi_request_fn+0x385/0x4d8
18) 1936 32 __generic_unplug_device+0x32/0x36
19) 1904 48 blk_execute_rq_nowait+0x77/0x9e
20) 1856 176 blk_execute_rq+0xa6/0xde
21) 1680 80 scsi_execute+0xf6/0x148
22) 1600 128 scsi_execute_req+0xa9/0xdb
23) 1472 96 sr_test_unit_ready+0x65/0xcb
24) 1376 160 sr_media_change+0x9f/0x2cd
25) 1216 48 media_changed+0x54/0x8c
26) 1168 16 cdrom_media_changed+0x31/0x37
27) 1152 16 sr_block_media_changed+0x19/0x1b
28) 1136 32 check_disk_change+0x29/0x5b
29) 1104 208 cdrom_open+0x3d7/0x4b2
30) 896 64 sr_block_open+0x90/0xad
31) 832 96 __blkdev_get+0xd3/0x358
32) 736 16 blkdev_get+0x10/0x12
33) 720 48 blkdev_open+0x76/0xac
34) 672 96 __dentry_open+0x199/0x2d2
35) 576 32 nameidata_to_filp+0x42/0x53
36) 544 320 do_filp_open+0x4f1/0x9d6
37) 224 80 do_sys_open+0x63/0x10f
38) 144 16 sys_open+0x20/0x22
39) 128 128 system_call_fastpath+0x16/0x1b
You can also use trace-cmd to handle this:
# trace-cmd stack --start
# trace-cmd stack
Depth Size Location (24 entries)
----- ---- --------
0) 2480 48 lock_timer_base+0x2c/0x52
1) 2432 96 __mod_timer+0x3e/0x15e
2) 2336 16 mod_timer_pending+0x15/0x17
3) 2320 64 __nf_ct_refresh_acct+0x60/0xd9
4) 2256 272 tcp_packet+0xe17/0x10e7
5) 1984 224 nf_conntrack_in+0x687/0x86e
6) 1760 16 ipv4_conntrack_local+0x40/0x49
7) 1744 80 nf_iterate+0x46/0x89
8) 1664 112 nf_hook_slow+0x6a/0xcb
9) 1552 32 nf_hook_thresh.clone.0+0x41/0x4a
10) 1520 16 __ip_local_out+0x7e/0x80
11) 1504 32 ip_local_out+0x16/0x29
12) 1472 176 ip_queue_xmit+0x30e/0x37f
13) 1296 128 tcp_transmit_skb+0x64d/0x68b
14) 1168 144 tcp_write_xmit+0x80d/0x8fc
15) 1024 32 __tcp_push_pending_frames+0x2f/0x5d
16) 992 16 tcp_push+0x88/0x8a
17) 976 176 tcp_sendmsg+0x77b/0x876
18) 800 64 __sock_sendmsg+0x61/0x6c
19) 736 176 sock_aio_write+0xc0/0xd4
20) 560 304 do_sync_write+0xe8/0x125
21) 256 64 vfs_write+0xc1/0x10b
22) 192 64 sys_write+0x4a/0x6e
23) 128 128 system_call_fastpath+0x16/0x1b
# trace-cmd stack --stop
This also shows you the size of stack each function took up.
-- Steve
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 2:27 ` Steven Rostedt
@ 2012-04-18 3:15 ` Dave Jones
2012-04-18 3:43 ` Steven Rostedt
0 siblings, 1 reply; 26+ messages in thread
From: Dave Jones @ 2012-04-18 3:15 UTC (permalink / raw)
To: Steven Rostedt; +Cc: Linus Torvalds, Linux Kernel
On Tue, Apr 17, 2012 at 10:27:05PM -0400, Steven Rostedt wrote:
> Note we have something that checks the stack, even on leaf functions.
>
> Enable CONFIG_STACK_TRACER
>
> and then enable it with the following:
>
> # echo 1 > /proc/sys/kernel/stack_tracer_enabled
> # cat /sys/kernel/debug/tracing/stack_trace
> Depth Size Location (40 entries)
> ----- ---- --------
> 0) 3056 208 select_task_rq_fair+0x30b/0x8b2
> 1) 2848 96 try_to_wake_up+0xc7/0x30e
That looks handy. Doesn't seem to work for me though on my test box.
(config option is enabled.)
# echo 1 > /proc/sys/kernel/stack_tracer_enabled
# cat /sys/kernel/debug/tracing/stack_trace
Depth Size Location (-1 entries)
----- ---- --------
That's all she wrote.
Dave
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 1:36 ` Linus Torvalds
2012-04-18 2:27 ` Steven Rostedt
@ 2012-04-18 3:19 ` Dave Jones
2012-04-18 17:02 ` Linus Torvalds
1 sibling, 1 reply; 26+ messages in thread
From: Dave Jones @ 2012-04-18 3:19 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Linux Kernel
On Tue, Apr 17, 2012 at 06:36:00PM -0700, Linus Torvalds wrote:
> On Tue, Apr 17, 2012 at 1:32 PM, Dave Jones <davej@redhat.com> wrote:
> >
> > Ok, this builds. I'll run with this for a while, and see what falls out.
>
> So assuming that works and doesn't have some silly thinko in it, I
> think it is a worthwhile addition to the whole stack debugging thing.
> Right now, the message about "process xyz used most stack, x bytes
> free" really is pretty useless. If it were to actually show "hey, this
> was the deepest actual stack chain", that sounds quite interesting.
>
> Of course, if the stack is largely used by some leaf function that
> just has a big stack frame, that won't show up in the stack trace, but
> that's presumably not the worst worry. And hopefully the caller of
> that would still be pretty deep and show up without having been
> overwritten.
So now that I'm instrumenting it, it's taking a lot longer to trigger
(how typical). Almost 6 hours in though, it's down to 72 bytes, and spewed
the traces below, which look pretty.. deep.
I'll leave it run overnight, to see if it triggers the 'overran stack' trace
again, but I suspect it'll look very similar to the ones shown below.
Dave
[ 2496.639872] trinity used greatest stack depth: 1560 bytes left
[ 3604.156895] trinity used greatest stack depth: 1352 bytes left
[ 5448.499455] trinity used greatest stack depth: 1264 bytes left
[ 9751.238389] trinity used greatest stack depth: 864 bytes left
[15000.946998] trinity used greatest stack depth: 800 bytes left
[15079.438172] trinity used greatest stack depth: 656 bytes left
[21137.019557] trinity used greatest stack depth: 456 bytes left
[21137.019982] Call Trace:
[21137.020353] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21137.020765] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21137.021718] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21137.022231] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21137.023327] [<ffffffff811a9eae>] ? deactivate_slab+0x3ce/0x5f0
[21137.023954] [<ffffffff8108c230>] ? __kernel_text_address+0x60/0x90
[21137.024989] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21137.025704] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21137.026157] [<ffffffff816aeca5>] ? _raw_spin_unlock+0x35/0x60
[21137.027200] [<ffffffff811aa02c>] ? deactivate_slab+0x54c/0x5f0
[21137.027710] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21137.028708] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21137.029558] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21137.030053] [<ffffffff811a7fcc>] ? set_track+0xac/0x1a0
[21137.031017] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21137.031669] [<ffffffff810a847a>] ? cpuacct_charge+0x2a/0x240
[21137.032782] [<ffffffff8109c04f>] ? finish_task_switch+0x3f/0x120
[21137.033295] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21137.033967] [<ffffffff816aebb1>] ? _raw_spin_unlock_irq+0x41/0x70
[21137.035386] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21137.035872] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21137.036376] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21137.037401] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21137.039119] [<ffffffff811a9eae>] ? deactivate_slab+0x3ce/0x5f0
[21137.040116] [<ffffffff8108c230>] ? __kernel_text_address+0x60/0x90
[21137.041752] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21137.042494] [<ffffffff810a2981>] ? get_parent_ip+0x11/0x50
[21137.043751] [<ffffffff816b30ad>] ? sub_preempt_count+0x9d/0xd0
[21137.044302] [<ffffffff816aeca5>] ? _raw_spin_unlock+0x35/0x60
[21137.044796] [<ffffffff811aa02c>] ? deactivate_slab+0x54c/0x5f0
[21137.045883] [<ffffffffa037d202>] ? alloc_extent_state+0x32/0x1c0 [btrfs]
[21137.046391] [<ffffffffa037d202>] ? alloc_extent_state+0x32/0x1c0 [btrfs]
[21137.046838] [<ffffffffa037d202>] ? alloc_extent_state+0x32/0x1c0 [btrfs]
[21137.047369] [<ffffffff811a7fcc>] ? set_track+0xac/0x1a0
[21137.048232] [<ffffffffa037d202>] ? alloc_extent_state+0x32/0x1c0 [btrfs]
[21137.049577] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21137.050115] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21137.050879] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21137.051399] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21137.052206] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21137.053551] [<ffffffff816ac00f>] ? __mutex_unlock_slowpath+0x14f/0x190
[21137.054026] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21137.055167] [<ffffffff816ac05e>] ? mutex_unlock+0xe/0x10
[21137.055655] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21137.056156] [<ffffffff81380165>] ? bit_putcs+0x2e5/0x560
[21137.057035] [<ffffffff814ae4cc>] ? qtd_fill+0xc/0xc0
[21137.058041] [<ffffffff816ac05e>] ? mutex_unlock+0xe/0x10
[21137.058644] [<ffffffff814b5609>] ? qh_append_tds+0xf9/0x690
[21137.059789] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21137.060326] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21137.061115] [<ffffffff816aec5e>] ? _raw_spin_unlock_irqrestore+0x7e/0x90
[21137.062180] [<ffffffff816ac00f>] ? __mutex_unlock_slowpath+0x14f/0x190
[21137.062731] [<ffffffff81050869>] ? __phys_addr+0x9/0x50
[21137.063832] [<ffffffff81345fb8>] ? swiotlb_dma_mapping_error+0x18/0x30
[21137.064337] [<ffffffff81499101>] ? usb_hcd_map_urb_for_dma+0x531/0x5b0
[21137.065074] [<ffffffff813806be>] ? soft_cursor+0x1ce/0x270
[21137.065919] [<ffffffffa00c73d2>] ? nouveau_fbcon_fillrect+0xc2/0x110 [nouveau]
[21137.066758] [<ffffffff8149922f>] ? usb_hcd_submit_urb+0xaf/0x860
[21137.068183] [<ffffffff8137fe3b>] ? bit_cursor+0x60b/0x650
[21137.068702] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21137.069802] [<ffffffff816aec5e>] ? _raw_spin_unlock_irqrestore+0x7e/0x90
[21137.070531] [<ffffffff814d336e>] ? usb_serial_generic_write_start+0x22e/0x270
[21137.071113] [<ffffffff81090667>] ? kfifo_copy_in+0x77/0x90
[21137.071588] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21137.072813] [<ffffffff814d3664>] ? usb_serial_generic_write+0x84/0xd0
[21137.073346] [<ffffffff814d4220>] ? usb_console_write+0x110/0x250
[21137.074210] [<ffffffff81335fae>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[21137.075545] [<ffffffff816af5b4>] ? retint_restore_args+0x13/0x13
[21137.076036] [<ffffffff81066ed9>] ? vprintk+0x389/0x600
[21137.076738] [<ffffffff810d375d>] ? trace_hardirqs_on+0xd/0x10
[21137.078066] [<ffffffff810d375d>] ? trace_hardirqs_on+0xd/0x10
[21137.078542] [<ffffffff810d375d>] ? trace_hardirqs_on+0xd/0x10
[21137.079263] [<ffffffff816a1b67>] ? printk+0x51/0x53
[21137.080295] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21137.080839] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21137.082013] [<ffffffff810e3145>] ? is_module_text_address+0x55/0x60
[21137.082566] [<ffffffff8108c230>] ? __kernel_text_address+0x60/0x90
[21137.083458] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21137.084787] [<ffffffff8101d54f>] ? dump_trace+0x17f/0x2f0
[21137.085246] [<ffffffff8101ea0c>] show_trace_log_lvl+0x5c/0x80
[21137.085962] [<ffffffff8101ea45>] show_trace+0x15/0x20
[21137.087188] [<ffffffff8106bbaf>] do_exit+0xb7f/0xb90
[21137.087629] [<ffffffff8106bf0f>] do_group_exit+0x4f/0xc0
[21137.088388] [<ffffffff8107eade>] get_signal_to_deliver+0x20e/0x880
[21137.089757] [<ffffffff8107bdb0>] ? __send_signal+0x150/0x7f0
[21137.090202] [<ffffffff811177de>] ? __rcu_read_unlock+0xe/0xa0
[21137.091126] [<ffffffff8108b830>] ? task_tgid_nr_ns+0x20/0x20
[21137.091701] [<ffffffff8101b315>] do_signal+0x65/0x5d0
[21137.092560] [<ffffffff816aebb1>] ? _raw_spin_unlock_irq+0x41/0x70
[21137.094242] [<ffffffff8107f322>] ? set_current_blocked+0x52/0x60
[21137.095524] [<ffffffff81335fae>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[21137.096777] [<ffffffff816aebb1>] ? _raw_spin_unlock_irq+0x41/0x70
[21137.098541] [<ffffffff8101b905>] do_notify_resume+0x65/0x80
[21137.099113] [<ffffffff81335fae>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[21137.099686] [<ffffffff816b73a2>] int_signal+0x12/0x17
[21162.880562] trinity used greatest stack depth: 280 bytes left
[21162.881249] Call Trace:
[21162.883712] [<ffffffff811605d4>] ? free_pages_prepare+0x1b4/0x290
[21162.884545] [<ffffffff811b8e20>] ? mem_cgroup_recursive_stat+0x70/0x70
[21162.885217] [<ffffffff810d375d>] ? trace_hardirqs_on+0xd/0x10
[21162.885673] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21162.886320] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21162.887113] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21162.887736] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21162.888600] [<ffffffff811a9eae>] ? deactivate_slab+0x3ce/0x5f0
[21162.889674] [<ffffffff8108c230>] ? __kernel_text_address+0x60/0x90
[21162.890501] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21162.891421] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21162.892022] [<ffffffff816aeca5>] ? _raw_spin_unlock+0x35/0x60
[21162.893069] [<ffffffff811aa02c>] ? deactivate_slab+0x54c/0x5f0
[21162.893717] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21162.895053] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21162.895490] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21162.896200] [<ffffffff811a7fcc>] ? set_track+0xac/0x1a0
[21162.897624] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21162.898462] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21162.899162] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21162.899775] [<ffffffff811a9eae>] ? deactivate_slab+0x3ce/0x5f0
[21162.900428] [<ffffffff8108c230>] ? __kernel_text_address+0x60/0x90
[21162.901518] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21162.902104] [<ffffffff810a2981>] ? get_parent_ip+0x11/0x50
[21162.902917] [<ffffffff816b30ad>] ? sub_preempt_count+0x9d/0xd0
[21162.904239] [<ffffffff816aeca5>] ? _raw_spin_unlock+0x35/0x60
[21162.904675] [<ffffffff811aa02c>] ? deactivate_slab+0x54c/0x5f0
[21162.905402] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21162.906195] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21162.907352] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21162.907863] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21162.908965] [<ffffffff811a9eae>] ? deactivate_slab+0x3ce/0x5f0
[21162.909507] [<ffffffff8108c230>] ? __kernel_text_address+0x60/0x90
[21162.910322] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21162.911307] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21162.911875] [<ffffffff816aeca5>] ? _raw_spin_unlock+0x35/0x60
[21162.912953] [<ffffffff811aa02c>] ? deactivate_slab+0x54c/0x5f0
[21162.913528] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21162.914363] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21162.915689] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21162.916138] [<ffffffff811a7fcc>] ? set_track+0xac/0x1a0
[21162.916812] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21162.918125] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21162.918571] [<ffffffff810ccf2d>] ? trace_hardirqs_off+0xd/0x10
[21162.919272] [<ffffffff811a9eae>] ? deactivate_slab+0x3ce/0x5f0
[21162.920287] [<ffffffff8108c230>] ? __kernel_text_address+0x60/0x90
[21162.920898] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21162.921917] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21162.922564] [<ffffffff816aeca5>] ? _raw_spin_unlock+0x35/0x60
[21162.923857] [<ffffffff811aa02c>] ? deactivate_slab+0x54c/0x5f0
[21162.924318] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21162.925040] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21162.926251] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21162.926741] [<ffffffff811a7fcc>] ? set_track+0xac/0x1a0
[21162.927492] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21162.928929] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21162.929416] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21162.930157] [<ffffffff8143d66a>] ? scsi_pool_alloc_command+0x4a/0x80
[21162.930697] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21162.931644] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21162.932587] [<ffffffff811a9eae>] ? deactivate_slab+0x3ce/0x5f0
[21162.933116] [<ffffffff8108c230>] ? __kernel_text_address+0x60/0x90
[21162.934177] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21162.934712] [<ffffffff810a2981>] ? get_parent_ip+0x11/0x50
[21162.935736] [<ffffffff816b30ad>] ? sub_preempt_count+0x9d/0xd0
[21162.936318] [<ffffffff816aeca5>] ? _raw_spin_unlock+0x35/0x60
[21162.937110] [<ffffffff811aa02c>] ? deactivate_slab+0x54c/0x5f0
[21162.938422] [<ffffffffa037d202>] ? alloc_extent_state+0x32/0x1c0 [btrfs]
[21162.938888] [<ffffffffa037d202>] ? alloc_extent_state+0x32/0x1c0 [btrfs]
[21162.939999] [<ffffffffa037d202>] ? alloc_extent_state+0x32/0x1c0 [btrfs]
[21162.940457] [<ffffffff811a7fcc>] ? set_track+0xac/0x1a0
[21162.941461] [<ffffffffa037d202>] ? alloc_extent_state+0x32/0x1c0 [btrfs]
[21162.941912] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21162.942806] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21162.943406] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21162.944445] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21162.944898] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21162.945692] [<ffffffff816aec5e>] ? _raw_spin_unlock_irqrestore+0x7e/0x90
[21162.946692] [<ffffffff816ac00f>] ? __mutex_unlock_slowpath+0x14f/0x190
[21162.947319] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21162.948440] [<ffffffff816ac05e>] ? mutex_unlock+0xe/0x10
[21162.948933] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21162.949744] [<ffffffff814ae4cc>] ? qtd_fill+0xc/0xc0
[21162.950960] [<ffffffff816ac05e>] ? mutex_unlock+0xe/0x10
[21162.951404] [<ffffffff814b5609>] ? qh_append_tds+0xf9/0x690
[21162.952143] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21162.953576] [<ffffffffa00a5e86>] ? nouveau_dma_wait+0x16/0x5b0 [nouveau]
[21162.954397] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21162.955171] [<ffffffff816aec5e>] ? _raw_spin_unlock_irqrestore+0x7e/0x90
[21162.955803] [<ffffffff816ac00f>] ? __mutex_unlock_slowpath+0x14f/0x190
[21162.956249] [<ffffffffa00c7180>] ? nouveau_fbcon_imageblit+0x90/0x110 [nouveau]
[21162.957053] [<ffffffff81050869>] ? __phys_addr+0x9/0x50
[21162.958037] [<ffffffff81345fb8>] ? swiotlb_dma_mapping_error+0x18/0x30
[21162.958692] [<ffffffff81499101>] ? usb_hcd_map_urb_for_dma+0x531/0x5b0
[21162.960092] [<ffffffffa00c73d2>] ? nouveau_fbcon_fillrect+0xc2/0x110 [nouveau]
[21162.961206] [<ffffffff8149922f>] ? usb_hcd_submit_urb+0xaf/0x860
[21162.962185] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21162.962649] [<ffffffff816aec5e>] ? _raw_spin_unlock_irqrestore+0x7e/0x90
[21162.963072] [<ffffffff814d336e>] ? usb_serial_generic_write_start+0x22e/0x270
[21162.963640] [<ffffffff81090667>] ? kfifo_copy_in+0x77/0x90
[21162.964801] [<ffffffff814d3664>] ? usb_serial_generic_write+0x84/0xd0
[21162.965701] [<ffffffff814d4220>] ? usb_console_write+0x110/0x250
[21162.966264] [<ffffffff81335fae>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[21162.967431] [<ffffffff816af5b4>] ? retint_restore_args+0x13/0x13
[21162.967995] [<ffffffff816aec5e>] ? _raw_spin_unlock_irqrestore+0x7e/0x90
[21162.969424] [<ffffffff81066ed9>] ? vprintk+0x389/0x600
[21162.969877] [<ffffffff810d375d>] ? trace_hardirqs_on+0xd/0x10
[21162.970836] [<ffffffff810d375d>] ? trace_hardirqs_on+0xd/0x10
[21162.971524] [<ffffffff810d375d>] ? trace_hardirqs_on+0xd/0x10
[21162.972502] [<ffffffff816a1b67>] ? printk+0x51/0x53
[21162.973192] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21162.974558] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21162.975000] [<ffffffff810e3145>] ? is_module_text_address+0x55/0x60
[21162.976554] [<ffffffff8108c230>] ? __kernel_text_address+0x60/0x90
[21162.978461] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21162.978907] [<ffffffff8101d54f>] ? dump_trace+0x17f/0x2f0
[21162.979574] [<ffffffff8101ea0c>] show_trace_log_lvl+0x5c/0x80
[21162.979977] [<ffffffff8101ea45>] show_trace+0x15/0x20
[21162.980819] [<ffffffff8106bbaf>] do_exit+0xb7f/0xb90
[21162.982280] [<ffffffff8106bf0f>] do_group_exit+0x4f/0xc0
[21162.982729] [<ffffffff8107eade>] get_signal_to_deliver+0x20e/0x880
[21162.983859] [<ffffffff8107bdb0>] ? __send_signal+0x150/0x7f0
[21162.984789] [<ffffffff811177de>] ? __rcu_read_unlock+0xe/0xa0
[21162.986147] [<ffffffff8108b830>] ? task_tgid_nr_ns+0x20/0x20
[21162.986616] [<ffffffff8101b315>] do_signal+0x65/0x5d0
[21162.987316] [<ffffffff816aebb1>] ? _raw_spin_unlock_irq+0x41/0x70
[21162.987761] [<ffffffff8107f322>] ? set_current_blocked+0x52/0x60
[21162.988680] [<ffffffff81335fae>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[21162.989521] [<ffffffff816aebb1>] ? _raw_spin_unlock_irq+0x41/0x70
[21162.990238] [<ffffffff8101b905>] do_notify_resume+0x65/0x80
[21162.991283] [<ffffffff81335fae>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[21162.992209] [<ffffffff816b73a2>] int_signal+0x12/0x17
[21171.661403] trinity used greatest stack depth: 72 bytes left
[21171.661978] Call Trace:
[21171.662403] [<ffffffff8108c230>] ? __kernel_text_address+0x60/0x90
[21171.663400] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21171.663856] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21171.664895] [<ffffffff8101d559>] ? dump_trace+0x189/0x2f0
[21171.665647] [<ffffffffa037d5aa>] ? free_extent_state+0x5a/0x1b0 [btrfs]
[21171.666475] [<ffffffffa037d5aa>] ? free_extent_state+0x5a/0x1b0 [btrfs]
[21171.667570] [<ffffffffa037d5aa>] ? free_extent_state+0x5a/0x1b0 [btrfs]
[21171.668140] [<ffffffff810a2981>] ? get_parent_ip+0x11/0x50
[21171.669035] [<ffffffff810a2981>] ? get_parent_ip+0x11/0x50
[21171.670080] [<ffffffff810d375d>] ? trace_hardirqs_on+0xd/0x10
[21171.670728] [<ffffffff8133db38>] ? debug_check_no_obj_freed+0x88/0x230
[21171.671778] [<ffffffff810a2981>] ? get_parent_ip+0x11/0x50
[21171.672654] [<ffffffff816b30ad>] ? sub_preempt_count+0x9d/0xd0
[21171.673285] [<ffffffff816aec2a>] ? _raw_spin_unlock_irqrestore+0x4a/0x90
[21171.674667] [<ffffffff8133dc2f>] ? debug_check_no_obj_freed+0x17f/0x230
[21171.675138] [<ffffffff811605d4>] ? free_pages_prepare+0x1b4/0x290
[21171.675722] [<ffffffff810d375d>] ? trace_hardirqs_on+0xd/0x10
[21171.676895] [<ffffffff81160c0e>] ? free_hot_cold_page_list+0x5e/0x1a0
[21171.677493] [<ffffffff811bd0be>] ? mem_cgroup_uncharge_cache_page+0x1e/0x30
[21171.678457] [<ffffffff8116d2da>] ? shrink_page_list+0x69a/0xac0
[21171.679757] [<ffffffffa0090205>] ? ttm_dma_pool_mm_shrink+0x105/0x190 [ttm]
[21171.680287] [<ffffffff810a2981>] ? get_parent_ip+0x11/0x50
[21171.680886] [<ffffffff816b30ad>] ? sub_preempt_count+0x9d/0xd0
[21171.682085] [<ffffffff811c668d>] ? put_super+0x1d/0x40
[21171.682662] [<ffffffffa0090205>] ? ttm_dma_pool_mm_shrink+0x105/0x190 [ttm]
[21171.684097] [<ffffffff810a2981>] ? get_parent_ip+0x11/0x50
[21171.684555] [<ffffffff816b30ad>] ? sub_preempt_count+0x9d/0xd0
[21171.685408] [<ffffffff816aeca5>] ? _raw_spin_unlock+0x35/0x60
[21171.685982] [<ffffffff811c66a1>] ? put_super+0x31/0x40
[21171.687135] [<ffffffff8111d081>] ? delayacct_end+0x71/0xa0
[21171.687675] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21171.688548] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21171.689918] [<ffffffff811a9eae>] ? deactivate_slab+0x3ce/0x5f0
[21171.690414] [<ffffffff8108c230>] ? __kernel_text_address+0x60/0x90
[21171.690989] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21171.691842] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21171.693212] [<ffffffff816aeca5>] ? _raw_spin_unlock+0x35/0x60
[21171.693704] [<ffffffff811aa02c>] ? deactivate_slab+0x54c/0x5f0
[21171.694509] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21171.695534] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21171.696104] [<ffffffff811a9eae>] ? deactivate_slab+0x3ce/0x5f0
[21171.697232] [<ffffffff8108c230>] ? __kernel_text_address+0x60/0x90
[21171.697821] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21171.699117] [<ffffffff810a2981>] ? get_parent_ip+0x11/0x50
[21171.699618] [<ffffffff816b30ad>] ? sub_preempt_count+0x9d/0xd0
[21171.700421] [<ffffffff816aeca5>] ? _raw_spin_unlock+0x35/0x60
[21171.701746] [<ffffffff811aa02c>] ? deactivate_slab+0x54c/0x5f0
[21171.702219] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21171.703277] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21171.703880] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21171.704810] [<ffffffff811a7fcc>] ? set_track+0xac/0x1a0
[21171.706108] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21171.706581] [<ffffffff810d375d>] ? trace_hardirqs_on+0xd/0x10
[21171.707560] [<ffffffff8115a5d5>] ? mempool_alloc_slab+0x15/0x20
[21171.708156] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21171.709302] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21171.709803] [<ffffffff811a9eae>] ? deactivate_slab+0x3ce/0x5f0
[21171.710925] [<ffffffff8108c230>] ? __kernel_text_address+0x60/0x90
[21171.711544] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21171.712409] [<ffffffff810a2981>] ? get_parent_ip+0x11/0x50
[21171.713734] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21171.714231] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21171.715304] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21171.716149] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21171.716902] [<ffffffff811a9eae>] ? deactivate_slab+0x3ce/0x5f0
[21171.717513] [<ffffffff8108c230>] ? __kernel_text_address+0x60/0x90
[21171.718660] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21171.719208] [<ffffffff810a2981>] ? get_parent_ip+0x11/0x50
[21171.720049] [<ffffffff816b30ad>] ? sub_preempt_count+0x9d/0xd0
[21171.721357] [<ffffffff816aeca5>] ? _raw_spin_unlock+0x35/0x60
[21171.721902] [<ffffffff811aa02c>] ? deactivate_slab+0x54c/0x5f0
[21171.722939] [<ffffffffa03a312e>] ? btrfs_add_delayed_tree_ref+0x8e/0x1d0 [btrfs]
[21171.723535] [<ffffffffa03a312e>] ? btrfs_add_delayed_tree_ref+0x8e/0x1d0 [btrfs]
[21171.724412] [<ffffffffa03a312e>] ? btrfs_add_delayed_tree_ref+0x8e/0x1d0 [btrfs]
[21171.725889] [<ffffffff811a7fcc>] ? set_track+0xac/0x1a0
[21171.726373] [<ffffffffa03a312e>] ? btrfs_add_delayed_tree_ref+0x8e/0x1d0 [btrfs]
[21171.726906] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21171.728478] [<ffffffff8133d384>] ? debug_object_activate+0x84/0x1a0
[21171.729179] [<ffffffff8133cf32>] ? __debug_object_init+0xa2/0x410
[21171.729714] [<ffffffff8133d9d7>] ? debug_object_active_state+0x67/0x140
[21171.730938] [<ffffffff811630fb>] ? __set_page_dirty_nobuffers+0x7b/0x1a0
[21171.731420] [<ffffffff8109c04f>] ? finish_task_switch+0x3f/0x120
[21171.732387] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21171.732991] [<ffffffff816aebb1>] ? _raw_spin_unlock_irq+0x41/0x70
[21171.734109] [<ffffffff8109c08c>] ? finish_task_switch+0x7c/0x120
[21171.734748] [<ffffffff8109c04f>] ? finish_task_switch+0x3f/0x120
[21171.735648] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21171.737086] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21171.737554] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21171.738120] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21171.739435] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21171.739889] [<ffffffff810e3123>] ? is_module_text_address+0x33/0x60
[21171.740964] [<ffffffff8108c230>] ? __kernel_text_address+0x60/0x90
[21171.741509] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21171.742293] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21171.743706] [<ffffffff8101d559>] ? dump_trace+0x189/0x2f0
[21171.744207] [<ffffffffa037d5aa>] ? free_extent_state+0x5a/0x1b0 [btrfs]
[21171.744857] [<ffffffffa037d5aa>] ? free_extent_state+0x5a/0x1b0 [btrfs]
[21171.746371] [<ffffffffa037d5aa>] ? free_extent_state+0x5a/0x1b0 [btrfs]
[21171.746883] [<ffffffff811a7fcc>] ? set_track+0xac/0x1a0
[21171.747827] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21171.748485] [<ffffffff810d375d>] ? trace_hardirqs_on+0xd/0x10
[21171.749276] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21171.749859] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21171.750886] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21171.751523] [<ffffffff8108ba29>] ? debug_lockdep_rcu_enabled+0x9/0x40
[21171.752646] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21171.753201] [<ffffffff816aec5e>] ? _raw_spin_unlock_irqrestore+0x7e/0x90
[21171.754044] [<ffffffff81198c0e>] ? dma_pool_alloc+0x22e/0x340
[21171.755396] [<ffffffffa00c7180>] ? nouveau_fbcon_imageblit+0x90/0x110 [nouveau]
[21171.755872] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21171.756904] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21171.757514] [<ffffffff814ae4cc>] ? qtd_fill+0xc/0xc0
[21171.758338] [<ffffffff814b5609>] ? qh_append_tds+0xf9/0x690
[21171.759728] [<ffffffffa00a5e86>] ? nouveau_dma_wait+0x16/0x5b0 [nouveau]
[21171.760393] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21171.761031] [<ffffffff816aec5e>] ? _raw_spin_unlock_irqrestore+0x7e/0x90
[21171.762309] [<ffffffff816ac00f>] ? __mutex_unlock_slowpath+0x14f/0x190
[21171.762794] [<ffffffff81050869>] ? __phys_addr+0x9/0x50
[21171.763690] [<ffffffff81345fb8>] ? swiotlb_dma_mapping_error+0x18/0x30
[21171.764358] [<ffffffff81499101>] ? usb_hcd_map_urb_for_dma+0x531/0x5b0
[21171.765536] [<ffffffffa00c73d2>] ? nouveau_fbcon_fillrect+0xc2/0x110 [nouveau]
[21171.766134] [<ffffffff8149922f>] ? usb_hcd_submit_urb+0xaf/0x860
[21171.766992] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21171.768393] [<ffffffff816aec5e>] ? _raw_spin_unlock_irqrestore+0x7e/0x90
[21171.768864] [<ffffffff814d336e>] ? usb_serial_generic_write_start+0x22e/0x270
[21171.769721] [<ffffffff81090667>] ? kfifo_copy_in+0x77/0x90
[21171.771230] [<ffffffff814d3664>] ? usb_serial_generic_write+0x84/0xd0
[21171.771902] [<ffffffff814d4220>] ? usb_console_write+0x110/0x250
[21171.772388] [<ffffffff81335fae>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[21171.773510] [<ffffffff816af5b4>] ? retint_restore_args+0x13/0x13
[21171.774121] [<ffffffff81066ed9>] ? vprintk+0x389/0x600
[21171.774964] [<ffffffff810d375d>] ? trace_hardirqs_on+0xd/0x10
[21171.776294] [<ffffffff810d375d>] ? trace_hardirqs_on+0xd/0x10
[21171.776891] [<ffffffff810d375d>] ? trace_hardirqs_on+0xd/0x10
[21171.777601] [<ffffffff816a1b67>] ? printk+0x51/0x53
[21171.779000] [<ffffffff810de7b0>] ? __module_address+0x10/0x160
[21171.779666] [<ffffffff816b301e>] ? sub_preempt_count+0xe/0xd0
[21171.780288] [<ffffffff810e3145>] ? is_module_text_address+0x55/0x60
[21171.781622] [<ffffffff8108c230>] ? __kernel_text_address+0x60/0x90
[21171.782202] [<ffffffff8101e861>] ? print_context_stack+0x81/0x140
[21171.782763] [<ffffffff8101d54f>] ? dump_trace+0x17f/0x2f0
[21171.784109] [<ffffffff8101ea0c>] show_trace_log_lvl+0x5c/0x80
[21171.784571] [<ffffffff8101ea45>] show_trace+0x15/0x20
[21171.785503] [<ffffffff8106bbaf>] do_exit+0xb7f/0xb90
[21171.786201] [<ffffffff8106bf0f>] do_group_exit+0x4f/0xc0
[21171.787277] [<ffffffff8107eade>] get_signal_to_deliver+0x20e/0x880
[21171.787886] [<ffffffff8107bdb0>] ? __send_signal+0x150/0x7f0
[21171.788749] [<ffffffff811177de>] ? __rcu_read_unlock+0xe/0xa0
[21171.790075] [<ffffffff8108b830>] ? task_tgid_nr_ns+0x20/0x20
[21171.790520] [<ffffffff8101b315>] do_signal+0x65/0x5d0
[21171.791316] [<ffffffff816aebb1>] ? _raw_spin_unlock_irq+0x41/0x70
[21171.792624] [<ffffffff8107f322>] ? set_current_blocked+0x52/0x60
[21171.793206] [<ffffffff81335fae>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[21171.793784] [<ffffffff816aebb1>] ? _raw_spin_unlock_irq+0x41/0x70
[21171.795181] [<ffffffff8101b905>] do_notify_resume+0x65/0x80
[21171.795614] [<ffffffff81335fae>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[21171.796675] [<ffffffff816b73a2>] int_signal+0x12/0x17
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 3:15 ` Dave Jones
@ 2012-04-18 3:43 ` Steven Rostedt
2012-04-18 3:50 ` Dave Jones
0 siblings, 1 reply; 26+ messages in thread
From: Steven Rostedt @ 2012-04-18 3:43 UTC (permalink / raw)
To: Dave Jones; +Cc: Linus Torvalds, Linux Kernel
On Tue, 2012-04-17 at 23:15 -0400, Dave Jones wrote:
> That looks handy. Doesn't seem to work for me though on my test box.
> (config option is enabled.)
>
> # echo 1 > /proc/sys/kernel/stack_tracer_enabled
> # cat /sys/kernel/debug/tracing/stack_trace
> Depth Size Location (-1 entries)
> ----- ---- --------
>
> That's all she wrote.
Hmm, was there any problems with the function tracer?
Messages in dmesg?
Just to make sure, what does
/sys/kernel/debug/tracing/enabled_functions
have?
This would show what functions the stack tracer is using.
I just compiled the latest vanilla kernel and tried it out, and it
worked for me.
-- Steve
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 3:43 ` Steven Rostedt
@ 2012-04-18 3:50 ` Dave Jones
2012-04-18 3:57 ` Steven Rostedt
0 siblings, 1 reply; 26+ messages in thread
From: Dave Jones @ 2012-04-18 3:50 UTC (permalink / raw)
To: Steven Rostedt; +Cc: Linus Torvalds, Linux Kernel
On Tue, Apr 17, 2012 at 11:43:31PM -0400, Steven Rostedt wrote:
> On Tue, 2012-04-17 at 23:15 -0400, Dave Jones wrote:
>
> > That looks handy. Doesn't seem to work for me though on my test box.
> > (config option is enabled.)
> >
> > # echo 1 > /proc/sys/kernel/stack_tracer_enabled
> > # cat /sys/kernel/debug/tracing/stack_trace
> > Depth Size Location (-1 entries)
> > ----- ---- --------
> >
> > That's all she wrote.
>
> Hmm, was there any problems with the function tracer?
> Messages in dmesg?
nope.
> Just to make sure, what does
>
> /sys/kernel/debug/tracing/enabled_functions
>
> have?
33503 lines.
> This would show what functions the stack tracer is using.
>
> I just compiled the latest vanilla kernel and tried it out, and it
> worked for me.
works on my other machines too with similar kernel configs, just not on the one I'm profiling.
Dave
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 3:50 ` Dave Jones
@ 2012-04-18 3:57 ` Steven Rostedt
2012-04-18 4:00 ` Steven Rostedt
2012-04-18 4:06 ` Dave Jones
0 siblings, 2 replies; 26+ messages in thread
From: Steven Rostedt @ 2012-04-18 3:57 UTC (permalink / raw)
To: Dave Jones; +Cc: Linus Torvalds, Linux Kernel
On Tue, 2012-04-17 at 23:50 -0400, Dave Jones wrote:
> > Just to make sure, what does
> >
> > /sys/kernel/debug/tracing/enabled_functions
> >
> > have?
>
> 33503 lines.
That sounds correct.
>
> > This would show what functions the stack tracer is using.
> >
> > I just compiled the latest vanilla kernel and tried it out, and it
> > worked for me.
>
> works on my other machines too with similar kernel configs, just not on the one I'm profiling.
I have to ask. Is this a x86 box?
Also, have you tested to see if function tracing works?
# cd /sys/kernel/debug/tracing
# echo function > current_tracer
# cat trace
If function tracing doesn't work this wouldn't either.
I'm also assuming you have CONFIG_STACKTRACE and
CONFIG_STACKTRACE_SUPPORT enabled. They should be selected, but configs
can always get screwed up.
-- Steve
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 3:57 ` Steven Rostedt
@ 2012-04-18 4:00 ` Steven Rostedt
2012-04-18 4:07 ` Dave Jones
2012-04-18 4:06 ` Dave Jones
1 sibling, 1 reply; 26+ messages in thread
From: Steven Rostedt @ 2012-04-18 4:00 UTC (permalink / raw)
To: Dave Jones; +Cc: Linus Torvalds, Linux Kernel
On Tue, 2012-04-17 at 23:57 -0400, Steven Rostedt wrote:
> Also, have you tested to see if function tracing works?
>
> # cd /sys/kernel/debug/tracing
> # echo function > current_tracer
> # cat trace
Also make sure that the max is zero:
# cat stack_max_size
# echo 0 > stack_max_size
-- Steve
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 3:57 ` Steven Rostedt
2012-04-18 4:00 ` Steven Rostedt
@ 2012-04-18 4:06 ` Dave Jones
2012-04-18 4:09 ` Steven Rostedt
1 sibling, 1 reply; 26+ messages in thread
From: Dave Jones @ 2012-04-18 4:06 UTC (permalink / raw)
To: Steven Rostedt; +Cc: Linus Torvalds, Linux Kernel
On Tue, Apr 17, 2012 at 11:57:21PM -0400, Steven Rostedt wrote:
> > > This would show what functions the stack tracer is using.
> > >
> > > I just compiled the latest vanilla kernel and tried it out, and it
> > > worked for me.
> >
> > works on my other machines too with similar kernel configs, just not on the one I'm profiling.
>
> I have to ask. Is this a x86 box?
yeah, 64-bit Xeon circa 2008.
> Also, have you tested to see if function tracing works?
>
> # cd /sys/kernel/debug/tracing
> # echo function > current_tracer
> # cat trace
>
> If function tracing doesn't work this wouldn't either.
That just prints out..
# tracer: function
#
# entries-in-buffer/entries-written: 0/0 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
> I'm also assuming you have CONFIG_STACKTRACE and
> CONFIG_STACKTRACE_SUPPORT enabled. They should be selected, but configs
> can always get screwed up.
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_STACKTRACE=y
Dave
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 4:00 ` Steven Rostedt
@ 2012-04-18 4:07 ` Dave Jones
0 siblings, 0 replies; 26+ messages in thread
From: Dave Jones @ 2012-04-18 4:07 UTC (permalink / raw)
To: Steven Rostedt; +Cc: Linus Torvalds, Linux Kernel
On Wed, Apr 18, 2012 at 12:00:44AM -0400, Steven Rostedt wrote:
> On Tue, 2012-04-17 at 23:57 -0400, Steven Rostedt wrote:
>
> > Also, have you tested to see if function tracing works?
> >
> > # cd /sys/kernel/debug/tracing
> > # echo function > current_tracer
> > # cat trace
>
> Also make sure that the max is zero:
>
> # cat stack_max_size
> # echo 0 > stack_max_size
Yep, it is zero.
Dave
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 4:06 ` Dave Jones
@ 2012-04-18 4:09 ` Steven Rostedt
2012-04-18 13:58 ` Dave Jones
0 siblings, 1 reply; 26+ messages in thread
From: Steven Rostedt @ 2012-04-18 4:09 UTC (permalink / raw)
To: Dave Jones; +Cc: Linus Torvalds, Linux Kernel
On Wed, 2012-04-18 at 00:06 -0400, Dave Jones wrote:
> That just prints out..
>
> # tracer: function
> #
> # entries-in-buffer/entries-written: 0/0 #P:4
> #
> # _-----=> irqs-off
> # / _----=> need-resched
> # | / _---=> hardirq/softirq
> # || / _--=> preempt-depth
> # ||| / delay
> # TASK-PID CPU# |||| TIMESTAMP FUNCTION
> # | | | |||| | |
>
> > I'm also assuming you have CONFIG_STACKTRACE and
> > CONFIG_STACKTRACE_SUPPORT enabled. They should be selected, but configs
> > can always get screwed up.
>
> CONFIG_STACKTRACE_SUPPORT=y
> CONFIG_STACKTRACE=y
>
OK, function tracing is broken on your box. Could you send me your full
config and I'll take a look at it tomorrow. Bed time for me now ;-)
-- Steve
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 4:09 ` Steven Rostedt
@ 2012-04-18 13:58 ` Dave Jones
2012-04-18 14:56 ` Steven Rostedt
0 siblings, 1 reply; 26+ messages in thread
From: Dave Jones @ 2012-04-18 13:58 UTC (permalink / raw)
To: Steven Rostedt; +Cc: Linus Torvalds, Linux Kernel
On Wed, Apr 18, 2012 at 12:09:24AM -0400, Steven Rostedt wrote:
> On Wed, 2012-04-18 at 00:06 -0400, Dave Jones wrote:
>
> > That just prints out..
> >
> > # tracer: function
> > #
> > # entries-in-buffer/entries-written: 0/0 #P:4
> > #
> > # _-----=> irqs-off
> > # / _----=> need-resched
> > # | / _---=> hardirq/softirq
> > # || / _--=> preempt-depth
> > # ||| / delay
> > # TASK-PID CPU# |||| TIMESTAMP FUNCTION
> > # | | | |||| | |
> >
> > > I'm also assuming you have CONFIG_STACKTRACE and
> > > CONFIG_STACKTRACE_SUPPORT enabled. They should be selected, but configs
> > > can always get screwed up.
> >
> > CONFIG_STACKTRACE_SUPPORT=y
> > CONFIG_STACKTRACE=y
> >
>
> OK, function tracing is broken on your box. Could you send me your full
> config and I'll take a look at it tomorrow. Bed time for me now ;-)
http://fpaste.org/b8PN/raw/
Dave
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 13:58 ` Dave Jones
@ 2012-04-18 14:56 ` Steven Rostedt
2012-04-18 15:27 ` Dave Jones
` (2 more replies)
0 siblings, 3 replies; 26+ messages in thread
From: Steven Rostedt @ 2012-04-18 14:56 UTC (permalink / raw)
To: Dave Jones; +Cc: Linus Torvalds, Linux Kernel
On Wed, 2012-04-18 at 09:58 -0400, Dave Jones wrote:
> > OK, function tracing is broken on your box. Could you send me your full
> > config and I'll take a look at it tomorrow. Bed time for me now ;-)
>
> http://fpaste.org/b8PN/raw/
Hmm, I had no problems with this config. But that wasn't on my Xeon. I'm
compiling for that box now.
Could you also paste your full dmesg.
Thanks,
-- Steve
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 14:56 ` Steven Rostedt
@ 2012-04-18 15:27 ` Dave Jones
2012-04-18 15:30 ` Dave Jones
2012-04-18 15:40 ` Steven Rostedt
2 siblings, 0 replies; 26+ messages in thread
From: Dave Jones @ 2012-04-18 15:27 UTC (permalink / raw)
To: Steven Rostedt; +Cc: Linus Torvalds, Linux Kernel
On Wed, Apr 18, 2012 at 10:56:05AM -0400, Steven Rostedt wrote:
> On Wed, 2012-04-18 at 09:58 -0400, Dave Jones wrote:
>
> > > OK, function tracing is broken on your box. Could you send me your full
> > > config and I'll take a look at it tomorrow. Bed time for me now ;-)
> >
> > http://fpaste.org/b8PN/raw/
>
> Hmm, I had no problems with this config. But that wasn't on my Xeon. I'm
> compiling for that box now.
>
> Could you also paste your full dmesg.
http://fpaste.org/QFmg/
Dave
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 14:56 ` Steven Rostedt
2012-04-18 15:27 ` Dave Jones
@ 2012-04-18 15:30 ` Dave Jones
2012-04-18 16:43 ` Steven Rostedt
2012-04-18 15:40 ` Steven Rostedt
2 siblings, 1 reply; 26+ messages in thread
From: Dave Jones @ 2012-04-18 15:30 UTC (permalink / raw)
To: Steven Rostedt; +Cc: Linus Torvalds, Linux Kernel
On Wed, Apr 18, 2012 at 10:56:05AM -0400, Steven Rostedt wrote:
> On Wed, 2012-04-18 at 09:58 -0400, Dave Jones wrote:
>
> > > OK, function tracing is broken on your box. Could you send me your full
> > > config and I'll take a look at it tomorrow. Bed time for me now ;-)
> >
> > http://fpaste.org/b8PN/raw/
>
> Hmm, I had no problems with this config. But that wasn't on my Xeon. I'm
> compiling for that box now.
>
> Could you also paste your full dmesg.
Oh, I guess it's this..
"PEBS disabled due to CPU errata."
Crap. Never even noticed that before.
Dave
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 14:56 ` Steven Rostedt
2012-04-18 15:27 ` Dave Jones
2012-04-18 15:30 ` Dave Jones
@ 2012-04-18 15:40 ` Steven Rostedt
2 siblings, 0 replies; 26+ messages in thread
From: Steven Rostedt @ 2012-04-18 15:40 UTC (permalink / raw)
To: Dave Jones; +Cc: Linus Torvalds, Linux Kernel
On Wed, 2012-04-18 at 10:56 -0400, Steven Rostedt wrote:
> On Wed, 2012-04-18 at 09:58 -0400, Dave Jones wrote:
>
> > > OK, function tracing is broken on your box. Could you send me your full
> > > config and I'll take a look at it tomorrow. Bed time for me now ;-)
> >
> > http://fpaste.org/b8PN/raw/
>
> Hmm, I had no problems with this config. But that wasn't on my Xeon. I'm
> compiling for that box now.
I ran your config on my own Xeon, and it worked just fine.
>
> Could you also paste your full dmesg.
I guess I need to examine what happens during boot up that may cause
this issue for you :-/
-- Steve
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 15:30 ` Dave Jones
@ 2012-04-18 16:43 ` Steven Rostedt
2012-05-23 14:37 ` Steven Rostedt
0 siblings, 1 reply; 26+ messages in thread
From: Steven Rostedt @ 2012-04-18 16:43 UTC (permalink / raw)
To: Dave Jones; +Cc: Linus Torvalds, Linux Kernel
On Wed, 2012-04-18 at 11:30 -0400, Dave Jones wrote:
> On Wed, Apr 18, 2012 at 10:56:05AM -0400, Steven Rostedt wrote:
> > On Wed, 2012-04-18 at 09:58 -0400, Dave Jones wrote:
> >
> > > > OK, function tracing is broken on your box. Could you send me your full
> > > > config and I'll take a look at it tomorrow. Bed time for me now ;-)
> > >
> > > http://fpaste.org/b8PN/raw/
> >
> > Hmm, I had no problems with this config. But that wasn't on my Xeon. I'm
> > compiling for that box now.
> >
> > Could you also paste your full dmesg.
>
> Oh, I guess it's this..
>
> "PEBS disabled due to CPU errata."
>
> Crap. Never even noticed that before.
Yeah, but that shouldn't stop the function tracer from working.
-- Steve
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 3:19 ` Dave Jones
@ 2012-04-18 17:02 ` Linus Torvalds
2012-04-18 17:14 ` Dave Jones
0 siblings, 1 reply; 26+ messages in thread
From: Linus Torvalds @ 2012-04-18 17:02 UTC (permalink / raw)
To: Dave Jones, Linus Torvalds, Linux Kernel
On Tue, Apr 17, 2012 at 8:19 PM, Dave Jones <davej@redhat.com> wrote:
>
> So now that I'm instrumenting it, it's taking a lot longer to trigger
> (how typical). Almost 6 hours in though, it's down to 72 bytes, and spewed
> the traces below, which look pretty.. deep.
Yeah. Sadly, they are less useful than I was hoping for. It's not some
single deep call-chain, it's almost all debug stuff and the "did we
release the RCU lock" or preemption checks, which I guess makes sense.
You have tons of options enabled in your kernel that makes for deeper
stack traces, and then all the interesting stuff gets overwritten by
what happened later.
For example, it looks like you have the USB serial console enabled,
and some of those deep stack traces are about that - and largely got
overwritten by the "dump_trace()" logic itself. So dump_trace()
printing stuff out also ended up overwriting the stack trace that we
were interested in.
I assume you have USB serial console on for a reason (ie: great for
catching oopses before the machine dies), but in this case it hurts.
Could you try just adding a
console_lock();
...
console_unlock();
around the show_trace() call. That will force the code to not actually
call down to the console layer until after the console_unlock(), so
the printing of the stack trace won't affect the stack *too* much.
That said, if you get the function tracer thing working, that will
give much nicer backtraces.
Linus
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 17:02 ` Linus Torvalds
@ 2012-04-18 17:14 ` Dave Jones
2012-04-18 18:24 ` Linus Torvalds
0 siblings, 1 reply; 26+ messages in thread
From: Dave Jones @ 2012-04-18 17:14 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Linux Kernel
On Wed, Apr 18, 2012 at 10:02:06AM -0700, Linus Torvalds wrote:
> > the traces below, which look pretty.. deep.
>
> Yeah. Sadly, they are less useful than I was hoping for. It's not some
> single deep call-chain, it's almost all debug stuff and the "did we
> release the RCU lock" or preemption checks, which I guess makes sense.
> You have tons of options enabled in your kernel that makes for deeper
> stack traces, and then all the interesting stuff gets overwritten by
> what happened later.
One thing I'm curious about.. Some of the function names are repeated
for a reason that doesn't seem obvious to me, when the call chain doesn't
call them in a loop. What's that about ?
> I assume you have USB serial console on for a reason (ie: great for
> catching oopses before the machine dies), but in this case it hurts.
Yeah, there's a (possibly related) problem where once a day some oops
gets triggered that just wedges the machine. I've not managed to capture
it yet, and the most I've gotten over the usb console was about
a dozen characters before it hung.
I've disabled the console blanking, and hooked up a monitor to it.
Perhaps that'll be enough to capture it without resorting to usb console.
> Could you try just adding a
>
> console_lock();
> ...
> console_unlock();
>
> around the show_trace() call. That will force the code to not actually
> call down to the console layer until after the console_unlock(), so
> the printing of the stack trace won't affect the stack *too* much.
That's a neat trick. I'll add that, in case I do have to fall back to usb console.
thanks,
Dave
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 17:14 ` Dave Jones
@ 2012-04-18 18:24 ` Linus Torvalds
0 siblings, 0 replies; 26+ messages in thread
From: Linus Torvalds @ 2012-04-18 18:24 UTC (permalink / raw)
To: Dave Jones, Linus Torvalds, Linux Kernel
On Wed, Apr 18, 2012 at 10:14 AM, Dave Jones <davej@redhat.com> wrote:
>
> One thing I'm curious about.. Some of the function names are repeated
> for a reason that doesn't seem obvious to me, when the call chain doesn't
> call them in a loop. What's that about ?
In a "real" stack trace, it's usually due to a pointer to a function
that just gets passed around as an argument.
However, in this case, I think it's simply because you are seeing a
lot of stack-traces "on top of each other", where the callers had
slightly different stack offsets, but old stack traces "shine through"
because the stack of the newer trace is not very dense.
Gcc often generates very sparse stack frames. Sometimes it is because
of stack alignment issues, but more often it's because gcc generates
the frame for all possible local variables, and then we took an early
exit from the function (or are still in the middle of the call chain)
and part of the stack simply hasn't been modified at all.
And sometimes it is because gcc stack space allocation simply sucks.
There was a time when gcc would never share stack space between local
variables even when they had no overlap in lifetime. I know they fixed
*some* of those cases, but I suspect it's still true that gcc just
often allocates way more stack than it really needs, and then you see
old stale stack contents shine through.
Linus
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-04-18 16:43 ` Steven Rostedt
@ 2012-05-23 14:37 ` Steven Rostedt
2012-05-23 15:05 ` Dave Jones
0 siblings, 1 reply; 26+ messages in thread
From: Steven Rostedt @ 2012-05-23 14:37 UTC (permalink / raw)
To: Dave Jones; +Cc: Linus Torvalds, Linux Kernel
On Wed, 2012-04-18 at 12:43 -0400, Steven Rostedt wrote:
> On Wed, 2012-04-18 at 11:30 -0400, Dave Jones wrote:
> > On Wed, Apr 18, 2012 at 10:56:05AM -0400, Steven Rostedt wrote:
> > > On Wed, 2012-04-18 at 09:58 -0400, Dave Jones wrote:
> > >
> > > > > OK, function tracing is broken on your box. Could you send me your full
> > > > > config and I'll take a look at it tomorrow. Bed time for me now ;-)
> > > >
> > > > http://fpaste.org/b8PN/raw/
> > >
> > > Hmm, I had no problems with this config. But that wasn't on my Xeon. I'm
> > > compiling for that box now.
> > >
> > > Could you also paste your full dmesg.
> >
> > Oh, I guess it's this..
> >
> > "PEBS disabled due to CPU errata."
> >
> > Crap. Never even noticed that before.
>
> Yeah, but that shouldn't stop the function tracer from working.
>
Hi Dave,
Have we ever figured out why this wasn't working? Did it start working
on newer kernels?
-- Steve
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-05-23 14:37 ` Steven Rostedt
@ 2012-05-23 15:05 ` Dave Jones
2012-05-23 15:14 ` Steven Rostedt
0 siblings, 1 reply; 26+ messages in thread
From: Dave Jones @ 2012-05-23 15:05 UTC (permalink / raw)
To: Steven Rostedt; +Cc: Linus Torvalds, Linux Kernel
On Wed, May 23, 2012 at 10:37:08AM -0400, Steven Rostedt wrote:
> > > Oh, I guess it's this..
> > >
> > > "PEBS disabled due to CPU errata."
> > >
> > > Crap. Never even noticed that before.
> >
> > Yeah, but that shouldn't stop the function tracer from working.
>
> Hi Dave,
>
> Have we ever figured out why this wasn't working? Did it start working
> on newer kernels?
I think that was where we left it.
But! Linus' current tree seems to have working tracing on that box.
Don't break it again! ;-)
Dave
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [3.4-rc3] Thread overran stack, or stack corrupted
2012-05-23 15:05 ` Dave Jones
@ 2012-05-23 15:14 ` Steven Rostedt
0 siblings, 0 replies; 26+ messages in thread
From: Steven Rostedt @ 2012-05-23 15:14 UTC (permalink / raw)
To: Dave Jones; +Cc: Linus Torvalds, Linux Kernel
On Wed, 2012-05-23 at 11:05 -0400, Dave Jones wrote:
> But! Linus' current tree seems to have working tracing on that box.
> Don't break it again! ;-)
You could install ktest and run a reverse bisect to find out what fixed
it ;-)
-- Steve
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2012-05-23 15:14 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-04-17 17:21 [3.4-rc3] Thread overran stack, or stack corrupted Dave Jones
2012-04-17 20:20 ` Linus Torvalds
2012-04-17 20:32 ` Dave Jones
2012-04-18 1:36 ` Linus Torvalds
2012-04-18 2:27 ` Steven Rostedt
2012-04-18 3:15 ` Dave Jones
2012-04-18 3:43 ` Steven Rostedt
2012-04-18 3:50 ` Dave Jones
2012-04-18 3:57 ` Steven Rostedt
2012-04-18 4:00 ` Steven Rostedt
2012-04-18 4:07 ` Dave Jones
2012-04-18 4:06 ` Dave Jones
2012-04-18 4:09 ` Steven Rostedt
2012-04-18 13:58 ` Dave Jones
2012-04-18 14:56 ` Steven Rostedt
2012-04-18 15:27 ` Dave Jones
2012-04-18 15:30 ` Dave Jones
2012-04-18 16:43 ` Steven Rostedt
2012-05-23 14:37 ` Steven Rostedt
2012-05-23 15:05 ` Dave Jones
2012-05-23 15:14 ` Steven Rostedt
2012-04-18 15:40 ` Steven Rostedt
2012-04-18 3:19 ` Dave Jones
2012-04-18 17:02 ` Linus Torvalds
2012-04-18 17:14 ` Dave Jones
2012-04-18 18:24 ` Linus Torvalds
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.