From: Peter Zijlstra <peterz@infradead.org>
To: Vince Weaver <vincent.weaver@maine.edu>
Cc: Ingo Molnar <mingo@kernel.org>,
linux-kernel@vger.kernel.org,
Thomas Gleixner <tglx@linutronix.de>,
Steven Rostedt <rostedt@goodmis.org>
Subject: Re: [perf] more perf_fuzzer memory corruption
Date: Fri, 18 Apr 2014 17:23:14 +0200 [thread overview]
Message-ID: <20140418152314.GY11182@twins.programming.kicks-ass.net> (raw)
In-Reply-To: <alpine.DEB.2.10.1404181042320.26918@vincent-weaver-1.um.maine.edu>
On Fri, Apr 18, 2014 at 10:45:47AM -0400, Vince Weaver wrote:
>
> OK, since the slab corruption was happening to event->hlist_entry->pprev
> I added a WARN() call to every modifier of pprev under
> include/linux/*list*.h to see what was stomping over freed memory.
>
> This is what came up:
>
> Apr 18 10:36:11 haswell kernel: [ 998.316177] ------------[ cut here ]------------
> Apr 18 10:36:11 haswell kernel: [ 998.321188] WARNING: CPU: 3 PID: 20717 at include/linux/rculist.h:410 perf_trace_add+0xc1/0x100()
> Apr 18 10:36:11 haswell kernel: [ 998.330681] Modules linked in: fuse x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi kvm snd_hda_intel iTCO_wdt snd_hda_controller i915 snd_hda_codec evdev crct10dif_pclmul drm_kms_helper iTCO_vendor_support snd_hwdep snd_pcm drm crc32_pclmul snd_seq mei_me parport_pc parport lpc_ich mfd_core psmouse ghash_clmulni_intel snd_timer snd_seq_device mei snd aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd soundcore pcspkr serio_raw i2c_i801 processor video i2c_algo_bit i2c_core wmi button battery tpm_tis tpm sg sd_mod sr_mod crc_t10dif crct10dif_common cdrom ahci ehci_pci libahci xhci_hcd e1000e libata ehci_hcd ptp scsi_mod crc32c_intel usbcore pps_core usb_common fan thermal thermal_sys
> Apr 18 10:36:11 haswell kernel: [ 998.405736] CPU: 3 PID: 20717 Comm: perf_fuzzer Not tainted 3.15.0-rc1+ #63
> Apr 18 10:36:11 haswell kernel: [ 998.413162] Hardware name: LENOVO 10AM000AUS/SHARKBAY, BIOS FBKT72AUS 01/26/2014
> Apr 18 10:36:11 haswell kernel: [ 998.420987] 0000000000000009 ffff880117923c70 ffffffff8164f753 0000000000000000
> Apr 18 10:36:11 haswell kernel: [ 998.429016] ffff880117923ca8 ffffffff810647cd ffffe8ffffcc30d8 ffff8800ce29f040
> Apr 18 10:36:11 haswell kernel: [ 998.437121] ffffffff81c1ba40 ffff8800cd3d9040 000000da35b18e50 ffff880117923cb8
> Apr 18 10:36:11 haswell kernel: [ 998.445246] Call Trace:
> Apr 18 10:36:11 haswell kernel: [ 998.447910] [<ffffffff8164f753>] dump_stack+0x45/0x56
> Apr 18 10:36:11 haswell kernel: [ 998.453451] [<ffffffff810647cd>] warn_slowpath_common+0x7d/0xa0
> Apr 18 10:36:11 haswell kernel: [ 998.459871] [<ffffffff810648aa>] warn_slowpath_null+0x1a/0x20
> Apr 18 10:36:11 haswell kernel: [ 998.466143] [<ffffffff81125a01>] perf_trace_add+0xc1/0x100
> Apr 18 10:36:11 haswell kernel: [ 998.472160] [<ffffffff81136640>] event_sched_in.isra.76+0x90/0x1e0
> Apr 18 10:36:11 haswell kernel: [ 998.478849] [<ffffffff811367f9>] group_sched_in+0x69/0x1e0
> Apr 18 10:36:11 haswell kernel: [ 998.484812] [<ffffffff81136e45>] __perf_event_enable+0x255/0x260
> Apr 18 10:36:11 haswell kernel: [ 998.491370] [<ffffffff81132340>] remote_function+0x40/0x50
> Apr 18 10:36:11 haswell kernel: [ 998.497311] [<ffffffff810de116>] generic_exec_single+0x126/0x170
> Apr 18 10:36:11 haswell kernel: [ 998.503764] [<ffffffff81132300>] ? task_clock_event_add+0x40/0x40
> Apr 18 10:36:11 haswell kernel: [ 998.510432] [<ffffffff810de1c7>] smp_call_function_single+0x67/0xa0
> Apr 18 10:36:11 haswell kernel: [ 998.517299] [<ffffffff811312b4>] task_function_call+0x44/0x50
> Apr 18 10:36:11 haswell kernel: [ 998.523539] [<ffffffff81136bf0>] ? perf_event_sched_in+0x90/0x90
> Apr 18 10:36:11 haswell kernel: [ 998.530085] [<ffffffff81131350>] perf_event_enable+0x90/0xf0
> Apr 18 10:36:11 haswell kernel: [ 998.536308] [<ffffffff811312c0>] ? task_function_call+0x50/0x50
> Apr 18 10:36:11 haswell kernel: [ 998.542761] [<ffffffff8113142a>] perf_event_for_each_child+0x3a/0xa0
> Apr 18 10:36:11 haswell kernel: [ 998.551512] [<ffffffff811379af>] perf_event_task_enable+0x4f/0x80
> Apr 18 10:36:11 haswell kernel: [ 998.560080] [<ffffffff8107c015>] SyS_prctl+0x255/0x4b0
> Apr 18 10:36:11 haswell kernel: [ 998.567605] [<ffffffff813c1406>] ? lockdep_sys_exit_thunk+0x35/0x67
> Apr 18 10:36:11 haswell kernel: [ 998.576333] [<ffffffff816609ed>] system_call_fastpath+0x1a/0x1f
> Apr 18 10:36:11 haswell kernel: [ 998.584698] ---[ end trace b175966afd57a174 ]---
> Apr 18 10:36:12 haswell kernel: [ 998.910691] ------------[ cut here ]------------
OK, that's a good clue. That looks like we're freeing events that still
are on the owner list, which would indicate we're freeing events that
have a refcount.
I added a WARN in free_event() to check the refcount, along with a
number of false positives (through the perf_event_open() fail path) I do
appear to be getting actual fails here.
At least I can 'reproduce' this. Earlier attempts, even based on your
.config only got me very mysterious lockups -- I suspect the corruption
happens on a slightly different spot or so and completely messes up the
machine.
next prev parent reply other threads:[~2014-04-18 15:23 UTC|newest]
Thread overview: 81+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-04-15 21:37 [perf] more perf_fuzzer memory corruption Vince Weaver
2014-04-15 21:49 ` Thomas Gleixner
2014-04-16 3:21 ` Vince Weaver
2014-04-16 4:18 ` Vince Weaver
2014-04-16 14:15 ` Peter Zijlstra
2014-04-16 17:30 ` Vince Weaver
2014-04-16 17:43 ` Vince Weaver
2014-04-16 17:47 ` Peter Zijlstra
2014-04-17 9:48 ` Ingo Molnar
2014-04-17 11:45 ` Peter Zijlstra
2014-04-17 14:22 ` Ingo Molnar
2014-04-17 14:42 ` Vince Weaver
2014-04-17 14:54 ` Peter Zijlstra
2014-04-17 15:35 ` Vince Weaver
2014-04-18 14:45 ` Vince Weaver
2014-04-18 14:51 ` Vince Weaver
2014-04-18 15:23 ` Peter Zijlstra [this message]
2014-04-18 16:59 ` Peter Zijlstra
2014-04-18 17:15 ` Peter Zijlstra
2014-04-23 20:58 ` Vince Weaver
2014-04-25 2:51 ` Vince Weaver
2014-04-28 14:21 ` Vince Weaver
2014-04-28 19:38 ` Vince Weaver
2014-04-29 9:46 ` Peter Zijlstra
2014-04-29 18:21 ` Vince Weaver
2014-04-29 19:01 ` Peter Zijlstra
2014-04-29 20:59 ` Vince Weaver
2014-04-30 18:44 ` Peter Zijlstra
2014-04-30 21:08 ` Vince Weaver
2014-04-30 22:51 ` Thomas Gleixner
2014-05-01 10:26 ` Peter Zijlstra
2014-05-01 11:50 ` Peter Zijlstra
2014-05-01 12:35 ` Thomas Gleixner
2014-05-01 13:12 ` Peter Zijlstra
2014-05-01 13:29 ` Thomas Gleixner
2014-05-01 13:22 ` Vince Weaver
2014-05-01 14:07 ` Vince Weaver
2014-05-01 14:27 ` Vince Weaver
2014-05-01 15:09 ` Peter Zijlstra
2014-05-01 15:50 ` Vince Weaver
2014-05-01 16:31 ` Thomas Gleixner
2014-05-01 17:18 ` Vince Weaver
2014-05-01 18:49 ` Vince Weaver
2014-05-01 21:32 ` Vince Weaver
2014-05-02 11:15 ` Peter Zijlstra
2014-05-02 15:42 ` Peter Zijlstra
2014-05-02 16:22 ` Vince Weaver
2014-05-02 16:22 ` Peter Zijlstra
2014-05-02 16:43 ` Vince Weaver
2014-05-02 17:27 ` Peter Zijlstra
2014-05-02 17:46 ` Vince Weaver
2014-05-02 19:12 ` Thomas Gleixner
2014-05-02 20:15 ` Vince Weaver
2014-05-02 20:45 ` Thomas Gleixner
2014-05-03 2:32 ` Vince Weaver
2014-05-03 3:02 ` Vince Weaver
2014-05-03 7:33 ` Peter Zijlstra
2014-05-05 9:31 ` Peter Zijlstra
2014-05-05 16:00 ` Vince Weaver
2014-05-05 17:10 ` Vince Weaver
2014-05-05 17:14 ` Peter Zijlstra
2014-05-05 18:47 ` Vince Weaver
2014-05-05 19:36 ` Peter Zijlstra
2014-05-05 19:51 ` Vince Weaver
2014-05-06 1:06 ` Vince Weaver
2014-05-06 16:57 ` Vince Weaver
2014-05-07 16:45 ` Peter Zijlstra
2014-05-08 10:40 ` [tip:perf/core] perf: Fix perf_event_init_context() tip-bot for Peter Zijlstra
2014-05-05 17:29 ` [perf] more perf_fuzzer memory corruption Ingo Molnar
2014-05-06 4:51 ` Vince Weaver
2014-05-06 17:06 ` Vince Weaver
2014-05-07 19:12 ` Ingo Molnar
2014-05-07 19:11 ` Ingo Molnar
2014-05-08 10:40 ` [tip:perf/core] perf: Fix race in removing an event tip-bot for Peter Zijlstra
2014-05-02 17:06 ` [perf] more perf_fuzzer memory corruption Vince Weaver
2014-05-02 17:04 ` Peter Zijlstra
2014-04-29 19:26 ` Steven Rostedt
2014-04-29 8:52 ` Peter Zijlstra
2014-04-29 18:11 ` Vince Weaver
2014-04-29 19:21 ` Steven Rostedt
2014-04-28 17:48 ` Thomas Gleixner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140418152314.GY11182@twins.programming.kicks-ass.net \
--to=peterz@infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=vincent.weaver@maine.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).