From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1755644AbbLGLFG (ORCPT <rfc822;w@1wt.eu>);
	Mon, 7 Dec 2015 06:05:06 -0500
Received: from mail-wm0-f52.google.com ([74.125.82.52]:35812 "EHLO
	mail-wm0-f52.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1751998AbbLGLFC (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Mon, 7 Dec 2015 06:05:02 -0500
MIME-Version: 1.0
In-Reply-To: <CACT4Y+a6O=5C_hMWC3FHw76n8WkmN+_-MSxwcDcLwHz46vZnNA@mail.gmail.com>
References: <CACT4Y+bQqHPsxdh5nhSA154KeCxRirk8QGwRe-5pMUToGE8yAg@mail.gmail.com>
 <20151204203212.GA64517@ast-mbp.thefacebook.com> <CACT4Y+a6O=5C_hMWC3FHw76n8WkmN+_-MSxwcDcLwHz46vZnNA@mail.gmail.com>
From: Dmitry Vyukov <dvyukov@google.com>
Date: Mon, 7 Dec 2015 12:04:41 +0100
Message-ID: <CACT4Y+bt13W2ijCVjGiyK1431wnCCXmQH0pGvWZZA8+WoQvj3A@mail.gmail.com>
Subject: Re: use-after-free in __perf_install_in_context
To: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>, Ingo Molnar <mingo@redhat.com>,
        Arnaldo Carvalho de Melo <acme@kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        syzkaller <syzkaller@googlegroups.com>,
        Kostya Serebryany <kcc@google.com>,
        Alexander Potapenko <glider@google.com>,
        Eric Dumazet <edumazet@google.com>,
        Sasha Levin <sasha.levin@oracle.com>
Content-Type: text/plain; charset=UTF-8
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Dec 4, 2015 at 10:00 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
> On Fri, Dec 4, 2015 at 9:32 PM, Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
>> On Fri, Dec 04, 2015 at 09:04:35PM +0100, Dmitry Vyukov wrote:
>>> Hello,
>>>
>>> While running syzkaller fuzzer I am seeing lots of the following
>>> use-after-free reports. Unfortunately all my numerous attempts to
>>> reproduce them in a controlled environment failed. They pop up during
>>> fuzzing periodically (once in several hours in a single VM), but
>>> whenever I try to stress-replay what happened in the VM before the
>>> report, the use-after-free does not reproduce. Can somebody
>>> knowledgeable in perf subsystem look at the report? Maybe it is
>>> possible to figure out what happened based purely on the report. I can
>>> pretty reliably test any proposed fixes.
>>> All reports look like this one. Then it is usually followed by other
>>> reports and eventually kernel hangs or dies. What happens in the
>>> fuzzer is essentially random syscalls with random arguments, tasks
>>> born and die concurrently and so on. I was able to reproduce it by
>>> restricting syscalls only to perf_event_open, perf ioctls and bpf
>>> syscall.
>>
>> For the sake of trying to narrow it down:
>> does the error disappear when you stop using bpf syscall in your fuzzing?
>> If yes, then I could have missed some interaction between perf_event_free,
>> kprobe free and bpf_prog_free.
>> There was a race there before.
>> May be there is still something else.
>
>
> It is a good question. I will test it.


Testing without bpf, so far hit the following warning on a non-tainted kernel:


------------[ cut here ]------------
WARNING: CPU: 0 PID: 26259 at kernel/events/core.c:2696
task_ctx_sched_out+0x12c/0x1c0()
Modules linked in:
CPU: 0 PID: 26259 Comm: syzkaller_execu Not tainted 4.4.0-rc3+ #150
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
 0000000000000000 ffff88006579f9a8 ffffffff82c6f2a8 0000000041b58ab3
 ffffffff8788bf8d ffffffff82c6f1f6 ffff88003dd32d00 ffff88003469b9c0
 ffffffff81626130 ffff88006579fa10 0000000000000000 0000000000006693
Call Trace:
 [<     inline     >] __dump_stack lib/dump_stack.c:15
 [<ffffffff82c6f2a8>] dump_stack+0xb2/0xfa lib/dump_stack.c:50
 [<ffffffff812ca446>] warn_slowpath_common+0xe6/0x170 kernel/panic.c:460
 [<ffffffff812ca699>] warn_slowpath_null+0x29/0x30 kernel/panic.c:493
 [<ffffffff8163302c>] task_ctx_sched_out+0x12c/0x1c0 kernel/events/core.c:2696
 [<     inline     >] perf_event_exit_task_context kernel/events/core.c:8815
 [<ffffffff8165669a>] perf_event_exit_task+0x27a/0xae0 kernel/events/core.c:8881
 [<ffffffff812d4a12>] do_exit+0x892/0x3050 kernel/exit.c:759
 [<ffffffff812d732c>] do_group_exit+0xec/0x390 kernel/exit.c:880
 [<ffffffff81302177>] get_signal+0x677/0x1bf0 kernel/signal.c:2307
 [<ffffffff8118645e>] do_signal+0x7e/0x2170 arch/x86/kernel/signal.c:709
 [<ffffffff81003a1e>] exit_to_usermode_loop+0xfe/0x1e0
arch/x86/entry/common.c:247
 [<     inline     >] prepare_exit_to_usermode arch/x86/entry/common.c:282
 [<ffffffff8100733b>] syscall_return_slowpath+0x16b/0x240
arch/x86/entry/common.c:344
 [<ffffffff868dafe2>] int_ret_from_sys_call+0x25/0x9f
arch/x86/entry/entry_64.S:281
---[ end trace c5e9cee9f12b7d5f ]---



static void task_ctx_sched_out(struct perf_event_context *ctx)
{
        struct perf_cpu_context *cpuctx = __get_cpu_context(ctx);

        if (!cpuctx->task_ctx)
                return;

        if (WARN_ON_ONCE(ctx != cpuctx->task_ctx))
                return;

        ctx_sched_out(ctx, cpuctx, EVENT_ALL);
        cpuctx->task_ctx = NULL;
}