perf: out-of-bounds write in perf_callchain_store

perf: out-of-bounds write in perf_callchain_store

[Dmitry Vyukov]

Hello,

The following program trigger an out-of-bounds write in
perf_callchain_store (if run in a parallel loop):

https://gist.githubusercontent.com/dvyukov/c05d883e776a353a1d063b670f50bde6/raw/1c8906b1aacfbd8a0cc0b5cf0cc4d0535345e497/gistfile1.txt


BUG: KASAN: slab-out-of-bounds in perf_callchain_user+0xe65/0xfc0 at
addr ffff88003e162840
Write of size 8 by task syz-executor/22516
CPU: 0 PID: 22516 Comm: syz-executor Not tainted 4.8.0-rc5-next-20160905+ #14
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
 ffffffff886b6fe0 ffff88003ec07738 ffffffff82db81a9 ffffffff00000000
 fffffbfff10d6dfc ffff88003e800a00 ffff88003e161740 ffff88003e163740
 0000000000000001 ffff88003e162840 ffff88003ec07760 ffffffff8180b2ec
Call Trace:
 [<ffffffff8180b9c7>] __asan_report_store8_noabort+0x17/0x20
mm/kasan/report.c:332
 [<     inline     >] perf_callchain_store include/linux/perf_event.h:1146
 [<ffffffff81014925>] perf_callchain_user+0xe65/0xfc0
arch/x86/events/core.c:2441
 [<ffffffff816c5f48>] get_perf_callchain+0x448/0x680
kernel/events/callchain.c:235
 [<ffffffff816c62cd>] perf_callchain+0x14d/0x1a0 kernel/events/callchain.c:191
 [<ffffffff816bbf8f>] perf_prepare_sample+0x7df/0xfc0 kernel/events/core.c:5849
 [<     inline     >] __perf_event_output kernel/events/core.c:5962
 [<ffffffff816bc865>] perf_event_output_forward+0xf5/0x220
kernel/events/core.c:5980
 [<ffffffff81691d27>] __perf_event_overflow+0x1c7/0x530
kernel/events/core.c:7059
 [<ffffffff81692b00>] perf_swevent_hrtimer+0x210/0x350 kernel/events/core.c:8247
 [<     inline     >] __run_hrtimer kernel/time/hrtimer.c:1238
 [<ffffffff81504425>] __hrtimer_run_queues+0x325/0xe70
kernel/time/hrtimer.c:1302
 [<ffffffff81506ddb>] hrtimer_interrupt+0x18b/0x420 kernel/time/hrtimer.c:1336
 [<ffffffff8126ceff>] local_apic_timer_interrupt+0x6f/0xe0
arch/x86/kernel/apic/apic.c:933
 [<ffffffff86e1d216>] smp_apic_timer_interrupt+0x76/0xa0
arch/x86/kernel/apic/apic.c:957
 [<ffffffff86e1c2cc>] apic_timer_interrupt+0x8c/0xa0
arch/x86/entry/entry_64.S:487
 <EOI>
Object at ffff88003e161740, in cache kmalloc-8192 size: 8192
Allocated:
PID = 22520
 [<ffffffff8122f486>] save_stack_trace+0x26/0x50 arch/x86/kernel/stacktrace.c:67
 [<ffffffff8180a6d6>] save_stack+0x46/0xd0 mm/kasan/kasan.c:479
 [<     inline     >] set_track mm/kasan/kasan.c:491
 [<ffffffff8180a94d>] kasan_kmalloc+0xad/0xe0 mm/kasan/kasan.c:582
 [<     inline     >] __do_kmalloc_node mm/slab.c:3709
 [<ffffffff81807b47>] __kmalloc_node+0x47/0x70 mm/slab.c:3716
 [<     inline     >] kmalloc_node include/linux/slab.h:532
 [<     inline     >] alloc_callchain_buffers kernel/events/callchain.c:89
 [<ffffffff816c5877>] get_callchain_buffers+0x1a7/0x3d0
kernel/events/callchain.c:136
 [<ffffffff816ac778>] perf_event_alloc+0x1cb8/0x24b0 kernel/events/core.c:9187
 [<ffffffff816b6084>] SYSC_perf_event_open+0x674/0x2250
kernel/events/core.c:9576
 [<ffffffff816bed89>] SyS_perf_event_open+0x39/0x50 kernel/events/core.c:9470
 [<ffffffff86e1a8c0>] entry_SYSCALL_64_fastpath+0x23/0xc1
Memory state around the buggy address:
 ffff88003e162700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 ffff88003e162780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff88003e162800: 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc
                                           ^
 ffff88003e162880: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
 ffff88003e162900: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================


On commit 0693c28cfc8e25f18c25b65a8942c026f1854a3c (Sep 5) of linux-next.

Re: perf: out-of-bounds write in perf_callchain_store

[Peter Zijlstra]

On Tue, Sep 06, 2016 at 03:42:40PM +0200, Dmitry Vyukov wrote:
> Hello,
> 
> The following program trigger an out-of-bounds write in
> perf_callchain_store (if run in a parallel loop):
> 
> https://gist.githubusercontent.com/dvyukov/c05d883e776a353a1d063b670f50bde6/raw/1c8906b1aacfbd8a0cc0b5cf0cc4d0535345e497/gistfile1.txt
> 
> 
> BUG: KASAN: slab-out-of-bounds in perf_callchain_user+0xe65/0xfc0 at
> addr ffff88003e162840
> Write of size 8 by task syz-executor/22516
> CPU: 0 PID: 22516 Comm: syz-executor Not tainted 4.8.0-rc5-next-20160905+ #14
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>  ffffffff886b6fe0 ffff88003ec07738 ffffffff82db81a9 ffffffff00000000
>  fffffbfff10d6dfc ffff88003e800a00 ffff88003e161740 ffff88003e163740
>  0000000000000001 ffff88003e162840 ffff88003ec07760 ffffffff8180b2ec
> Call Trace:
>  [<ffffffff8180b9c7>] __asan_report_store8_noabort+0x17/0x20
> mm/kasan/report.c:332
>  [<     inline     >] perf_callchain_store include/linux/perf_event.h:1146
>  [<ffffffff81014925>] perf_callchain_user+0xe65/0xfc0 arch/x86/events/core.c:2441
>  [<ffffffff816c5f48>] get_perf_callchain+0x448/0x680 kernel/events/callchain.c:235
>  [<ffffffff816c62cd>] perf_callchain+0x14d/0x1a0 kernel/events/callchain.c:191

Urgh, that callchain code is a pain with that context/entries
separation. But I can't see an obvious overrun there.

But WTF is max_contexts a sysctl? that doesn't seen to make any kind of
sense.

Acme, can you untangle that stuff and spot the fail?

Re: perf: out-of-bounds write in perf_callchain_store

[Arnaldo Carvalho de Melo]

Em Tue, Sep 06, 2016 at 04:36:08PM +0200, Peter Zijlstra escreveu:
> On Tue, Sep 06, 2016 at 03:42:40PM +0200, Dmitry Vyukov wrote:
> > Hello,
> > 
> > The following program trigger an out-of-bounds write in
> > perf_callchain_store (if run in a parallel loop):
> > 
> > https://gist.githubusercontent.com/dvyukov/c05d883e776a353a1d063b670f50bde6/raw/1c8906b1aacfbd8a0cc0b5cf0cc4d0535345e497/gistfile1.txt
> > 
> > 
> > BUG: KASAN: slab-out-of-bounds in perf_callchain_user+0xe65/0xfc0 at
> > addr ffff88003e162840
> > Write of size 8 by task syz-executor/22516
> > CPU: 0 PID: 22516 Comm: syz-executor Not tainted 4.8.0-rc5-next-20160905+ #14
> > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> >  ffffffff886b6fe0 ffff88003ec07738 ffffffff82db81a9 ffffffff00000000
> >  fffffbfff10d6dfc ffff88003e800a00 ffff88003e161740 ffff88003e163740
> >  0000000000000001 ffff88003e162840 ffff88003ec07760 ffffffff8180b2ec
> > Call Trace:
> >  [<ffffffff8180b9c7>] __asan_report_store8_noabort+0x17/0x20
> > mm/kasan/report.c:332
> >  [<     inline     >] perf_callchain_store include/linux/perf_event.h:1146
> >  [<ffffffff81014925>] perf_callchain_user+0xe65/0xfc0 arch/x86/events/core.c:2441
> >  [<ffffffff816c5f48>] get_perf_callchain+0x448/0x680 kernel/events/callchain.c:235
> >  [<ffffffff816c62cd>] perf_callchain+0x14d/0x1a0 kernel/events/callchain.c:191
> 
> Urgh, that callchain code is a pain with that context/entries
> separation. But I can't see an obvious overrun there.
> 
> But WTF is max_contexts a sysctl? that doesn't seen to make any kind of
> sense.
> 
> Acme, can you untangle that stuff and spot the fail?

I looked at it briefly some moments ago, couldn't find it so far, have
to look at what was behind adding a sysctl for that :-\

And yeah, that entry/ctx thing, IIRC, was done to reduce patch size,
probably needs some polishing to become clearer.

- Arnaldo

Re: perf: out-of-bounds write in perf_callchain_store

[Dmitry Vyukov]

On Tue, Sep 6, 2016 at 4:44 PM, Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
> Em Tue, Sep 06, 2016 at 04:36:08PM +0200, Peter Zijlstra escreveu:
>> On Tue, Sep 06, 2016 at 03:42:40PM +0200, Dmitry Vyukov wrote:
>> > Hello,
>> >
>> > The following program trigger an out-of-bounds write in
>> > perf_callchain_store (if run in a parallel loop):
>> >
>> > https://gist.githubusercontent.com/dvyukov/c05d883e776a353a1d063b670f50bde6/raw/1c8906b1aacfbd8a0cc0b5cf0cc4d0535345e497/gistfile1.txt
>> >
>> >
>> > BUG: KASAN: slab-out-of-bounds in perf_callchain_user+0xe65/0xfc0 at
>> > addr ffff88003e162840
>> > Write of size 8 by task syz-executor/22516
>> > CPU: 0 PID: 22516 Comm: syz-executor Not tainted 4.8.0-rc5-next-20160905+ #14
>> > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>> >  ffffffff886b6fe0 ffff88003ec07738 ffffffff82db81a9 ffffffff00000000
>> >  fffffbfff10d6dfc ffff88003e800a00 ffff88003e161740 ffff88003e163740
>> >  0000000000000001 ffff88003e162840 ffff88003ec07760 ffffffff8180b2ec
>> > Call Trace:
>> >  [<ffffffff8180b9c7>] __asan_report_store8_noabort+0x17/0x20
>> > mm/kasan/report.c:332
>> >  [<     inline     >] perf_callchain_store include/linux/perf_event.h:1146
>> >  [<ffffffff81014925>] perf_callchain_user+0xe65/0xfc0 arch/x86/events/core.c:2441
>> >  [<ffffffff816c5f48>] get_perf_callchain+0x448/0x680 kernel/events/callchain.c:235
>> >  [<ffffffff816c62cd>] perf_callchain+0x14d/0x1a0 kernel/events/callchain.c:191
>>
>> Urgh, that callchain code is a pain with that context/entries
>> separation. But I can't see an obvious overrun there.
>>
>> But WTF is max_contexts a sysctl? that doesn't seen to make any kind of
>> sense.
>>
>> Acme, can you untangle that stuff and spot the fail?
>
> I looked at it briefly some moments ago, couldn't find it so far, have
> to look at what was behind adding a sysctl for that :-\
>
> And yeah, that entry/ctx thing, IIRC, was done to reduce patch size,
> probably needs some polishing to become clearer.


I believe fuzzer wasn't messing with sysctl's.
But, yeah, I guess it's really bad idea to try to change them on a
running system.

Re: perf: out-of-bounds write in perf_callchain_store

[Arnaldo Carvalho de Melo]

Em Tue, Sep 06, 2016 at 04:49:29PM +0200, Dmitry Vyukov escreveu:
> On Tue, Sep 6, 2016 at 4:44 PM, Arnaldo Carvalho de Melo
> <acme@kernel.org> wrote:
> > Em Tue, Sep 06, 2016 at 04:36:08PM +0200, Peter Zijlstra escreveu:
> >> On Tue, Sep 06, 2016 at 03:42:40PM +0200, Dmitry Vyukov wrote:
> >> > Hello,
> >> >
> >> > The following program trigger an out-of-bounds write in
> >> > perf_callchain_store (if run in a parallel loop):
> >> >
> >> > https://gist.githubusercontent.com/dvyukov/c05d883e776a353a1d063b670f50bde6/raw/1c8906b1aacfbd8a0cc0b5cf0cc4d0535345e497/gistfile1.txt
> >> >
> >> >
> >> > BUG: KASAN: slab-out-of-bounds in perf_callchain_user+0xe65/0xfc0 at
> >> > addr ffff88003e162840
> >> > Write of size 8 by task syz-executor/22516
> >> > CPU: 0 PID: 22516 Comm: syz-executor Not tainted 4.8.0-rc5-next-20160905+ #14
> >> > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> >> >  ffffffff886b6fe0 ffff88003ec07738 ffffffff82db81a9 ffffffff00000000
> >> >  fffffbfff10d6dfc ffff88003e800a00 ffff88003e161740 ffff88003e163740
> >> >  0000000000000001 ffff88003e162840 ffff88003ec07760 ffffffff8180b2ec
> >> > Call Trace:
> >> >  [<ffffffff8180b9c7>] __asan_report_store8_noabort+0x17/0x20
> >> > mm/kasan/report.c:332
> >> >  [<     inline     >] perf_callchain_store include/linux/perf_event.h:1146
> >> >  [<ffffffff81014925>] perf_callchain_user+0xe65/0xfc0 arch/x86/events/core.c:2441
> >> >  [<ffffffff816c5f48>] get_perf_callchain+0x448/0x680 kernel/events/callchain.c:235
> >> >  [<ffffffff816c62cd>] perf_callchain+0x14d/0x1a0 kernel/events/callchain.c:191
> >>
> >> Urgh, that callchain code is a pain with that context/entries
> >> separation. But I can't see an obvious overrun there.
> >>
> >> But WTF is max_contexts a sysctl? that doesn't seen to make any kind of
> >> sense.
> >>
> >> Acme, can you untangle that stuff and spot the fail?
> >
> > I looked at it briefly some moments ago, couldn't find it so far, have
> > to look at what was behind adding a sysctl for that :-\
> >
> > And yeah, that entry/ctx thing, IIRC, was done to reduce patch size,
> > probably needs some polishing to become clearer.
> 
> 
> I believe fuzzer wasn't messing with sysctl's.
> But, yeah, I guess it's really bad idea to try to change them on a
> running system.

Why? They start with a reasonable value, but that is not ok for all
cases, thus they were introduced, and they only can get changed if there
are _no_ callchain users, so what would be the problem?

- Arnaldo

Re: perf: out-of-bounds write in perf_callchain_store

[Peter Zijlstra]

On Tue, Sep 06, 2016 at 04:49:29PM +0200, Dmitry Vyukov wrote:
> On Tue, Sep 6, 2016 at 4:44 PM, Arnaldo Carvalho de Melo
> > I looked at it briefly some moments ago, couldn't find it so far, have
> > to look at what was behind adding a sysctl for that :-\
> >
> > And yeah, that entry/ctx thing, IIRC, was done to reduce patch size,
> > probably needs some polishing to become clearer.
> 
> 
> I believe fuzzer wasn't messing with sysctl's.
> But, yeah, I guess it's really bad idea to try to change them on a
> running system.

The sysctl's should return -EBUSY if its in use... 

Re: perf: out-of-bounds write in perf_callchain_store

[Dmitry Vyukov]

On Tue, Sep 6, 2016 at 5:05 PM, Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
> Em Tue, Sep 06, 2016 at 04:49:29PM +0200, Dmitry Vyukov escreveu:
>> On Tue, Sep 6, 2016 at 4:44 PM, Arnaldo Carvalho de Melo
>> <acme@kernel.org> wrote:
>> > Em Tue, Sep 06, 2016 at 04:36:08PM +0200, Peter Zijlstra escreveu:
>> >> On Tue, Sep 06, 2016 at 03:42:40PM +0200, Dmitry Vyukov wrote:
>> >> > Hello,
>> >> >
>> >> > The following program trigger an out-of-bounds write in
>> >> > perf_callchain_store (if run in a parallel loop):
>> >> >
>> >> > https://gist.githubusercontent.com/dvyukov/c05d883e776a353a1d063b670f50bde6/raw/1c8906b1aacfbd8a0cc0b5cf0cc4d0535345e497/gistfile1.txt
>> >> >
>> >> >
>> >> > BUG: KASAN: slab-out-of-bounds in perf_callchain_user+0xe65/0xfc0 at
>> >> > addr ffff88003e162840
>> >> > Write of size 8 by task syz-executor/22516
>> >> > CPU: 0 PID: 22516 Comm: syz-executor Not tainted 4.8.0-rc5-next-20160905+ #14
>> >> > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>> >> >  ffffffff886b6fe0 ffff88003ec07738 ffffffff82db81a9 ffffffff00000000
>> >> >  fffffbfff10d6dfc ffff88003e800a00 ffff88003e161740 ffff88003e163740
>> >> >  0000000000000001 ffff88003e162840 ffff88003ec07760 ffffffff8180b2ec
>> >> > Call Trace:
>> >> >  [<ffffffff8180b9c7>] __asan_report_store8_noabort+0x17/0x20
>> >> > mm/kasan/report.c:332
>> >> >  [<     inline     >] perf_callchain_store include/linux/perf_event.h:1146
>> >> >  [<ffffffff81014925>] perf_callchain_user+0xe65/0xfc0 arch/x86/events/core.c:2441
>> >> >  [<ffffffff816c5f48>] get_perf_callchain+0x448/0x680 kernel/events/callchain.c:235
>> >> >  [<ffffffff816c62cd>] perf_callchain+0x14d/0x1a0 kernel/events/callchain.c:191
>> >>
>> >> Urgh, that callchain code is a pain with that context/entries
>> >> separation. But I can't see an obvious overrun there.
>> >>
>> >> But WTF is max_contexts a sysctl? that doesn't seen to make any kind of
>> >> sense.
>> >>
>> >> Acme, can you untangle that stuff and spot the fail?
>> >
>> > I looked at it briefly some moments ago, couldn't find it so far, have
>> > to look at what was behind adding a sysctl for that :-\
>> >
>> > And yeah, that entry/ctx thing, IIRC, was done to reduce patch size,
>> > probably needs some polishing to become clearer.
>>
>>
>> I believe fuzzer wasn't messing with sysctl's.
>> But, yeah, I guess it's really bad idea to try to change them on a
>> running system.
>
> Why? They start with a reasonable value, but that is not ok for all
> cases, thus they were introduced, and they only can get changed if there
> are _no_ callchain users, so what would be the problem?


Sorry, I missed the "they only can get changed if there are _no_
callchain users" part.