kernel panic: stack is corrupted in pointer

kernel panic: stack is corrupted in pointer

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit:    1438cde7 Add linux-next specific files for 20190716
git tree:       linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=13988058600000
kernel config:  https://syzkaller.appspot.com/x/.config?x=3430a151e1452331
dashboard link: https://syzkaller.appspot.com/bug?extid=79f5f028005a77ecb6bb
compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=111fc8afa00000

The bug was bisected to:

commit 96a5d8d4915f3e241ebb48d5decdd110ab9c7dcf
Author: Leo Liu <leo.liu@amd.com>
Date:   Fri Jul 13 15:26:28 2018 +0000

     drm/amdgpu: Make sure IB tests flushed after IP resume

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=14a46200600000
final crash:    https://syzkaller.appspot.com/x/report.txt?x=16a46200600000
console output: https://syzkaller.appspot.com/x/log.txt?x=12a46200600000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+79f5f028005a77ecb6bb@syzkaller.appspotmail.com
Fixes: 96a5d8d4915f ("drm/amdgpu: Make sure IB tests flushed after IP  
resume")

Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in:  
pointer+0x702/0x750 lib/vsprintf.c:2187
Shutting down cpus with NMI
Kernel Offset: disabled


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
syzbot can test patches for this bug, for details see:
https://goo.gl/tpsmEJ#testing-patches

Re: kernel panic: stack is corrupted in pointer

[Dmitry Vyukov]

On Wed, Jul 17, 2019 at 10:58 AM syzbot
<syzbot+79f5f028005a77ecb6bb@syzkaller.appspotmail.com> wrote:
>
> Hello,
>
> syzbot found the following crash on:
>
> HEAD commit:    1438cde7 Add linux-next specific files for 20190716
> git tree:       linux-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=13988058600000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=3430a151e1452331
> dashboard link: https://syzkaller.appspot.com/bug?extid=79f5f028005a77ecb6bb
> compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=111fc8afa00000

From the repro it looks like the same bpf stack overflow bug. +John
We need to dup them onto some canonical report for this bug, or this
becomes unmanageable.

#syz dup: kernel panic: corrupted stack end in dput

> The bug was bisected to:
>
> commit 96a5d8d4915f3e241ebb48d5decdd110ab9c7dcf
> Author: Leo Liu <leo.liu@amd.com>
> Date:   Fri Jul 13 15:26:28 2018 +0000
>
>      drm/amdgpu: Make sure IB tests flushed after IP resume
>
> bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=14a46200600000
> final crash:    https://syzkaller.appspot.com/x/report.txt?x=16a46200600000
> console output: https://syzkaller.appspot.com/x/log.txt?x=12a46200600000
>
> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: syzbot+79f5f028005a77ecb6bb@syzkaller.appspotmail.com
> Fixes: 96a5d8d4915f ("drm/amdgpu: Make sure IB tests flushed after IP
> resume")
>
> Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in:
> pointer+0x702/0x750 lib/vsprintf.c:2187
> Shutting down cpus with NMI
> Kernel Offset: disabled
>
>
> ---
> This bug is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at syzkaller@googlegroups.com.
>
> syzbot will keep track of this bug report. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> For information about bisection process see: https://goo.gl/tpsmEJ#bisection
> syzbot can test patches for this bug, for details see:
> https://goo.gl/tpsmEJ#testing-patches

Re: kernel panic: stack is corrupted in pointer

[John Fastabend]

Dmitry Vyukov wrote:
> On Wed, Jul 17, 2019 at 10:58 AM syzbot
> <syzbot+79f5f028005a77ecb6bb@syzkaller.appspotmail.com> wrote:
> >
> > Hello,
> >
> > syzbot found the following crash on:
> >
> > HEAD commit:    1438cde7 Add linux-next specific files for 20190716
> > git tree:       linux-next
> > console output: https://syzkaller.appspot.com/x/log.txt?x=13988058600000
> > kernel config:  https://syzkaller.appspot.com/x/.config?x=3430a151e1452331
> > dashboard link: https://syzkaller.appspot.com/bug?extid=79f5f028005a77ecb6bb
> > compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
> > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=111fc8afa00000
> 
> From the repro it looks like the same bpf stack overflow bug. +John
> We need to dup them onto some canonical report for this bug, or this
> becomes unmanageable.

Fixes in bpf tree should fix this. Hopefully, we will squash this once fixes
percolate up.

#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git

> 
> #syz dup: kernel panic: corrupted stack end in dput
> 
> > The bug was bisected to:
> >
> > commit 96a5d8d4915f3e241ebb48d5decdd110ab9c7dcf
> > Author: Leo Liu <leo.liu@amd.com>
> > Date:   Fri Jul 13 15:26:28 2018 +0000
> >
> >      drm/amdgpu: Make sure IB tests flushed after IP resume
> >
> > bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=14a46200600000
> > final crash:    https://syzkaller.appspot.com/x/report.txt?x=16a46200600000
> > console output: https://syzkaller.appspot.com/x/log.txt?x=12a46200600000
> >
> > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > Reported-by: syzbot+79f5f028005a77ecb6bb@syzkaller.appspotmail.com
> > Fixes: 96a5d8d4915f ("drm/amdgpu: Make sure IB tests flushed after IP
> > resume")
> >
> > Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in:
> > pointer+0x702/0x750 lib/vsprintf.c:2187
> > Shutting down cpus with NMI
> > Kernel Offset: disabled
> >
> >
> > ---
> > This bug is generated by a bot. It may contain errors.
> > See https://goo.gl/tpsmEJ for more information about syzbot.
> > syzbot engineers can be reached at syzkaller@googlegroups.com.
> >
> > syzbot will keep track of this bug report. See:
> > https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> > For information about bisection process see: https://goo.gl/tpsmEJ#bisection
> > syzbot can test patches for this bug, for details see:
> > https://goo.gl/tpsmEJ#testing-patches

Re: Re: kernel panic: stack is corrupted in pointer

[syzbot]

> Dmitry Vyukov wrote:
>> On Wed, Jul 17, 2019 at 10:58 AM syzbot
>> <syzbot+79f5f028005a77ecb6bb@syzkaller.appspotmail.com> wrote:
>> >
>> > Hello,
>> >
>> > syzbot found the following crash on:
>> >
>> > HEAD commit:    1438cde7 Add linux-next specific files for 20190716
>> > git tree:       linux-next
>> > console output:  
>> https://syzkaller.appspot.com/x/log.txt?x=13988058600000
>> > kernel config:   
>> https://syzkaller.appspot.com/x/.config?x=3430a151e1452331
>> > dashboard link:  
>> https://syzkaller.appspot.com/bug?extid=79f5f028005a77ecb6bb
>> > compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
>> > syz repro:       
>> https://syzkaller.appspot.com/x/repro.syz?x=111fc8afa00000

>>  From the repro it looks like the same bpf stack overflow bug. +John
>> We need to dup them onto some canonical report for this bug, or this
>> becomes unmanageable.

> Fixes in bpf tree should fix this. Hopefully, we will squash this once  
> fixes
> percolate up.

> #syz test: git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git

">" does not look like a valid git branch or commit.



>> #syz dup: kernel panic: corrupted stack end in dput

>> > The bug was bisected to:
>> >
>> > commit 96a5d8d4915f3e241ebb48d5decdd110ab9c7dcf
>> > Author: Leo Liu <leo.liu@amd.com>
>> > Date:   Fri Jul 13 15:26:28 2018 +0000
>> >
>> >      drm/amdgpu: Make sure IB tests flushed after IP resume
>> >
>> > bisection log:   
>> https://syzkaller.appspot.com/x/bisect.txt?x=14a46200600000
>> > final crash:     
>> https://syzkaller.appspot.com/x/report.txt?x=16a46200600000
>> > console output:  
>> https://syzkaller.appspot.com/x/log.txt?x=12a46200600000
>> >
>> > IMPORTANT: if you fix the bug, please add the following tag to the  
>> commit:
>> > Reported-by: syzbot+79f5f028005a77ecb6bb@syzkaller.appspotmail.com
>> > Fixes: 96a5d8d4915f ("drm/amdgpu: Make sure IB tests flushed after IP
>> > resume")
>> >
>> > Kernel panic - not syncing: stack-protector: Kernel stack is corrupted  
>> in:
>> > pointer+0x702/0x750 lib/vsprintf.c:2187
>> > Shutting down cpus with NMI
>> > Kernel Offset: disabled
>> >
>> >
>> > ---
>> > This bug is generated by a bot. It may contain errors.
>> > See https://goo.gl/tpsmEJ for more information about syzbot.
>> > syzbot engineers can be reached at syzkaller@googlegroups.com.
>> >
>> > syzbot will keep track of this bug report. See:
>> > https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>> > For information about bisection process see:  
>> https://goo.gl/tpsmEJ#bisection
>> > syzbot can test patches for this bug, for details see:
>> > https://goo.gl/tpsmEJ#testing-patches

Re: Re: kernel panic: stack is corrupted in pointer

[John Fastabend]

syzbot wrote:
> > Dmitry Vyukov wrote:
> >> On Wed, Jul 17, 2019 at 10:58 AM syzbot
> >> <syzbot+79f5f028005a77ecb6bb@syzkaller.appspotmail.com> wrote:
> >> >
> >> > Hello,
> >> >
> >> > syzbot found the following crash on:
> >> >
> >> > HEAD commit:    1438cde7 Add linux-next specific files for 20190716
> >> > git tree:       linux-next
> >> > console output:  
> >> https://syzkaller.appspot.com/x/log.txt?x=13988058600000
> >> > kernel config:   
> >> https://syzkaller.appspot.com/x/.config?x=3430a151e1452331
> >> > dashboard link:  
> >> https://syzkaller.appspot.com/bug?extid=79f5f028005a77ecb6bb
> >> > compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
> >> > syz repro:       
> >> https://syzkaller.appspot.com/x/repro.syz?x=111fc8afa00000
> 
> >>  From the repro it looks like the same bpf stack overflow bug. +John
> >> We need to dup them onto some canonical report for this bug, or this
> >> becomes unmanageable.
> 
> > Fixes in bpf tree should fix this. Hopefully, we will squash this once  
> > fixes
> > percolate up.
> 
> > #syz test: git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git
> 
> ">" does not look like a valid git branch or commit.
> 

try again,

#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git master

Re: kernel panic: stack is corrupted in pointer

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger  
crash:

Reported-and-tested-by:  
syzbot+79f5f028005a77ecb6bb@syzkaller.appspotmail.com

Tested on:

commit:         decb705e libbpf: fix using uninitialized ioctl results
git tree:       bpf
kernel config:  https://syzkaller.appspot.com/x/.config?x=87305c3ca9c25c70
compiler:       gcc (GCC) 9.0.0 20181231 (experimental)

Note: testing is done by a robot and is best-effort only.

Re: kernel panic: stack is corrupted in pointer

[Dmitry Vyukov]

On Tue, Jul 23, 2019 at 7:26 PM John Fastabend <john.fastabend@gmail.com> wrote:
>
> Dmitry Vyukov wrote:
> > On Wed, Jul 17, 2019 at 10:58 AM syzbot
> > <syzbot+79f5f028005a77ecb6bb@syzkaller.appspotmail.com> wrote:
> > >
> > > Hello,
> > >
> > > syzbot found the following crash on:
> > >
> > > HEAD commit:    1438cde7 Add linux-next specific files for 20190716
> > > git tree:       linux-next
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=13988058600000
> > > kernel config:  https://syzkaller.appspot.com/x/.config?x=3430a151e1452331
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=79f5f028005a77ecb6bb
> > > compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
> > > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=111fc8afa00000
> >
> > From the repro it looks like the same bpf stack overflow bug. +John
> > We need to dup them onto some canonical report for this bug, or this
> > becomes unmanageable.
>
> Fixes in bpf tree should fix this. Hopefully, we will squash this once fixes
> percolate up.
>
> #syz test: git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git

Cool! What is the fix?
We don't need to wait for the fix to percolate up (and then down
too!). syzbot gracefully handles when a patch is not yet present
everywhere (it happens all the time).

Btw, this was due to a stack overflow, right? Or something else?
We are trying to make KASAN configuration detect stack overflows too,
so that it does not cause havoc next time. But it turns out to be
non-trivial and our current attempt seems to fail:
https://groups.google.com/forum/#!topic/kasan-dev/IhYv7QYhLfY


> > #syz dup: kernel panic: corrupted stack end in dput
> >
> > > The bug was bisected to:
> > >
> > > commit 96a5d8d4915f3e241ebb48d5decdd110ab9c7dcf
> > > Author: Leo Liu <leo.liu@amd.com>
> > > Date:   Fri Jul 13 15:26:28 2018 +0000
> > >
> > >      drm/amdgpu: Make sure IB tests flushed after IP resume
> > >
> > > bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=14a46200600000
> > > final crash:    https://syzkaller.appspot.com/x/report.txt?x=16a46200600000
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=12a46200600000
> > >
> > > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > > Reported-by: syzbot+79f5f028005a77ecb6bb@syzkaller.appspotmail.com
> > > Fixes: 96a5d8d4915f ("drm/amdgpu: Make sure IB tests flushed after IP
> > > resume")
> > >
> > > Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in:
> > > pointer+0x702/0x750 lib/vsprintf.c:2187
> > > Shutting down cpus with NMI
> > > Kernel Offset: disabled
> > >
> > >
> > > ---
> > > This bug is generated by a bot. It may contain errors.
> > > See https://goo.gl/tpsmEJ for more information about syzbot.
> > > syzbot engineers can be reached at syzkaller@googlegroups.com.
> > >
> > > syzbot will keep track of this bug report. See:
> > > https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> > > For information about bisection process see: https://goo.gl/tpsmEJ#bisection
> > > syzbot can test patches for this bug, for details see:
> > > https://goo.gl/tpsmEJ#testing-patches
>
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/5d37433a832d_3aba2ae4f6ec05bc3a%40john-XPS-13-9370.notmuch.

Re: kernel panic: stack is corrupted in pointer

[John Fastabend]

Dmitry Vyukov wrote:
> On Tue, Jul 23, 2019 at 7:26 PM John Fastabend <john.fastabend@gmail.com> wrote:
> >
> > Dmitry Vyukov wrote:
> > > On Wed, Jul 17, 2019 at 10:58 AM syzbot
> > > <syzbot+79f5f028005a77ecb6bb@syzkaller.appspotmail.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > syzbot found the following crash on:
> > > >
> > > > HEAD commit:    1438cde7 Add linux-next specific files for 20190716
> > > > git tree:       linux-next
> > > > console output: https://syzkaller.appspot.com/x/log.txt?x=13988058600000
> > > > kernel config:  https://syzkaller.appspot.com/x/.config?x=3430a151e1452331
> > > > dashboard link: https://syzkaller.appspot.com/bug?extid=79f5f028005a77ecb6bb
> > > > compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
> > > > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=111fc8afa00000
> > >
> > > From the repro it looks like the same bpf stack overflow bug. +John
> > > We need to dup them onto some canonical report for this bug, or this
> > > becomes unmanageable.
> >
> > Fixes in bpf tree should fix this. Hopefully, we will squash this once fixes
> > percolate up.
> >
> > #syz test: git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git
> 
> Cool! What is the fix?

It took a series of patches here,

https://www.spinics.net/lists/netdev/msg586986.html

The fix commits from bpf tree are,

(git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git) 

318892ac068397f40ff81d9155898da01493b1d2
ac78fc148d8249dbf382c2127456dd08ec5b161c
f87e62d45e51b12d48d2cb46b5cde8f83b866bc4
313ab004805cf52a42673b15852b3842474ccd87
32857cf57f920cdc03b5095f08febec94cf9c36b
45a4521dcbd92e71c9e53031b40e34211d3b4feb
2bb90e5cc90e1d09f631aeab041a9cf913a5bbe5
0e858739c2d2eedeeac1d35bfa0ec3cc2a7190d8
95fa145479fbc0a0c1fd3274ceb42ec03c042a4a

The last commit fixes this paticular syzbot issue,

commit 95fa145479fbc0a0c1fd3274ceb42ec03c042a4a
Author: John Fastabend <john.fastabend@gmail.com>
Date:   Fri Jul 19 10:29:22 2019 -0700

    bpf: sockmap/tls, close can race with map free

The other commits address some other issues found while testing.

> We don't need to wait for the fix to percolate up (and then down
> too!). syzbot gracefully handles when a patch is not yet present
> everywhere (it happens all the time).

Great. By the way the above should fix many of the outstanding
reports against bpf sockmap and tls side. I'll have to walk through
each one individually to double check though. I guess we can mark
them as dup reports and syzbot should sort it out?

> 
> Btw, this was due to a stack overflow, right? Or something else?

Right, stack overflow due to race in updating sock ops where build a
circular call chain.

> We are trying to make KASAN configuration detect stack overflows too,
> so that it does not cause havoc next time. But it turns out to be
> non-trivial and our current attempt seems to fail:
> https://groups.google.com/forum/#!topic/kasan-dev/IhYv7QYhLfY
> 
>