[syzbot] kernel panic: corrupted stack end in openat

[syzbot] kernel panic: corrupted stack end in openat

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    1e28eed1 Linux 5.12-rc3
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=167535e6d00000
kernel config:  https://syzkaller.appspot.com/x/.config?x=e0cee1f53de33ca3
dashboard link: https://syzkaller.appspot.com/bug?extid=0b06ef9b44d00d600183
userspace arch: arm

Unfortunately, I don't have any reproducer for this issue yet.

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+0b06ef9b44d00d600183@syzkaller.appspotmail.com

Kernel panic - not syncing: corrupted stack end detected inside scheduler
CPU: 0 PID: 3263 Comm: syz-fuzzer Not tainted 5.12.0-rc3-syzkaller #0
Hardware name: ARM-Versatile Express
Backtrace: 
[<81802700>] (dump_backtrace) from [<81802974>] (show_stack+0x18/0x1c arch/arm/kernel/traps.c:252)
 r7:00000080 r6:60000093 r5:00000000 r4:82b58544
[<8180295c>] (show_stack) from [<8180a048>] (__dump_stack lib/dump_stack.c:79 [inline])
[<8180295c>] (show_stack) from [<8180a048>] (dump_stack+0xb8/0xe8 lib/dump_stack.c:120)
[<81809f90>] (dump_stack) from [<81803508>] (panic+0x130/0x378 kernel/panic.c:231)
 r7:81f4bdc0 r6:82a392a4 r5:00000000 r4:82c6b0d0
[<818033d8>] (panic) from [<81820270>] (schedule_debug kernel/sched/core.c:4822 [inline])
[<818033d8>] (panic) from [<81820270>] (__schedule+0xcb8/0xcc8 kernel/sched/core.c:4967)
 r3:57ac6e9d r2:00040000 r1:81f5a53c r0:81f4bdc0
 r7:00000001
[<8181f5b8>] (__schedule) from [<8182046c>] (preempt_schedule_common+0x3c/0xac kernel/sched/core.c:5233)
 r10:0000071f r9:ffefd000 r8:00000001 r7:81820510 r6:00000001 r5:81820510
 r4:85888000
[<81820430>] (preempt_schedule_common) from [<81820510>] (preempt_schedule+0x34/0x38 kernel/sched/core.c:5258)
 r7:82c6a4e0 r6:00000001 r5:85888000 r4:df48d420
[<818204dc>] (preempt_schedule) from [<8048c884>] (__kunmap_atomic include/linux/highmem-internal.h:114 [inline])
[<818204dc>] (preempt_schedule) from [<8048c884>] (clear_highpage include/linux/highmem.h:204 [inline])
[<818204dc>] (preempt_schedule) from [<8048c884>] (kernel_init_free_pages+0xc4/0xd0 mm/page_alloc.c:1212)
[<8048c7c0>] (kernel_init_free_pages) from [<80492ce8>] (post_alloc_hook mm/page_alloc.c:2305 [inline])
[<8048c7c0>] (kernel_init_free_pages) from [<80492ce8>] (prep_new_page mm/page_alloc.c:2311 [inline])
[<8048c7c0>] (kernel_init_free_pages) from [<80492ce8>] (get_page_from_freelist+0x163c/0x1698 mm/page_alloc.c:3951)
 r10:df48d3f0 r9:82bf89c0 r8:df48d3f0 r7:0000000b r6:00000002 r5:00000001
 r4:df48d3f8 r3:00000001
[<804916ac>] (get_page_from_freelist) from [<804933c8>] (__alloc_pages_nodemask+0x164/0x1850 mm/page_alloc.c:5001)
 r10:00000000 r9:860a9a80 r8:00112cca r7:0000000b r6:00000081 r5:00000008
 r4:00000000
[<80493264>] (__alloc_pages_nodemask) from [<8042f0f8>] (__alloc_pages include/linux/gfp.h:525 [inline])
[<80493264>] (__alloc_pages_nodemask) from [<8042f0f8>] (__alloc_pages_node include/linux/gfp.h:538 [inline])
[<80493264>] (__alloc_pages_nodemask) from [<8042f0f8>] (alloc_pages_node include/linux/gfp.h:552 [inline])
[<80493264>] (__alloc_pages_nodemask) from [<8042f0f8>] (alloc_pages include/linux/gfp.h:571 [inline])
[<80493264>] (__alloc_pages_nodemask) from [<8042f0f8>] (__page_cache_alloc include/linux/pagemap.h:289 [inline])
[<80493264>] (__alloc_pages_nodemask) from [<8042f0f8>] (page_cache_ra_unbounded+0xc4/0x294 mm/readahead.c:216)
 r10:860a9a84 r9:860a9a80 r8:8588956c r7:0000000b r6:00000107 r5:85889688
 r4:df48d3c0
[<8042f034>] (page_cache_ra_unbounded) from [<8042f3c4>] (do_page_cache_ra+0xfc/0x150 mm/readahead.c:267)
 r10:860a99ac r9:00000001 r8:00000020 r7:80000013 r6:8042f624 r5:85889688
 r4:860a9908
[<8042f2c8>] (do_page_cache_ra) from [<8042f624>] (ondemand_readahead+0x20c/0x47c mm/readahead.c:549)
 r10:00000001 r9:000000fc r8:00000020 r7:000000dc r6:85889688 r5:00000000
 r4:85aa8ea0
[<8042f418>] (ondemand_readahead) from [<8042f958>] (page_cache_async_ra mm/readahead.c:607 [inline])
[<8042f418>] (ondemand_readahead) from [<8042f958>] (page_cache_async_ra+0xc4/0x110 mm/readahead.c:581)
 r10:85889818 r9:df4a4be0 r8:860a9a80 r7:85889714 r6:00000000 r5:85889688
 r4:85aa8ea0
[<8042f894>] (page_cache_async_ra) from [<80420d1c>] (page_cache_async_readahead include/linux/pagemap.h:863 [inline])
[<8042f894>] (page_cache_async_ra) from [<80420d1c>] (filemap_readahead mm/filemap.c:2350 [inline])
[<8042f894>] (page_cache_async_ra) from [<80420d1c>] (filemap_get_pages+0x254/0x648 mm/filemap.c:2391)
 r7:85889714 r6:000000db r5:85889830 r4:000000dc
[<80420ac8>] (filemap_get_pages) from [<804211d8>] (filemap_read+0xc8/0x4e0 mm/filemap.c:2458)
 r10:85889818 r9:860a9908 r8:805ff484 r7:85889830 r6:00000000 r5:85889818
 r4:85889830
[<80421110>] (filemap_read) from [<80421788>] (generic_file_read_iter+0x198/0x234 mm/filemap.c:2609)
 r10:00001000 r9:00000000 r8:805ff484 r7:00001000 r6:00000000 r5:85889818
 r4:85889830
[<804215f0>] (generic_file_read_iter) from [<805ff484>] (ext4_file_read_iter fs/ext4/file.c:130 [inline])
[<804215f0>] (generic_file_read_iter) from [<805ff484>] (ext4_file_read_iter+0x54/0x118 fs/ext4/file.c:113)
 r10:00001000 r9:00000000 r8:00001000 r7:85889888 r6:860a9908 r5:85889830
 r4:85889818
[<805ff430>] (ext4_file_read_iter) from [<804da4fc>] (__kernel_read+0x130/0x314 fs/read_write.c:454)
 r7:85889888 r6:00000000 r5:00000000 r4:85aa8dc0
[<804da3cc>] (__kernel_read) from [<8073774c>] (integrity_kernel_read+0x20/0x28 security/integrity/iint.c:191)
 r9:00000000 r8:00400000 r7:83685000 r6:00000000 r5:85aa8dc0 r4:000db000
[<8073772c>] (integrity_kernel_read) from [<8073a904>] (ima_calc_file_hash_tfm+0x178/0x228 security/integrity/ima/ima_crypto.c:484)
[<8073a78c>] (ima_calc_file_hash_tfm) from [<8073ae2c>] (ima_calc_file_shash security/integrity/ima/ima_crypto.c:515 [inline])
[<8073a78c>] (ima_calc_file_hash_tfm) from [<8073ae2c>] (ima_calc_file_hash+0x124/0x8b8 security/integrity/ima/ima_crypto.c:572)
 r10:860a99ac r9:00000000 r8:00000001 r7:00000013 r6:832fab90 r5:8570f900
 r4:85aa8dc0
[<8073ad08>] (ima_calc_file_hash) from [<8073c1a8>] (ima_collect_measurement+0x224/0x260 security/integrity/ima/ima_api.c:252)
 r10:860a3344 r9:860a99c4 r8:858796e8 r7:00000003 r6:00000000 r5:858796e8
 r4:85aa8dc0
[<8073bf84>] (ima_collect_measurement) from [<80739458>] (process_measurement+0x46c/0x7b0 security/integrity/ima/ima_main.c:330)
 r10:00000000 r9:860a99c4 r8:858796e8 r7:00000001 r6:00000001 r5:00000000
 r4:85aa8dc0
[<80738fec>] (process_measurement) from [<80739814>] (ima_file_check+0x78/0xa0 security/integrity/ima/ima_main.c:499)
 r10:00000000 r9:00000000 r8:85aa8dc0 r7:00000000 r6:00000000 r5:85889d48
 r4:00000006
[<8073979c>] (ima_file_check) from [<804ec878>] (do_open fs/namei.c:3367 [inline])
[<8073979c>] (ima_file_check) from [<804ec878>] (path_openat+0x20c/0x10f8 fs/namei.c:3498)
 r7:85889e58 r6:82a3c59c r5:85889f20 r4:00020002
[<804ec66c>] (path_openat) from [<804ef6ec>] (do_filp_open+0x7c/0x12c fs/namei.c:3525)
 r10:00000142 r9:85888000 r8:80200224 r7:00000001 r6:85889f20 r5:85889e58
 r4:85889f58
[<804ef670>] (do_filp_open) from [<804d7a6c>] (do_sys_openat2+0xa8/0x160 fs/open.c:1187)
 r7:ffffff9c r6:00000009 r5:8361b000 r4:85889f58
[<804d79c4>] (do_sys_openat2) from [<804d7f10>] (do_sys_open fs/open.c:1203 [inline])
[<804d79c4>] (do_sys_openat2) from [<804d7f10>] (__do_sys_openat fs/open.c:1219 [inline])
[<804d79c4>] (do_sys_openat2) from [<804d7f10>] (sys_openat+0xa4/0xcc fs/open.c:1214)
 r7:00000142 r6:00000000 r5:03af21c0 r4:ffffff9c
[<804d7e6c>] (sys_openat) from [<80200060>] (ret_fast_syscall+0x0/0x2c arch/arm/mm/proc-v7.S:64)
Exception stack(0x85889fa8 to 0x85889ff0)
9fa0:                   00000000 00000000 ffffff9c 03af21c0 000a0002 000001a4
9fc0: 00000000 00000000 00000000 00000142 00000005 7fff6f7f 00c000e0 00a1f128
9fe0: 03af21c3 00e4cb78 00012368 000b7738
 r5:00000000 r4:00000000
Dumping ftrace buffer:
   (ftrace buffer empty)
Rebooting in 1 seconds..


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

Re: [syzbot] kernel panic: corrupted stack end in openat

[Dmitry Vyukov]

On Tue, Mar 16, 2021 at 8:18 AM syzbot
<syzbot+0b06ef9b44d00d600183@syzkaller.appspotmail.com> wrote:
>
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit:    1e28eed1 Linux 5.12-rc3
> git tree:       upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=167535e6d00000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=e0cee1f53de33ca3
> dashboard link: https://syzkaller.appspot.com/bug?extid=0b06ef9b44d00d600183
> userspace arch: arm
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+0b06ef9b44d00d600183@syzkaller.appspotmail.com

+arm32 maintainer
I think this is a real stack overflow on arm32, the stack is indeed deep.

> Kernel panic - not syncing: corrupted stack end detected inside scheduler
> CPU: 0 PID: 3263 Comm: syz-fuzzer Not tainted 5.12.0-rc3-syzkaller #0
> Hardware name: ARM-Versatile Express
> Backtrace:
> [<81802700>] (dump_backtrace) from [<81802974>] (show_stack+0x18/0x1c arch/arm/kernel/traps.c:252)
>  r7:00000080 r6:60000093 r5:00000000 r4:82b58544
> [<8180295c>] (show_stack) from [<8180a048>] (__dump_stack lib/dump_stack.c:79 [inline])
> [<8180295c>] (show_stack) from [<8180a048>] (dump_stack+0xb8/0xe8 lib/dump_stack.c:120)
> [<81809f90>] (dump_stack) from [<81803508>] (panic+0x130/0x378 kernel/panic.c:231)
>  r7:81f4bdc0 r6:82a392a4 r5:00000000 r4:82c6b0d0
> [<818033d8>] (panic) from [<81820270>] (schedule_debug kernel/sched/core.c:4822 [inline])
> [<818033d8>] (panic) from [<81820270>] (__schedule+0xcb8/0xcc8 kernel/sched/core.c:4967)
>  r3:57ac6e9d r2:00040000 r1:81f5a53c r0:81f4bdc0
>  r7:00000001
> [<8181f5b8>] (__schedule) from [<8182046c>] (preempt_schedule_common+0x3c/0xac kernel/sched/core.c:5233)
>  r10:0000071f r9:ffefd000 r8:00000001 r7:81820510 r6:00000001 r5:81820510
>  r4:85888000
> [<81820430>] (preempt_schedule_common) from [<81820510>] (preempt_schedule+0x34/0x38 kernel/sched/core.c:5258)
>  r7:82c6a4e0 r6:00000001 r5:85888000 r4:df48d420
> [<818204dc>] (preempt_schedule) from [<8048c884>] (__kunmap_atomic include/linux/highmem-internal.h:114 [inline])
> [<818204dc>] (preempt_schedule) from [<8048c884>] (clear_highpage include/linux/highmem.h:204 [inline])
> [<818204dc>] (preempt_schedule) from [<8048c884>] (kernel_init_free_pages+0xc4/0xd0 mm/page_alloc.c:1212)
> [<8048c7c0>] (kernel_init_free_pages) from [<80492ce8>] (post_alloc_hook mm/page_alloc.c:2305 [inline])
> [<8048c7c0>] (kernel_init_free_pages) from [<80492ce8>] (prep_new_page mm/page_alloc.c:2311 [inline])
> [<8048c7c0>] (kernel_init_free_pages) from [<80492ce8>] (get_page_from_freelist+0x163c/0x1698 mm/page_alloc.c:3951)
>  r10:df48d3f0 r9:82bf89c0 r8:df48d3f0 r7:0000000b r6:00000002 r5:00000001
>  r4:df48d3f8 r3:00000001
> [<804916ac>] (get_page_from_freelist) from [<804933c8>] (__alloc_pages_nodemask+0x164/0x1850 mm/page_alloc.c:5001)
>  r10:00000000 r9:860a9a80 r8:00112cca r7:0000000b r6:00000081 r5:00000008
>  r4:00000000
> [<80493264>] (__alloc_pages_nodemask) from [<8042f0f8>] (__alloc_pages include/linux/gfp.h:525 [inline])
> [<80493264>] (__alloc_pages_nodemask) from [<8042f0f8>] (__alloc_pages_node include/linux/gfp.h:538 [inline])
> [<80493264>] (__alloc_pages_nodemask) from [<8042f0f8>] (alloc_pages_node include/linux/gfp.h:552 [inline])
> [<80493264>] (__alloc_pages_nodemask) from [<8042f0f8>] (alloc_pages include/linux/gfp.h:571 [inline])
> [<80493264>] (__alloc_pages_nodemask) from [<8042f0f8>] (__page_cache_alloc include/linux/pagemap.h:289 [inline])
> [<80493264>] (__alloc_pages_nodemask) from [<8042f0f8>] (page_cache_ra_unbounded+0xc4/0x294 mm/readahead.c:216)
>  r10:860a9a84 r9:860a9a80 r8:8588956c r7:0000000b r6:00000107 r5:85889688
>  r4:df48d3c0
> [<8042f034>] (page_cache_ra_unbounded) from [<8042f3c4>] (do_page_cache_ra+0xfc/0x150 mm/readahead.c:267)
>  r10:860a99ac r9:00000001 r8:00000020 r7:80000013 r6:8042f624 r5:85889688
>  r4:860a9908
> [<8042f2c8>] (do_page_cache_ra) from [<8042f624>] (ondemand_readahead+0x20c/0x47c mm/readahead.c:549)
>  r10:00000001 r9:000000fc r8:00000020 r7:000000dc r6:85889688 r5:00000000
>  r4:85aa8ea0
> [<8042f418>] (ondemand_readahead) from [<8042f958>] (page_cache_async_ra mm/readahead.c:607 [inline])
> [<8042f418>] (ondemand_readahead) from [<8042f958>] (page_cache_async_ra+0xc4/0x110 mm/readahead.c:581)
>  r10:85889818 r9:df4a4be0 r8:860a9a80 r7:85889714 r6:00000000 r5:85889688
>  r4:85aa8ea0
> [<8042f894>] (page_cache_async_ra) from [<80420d1c>] (page_cache_async_readahead include/linux/pagemap.h:863 [inline])
> [<8042f894>] (page_cache_async_ra) from [<80420d1c>] (filemap_readahead mm/filemap.c:2350 [inline])
> [<8042f894>] (page_cache_async_ra) from [<80420d1c>] (filemap_get_pages+0x254/0x648 mm/filemap.c:2391)
>  r7:85889714 r6:000000db r5:85889830 r4:000000dc
> [<80420ac8>] (filemap_get_pages) from [<804211d8>] (filemap_read+0xc8/0x4e0 mm/filemap.c:2458)
>  r10:85889818 r9:860a9908 r8:805ff484 r7:85889830 r6:00000000 r5:85889818
>  r4:85889830
> [<80421110>] (filemap_read) from [<80421788>] (generic_file_read_iter+0x198/0x234 mm/filemap.c:2609)
>  r10:00001000 r9:00000000 r8:805ff484 r7:00001000 r6:00000000 r5:85889818
>  r4:85889830
> [<804215f0>] (generic_file_read_iter) from [<805ff484>] (ext4_file_read_iter fs/ext4/file.c:130 [inline])
> [<804215f0>] (generic_file_read_iter) from [<805ff484>] (ext4_file_read_iter+0x54/0x118 fs/ext4/file.c:113)
>  r10:00001000 r9:00000000 r8:00001000 r7:85889888 r6:860a9908 r5:85889830
>  r4:85889818
> [<805ff430>] (ext4_file_read_iter) from [<804da4fc>] (__kernel_read+0x130/0x314 fs/read_write.c:454)
>  r7:85889888 r6:00000000 r5:00000000 r4:85aa8dc0
> [<804da3cc>] (__kernel_read) from [<8073774c>] (integrity_kernel_read+0x20/0x28 security/integrity/iint.c:191)
>  r9:00000000 r8:00400000 r7:83685000 r6:00000000 r5:85aa8dc0 r4:000db000
> [<8073772c>] (integrity_kernel_read) from [<8073a904>] (ima_calc_file_hash_tfm+0x178/0x228 security/integrity/ima/ima_crypto.c:484)
> [<8073a78c>] (ima_calc_file_hash_tfm) from [<8073ae2c>] (ima_calc_file_shash security/integrity/ima/ima_crypto.c:515 [inline])
> [<8073a78c>] (ima_calc_file_hash_tfm) from [<8073ae2c>] (ima_calc_file_hash+0x124/0x8b8 security/integrity/ima/ima_crypto.c:572)
>  r10:860a99ac r9:00000000 r8:00000001 r7:00000013 r6:832fab90 r5:8570f900
>  r4:85aa8dc0
> [<8073ad08>] (ima_calc_file_hash) from [<8073c1a8>] (ima_collect_measurement+0x224/0x260 security/integrity/ima/ima_api.c:252)
>  r10:860a3344 r9:860a99c4 r8:858796e8 r7:00000003 r6:00000000 r5:858796e8
>  r4:85aa8dc0
> [<8073bf84>] (ima_collect_measurement) from [<80739458>] (process_measurement+0x46c/0x7b0 security/integrity/ima/ima_main.c:330)
>  r10:00000000 r9:860a99c4 r8:858796e8 r7:00000001 r6:00000001 r5:00000000
>  r4:85aa8dc0
> [<80738fec>] (process_measurement) from [<80739814>] (ima_file_check+0x78/0xa0 security/integrity/ima/ima_main.c:499)
>  r10:00000000 r9:00000000 r8:85aa8dc0 r7:00000000 r6:00000000 r5:85889d48
>  r4:00000006
> [<8073979c>] (ima_file_check) from [<804ec878>] (do_open fs/namei.c:3367 [inline])
> [<8073979c>] (ima_file_check) from [<804ec878>] (path_openat+0x20c/0x10f8 fs/namei.c:3498)
>  r7:85889e58 r6:82a3c59c r5:85889f20 r4:00020002
> [<804ec66c>] (path_openat) from [<804ef6ec>] (do_filp_open+0x7c/0x12c fs/namei.c:3525)
>  r10:00000142 r9:85888000 r8:80200224 r7:00000001 r6:85889f20 r5:85889e58
>  r4:85889f58
> [<804ef670>] (do_filp_open) from [<804d7a6c>] (do_sys_openat2+0xa8/0x160 fs/open.c:1187)
>  r7:ffffff9c r6:00000009 r5:8361b000 r4:85889f58
> [<804d79c4>] (do_sys_openat2) from [<804d7f10>] (do_sys_open fs/open.c:1203 [inline])
> [<804d79c4>] (do_sys_openat2) from [<804d7f10>] (__do_sys_openat fs/open.c:1219 [inline])
> [<804d79c4>] (do_sys_openat2) from [<804d7f10>] (sys_openat+0xa4/0xcc fs/open.c:1214)
>  r7:00000142 r6:00000000 r5:03af21c0 r4:ffffff9c
> [<804d7e6c>] (sys_openat) from [<80200060>] (ret_fast_syscall+0x0/0x2c arch/arm/mm/proc-v7.S:64)
> Exception stack(0x85889fa8 to 0x85889ff0)
> 9fa0:                   00000000 00000000 ffffff9c 03af21c0 000a0002 000001a4
> 9fc0: 00000000 00000000 00000000 00000142 00000005 7fff6f7f 00c000e0 00a1f128
> 9fe0: 03af21c3 00e4cb78 00012368 000b7738
>  r5:00000000 r4:00000000
> Dumping ftrace buffer:
>    (ftrace buffer empty)
> Rebooting in 1 seconds..
>
>
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at syzkaller@googlegroups.com.
>
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/00000000000069802205bda22b7f%40google.com.

Re: [syzbot] kernel panic: corrupted stack end in openat

[Russell King - ARM Linux admin]

On Tue, Mar 16, 2021 at 08:59:17AM +0100, Dmitry Vyukov wrote:
> On Tue, Mar 16, 2021 at 8:18 AM syzbot
> <syzbot+0b06ef9b44d00d600183@syzkaller.appspotmail.com> wrote:
> >
> > Hello,
> >
> > syzbot found the following issue on:
> >
> > HEAD commit:    1e28eed1 Linux 5.12-rc3
> > git tree:       upstream
> > console output: https://syzkaller.appspot.com/x/log.txt?x=167535e6d00000
> > kernel config:  https://syzkaller.appspot.com/x/.config?x=e0cee1f53de33ca3
> > dashboard link: https://syzkaller.appspot.com/bug?extid=0b06ef9b44d00d600183
> > userspace arch: arm
> >
> > Unfortunately, I don't have any reproducer for this issue yet.
> >
> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > Reported-by: syzbot+0b06ef9b44d00d600183@syzkaller.appspotmail.com
> 
> +arm32 maintainer
> I think this is a real stack overflow on arm32, the stack is indeed deep.

There's no way to know for sure because there's no indication of the
stack pointer in this, so we don't know how much space remains.
Therefore we don't know whether this is something in the dumped
path, or an interrupt causing it.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: [syzbot] kernel panic: corrupted stack end in openat

[Dmitry Vyukov]

On Tue, Mar 16, 2021 at 10:24 AM Russell King - ARM Linux admin
<linux@armlinux.org.uk> wrote:
>
> On Tue, Mar 16, 2021 at 08:59:17AM +0100, Dmitry Vyukov wrote:
> > On Tue, Mar 16, 2021 at 8:18 AM syzbot
> > <syzbot+0b06ef9b44d00d600183@syzkaller.appspotmail.com> wrote:
> > >
> > > Hello,
> > >
> > > syzbot found the following issue on:
> > >
> > > HEAD commit:    1e28eed1 Linux 5.12-rc3
> > > git tree:       upstream
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=167535e6d00000
> > > kernel config:  https://syzkaller.appspot.com/x/.config?x=e0cee1f53de33ca3
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=0b06ef9b44d00d600183
> > > userspace arch: arm
> > >
> > > Unfortunately, I don't have any reproducer for this issue yet.
> > >
> > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > Reported-by: syzbot+0b06ef9b44d00d600183@syzkaller.appspotmail.com
> >
> > +arm32 maintainer
> > I think this is a real stack overflow on arm32, the stack is indeed deep.
>
> There's no way to know for sure because there's no indication of the
> stack pointer in this, so we don't know how much space remains.
> Therefore we don't know whether this is something in the dumped
> path, or an interrupt causing it.

Agree, to know for sure we would need support for VMAP_STACK.
But do we really need to know it? If it's an interrupt on top, it does
not make any difference?

Re: [syzbot] kernel panic: corrupted stack end in openat

[Arnd Bergmann]

On Tue, Mar 16, 2021 at 8:59 AM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Tue, Mar 16, 2021 at 8:18 AM syzbot
> <syzbot+0b06ef9b44d00d600183@syzkaller.appspotmail.com> wrote:
> >
> > Hello,
> >
> > syzbot found the following issue on:
> >
> > HEAD commit:    1e28eed1 Linux 5.12-rc3
> > git tree:       upstream
> > console output: https://syzkaller.appspot.com/x/log.txt?x=167535e6d00000
> > kernel config:  https://syzkaller.appspot.com/x/.config?x=e0cee1f53de33ca3
> > dashboard link: https://syzkaller.appspot.com/bug?extid=0b06ef9b44d00d600183
> > userspace arch: arm
> >
> > Unfortunately, I don't have any reproducer for this issue yet.
> >
> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > Reported-by: syzbot+0b06ef9b44d00d600183@syzkaller.appspotmail.com
>
> +arm32 maintainer
> I think this is a real stack overflow on arm32, the stack is indeed deep.

Nice find. I see there was already a second report, so it seems to be
reproducible as well.
If you are able to trigger this reliably, you could try printing the frame
pointer while unwinding to see what is actually going on:

--- a/arch/arm/kernel/traps.c
+++ b/arch/arm/kernel/traps.c
@@ -68,8 +68,8 @@ void dump_backtrace_entry(unsigned long where,
unsigned long from,
        unsigned long end = frame + 4 + sizeof(struct pt_regs);

 #ifdef CONFIG_KALLSYMS
-       printk("%s[<%08lx>] (%ps) from [<%08lx>] (%pS)\n",
-               loglvl, where, (void *)where, from, (void *)from);
+       printk("%s[<%08lx>] (%ps) from [<%08lx>] (%pS), frame %08lx\n",
+               loglvl, where, (void *)where, from, (void *)from, frame);
 #else
        printk("%sFunction entered at [<%08lx>] from [<%08lx>]\n",
                loglvl, where, from);

If that doesn't help, I could have a look at the binary to see which
functions in the call chain take a lot of stack space, if any.

Which exact compiler version do you use for building these
kernels? I can try doing a build with the same commit and config.

This one function is one that I have seen before when looking at build
warnings with KASAN:

> > [<8073772c>] (integrity_kernel_read) from [<8073a904>] (ima_calc_file_hash_tfm+0x178/0x228 security/integrity/ima/ima_crypto.c:484)
> > [<8073a78c>] (ima_calc_file_hash_tfm) from [<8073ae2c>] (ima_calc_file_shash security/integrity/ima/ima_crypto.c:515 [inline])
> > [<8073a78c>] (ima_calc_file_hash_tfm) from [<8073ae2c>] (ima_calc_file_hash+0x124/0x8b8 security/integrity/ima/ima_crypto.c:572)

ima_calc_file_hash_tfm() has a SHASH_DESC_ON_STACK(), which by itself can
use up 512 bytes, but KASAN sometimes triples this number. However, I see
you do not actually have KASAN enabled, so there is probably more to it.

       Arnd

Re: [syzbot] kernel panic: corrupted stack end in openat

[Dmitry Vyukov]

On Tue, Mar 16, 2021 at 11:02 AM Arnd Bergmann <arnd@arndb.de> wrote:
> > On Tue, Mar 16, 2021 at 8:18 AM syzbot
> > <syzbot+0b06ef9b44d00d600183@syzkaller.appspotmail.com> wrote:
> > >
> > > Hello,
> > >
> > > syzbot found the following issue on:
> > >
> > > HEAD commit:    1e28eed1 Linux 5.12-rc3
> > > git tree:       upstream
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=167535e6d00000
> > > kernel config:  https://syzkaller.appspot.com/x/.config?x=e0cee1f53de33ca3
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=0b06ef9b44d00d600183
> > > userspace arch: arm
> > >
> > > Unfortunately, I don't have any reproducer for this issue yet.
> > >
> > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > Reported-by: syzbot+0b06ef9b44d00d600183@syzkaller.appspotmail.com
> >
> > +arm32 maintainer
> > I think this is a real stack overflow on arm32, the stack is indeed deep.
>
> Nice find. I see there was already a second report, so it seems to be
> reproducible as well.
> If you are able to trigger this reliably, you could try printing the frame
> pointer while unwinding to see what is actually going on:
>
> --- a/arch/arm/kernel/traps.c
> +++ b/arch/arm/kernel/traps.c
> @@ -68,8 +68,8 @@ void dump_backtrace_entry(unsigned long where,
> unsigned long from,
>         unsigned long end = frame + 4 + sizeof(struct pt_regs);
>
>  #ifdef CONFIG_KALLSYMS
> -       printk("%s[<%08lx>] (%ps) from [<%08lx>] (%pS)\n",
> -               loglvl, where, (void *)where, from, (void *)from);
> +       printk("%s[<%08lx>] (%ps) from [<%08lx>] (%pS), frame %08lx\n",
> +               loglvl, where, (void *)where, from, (void *)from, frame);
>  #else
>         printk("%sFunction entered at [<%08lx>] from [<%08lx>]\n",
>                 loglvl, where, from);
>
> If that doesn't help, I could have a look at the binary to see which
> functions in the call chain take a lot of stack space, if any.
>
> Which exact compiler version do you use for building these
> kernels? I can try doing a build with the same commit and config.
>
> This one function is one that I have seen before when looking at build
> warnings with KASAN:
>
> > > [<8073772c>] (integrity_kernel_read) from [<8073a904>] (ima_calc_file_hash_tfm+0x178/0x228 security/integrity/ima/ima_crypto.c:484)
> > > [<8073a78c>] (ima_calc_file_hash_tfm) from [<8073ae2c>] (ima_calc_file_shash security/integrity/ima/ima_crypto.c:515 [inline])
> > > [<8073a78c>] (ima_calc_file_hash_tfm) from [<8073ae2c>] (ima_calc_file_hash+0x124/0x8b8 security/integrity/ima/ima_crypto.c:572)
>
> ima_calc_file_hash_tfm() has a SHASH_DESC_ON_STACK(), which by itself can
> use up 512 bytes, but KASAN sometimes triples this number. However, I see
> you do not actually have KASAN enabled, so there is probably more to it.

The compiler is gcc version 10.2.1 20210110 (Debian 10.2.1-6)
It's available in gcr.io/syzkaller/syzbot container.
(syzbot should have been provided the compiler version, something
broke, I've filed https://github.com/google/syzkaller/issues/2498 for
this)

Yes, KASAN is not enabled on arm32 for now.

Re printing FP, syzbot does not use custom patches:
http://bit.do/syzbot#no-custom-patches
But this does not seem to be syzbot-specific. It seems that any arm32
stack overflow report will be unactionable, so I think it would be
useful to include this into the mainline kernel to make overflow
reports useful for everybody (and for syzbot as a side effect).

Re: [syzbot] kernel panic: corrupted stack end in openat

[Ard Biesheuvel]

On Tue, 16 Mar 2021 at 11:04, Arnd Bergmann <arnd@arndb.de> wrote:
>
> On Tue, Mar 16, 2021 at 8:59 AM Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> > On Tue, Mar 16, 2021 at 8:18 AM syzbot
> > <syzbot+0b06ef9b44d00d600183@syzkaller.appspotmail.com> wrote:
> > >
> > > Hello,
> > >
> > > syzbot found the following issue on:
> > >
> > > HEAD commit:    1e28eed1 Linux 5.12-rc3
> > > git tree:       upstream
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=167535e6d00000
> > > kernel config:  https://syzkaller.appspot.com/x/.config?x=e0cee1f53de33ca3
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=0b06ef9b44d00d600183
> > > userspace arch: arm
> > >
> > > Unfortunately, I don't have any reproducer for this issue yet.
> > >
> > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > Reported-by: syzbot+0b06ef9b44d00d600183@syzkaller.appspotmail.com
> >
> > +arm32 maintainer
> > I think this is a real stack overflow on arm32, the stack is indeed deep.
>
> Nice find. I see there was already a second report, so it seems to be
> reproducible as well.
> If you are able to trigger this reliably, you could try printing the frame
> pointer while unwinding to see what is actually going on:
>
> --- a/arch/arm/kernel/traps.c
> +++ b/arch/arm/kernel/traps.c
> @@ -68,8 +68,8 @@ void dump_backtrace_entry(unsigned long where,
> unsigned long from,
>         unsigned long end = frame + 4 + sizeof(struct pt_regs);
>
>  #ifdef CONFIG_KALLSYMS
> -       printk("%s[<%08lx>] (%ps) from [<%08lx>] (%pS)\n",
> -               loglvl, where, (void *)where, from, (void *)from);
> +       printk("%s[<%08lx>] (%ps) from [<%08lx>] (%pS), frame %08lx\n",
> +               loglvl, where, (void *)where, from, (void *)from, frame);
>  #else
>         printk("%sFunction entered at [<%08lx>] from [<%08lx>]\n",
>                 loglvl, where, from);
>
> If that doesn't help, I could have a look at the binary to see which
> functions in the call chain take a lot of stack space, if any.
>
> Which exact compiler version do you use for building these
> kernels? I can try doing a build with the same commit and config.
>
> This one function is one that I have seen before when looking at build
> warnings with KASAN:
>
> > > [<8073772c>] (integrity_kernel_read) from [<8073a904>] (ima_calc_file_hash_tfm+0x178/0x228 security/integrity/ima/ima_crypto.c:484)
> > > [<8073a78c>] (ima_calc_file_hash_tfm) from [<8073ae2c>] (ima_calc_file_shash security/integrity/ima/ima_crypto.c:515 [inline])
> > > [<8073a78c>] (ima_calc_file_hash_tfm) from [<8073ae2c>] (ima_calc_file_hash+0x124/0x8b8 security/integrity/ima/ima_crypto.c:572)
>
> ima_calc_file_hash_tfm() has a SHASH_DESC_ON_STACK(), which by itself can
> use up 512 bytes, but KASAN sometimes triples this number. However, I see
> you do not actually have KASAN enabled, so there is probably more to it.
>

FYI, as an aside, the SHASH_DESC_ON_STACK() issue was fixed in

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=660d2062190db131d2feaf19914e90f868fe285c

(note that the size of SHASH_DESC_ON_STACK() accounts for two struct
shash_desc instances)

Re: [syzbot] kernel panic: corrupted stack end in openat

[Arnd Bergmann]

On Tue, Mar 16, 2021 at 11:17 AM Dmitry Vyukov <dvyukov@google.com> wrote:
> On Tue, Mar 16, 2021 at 11:02 AM Arnd Bergmann <arnd@arndb.de> wrote:
> > > On Tue, Mar 16, 2021 at 8:18 AM syzbot
>
> > > > [<8073772c>] (integrity_kernel_read) from [<8073a904>] (ima_calc_file_hash_tfm+0x178/0x228 security/integrity/ima/ima_crypto.c:484)
> > > > [<8073a78c>] (ima_calc_file_hash_tfm) from [<8073ae2c>] (ima_calc_file_shash security/integrity/ima/ima_crypto.c:515 [inline])
> > > > [<8073a78c>] (ima_calc_file_hash_tfm) from [<8073ae2c>] (ima_calc_file_hash+0x124/0x8b8 security/integrity/ima/ima_crypto.c:572)
> >
> > ima_calc_file_hash_tfm() has a SHASH_DESC_ON_STACK(), which by itself can
> > use up 512 bytes, but KASAN sometimes triples this number. However, I see
> > you do not actually have KASAN enabled, so there is probably more to it.
>
> The compiler is gcc version 10.2.1 20210110 (Debian 10.2.1-6)

Ok, building with Ubuntu 10.2.1-1ubuntu1 20201207 locally, that's
the closest I have installed, and I think the Debian and Ubuntu versions
are generally quite close in case of gcc since they are maintained by
the same packagers.

I see ima_calc_field_array_hash_tfm() shows up as one of the larger
stack users, but not alarmingly high:
../security/integrity/ima/ima_crypto.c: In function
‘ima_calc_field_array_hash_tfm’:
../security/integrity/ima/ima_crypto.c:624:1: warning: the frame size
of 664 bytes is larger than 600 bytes [-Wframe-larger-than=]
none of the other functions from the call chain have more than 600 bytes
in this combination of config/compiler/sourcetree.

In combination, I don't get to more than ~2300 bytes:

    [<818033d8>] (panic)
 52 [<8181f5b8>] (__schedule)
  0 [<81820430>] (preempt_schedule_common)
  0 [<818204dc>] (preempt_schedule)
  0 [<8048c7c0>] (kernel_init_free_pages)
148 [<804916ac>] (get_page_from_freelist
212 [<80493264>] (__alloc_pages_nodemask)
 44 [<8042f034>] (page_cache_ra_unbounded)
 36 [<8042f2c8>] (do_page_cache_ra)
 28 [<8042f418>] (ondemand_readahead)
  0 [<8042f894>] (page_cache_async_ra)
 68 [<80420ac8>] (filemap_get_pages)
120 [<80421110>] (filemap_read)
 36 [<804215f0>] (generic_file_read_iter)
  8 [<805ff430>] (ext4_file_read_iter)
 96 [<804da3cc>] (__kernel_read)
  8 [<8073772c>] (integrity_kernel_read)
412 [<8073a78c>] (ima_calc_file_hash_tfm)
164 [<8073ad08>] (ima_calc_file_hash)
106 [<8073bf84>] (ima_collect_measurement)
332 [<80738fec>] (process_measurement)
 24 [<8073979c>] (ima_file_check)
172 [<804ec66c>] (path_openat)
152 [<804ef670>] (do_filp_open)
 40 [<804d79c4>] (do_sys_openat2)

> Re printing FP, syzbot does not use custom patches:
> http://bit.do/syzbot#no-custom-patches
> But this does not seem to be syzbot-specific. It seems that any arm32
> stack overflow report will be unactionable, so I think it would be
> useful to include this into the mainline kernel to make overflow
> reports useful for everybody (and for syzbot as a side effect).

ok.

       Arnd

Re: [syzbot] kernel panic: corrupted stack end in openat

[Russell King - ARM Linux admin]

On Tue, Mar 16, 2021 at 04:44:45PM +0100, Arnd Bergmann wrote:
> On Tue, Mar 16, 2021 at 11:17 AM Dmitry Vyukov <dvyukov@google.com> wrote:
> > On Tue, Mar 16, 2021 at 11:02 AM Arnd Bergmann <arnd@arndb.de> wrote:
> > > > On Tue, Mar 16, 2021 at 8:18 AM syzbot
> >
> > > > > [<8073772c>] (integrity_kernel_read) from [<8073a904>] (ima_calc_file_hash_tfm+0x178/0x228 security/integrity/ima/ima_crypto.c:484)
> > > > > [<8073a78c>] (ima_calc_file_hash_tfm) from [<8073ae2c>] (ima_calc_file_shash security/integrity/ima/ima_crypto.c:515 [inline])
> > > > > [<8073a78c>] (ima_calc_file_hash_tfm) from [<8073ae2c>] (ima_calc_file_hash+0x124/0x8b8 security/integrity/ima/ima_crypto.c:572)
> > >
> > > ima_calc_file_hash_tfm() has a SHASH_DESC_ON_STACK(), which by itself can
> > > use up 512 bytes, but KASAN sometimes triples this number. However, I see
> > > you do not actually have KASAN enabled, so there is probably more to it.
> >
> > The compiler is gcc version 10.2.1 20210110 (Debian 10.2.1-6)
> 
> Ok, building with Ubuntu 10.2.1-1ubuntu1 20201207 locally, that's
> the closest I have installed, and I think the Debian and Ubuntu versions
> are generally quite close in case of gcc since they are maintained by
> the same packagers.
> 
> I see ima_calc_field_array_hash_tfm() shows up as one of the larger
> stack users, but not alarmingly high:
> ../security/integrity/ima/ima_crypto.c: In function
> ‘ima_calc_field_array_hash_tfm’:
> ../security/integrity/ima/ima_crypto.c:624:1: warning: the frame size
> of 664 bytes is larger than 600 bytes [-Wframe-larger-than=]
> none of the other functions from the call chain have more than 600 bytes
> in this combination of config/compiler/sourcetree.
> 
> In combination, I don't get to more than ~2300 bytes:

... which shouldn't be a problem - that's just over 1/4 of the stack
space. Could it be the syzbot's gcc is doing something weird and
inflating the stack frames?

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: [syzbot] kernel panic: corrupted stack end in openat

[Dmitry Vyukov]

On Tue, Mar 16, 2021 at 4:51 PM Russell King - ARM Linux admin
<linux@armlinux.org.uk> wrote:
>
> On Tue, Mar 16, 2021 at 04:44:45PM +0100, Arnd Bergmann wrote:
> > On Tue, Mar 16, 2021 at 11:17 AM Dmitry Vyukov <dvyukov@google.com> wrote:
> > > On Tue, Mar 16, 2021 at 11:02 AM Arnd Bergmann <arnd@arndb.de> wrote:
> > > > > On Tue, Mar 16, 2021 at 8:18 AM syzbot
> > >
> > > > > > [<8073772c>] (integrity_kernel_read) from [<8073a904>] (ima_calc_file_hash_tfm+0x178/0x228 security/integrity/ima/ima_crypto.c:484)
> > > > > > [<8073a78c>] (ima_calc_file_hash_tfm) from [<8073ae2c>] (ima_calc_file_shash security/integrity/ima/ima_crypto.c:515 [inline])
> > > > > > [<8073a78c>] (ima_calc_file_hash_tfm) from [<8073ae2c>] (ima_calc_file_hash+0x124/0x8b8 security/integrity/ima/ima_crypto.c:572)
> > > >
> > > > ima_calc_file_hash_tfm() has a SHASH_DESC_ON_STACK(), which by itself can
> > > > use up 512 bytes, but KASAN sometimes triples this number. However, I see
> > > > you do not actually have KASAN enabled, so there is probably more to it.
> > >
> > > The compiler is gcc version 10.2.1 20210110 (Debian 10.2.1-6)
> >
> > Ok, building with Ubuntu 10.2.1-1ubuntu1 20201207 locally, that's
> > the closest I have installed, and I think the Debian and Ubuntu versions
> > are generally quite close in case of gcc since they are maintained by
> > the same packagers.
> >
> > I see ima_calc_field_array_hash_tfm() shows up as one of the larger
> > stack users, but not alarmingly high:
> > ../security/integrity/ima/ima_crypto.c: In function
> > ‘ima_calc_field_array_hash_tfm’:
> > ../security/integrity/ima/ima_crypto.c:624:1: warning: the frame size
> > of 664 bytes is larger than 600 bytes [-Wframe-larger-than=]
> > none of the other functions from the call chain have more than 600 bytes
> > in this combination of config/compiler/sourcetree.
> >
> > In combination, I don't get to more than ~2300 bytes:
>
> ... which shouldn't be a problem - that's just over 1/4 of the stack
> space. Could it be the syzbot's gcc is doing something weird and
> inflating the stack frames?

It's just a stock Debian-provided gcc. Which I would assume also just
a plain gcc.

Re: [syzbot] kernel panic: corrupted stack end in openat

[Arnd Bergmann]

On Tue, Mar 16, 2021 at 4:51 PM Russell King - ARM Linux admin
<linux@armlinux.org.uk> wrote:
> On Tue, Mar 16, 2021 at 04:44:45PM +0100, Arnd Bergmann wrote:
> > On Tue, Mar 16, 2021 at 11:17 AM Dmitry Vyukov <dvyukov@google.com> wrote:
> > > The compiler is gcc version 10.2.1 20210110 (Debian 10.2.1-6)
> >
> > Ok, building with Ubuntu 10.2.1-1ubuntu1 20201207 locally, that's
> > the closest I have installed, and I think the Debian and Ubuntu versions
> > are generally quite close in case of gcc since they are maintained by
> > the same packagers.
>
> ... which shouldn't be a problem - that's just over 1/4 of the stack
> space. Could it be the syzbot's gcc is doing something weird and
> inflating the stack frames?

It's possible, I think that's really unlikely given that it's just Debian's
gcc, which is as close to mainline as the version I was using.

Uwe's DEBUG_STACKOVERFLOW patch from a while ago might
help if this was the problem though:
https://lore.kernel.org/linux-arm-kernel/20200108082913.29710-1-u.kleine-koenig@pengutronix.de/

My best guess is something going wrong in the interrupt
that triggered the preempt_schedule() which ended up calling
task_stack_end_corrupted() in schedule_debug(), as you suggested
earlier.

       Arnd

Re: [syzbot] kernel panic: corrupted stack end in openat

[Dmitry Vyukov]

On Tue, Mar 16, 2021 at 5:03 PM Arnd Bergmann <arnd@arndb.de> wrote:
>
> On Tue, Mar 16, 2021 at 4:51 PM Russell King - ARM Linux admin
> <linux@armlinux.org.uk> wrote:
> > On Tue, Mar 16, 2021 at 04:44:45PM +0100, Arnd Bergmann wrote:
> > > On Tue, Mar 16, 2021 at 11:17 AM Dmitry Vyukov <dvyukov@google.com> wrote:
> > > > The compiler is gcc version 10.2.1 20210110 (Debian 10.2.1-6)
> > >
> > > Ok, building with Ubuntu 10.2.1-1ubuntu1 20201207 locally, that's
> > > the closest I have installed, and I think the Debian and Ubuntu versions
> > > are generally quite close in case of gcc since they are maintained by
> > > the same packagers.
> >
> > ... which shouldn't be a problem - that's just over 1/4 of the stack
> > space. Could it be the syzbot's gcc is doing something weird and
> > inflating the stack frames?
>
> It's possible, I think that's really unlikely given that it's just Debian's
> gcc, which is as close to mainline as the version I was using.
>
> Uwe's DEBUG_STACKOVERFLOW patch from a while ago might
> help if this was the problem though:
> https://lore.kernel.org/linux-arm-kernel/20200108082913.29710-1-u.kleine-koenig@pengutronix.de/
>
> My best guess is something going wrong in the interrupt
> that triggered the preempt_schedule() which ended up calling
> task_stack_end_corrupted() in schedule_debug(), as you suggested
> earlier.

FWIW I see slightly larger frames with the config:

073ab64 <ima_calc_field_array_hash_tfm>:
8073ab64:       e1a0c00d        mov     ip, sp
8073ab68:       e92ddff0        push    {r4, r5, r6, r7, r8, r9, sl,
fp, ip, lr, pc}
8073ab6c:       e24cb004        sub     fp, ip, #4
8073ab70:       e24ddfa7        sub     sp, sp, #668    ; 0x29c

page_alloc can also do reclaim, I had the impression that reclaim can
be quite heavy-weight in all respects.

Re: [syzbot] kernel panic: corrupted stack end in openat

[Arnd Bergmann]

On Tue, Mar 16, 2021 at 5:13 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Tue, Mar 16, 2021 at 5:03 PM Arnd Bergmann <arnd@arndb.de> wrote:
> >
> > On Tue, Mar 16, 2021 at 4:51 PM Russell King - ARM Linux admin
> > <linux@armlinux.org.uk> wrote:
> > > On Tue, Mar 16, 2021 at 04:44:45PM +0100, Arnd Bergmann wrote:
> > > > On Tue, Mar 16, 2021 at 11:17 AM Dmitry Vyukov <dvyukov@google.com> wrote:
> > > > > The compiler is gcc version 10.2.1 20210110 (Debian 10.2.1-6)
> > > >
> > > > Ok, building with Ubuntu 10.2.1-1ubuntu1 20201207 locally, that's
> > > > the closest I have installed, and I think the Debian and Ubuntu versions
> > > > are generally quite close in case of gcc since they are maintained by
> > > > the same packagers.
> > >
> > > ... which shouldn't be a problem - that's just over 1/4 of the stack
> > > space. Could it be the syzbot's gcc is doing something weird and
> > > inflating the stack frames?
> >
> > It's possible, I think that's really unlikely given that it's just Debian's
> > gcc, which is as close to mainline as the version I was using.
> >
> > Uwe's DEBUG_STACKOVERFLOW patch from a while ago might
> > help if this was the problem though:
> > https://lore.kernel.org/linux-arm-kernel/20200108082913.29710-1-u.kleine-koenig@pengutronix.de/
> >
> > My best guess is something going wrong in the interrupt
> > that triggered the preempt_schedule() which ended up calling
> > task_stack_end_corrupted() in schedule_debug(), as you suggested
> > earlier.
>
> FWIW I see slightly larger frames with the config:
>
> 073ab64 <ima_calc_field_array_hash_tfm>:
> 8073ab64:       e1a0c00d        mov     ip, sp
> 8073ab68:       e92ddff0        push    {r4, r5, r6, r7, r8, r9, sl,
> fp, ip, lr, pc}
> 8073ab6c:       e24cb004        sub     fp, ip, #4
> 8073ab70:       e24ddfa7        sub     sp, sp, #668    ; 0x29c

Yes, this is the one that the compiler complained about when warning
for stack over 600 bytes. It's not called in this call chain though.

> page_alloc can also do reclaim, I had the impression that reclaim can
> be quite heavy-weight in all respects.

Yes, that is another possibility. What writable file systems or swap
do you normally have mounted that it could be writing to, and on
what storage device?

       Arnd

Re: [syzbot] kernel panic: corrupted stack end in openat

[Dmitry Vyukov]

On Tue, Mar 16, 2021 at 5:28 PM Arnd Bergmann <arnd@arndb.de> wrote:
>
> On Tue, Mar 16, 2021 at 5:13 PM Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> > On Tue, Mar 16, 2021 at 5:03 PM Arnd Bergmann <arnd@arndb.de> wrote:
> > >
> > > On Tue, Mar 16, 2021 at 4:51 PM Russell King - ARM Linux admin
> > > <linux@armlinux.org.uk> wrote:
> > > > On Tue, Mar 16, 2021 at 04:44:45PM +0100, Arnd Bergmann wrote:
> > > > > On Tue, Mar 16, 2021 at 11:17 AM Dmitry Vyukov <dvyukov@google.com> wrote:
> > > > > > The compiler is gcc version 10.2.1 20210110 (Debian 10.2.1-6)
> > > > >
> > > > > Ok, building with Ubuntu 10.2.1-1ubuntu1 20201207 locally, that's
> > > > > the closest I have installed, and I think the Debian and Ubuntu versions
> > > > > are generally quite close in case of gcc since they are maintained by
> > > > > the same packagers.
> > > >
> > > > ... which shouldn't be a problem - that's just over 1/4 of the stack
> > > > space. Could it be the syzbot's gcc is doing something weird and
> > > > inflating the stack frames?
> > >
> > > It's possible, I think that's really unlikely given that it's just Debian's
> > > gcc, which is as close to mainline as the version I was using.
> > >
> > > Uwe's DEBUG_STACKOVERFLOW patch from a while ago might
> > > help if this was the problem though:
> > > https://lore.kernel.org/linux-arm-kernel/20200108082913.29710-1-u.kleine-koenig@pengutronix.de/
> > >
> > > My best guess is something going wrong in the interrupt
> > > that triggered the preempt_schedule() which ended up calling
> > > task_stack_end_corrupted() in schedule_debug(), as you suggested
> > > earlier.
> >
> > FWIW I see slightly larger frames with the config:
> >
> > 073ab64 <ima_calc_field_array_hash_tfm>:
> > 8073ab64:       e1a0c00d        mov     ip, sp
> > 8073ab68:       e92ddff0        push    {r4, r5, r6, r7, r8, r9, sl,
> > fp, ip, lr, pc}
> > 8073ab6c:       e24cb004        sub     fp, ip, #4
> > 8073ab70:       e24ddfa7        sub     sp, sp, #668    ; 0x29c
>
> Yes, this is the one that the compiler complained about when warning
> for stack over 600 bytes. It's not called in this call chain though.
>
> > page_alloc can also do reclaim, I had the impression that reclaim can
> > be quite heavy-weight in all respects.
>
> Yes, that is another possibility. What writable file systems or swap
> do you normally have mounted that it could be writing to, and on
> what storage device?

The root fs is ext4 on virtio-blk.

There are also several dozens of shrinkers that can be called during reclaim:
https://elixir.bootlin.com/linux/latest/C/ident/unregister_shrinker

Re: [syzbot] kernel panic: corrupted stack end in openat

[Arnd Bergmann]

On Wed, Mar 17, 2021 at 8:52 AM Dmitry Vyukov <dvyukov@google.com> wrote:
> On Tue, Mar 16, 2021 at 5:28 PM Arnd Bergmann <arnd@arndb.de> wrote:
> > On Tue, Mar 16, 2021 at 5:13 PM Dmitry Vyukov <dvyukov@google.com> wrote:
> > > On Tue, Mar 16, 2021 at 5:03 PM Arnd Bergmann <arnd@arndb.de> wrote:
> > > > On Tue, Mar 16, 2021 at 4:51 PM Russell King - ARM Linux admin
> > > > <linux@armlinux.org.uk> wrote:
> > > > > On Tue, Mar 16, 2021 at 04:44:45PM +0100, Arnd Bergmann wrote:
> > > > > > On Tue, Mar 16, 2021 at 11:17 AM Dmitry Vyukov <dvyukov@google.com> wrote:
> > > > > > > The compiler is gcc version 10.2.1 20210110 (Debian 10.2.1-6)
> > > > > >
> > > > > > Ok, building with Ubuntu 10.2.1-1ubuntu1 20201207 locally, that's
> > > > > > the closest I have installed, and I think the Debian and Ubuntu versions
> > > > > > are generally quite close in case of gcc since they are maintained by
> > > > > > the same packagers.
> > > > >
> > > > > ... which shouldn't be a problem - that's just over 1/4 of the stack
> > > > > space. Could it be the syzbot's gcc is doing something weird and
> > > > > inflating the stack frames?
> > > >
> > > > It's possible, I think that's really unlikely given that it's just Debian's
> > > > gcc, which is as close to mainline as the version I was using.
> > > >
> > > > Uwe's DEBUG_STACKOVERFLOW patch from a while ago might
> > > > help if this was the problem though:
> > > > https://lore.kernel.org/linux-arm-kernel/20200108082913.29710-1-u.kleine-koenig@pengutronix.de/
> > > >
> > > > My best guess is something going wrong in the interrupt
> > > > that triggered the preempt_schedule() which ended up calling
> > > > task_stack_end_corrupted() in schedule_debug(), as you suggested
> > > > earlier.
> > >
> > > FWIW I see slightly larger frames with the config:
> > >
> > > 073ab64 <ima_calc_field_array_hash_tfm>:
> > > 8073ab64:       e1a0c00d        mov     ip, sp
> > > 8073ab68:       e92ddff0        push    {r4, r5, r6, r7, r8, r9, sl,
> > > fp, ip, lr, pc}
> > > 8073ab6c:       e24cb004        sub     fp, ip, #4
> > > 8073ab70:       e24ddfa7        sub     sp, sp, #668    ; 0x29c
> >
> > Yes, this is the one that the compiler complained about when warning
> > for stack over 600 bytes. It's not called in this call chain though.
> >
> > > page_alloc can also do reclaim, I had the impression that reclaim can
> > > be quite heavy-weight in all respects.
> >
> > Yes, that is another possibility. What writable file systems or swap
> > do you normally have mounted that it could be writing to, and on
> > what storage device?
>
> The root fs is ext4 on virtio-blk.
>
> There are also several dozens of shrinkers that can be called during reclaim:
> https://elixir.bootlin.com/linux/latest/C/ident/unregister_shrinker

Right, unfortunately I don't see a smoking gun there either, unless you are
also using NFS or devicemapper.

Implementing VMAP_STACK as you suggested earlier is probably the
best way to figure out if there is an actual overrun of the stack.
Alternatively, adding support for GCC_PLUGIN_STACKLEAK might
also help find out if we ever get close to the limit. This is probably
less work, but it might not actually help in this case.

        Arnd

Re: [syzbot] kernel panic: corrupted stack end in openat

[Dmitry Vyukov]

On Wed, Mar 17, 2021 at 9:32 AM Arnd Bergmann <arnd@arndb.de> wrote:
> > > > > <linux@armlinux.org.uk> wrote:
> > > > > > On Tue, Mar 16, 2021 at 04:44:45PM +0100, Arnd Bergmann wrote:
> > > > > > > On Tue, Mar 16, 2021 at 11:17 AM Dmitry Vyukov <dvyukov@google.com> wrote:
> > > > > > > > The compiler is gcc version 10.2.1 20210110 (Debian 10.2.1-6)
> > > > > > >
> > > > > > > Ok, building with Ubuntu 10.2.1-1ubuntu1 20201207 locally, that's
> > > > > > > the closest I have installed, and I think the Debian and Ubuntu versions
> > > > > > > are generally quite close in case of gcc since they are maintained by
> > > > > > > the same packagers.
> > > > > >
> > > > > > ... which shouldn't be a problem - that's just over 1/4 of the stack
> > > > > > space. Could it be the syzbot's gcc is doing something weird and
> > > > > > inflating the stack frames?
> > > > >
> > > > > It's possible, I think that's really unlikely given that it's just Debian's
> > > > > gcc, which is as close to mainline as the version I was using.
> > > > >
> > > > > Uwe's DEBUG_STACKOVERFLOW patch from a while ago might
> > > > > help if this was the problem though:
> > > > > https://lore.kernel.org/linux-arm-kernel/20200108082913.29710-1-u.kleine-koenig@pengutronix.de/
> > > > >
> > > > > My best guess is something going wrong in the interrupt
> > > > > that triggered the preempt_schedule() which ended up calling
> > > > > task_stack_end_corrupted() in schedule_debug(), as you suggested
> > > > > earlier.
> > > >
> > > > FWIW I see slightly larger frames with the config:
> > > >
> > > > 073ab64 <ima_calc_field_array_hash_tfm>:
> > > > 8073ab64:       e1a0c00d        mov     ip, sp
> > > > 8073ab68:       e92ddff0        push    {r4, r5, r6, r7, r8, r9, sl,
> > > > fp, ip, lr, pc}
> > > > 8073ab6c:       e24cb004        sub     fp, ip, #4
> > > > 8073ab70:       e24ddfa7        sub     sp, sp, #668    ; 0x29c
> > >
> > > Yes, this is the one that the compiler complained about when warning
> > > for stack over 600 bytes. It's not called in this call chain though.
> > >
> > > > page_alloc can also do reclaim, I had the impression that reclaim can
> > > > be quite heavy-weight in all respects.
> > >
> > > Yes, that is another possibility. What writable file systems or swap
> > > do you normally have mounted that it could be writing to, and on
> > > what storage device?
> >
> > The root fs is ext4 on virtio-blk.
> >
> > There are also several dozens of shrinkers that can be called during reclaim:
> > https://elixir.bootlin.com/linux/latest/C/ident/unregister_shrinker
>
> Right, unfortunately I don't see a smoking gun there either, unless you are
> also using NFS or devicemapper.
>
> Implementing VMAP_STACK as you suggested earlier is probably the
> best way to figure out if there is an actual overrun of the stack.
> Alternatively, adding support for GCC_PLUGIN_STACKLEAK might
> also help find out if we ever get close to the limit. This is probably
> less work, but it might not actually help in this case.

VMAP_STACK is quite intrusive as far as I understand. For KASAN I
considered a simpler option: have a debug config that allocates an
extra page after the stack and mprotect's it. It wastes a physical
page per task (fine for a debug config), but I would assume should be
radically simpler to implement. In the end somebody implemented proper
VMAP_STACK support for KASAN, but I still think it may be a reasonable
compromise between time investment and value.