* RISC-V Linux kernel not booting up with KASAN enabled @ 2023-03-02 3:13 Chathura Rajapaksha 2023-03-02 7:50 ` Alexandre Ghiti 0 siblings, 1 reply; 10+ messages in thread From: Chathura Rajapaksha @ 2023-03-02 3:13 UTC (permalink / raw) To: linux-riscv Hi All, I observed that RISC-V Linux hangs when I enable KASAN. Without KASAN it works fine with QEMU. I am using the commit ae3419fbac845b4d3f3a9fae4cc80c68d82cdf6e When KASAN is enabled, QEMU hangs after OpenSBI prints. I noticed a similar issue was reported before in https://lore.kernel.org/lkml/CACT4Y+ZmuOpyf_0vHTT4t3wkmJuW8Ezvcg7v6yDVd8YOViS=GA@mail.gmail.com/t/ But I believe I have the patch mentioned in that thread. My kernel config: https://drive.google.com/file/d/1j9nU7f9MxCc_i-UHUCTvo7o6nDrcUz0w/view?usp=sharing Best regards, Chath _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RISC-V Linux kernel not booting up with KASAN enabled 2023-03-02 3:13 RISC-V Linux kernel not booting up with KASAN enabled Chathura Rajapaksha @ 2023-03-02 7:50 ` Alexandre Ghiti 2023-03-02 16:25 ` Chathura Rajapaksha 0 siblings, 1 reply; 10+ messages in thread From: Alexandre Ghiti @ 2023-03-02 7:50 UTC (permalink / raw) To: Chathura Rajapaksha, linux-riscv Hi Chatura, On 3/2/23 04:13, Chathura Rajapaksha wrote: > Hi All, > > I observed that RISC-V Linux hangs when I enable KASAN. > Without KASAN it works fine with QEMU. > I am using the commit ae3419fbac845b4d3f3a9fae4cc80c68d82cdf6e > > When KASAN is enabled, QEMU hangs after OpenSBI prints. > > I noticed a similar issue was reported before in > https://lore.kernel.org/lkml/CACT4Y+ZmuOpyf_0vHTT4t3wkmJuW8Ezvcg7v6yDVd8YOViS=GA@mail.gmail.com/t/ > But I believe I have the patch mentioned in that thread. I proposed a series that will be included in 6.3 regarding KASAN issues here: https://patchwork.kernel.org/project/linux-riscv/list/?series=718458 Can you give it a try and tell me if it works better? Thanks, Alex > > My kernel config: > https://drive.google.com/file/d/1j9nU7f9MxCc_i-UHUCTvo7o6nDrcUz0w/view?usp=sharing > > Best regards, > Chath > > _______________________________________________ > linux-riscv mailing list > linux-riscv@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-riscv _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RISC-V Linux kernel not booting up with KASAN enabled 2023-03-02 7:50 ` Alexandre Ghiti @ 2023-03-02 16:25 ` Chathura Rajapaksha 2023-03-02 18:01 ` Chathura Rajapaksha 2023-03-02 19:49 ` Alexandre Ghiti 0 siblings, 2 replies; 10+ messages in thread From: Chathura Rajapaksha @ 2023-03-02 16:25 UTC (permalink / raw) To: Alexandre Ghiti; +Cc: linux-riscv Hi Alex, Thank you very much, kernel booted up with the patches you mentioned. Bootup was pretty slow compared to before though (on a dev board). I guess that is kind of expected with KASAN enabled. Thanks again. Regards, Chath On Thu, Mar 2, 2023 at 2:50 AM Alexandre Ghiti <alex@ghiti.fr> wrote: > > Hi Chatura, > > On 3/2/23 04:13, Chathura Rajapaksha wrote: > > Hi All, > > > > I observed that RISC-V Linux hangs when I enable KASAN. > > Without KASAN it works fine with QEMU. > > I am using the commit ae3419fbac845b4d3f3a9fae4cc80c68d82cdf6e > > > > When KASAN is enabled, QEMU hangs after OpenSBI prints. > > > > I noticed a similar issue was reported before in > > https://lore.kernel.org/lkml/CACT4Y+ZmuOpyf_0vHTT4t3wkmJuW8Ezvcg7v6yDVd8YOViS=GA@mail.gmail.com/t/ > > But I believe I have the patch mentioned in that thread. > > > I proposed a series that will be included in 6.3 regarding KASAN issues > here: https://patchwork.kernel.org/project/linux-riscv/list/?series=718458 > > Can you give it a try and tell me if it works better? > > Thanks, > > Alex > > > > > > My kernel config: > > https://drive.google.com/file/d/1j9nU7f9MxCc_i-UHUCTvo7o6nDrcUz0w/view?usp=sharing > > > > Best regards, > > Chath > > > > _______________________________________________ > > linux-riscv mailing list > > linux-riscv@lists.infradead.org > > http://lists.infradead.org/mailman/listinfo/linux-riscv _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RISC-V Linux kernel not booting up with KASAN enabled 2023-03-02 16:25 ` Chathura Rajapaksha @ 2023-03-02 18:01 ` Chathura Rajapaksha 2023-03-02 20:11 ` Alexandre Ghiti 2023-03-02 19:49 ` Alexandre Ghiti 1 sibling, 1 reply; 10+ messages in thread From: Chathura Rajapaksha @ 2023-03-02 18:01 UTC (permalink / raw) To: Alexandre Ghiti; +Cc: linux-riscv Hi Alex/All, Kernel is booting now but I get the following KASAN failure in the bootup itself. I didn't see this bug was reported before anywhere. [ 0.000000] Memory: 63436K/129024K available (20385K kernel code, 7120K rwdata, 4096K rodata, 2138K init, 476K bss, 65588K reserved, 0K cma-reserved) [ 0.000000] ================================================================== [ 0.000000] BUG: KASAN: stack-out-of-bounds in walk_stackframe+0x1b2/0x1e2 [ 0.000000] Read of size 8 at addr ffffffff81e07c40 by task swapper/0 [ 0.000000] [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 6.2.0-gae3419fbac84-dirty #7 [ 0.000000] Hardware name: riscv-virtio,qemu (DT) [ 0.000000] Call Trace: [ 0.000000] [<ffffffff8000ab9e>] walk_stackframe+0x0/0x1e2 [ 0.000000] [<ffffffff80108508>] init_param_lock+0x26/0x2a [ 0.000000] [<ffffffff8000ad4c>] walk_stackframe+0x1ae/0x1e2 [ 0.000000] [<ffffffff813d86e0>] dump_stack_lvl+0x22/0x36 [ 0.000000] [<ffffffff813bd17a>] print_report+0x198/0x4a8 [ 0.000000] [<ffffffff80108508>] init_param_lock+0x26/0x2a [ 0.000000] [<ffffffff8000ad4c>] walk_stackframe+0x1ae/0x1e2 [ 0.000000] [<ffffffff8023bd52>] kasan_report+0x9a/0xc8 [ 0.000000] [<ffffffff8000ad4c>] walk_stackframe+0x1ae/0x1e2 [ 0.000000] [<ffffffff8000ad4c>] walk_stackframe+0x1ae/0x1e2 [ 0.000000] [<ffffffff80108748>] stack_trace_save+0x88/0xa6 [ 0.000000] [<ffffffff801086bc>] filter_irq_stacks+0x8a/0x8e [ 0.000000] [<ffffffff800b65e2>] devkmsg_read+0x3f8/0x3fc [ 0.000000] [<ffffffff8023b2de>] kasan_save_stack+0x2c/0x56 [ 0.000000] [<ffffffff80108744>] stack_trace_save+0x84/0xa6 [ 0.000000] [<ffffffff8023b31a>] kasan_set_track+0x12/0x20 [ 0.000000] [<ffffffff8023b8f6>] __kasan_slab_alloc+0x58/0x5e [ 0.000000] [<ffffffff8023aeae>] __kmem_cache_create+0x21e/0x39a [ 0.000000] [<ffffffff8141623e>] create_boot_cache+0x70/0x9c [ 0.000000] [<ffffffff8141b5f6>] kmem_cache_init+0x6c/0x11e [ 0.000000] [<ffffffff8140125a>] mm_init+0xd8/0xfe [ 0.000000] [<ffffffff8140145c>] start_kernel+0x190/0x3ca [ 0.000000] [ 0.000000] The buggy address belongs to stack of task swapper/0 [ 0.000000] and is located at offset 0 in frame: [ 0.000000] stack_trace_save+0x0/0xa6 [ 0.000000] [ 0.000000] This frame has 1 object: [ 0.000000] [32, 56) 'c' [ 0.000000] [ 0.000000] The buggy address belongs to the physical page: [ 0.000000] page:(____ptrval____) refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x82007 [ 0.000000] flags: 0x1000(reserved|zone=0) [ 0.000000] raw: 0000000000001000 ff60000007ca5090 ff60000007ca5090 0000000000000000 [ 0.000000] raw: 0000000000000000 0000000000000000 00000001ffffffff [ 0.000000] page dumped because: kasan: bad access detected [ 0.000000] [ 0.000000] Memory state around the buggy address: [ 0.000000] ffffffff81e07b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.000000] ffffffff81e07b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.000000] >ffffffff81e07c00: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 00 00 f3 [ 0.000000] ^ [ 0.000000] ffffffff81e07c80: f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.000000] ffffffff81e07d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 0.000000] ================================================================== Best, Chath On Thu, Mar 2, 2023 at 11:25 AM Chathura Rajapaksha <chathura.abeyrathne.lk@gmail.com> wrote: > > Hi Alex, > > Thank you very much, kernel booted up with the patches you mentioned. > Bootup was pretty slow compared to before though (on a dev board). > I guess that is kind of expected with KASAN enabled. > Thanks again. > > Regards, > Chath > > On Thu, Mar 2, 2023 at 2:50 AM Alexandre Ghiti <alex@ghiti.fr> wrote: > > > > Hi Chatura, > > > > On 3/2/23 04:13, Chathura Rajapaksha wrote: > > > Hi All, > > > > > > I observed that RISC-V Linux hangs when I enable KASAN. > > > Without KASAN it works fine with QEMU. > > > I am using the commit ae3419fbac845b4d3f3a9fae4cc80c68d82cdf6e > > > > > > When KASAN is enabled, QEMU hangs after OpenSBI prints. > > > > > > I noticed a similar issue was reported before in > > > https://lore.kernel.org/lkml/CACT4Y+ZmuOpyf_0vHTT4t3wkmJuW8Ezvcg7v6yDVd8YOViS=GA@mail.gmail.com/t/ > > > But I believe I have the patch mentioned in that thread. > > > > > > I proposed a series that will be included in 6.3 regarding KASAN issues > > here: https://patchwork.kernel.org/project/linux-riscv/list/?series=718458 > > > > Can you give it a try and tell me if it works better? > > > > Thanks, > > > > Alex > > > > > > > > > > My kernel config: > > > https://drive.google.com/file/d/1j9nU7f9MxCc_i-UHUCTvo7o6nDrcUz0w/view?usp=sharing > > > > > > Best regards, > > > Chath > > > > > > _______________________________________________ > > > linux-riscv mailing list > > > linux-riscv@lists.infradead.org > > > http://lists.infradead.org/mailman/listinfo/linux-riscv -- Chathura Rajapaksha Ph.D. Candidate, Boston University _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RISC-V Linux kernel not booting up with KASAN enabled 2023-03-02 18:01 ` Chathura Rajapaksha @ 2023-03-02 20:11 ` Alexandre Ghiti 2023-03-03 5:44 ` Dmitry Vyukov 0 siblings, 1 reply; 10+ messages in thread From: Alexandre Ghiti @ 2023-03-02 20:11 UTC (permalink / raw) To: Chathura Rajapaksha Cc: linux-riscv, dvyukov@google.com >> Dmitry Vyukov, kasan-dev +cc Dmitry and kasan-dev, in case they know about this but I did not find anything related On 3/2/23 19:01, Chathura Rajapaksha wrote: > Hi Alex/All, > > Kernel is booting now but I get the following KASAN failure in the > bootup itself. > I didn't see this bug was reported before anywhere. > > [ 0.000000] Memory: 63436K/129024K available (20385K kernel code, > 7120K rwdata, 4096K rodata, 2138K init, 476K bss, 65588K reserved, 0K > cma-reserved) > [ 0.000000] ================================================================== > [ 0.000000] BUG: KASAN: stack-out-of-bounds in walk_stackframe+0x1b2/0x1e2 > [ 0.000000] Read of size 8 at addr ffffffff81e07c40 by task swapper/0 > [ 0.000000] > [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted > 6.2.0-gae3419fbac84-dirty #7 > [ 0.000000] Hardware name: riscv-virtio,qemu (DT) > [ 0.000000] Call Trace: > [ 0.000000] [<ffffffff8000ab9e>] walk_stackframe+0x0/0x1e2 > [ 0.000000] [<ffffffff80108508>] init_param_lock+0x26/0x2a > [ 0.000000] [<ffffffff8000ad4c>] walk_stackframe+0x1ae/0x1e2 > [ 0.000000] [<ffffffff813d86e0>] dump_stack_lvl+0x22/0x36 > [ 0.000000] [<ffffffff813bd17a>] print_report+0x198/0x4a8 > [ 0.000000] [<ffffffff80108508>] init_param_lock+0x26/0x2a > [ 0.000000] [<ffffffff8000ad4c>] walk_stackframe+0x1ae/0x1e2 > [ 0.000000] [<ffffffff8023bd52>] kasan_report+0x9a/0xc8 > [ 0.000000] [<ffffffff8000ad4c>] walk_stackframe+0x1ae/0x1e2 > [ 0.000000] [<ffffffff8000ad4c>] walk_stackframe+0x1ae/0x1e2 > [ 0.000000] [<ffffffff80108748>] stack_trace_save+0x88/0xa6 > [ 0.000000] [<ffffffff801086bc>] filter_irq_stacks+0x8a/0x8e > [ 0.000000] [<ffffffff800b65e2>] devkmsg_read+0x3f8/0x3fc > [ 0.000000] [<ffffffff8023b2de>] kasan_save_stack+0x2c/0x56 > [ 0.000000] [<ffffffff80108744>] stack_trace_save+0x84/0xa6 > [ 0.000000] [<ffffffff8023b31a>] kasan_set_track+0x12/0x20 > [ 0.000000] [<ffffffff8023b8f6>] __kasan_slab_alloc+0x58/0x5e > [ 0.000000] [<ffffffff8023aeae>] __kmem_cache_create+0x21e/0x39a > [ 0.000000] [<ffffffff8141623e>] create_boot_cache+0x70/0x9c > [ 0.000000] [<ffffffff8141b5f6>] kmem_cache_init+0x6c/0x11e > [ 0.000000] [<ffffffff8140125a>] mm_init+0xd8/0xfe > [ 0.000000] [<ffffffff8140145c>] start_kernel+0x190/0x3ca > [ 0.000000] > [ 0.000000] The buggy address belongs to stack of task swapper/0 > [ 0.000000] and is located at offset 0 in frame: > [ 0.000000] stack_trace_save+0x0/0xa6 > [ 0.000000] > [ 0.000000] This frame has 1 object: > [ 0.000000] [32, 56) 'c' > [ 0.000000] > [ 0.000000] The buggy address belongs to the physical page: > [ 0.000000] page:(____ptrval____) refcount:1 mapcount:0 > mapping:0000000000000000 index:0x0 pfn:0x82007 > [ 0.000000] flags: 0x1000(reserved|zone=0) > [ 0.000000] raw: 0000000000001000 ff60000007ca5090 ff60000007ca5090 > 0000000000000000 > [ 0.000000] raw: 0000000000000000 0000000000000000 00000001ffffffff > [ 0.000000] page dumped because: kasan: bad access detected > [ 0.000000] > [ 0.000000] Memory state around the buggy address: > [ 0.000000] ffffffff81e07b00: 00 00 00 00 00 00 00 00 00 00 00 00 > 00 00 00 00 > [ 0.000000] ffffffff81e07b80: 00 00 00 00 00 00 00 00 00 00 00 00 > 00 00 00 00 > [ 0.000000] >ffffffff81e07c00: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 > 00 00 00 f3 > [ 0.000000] ^ > [ 0.000000] ffffffff81e07c80: f3 f3 f3 f3 00 00 00 00 00 00 00 00 > 00 00 00 00 > [ 0.000000] ffffffff81e07d00: 00 00 00 00 00 00 00 00 00 00 00 00 > 00 00 00 00 > [ 0.000000] ================================================================== I was able to reproduce the exact same trace, I'll debug that tomorrow, I hope it is a real bug :) Thanks for the report Chatura, Alex > > Best, > Chath > > On Thu, Mar 2, 2023 at 11:25 AM Chathura Rajapaksha > <chathura.abeyrathne.lk@gmail.com> wrote: >> Hi Alex, >> >> Thank you very much, kernel booted up with the patches you mentioned. >> Bootup was pretty slow compared to before though (on a dev board). >> I guess that is kind of expected with KASAN enabled. >> Thanks again. >> >> Regards, >> Chath >> >> On Thu, Mar 2, 2023 at 2:50 AM Alexandre Ghiti <alex@ghiti.fr> wrote: >>> Hi Chatura, >>> >>> On 3/2/23 04:13, Chathura Rajapaksha wrote: >>>> Hi All, >>>> >>>> I observed that RISC-V Linux hangs when I enable KASAN. >>>> Without KASAN it works fine with QEMU. >>>> I am using the commit ae3419fbac845b4d3f3a9fae4cc80c68d82cdf6e >>>> >>>> When KASAN is enabled, QEMU hangs after OpenSBI prints. >>>> >>>> I noticed a similar issue was reported before in >>>> https://lore.kernel.org/lkml/CACT4Y+ZmuOpyf_0vHTT4t3wkmJuW8Ezvcg7v6yDVd8YOViS=GA@mail.gmail.com/t/ >>>> But I believe I have the patch mentioned in that thread. >>> >>> I proposed a series that will be included in 6.3 regarding KASAN issues >>> here: https://patchwork.kernel.org/project/linux-riscv/list/?series=718458 >>> >>> Can you give it a try and tell me if it works better? >>> >>> Thanks, >>> >>> Alex >>> >>> >>>> My kernel config: >>>> https://drive.google.com/file/d/1j9nU7f9MxCc_i-UHUCTvo7o6nDrcUz0w/view?usp=sharing >>>> >>>> Best regards, >>>> Chath >>>> >>>> _______________________________________________ >>>> linux-riscv mailing list >>>> linux-riscv@lists.infradead.org >>>> http://lists.infradead.org/mailman/listinfo/linux-riscv > > _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RISC-V Linux kernel not booting up with KASAN enabled 2023-03-02 20:11 ` Alexandre Ghiti @ 2023-03-03 5:44 ` Dmitry Vyukov 2023-03-03 14:16 ` Alexandre Ghiti 2023-03-07 2:57 ` Chathura Rajapaksha 0 siblings, 2 replies; 10+ messages in thread From: Dmitry Vyukov @ 2023-03-03 5:44 UTC (permalink / raw) To: alex; +Cc: Chathura Rajapaksha, linux-riscv, kasan-dev On Thu, 2 Mar 2023 at 21:11, Alexandre Ghiti <alex@ghiti.fr> wrote: > > +cc Dmitry and kasan-dev, in case they know about this but I did not > find anything related Hard to say anything w/o commit/symbolized report. If it's stack unwinder and it's supposed to be precise, then it may be a bug in the unwinder where it reads a wrong location and is imprecise (not the frame pointer). If it's supposed to be imprecise, then it should use READ_ONCE_NOCHECK to read random stack locations. > On 3/2/23 19:01, Chathura Rajapaksha wrote: > > Hi Alex/All, > > > > Kernel is booting now but I get the following KASAN failure in the > > bootup itself. > > I didn't see this bug was reported before anywhere. > > > > [ 0.000000] Memory: 63436K/129024K available (20385K kernel code, > > 7120K rwdata, 4096K rodata, 2138K init, 476K bss, 65588K reserved, 0K > > cma-reserved) > > [ 0.000000] ================================================================== > > [ 0.000000] BUG: KASAN: stack-out-of-bounds in walk_stackframe+0x1b2/0x1e2 > > [ 0.000000] Read of size 8 at addr ffffffff81e07c40 by task swapper/0 > > [ 0.000000] > > [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted > > 6.2.0-gae3419fbac84-dirty #7 > > [ 0.000000] Hardware name: riscv-virtio,qemu (DT) > > [ 0.000000] Call Trace: > > [ 0.000000] [<ffffffff8000ab9e>] walk_stackframe+0x0/0x1e2 > > [ 0.000000] [<ffffffff80108508>] init_param_lock+0x26/0x2a > > [ 0.000000] [<ffffffff8000ad4c>] walk_stackframe+0x1ae/0x1e2 > > [ 0.000000] [<ffffffff813d86e0>] dump_stack_lvl+0x22/0x36 > > [ 0.000000] [<ffffffff813bd17a>] print_report+0x198/0x4a8 > > [ 0.000000] [<ffffffff80108508>] init_param_lock+0x26/0x2a > > [ 0.000000] [<ffffffff8000ad4c>] walk_stackframe+0x1ae/0x1e2 > > [ 0.000000] [<ffffffff8023bd52>] kasan_report+0x9a/0xc8 > > [ 0.000000] [<ffffffff8000ad4c>] walk_stackframe+0x1ae/0x1e2 > > [ 0.000000] [<ffffffff8000ad4c>] walk_stackframe+0x1ae/0x1e2 > > [ 0.000000] [<ffffffff80108748>] stack_trace_save+0x88/0xa6 > > [ 0.000000] [<ffffffff801086bc>] filter_irq_stacks+0x8a/0x8e > > [ 0.000000] [<ffffffff800b65e2>] devkmsg_read+0x3f8/0x3fc > > [ 0.000000] [<ffffffff8023b2de>] kasan_save_stack+0x2c/0x56 > > [ 0.000000] [<ffffffff80108744>] stack_trace_save+0x84/0xa6 > > [ 0.000000] [<ffffffff8023b31a>] kasan_set_track+0x12/0x20 > > [ 0.000000] [<ffffffff8023b8f6>] __kasan_slab_alloc+0x58/0x5e > > [ 0.000000] [<ffffffff8023aeae>] __kmem_cache_create+0x21e/0x39a > > [ 0.000000] [<ffffffff8141623e>] create_boot_cache+0x70/0x9c > > [ 0.000000] [<ffffffff8141b5f6>] kmem_cache_init+0x6c/0x11e > > [ 0.000000] [<ffffffff8140125a>] mm_init+0xd8/0xfe > > [ 0.000000] [<ffffffff8140145c>] start_kernel+0x190/0x3ca > > [ 0.000000] > > [ 0.000000] The buggy address belongs to stack of task swapper/0 > > [ 0.000000] and is located at offset 0 in frame: > > [ 0.000000] stack_trace_save+0x0/0xa6 > > [ 0.000000] > > [ 0.000000] This frame has 1 object: > > [ 0.000000] [32, 56) 'c' > > [ 0.000000] > > [ 0.000000] The buggy address belongs to the physical page: > > [ 0.000000] page:(____ptrval____) refcount:1 mapcount:0 > > mapping:0000000000000000 index:0x0 pfn:0x82007 > > [ 0.000000] flags: 0x1000(reserved|zone=0) > > [ 0.000000] raw: 0000000000001000 ff60000007ca5090 ff60000007ca5090 > > 0000000000000000 > > [ 0.000000] raw: 0000000000000000 0000000000000000 00000001ffffffff > > [ 0.000000] page dumped because: kasan: bad access detected > > [ 0.000000] > > [ 0.000000] Memory state around the buggy address: > > [ 0.000000] ffffffff81e07b00: 00 00 00 00 00 00 00 00 00 00 00 00 > > 00 00 00 00 > > [ 0.000000] ffffffff81e07b80: 00 00 00 00 00 00 00 00 00 00 00 00 > > 00 00 00 00 > > [ 0.000000] >ffffffff81e07c00: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 > > 00 00 00 f3 > > [ 0.000000] ^ > > [ 0.000000] ffffffff81e07c80: f3 f3 f3 f3 00 00 00 00 00 00 00 00 > > 00 00 00 00 > > [ 0.000000] ffffffff81e07d00: 00 00 00 00 00 00 00 00 00 00 00 00 > > 00 00 00 00 > > [ 0.000000] ================================================================== > > > I was able to reproduce the exact same trace, I'll debug that tomorrow, > I hope it is a real bug :) > > Thanks for the report Chatura, > > Alex > > > > > > Best, > > Chath > > > > On Thu, Mar 2, 2023 at 11:25 AM Chathura Rajapaksha > > <chathura.abeyrathne.lk@gmail.com> wrote: > >> Hi Alex, > >> > >> Thank you very much, kernel booted up with the patches you mentioned. > >> Bootup was pretty slow compared to before though (on a dev board). > >> I guess that is kind of expected with KASAN enabled. > >> Thanks again. > >> > >> Regards, > >> Chath > >> > >> On Thu, Mar 2, 2023 at 2:50 AM Alexandre Ghiti <alex@ghiti.fr> wrote: > >>> Hi Chatura, > >>> > >>> On 3/2/23 04:13, Chathura Rajapaksha wrote: > >>>> Hi All, > >>>> > >>>> I observed that RISC-V Linux hangs when I enable KASAN. > >>>> Without KASAN it works fine with QEMU. > >>>> I am using the commit ae3419fbac845b4d3f3a9fae4cc80c68d82cdf6e > >>>> > >>>> When KASAN is enabled, QEMU hangs after OpenSBI prints. > >>>> > >>>> I noticed a similar issue was reported before in > >>>> https://lore.kernel.org/lkml/CACT4Y+ZmuOpyf_0vHTT4t3wkmJuW8Ezvcg7v6yDVd8YOViS=GA@mail.gmail.com/t/ > >>>> But I believe I have the patch mentioned in that thread. > >>> > >>> I proposed a series that will be included in 6.3 regarding KASAN issues > >>> here: https://patchwork.kernel.org/project/linux-riscv/list/?series=718458 > >>> > >>> Can you give it a try and tell me if it works better? > >>> > >>> Thanks, > >>> > >>> Alex > >>> > >>> > >>>> My kernel config: > >>>> https://drive.google.com/file/d/1j9nU7f9MxCc_i-UHUCTvo7o6nDrcUz0w/view?usp=sharing > >>>> > >>>> Best regards, > >>>> Chath > >>>> > >>>> _______________________________________________ > >>>> linux-riscv mailing list > >>>> linux-riscv@lists.infradead.org > >>>> http://lists.infradead.org/mailman/listinfo/linux-riscv > > > > _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RISC-V Linux kernel not booting up with KASAN enabled 2023-03-03 5:44 ` Dmitry Vyukov @ 2023-03-03 14:16 ` Alexandre Ghiti 2023-03-07 2:57 ` Chathura Rajapaksha 1 sibling, 0 replies; 10+ messages in thread From: Alexandre Ghiti @ 2023-03-03 14:16 UTC (permalink / raw) To: Dmitry Vyukov; +Cc: Chathura Rajapaksha, linux-riscv, kasan-dev On 3/3/23 06:44, Dmitry Vyukov wrote: > On Thu, 2 Mar 2023 at 21:11, Alexandre Ghiti <alex@ghiti.fr> wrote: >> +cc Dmitry and kasan-dev, in case they know about this but I did not >> find anything related > Hard to say anything w/o commit/symbolized report. > If it's stack unwinder and it's supposed to be precise, then it may be > a bug in the unwinder where it reads a wrong location and is imprecise > (not the frame pointer). > If it's supposed to be imprecise, then it should use READ_ONCE_NOCHECK > to read random stack locations. Please correct me if I say something obviously wrong. The config used to generate this trace does not set CONFIG_FRAME_POINTER: we were then in an imprecise stack unwinding mode. When set, the backtrace disappears: so IIUC, the issue lies in the stack unwinding function that reads the stack randomly and KASAN does not like that. So as you suggested, I used READ_ONCE_NOCHECK when reading the stack and the backtrace also disappears. So the following patch would be the fix for this, is that correct? diff --git a/arch/riscv/kernel/stacktrace.c b/arch/riscv/kernel/stacktrace.c index f9a5a7c90ff0..64a9c093aef9 100644 --- a/arch/riscv/kernel/stacktrace.c +++ b/arch/riscv/kernel/stacktrace.c @@ -101,7 +101,7 @@ void notrace walk_stackframe(struct task_struct *task, while (!kstack_end(ksp)) { if (__kernel_text_address(pc) && unlikely(!fn(arg, pc))) break; - pc = (*ksp++) - 0x4; + pc = READ_ONCE_NOCHECK(*ksp++) - 0x4; } } Thanks for your quick answer, Alex > >> On 3/2/23 19:01, Chathura Rajapaksha wrote: >>> Hi Alex/All, >>> >>> Kernel is booting now but I get the following KASAN failure in the >>> bootup itself. >>> I didn't see this bug was reported before anywhere. >>> >>> [ 0.000000] Memory: 63436K/129024K available (20385K kernel code, >>> 7120K rwdata, 4096K rodata, 2138K init, 476K bss, 65588K reserved, 0K >>> cma-reserved) >>> [ 0.000000] ================================================================== >>> [ 0.000000] BUG: KASAN: stack-out-of-bounds in walk_stackframe+0x1b2/0x1e2 >>> [ 0.000000] Read of size 8 at addr ffffffff81e07c40 by task swapper/0 >>> [ 0.000000] >>> [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted >>> 6.2.0-gae3419fbac84-dirty #7 >>> [ 0.000000] Hardware name: riscv-virtio,qemu (DT) >>> [ 0.000000] Call Trace: >>> [ 0.000000] [<ffffffff8000ab9e>] walk_stackframe+0x0/0x1e2 >>> [ 0.000000] [<ffffffff80108508>] init_param_lock+0x26/0x2a >>> [ 0.000000] [<ffffffff8000ad4c>] walk_stackframe+0x1ae/0x1e2 >>> [ 0.000000] [<ffffffff813d86e0>] dump_stack_lvl+0x22/0x36 >>> [ 0.000000] [<ffffffff813bd17a>] print_report+0x198/0x4a8 >>> [ 0.000000] [<ffffffff80108508>] init_param_lock+0x26/0x2a >>> [ 0.000000] [<ffffffff8000ad4c>] walk_stackframe+0x1ae/0x1e2 >>> [ 0.000000] [<ffffffff8023bd52>] kasan_report+0x9a/0xc8 >>> [ 0.000000] [<ffffffff8000ad4c>] walk_stackframe+0x1ae/0x1e2 >>> [ 0.000000] [<ffffffff8000ad4c>] walk_stackframe+0x1ae/0x1e2 >>> [ 0.000000] [<ffffffff80108748>] stack_trace_save+0x88/0xa6 >>> [ 0.000000] [<ffffffff801086bc>] filter_irq_stacks+0x8a/0x8e >>> [ 0.000000] [<ffffffff800b65e2>] devkmsg_read+0x3f8/0x3fc >>> [ 0.000000] [<ffffffff8023b2de>] kasan_save_stack+0x2c/0x56 >>> [ 0.000000] [<ffffffff80108744>] stack_trace_save+0x84/0xa6 >>> [ 0.000000] [<ffffffff8023b31a>] kasan_set_track+0x12/0x20 >>> [ 0.000000] [<ffffffff8023b8f6>] __kasan_slab_alloc+0x58/0x5e >>> [ 0.000000] [<ffffffff8023aeae>] __kmem_cache_create+0x21e/0x39a >>> [ 0.000000] [<ffffffff8141623e>] create_boot_cache+0x70/0x9c >>> [ 0.000000] [<ffffffff8141b5f6>] kmem_cache_init+0x6c/0x11e >>> [ 0.000000] [<ffffffff8140125a>] mm_init+0xd8/0xfe >>> [ 0.000000] [<ffffffff8140145c>] start_kernel+0x190/0x3ca >>> [ 0.000000] >>> [ 0.000000] The buggy address belongs to stack of task swapper/0 >>> [ 0.000000] and is located at offset 0 in frame: >>> [ 0.000000] stack_trace_save+0x0/0xa6 >>> [ 0.000000] >>> [ 0.000000] This frame has 1 object: >>> [ 0.000000] [32, 56) 'c' >>> [ 0.000000] >>> [ 0.000000] The buggy address belongs to the physical page: >>> [ 0.000000] page:(____ptrval____) refcount:1 mapcount:0 >>> mapping:0000000000000000 index:0x0 pfn:0x82007 >>> [ 0.000000] flags: 0x1000(reserved|zone=0) >>> [ 0.000000] raw: 0000000000001000 ff60000007ca5090 ff60000007ca5090 >>> 0000000000000000 >>> [ 0.000000] raw: 0000000000000000 0000000000000000 00000001ffffffff >>> [ 0.000000] page dumped because: kasan: bad access detected >>> [ 0.000000] >>> [ 0.000000] Memory state around the buggy address: >>> [ 0.000000] ffffffff81e07b00: 00 00 00 00 00 00 00 00 00 00 00 00 >>> 00 00 00 00 >>> [ 0.000000] ffffffff81e07b80: 00 00 00 00 00 00 00 00 00 00 00 00 >>> 00 00 00 00 >>> [ 0.000000] >ffffffff81e07c00: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 >>> 00 00 00 f3 >>> [ 0.000000] ^ >>> [ 0.000000] ffffffff81e07c80: f3 f3 f3 f3 00 00 00 00 00 00 00 00 >>> 00 00 00 00 >>> [ 0.000000] ffffffff81e07d00: 00 00 00 00 00 00 00 00 00 00 00 00 >>> 00 00 00 00 >>> [ 0.000000] ================================================================== >> >> I was able to reproduce the exact same trace, I'll debug that tomorrow, >> I hope it is a real bug :) >> >> Thanks for the report Chatura, >> >> Alex >> >> >>> Best, >>> Chath >>> >>> On Thu, Mar 2, 2023 at 11:25 AM Chathura Rajapaksha >>> <chathura.abeyrathne.lk@gmail.com> wrote: >>>> Hi Alex, >>>> >>>> Thank you very much, kernel booted up with the patches you mentioned. >>>> Bootup was pretty slow compared to before though (on a dev board). >>>> I guess that is kind of expected with KASAN enabled. >>>> Thanks again. >>>> >>>> Regards, >>>> Chath >>>> >>>> On Thu, Mar 2, 2023 at 2:50 AM Alexandre Ghiti <alex@ghiti.fr> wrote: >>>>> Hi Chatura, >>>>> >>>>> On 3/2/23 04:13, Chathura Rajapaksha wrote: >>>>>> Hi All, >>>>>> >>>>>> I observed that RISC-V Linux hangs when I enable KASAN. >>>>>> Without KASAN it works fine with QEMU. >>>>>> I am using the commit ae3419fbac845b4d3f3a9fae4cc80c68d82cdf6e >>>>>> >>>>>> When KASAN is enabled, QEMU hangs after OpenSBI prints. >>>>>> >>>>>> I noticed a similar issue was reported before in >>>>>> https://lore.kernel.org/lkml/CACT4Y+ZmuOpyf_0vHTT4t3wkmJuW8Ezvcg7v6yDVd8YOViS=GA@mail.gmail.com/t/ >>>>>> But I believe I have the patch mentioned in that thread. >>>>> I proposed a series that will be included in 6.3 regarding KASAN issues >>>>> here: https://patchwork.kernel.org/project/linux-riscv/list/?series=718458 >>>>> >>>>> Can you give it a try and tell me if it works better? >>>>> >>>>> Thanks, >>>>> >>>>> Alex >>>>> >>>>> >>>>>> My kernel config: >>>>>> https://drive.google.com/file/d/1j9nU7f9MxCc_i-UHUCTvo7o6nDrcUz0w/view?usp=sharing >>>>>> >>>>>> Best regards, >>>>>> Chath >>>>>> >>>>>> _______________________________________________ >>>>>> linux-riscv mailing list >>>>>> linux-riscv@lists.infradead.org >>>>>> http://lists.infradead.org/mailman/listinfo/linux-riscv >>> > _______________________________________________ > linux-riscv mailing list > linux-riscv@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-riscv _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: RISC-V Linux kernel not booting up with KASAN enabled 2023-03-03 5:44 ` Dmitry Vyukov 2023-03-03 14:16 ` Alexandre Ghiti @ 2023-03-07 2:57 ` Chathura Rajapaksha 2023-03-08 9:10 ` Alexandre Ghiti 1 sibling, 1 reply; 10+ messages in thread From: Chathura Rajapaksha @ 2023-03-07 2:57 UTC (permalink / raw) To: Dmitry Vyukov; +Cc: alex, linux-riscv, kasan-dev Thanks, Dmitry and Alex. Let me know if you need anything else from me. Please let me know if you have a fix for this bug, I will be happy to verify. Best regards, Chath _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RISC-V Linux kernel not booting up with KASAN enabled 2023-03-07 2:57 ` Chathura Rajapaksha @ 2023-03-08 9:10 ` Alexandre Ghiti 0 siblings, 0 replies; 10+ messages in thread From: Alexandre Ghiti @ 2023-03-08 9:10 UTC (permalink / raw) To: Chathura Rajapaksha, Dmitry Vyukov; +Cc: linux-riscv, kasan-dev On 3/7/23 03:57, Chathura Rajapaksha wrote: > Thanks, Dmitry and Alex. Let me know if you need anything else from me. > Please let me know if you have a fix for this bug, I will be happy to verify. > > Best regards, > Chath > > _______________________________________________ > linux-riscv mailing list > linux-riscv@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-riscv I'm about to propose a patch for this issue, if you can test it, that would be nice, Thanks, Alex _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RISC-V Linux kernel not booting up with KASAN enabled 2023-03-02 16:25 ` Chathura Rajapaksha 2023-03-02 18:01 ` Chathura Rajapaksha @ 2023-03-02 19:49 ` Alexandre Ghiti 1 sibling, 0 replies; 10+ messages in thread From: Alexandre Ghiti @ 2023-03-02 19:49 UTC (permalink / raw) To: Chathura Rajapaksha; +Cc: linux-riscv On 3/2/23 17:25, Chathura Rajapaksha wrote: > Hi Alex, > > Thank you very much, kernel booted up with the patches you mentioned. Great, thanks for giving it a try! > Bootup was pretty slow compared to before though (on a dev board). > I guess that is kind of expected with KASAN enabled. Only the page table creation changes, that's weird: I saw in your config you were using outline instrumentation, can you try to use the inline instrumentation (select KASAN_INLINE instead of KASAN_OUTLINE) and see if that's faster? > Thanks again. > > Regards, > Chath Thanks for testing, Alex > > On Thu, Mar 2, 2023 at 2:50 AM Alexandre Ghiti <alex@ghiti.fr> wrote: >> Hi Chatura, >> >> On 3/2/23 04:13, Chathura Rajapaksha wrote: >>> Hi All, >>> >>> I observed that RISC-V Linux hangs when I enable KASAN. >>> Without KASAN it works fine with QEMU. >>> I am using the commit ae3419fbac845b4d3f3a9fae4cc80c68d82cdf6e >>> >>> When KASAN is enabled, QEMU hangs after OpenSBI prints. >>> >>> I noticed a similar issue was reported before in >>> https://lore.kernel.org/lkml/CACT4Y+ZmuOpyf_0vHTT4t3wkmJuW8Ezvcg7v6yDVd8YOViS=GA@mail.gmail.com/t/ >>> But I believe I have the patch mentioned in that thread. >> >> I proposed a series that will be included in 6.3 regarding KASAN issues >> here: https://patchwork.kernel.org/project/linux-riscv/list/?series=718458 >> >> Can you give it a try and tell me if it works better? >> >> Thanks, >> >> Alex >> >> >>> My kernel config: >>> https://drive.google.com/file/d/1j9nU7f9MxCc_i-UHUCTvo7o6nDrcUz0w/view?usp=sharing >>> >>> Best regards, >>> Chath >>> >>> _______________________________________________ >>> linux-riscv mailing list >>> linux-riscv@lists.infradead.org >>> http://lists.infradead.org/mailman/listinfo/linux-riscv > _______________________________________________ > linux-riscv mailing list > linux-riscv@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-riscv _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2023-03-08 9:10 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-03-02 3:13 RISC-V Linux kernel not booting up with KASAN enabled Chathura Rajapaksha 2023-03-02 7:50 ` Alexandre Ghiti 2023-03-02 16:25 ` Chathura Rajapaksha 2023-03-02 18:01 ` Chathura Rajapaksha 2023-03-02 20:11 ` Alexandre Ghiti 2023-03-03 5:44 ` Dmitry Vyukov 2023-03-03 14:16 ` Alexandre Ghiti 2023-03-07 2:57 ` Chathura Rajapaksha 2023-03-08 9:10 ` Alexandre Ghiti 2023-03-02 19:49 ` Alexandre Ghiti
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.