From: "Chan Kim" <ckim@etri.re.kr> To: "'qemu-devel'" <qemu-devel@nongnu.org>, "'kernelnewbies'" <kernelnewbies@kernelnewbies.org> Subject: RE: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build Date: Tue, 26 Apr 2022 10:46:45 +0900 [thread overview] Message-ID: <038001d8590f$7ccd7210$76685630$@etri.re.kr> (raw) In-Reply-To: [-- Attachment #1.1: Type: text/plain, Size: 6552 bytes --] Hello, I hope somebody could check this case. It’s easily reproducible for anybody working with qemu and arm64 linux. I returned to this problem and made another observation.(showing the back-trace is really broken). In another incident of breakpoint at function __driver_attach, (right after it stopped at the function) (gdb) bt #0 __driver_attach (dev=dev@entry=0xffff0000401d1810, data=data@entry=0xffff800011bbbbb8 <mxc_gpio_driver+40>) at drivers/base/dd.c:1046 #1 0xffff8000107684f8 in bus_for_each_dev (bus=0xffff800011cba910 <platform_bus_type>, start=0x0, data=0xffff800011bbbbb8 <mxc_gpio_driver+40>, fn=0xffff80001076b860 <__driver_attach>) at drivers/base/bus.c:307 #2 0xb8cd80001076a594 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) x/5g $sp 0xffff800011dcbcc0: 0xffff800011dcbd20 0xb8cd80001076a594 0xffff800011dcbcd0: 0xffff80001076b860 0xffff800011bbbbb8 0xffff800011dcbce0: 0x0000000000000000 Because it's right after the pc reached the function __driver_attach, the sp is still not updated from previous function (bus_for_each_dev). And the first two values at the $sp are supposed to be the fp and lr of the previous function (see https://stackoverflow.com/questions/66098678/understanding-aarch64-assembly-function-call-how-is-stack-operated arm64 stores previous function's fp and lr at the bottom of new stack frame as it enters a function). The lr (link register, the address to return after this bus_for_each_dev function) is really 0xb8cd80001076a594 which is weird (not a kernel address). The following 3 values are function arguments for bus_for_each_dev and they look correct. From: Chan Kim <ckim@etri.re.kr> Sent: Saturday, April 23, 2022 8:37 AM To: 'Mulyadi Santosa' <mulyadi.santosa@gmail.com> Cc: 'qemu-devel' <qemu-devel@nongnu.org>; 'kernelnewbies' <kernelnewbies@kernelnewbies.org> Subject: RE: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build Hi, Mulyadi Thank you for replying. I found CONFIG_DEBUG_FRAME_POINTER, CONFIG_DEBUG_INFO are already set by default. And I tried adding CONFIG_DEBUG_KERNEL, CONFIG_KGDB, CONFIG_GDB_SCRIPTS, CONFIG_STACKTRACE all to no avail. Regards, Chan From: Mulyadi Santosa <mulyadi.santosa@gmail.com <mailto:mulyadi.santosa@gmail.com> > Sent: Friday, April 22, 2022 11:53 PM To: Chan Kim <ckim@etri.re.kr <mailto:ckim@etri.re.kr> > Cc: qemu-devel <qemu-devel@nongnu.org <mailto:qemu-devel@nongnu.org> >; kernelnewbies <kernelnewbies@kernelnewbies.org <mailto:kernelnewbies@kernelnewbies.org> > Subject: Re: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build On Fri, Apr 22, 2022 at 7:30 PM Chan Kim <ckim@etri.re.kr <mailto:ckim@etri.re.kr> > wrote: Hello all, Really strange thing happening here.. I can't see the full stack trace with 'bt' command in gdb. So I tried with fresh linux-5.10.122 source and qemu-6.2.0 source and it's happening too! (it's happening when I do combinations with linux 5.10.0 and qemu-5.1.0. But it's not happening with linux-5.4.21) I would be grateful if somebody could check if this happens to other people or just me. 1. download linux-5.1.122 tarball from https://www.kernel.org/ 2. uncompress it and set env variable ARCH=arm64, CROSS_COMPILE=aarch64-none-elf- , do "make defconfig" and "make -j`nproc` Image" 3. download qemu-6.2.0 from https://www.qemu.org/ 4. uncompress it and do "mkdir build" "cd build" "../configure --target-list=aarch64-softmmu --enable-debug" 5. run qemu and wait for debugger to attach. qemu-6.2.0/build/aarch64-softmmu/qemu-system-aarch64 -machine virt,gic-version=max,secure=off,virtualization=true -cpu max -kernel linux-5.10.112/arch/arm64/boot/Image -m 2G -nographic -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/srv/tftp -device virtio-net-pci,netdev=vnet -machine iommu=smmuv3 --append "root=/dev/ram init=/init nokaslr earlycon ip=dhcp hugepages=16" -s -S 6. run debugger, do "aarch64-none-elf-gdb linux-6.10.112/vmlinux -x\ It has been long time since I compiled linux kernel but I guess, either you need to compile kernel with enabled frame pointer, and/or you need to enable debug symbol embedded into final kernel image. cmiiw gdb_script" (gdb_script content : target remote :1234 layout src b start_kernel b __driver_attach ) Now, in gdb, when you press 'c' twice, it'll stop at the first __driver_attach. (first one stops at start_kernel). When you are at __attach_driver, type 'bt'. See if you see the full function stack trace. This is what I see. (gdb) bt #0 __driver_attach (dev=0xffff000002582810, data=0xffff800011dc2358 <dummy_regulator_driver+40>) at drivers/base/dd.c:1060 #1 0xffff8000107a3ed0 in bus_for_each_dev (bus=<optimized out>, start=<optimized out>, data=0xffff800011dc2358 <dummy_regulator_driver+40>, fn=0xffff8000107a6f60 <__driver_attach>) at drivers/base/bus.c:305 #2 0xd6d78000107a5c58 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) I used to see more thatn 20 stacks trace but strangely I see only two. I can still see many stacks for linux-5.4.21 that I was working with in the past. Could anyone check if this happens to anyone? I think if I add BLK_DEV_RAM and set initramfs.cpio.gz in the linux build, the kernel will boot ok to the shell prompt. Only the gdb can't show the stack levels. My OS : ubuntu-20.04 5.13.0-35-generic $ aarch64-none-elf-gdb --version GNU gdb (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16)) 10.1.90.20201028-git Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Thank you. Chan Kim _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org <mailto:Kernelnewbies@kernelnewbies.org> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com <http://the-hydra.blogspot.com> training: mulyaditraining.blogspot.com <http://mulyaditraining.blogspot.com> [-- Attachment #1.2: Type: text/html, Size: 15182 bytes --] [-- Attachment #2: Type: text/plain, Size: 170 bytes --] _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
WARNING: multiple messages have this Message-ID (diff)
From: "Chan Kim" <ckim@etri.re.kr> To: "'qemu-devel'" <qemu-devel@nongnu.org>, "'kernelnewbies'" <kernelnewbies@kernelnewbies.org> Subject: RE: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build Date: Tue, 26 Apr 2022 10:46:45 +0900 [thread overview] Message-ID: <038001d8590f$7ccd7210$76685630$@etri.re.kr> (raw) In-Reply-To: [-- Attachment #1: Type: text/plain, Size: 6552 bytes --] Hello, I hope somebody could check this case. It’s easily reproducible for anybody working with qemu and arm64 linux. I returned to this problem and made another observation.(showing the back-trace is really broken). In another incident of breakpoint at function __driver_attach, (right after it stopped at the function) (gdb) bt #0 __driver_attach (dev=dev@entry=0xffff0000401d1810, data=data@entry=0xffff800011bbbbb8 <mxc_gpio_driver+40>) at drivers/base/dd.c:1046 #1 0xffff8000107684f8 in bus_for_each_dev (bus=0xffff800011cba910 <platform_bus_type>, start=0x0, data=0xffff800011bbbbb8 <mxc_gpio_driver+40>, fn=0xffff80001076b860 <__driver_attach>) at drivers/base/bus.c:307 #2 0xb8cd80001076a594 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) x/5g $sp 0xffff800011dcbcc0: 0xffff800011dcbd20 0xb8cd80001076a594 0xffff800011dcbcd0: 0xffff80001076b860 0xffff800011bbbbb8 0xffff800011dcbce0: 0x0000000000000000 Because it's right after the pc reached the function __driver_attach, the sp is still not updated from previous function (bus_for_each_dev). And the first two values at the $sp are supposed to be the fp and lr of the previous function (see https://stackoverflow.com/questions/66098678/understanding-aarch64-assembly-function-call-how-is-stack-operated arm64 stores previous function's fp and lr at the bottom of new stack frame as it enters a function). The lr (link register, the address to return after this bus_for_each_dev function) is really 0xb8cd80001076a594 which is weird (not a kernel address). The following 3 values are function arguments for bus_for_each_dev and they look correct. From: Chan Kim <ckim@etri.re.kr> Sent: Saturday, April 23, 2022 8:37 AM To: 'Mulyadi Santosa' <mulyadi.santosa@gmail.com> Cc: 'qemu-devel' <qemu-devel@nongnu.org>; 'kernelnewbies' <kernelnewbies@kernelnewbies.org> Subject: RE: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build Hi, Mulyadi Thank you for replying. I found CONFIG_DEBUG_FRAME_POINTER, CONFIG_DEBUG_INFO are already set by default. And I tried adding CONFIG_DEBUG_KERNEL, CONFIG_KGDB, CONFIG_GDB_SCRIPTS, CONFIG_STACKTRACE all to no avail. Regards, Chan From: Mulyadi Santosa <mulyadi.santosa@gmail.com <mailto:mulyadi.santosa@gmail.com> > Sent: Friday, April 22, 2022 11:53 PM To: Chan Kim <ckim@etri.re.kr <mailto:ckim@etri.re.kr> > Cc: qemu-devel <qemu-devel@nongnu.org <mailto:qemu-devel@nongnu.org> >; kernelnewbies <kernelnewbies@kernelnewbies.org <mailto:kernelnewbies@kernelnewbies.org> > Subject: Re: Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build On Fri, Apr 22, 2022 at 7:30 PM Chan Kim <ckim@etri.re.kr <mailto:ckim@etri.re.kr> > wrote: Hello all, Really strange thing happening here.. I can't see the full stack trace with 'bt' command in gdb. So I tried with fresh linux-5.10.122 source and qemu-6.2.0 source and it's happening too! (it's happening when I do combinations with linux 5.10.0 and qemu-5.1.0. But it's not happening with linux-5.4.21) I would be grateful if somebody could check if this happens to other people or just me. 1. download linux-5.1.122 tarball from https://www.kernel.org/ 2. uncompress it and set env variable ARCH=arm64, CROSS_COMPILE=aarch64-none-elf- , do "make defconfig" and "make -j`nproc` Image" 3. download qemu-6.2.0 from https://www.qemu.org/ 4. uncompress it and do "mkdir build" "cd build" "../configure --target-list=aarch64-softmmu --enable-debug" 5. run qemu and wait for debugger to attach. qemu-6.2.0/build/aarch64-softmmu/qemu-system-aarch64 -machine virt,gic-version=max,secure=off,virtualization=true -cpu max -kernel linux-5.10.112/arch/arm64/boot/Image -m 2G -nographic -netdev user,id=vnet,hostfwd=:127.0.0.1:0-:22,tftp=/srv/tftp -device virtio-net-pci,netdev=vnet -machine iommu=smmuv3 --append "root=/dev/ram init=/init nokaslr earlycon ip=dhcp hugepages=16" -s -S 6. run debugger, do "aarch64-none-elf-gdb linux-6.10.112/vmlinux -x\ It has been long time since I compiled linux kernel but I guess, either you need to compile kernel with enabled frame pointer, and/or you need to enable debug symbol embedded into final kernel image. cmiiw gdb_script" (gdb_script content : target remote :1234 layout src b start_kernel b __driver_attach ) Now, in gdb, when you press 'c' twice, it'll stop at the first __driver_attach. (first one stops at start_kernel). When you are at __attach_driver, type 'bt'. See if you see the full function stack trace. This is what I see. (gdb) bt #0 __driver_attach (dev=0xffff000002582810, data=0xffff800011dc2358 <dummy_regulator_driver+40>) at drivers/base/dd.c:1060 #1 0xffff8000107a3ed0 in bus_for_each_dev (bus=<optimized out>, start=<optimized out>, data=0xffff800011dc2358 <dummy_regulator_driver+40>, fn=0xffff8000107a6f60 <__driver_attach>) at drivers/base/bus.c:305 #2 0xd6d78000107a5c58 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) I used to see more thatn 20 stacks trace but strangely I see only two. I can still see many stacks for linux-5.4.21 that I was working with in the past. Could anyone check if this happens to anyone? I think if I add BLK_DEV_RAM and set initramfs.cpio.gz in the linux build, the kernel will boot ok to the shell prompt. Only the gdb can't show the stack levels. My OS : ubuntu-20.04 5.13.0-35-generic $ aarch64-none-elf-gdb --version GNU gdb (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16)) 10.1.90.20201028-git Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Thank you. Chan Kim _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org <mailto:Kernelnewbies@kernelnewbies.org> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies -- regards, Mulyadi Santosa Freelance Linux trainer and consultant blog: the-hydra.blogspot.com <http://the-hydra.blogspot.com> training: mulyaditraining.blogspot.com <http://mulyaditraining.blogspot.com> [-- Attachment #2: Type: text/html, Size: 15182 bytes --]
next prev parent reply other threads:[~2022-04-26 1:47 UTC|newest] Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-04-22 12:14 Backtrace stopped: previous frame identical to this frame (corrupt stack?) , even with fresh qemu and linux build Chan Kim 2022-04-22 12:14 ` Chan Kim 2022-04-22 14:52 ` Mulyadi Santosa 2022-04-22 14:52 ` Mulyadi Santosa 2022-04-22 23:36 ` Chan Kim 2022-04-22 23:36 ` Chan Kim 2022-04-26 1:46 ` Chan Kim [this message] 2022-04-26 1:46 ` Chan Kim 2022-04-27 9:05 ` Chan Kim 2022-04-27 9:05 ` Chan Kim 2022-04-27 9:19 ` Chan Kim 2022-04-27 9:19 ` Chan Kim
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to='038001d8590f$7ccd7210$76685630$@etri.re.kr' \ --to=ckim@etri.re.kr \ --cc=kernelnewbies@kernelnewbies.org \ --cc=qemu-devel@nongnu.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.