* Kernel panic when net booting OpenBMC
@ 2020-10-14 23:42 Nirenjan Krishnan
2020-10-15 1:56 ` Lei Yu
2020-10-15 11:08 ` Deepak Kodihalli
0 siblings, 2 replies; 6+ messages in thread
From: Nirenjan Krishnan @ 2020-10-14 23:42 UTC (permalink / raw)
To: openbmc
Hi,
I’m trying to netboot an OpenBMC image on a custom target board that we are building. The board boots fine when we boot from flash, but when booting from the network, I run into the following kernel panic:
[ 1.195106] /dev/root: Can't open blockdev
[ 1.195425] VFS: Cannot open root device "nfs" or unknown-block(0,255): error -6
[ 1.195736] Please append a correct "root=" boot option; here are the available partitions:
[ 1.196391] 1f00 32768 mtdblock0
[ 1.196438] (driver?)
[ 1.196769] 1f01 384 mtdblock1
[ 1.196775] (driver?)
[ 1.197074] 1f02 128 mtdblock2
[ 1.197079] (driver?)
[ 1.197357] 1f03 4352 mtdblock3
[ 1.197362] (driver?)
[ 1.197647] 1f04 23808 mtdblock4
[ 1.197652] (driver?)
[ 1.197949] 1f05 4096 mtdblock5
[ 1.197955] (driver?)
[ 1.198246] 1f06 32768 mtdblock6
[ 1.198252] (driver?)
[ 1.198713] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,255)
[ 1.199310] CPU: 0 PID: 1 Comm: swapper Not tainted 5.8.0-671550e-dirty-3ae2c18 #1
[ 1.199860] Hardware name: Generic DT based system
[ 1.200228] Backtrace:
[ 1.200671] [<80106f1c>] (dump_backtrace) from [<80107170>] (show_stack+0x20/0x24)
[ 1.201155] r7:00008000 r6:808ec4b4 r5:00000000 r4:808fe258
[ 1.201512] [<80107150>] (show_stack) from [<807a5414>] (dump_stack+0x28/0x30)
[ 1.201882] [<807a53ec>] (dump_stack) from [<80116160>] (panic+0xf8/0x320)
[ 1.202229] r5:00000000 r4:80b84290
[ 1.202499] [<80116068>] (panic) from [<80a01aa0>] (mount_block_root+0x2e4/0x398)
[ 1.202881] r3:80b03008 r2:00000000 r1:8e4a1ec4 r0:808ec4b4
[ 1.203168] r7:00008000
[ 1.203316] [<80a017bc>] (mount_block_root) from [<80a01bcc>] (mount_root+0x78/0x94)
[ 1.203718] r10:8098cf04 r9:80a2983c r8:80a2985c r7:00000008 r6:8e4966e0 r5:80a29875
[ 1.204102] r4:000000ff
[ 1.204242] [<80a01b54>] (mount_root) from [<80a01d24>] (prepare_namespace+0x13c/0x194)
[ 1.204640] r5:80a29875 r4:80b84028
[ 1.204840] [<80a01be8>] (prepare_namespace) from [<80a0154c>] (kernel_init_freeable+0x1b0/0x1f8)
[ 1.205282] r5:80b84000 r4:00000096
[ 1.205495] [<80a0139c>] (kernel_init_freeable) from [<807bedc4>] (kernel_init+0x18/0x120)
[ 1.205913] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:807bedac
[ 1.206305] r4:00000000
[ 1.206454] [<807bedac>] (kernel_init) from [<80100128>] (ret_from_fork+0x14/0x2c)
[ 1.206898] Exception stack(0x8e4a1fb0 to 0x8e4a1ff8)
[ 1.207303] 1fa0: 00000000 00000000 00000000 00000000
[ 1.207831] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 1.208353] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 1.208739] r5:807bedac r4:00000000
[ 1.209505] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,255) ]---
I enabled CONFIG_NETWORK_FILESYSTEMS, CONFIG_NFS_FS and CONFIG_ROOT_NFS in the kernel configuration, rebuilt the kernel and verified that the flash boot works fine.
These are the commands that I ran to netboot
ast# tftp 83000000 uImage
ast# tftp 84000000 aspeed.dtb
ast# setenv serverip 10.x.x.x
ast# setenv bootargs root=/dev/nfs rw nfsroot=${serverip}:/data/nfs/test,nolock,tcp console=ttyS4,115200n8 mem.devmem=1 ip=${ipaddr}:${serverip}:${serverip}:255.0.0.0::eth0 nfsrootdebug
ast# bootm 83000000 - 84000000
I also used the fitImage directly instead of a separate uImage and DTB, and ran into the same error. I tried enabling CPIO image builds by adding the following line in local.conf:
IMAGE_FSTYPES += "cpio.${INITRAMFS_CTYPE}.u-boot"
However, if I use the generated CPIO image directly as the initramfs, then all I see is "Starting kernel ..." and no further output from the system.
Could you please suggest how I could go about debugging this? I've followed all the netboot suggestions from these threads on this mailing list:\
- https://lists.ozlabs.org/pipermail/openbmc/2018-June/012054.html
- https://lists.ozlabs.org/pipermail/openbmc/2015-October/000010.html
Thanks and regards
Nirenjan Krishnan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Kernel panic when net booting OpenBMC
2020-10-14 23:42 Kernel panic when net booting OpenBMC Nirenjan Krishnan
@ 2020-10-15 1:56 ` Lei Yu
2020-10-15 5:35 ` Nirenjan Krishnan
2020-10-15 11:08 ` Deepak Kodihalli
1 sibling, 1 reply; 6+ messages in thread
From: Lei Yu @ 2020-10-15 1:56 UTC (permalink / raw)
To: Nirenjan Krishnan; +Cc: openbmc
On Thu, Oct 15, 2020 at 7:43 AM Nirenjan Krishnan <nkrishnan@nvidia.com> wrote:
> Could you please suggest how I could go about debugging this? I've followed all the netboot suggestions from these threads on this mailing list:\
> - https://lists.ozlabs.org/pipermail/openbmc/2018-June/012054.html
> - https://lists.ozlabs.org/pipermail/openbmc/2015-October/000010.html
>
Probably you could follow this one
https://lists.ozlabs.org/pipermail/openbmc/2017-December/010264.html,
which successfully boots the BMC from TFTP + NFS rootfs.
--
BRs,
Lei YU
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Kernel panic when net booting OpenBMC
2020-10-15 1:56 ` Lei Yu
@ 2020-10-15 5:35 ` Nirenjan Krishnan
2020-10-15 6:10 ` Lei Yu
0 siblings, 1 reply; 6+ messages in thread
From: Nirenjan Krishnan @ 2020-10-15 5:35 UTC (permalink / raw)
To: Lei Yu; +Cc: openbmc
I tried that too, same error. I also tried adding `debug-init-sh enable-initrd-debug-sh` to bootargs to try and debug this further, but it panics before it gets to the debug shell.
On 10/14/20, 6:56 PM, "Lei Yu" <yulei.sh@bytedance.com> wrote:
External email: Use caution opening links or attachments
On Thu, Oct 15, 2020 at 7:43 AM Nirenjan Krishnan <nkrishnan@nvidia.com> wrote:
> Could you please suggest how I could go about debugging this? I've followed all the netboot suggestions from these threads on this mailing list:\
> - https://lists.ozlabs.org/pipermail/openbmc/2018-June/012054.html
> - https://lists.ozlabs.org/pipermail/openbmc/2015-October/000010.html
>
Probably you could follow this one
https://lists.ozlabs.org/pipermail/openbmc/2017-December/010264.html,
which successfully boots the BMC from TFTP + NFS rootfs.
--
BRs,
Lei YU
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Kernel panic when net booting OpenBMC
2020-10-15 5:35 ` Nirenjan Krishnan
@ 2020-10-15 6:10 ` Lei Yu
0 siblings, 0 replies; 6+ messages in thread
From: Lei Yu @ 2020-10-15 6:10 UTC (permalink / raw)
To: Nirenjan Krishnan; +Cc: openbmc
On Thu, Oct 15, 2020 at 1:35 PM Nirenjan Krishnan <nkrishnan@nvidia.com> wrote:
>
> I tried that too, same error. I also tried adding `debug-init-sh enable-initrd-debug-sh` to bootargs to try and debug this further, but it panics before it gets to the debug shell.
>
I would suggest checking:
1. If the kernel config really enables the NFS related configs, e.g.
CONFIG_NFS_FS, CONFIG_ROOT_NFS, etc.
2. If the NFS server is working correctly.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Kernel panic when net booting OpenBMC
2020-10-14 23:42 Kernel panic when net booting OpenBMC Nirenjan Krishnan
2020-10-15 1:56 ` Lei Yu
@ 2020-10-15 11:08 ` Deepak Kodihalli
2020-10-16 18:56 ` Nirenjan Krishnan
1 sibling, 1 reply; 6+ messages in thread
From: Deepak Kodihalli @ 2020-10-15 11:08 UTC (permalink / raw)
To: Nirenjan Krishnan; +Cc: openbmc
[-- Attachment #1: Type: text/plain, Size: 5360 bytes --]
Hi Nirenjan,
On Thu, Oct 15, 2020 at 5:13 AM Nirenjan Krishnan <nkrishnan@nvidia.com>
wrote:
> Hi,
>
> I’m trying to netboot an OpenBMC image on a custom target board that we
> are building. The board boots fine when we boot from flash, but when
> booting from the network, I run into the following kernel panic:
>
> [ 1.195106] /dev/root: Can't open blockdev
> [ 1.195425] VFS: Cannot open root device "nfs" or unknown-block(0,255):
> error -6
> [ 1.195736] Please append a correct "root=" boot option; here are the
> available partitions:
> [ 1.196391] 1f00 32768 mtdblock0
> [ 1.196438] (driver?)
> [ 1.196769] 1f01 384 mtdblock1
> [ 1.196775] (driver?)
> [ 1.197074] 1f02 128 mtdblock2
> [ 1.197079] (driver?)
> [ 1.197357] 1f03 4352 mtdblock3
> [ 1.197362] (driver?)
> [ 1.197647] 1f04 23808 mtdblock4
> [ 1.197652] (driver?)
> [ 1.197949] 1f05 4096 mtdblock5
> [ 1.197955] (driver?)
> [ 1.198246] 1f06 32768 mtdblock6
> [ 1.198252] (driver?)
> [ 1.198713] Kernel panic - not syncing: VFS: Unable to mount root fs on
> unknown-block(0,255)
> [ 1.199310] CPU: 0 PID: 1 Comm: swapper Not tainted
> 5.8.0-671550e-dirty-3ae2c18 #1
> [ 1.199860] Hardware name: Generic DT based system
> [ 1.200228] Backtrace:
> [ 1.200671] [<80106f1c>] (dump_backtrace) from [<80107170>]
> (show_stack+0x20/0x24)
> [ 1.201155] r7:00008000 r6:808ec4b4 r5:00000000 r4:808fe258
> [ 1.201512] [<80107150>] (show_stack) from [<807a5414>]
> (dump_stack+0x28/0x30)
> [ 1.201882] [<807a53ec>] (dump_stack) from [<80116160>]
> (panic+0xf8/0x320)
> [ 1.202229] r5:00000000 r4:80b84290
> [ 1.202499] [<80116068>] (panic) from [<80a01aa0>]
> (mount_block_root+0x2e4/0x398)
> [ 1.202881] r3:80b03008 r2:00000000 r1:8e4a1ec4 r0:808ec4b4
> [ 1.203168] r7:00008000
> [ 1.203316] [<80a017bc>] (mount_block_root) from [<80a01bcc>]
> (mount_root+0x78/0x94)
> [ 1.203718] r10:8098cf04 r9:80a2983c r8:80a2985c r7:00000008
> r6:8e4966e0 r5:80a29875
> [ 1.204102] r4:000000ff
> [ 1.204242] [<80a01b54>] (mount_root) from [<80a01d24>]
> (prepare_namespace+0x13c/0x194)
> [ 1.204640] r5:80a29875 r4:80b84028
> [ 1.204840] [<80a01be8>] (prepare_namespace) from [<80a0154c>]
> (kernel_init_freeable+0x1b0/0x1f8)
> [ 1.205282] r5:80b84000 r4:00000096
> [ 1.205495] [<80a0139c>] (kernel_init_freeable) from [<807bedc4>]
> (kernel_init+0x18/0x120)
> [ 1.205913] r10:00000000 r9:00000000 r8:00000000 r7:00000000
> r6:00000000 r5:807bedac
> [ 1.206305] r4:00000000
> [ 1.206454] [<807bedac>] (kernel_init) from [<80100128>]
> (ret_from_fork+0x14/0x2c)
> [ 1.206898] Exception stack(0x8e4a1fb0 to 0x8e4a1ff8)
> [ 1.207303] 1fa0: 00000000 00000000
> 00000000 00000000
> [ 1.207831] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000
> 00000000 00000000
> [ 1.208353] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
> [ 1.208739] r5:807bedac r4:00000000
> [ 1.209505] ---[ end Kernel panic - not syncing: VFS: Unable to mount
> root fs on unknown-block(0,255) ]---
>
> I enabled CONFIG_NETWORK_FILESYSTEMS, CONFIG_NFS_FS and CONFIG_ROOT_NFS in
> the kernel configuration, rebuilt the kernel and verified that the flash
> boot works fine.
>
> These are the commands that I ran to netboot
>
> ast# tftp 83000000 uImage
> ast# tftp 84000000 aspeed.dtb
> ast# setenv serverip 10.x.x.x
> ast# setenv bootargs root=/dev/nfs rw
> nfsroot=${serverip}:/data/nfs/test,nolock,tcp console=ttyS4,115200n8
> mem.devmem=1 ip=${ipaddr}:${serverip}:${serverip}:255.0.0.0::eth0
> nfsrootdebug
> ast# bootm 83000000 - 84000000
>
> I also used the fitImage directly instead of a separate uImage and DTB,
> and ran into the same error. I tried enabling CPIO image builds by adding
> the following line in local.conf:
>
> IMAGE_FSTYPES += "cpio.${INITRAMFS_CTYPE}.u-boot"
>
It seems like the rootfs can't be found. The fitImage based approach
usually works well for me. Can you elaborate what you did with the fitImage
based approach? I typically do this (on an AST2500) :
1) Add IMAGE_FSTYPES += "cpio.${INITRAMFS_CTYPE}.u-boot" to the conf
2) Copy fitImage and obmc-phosphor-image-<platform>.cpio.lzma.u-boot to the
tftp server.
3) Setup the tftp server and the BMC network
4) tftp 0x83000000 fitImage
5) tftp 0x93000000 initrd (in step 2 I had named the cpio image as initrd
on the tftp server). At this stage 'iminfo' should show what images have
been loaded.
6) bootm 0x83000000 0x93000000
If you want to instead boot from NFS, then I see Yu Lei has some
suggestions/questions for you.
Regards,
Deepak
> However, if I use the generated CPIO image directly as the initramfs, then
> all I see is "Starting kernel ..." and no further output from the system.
>
> Could you please suggest how I could go about debugging this? I've
> followed all the netboot suggestions from these threads on this mailing
> list:\
> - https://lists.ozlabs.org/pipermail/openbmc/2018-June/012054.html
> - https://lists.ozlabs.org/pipermail/openbmc/2015-October/000010.html
>
> Thanks and regards
> Nirenjan Krishnan
>
>
[-- Attachment #2: Type: text/html, Size: 6605 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Kernel panic when net booting OpenBMC
2020-10-15 11:08 ` Deepak Kodihalli
@ 2020-10-16 18:56 ` Nirenjan Krishnan
0 siblings, 0 replies; 6+ messages in thread
From: Nirenjan Krishnan @ 2020-10-16 18:56 UTC (permalink / raw)
To: Deepak Kodihalli; +Cc: openbmc
Hi Deepak,
> It seems like the rootfs can't be found. The fitImage based approach usually works well for me. Can you elaborate what you did with the fitImage
> based approach? I typically do this (on an AST2500) :
> 1) Add IMAGE_FSTYPES += "cpio.${INITRAMFS_CTYPE}.u-boot" to the conf
> 2) Copy fitImage and obmc-phosphor-image-<platform>.cpio.lzma.u-boot to the tftp server.
> 3) Setup the tftp server and the BMC network
> 4) tftp 0x83000000 fitImage
> 5) tftp 0x93000000 initrd (in step 2 I had named the cpio image as initrd on the tftp server). At this stage 'iminfo' should show what images have > been loaded.
> 6) bootm 0x83000000 0x93000000
I have added the CPIO image to local.conf, but I don't see an obmc-phosphor-image-<platform>.cpio.*.u-boot, only obmc-phosphor-initramfs-<platform>.cpio.xz.u-boot. I figured this might be what you intended, but that doesn't work either. Perhaps it's an issue with the load address and entry point? Both show as 00000000 on the generated CPIO image, but if I look at the initramfs section of the fitImage, then these addresses are "unavailable".
When I use the corresponding initramfs.cpio directly, the system hangs at Loading kernel, I don't see any output on the console after this. If I change the initramfs load address and entry point to an address inside DRAM, then it starts the kernel, but then I get a kernel panic as shown below.
[ 1.248074] /dev/root: Can't open blockdev
[ 1.248634] VFS: Cannot open root device "ram" or unknown-block(1,0): error -6
[ 1.249139] Please append a correct "root=" boot option; here are the available partitions:
[ 1.250056] 1f00 32768 mtdblock0
[ 1.250094] (driver?)
[ 1.250626] 1f01 384 mtdblock1
[ 1.250633] (driver?)
[ 1.251036] 1f02 128 mtdblock2
[ 1.251042] (driver?)
[ 1.251509] 1f03 4352 mtdblock3
[ 1.251515] (driver?)
[ 1.251978] 1f04 23808 mtdblock4
[ 1.251983] (driver?)
[ 1.252450] 1f05 4096 mtdblock5
[ 1.252456] (driver?)
[ 1.252938] 1f06 32768 mtdblock6
[ 1.252944] (driver?)
[ 1.253585] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)
One observation that I had is that by using the initramfs that is integrated into the fitImage, I'm able to boot into an initrd shell by appending the following to bootargs - debug-init-sh enable-initrd-debug-sh.
Based on that shell, it appears that the initrd does not pick up the kernel IP configuration, which is perhaps why the NFS mount is failing. Is there a standard location for the init script that is built into the initramfs?
Thanks and regards
Nirenjan Krishnan
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-10-16 19:24 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-14 23:42 Kernel panic when net booting OpenBMC Nirenjan Krishnan
2020-10-15 1:56 ` Lei Yu
2020-10-15 5:35 ` Nirenjan Krishnan
2020-10-15 6:10 ` Lei Yu
2020-10-15 11:08 ` Deepak Kodihalli
2020-10-16 18:56 ` Nirenjan Krishnan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).