Looking for advise on debugging a non-boot kernel on qemu-system-sh4

Looking for advise on debugging a non-boot kernel on qemu-system-sh4

[John Paul Adrian Glaubitz]

Hello!

I'm regularly building debian-installer packages for Debian's unofficial ports
which includes sh4 among others. The kernel package and therefore the installer
package contains a kernel for the SH7751R machine which is emulated by QEMU when
choosing the "r2d" type.

Unfortunately, I have not yet been able to boot a current kernel on qemu-system-sh4,
the screen remains blank and there are no error messages. Booting an older 2.6 kernel
works just fine.

I'm using qemu-system-sh4 as follows:

$ qemu-system-sh4 -M r2d -kernel vmlinuz-2.6.32-5-sh7751r -initrd initrd.gz -hda debian.img \
  -append "root=/dev/sda1 console=tty0 noiotrap"

The old 2.6 kernel from [1] boots fine while the current 5.14.x kernel from [2] does
not produce any output.

Can anyone enlighten me what I might be missing?

Thanks,
Adrian

PS: Please CC me as I am subscribed without getting messages.

> [1] https://people.debian.org/~aurel32/qemu/sh4/
> [2] https://cdimage.debian.org/cdimage/ports/current-debian-installer/sh4/

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Looking for advise on debugging a non-boot kernel on qemu-system-sh4

[BALATON Zoltan]

Hello,

On Thu, 21 Oct 2021, John Paul Adrian Glaubitz wrote:
> I'm regularly building debian-installer packages for Debian's unofficial ports
> which includes sh4 among others. The kernel package and therefore the installer
> package contains a kernel for the SH7751R machine which is emulated by QEMU when
> choosing the "r2d" type.
>
> Unfortunately, I have not yet been able to boot a current kernel on qemu-system-sh4,
> the screen remains blank and there are no error messages. Booting an older 2.6 kernel
> works just fine.
>
> I'm using qemu-system-sh4 as follows:
>
> $ qemu-system-sh4 -M r2d -kernel vmlinuz-2.6.32-5-sh7751r -initrd initrd.gz -hda debian.img \
>  -append "root=/dev/sda1 console=tty0 noiotrap"
>
> The old 2.6 kernel from [1] boots fine while the current 5.14.x kernel from [2] does
> not produce any output.
>
> Can anyone enlighten me what I might be missing?

Adding -d in_asm shows it seems to loop early in the kernel but not sure 
where. Maybe try to compare addresses with System.map to find out where 
it's getting stuck (but System.map was not included in your installer 
image). Also if it works on earlier kernel you might try to bisect which 
kernel commit caused the problem. Maybe knowing that helps to tell where 
to look further.

Regards,
BALATON Zoltan

Re: Looking for advise on debugging a non-boot kernel on qemu-system-sh4

[John Paul Adrian Glaubitz]

Hi Zoltan!

On 10/21/21 14:12, BALATON Zoltan wrote:
> Adding -d in_asm shows it seems to loop early in the kernel but not sure where.
> Maybe try to compare addresses with System.map to find out where it's getting
> stuck (but System.map was not included in your installer image).

Here is the System.map if that helps [1].

> Also if it works on earlier kernel you might try to bisect which kernel commit
> caused the problem. Maybe knowing that helps to tell where to look further.

If nothing else helps, I will try that.

Adrian

> [1] https://people.debian.org/~glaubitz/System.map-5.14.0-3-sh7751r.gz

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Looking for advise on debugging a non-boot kernel on qemu-system-sh4

[Thomas Huth]

On 21/10/2021 11.49, John Paul Adrian Glaubitz wrote:
> Hello!
> 
> I'm regularly building debian-installer packages for Debian's unofficial ports
> which includes sh4 among others. The kernel package and therefore the installer
> package contains a kernel for the SH7751R machine which is emulated by QEMU when
> choosing the "r2d" type.
> 
> Unfortunately, I have not yet been able to boot a current kernel on qemu-system-sh4,
> the screen remains blank and there are no error messages. Booting an older 2.6 kernel
> works just fine.
> 
> I'm using qemu-system-sh4 as follows:
> 
> $ qemu-system-sh4 -M r2d -kernel vmlinuz-2.6.32-5-sh7751r -initrd initrd.gz -hda debian.img \
>    -append "root=/dev/sda1 console=tty0 noiotrap"
> 
> The old 2.6 kernel from [1] boots fine while the current 5.14.x kernel from [2] does
> not produce any output.
> 
> Can anyone enlighten me what I might be missing?

I can't say much about very recent kernels, but FWIW, it was still working 
fine with kernel 4.9 in 2018, see:

  https://www.qemu-advent-calendar.org/2018/#day-9

In case it's just the video driver that is not working anymore, try:

  -append "console=ttySC1" -serial null -serial stdio

... that should hopefully redirect the kernel output to the serial console 
on stdio of the host terminal (at least it does so with the advent calendar 
image that I mentioned above).

  HTH,
   Thomas

Re: Looking for advise on debugging a non-boot kernel on qemu-system-sh4

[BALATON Zoltan]

On Thu, 21 Oct 2021, John Paul Adrian Glaubitz wrote:
> Hi Zoltan!
>
> On 10/21/21 14:12, BALATON Zoltan wrote:
>> Adding -d in_asm shows it seems to loop early in the kernel but not sure where.
>> Maybe try to compare addresses with System.map to find out where it's getting
>> stuck (but System.map was not included in your installer image).
>
> Here is the System.map if that helps [1].
>
>> Also if it works on earlier kernel you might try to bisect which kernel commit
>> caused the problem. Maybe knowing that helps to tell where to look further.
>
> If nothing else helps, I will try that.
>
> Adrian
>
>> [1] https://people.debian.org/~glaubitz/System.map-5.14.0-3-sh7751r.gz

I could not find any addresses that look like those in the map but I now 
see it seems to reboot on encountering an invalid instruction maybe before 
(during) uncomressing the kernel:

start:
0xac800000:  mov.l	0xac80007c,r1	! 0x500000f0
[,,,]
0x8c80085e:  mov.l	r1,@(4,r8)
0x8c800860:  bra	0x8c800b84
0x8c800862:  mov	r6,r0

----------------
IN:
0x8c80058c:  .word 0x0000

----------------
IN:
0xac800000:  mov.l	0xac80007c,r1	! 0x500000f0

and then this repeats. I wonder what's that zero opcode is and why is it 
there. Previously before it gets zero there was running it and there was 
still code there:

IN:
0x8c800ad4:  mov.l      r1,@(36,r9)
0x8c800ad6:  mov.l      @(28,r9),r1
0x8c800ad8:  mov.l      @(8,r15),r5
0x8c800ada:  mov.l      r1,@(32,r9)
0x8c800adc:  mov.l      @(24,r9),r1
0x8c800ade:  mov.l      r1,@(28,r9)
0x8c800ae0:  mov.l      @(20,r9),r1
0x8c800ae2:  mov.l      r1,@(24,r9)
0x8c800ae4:  mov.l      0x8c800c44,r1   ! 0x8c80058c
0x8c800ae6:  jsr        @r1
0x8c800ae8:  mov        r8,r4

----------------
IN:
0x8c80058c:  mov.l      r8,@-r15
0x8c80058e:  mov.l      r9,@-r15
0x8c800590:  mov.l      r10,@-r15
0x8c800592:  mov.l      r11,@-r15
0x8c800594:  mov.l      @r4,r2
0x8c800596:  mov.l      0x8c800718,r1   ! 0xffffff
0x8c800598:  mov.l      r12,@-r15
0x8c80059a:  cmp/hi     r1,r2
0x8c80059c:  bt.s       0x8c8005ba
0x8c80059e:  mov.l      r13,@-r15

So somthing seems to overwrite it. Maybe you can try building an 
uncompressed kernel or one using a different compression and see if that 
does the same, at least that way we can see if it's in the decompressing 
or later. I think it's past linux/arch/sh/boot/compressed/head32.S and 
maybe somewhere in decompress_kernel but not sure which decompression it 
uses. Kernel config is also missing to check but I probably give up at 
this point and let you experiment some more.

Regards,
BALATON Zoltan

Re: Looking for advise on debugging a non-boot kernel on qemu-system-sh4

[John Paul Adrian Glaubitz]

Hi Zoltan!

On 10/21/21 15:49, BALATON Zoltan wrote:
> So somthing seems to overwrite it. Maybe you can try building an uncompressed
> kernel or one using a different compression and see if that does the same, at
> least that way we can see if it's in the decompressing or later. I think it's
> past linux/arch/sh/boot/compressed/head32.S and maybe somewhere in decompress_kernel
> but not sure which decompression it uses. Kernel config is also missing to check
> but I probably give up at this point and let you experiment some more.

I think I've seen problems with compressed kernel images and QEMU before. I will switch
to an uncompressed kernel and try again.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Looking for advise on debugging a non-boot kernel on qemu-system-sh4

[BALATON Zoltan]

On Thu, 21 Oct 2021, John Paul Adrian Glaubitz wrote:
> On 10/21/21 15:49, BALATON Zoltan wrote:
>> So somthing seems to overwrite it. Maybe you can try building an uncompressed
>> kernel or one using a different compression and see if that does the same, at
>> least that way we can see if it's in the decompressing or later. I think it's
>> past linux/arch/sh/boot/compressed/head32.S and maybe somewhere in decompress_kernel
>> but not sure which decompression it uses. Kernel config is also missing to check
>> but I probably give up at this point and let you experiment some more.
>
> I think I've seen problems with compressed kernel images and QEMU before. I will switch
> to an uncompressed kernel and try again.

How did you compile the kernel that does not boot? What config have you 
used? I've tried to reproduce it by compiling a kernel with 
rts7751r2d1_defconfig and different compression methods but it did start 
and never got the problem seen with your kernel. Maybe it's the gcc 
version? My cross compiler is 8.4.0 and you seem to use 10.x. Maybe newer 
gcc uses something that's not emulated correctly? It would be interesting 
to identify what's causing the problem.

Regards,
BALATON Zoltan

Re: Looking for advise on debugging a non-boot kernel on qemu-system-sh4

[John Paul Adrian Glaubitz]

Hi Zoltan!

Thanks a lot for helping me to investigate the problem. Much appreciated!

On 10/22/21 23:06, BALATON Zoltan wrote:
>> I think I've seen problems with compressed kernel images and QEMU before. I will switch
>> to an uncompressed kernel and try again.
> 
> How did you compile the kernel that does not boot? What config have you used?

The config is constructed from the Debian kernel configuration tree. I have uploaded
the resulting config file here:

> https://people.debian.org/~glaubitz/config-5.14.0-3-sh7751r.gz

I've tried to reproduce it by compiling a kernel with rts7751r2d1_defconfig and different
> compression methods but it did start and never got the problem seen with your kernel.

Oh, that's very interesting. How big were the kernel images you got? My suspicion was
that the current Debian kernel might be too much.

> Maybe it's the gcc version? My cross compiler is 8.4.0 and you seem to use 10.x. Maybe
> newer gcc uses something that's not emulated correctly?

Yes, it has been built with gcc-10 which is currently Debian's default kernel for building
the kernel.

> It would be interesting to identify what's causing the problem.

Indeed. Thanks for helping me with that.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Looking for advise on debugging a non-boot kernel on qemu-system-sh4

[John Paul Adrian Glaubitz]

Hi Zoltan!

On 10/22/21 23:49, John Paul Adrian Glaubitz wrote:
>> How did you compile the kernel that does not boot? What config have you used?
> 
> The config is constructed from the Debian kernel configuration tree. I have uploaded
> the resulting config file here:
> 
>> https://people.debian.org/~glaubitz/config-5.14.0-3-sh7751r.gz
> 
>> I've tried to reproduce it by compiling a kernel with rts7751r2d1_defconfig and different
>> compression methods but it did start and never got the problem seen with your kernel.
> 
> Oh, that's very interesting. How big were the kernel images you got? My suspicion was
> that the current Debian kernel might be too much.

I can confirm that the default config works for me, too. Both with gcc-8 and gcc-11.

Using that kernel, however, I cannot use the debian-installer initrd.gz, even with
CONFIG_BLK_DEV_INITRD enabled in the kernel configuration.

The kernel prints a message saying that the initramfs is uncompressed, but whatever I
do I cannot get it to mount the initramfs. All I get is this:

[    0.096000] Trying to unpack rootfs image as initramfs...

and then later:

[    0.480000] ---[ end trace 46a3adcb34136251 ]---
[    0.480000] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    0.480000] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Looking for advise on debugging a non-boot kernel on qemu-system-sh4

[BALATON Zoltan]

On Sat, 23 Oct 2021, John Paul Adrian Glaubitz wrote:
> On 10/22/21 23:49, John Paul Adrian Glaubitz wrote:
>>> How did you compile the kernel that does not boot? What config have you used?
>>
>> The config is constructed from the Debian kernel configuration tree. I have uploaded
>> the resulting config file here:
>>
>>> https://people.debian.org/~glaubitz/config-5.14.0-3-sh7751r.gz
>>
>>> I've tried to reproduce it by compiling a kernel with rts7751r2d1_defconfig and different
>>> compression methods but it did start and never got the problem seen with your kernel.
>>
>> Oh, that's very interesting. How big were the kernel images you got? My suspicion was
>> that the current Debian kernel might be too much.
>
> I can confirm that the default config works for me, too. Both with gcc-8 and gcc-11.

OK with your config I can reproduce the problem too but the kernel with 
that config is 177MB and the r2d board has 64MB RAM so this can't work 
that way. Then it's likely not a but but a too big kernel.

> Using that kernel, however, I cannot use the debian-installer initrd.gz, even with
> CONFIG_BLK_DEV_INITRD enabled in the kernel configuration.
>
> The kernel prints a message saying that the initramfs is uncompressed, but whatever I
> do I cannot get it to mount the initramfs. All I get is this:
>
> [    0.096000] Trying to unpack rootfs image as initramfs...
>
> and then later:
>
> [    0.480000] ---[ end trace 46a3adcb34136251 ]---
> [    0.480000] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
> [    0.480000] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

I don't know, you have to find the needed config options to have what's 
needed to use this initrd. You could either try to strip down the debian 
config or add more to the default until you get a working kernel that fits 
in the memory (and there's some left to run other processes).

Regards,
BALATON Zoltan

Re: Looking for advise on debugging a non-boot kernel on qemu-system-sh4

[John Paul Adrian Glaubitz]

Hello Zoltan!

On 10/23/21 03:07, BALATON Zoltan wrote:
>> I can confirm that the default config works for me, too. Both with gcc-8 and gcc-11.
> 
> OK with your config I can reproduce the problem too but the kernel with that config
> is 177MB and the r2d board has 64MB RAM so this can't work that way. Then it's likely
> not a but but a too big kernel.

You either need to strip the kernel with "strip vmlinux" or use the image from arch/sh/
boot/zImage.

>> Using that kernel, however, I cannot use the debian-installer initrd.gz, even with
>> CONFIG_BLK_DEV_INITRD enabled in the kernel configuration.
>>
>> The kernel prints a message saying that the initramfs is uncompressed, but whatever I
>> do I cannot get it to mount the initramfs. All I get is this:
>>
>> [    0.096000] Trying to unpack rootfs image as initramfs...
>>
>> and then later:
>>
>> [    0.480000] ---[ end trace 46a3adcb34136251 ]---
>> [    0.480000] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
>> [    0.480000] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---
> 
> I don't know, you have to find the needed config options to have what's needed to use this initrd.
> You could either try to strip down the debian config or add more to the default until you get a
> working kernel that fits in the memory (and there's some left to run other processes).

I'll keep digging.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Looking for advise on debugging a non-boot kernel on qemu-system-sh4

[BALATON Zoltan]

Hello,

On Sat, 23 Oct 2021, John Paul Adrian Glaubitz wrote:
> On 10/23/21 03:07, BALATON Zoltan wrote:
>>> I can confirm that the default config works for me, too. Both with gcc-8 and gcc-11.
>>
>> OK with your config I can reproduce the problem too but the kernel with that config
>> is 177MB and the r2d board has 64MB RAM so this can't work that way. Then it's likely
>> not a but but a too big kernel.
>
> You either need to strip the kernel with "strip vmlinux" or use the image from arch/sh/
> boot/zImage.

I've actually used that kernel but looked at the wrong uncompressed size, 
it's indeed just 9.2MB when stripped so that should work. I was trying to 
debug further and found two problems:

Commit abb0cd93494 (accel/tcg: Split out log_cpu_exec) seems to have 
broken -singlestep -d in_asm,cpu output with sh after a delay slot. Since 
that commit I get:

pc=0xac80003e sr=0x500000f1 pr=0x00000000 fpscr=0x00040001
spc=0x00000000 ssr=0x00000000 gbr=0x00000000 vbr=0x00000000
sgr=0x00000000 dbr=0x00000000 delayed_pc=0x00000000 fpul=0x00000000
r0=0x8cc9d000 r1=0xacc9d000 r2=0xe0000000 r3=0x8c800000
r4=0x00000000 r5=0x00000000 r6=0x00000000 r7=0x00000000
r8=0x00000000 r9=0x00000000 r10=0x00000000 r11=0x00000000
r12=0x00000000 r13=0x00000000 r14=0x00000000 r15=0x00000000
r16=0x00000000 r17=0x500000f0 r18=0x00000000 r19=0x00000000
r20=0x00000000 r21=0x00000000 r22=0x00000000 r23=0x00000000
----------------
IN:
0xac800040:  bt.s       0xac80001a

pc=0xac800040 sr=0x500000f1 pr=0x00000000 fpscr=0x00040001
spc=0x00000000 ssr=0x00000000 gbr=0x00000000 vbr=0x00000000
sgr=0x00000000 dbr=0x00000000 delayed_pc=0x00000000 fpul=0x00000000
r0=0x8cc9cfe0 r1=0xacc9d000 r2=0xe0000000 r3=0x8c800000
r4=0x00000000 r5=0x00000000 r6=0x00000000 r7=0x00000000
r8=0x00000000 r9=0x00000000 r10=0x00000000 r11=0x00000000
r12=0x00000000 r13=0x00000000 r14=0x00000000 r15=0x00000000
r16=0x00000000 r17=0x500000f0 r18=0x00000000 r19=0x00000000
r20=0x00000000 r21=0x00000000 r22=0x00000000 r23=0x00000000
----------------
IN:
0xac800042:  add        #-32,r1

pc=0xac800042 sr=0x500000f1 pr=0x00000000 fpscr=0x00040001
spc=0x00000000 ssr=0x00000000 gbr=0x00000000 vbr=0x00000000
sgr=0x00000000 dbr=0x00000000 delayed_pc=0xac80001a fpul=0x00000000
r0=0x8cc9cfe0 r1=0xacc9d000 r2=0xe0000000 r3=0x8c800000
r4=0x00000000 r5=0x00000000 r6=0x00000000 r7=0x00000000
r8=0x00000000 r9=0x00000000 r10=0x00000000 r11=0x00000000
r12=0x00000000 r13=0x00000000 r14=0x00000000 r15=0x00000000
r16=0x00000000 r17=0x500000f0 r18=0x00000000 r19=0x00000000
r20=0x00000000 r21=0x00000000 r22=0x00000000 r23=0x00000000
in conditional delay slot (delayed_pc=0xac80001a)
pc=0xac80001a sr=0x500000f1 pr=0x00000000 fpscr=0x00040001
spc=0x00000000 ssr=0x00000000 gbr=0x00000000 vbr=0x00000000
sgr=0x00000000 dbr=0x00000000 delayed_pc=0xac80001a fpul=0x00000000
r0=0x8cc9cfe0 r1=0xacc9cfe0 r2=0xe0000000 r3=0x8c800000
r4=0x00000000 r5=0x00000000 r6=0x00000000 r7=0x00000000
r8=0x00000000 r9=0x00000000 r10=0x00000000 r11=0x00000000
r12=0x00000000 r13=0x00000000 r14=0x00000000 r15=0x00000000
r16=0x00000000 r17=0x500000f0 r18=0x00000000 r19=0x00000000
r20=0x00000000 r21=0x00000000 r22=0x00000000 r23=0x00000000
pc=0xac80001c sr=0x500000f1 pr=0x00000000 fpscr=0x00040001
spc=0x00000000 ssr=0x00000000 gbr=0x00000000 vbr=0x00000000
sgr=0x00000000 dbr=0x00000000 delayed_pc=0xac80001a fpul=0x00000000
r0=0x8cc9cfe0 r1=0xacc9cfe0 r2=0xe0000000 r3=0x8c800000
r4=0x00000000 r5=0x00000000 r6=0x00000000 r7=0x00000000
r8=0x00000000 r9=0x00000000 r10=0x00000000 r11=0x00000000
r12=0x00000000 r13=0x00000000 r14=0x00000000 r15=0x00000000
r16=0x00000000 r17=0x500000f0 r18=0x00000000 r19=0x00000000
r20=0x00000000 r21=0x00000000 r22=0x00000000 r23=0x00000000

After the first delay slot no more in_asm output is printed. Going back to 
the commit before I get normal output. Then running zImage with your 
config I see:

----------------
IN:
0x8c801574:  bra        0x8c801528

pc=0x8c801574 sr=0x500000f0 pr=0x8c8013d6 fpscr=0x00040001
spc=0x00000000 ssr=0x00000000 gbr=0x00000000 vbr=0x00000000
sgr=0x00000000 dbr=0x00000000 delayed_pc=0x8c801594 fpul=0x00000000
r0=0x00000007 r1=0x0000000e r2=0x8cca1084 r3=0xfffffff9
r4=0x00000137 r5=0xfffffffa r6=0x8cca1570 r7=0x00000012
r8=0x8cca1044 r9=0x00000011 r10=0x00000005 r11=0x00097d36
r12=0x8cca1014 r13=0x0000000f r14=0x8cc0183c r15=0x8cca0f80
r16=0x00000000 r17=0x500000f0 r18=0x00000000 r19=0x00000000
r20=0x00000000 r21=0x00000000 r22=0x00000000 r23=0x00000000
----------------
IN:
0x8c801576:  add        #-7,r5

pc=0x8c801576 sr=0x500000f0 pr=0x8c8013d6 fpscr=0x00040001
spc=0x00000000 ssr=0x00000000 gbr=0x00000000 vbr=0x00000000
sgr=0x00000000 dbr=0x00000000 delayed_pc=0x8c801528 fpul=0x00000000
r0=0x00000007 r1=0x0000000e r2=0x8cca1084 r3=0xfffffff9
r4=0x00000137 r5=0xfffffffa r6=0x8cca1570 r7=0x00000012
r8=0x8cca1044 r9=0x00000011 r10=0x00000005 r11=0x00097d36
r12=0x8cca1014 r13=0x0000000f r14=0x8cc0183c r15=0x8cca0f80
r16=0x00000000 r17=0x500000f0 r18=0x00000000 r19=0x00000000
r20=0x00000000 r21=0x00000000 r22=0x00000000 r23=0x00000000
in delay slot (delayed_pc=0x8c801528)
----------------
IN:
0x8c800964:  .word 0x0000

pc=0x8c800964 sr=0x500000f1 pr=0x8c801654 fpscr=0x00040001
spc=0x00000000 ssr=0x00000000 gbr=0x00000000 vbr=0x00000000
sgr=0x00000000 dbr=0x00000000 delayed_pc=0x8c800964 fpul=0x00000000
r0=0x0000001b r1=0xac8009ca r2=0x8cc9956d r3=0xfffffefe
r4=0x8cca1014 r5=0x00000000 r6=0x0142850a r7=0x8cc5001e
r8=0x8cca1044 r9=0x00000102 r10=0x00000000 r11=0x00000000
r12=0xac8009ca r13=0xac8009aa r14=0x00000000 r15=0x8cca0f28
r16=0x00000000 r17=0x500000f0 r18=0x00000000 r19=0x00000000
r20=0x00000000 r21=0x00000000 r22=0x00000000 r23=0x00000000
----------------
IN:
0xac800000:  mov.l      0xac80007c,r1   ! 0x500000f0

pc=0xac800000 sr=0x700000f0 pr=0x00000000 fpscr=0x00040001
spc=0x00000000 ssr=0x00000000 gbr=0x00000000 vbr=0x00000000
sgr=0x00000000 dbr=0x00000000 delayed_pc=0x00000000 fpul=0x00000000
r0=0x00000000 r1=0x00000000 r2=0x00000000 r3=0x00000000
r4=0x00000000 r5=0x00000000 r6=0x00000000 r7=0x00000000
r8=0x00000000 r9=0x00000000 r10=0x00000000 r11=0x00000000
r12=0x00000000 r13=0x00000000 r14=0x00000000 r15=0x00000000
r16=0x00000000 r17=0x00000000 r18=0x00000000 r19=0x00000000
r20=0x00000000 r21=0x00000000 r22=0x00000000 r23=0x00000000

This seems to take a wrong turn at the delayed branch and somehow ends up 
at 0x8c800964 instead of 0x8c801528 but I'm not sure where to look firther 
why. I'm cc-ing Richard for both the -d cpu and this hoping he has some 
more insight.

Regards,
BALATON Zoltan

Re: Looking for advise on debugging a non-boot kernel on qemu-system-sh4

[John Paul Adrian Glaubitz]

Hi Zoltan!

On 10/23/21 15:22, BALATON Zoltan wrote:
>> You either need to strip the kernel with "strip vmlinux" or use the image from arch/sh/
>> boot/zImage.
> 
> I've actually used that kernel but looked at the wrong uncompressed size, it's indeed just
> 9.2MB when stripped so that should work. I was trying to debug further and found two problems:
> 
> Commit abb0cd93494 (accel/tcg: Split out log_cpu_exec) seems to have broken -singlestep -d in_asm,cpu
> output with sh after a delay slot. Since that commit I get:
> (...) 
> This seems to take a wrong turn at the delayed branch and somehow ends up at 0x8c800964 instead of
> 0x8c801528 but I'm not sure where to look firther why. I'm cc-ing Richard for both the -d cpu and
> this hoping he has some more insight.

Shall we open a bug report?

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Looking for advise on debugging a non-boot kernel on qemu-system-sh4

[BALATON Zoltan]

On Tue, 26 Oct 2021, John Paul Adrian Glaubitz wrote:
> Hi Zoltan!
>
> On 10/23/21 15:22, BALATON Zoltan wrote:
>>> You either need to strip the kernel with "strip vmlinux" or use the image from arch/sh/
>>> boot/zImage.
>>
>> I've actually used that kernel but looked at the wrong uncompressed size, it's indeed just
>> 9.2MB when stripped so that should work. I was trying to debug further and found two problems:
>>
>> Commit abb0cd93494 (accel/tcg: Split out log_cpu_exec) seems to have broken -singlestep -d in_asm,cpu
>> output with sh after a delay slot. Since that commit I get:
>> (...)
>> This seems to take a wrong turn at the delayed branch and somehow ends up at 0x8c800964 instead of
>> 0x8c801528 but I'm not sure where to look firther why. I'm cc-ing Richard for both the -d cpu and
>> this hoping he has some more insight.
>
> Shall we open a bug report?

Well, we don't know yet what to put in the bug report apart from there is 
some bug somewhere. That's not too useful. I now understand that the -d 
output is not showing already translated TBs (I knew this but most of the 
time with -singlestep it gives good results anyway) but here it runs the 
loops without further output then we only see the first loop iteration and 
the end result. So the problem is not that it goes to 0x8c800964 as I 
think that's part of the loop for decompressing the kernel but it seems 
something is overwriting 0x8c800964 while it still expects to run code 
from there but I don't know what and why. One way to find could be to 
disassemble the kernel code and compare that with the -d output and check 
every instruction manually but that takes a lot of time or if you have a 
cross debugger you could try attaching that to QEMU and try to debug it 
that way but I don't have that either. Any other idea how to find out what 
is happening?

Regards,
BALATON Zoltan

Re: Looking for advise on debugging a non-boot kernel on qemu-system-sh4

[John Paul Adrian Glaubitz]

Hi Zoltan!

On 10/26/21 00:40, BALATON Zoltan wrote:
> On Tue, 26 Oct 2021, John Paul Adrian Glaubitz wrote:
>> Hi Zoltan!
>>
>> On 10/23/21 15:22, BALATON Zoltan wrote:
>>>> You either need to strip the kernel with "strip vmlinux" or use the image from arch/sh/
>>>> boot/zImage.
>>>
>>> I've actually used that kernel but looked at the wrong uncompressed size, it's indeed just
>>> 9.2MB when stripped so that should work. I was trying to debug further and found two problems:
>>>
>>> Commit abb0cd93494 (accel/tcg: Split out log_cpu_exec) seems to have broken -singlestep -d in_asm,cpu
>>> output with sh after a delay slot. Since that commit I get:
>>> (...)
>>> This seems to take a wrong turn at the delayed branch and somehow ends up at 0x8c800964 instead of
>>> 0x8c801528 but I'm not sure where to look firther why. I'm cc-ing Richard for both the -d cpu and
>>> this hoping he has some more insight.
>>
>> Shall we open a bug report?
> 
> Well, we don't know yet what to put in the bug report apart from there is some bug somewhere. That's
> not too useful. I now understand that the -d output is not showing already translated TBs (I knew this
> but most of the time with -singlestep it gives good results anyway) but here it runs the loops without
> further output then we only see the first loop iteration and the end result. So the problem is not that
> it goes to 0x8c800964 as I think that's part of the loop for decompressing the kernel but it seems
> something is overwriting 0x8c800964 while it still expects to run code from there but I don't know what
> and why. One way to find could be to disassemble the kernel code and compare that with the -d output and
> check every instruction manually but that takes a lot of time or if you have a cross debugger you could
> try attaching that to QEMU and try to debug it that way but I don't have that either. Any other idea how
> to find out what is happening?

Robert Święcki (CC'ed) found out that disabling tracing support makes Debian's kernel bootable [1].

Not sure if this is a kernel bug or a QEMU bug then. Does QEMU have any support for kernel tracing?

Adrian

> [1] https://marc.info/?l=linux-sh&m=164193147916418&w=2

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Looking for advise on debugging a non-boot kernel on qemu-system-sh4

[BALATON Zoltan]

[-- Attachment #1: Type: text/plain, Size: 3068 bytes --]

Hello,

On Wed, 12 Jan 2022, John Paul Adrian Glaubitz wrote:
> Hi Zoltan!
>
> On 10/26/21 00:40, BALATON Zoltan wrote:
>> On Tue, 26 Oct 2021, John Paul Adrian Glaubitz wrote:
>>> Hi Zoltan!
>>>
>>> On 10/23/21 15:22, BALATON Zoltan wrote:
>>>>> You either need to strip the kernel with "strip vmlinux" or use the image from arch/sh/
>>>>> boot/zImage.
>>>>
>>>> I've actually used that kernel but looked at the wrong uncompressed size, it's indeed just
>>>> 9.2MB when stripped so that should work. I was trying to debug further and found two problems:
>>>>
>>>> Commit abb0cd93494 (accel/tcg: Split out log_cpu_exec) seems to have broken -singlestep -d in_asm,cpu
>>>> output with sh after a delay slot. Since that commit I get:
>>>> (...)
>>>> This seems to take a wrong turn at the delayed branch and somehow ends up at 0x8c800964 instead of
>>>> 0x8c801528 but I'm not sure where to look firther why. I'm cc-ing Richard for both the -d cpu and
>>>> this hoping he has some more insight.
>>>
>>> Shall we open a bug report?
>>
>> Well, we don't know yet what to put in the bug report apart from there is some bug somewhere. That's
>> not too useful. I now understand that the -d output is not showing already translated TBs (I knew this
>> but most of the time with -singlestep it gives good results anyway) but here it runs the loops without
>> further output then we only see the first loop iteration and the end result. So the problem is not that
>> it goes to 0x8c800964 as I think that's part of the loop for decompressing the kernel but it seems
>> something is overwriting 0x8c800964 while it still expects to run code from there but I don't know what
>> and why. One way to find could be to disassemble the kernel code and compare that with the -d output and
>> check every instruction manually but that takes a lot of time or if you have a cross debugger you could
>> try attaching that to QEMU and try to debug it that way but I don't have that either. Any other idea how
>> to find out what is happening?
>
> Robert Święcki (CC'ed) found out that disabling tracing support makes Debian's kernel bootable [1].
>
> Not sure if this is a kernel bug or a QEMU bug then. Does QEMU have any support for kernel tracing?
>
> Adrian
>
>> [1] https://marc.info/?l=linux-sh&m=164193147916418&w=2

I don't think there's any support in QEMU that interacts with kernel 
tracing. I think more likely is that disabling this option either avoids 
some code path in the kernel where the problem happens or it changes the 
memory layout (e.g. making the kernel take less space) so it does not 
overwrite the part where it crashes (if it's really a problem with writing 
to wrong address it may still corrupt something else but that may not 
cause immediate crash). So while it boots it may still have a problem 
later if it happens to write to used memory. Best way would be to debug 
with cross debugger and find where exactly it's crashing and find out why 
but I don't have a cross debugger for SH4 at the moment nor time to look 
into this.

Regards,
BALATON Zoltan