illegal hardware instruction during MIPS-I ELF linux user emulation

illegal hardware instruction during MIPS-I ELF linux user emulation

[Libo Zhou]

Hi all,

I have an binary file generated by a cross compiler. The 'file' command says

$ file test
test: ELF 32-bit LSB LSB executable, MIPS, MIPS-I version 1 (SYSV), statically linked, with debug_info, not stripped.

When I executed it with

$./qemu-mipsel test
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
[1] 11088 illegal hardware instruction (core dumped) ./qemu-mipsel test

However, when I use another cross compiler that generates MIPS32 rel2 binaries, it worked fine.

Can anyone tell me what's going wrong?

Cheers,
Libo Zhou

Re:illegal hardware instruction during MIPS-I ELF linux user emulation

[Libo Zhou]

Any help guys? Is there a way to look at the ELF binary code to see if the instructions have invalid encoding scheme? Since I used a custom compiler that claims to have MIPS-I instructions. I doubt it though because 'file' command gives a reasonable output saying it surely is MIPS-I.

Re: illegal hardware instruction during MIPS-I ELF linux user emulation

[Peter Maydell]

On Mon, 23 Sep 2019 at 13:08, Libo Zhou <zhlb29@foxmail.com> wrote:
>
> Any help guys? Is there a way to look at the ELF binary code to see if the instructions have invalid encoding scheme? Since I used a custom compiler that claims to have MIPS-I instructions. I doubt it though because 'file' command gives a reasonable output saying it surely is MIPS-I.

'file' just looks at the header on the file to see what it
claims to be. It doesn't look through the rest of the file
to check what actual instructions the compiler emitted.
You can use 'objdump' if you want to disassemble a file.

I would start by using the QEMU gdbstub to connect a
MIPS-aware gdb. Then when the SIGILL arrives you can see
what instruction the guest program was trying to execute.

thanks
-- PMM

Re: illegal hardware instruction during MIPS-I ELF linux user emulation

[Philippe Mathieu-Daudé]

Hi Libo,

On 9/23/19 10:15 AM, Libo Zhou wrote:
> Hi all,
> 
> I have an binary file generated by a cross compiler. The 'file' command says
> 
> $ file test
> test: ELF 32-bit LSB LSB executable, MIPS, MIPS-I version 1 (SYSV), statically linked, with debug_info, not stripped.

If you look at the mips_defs[] array in
target/mips/translate_init.inc.c, the older ISA implemented is MIPS-II:

$ git grep .insn_flags target/mips/translate_init.inc.c
translate_init.inc.c:75:        .insn_flags = CPU_MIPS32,
translate_init.inc.c:97:        .insn_flags = CPU_MIPS32 | ASE_MIPS16,
translate_init.inc.c:117:        .insn_flags = CPU_MIPS32,
translate_init.inc.c:137:        .insn_flags = CPU_MIPS32 | ASE_MIPS16,
translate_init.inc.c:158:        .insn_flags = CPU_MIPS32R2,
translate_init.inc.c:179:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
translate_init.inc.c:201:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
translate_init.inc.c:223:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16
| ASE_DSP,
translate_init.inc.c:249:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
translate_init.inc.c:297:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16
| ASE_DSP | ASE_MT,
translate_init.inc.c:323:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16
| ASE_DSP | ASE_DSP_R2,
translate_init.inc.c:343:        .insn_flags = CPU_MIPS32R2 | ASE_MICROMIPS,
translate_init.inc.c:364:        .insn_flags = CPU_MIPS32R2 | ASE_MICROMIPS,
translate_init.inc.c:410:        .insn_flags = CPU_MIPS32R5 | ASE_MSA,
translate_init.inc.c:449:        .insn_flags = CPU_MIPS32R6 | ASE_MICROMIPS,
translate_init.inc.c:488:        .insn_flags = CPU_NANOMIPS32 | ASE_DSP
| ASE_DSP_R2 | ASE_DSP_R3 |
translate_init.inc.c:511:        .insn_flags = CPU_MIPS3,
translate_init.inc.c:531:        .insn_flags = CPU_VR54XX,
translate_init.inc.c:552:        .insn_flags = CPU_MIPS64,
translate_init.inc.c:578:        .insn_flags = CPU_MIPS64,
translate_init.inc.c:607:        .insn_flags = CPU_MIPS64 | ASE_MIPS3D,
translate_init.inc.c:636:        .insn_flags = CPU_MIPS64R2 | ASE_MIPS3D,
translate_init.inc.c:657:        .insn_flags = CPU_MIPS64R2,
translate_init.inc.c:681:        .insn_flags = CPU_MIPS64R2,
translate_init.inc.c:721:        .insn_flags = CPU_MIPS64R6 | ASE_MSA,
translate_init.inc.c:761:        .insn_flags = CPU_MIPS64R6 | ASE_MSA,
translate_init.inc.c:781:        .insn_flags = CPU_LOONGSON2E,
translate_init.inc.c:801:        .insn_flags = CPU_LOONGSON2F,
translate_init.inc.c:830:        .insn_flags = CPU_MIPS64R2 | ASE_DSP |
ASE_DSP_R2,

So currently there is no MIPS-I only CPU.
Note that the code got written with MIPS32 in mind, and implementing
MIPS-I requires a considerable amount of change in the codebase.

> When I executed it with
> 
> $./qemu-mipsel test
> qemu: uncaught target signal 4 (Illegal instruction) - core dumped
> [1] 11088 illegal hardware instruction (core dumped) ./qemu-mipsel test

You can try ./qemu-mipsel -d in_asm,int test to display debugging
information, you might see which instruction trapped.

IMO it is likely the RFE (Return from Exception) instruction.

Regards,

Phil.

> However, when I use another cross compiler that generates MIPS32 rel2 binaries, it worked fine.
> 
> Can anyone tell me what's going wrong?
> 
> Cheers,
> Libo Zhou
> 

Re: illegal hardware instruction during MIPS-I ELF linux user emulation

[Peter Maydell]

On Mon, 23 Sep 2019 at 15:40, Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> So currently there is no MIPS-I only CPU.
> Note that the code got written with MIPS32 in mind, and implementing
> MIPS-I requires a considerable amount of change in the codebase.

...but MIPS-I binaries should run on MIPS-II and newer CPUs, shouldn't
they?

> IMO it is likely the RFE (Return from Exception) instruction.

It seems unlikely that a linux userspace binary would be trying to
execute RFE...

thanks
-- PMM

Re: illegal hardware instruction during MIPS-I ELF linux user emulation

[Philippe Mathieu-Daudé]

On 9/23/19 4:42 PM, Peter Maydell wrote:
> On Mon, 23 Sep 2019 at 15:40, Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
>> So currently there is no MIPS-I only CPU.
>> Note that the code got written with MIPS32 in mind, and implementing
>> MIPS-I requires a considerable amount of change in the codebase.
> 
> ...but MIPS-I binaries should run on MIPS-II and newer CPUs, shouldn't
> they?

Some MIPS-I instructions where removed for MIPS-II (as RFE) and they are
not implemented. Also some CP0 registers are different.

>> IMO it is likely the RFE (Return from Exception) instruction.
> 
> It seems unlikely that a linux userspace binary would be trying to
> execute RFE...

Oh I thought it was system emulation, indeed it can't be RFE.

One GCC release targetting R3000 (Philips PR31700, Toshiba TX39) doesn't
emit NOP for branch's delay slot. I remember QEMU fails to run the
binaries it generates, but I don't remember how it fails.

Libo, can you provide more information about the cross-compiler you use
and the flags you use when calling it please?

Thanks,

Phil.

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Libo Zhou]

> I would start by using the QEMU gdbstub to connect a
> MIPS-aware gdb. Then when the SIGILL arrives you can see
> what instruction the guest program was trying to execute.

Just tried it and found something interesting.
I connected gdb-multiarch to QEMU gdbstub. gdb-multiarch's architecture was set to mips:3000 automatically (and Wikipedia says r3k uses MIPS-I).

When I did 'layout asm', and compared the instructions displayed against test.s generated by my mipsel-linux-unknown-gcc, they appeared to be a little bit different.

The 'store word' instruction in test.s is shown as a hex '0x7f......(don't remember the rest)';
'load word' is shown as '0x5f......';
'load immediate' is seen as 'addi';
'j' as 'jr';

When I single-stepped the instructions, the SIGILL was thrown immediately after the first unrecognized 0x7f......, which is supposed to be a store word (sw).
 
Hence, can I conclude that MIPS-I is not implemented in QEMU out of the box?

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Libo Zhou]

[-- Attachment #1: Type: text/plain, Size: 1090 bytes --]

> I would start by using the QEMU gdbstub to connect a
> MIPS-aware gdb. Then when the SIGILL arrives you can see
> what instruction the guest program was trying to execute.

Just tried it and found something interesting.
I connected gdb-multiarch to QEMU gdbstub. gdb-multiarch's architecture was set to mips:3000 automatically (and Wikipedia says r3k uses MIPS-I).

When I did 'layout asm', and compared the instructions displayed against test.s generated by my mipsel-linux-unknown-gcc, they appeared to be a little bit different.

The 'store word' instruction in test.s is shown as a hex '0x7f......(don't remember the rest)';
'load word' is shown as '0x5f......';
'load immediate' is seen as 'addi';
'j' as 'jr';

When I single-stepped the instructions, the SIGILL was thrown immediately after the first unrecognized 0x7f......, which is supposed to be a store word (sw).

Hence, can I conclude that MIPS-I is not implemented in QEMU out of the box?


EDIT: Or my compiler isn't implementing MIPS-I correctly because gdb-multiarch's r3k doesn't recognize some hexes?

[-- Attachment #2: Type: text/html, Size: 1166 bytes --]

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Libo Zhou]

> If you look at the mips_defs[] array in
> target/mips/translate_init.inc.c, the older ISA implemented is MIPS-II:

> $ git grep .insn_flags target/mips/translate_init.inc.c
> translate_init.inc.c:75:        .insn_flags = CPU_MIPS32,
> translate_init.inc.c:97:        .insn_flags = CPU_MIPS32 | ASE_MIPS16,
> translate_init.inc.c:117:        .insn_flags = CPU_MIPS32,
> translate_init.inc.c:137:        .insn_flags = CPU_MIPS32 | ASE_MIPS16,
> translate_init.inc.c:158:        .insn_flags = CPU_MIPS32R2,
> translate_init.inc.c:179:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
> translate_init.inc.c:201:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
> translate_init.inc.c:223:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16 | ASE_DSP,
> translate_init.inc.c:249:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
> translate_init.inc.c:297:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16 | ASE_DSP | ASE_MT,
> translate_init.inc.c:323:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16 | ASE_DSP | ASE_DSP_R2,
> translate_init.inc.c:343:        .insn_flags = CPU_MIPS32R2 | ASE_MICROMIPS,
> translate_init.inc.c:364:        .insn_flags = CPU_MIPS32R2 | ASE_MICROMIPS,
> translate_init.inc.c:410:        .insn_flags = CPU_MIPS32R5 | ASE_MSA,
> translate_init.inc.c:449:        .insn_flags = CPU_MIPS32R6 | ASE_MICROMIPS,
> translate_init.inc.c:488:        .insn_flags = CPU_NANOMIPS32 | ASE_DSP | ASE_DSP_R2 | ASE_DSP_R3 |
> translate_init.inc.c:511:        .insn_flags = CPU_MIPS3,
> translate_init.inc.c:531:        .insn_flags = CPU_VR54XX,
> translate_init.inc.c:552:        .insn_flags = CPU_MIPS64,
> translate_init.inc.c:578:        .insn_flags = CPU_MIPS64,
> translate_init.inc.c:607:        .insn_flags = CPU_MIPS64 | ASE_MIPS3D,
> translate_init.inc.c:636:        .insn_flags = CPU_MIPS64R2 | ASE_MIPS3D,
> translate_init.inc.c:657:        .insn_flags = CPU_MIPS64R2,
> translate_init.inc.c:681:        .insn_flags = CPU_MIPS64R2,
> translate_init.inc.c:721:        .insn_flags = CPU_MIPS64R6 | ASE_MSA,
> translate_init.inc.c:761:        .insn_flags = CPU_MIPS64R6 | ASE_MSA,
> translate_init.inc.c:781:        .insn_flags = CPU_LOONGSON2E,
> translate_init.inc.c:801:        .insn_flags = CPU_LOONGSON2F,
> translate_init.inc.c:830:        .insn_flags = CPU_MIPS64R2 | ASE_DSP | ASE_DSP_R2,

> So currently there is no MIPS-I only CPU.
> Note that the code got written with MIPS32 in mind, and implementing
> MIPS-I requires a considerable amount of change in the codebase.

Hi Philippe,

I just figured out what the problem was. The custom compiler I used just modified the opcode fields of sw and lw instructions of MIPS, so QEMU didn't recognize them out of the box.
I just added the support in decode_opc function in translate.c, and I also added my own CPU model in translate_init.inc.c. However, the illegal instruction exception is still there.

I am suspecting that the way I added my own CPU model in translate_init.inc.c is wrong. Below is what I added:
...
+{
+    .name = "MyCPU",
+    .insn_flags = CPU_MIPS1 | INSN_MYCPU,
+},
...
I just need to simulate it's instruction set in linux user emulation, I didn't include CP0* items in the list. Is this good enough to add a new CPU model?

Thanks,
Libo Zhou

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Philippe Mathieu-Daudé]

On 9/26/19 4:31 PM, Libo Zhou wrote:
>> If you look at the mips_defs[] array in
>> target/mips/translate_init.inc.c, the older ISA implemented is MIPS-II:
> 
>> $ git grep .insn_flags target/mips/translate_init.inc.c
>> translate_init.inc.c:75:        .insn_flags = CPU_MIPS32,
>> translate_init.inc.c:97:        .insn_flags = CPU_MIPS32 | ASE_MIPS16,
>> translate_init.inc.c:117:        .insn_flags = CPU_MIPS32,
>> translate_init.inc.c:137:        .insn_flags = CPU_MIPS32 | ASE_MIPS16,
>> translate_init.inc.c:158:        .insn_flags = CPU_MIPS32R2,
>> translate_init.inc.c:179:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
>> translate_init.inc.c:201:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
>> translate_init.inc.c:223:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16 | ASE_DSP,
>> translate_init.inc.c:249:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16,
>> translate_init.inc.c:297:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16 | ASE_DSP | ASE_MT,
>> translate_init.inc.c:323:        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16 | ASE_DSP | ASE_DSP_R2,
>> translate_init.inc.c:343:        .insn_flags = CPU_MIPS32R2 | ASE_MICROMIPS,
>> translate_init.inc.c:364:        .insn_flags = CPU_MIPS32R2 | ASE_MICROMIPS,
>> translate_init.inc.c:410:        .insn_flags = CPU_MIPS32R5 | ASE_MSA,
>> translate_init.inc.c:449:        .insn_flags = CPU_MIPS32R6 | ASE_MICROMIPS,
>> translate_init.inc.c:488:        .insn_flags = CPU_NANOMIPS32 | ASE_DSP | ASE_DSP_R2 | ASE_DSP_R3 |
>> translate_init.inc.c:511:        .insn_flags = CPU_MIPS3,
>> translate_init.inc.c:531:        .insn_flags = CPU_VR54XX,
>> translate_init.inc.c:552:        .insn_flags = CPU_MIPS64,
>> translate_init.inc.c:578:        .insn_flags = CPU_MIPS64,
>> translate_init.inc.c:607:        .insn_flags = CPU_MIPS64 | ASE_MIPS3D,
>> translate_init.inc.c:636:        .insn_flags = CPU_MIPS64R2 | ASE_MIPS3D,
>> translate_init.inc.c:657:        .insn_flags = CPU_MIPS64R2,
>> translate_init.inc.c:681:        .insn_flags = CPU_MIPS64R2,
>> translate_init.inc.c:721:        .insn_flags = CPU_MIPS64R6 | ASE_MSA,
>> translate_init.inc.c:761:        .insn_flags = CPU_MIPS64R6 | ASE_MSA,
>> translate_init.inc.c:781:        .insn_flags = CPU_LOONGSON2E,
>> translate_init.inc.c:801:        .insn_flags = CPU_LOONGSON2F,
>> translate_init.inc.c:830:        .insn_flags = CPU_MIPS64R2 | ASE_DSP | ASE_DSP_R2,
> 
>> So currently there is no MIPS-I only CPU.
>> Note that the code got written with MIPS32 in mind, and implementing
>> MIPS-I requires a considerable amount of change in the codebase.
> 
> Hi Philippe,
> 
> I just figured out what the problem was. The custom compiler I used just modified the opcode fields of sw and lw instructions of MIPS, so QEMU didn't recognize them out of the box.
> I just added the support in decode_opc function in translate.c, and I also added my own CPU model in translate_init.inc.c. However, the illegal instruction exception is still there.
> 
> I am suspecting that the way I added my own CPU model in translate_init.inc.c is wrong. Below is what I added:
> ...
> +{
> +    .name = "MyCPU",
> +    .insn_flags = CPU_MIPS1 | INSN_MYCPU,
> +},
> ...
> I just need to simulate it's instruction set in linux user emulation, I didn't include CP0* items in the list. Is this good enough to add a new CPU model?

Something like that might be acceptable for linux-user.
You should at least set CP0_PRid/Config0/Status_mask.

Look at this patch where Aleksandar accepted the R5900 CPU:
https://git.qemu.org/?p=qemu.git;a=commitdiff;h=ed4f49ba9bb56

Can you share what is your CPU?

Re: illegal hardware instruction during MIPS-I ELF linux useremulation

[Libo Zhou]

> > +{
> > +    .name = "MyCPU",
> > +    .insn_flags = CPU_MIPS1 | INSN_MYCPU,
> > +},
> > ...
> > I just need to simulate it's instruction set in linux user emulation, I didn't include CP0* items in the list. Is this good enough to add a new CPU model?

> Something like that might be acceptable for linux-user.
> You should at least set CP0_PRid/Config0/Status_mask.

> Look at this patch where Aleksandar accepted the R5900 CPU:
> https://git.qemu.org/?p=qemu.git;a=commitdiff;h=ed4f49ba9bb56

> Can you share what is your CPU?

It's working now. When I put the -cpu flag at the end of the command line, my cpu was not recognized.

Thanks,
Libo