[Qemu-devel] bug: qemu-0.9.0 emulating mipsel (32-bit R3000) on amd64

[Qemu-devel] bug: qemu-0.9.0 emulating mipsel (32-bit R3000) on amd64

[John Reiser]

Hi,

qemu-0.9.0 compiled and running on Debian 2.6.18-4-amd64,
[compiled by gcc (GCC) 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)],
emulating Debian 2.6.18-4-qemu mipsel (32-bit MIPS R3000 little endian),
errs when gdb 6.4.90-debian (running on the emulated mipsel)
single-steps the user-mode instruction:
	lw      a2,-44(s7)
After single-stepping the 'lw', then register a2 contains garbage
instead of the memory contents at address -44(s7).
This is reproducible every time (and the bad value is the same.)
Also, executing the 'lw' by some means other than single stepping
(such as by setting a breakpoint _beyond_ the 'lw' and continuing)
apparently works correctly.

Here is the gdb console log:
-----
0x001060e4 in ?? ()
0x1060e4:       lw      a2,-44(s7)
(gdb) p $a2
$3 = 0x0
(gdb) x/x $s7-44
0x105458:       0x00120000
(gdb) g   # step one instruction and show next instruction
warning: GDB can't find the start of the function at 0x1060e8.
warning: GDB can't find the start of the function at 0x1060e8.
0x001060e8 in ?? ()
0x1060e8:       addiu   sp,sp,-32
(gdb) p $a2
$4 = 0x5000d
-----
where 'g' is a single-step macro for gdb:
   define g
   stepi
   x/i $pc
   end

I have posted the user-mode executable (28KB) at
    http://bitwagon.com/ftp/date.qemu-0.9.0.bug
The complete gdb session is eight (8) instructions:
-----
(gdb) set output-radix 16
(gdb) run
Program received signal SIGTRAP, Trace/breakpoint trap.
0x00105478 in ?? ()   # the entry point
(gdb) x/i $pc
0x105478:       break
(gdb) set $pc+=4   # skip over the 'break' at entry
(gdb) x/i $pc
0x10547c:       bal     0x10617c    # use 'g' or 'stepi' 5 times
0x105480:         addiu   s7,ra,0   # delay slot; executed but not stopped
0x10617c:       lw      s5,-48(s7)  # this 'lw' executes correctly
0x106180:       bal     0x1060e0
0x106184:         move    s6,ra   # delay slot
0x1060e0:       addiu   s5,s5,-92
0x1060e4:       lw      a2,-44(s7)  # bug: register a2 gets bad value
0x1060e8:       addiu   sp,sp,-32
(gdb) p $a2
$1 = 0x5000d   # (Even after allowing for 1-cycle load delay.)
(gdb) x/x $s7-44
0x105458:       0x00120000   # value that should be in register a2, but is not
-----

The emulated Debian system was installed using the directions at
   http://www.aurel32.net/info/debian_mips_qemu.php
and the actual installed kernel and initrd was:
   http://people.debian.org/~ths/d-i/mipsel/images/20070503-02:00/qemu/netboot/initrd.gz
   http://people.debian.org/~ths/d-i/mipsel/images/20070503-02:00/qemu/netboot/vmlinux-2.6.18-4-qemu
After installation, then the emulation is invoked by:
   qemu-system-mipsel -kernel vmlinux-2.6.18-4-qemu -initrd initrd.gz \
        -hda hda.img -append "root=/dev/hda1 console=ttyS0" -nographic \
        -net nic -net tap
The gdb transcript was copy+paste from an xterm running ssh into the
emulated system.  ["apt-get install ssh" on the emulated system.]

Please suggest how to find and fix this bug?
(It's hard to remember to avoid single-stepping 'lw'.)

-- 
John Reiser, jreiser@BitWagon.com

Re: [Qemu-devel] bug: qemu-0.9.0 emulating mipsel (32-bit R3000) on amd64

[Thiemo Seufer]

John Reiser wrote:
> Hi,
> 
> qemu-0.9.0 compiled and running on Debian 2.6.18-4-amd64,
> [compiled by gcc (GCC) 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)],
> emulating Debian 2.6.18-4-qemu mipsel (32-bit MIPS R3000 little endian),
> errs when gdb 6.4.90-debian (running on the emulated mipsel)
> single-steps the user-mode instruction:
> 	lw      a2,-44(s7)
> After single-stepping the 'lw', then register a2 contains garbage
> instead of the memory contents at address -44(s7).

This looks like another instance of "Qemu/MIPS doesn't handle
self-modifying code correctly" (the break instructions inserted
by gdb are exactly this).

A gross workaround is
http://lists.nongnu.org/archive/html/qemu-devel/2007-05/msg00037.html


Thiemo

Re: [Qemu-devel] bug: qemu-0.9.0 emulating mipsel (32-bit R3000) on amd64

[Thiemo Seufer]

ths wrote:
> John Reiser wrote:
> > Hi,
> > 
> > qemu-0.9.0 compiled and running on Debian 2.6.18-4-amd64,
> > [compiled by gcc (GCC) 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)],
> > emulating Debian 2.6.18-4-qemu mipsel (32-bit MIPS R3000 little endian),
> > errs when gdb 6.4.90-debian (running on the emulated mipsel)
> > single-steps the user-mode instruction:
> > 	lw      a2,-44(s7)
> > After single-stepping the 'lw', then register a2 contains garbage
> > instead of the memory contents at address -44(s7).
> 
> This looks like another instance of "Qemu/MIPS doesn't handle
> self-modifying code correctly" (the break instructions inserted
> by gdb are exactly this).
> 
> A gross workaround is
> http://lists.nongnu.org/archive/html/qemu-devel/2007-05/msg00037.html

That is, that's a workaround for another instance of the problem.


Thiemo

Re: [Qemu-devel] bug: qemu-0.9.0 emulating mipsel (32-bit R3000) on amd64

[Daniel Jacobowitz]

On Sun, May 06, 2007 at 09:04:52PM +0100, Thiemo Seufer wrote:
> This looks like another instance of "Qemu/MIPS doesn't handle
> self-modifying code correctly" (the break instructions inserted
> by gdb are exactly this).
> 
> A gross workaround is
> http://lists.nongnu.org/archive/html/qemu-devel/2007-05/msg00037.html

Someone might want to try:
  http://lists.nongnu.org/archive/html/qemu-devel/2007-04/msg00514.html

-- 
Daniel Jacobowitz
CodeSourcery

Re: [Qemu-devel] bug: qemu-0.9.0 emulating mipsel (32-bit R3000) on amd64

[Paul Brook]

On Monday 07 May 2007, Daniel Jacobowitz wrote:
> On Sun, May 06, 2007 at 09:04:52PM +0100, Thiemo Seufer wrote:
> > This looks like another instance of "Qemu/MIPS doesn't handle
> > self-modifying code correctly" (the break instructions inserted
> > by gdb are exactly this).
> >
> > A gross workaround is
> > http://lists.nongnu.org/archive/html/qemu-devel/2007-05/msg00037.html
>
> Someone might want to try:
>   http://lists.nongnu.org/archive/html/qemu-devel/2007-04/msg00514.html

I think breakpoints and FPU are separate bugs.

I notice that the FPU enable bit (CP0C1_FP) is not included in the TB flags. 
My guess is you need to modify exec.c:tb_find_fast to include this.

Paul

Re: [Qemu-devel] workaround: qemu-0.9.0 emulating mipsel (32-bit R3000) on amd64

[John Reiser]

Daniel Jacobowitz wrote:
> On Sun, May 06, 2007 at 09:04:52PM +0100, Thiemo Seufer wrote:
> 
>>This looks like another instance of "Qemu/MIPS doesn't handle
>>self-modifying code correctly" (the break instructions inserted
>>by gdb are exactly this).
>>
>>A gross workaround is
>>http://lists.nongnu.org/archive/html/qemu-devel/2007-05/msg00037.html
> 
> 
> Someone might want to try:
>   http://lists.nongnu.org/archive/html/qemu-devel/2007-04/msg00514.html
> 

This works for me so far: I can single-step reliably in gdb.

--- a/qemu-0.9.0/target-mips/helper.c   2007-05-05 15:39:21.000000000 -0700
+++ b/qemu-0.9.0/target-mips/helper.c   2007-05-07 13:24:50.000000000 -0700
@@ -358,6 +358,7 @@
         goto set_EPC;
     case EXCP_BREAK:
         cause = 9;
+       tlb_flush_page(env, env->PC);
         goto set_EPC;
     case EXCP_RI:
         cause = 10;


-- 
John Reiser, jreiser@BitWagon.com

Re: [Qemu-devel] workaround: qemu-0.9.0 emulating mipsel (32-bit R3000) on amd64

[Paul Brook]

On Monday 07 May 2007, John Reiser wrote:
> Daniel Jacobowitz wrote:
> > On Sun, May 06, 2007 at 09:04:52PM +0100, Thiemo Seufer wrote:
> >>This looks like another instance of "Qemu/MIPS doesn't handle
> >>self-modifying code correctly" (the break instructions inserted
> >>by gdb are exactly this).
> >>
> >>A gross workaround is
> >>http://lists.nongnu.org/archive/html/qemu-devel/2007-05/msg00037.html
> >
> > Someone might want to try:
> >   http://lists.nongnu.org/archive/html/qemu-devel/2007-04/msg00514.html
>
> This works for me so far: I can single-step reliably in gdb.
>
> --- a/qemu-0.9.0/target-mips/helper.c   2007-05-05 15:39:21.000000000 -0700
> +++ b/qemu-0.9.0/target-mips/helper.c   2007-05-07 13:24:50.000000000 -0700
> @@ -358,6 +358,7 @@
>          goto set_EPC;
>      case EXCP_BREAK:
>          cause = 9;
> +       tlb_flush_page(env, env->PC);
>          goto set_EPC;
>      case EXCP_RI:
>          cause = 10;

I think this is still broken if the breakpoint is the first instruction on a 
page. The changes Daniel mentioned should make this sort of flushing 
unnecessary.

Paul

Re: [Qemu-devel] workaround: qemu-0.9.0 emulating mipsel (32-bit R3000) on amd64

[Paul Brook]

> > --- a/qemu-0.9.0/target-mips/helper.c   2007-05-05 15:39:21.000000000
> > -0700 +++ b/qemu-0.9.0/target-mips/helper.c   2007-05-07
> > 13:24:50.000000000 -0700 @@ -358,6 +358,7 @@
> >          goto set_EPC;
> >      case EXCP_BREAK:
> >          cause = 9;
> > +       tlb_flush_page(env, env->PC);
> >          goto set_EPC;
> >      case EXCP_RI:
> >          cause = 10;
>
> I think this is still broken if the breakpoint is the first instruction on
> a page. The changes Daniel mentioned should make this sort of flushing
> unnecessary.

On second thoughts it probably does work, but I think it is the wrong way to 
fix this problem.

Paul

Re: [Qemu-devel] bug: qemu-0.9.0 emulating mipsel (32-bit R3000) on amd64

[Stefan Weil]

This won't help for the problems with MIPS FPU emulation, will it?

Both breakpoints and the FPU emulation in the Linux kernel use
self-modifying code, so there should be a general solution for
both (and more related) problems.

As long as this general solution is missing, the published code
patches help.

Stefan


Daniel Jacobowitz schrieb:
> On Sun, May 06, 2007 at 09:04:52PM +0100, Thiemo Seufer wrote:
>> This looks like another instance of "Qemu/MIPS doesn't handle
>> self-modifying code correctly" (the break instructions inserted
>> by gdb are exactly this).
>>
>> A gross workaround is
>> http://lists.nongnu.org/archive/html/qemu-devel/2007-05/msg00037.html
>
> Someone might want to try:
> http://lists.nongnu.org/archive/html/qemu-devel/2007-04/msg00514.html

Re: [Qemu-devel] bug: qemu-0.9.0 emulating mipsel (32-bit R3000) on amd64

[John Reiser]

>>qemu-0.9.0 ...
>>emulating Debian 2.6.18-4-qemu mipsel ...
>>errs when gdb 6.4.90-debian (running on the emulated mipsel)
>>single-steps the user-mode instruction ...

> This looks like another instance of "Qemu/MIPS doesn't handle
> self-modifying code correctly" (the break instructions inserted
> by gdb are exactly this).

No, the usage by gdb does *not* qualify as "self-modifying code."
gdb uses the system call ptrace(PTRACE_POKETEXT, pid, addr, data)
to have the emulated operating system kernel itself modify the memory
of the child process.  Nobody has to guess or to "snoop" the memory
bus in order to discover that the instruction stream is being modified.
Instead, there is direct notification of what is happening.  If nothing
else, then under CONFIG_QEMU the implementation of sys_ptrace()
must notify the emulator to flush the appropriate translations.

-- 
John Reiser, jreiser@BitWagon.com

Re: [Qemu-devel] bug: qemu-0.9.0 emulating mipsel (32-bit R3000) on amd64

[Thiemo Seufer]

John Reiser wrote:
> >>qemu-0.9.0 ...
> >>emulating Debian 2.6.18-4-qemu mipsel ...
> >>errs when gdb 6.4.90-debian (running on the emulated mipsel)
> >>single-steps the user-mode instruction ...
> 
> > This looks like another instance of "Qemu/MIPS doesn't handle
> > self-modifying code correctly" (the break instructions inserted
> > by gdb are exactly this).
> 
> No, the usage by gdb does *not* qualify as "self-modifying code."

In the context of Qemu system emulation it does...

> gdb uses the system call ptrace(PTRACE_POKETEXT, pid, addr, data)
> to have the emulated operating system kernel itself modify the memory
> of the child process.

... since "child processes" etc. run by the guest kernel are just a
foreign thing to Qemu.

> Nobody has to guess or to "snoop" the memory
> bus in order to discover that the instruction stream is being modified.
> Instead, there is direct notification of what is happening.

The Linux kernel happily does cache flushes, and Qemu happily ignores
them, since it doesn't implement a cache model. (A cache model is not
the answer. It would be slow, it would only paper over the problem,
it wouldn't help for uncached accesses or cacheless systems).

> If nothing
> else, then under CONFIG_QEMU the implementation of sys_ptrace()
> must notify the emulator to flush the appropriate translations.

Hacking special facilities in the guest kernel just to work around
a Qemu bug is IMHO the wrong approach.


Thiemo