Re: [Qemu-devel] qemu crashed when starting vm(kvm) with vnc connect

Re: [Qemu-devel] qemu crashed when starting vm(kvm) with vnc connect

[Stefan Hajnoczi]

On Tue, Apr 02, 2013 at 09:02:02AM +0000, Zhanghaoyu (A) wrote:
> I start a kvm VM with vnc(using the zrle protocol) connect, sometimes qemu program crashed during starting period, received signal SIGABRT.
> Trying about 20 times, this crash may be reproduced.
> I guess the cause memory corruption or double free.

Which version of QEMU are you running?

Please try qemu.git/master.

Stefan

Re: 答复: [Qemu-devel] qemu crashed when starting vm(kvm) with vnc connect

[Stefan Hajnoczi]

On Sun, Apr 07, 2013 at 04:58:07AM +0000, Zhanghaoyu (A) wrote:
> >> I start a kvm VM with vnc(using the zrle protocol) connect, sometimes qemu program crashed during starting period, received signal SIGABRT.
> >> Trying about 20 times, this crash may be reproduced.
> >> I guess the cause memory corruption or double free.
> >
> > Which version of QEMU are you running?
> > 
> > Please try qemu.git/master.
> > 
> > Stefan
> 
> I used the QEMU download from qemu.git (http://git.qemu.org/git/qemu.git).

Great, thanks!  Can you please post a backtrace?

The easiest way is:

  $ ulimit -c unlimited
  $ qemu-system-x86_64 -enable-kvm -m 1024 ...
  ...crash...
  $ gdb -c qemu-system-x86_64.core
  (gdb) bt

Depending on how your system is configured the core file might have a
different filename but there should be a file name *core* the current
working directory after the crash.

The backtrace will make it possible to find out where the crash
occurred.

Thanks,
Stefan

reply: reply: [Qemu-devel] qemu crashed when starting vm(kvm) with vnc connect

[Zhanghaoyu (A)]

On Sun, Apr 07, 2013 at 04:58:07AM +0000, Zhanghaoyu (A) wrote:
> >>> I start a kvm VM with vnc(using the zrle protocol) connect, sometimes qemu program crashed during starting period, received signal SIGABRT.
> >>> Trying about 20 times, this crash may be reproduced.
> >>> I guess the cause memory corruption or double free.
> >>
> >> Which version of QEMU are you running?
> >> 
> >> Please try qemu.git/master.
> >> 
> >> Stefan
> >
> >I used the QEMU download from qemu.git (http://git.qemu.org/git/qemu.git).
>
> Great, thanks!  Can you please post a backtrace?
> 
> The easiest way is:
> 
>  $ ulimit -c unlimited
>  $ qemu-system-x86_64 -enable-kvm -m 1024 ...
>  ...crash...
>  $ gdb -c qemu-system-x86_64.core
>  (gdb) bt
> 
> Depending on how your system is configured the core file might have a different filename but there should be a file name *core* the current working directory
after the crash.
> 
> The backtrace will make it possible to find out where the crash occurred.
> 
> Thanks,
> Stefan

backtrace from core file is shown as below:

Program received signal SIGABRT, Aborted.
0x00007f32eda3dd95 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f32eda3dd95 in raise () from /lib64/libc.so.6
#1  0x00007f32eda3f2ab in abort () from /lib64/libc.so.6
#2  0x00007f32eda77ece in __libc_message () from /lib64/libc.so.6
#3  0x00007f32eda7dc06 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007f32eda7ecda in _int_free () from /lib64/libc.so.6
#5  0x00007f32efd3452c in free_and_trace (mem=0x7f3290000cd0) at vl.c:2880
#6  0x00007f32efd251a1 in buffer_free (buffer=0x7f32f0c82890) at ui/vnc.c:505
#7  0x00007f32efd20c56 in vnc_zrle_clear (vs=0x7f32f0c762d0)
    at ui/vnc-enc-zrle.c:364
#8  0x00007f32efd26d07 in vnc_disconnect_finish (vs=0x7f32f0c762d0)
    at ui/vnc.c:1050
#9  0x00007f32efd275c5 in vnc_client_read (opaque=0x7f32f0c762d0)
    at ui/vnc.c:1349
#10 0x00007f32efcb397c in qemu_iohandler_poll (readfds=0x7f32f074d020,
    writefds=0x7f32f074d0a0, xfds=0x7f32f074d120, ret=1) at iohandler.c:124
#11 0x00007f32efcb46e8 in main_loop_wait (nonblocking=0) at main-loop.c:417
#12 0x00007f32efd31159 in main_loop () at vl.c:2133
#13 0x00007f32efd38070 in main (argc=46, argv=0x7fff7f5df178,
    envp=0x7fff7f5df2f0) at vl.c:4481

Zhang Haoyu

Re: [Qemu-devel] reply: reply: qemu crashed when starting vm(kvm) with vnc connect

[Stefan Hajnoczi]

On Mon, Apr 08, 2013 at 12:27:06PM +0000, Zhanghaoyu (A) wrote:
> On Sun, Apr 07, 2013 at 04:58:07AM +0000, Zhanghaoyu (A) wrote:
> > >>> I start a kvm VM with vnc(using the zrle protocol) connect, sometimes qemu program crashed during starting period, received signal SIGABRT.
> > >>> Trying about 20 times, this crash may be reproduced.
> > >>> I guess the cause memory corruption or double free.
> > >>
> > >> Which version of QEMU are you running?
> > >> 
> > >> Please try qemu.git/master.
> > >> 
> > >> Stefan
> > >
> > >I used the QEMU download from qemu.git (http://git.qemu.org/git/qemu.git).
> >
> > Great, thanks!  Can you please post a backtrace?
> > 
> > The easiest way is:
> > 
> >  $ ulimit -c unlimited
> >  $ qemu-system-x86_64 -enable-kvm -m 1024 ...
> >  ...crash...
> >  $ gdb -c qemu-system-x86_64.core
> >  (gdb) bt
> > 
> > Depending on how your system is configured the core file might have a different filename but there should be a file name *core* the current working directory
> after the crash.
> > 
> > The backtrace will make it possible to find out where the crash occurred.
> > 
> > Thanks,
> > Stefan
> 
> backtrace from core file is shown as below:
> 
> Program received signal SIGABRT, Aborted.
> 0x00007f32eda3dd95 in raise () from /lib64/libc.so.6
> (gdb) bt
> #0  0x00007f32eda3dd95 in raise () from /lib64/libc.so.6
> #1  0x00007f32eda3f2ab in abort () from /lib64/libc.so.6
> #2  0x00007f32eda77ece in __libc_message () from /lib64/libc.so.6
> #3  0x00007f32eda7dc06 in malloc_printerr () from /lib64/libc.so.6
> #4  0x00007f32eda7ecda in _int_free () from /lib64/libc.so.6
> #5  0x00007f32efd3452c in free_and_trace (mem=0x7f3290000cd0) at vl.c:2880
> #6  0x00007f32efd251a1 in buffer_free (buffer=0x7f32f0c82890) at ui/vnc.c:505
> #7  0x00007f32efd20c56 in vnc_zrle_clear (vs=0x7f32f0c762d0)
>     at ui/vnc-enc-zrle.c:364
> #8  0x00007f32efd26d07 in vnc_disconnect_finish (vs=0x7f32f0c762d0)
>     at ui/vnc.c:1050
> #9  0x00007f32efd275c5 in vnc_client_read (opaque=0x7f32f0c762d0)
>     at ui/vnc.c:1349
> #10 0x00007f32efcb397c in qemu_iohandler_poll (readfds=0x7f32f074d020,
>     writefds=0x7f32f074d0a0, xfds=0x7f32f074d120, ret=1) at iohandler.c:124
> #11 0x00007f32efcb46e8 in main_loop_wait (nonblocking=0) at main-loop.c:417
> #12 0x00007f32efd31159 in main_loop () at vl.c:2133
> #13 0x00007f32efd38070 in main (argc=46, argv=0x7fff7f5df178,
>     envp=0x7fff7f5df2f0) at vl.c:4481

CCing Corentin and Gerd who are more familiar with the VNC code than me.

Stefan

Re: [Qemu-devel] reply: reply: qemu crashed when starting vm(kvm) with vnc connect

[Gerd Hoffmann]

On 04/11/13 11:29, Stefan Hajnoczi wrote:
> On Mon, Apr 08, 2013 at 12:27:06PM +0000, Zhanghaoyu (A) wrote:
>> On Sun, Apr 07, 2013 at 04:58:07AM +0000, Zhanghaoyu (A) wrote:
>>>>>> I start a kvm VM with vnc(using the zrle protocol) connect, sometimes qemu program crashed during starting period, received signal SIGABRT.
>>>>>> Trying about 20 times, this crash may be reproduced.
>>>>>> I guess the cause memory corruption or double free.
>>>>>
>>>>> Which version of QEMU are you running?
>>>>>
>>>>> Please try qemu.git/master.

Please try again with latest master, might be fixed meanwhile.

If it still happens pleas provide full qemu and vnc client command lines.

>> backtrace from core file is shown as below:
>>
>> Program received signal SIGABRT, Aborted.

>> #8  0x00007f32efd26d07 in vnc_disconnect_finish (vs=0x7f32f0c762d0)
>>     at ui/vnc.c:1050

Do you have a vnc client connected?  Do you close it?

Any errors reported by the vnc client (maybe it disconnects due to an
error in the data stream)?

cheers,
  Gerd

reply: [Qemu-devel] reply: reply: qemu crashed when starting vm(kvm) with vnc connect

[Zhanghaoyu (A)]

> > On Mon, Apr 08, 2013 at 12:27:06PM +0000, Zhanghaoyu (A) wrote:
> >> On Sun, Apr 07, 2013 at 04:58:07AM +0000, Zhanghaoyu (A) wrote:
> >>>>>> I start a kvm VM with vnc(using the zrle protocol) connect, sometimes qemu program crashed during starting period, received signal SIGABRT.
> >>>>>> Trying about 20 times, this crash may be reproduced.
> >>>>>> I guess the cause memory corruption or double free.
> >>>>>
> >>>>> Which version of QEMU are you running?
> >>>>>
> >>>>> Please try qemu.git/master.
> 
> Please try again with latest master, might be fixed meanwhile.
> 
> If it still happens pleas provide full qemu and vnc client command lines.
> 
> >> backtrace from core file is shown as below:
> >>
> >> Program received signal SIGABRT, Aborted.
> 
> >> #8  0x00007f32efd26d07 in vnc_disconnect_finish (vs=0x7f32f0c762d0)
> >>     at ui/vnc.c:1050
> 
> Do you have a vnc client connected?  Do you close it?
> 
I have a vnc client connected, it was auto closed while qemu crashed.

> Any errors reported by the vnc client (maybe it disconnects due to an error in the data stream)?
> 
No errors reported by the vnc client, just popup a reconnect window.

And, I have tried to fix this bug, not reproduce this crash after tried about 100 times, patch is shown as below,
--- a/ui/vnc-jobs.c     2013-04-18 20:10:07.000000000 +0800
+++ b/ui/vnc-jobs.c     2013-04-18 20:14:06.000000000 +0800
@@ -234,7 +234,6 @@ static int vnc_worker_thread_loop(VncJob
         vnc_unlock_output(job->vs);
         goto disconnected;
     }
-    vnc_unlock_output(job->vs);

     /* Make a local copy of vs and switch output buffers */
     vnc_async_encoding_start(job->vs, &vs);
@@ -252,6 +251,8 @@ static int vnc_worker_thread_loop(VncJob

         if (job->vs->csock == -1) {
             vnc_unlock_display(job->vs->vd);
+            vnc_async_encoding_end(job->vs, &vs);
+            vnc_unlock_output(job->vs);
             goto disconnected;
         }

@@ -269,7 +270,6 @@ static int vnc_worker_thread_loop(VncJob
     vs.output.buffer[saved_offset] = (n_rectangles >> 8) & 0xFF;
     vs.output.buffer[saved_offset + 1] = n_rectangles & 0xFF;

-    vnc_lock_output(job->vs);
     if (job->vs->csock != -1) {
         buffer_reserve(&job->vs->jobs_buffer, vs.output.offset);
         buffer_append(&job->vs->jobs_buffer, vs.output.buffer,
@@ -278,6 +278,8 @@ static int vnc_worker_thread_loop(VncJob
         vnc_async_encoding_end(job->vs, &vs);

        qemu_bh_schedule(job->vs->bh);
+    } else {
+        vnc_async_encoding_end(job->vs, &vs);
     }
     vnc_unlock_output(job->vs);

Thanks,
Zhang Haoyu