[Qemu-devel] Crash with odd chardev setup

[Qemu-devel] Crash with odd chardev setup

[Dr. David Alan Gilbert]

Hi Dan,
  I've got a crash in head (and 2.10) which is a bit of a heisenbug;
I can trigger it with:

./qemu-system-x86_64 -netdev tap,id=hostnet0,vhost=on,fd=10   -chardev socket,id=charchannel0,path=/tmp/org.qemu.guest_agent.0,server,nowait  -monitor stdio -vnc :0

and then 'q' to quit.

Note I'm not doing a redirect in of fd 10.
It goes away if I remove either the -netdev or the -chardev option.

It doesn't trigger under gdb, but fortunately we get a core:

#0  0x000055a226d94a2e in socket_listen_cleanup (fd=<optimized out>, errp=errp@entry=0x7fff3585e8c0)
    at /root/qemu/util/qemu-sockets.c:1077
    1077	    if (addr->type == SOCKET_ADDRESS_TYPE_UNIX
    1078	        && addr->u.q_unix.path) {
    1079	        if (unlink(addr->u.q_unix.path) < 0 && errno != ENOENT) {

#1  0x000055a226d4242b in qio_channel_socket_finalize (obj=<optimized out>) at /root/qemu/io/channel-socket.c:388
#2  0x000055a226cbca92 in object_unref (type=<optimized out>, obj=0x55a2290ab400) at /root/qemu/qom/object.c:462
#3  0x000055a226cbca92 in object_unref (data=0x55a2290ab400) at /root/qemu/qom/object.c:476
#4  0x000055a226cbca92 in object_unref (obj=0x55a2290ab400) at /root/qemu/qom/object.c:911
#5  0x000055a226d334c9 in char_socket_finalize (obj=<optimized out>) at /root/qemu/chardev/char-socket.c:805
#6  0x000055a226cbca92 in object_unref (type=<optimized out>, obj=0x55a2290aa7f0) at /root/qemu/qom/object.c:462
#7  0x000055a226cbca92 in object_unref (data=0x55a2290aa7f0) at /root/qemu/qom/object.c:476
#8  0x000055a226cbca92 in object_unref (obj=0x55a2290aa7f0) at /root/qemu/qom/object.c:911
#9  0x000055a226cbcacc in object_unref (obj=0x55a2290ab7a0) at /root/qemu/qom/object.c:413
#10 0x000055a226cbcacc in object_unref (data=0x55a2290ab7a0) at /root/qemu/qom/object.c:475
#11 0x000055a226cbcacc in object_unref (obj=0x55a2290ab7a0) at /root/qemu/qom/object.c:911
#12 0x000055a226cbba87 in object_property_del_child (obj=0x55a22909e770, child=0x55a2290ab7a0, errp=0x0)
    at /root/qemu/qom/object.c:436
#13 0x000055a226cbc2e4 in object_unparent (obj=<optimized out>) at /root/qemu/qom/object.c:455
#14 0x000055a226d2e7e5 in qemu_chr_cleanup () at /root/qemu/chardev/char.c:1090
#15 0x000055a2269c73ff in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
    at /root/qemu/vl.c:4914

Dave

--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] Crash with odd chardev setup

[Daniel P. Berrange]

On Wed, Oct 25, 2017 at 07:00:14PM +0100, Dr. David Alan Gilbert wrote:
> Hi Dan,
>   I've got a crash in head (and 2.10) which is a bit of a heisenbug;
> I can trigger it with:
> 
> ./qemu-system-x86_64 -netdev tap,id=hostnet0,vhost=on,fd=10   -chardev socket,id=charchannel0,path=/tmp/org.qemu.guest_agent.0,server,nowait  -monitor stdio -vnc :0
> 
> and then 'q' to quit.

Hmm, that doesn't trigger for me on git master at least.

> Note I'm not doing a redirect in of fd 10.

So it's trying & failing to setup the tap dev, right ?

eg you see this:

#  ./x86_64-softmmu/qemu-system-x86_64 -netdev tap,id=hostnet0,vhost=on,fd=10   -chardev socket,id=charchannel0,path=/tmp/org.qemu.guest_agent.0,server,nowait  -monitor stdio -vnc :0
qemu-system-x86_64: -netdev tap,id=hostnet0,vhost=on,fd=10: TUNGETIFF ioctl() failed: Invalid argument
QEMU 2.10.50 monitor - type 'help' for more information
(qemu) qemu-system-x86_64: warning: netdev hostnet0 has no peer

(qemu) q


Except it crashes at the end ?


> It goes away if I remove either the -netdev or the -chardev option.
> 
> It doesn't trigger under gdb, but fortunately we get a core:
> 
> #0  0x000055a226d94a2e in socket_listen_cleanup (fd=<optimized out>, errp=errp@entry=0x7fff3585e8c0)
>     at /root/qemu/util/qemu-sockets.c:1077
>     1077	    if (addr->type == SOCKET_ADDRESS_TYPE_UNIX
>     1078	        && addr->u.q_unix.path) {
>     1079	        if (unlink(addr->u.q_unix.path) < 0 && errno != ENOENT) {

Can you see from the core whether one of those pointers is NULL, or is there
a complete garbage pointer ?

I wonder if it triggers if you run QEMU under valgrind ? 


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Crash with odd chardev setup

[Dr. David Alan Gilbert]

* Daniel P. Berrange (berrange@redhat.com) wrote:
> On Wed, Oct 25, 2017 at 07:00:14PM +0100, Dr. David Alan Gilbert wrote:
> > Hi Dan,
> >   I've got a crash in head (and 2.10) which is a bit of a heisenbug;
> > I can trigger it with:
> > 
> > ./qemu-system-x86_64 -netdev tap,id=hostnet0,vhost=on,fd=10   -chardev socket,id=charchannel0,path=/tmp/org.qemu.guest_agent.0,server,nowait  -monitor stdio -vnc :0
> > 
> > and then 'q' to quit.
> 
> Hmm, that doesn't trigger for me on git master at least.

Hmm.

> > Note I'm not doing a redirect in of fd 10.
> 
> So it's trying & failing to setup the tap dev, right ?
> 
> eg you see this:
> 
> #  ./x86_64-softmmu/qemu-system-x86_64 -netdev tap,id=hostnet0,vhost=on,fd=10   -chardev socket,id=charchannel0,path=/tmp/org.qemu.guest_agent.0,server,nowait  -monitor stdio -vnc :0
> qemu-system-x86_64: -netdev tap,id=hostnet0,vhost=on,fd=10: TUNGETIFF ioctl() failed: Invalid argument
> QEMU 2.10.50 monitor - type 'help' for more information
> (qemu) qemu-system-x86_64: warning: netdev hostnet0 has no peer
> 
> (qemu) q
> 
> 
> Except it crashes at the end ?

Right.

> 
> 
> > It goes away if I remove either the -netdev or the -chardev option.
> > 
> > It doesn't trigger under gdb, but fortunately we get a core:
> > 
> > #0  0x000055a226d94a2e in socket_listen_cleanup (fd=<optimized out>, errp=errp@entry=0x7fff3585e8c0)
> >     at /root/qemu/util/qemu-sockets.c:1077
> >     1077	    if (addr->type == SOCKET_ADDRESS_TYPE_UNIX
> >     1078	        && addr->u.q_unix.path) {
> >     1079	        if (unlink(addr->u.q_unix.path) < 0 && errno != ENOENT) {
> 
> Can you see from the core whether one of those pointers is NULL, or is there
> a complete garbage pointer ?

Gdb showed all the pointers as optimised out I think.

> I wonder if it triggers if you run QEMU under valgrind ? 

It does, but it shows: 
==29930== Thread 1:
==29930== Invalid read of size 4
==29930==    at 0x6F3A2E: socket_listen_cleanup (qemu-sockets.c:1077)
==29930==    by 0x6A142A: qio_channel_socket_finalize (channel-socket.c:388)
==29930==    by 0x61BA91: object_deinit (object.c:462)
==29930==    by 0x61BA91: object_finalize (object.c:476)
==29930==    by 0x61BA91: object_unref (object.c:911)
==29930==    by 0x6924C8: char_socket_finalize (char-socket.c:805)
==29930==    by 0x61BA91: object_deinit (object.c:462)
==29930==    by 0x61BA91: object_finalize (object.c:476)
==29930==    by 0x61BA91: object_unref (object.c:911)
==29930==    by 0x61BACB: object_property_del_all (object.c:413)
==29930==    by 0x61BACB: object_finalize (object.c:475)
==29930==    by 0x61BACB: object_unref (object.c:911)
==29930==    by 0x61AA86: object_property_del_child.isra.7 (object.c:436)
==29930==    by 0x3263FE: main (vl.c:4914)
==29930==  Address 0x0 is not stack'd, malloc'd or (recently) free'd

so I guess one of the pointers is NULL;   I guess I need a few printf's.

Dave

> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] Crash with odd chardev setup

[Dr. David Alan Gilbert]

* Dr. David Alan Gilbert (dgilbert@redhat.com) wrote:
> * Daniel P. Berrange (berrange@redhat.com) wrote:
> > On Wed, Oct 25, 2017 at 07:00:14PM +0100, Dr. David Alan Gilbert wrote:
> > > Hi Dan,
> > >   I've got a crash in head (and 2.10) which is a bit of a heisenbug;
> > > I can trigger it with:
> > > 
> > > ./qemu-system-x86_64 -netdev tap,id=hostnet0,vhost=on,fd=10   -chardev socket,id=charchannel0,path=/tmp/org.qemu.guest_agent.0,server,nowait  -monitor stdio -vnc :0
> > > 
> > > and then 'q' to quit.
> > 
> > Hmm, that doesn't trigger for me on git master at least.
> 
> Hmm.
> 
> > > Note I'm not doing a redirect in of fd 10.
> > 
> > So it's trying & failing to setup the tap dev, right ?
> > 
> > eg you see this:
> > 
> > #  ./x86_64-softmmu/qemu-system-x86_64 -netdev tap,id=hostnet0,vhost=on,fd=10   -chardev socket,id=charchannel0,path=/tmp/org.qemu.guest_agent.0,server,nowait  -monitor stdio -vnc :0
> > qemu-system-x86_64: -netdev tap,id=hostnet0,vhost=on,fd=10: TUNGETIFF ioctl() failed: Invalid argument
> > QEMU 2.10.50 monitor - type 'help' for more information
> > (qemu) qemu-system-x86_64: warning: netdev hostnet0 has no peer
> > 
> > (qemu) q
> > 
> > 
> > Except it crashes at the end ?
> 
> Right.
> 
> > 
> > 
> > > It goes away if I remove either the -netdev or the -chardev option.
> > > 
> > > It doesn't trigger under gdb, but fortunately we get a core:
> > > 
> > > #0  0x000055a226d94a2e in socket_listen_cleanup (fd=<optimized out>, errp=errp@entry=0x7fff3585e8c0)
> > >     at /root/qemu/util/qemu-sockets.c:1077
> > >     1077	    if (addr->type == SOCKET_ADDRESS_TYPE_UNIX
> > >     1078	        && addr->u.q_unix.path) {
> > >     1079	        if (unlink(addr->u.q_unix.path) < 0 && errno != ENOENT) {
> > 
> > Can you see from the core whether one of those pointers is NULL, or is there
> > a complete garbage pointer ?
> 
> Gdb showed all the pointers as optimised out I think.
> 
> > I wonder if it triggers if you run QEMU under valgrind ? 
> 
> It does, but it shows: 
> ==29930== Thread 1:
> ==29930== Invalid read of size 4
> ==29930==    at 0x6F3A2E: socket_listen_cleanup (qemu-sockets.c:1077)
> ==29930==    by 0x6A142A: qio_channel_socket_finalize (channel-socket.c:388)
> ==29930==    by 0x61BA91: object_deinit (object.c:462)
> ==29930==    by 0x61BA91: object_finalize (object.c:476)
> ==29930==    by 0x61BA91: object_unref (object.c:911)
> ==29930==    by 0x6924C8: char_socket_finalize (char-socket.c:805)
> ==29930==    by 0x61BA91: object_deinit (object.c:462)
> ==29930==    by 0x61BA91: object_finalize (object.c:476)
> ==29930==    by 0x61BA91: object_unref (object.c:911)
> ==29930==    by 0x61BACB: object_property_del_all (object.c:413)
> ==29930==    by 0x61BACB: object_finalize (object.c:475)
> ==29930==    by 0x61BACB: object_unref (object.c:911)
> ==29930==    by 0x61AA86: object_property_del_child.isra.7 (object.c:436)
> ==29930==    by 0x3263FE: main (vl.c:4914)
> ==29930==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
> 
> so I guess one of the pointers is NULL;   I guess I need a few printf's.

I can get rid of the actual crash by changing socket_listen_cleanup;

@@ -1074,7 +1074,7 @@ void socket_listen_cleanup(int fd, Error **errp)

     addr = socket_local_address(fd, errp);

-    if (addr->type == SOCKET_ADDRESS_TYPE_UNIX
+    if (addr && addr->type == SOCKET_ADDRESS_TYPE_UNIX
         && addr->u.q_unix.path) {

socket_local_address is failing with :
socket_listen_cleanup: fd=10 addr=(nil)
qemu-system-x86_64: Unable to query local socket address: Bad file descriptor

The difference under gdb seems to be the fd in cleanup is fd=14 so isn't
the same one as the bad fd passed in (I don't really understand why).

Dave
> Dave
> 
> > 
> > Regards,
> > Daniel
> > -- 
> > |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> > |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> > |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK