[Qemu-devel] 9pfs is broken in 2.5 (SIGSEGV) (bisected)

[Qemu-devel] 9pfs is broken in 2.5 (SIGSEGV) (bisected)

[Michael Tokarev]

When trying to mount a 9p filesystem in guest in 2.5 qemu,
qemu immediately segfaults.

 qemu-system-x86_64 ...
  -fsdev local,security_model=none,id=fsd0,path=/tmp
  -device virtio-9p-pci,fsdev=fsd0,mount_tag=fsd0


 mount -t 9p -o trans=virtio,version=9p2000.u fsd0 /mnt
 (SIGSEGV)

The problem goes down to the following commit:

commit ebac1202c95a4f1b76b6ef3f0f63926fa76e753e
Author: Paolo Bonzini <pbonzini@redhat.com>
Date:   Fri Nov 27 12:43:06 2015 +0100

    virtio-9p: use QEMU thread pool

    The QEMU thread pool already has a mechanism to invoke callbacks in the main
    thread.  It does not need an EventNotifier and it is more efficient too.
    Use it instead of GAsyncQueue + GThreadPool + glue.

    As a side effect, it silences Coverity's complaint about an unchecked
    return value for event_notifier_init.

    Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
    Reviewed-by: Greg Kurz <gkurz@linux.vnet.ibm.com>
    (removed no more needed #include <glib.h> from virtio-9p-coth.h)
    Signed-off-by: Greg Kurz <gkurz@linux.vnet.ibm.com>

This is a heads-up for now, digging further.

/mjt

Re: [Qemu-devel] 9pfs is broken in 2.5 (SIGSEGV) (bisected)

[Michael Tokarev]

19.12.2015 11:05, Michael Tokarev wrote:

> commit ebac1202c95a4f1b76b6ef3f0f63926fa76e753e
> Author: Paolo Bonzini <pbonzini@redhat.com>
> Date:   Fri Nov 27 12:43:06 2015 +0100
>     virtio-9p: use QEMU thread pool

Program received signal SIGSEGV, Segmentation fault.
thread_pool_submit_aio (pool=0x0, func=0x55555580f6a0 <coroutine_enter_func>, arg=0x555557698870, cb=<optimized out>,
    opaque=<optimized out>) at /build/kvm/git/thread-pool.c:246
246	    QLIST_INSERT_HEAD(&pool->head, req, all);
(gdb) bt
#0  thread_pool_submit_aio (pool=0x0, func=0x55555580f6a0 <coroutine_enter_func>, arg=0x555557698870, cb=<optimized out>,
    opaque=<optimized out>) at /build/kvm/git/thread-pool.c:246
#1  0x000055555592c9ed in aio_bh_call (bh=<optimized out>) at /build/kvm/git/async.c:64
#2  aio_bh_poll (ctx=ctx@entry=0x5555563c9360) at /build/kvm/git/async.c:92
#3  0x0000555555939390 in aio_dispatch (ctx=0x5555563c9360) at /build/kvm/git/aio-posix.c:305
#4  0x000055555592c7de in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>)
    at /build/kvm/git/async.c:231
#5  0x00007ffff49d1c5d in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#6  0x0000555555937cf8 in glib_pollfds_poll () at /build/kvm/git/main-loop.c:211
#7  os_host_main_loop_wait (timeout=<optimized out>) at /build/kvm/git/main-loop.c:256
#8  main_loop_wait (nonblocking=<optimized out>) at /build/kvm/git/main-loop.c:504
#9  0x00005555556dbe3e in main_loop () at /build/kvm/git/vl.c:1923
#10 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /build/kvm/git/vl.c:4684
(gdb) p pool
$2 = (ThreadPool *) 0x0

Woops.

/mjt

Re: [Qemu-devel] 9pfs is broken in 2.5 (SIGSEGV) (bisected)

[Michael Tokarev]

19.12.2015 11:15, Michael Tokarev wrote:
>> commit ebac1202c95a4f1b76b6ef3f0f63926fa76e753e
>> Author: Paolo Bonzini <pbonzini@redhat.com>
>> Date:   Fri Nov 27 12:43:06 2015 +0100
>>     virtio-9p: use QEMU thread pool
> 
> Program received signal SIGSEGV, Segmentation fault.
> thread_pool_submit_aio (pool=0x0, func=0x55555580f6a0 <coroutine_enter_func>, arg=0x555557698870, cb=<optimized out>,
>     opaque=<optimized out>) at /build/kvm/git/thread-pool.c:246
> 246	    QLIST_INSERT_HEAD(&pool->head, req, all);
> (gdb) bt
> #0  thread_pool_submit_aio (pool=0x0, func=0x55555580f6a0 <coroutine_enter_func>, arg=0x555557698870, cb=<optimized out>,
>     opaque=<optimized out>) at /build/kvm/git/thread-pool.c:246
> #1  0x000055555592c9ed in aio_bh_call (bh=<optimized out>) at /build/kvm/git/async.c:64
> #2  aio_bh_poll (ctx=ctx@entry=0x5555563c9360) at /build/kvm/git/async.c:92
> #3  0x0000555555939390 in aio_dispatch (ctx=0x5555563c9360) at /build/kvm/git/aio-posix.c:305
> #4  0x000055555592c7de in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>)
>     at /build/kvm/git/async.c:231
> #5  0x00007ffff49d1c5d in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
> #6  0x0000555555937cf8 in glib_pollfds_poll () at /build/kvm/git/main-loop.c:211
> #7  os_host_main_loop_wait (timeout=<optimized out>) at /build/kvm/git/main-loop.c:256
> #8  main_loop_wait (nonblocking=<optimized out>) at /build/kvm/git/main-loop.c:504
> #9  0x00005555556dbe3e in main_loop () at /build/kvm/git/vl.c:1923
> #10 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /build/kvm/git/vl.c:4684
> (gdb) p pool
> $2 = (ThreadPool *) 0x0

In 9pfs/virtio-9p-coth.c:co_run_in_worker_bh(), in this context:

void co_run_in_worker_bh(void *opaque)
{
    Coroutine *co = opaque;
    thread_pool_submit_aio(qemu_get_aio_context()->thread_pool,
                           coroutine_enter_func, co, coroutine_enter_cb, co);
}

qemu_get_aio_context()->thread_pool is NULL.

Thanks,

/mjt

Re: [Qemu-devel] 9pfs is broken in 2.5 (SIGSEGV) (bisected)

[Greg Kurz]

On Sat, 19 Dec 2015 12:16:58 +0300
Michael Tokarev <mjt@tls.msk.ru> wrote:

> 19.12.2015 11:15, Michael Tokarev wrote:
> >> commit ebac1202c95a4f1b76b6ef3f0f63926fa76e753e
> >> Author: Paolo Bonzini <pbonzini@redhat.com>
> >> Date:   Fri Nov 27 12:43:06 2015 +0100
> >>     virtio-9p: use QEMU thread pool
> > 
> > Program received signal SIGSEGV, Segmentation fault.
> > thread_pool_submit_aio (pool=0x0, func=0x55555580f6a0 <coroutine_enter_func>, arg=0x555557698870, cb=<optimized out>,
> >     opaque=<optimized out>) at /build/kvm/git/thread-pool.c:246
> > 246	    QLIST_INSERT_HEAD(&pool->head, req, all);
> > (gdb) bt
> > #0  thread_pool_submit_aio (pool=0x0, func=0x55555580f6a0 <coroutine_enter_func>, arg=0x555557698870, cb=<optimized out>,
> >     opaque=<optimized out>) at /build/kvm/git/thread-pool.c:246
> > #1  0x000055555592c9ed in aio_bh_call (bh=<optimized out>) at /build/kvm/git/async.c:64
> > #2  aio_bh_poll (ctx=ctx@entry=0x5555563c9360) at /build/kvm/git/async.c:92
> > #3  0x0000555555939390 in aio_dispatch (ctx=0x5555563c9360) at /build/kvm/git/aio-posix.c:305
> > #4  0x000055555592c7de in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>)
> >     at /build/kvm/git/async.c:231
> > #5  0x00007ffff49d1c5d in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
> > #6  0x0000555555937cf8 in glib_pollfds_poll () at /build/kvm/git/main-loop.c:211
> > #7  os_host_main_loop_wait (timeout=<optimized out>) at /build/kvm/git/main-loop.c:256
> > #8  main_loop_wait (nonblocking=<optimized out>) at /build/kvm/git/main-loop.c:504
> > #9  0x00005555556dbe3e in main_loop () at /build/kvm/git/vl.c:1923
> > #10 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /build/kvm/git/vl.c:4684
> > (gdb) p pool
> > $2 = (ThreadPool *) 0x0
> 
> In 9pfs/virtio-9p-coth.c:co_run_in_worker_bh(), in this context:
> 
> void co_run_in_worker_bh(void *opaque)
> {
>     Coroutine *co = opaque;
>     thread_pool_submit_aio(qemu_get_aio_context()->thread_pool,
>                            coroutine_enter_func, co, coroutine_enter_cb, co);
> }
> 
> qemu_get_aio_context()->thread_pool is NULL.
> 
> Thanks,
> 
> /mjt
> 

Hi Michael,

It works-for-me-of-course on ppc64le and x86_64. I could verify with gdb that
thread_pool gets set early:

#0  aio_get_thread_pool (ctx=0x10e6b7a0)
    at /home/greg/Work/qemu/qemu-master/async.c:280
#1  0x00000000105e06fc in paio_submit (bs=0x10e8d1c0, fd=15, sector_num=0, 
    qiov=0x3fffffffe0e0, nb_sectors=1, cb=0x105f03e0 <bdrv_co_io_em_complete>, 
    opaque=0x3fffb6e2fc88, type=1)
    at /home/greg/Work/qemu/qemu-master/block/raw-posix.c:1309
#2  0x00000000105e08a0 in raw_aio_submit (bs=0x10e8d1c0, sector_num=0, 
    qiov=0x3fffffffe0e0, nb_sectors=1, cb=0x105f03e0 <bdrv_co_io_em_complete>, 
    opaque=0x3fffb6e2fc88, type=1)
    at /home/greg/Work/qemu/qemu-master/block/raw-posix.c:1339
#3  0x00000000105e0a98 in raw_aio_readv (bs=0x10e8d1c0, sector_num=0, 
    qiov=0x3fffffffe0e0, nb_sectors=1, cb=0x105f03e0 <bdrv_co_io_em_complete>, 
    opaque=0x3fffb6e2fc88)
    at /home/greg/Work/qemu/qemu-master/block/raw-posix.c:1377
#4  0x00000000105f055c in bdrv_co_io_em (bs=0x10e8d1c0, sector_num=0, 
    nb_sectors=1, iov=0x3fffffffe0e0, is_write=false)
    at /home/greg/Work/qemu/qemu-master/block/io.c:2300
#5  0x00000000105f0648 in bdrv_co_readv_em (bs=0x10e8d1c0, sector_num=0, 
    nb_sectors=1, iov=0x3fffffffe0e0)
    at /home/greg/Work/qemu/qemu-master/block/io.c:2317
#6  0x00000000105eba90 in bdrv_aligned_preadv (bs=0x10e8d1c0, 
    req=0x3fffb6e2fe48, offset=0, bytes=512, align=512, qiov=0x3fffffffe0e0, 
    flags=0) at /home/greg/Work/qemu/qemu-master/block/io.c:900
#7  0x00000000105ebfd4 in bdrv_co_do_preadv (bs=0x10e8d1c0, offset=0, 
    bytes=512, qiov=0x3fffffffe0e0, flags=(unknown: 0))
    at /home/greg/Work/qemu/qemu-master/block/io.c:992
#8  0x00000000105eaa28 in bdrv_rw_co_entry (opaque=0x3fffffffe040)
    at /home/greg/Work/qemu/qemu-master/block/io.c:534
#9  0x0000000010675bc4 in coroutine_trampoline (i0=283706576, i1=0)
    at /home/greg/Work/qemu/qemu-master/util/coroutine-ucontext.c:80
#10 0x00003fffb76c1c4c in makecontext () from /lib64/power8/libc.so.6

Probably other options on the command line make the difference. Anyway it
would seem we can reach co_run_in_worker_bh() without the main thread pool
being initialized... I'll work this out early next week.

Thanks.

--
Greg