From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46326) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WhIHp-0002LO-Et for qemu-devel@nongnu.org; Mon, 05 May 2014 08:46:33 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WhIHh-0005Ju-NV for qemu-devel@nongnu.org; Mon, 05 May 2014 08:46:25 -0400 Received: from e06smtp10.uk.ibm.com ([195.75.94.106]:37458) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WhIHh-0005JE-7d for qemu-devel@nongnu.org; Mon, 05 May 2014 08:46:17 -0400 Received: from /spool/local by e06smtp10.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 5 May 2014 13:46:14 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (d06relay13.portsmouth.uk.ibm.com [9.149.109.198]) by d06dlp01.portsmouth.uk.ibm.com (Postfix) with ESMTP id E547A17D806A for ; Mon, 5 May 2014 13:47:16 +0100 (BST) Received: from d06av02.portsmouth.uk.ibm.com (d06av02.portsmouth.uk.ibm.com [9.149.37.228]) by b06cxnps4076.portsmouth.uk.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id s45CkCV6786734 for ; Mon, 5 May 2014 12:46:12 GMT Received: from d06av02.portsmouth.uk.ibm.com (localhost [127.0.0.1]) by d06av02.portsmouth.uk.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id s45CkAdG015916 for ; Mon, 5 May 2014 06:46:11 -0600 Message-ID: <53678811.7030403@de.ibm.com> Date: Mon, 05 May 2014 14:46:09 +0200 From: Christian Borntraeger MIME-Version: 1.0 References: <1398956086-20171-1-git-send-email-stefanha@redhat.com> <53675738.7040704@de.ibm.com> <20140505120525.GD16173@stefanha-thinkpad.redhat.com> In-Reply-To: <20140505120525.GD16173@stefanha-thinkpad.redhat.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH 00/22] dataplane: use QEMU block layer List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Stefan Hajnoczi Cc: Kevin Wolf , "Shergill, Gurinder" , qemu-devel@nongnu.org, Stefan Hajnoczi , Paolo Bonzini , "Vinod, Chegu" On 05/05/14 14:05, Stefan Hajnoczi wrote: > On Mon, May 05, 2014 at 11:17:44AM +0200, Christian Borntraeger wrote: >> On 01/05/14 16:54, Stefan Hajnoczi wrote: >>> This patch series switches virtio-blk data-plane from a custom Linux AIO >>> request queue to the QEMU block layer. The previous "raw files only" >>> limitation is lifted. All image formats and protocols can now be used with >>> virtio-blk data-plane. >> >> Nice. Is there a git branch somewhere, so that we can test this on s390? > > Hi Christian, > I'm getting to work on v2 but you can grab this v1 series from git in > the meantime: > > https://github.com/stefanha/qemu.git bdrv_set_aio_context > > Stefan > In general the main path seems to work fine. With lots of devices (one qcow2, 23 raw scsi disks) I get a hang on shutdown. kvm_stat claims that nothing is going on any more, but somehow threads are stuck in ppoll. gdb tells me that all cpus have #0 0x000003fffcde0ba0 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x000003fffcde3c0c in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0 #2 0x000003fffcddc99a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #3 0x00000000801f183a in qemu_cond_wait (cond=, mutex=mutex@entry=0x8072ba30 ) at /home/cborntra/REPOS/qemu/util/qemu-thread-posix.c:135 #4 0x00000000801512f2 in qemu_kvm_wait_io_event (cpu=) at /home/cborntra/REPOS/qemu/cpus.c:842 #5 qemu_kvm_cpu_thread_fn (arg=0x80a53e10) at /home/cborntra/REPOS/qemu/cpus.c:878 all iothreads have #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x3fff4001b00, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x807dd610, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x807dd4c8) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 the main thread has Thread 1 (Thread 0x3fff9e5c9b0 (LWP 33684)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x80ae8030, nfds=nfds@entry=4, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=ctx@entry=0x809a7ea0, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x0000000080030c46 in bdrv_flush (bs=bs@entry=0x807e5900) at /home/cborntra/REPOS/qemu/block.c:4904 #5 0x0000000080030ce8 in bdrv_flush_all () at /home/cborntra/REPOS/qemu/block.c:3723 #6 0x0000000080152fe8 in do_vm_stop (state=) at /home/cborntra/REPOS/qemu/cpus.c:538 #7 vm_stop (state=) at /home/cborntra/REPOS/qemu/cpus.c:1219 #8 0x0000000000000000 in ?? () How are the ppoll calls supposed to return if there is nothing going on? PS: I think I have seen this before recently during managedsave, so it might have been introduced with the iothread rework instead of this one. ---- full trace ---- Thread 34 (Thread 0x3fff919c910 (LWP 33696)): #0 0x000003fffcde0b5e in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x000003fffcde3c0c in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0 #2 0x000003fffcddc99a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #3 0x00000000801f183a in qemu_cond_wait (cond=, mutex=mutex@entry=0x8072ba30 ) at /home/cborntra/REPOS/qemu/util/qemu-thread-posix.c:135 #4 0x00000000801512f2 in qemu_kvm_wait_io_event (cpu=) at /home/cborntra/REPOS/qemu/cpus.c:842 #5 qemu_kvm_cpu_thread_fn (arg=0x809e1f00) at /home/cborntra/REPOS/qemu/cpus.c:878 #6 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #7 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 33 (Thread 0x3fff899c910 (LWP 33697)): #0 0x000003fffcde0ba0 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x000003fffcde3c0c in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0 #2 0x000003fffcddc99a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #3 0x00000000801f183a in qemu_cond_wait (cond=, mutex=mutex@entry=0x8072ba30 ) at /home/cborntra/REPOS/qemu/util/qemu-thread-posix.c:135 #4 0x00000000801512f2 in qemu_kvm_wait_io_event (cpu=) at /home/cborntra/REPOS/qemu/cpus.c:842 #5 qemu_kvm_cpu_thread_fn (arg=0x809f2370) at /home/cborntra/REPOS/qemu/cpus.c:878 #6 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #7 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 32 (Thread 0x3fef3fff910 (LWP 33698)): #0 0x000003fffcde0ba0 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x000003fffcde3c0c in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0 #2 0x000003fffcddc99a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #3 0x00000000801f183a in qemu_cond_wait (cond=, mutex=mutex@entry=0x8072ba30 ) at /home/cborntra/REPOS/qemu/util/qemu-thread-posix.c:135 #4 0x00000000801512f2 in qemu_kvm_wait_io_event (cpu=) at /home/cborntra/REPOS/qemu/cpus.c:842 #5 qemu_kvm_cpu_thread_fn (arg=0x80a027e0) at /home/cborntra/REPOS/qemu/cpus.c:878 #6 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #7 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 31 (Thread 0x3fef37ff910 (LWP 33699)): #0 0x000003fffcde0b5e in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x000003fffcde3c0c in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0 #2 0x000003fffcddc99a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #3 0x00000000801f183a in qemu_cond_wait (cond=, mutex=mutex@entry=0x8072ba30 ) at /home/cborntra/REPOS/qemu/util/qemu-thread-posix.c:135 #4 0x00000000801512f2 in qemu_kvm_wait_io_event (cpu=) at /home/cborntra/REPOS/qemu/cpus.c:842 #5 qemu_kvm_cpu_thread_fn (arg=0x80a12c50) at /home/cborntra/REPOS/qemu/cpus.c:878 #6 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #7 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 30 (Thread 0x3fef2fff910 (LWP 33700)): #0 0x000003fffcde0ba0 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x000003fffcde3c0c in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0 #2 0x000003fffcddc99a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #3 0x00000000801f183a in qemu_cond_wait (cond=, mutex=mutex@entry=0x8072ba30 ) at /home/cborntra/REPOS/qemu/util/qemu-thread-posix.c:135 #4 0x00000000801512f2 in qemu_kvm_wait_io_event (cpu=) at /home/cborntra/REPOS/qemu/cpus.c:842 #5 qemu_kvm_cpu_thread_fn (arg=0x80a230c0) at /home/cborntra/REPOS/qemu/cpus.c:878 #6 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #7 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 29 (Thread 0x3fef27ff910 (LWP 33701)): #0 0x000003fffcde0ba0 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x000003fffcde3c0c in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0 #2 0x000003fffcddc99a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #3 0x00000000801f183a in qemu_cond_wait (cond=, mutex=mutex@entry=0x8072ba30 ) at /home/cborntra/REPOS/qemu/util/qemu-thread-posix.c:135 #4 0x00000000801512f2 in qemu_kvm_wait_io_event (cpu=) at /home/cborntra/REPOS/qemu/cpus.c:842 #5 qemu_kvm_cpu_thread_fn (arg=0x80a33530) at /home/cborntra/REPOS/qemu/cpus.c:878 #6 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #7 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 28 (Thread 0x3fef1fff910 (LWP 33702)): #0 0x000003fffcde0ba0 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x000003fffcde3c0c in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0 #2 0x000003fffcddc99a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #3 0x00000000801f183a in qemu_cond_wait (cond=, mutex=mutex@entry=0x8072ba30 ) at /home/cborntra/REPOS/qemu/util/qemu-thread-posix.c:135 #4 0x00000000801512f2 in qemu_kvm_wait_io_event (cpu=) at /home/cborntra/REPOS/qemu/cpus.c:842 #5 qemu_kvm_cpu_thread_fn (arg=0x80a439a0) at /home/cborntra/REPOS/qemu/cpus.c:878 #6 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #7 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 27 (Thread 0x3fef17ff910 (LWP 33703)): #0 0x000003fffcde0ba0 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x000003fffcde3c0c in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0 #2 0x000003fffcddc99a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #3 0x00000000801f183a in qemu_cond_wait (cond=, mutex=mutex@entry=0x8072ba30 ) at /home/cborntra/REPOS/qemu/util/qemu-thread-posix.c:135 #4 0x00000000801512f2 in qemu_kvm_wait_io_event (cpu=) at /home/cborntra/REPOS/qemu/cpus.c:842 #5 qemu_kvm_cpu_thread_fn (arg=0x80a53e10) at /home/cborntra/REPOS/qemu/cpus.c:878 #6 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #7 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 26 (Thread 0x3fef0fff910 (LWP 33704)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x80ae8030, nfds=nfds@entry=4, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x809a7ea0, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x809a7d58) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 25 (Thread 0x3fef07ff910 (LWP 33705)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x3fff4001b00, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x807dd610, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x807dd4c8) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 24 (Thread 0x3feeffff910 (LWP 33706)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x807e5470, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x807e0130, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x807e0038) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 23 (Thread 0x3feef7ff910 (LWP 33707)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x3fff400e350, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80a6f440, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80a6f348) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 22 (Thread 0x3feeefff910 (LWP 33708)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x80ad5dd0, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80a65db0, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80a65c68) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 21 (Thread 0x3feee7ff910 (LWP 33709)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x3fff400e380, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80a69f00, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80a69e08) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 20 (Thread 0x3feedfff910 (LWP 33710)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x80ae8580, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80a7cb50, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80a7ca58) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 19 (Thread 0x3feed7ff910 (LWP 33711)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x3fff4001180, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80a87050, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80a86f08) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 18 (Thread 0x3feecfff910 (LWP 33712)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x80acf3d0, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80a83280, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80a83188) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 17 (Thread 0x3feec7ff910 (LWP 33713)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x3fff4008fd0, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80a74bc0, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80a74a78) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 16 (Thread 0x3feebfff910 (LWP 33714)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x80ae61d0, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80a78cf0, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80a78bf8) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 15 (Thread 0x3feeb7ff910 (LWP 33715)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x3fff40011b0, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80a911a0, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80a910a8) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 14 (Thread 0x3feeafff910 (LWP 33716)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x80af53d0, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80a9b680, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80a9b538) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 13 (Thread 0x3feea7ff910 (LWP 33717)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x3fff4002bd0, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80a978b0, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80a977b8) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 12 (Thread 0x3fee9fff910 (LWP 33718)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x80ae85b0, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80a891d0, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80a89088) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 11 (Thread 0x3fee97ff910 (LWP 33719)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x3fff400a7d0, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80a8d320, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80a8d228) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 10 (Thread 0x3fee8fff910 (LWP 33720)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x80ae27d0, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80aa57d0, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80aa56d8) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 9 (Thread 0x3fee87ff910 (LWP 33721)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x3fff4018fd0, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80aafcd0, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80aafb88) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 8 (Thread 0x3fee7fff910 (LWP 33722)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x80b02fd0, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80aabf00, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80aabe08) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 7 (Thread 0x3fee77ff910 (LWP 33723)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x3fff4017580, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80a9d840, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80a9d6f8) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 6 (Thread 0x3fee6fff910 (LWP 33724)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x80abc4d0, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80aa1970, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80aa1878) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 5 (Thread 0x3fee67ff910 (LWP 33725)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x3fff40067d0, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80aba760, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80aba668) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 4 (Thread 0x3fee5fff910 (LWP 33726)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x80af7bd0, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80ab2630, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80ab24e8) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 3 (Thread 0x3fee57ff910 (LWP 33727)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x3fff400e2e0, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80ab5170, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80ab5078) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 2 (Thread 0x3fee4fff910 (LWP 33728)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x80b02980, nfds=nfds@entry=3, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=0x80ac5ac0, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x00000000800b2f6c in iothread_run (opaque=0x80ac59c8) at /home/cborntra/REPOS/qemu/iothread.c:41 #5 0x000003fffcdd8412 in start_thread () from /lib64/libpthread.so.0 #6 0x000003fffbc3f0ae in thread_start () from /lib64/libc.so.6 Thread 1 (Thread 0x3fff9e5c9b0 (LWP 33684)): #0 0x000003fffbc348e0 in ppoll () from /lib64/libc.so.6 #1 0x00000000800fcce6 in ppoll (__ss=0x0, __timeout=0x0, __nfds=, __fds=) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=fds@entry=0x80ae8030, nfds=nfds@entry=4, timeout=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:311 #3 0x000000008001ae4c in aio_poll (ctx=ctx@entry=0x809a7ea0, blocking=blocking@entry=true) at /home/cborntra/REPOS/qemu/aio-posix.c:221 #4 0x0000000080030c46 in bdrv_flush (bs=bs@entry=0x807e5900) at /home/cborntra/REPOS/qemu/block.c:4904 #5 0x0000000080030ce8 in bdrv_flush_all () at /home/cborntra/REPOS/qemu/block.c:3723 #6 0x0000000080152fe8 in do_vm_stop (state=) at /home/cborntra/REPOS/qemu/cpus.c:538 #7 vm_stop (state=) at /home/cborntra/REPOS/qemu/cpus.c:1219 #8 0x0000000000000000 in ?? ()