From: Christian Borntraeger <borntraeger@de.ibm.com>
To: Paolo Bonzini <pbonzini@redhat.com>,
tu bo <tubo@linux.vnet.ibm.com>, Fam Zheng <famz@redhat.com>,
qemu-devel@nongnu.org
Cc: Kevin Wolf <kwolf@redhat.com>,
cornelia.huck@de.ibm.com, Stefan Hajnoczi <stefanha@redhat.com>,
qemu-block@nongnu.org, "Michael S. Tsirkin" <mst@redhat.com>
Subject: Re: [Qemu-devel] [PATCH 0/4] Tweaks around virtio-blk start/stop
Date: Thu, 17 Mar 2016 16:16:52 +0100 [thread overview]
Message-ID: <56EACA64.9060402@de.ibm.com> (raw)
In-Reply-To: <56EACA22.2020505@de.ibm.com>
On 03/17/2016 04:15 PM, Christian Borntraeger wrote:
> On 03/17/2016 04:02 PM, Paolo Bonzini wrote:
>>
>>
>> On 17/03/2016 13:39, Christian Borntraeger wrote:
>>> As an interesting side note, I updated my system from F20 to F23 some days ago
>>> (after the initial report). While To Bo is still on a F20 system. I was not able
>>> to reproduce the original crash on f23. but going back to F20 made this
>>> problem re-appear.
>>>
>>> Stack trace of thread 26429:
>>> #0 0x00000000802008aa tracked_request_begin (qemu-system-s390x)
>>> #1 0x0000000080203f3c bdrv_co_do_preadv (qemu-system-s390x)
>>> #2 0x000000008020567c bdrv_co_do_readv (qemu-system-s390x)
>>> #3 0x000000008025d0f4 coroutine_trampoline (qemu-system-s390x)
>>> #4 0x000003ff943d150a __makecontext_ret (libc.so.6)
>>>
>>> this is with patch 2-4 plus the removal of virtio_queue_host_notifier_read.
>>>
>>> Without removing virtio_queue_host_notifier_read, I get the same mutex lockup (as expected).
>>>
>>> Maybe we have two independent issues here and this is some old bug in glibc or
>>> whatever?
>>
>> I'm happy to try and reproduce on x86 if you give me some instruction
>> (RHEL7 should be close enough to Fedora 20).
>>
>> Can you add an assert in virtio_blk_handle_output to catch reentrancy, like
>
> that was quick (let me know if I should recompile with debugging)
>
> (gdb) thread apply all bt
>
> Thread 5 (Thread 0x3ff7b8ff910 (LWP 236419)):
> #0 0x000003ff7cdfcf56 in syscall () from /lib64/libc.so.6
> #1 0x000000001022452e in futex_wait (val=<optimized out>, ev=<optimized out>) at /home/cborntra/REPOS/qemu/util/qemu-thread-posix.c:292
> #2 qemu_event_wait (ev=ev@entry=0x1082b5c4 <rcu_call_ready_event>) at /home/cborntra/REPOS/qemu/util/qemu-thread-posix.c:399
> #3 0x000000001023353a in call_rcu_thread (opaque=<optimized out>) at /home/cborntra/REPOS/qemu/util/rcu.c:250
> #4 0x000003ff7cf084c6 in start_thread () from /lib64/libpthread.so.0
> #5 0x000003ff7ce02ec2 in thread_start () from /lib64/libc.so.6
>
> Thread 4 (Thread 0x3ff78eca910 (LWP 236426)):
> #0 0x000003ff7cdf819a in ioctl () from /lib64/libc.so.6
> #1 0x000000001005ddf8 in kvm_vcpu_ioctl (cpu=cpu@entry=0x10c27d40, type=type@entry=44672) at /home/cborntra/REPOS/qemu/kvm-all.c:1984
> #2 0x000000001005df1c in kvm_cpu_exec (cpu=cpu@entry=0x10c27d40) at /home/cborntra/REPOS/qemu/kvm-all.c:1834
> #3 0x000000001004b1be in qemu_kvm_cpu_thread_fn (arg=0x10c27d40) at /home/cborntra/REPOS/qemu/cpus.c:1050
> #4 0x000003ff7cf084c6 in start_thread () from /lib64/libpthread.so.0
> #5 0x000003ff7ce02ec2 in thread_start () from /lib64/libc.so.6
>
> Thread 3 (Thread 0x3ff7e8dcbb0 (LWP 236395)):
> #0 0x000003ff7cdf66e6 in ppoll () from /lib64/libc.so.6
> #1 0x00000000101a5e08 in ppoll (__ss=0x0, __timeout=0x3ffd6afe8a0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:77
> #2 qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=1034000000) at /home/cborntra/REPOS/qemu/qemu-timer.c:325
> #3 0x00000000101a56f2 in os_host_main_loop_wait (timeout=1034000000) at /home/cborntra/REPOS/qemu/main-loop.c:251
> #4 main_loop_wait (nonblocking=<optimized out>) at /home/cborntra/REPOS/qemu/main-loop.c:505
> #5 0x00000000100136d6 in main_loop () at /home/cborntra/REPOS/qemu/vl.c:1933
> #6 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /home/cborntra/REPOS/qemu/vl.c:4656
>
> Thread 2 (Thread 0x3ff7b0ff910 (LWP 236421)):
> #0 0x000003ff7cdf66e6 in ppoll () from /lib64/libc.so.6
> #1 0x00000000101a5e28 in ppoll (__ss=0x0, __timeout=0x0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:77
> #2 qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=-1) at /home/cborntra/REPOS/qemu/qemu-timer.c:313
> #3 0x00000000101a727c in aio_poll (ctx=0x10880560, blocking=<optimized out>) at /home/cborntra/REPOS/qemu/aio-posix.c:453
> #4 0x00000000100d39f0 in iothread_run (opaque=0x10880020) at /home/cborntra/REPOS/qemu/iothread.c:46
> #5 0x000003ff7cf084c6 in start_thread () from /lib64/libpthread.so.0
> #6 0x000003ff7ce02ec2 in thread_start () from /lib64/libc.so.6
>
> Thread 1 (Thread 0x3ff57fff910 (LWP 236427)):
> #0 0x000003ff7cd3b650 in raise () from /lib64/libc.so.6
> #1 0x000003ff7cd3ced8 in abort () from /lib64/libc.so.6
> #2 0x000003ff7cd33666 in __assert_fail_base () from /lib64/libc.so.6
> #3 0x000003ff7cd336f4 in __assert_fail () from /lib64/libc.so.6
> #4 0x000000001007a3c4 in virtio_blk_handle_output (vdev=<optimized out>, vq=<optimized out>) at /home/cborntra/REPOS/qemu/hw/block/virtio-blk.c:595
> #5 0x000000001009390e in virtio_queue_notify_vq (vq=0x10d77c70) at /home/cborntra/REPOS/qemu/hw/virtio/virtio.c:1095
> #6 0x0000000010095894 in virtio_queue_notify_vq (vq=<optimized out>) at /home/cborntra/REPOS/qemu/hw/virtio/virtio.c:1091
> #7 virtio_queue_notify (vdev=<optimized out>, n=n@entry=0) at /home/cborntra/REPOS/qemu/hw/virtio/virtio.c:1101
> #8 0x00000000100a17c8 in virtio_ccw_hcall_notify (args=<optimized out>) at /home/cborntra/REPOS/qemu/hw/s390x/s390-virtio-ccw.c:66
> #9 0x000000001009c210 in s390_virtio_hypercall (env=env@entry=0x10c75aa0) at /home/cborntra/REPOS/qemu/hw/s390x/s390-virtio-hcall.c:35
> #10 0x00000000100cb4e8 in handle_hypercall (run=<optimized out>, cpu=0x10c6d7d0) at /home/cborntra/REPOS/qemu/target-s390x/kvm.c:1283
> #11 handle_diag (ipb=<optimized out>, run=0x3ff78680000, cpu=0x10c6d7d0) at /home/cborntra/REPOS/qemu/target-s390x/kvm.c:1352
FWIW, this looks like that we still have a case, without eventfd during reboot or startup
> #12 handle_instruction (run=0x3ff78680000, cpu=0x10c6d7d0) at /home/cborntra/REPOS/qemu/target-s390x/kvm.c:1799
> #13 handle_intercept (cpu=0x10c6d7d0) at /home/cborntra/REPOS/qemu/target-s390x/kvm.c:1842
> #14 kvm_arch_handle_exit (cs=cs@entry=0x10c6d7d0, run=run@entry=0x3ff78680000) at /home/cborntra/REPOS/qemu/target-s390x/kvm.c:2028
> #15 0x000000001005df70 in kvm_cpu_exec (cpu=cpu@entry=0x10c6d7d0) at /home/cborntra/REPOS/qemu/kvm-all.c:1921
> #16 0x000000001004b1be in qemu_kvm_cpu_thread_fn (arg=0x10c6d7d0) at /home/cborntra/REPOS/qemu/cpus.c:1050
> #17 0x000003ff7cf084c6 in start_thread () from /lib64/libpthread.so.0
> #18 0x000003ff7ce02ec2 in thread_start () from /lib64/libc.so.6
>
next prev parent reply other threads:[~2016-03-17 15:17 UTC|newest]
Thread overview: 70+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-03-16 10:10 [Qemu-devel] [PATCH 0/4] Tweaks around virtio-blk start/stop Fam Zheng
2016-03-16 10:10 ` [Qemu-devel] [PATCH 1/4] block: Use drained section in bdrv_set_aio_context Fam Zheng
2016-03-16 10:27 ` Paolo Bonzini
2016-03-16 10:51 ` Fam Zheng
2016-03-16 10:10 ` [Qemu-devel] [PATCH 2/4] block-backend: Introduce blk_drained_begin/end Fam Zheng
2016-03-16 10:10 ` [Qemu-devel] [PATCH 3/4] virtio-blk: Use blk_drained_begin/end around dataplane stop Fam Zheng
2016-03-16 10:10 ` [Qemu-devel] [PATCH 4/4] virtio-blk: Clean up start/stop with mutex and BH Fam Zheng
2016-03-17 15:00 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2016-03-17 15:07 ` Paolo Bonzini
2016-03-22 12:52 ` Fam Zheng
2016-03-22 18:05 ` Paolo Bonzini
2016-03-23 8:10 ` Cornelia Huck
2016-03-23 9:08 ` Paolo Bonzini
2016-03-23 9:12 ` Christian Borntraeger
2016-03-24 8:19 ` tu bo
2016-03-24 8:32 ` Cornelia Huck
2016-03-24 8:47 ` Cornelia Huck
2016-03-24 9:31 ` Cornelia Huck
2016-03-16 10:28 ` [Qemu-devel] [PATCH 0/4] Tweaks around virtio-blk start/stop Paolo Bonzini
2016-03-16 10:49 ` Christian Borntraeger
2016-03-16 11:09 ` Paolo Bonzini
2016-03-16 11:24 ` Christian Borntraeger
2016-03-16 12:55 ` Paolo Bonzini
2016-03-16 13:38 ` Christian Borntraeger
2016-03-16 13:45 ` Paolo Bonzini
2016-03-17 0:39 ` Fam Zheng
2016-03-17 11:03 ` tu bo
2016-03-21 10:57 ` Fam Zheng
2016-03-21 11:15 ` Cornelia Huck
2016-03-21 12:45 ` Fam Zheng
2016-03-21 13:02 ` Cornelia Huck
2016-03-21 23:45 ` Fam Zheng
2016-03-22 8:06 ` Cornelia Huck
2016-03-22 7:10 ` tu bo
2016-03-22 7:18 ` Fam Zheng
2016-03-22 9:07 ` Cornelia Huck
2016-03-22 9:46 ` Paolo Bonzini
2016-03-22 11:59 ` Cornelia Huck
2016-03-22 12:11 ` Paolo Bonzini
2016-03-22 12:54 ` Cornelia Huck
2016-03-17 12:22 ` tu bo
2016-03-17 12:39 ` Christian Borntraeger
2016-03-17 13:02 ` Cornelia Huck
2016-03-17 15:02 ` Paolo Bonzini
2016-03-17 15:07 ` Christian Borntraeger
2016-03-17 15:15 ` Christian Borntraeger
2016-03-17 15:16 ` Christian Borntraeger [this message]
2016-03-17 16:08 ` Christian Borntraeger
2016-03-18 15:03 ` Paolo Bonzini
2016-03-21 9:42 ` Fam Zheng
2016-03-21 11:10 ` Christian Borntraeger
2016-03-21 12:17 ` Cornelia Huck
2016-03-21 13:47 ` TU BO
2016-03-21 13:54 ` Paolo Bonzini
2016-03-21 14:19 ` Cornelia Huck
2016-03-22 0:31 ` Fam Zheng
2016-03-16 11:32 ` Cornelia Huck
2016-03-16 11:48 ` Paolo Bonzini
2016-03-16 11:56 ` Cornelia Huck
2016-03-16 11:59 ` Paolo Bonzini
2016-03-16 12:22 ` Cornelia Huck
2016-03-16 12:32 ` Paolo Bonzini
2016-03-16 12:42 ` Cornelia Huck
2016-03-16 12:49 ` Paolo Bonzini
2016-03-16 13:04 ` Cornelia Huck
2016-03-16 13:10 ` Paolo Bonzini
2016-03-16 13:14 ` Cornelia Huck
2016-03-16 13:15 ` Paolo Bonzini
2016-03-16 11:52 ` Cornelia Huck
2016-03-16 11:54 ` Paolo Bonzini
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=56EACA64.9060402@de.ibm.com \
--to=borntraeger@de.ibm.com \
--cc=cornelia.huck@de.ibm.com \
--cc=famz@redhat.com \
--cc=kwolf@redhat.com \
--cc=mst@redhat.com \
--cc=pbonzini@redhat.com \
--cc=qemu-block@nongnu.org \
--cc=qemu-devel@nongnu.org \
--cc=stefanha@redhat.com \
--cc=tubo@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.