From: Paolo Bonzini <pbonzini@redhat.com>
To: Stefan Weil <sw@weilnetz.de>
Cc: "Michael W. Bombardieri" <mb@ii.net>,
qemu-devel@nongnu.org, Stefan Hajnoczi <stefanha@redhat.com>
Subject: [Qemu-devel] broken win32 coroutines (was Re: qemu 1.6.1)
Date: Mon, 23 Jun 2014 16:39:30 +0200 [thread overview]
Message-ID: <53A83C22.6090208@redhat.com> (raw)
In-Reply-To: <526B908B.20104@weilnetz.de>
Il 26/10/2013 11:51, Stefan Weil ha scritto:
> unpatched QEMU, crash with assertion
>
> 00448670 <_qemu_coroutine_switch>:
> 448670: 53 push %ebx
> 448671: 83 ec 18 sub $0x18,%esp
> 448674: c7 04 24 a8 62 6d 00 movl $0x6d62a8,(%esp)
> 44867b: 8b 5c 24 24 mov 0x24(%esp),%ebx
> 44867f: e8 ec 9e 27 00 call 6c2570
> <___emutls_get_address>
> 448684: 89 18 mov %ebx,(%eax)
> 448686: 8b 44 24 28 mov 0x28(%esp),%eax
> 44868a: 89 43 24 mov %eax,0x24(%ebx)
> 44868d: 8b 43 20 mov 0x20(%ebx),%eax
> 448690: 89 04 24 mov %eax,(%esp)
> 448693: ff 15 c0 5f 83 00 call *0x835fc0
> 448699: 83 ec 04 sub $0x4,%esp
> 44869c: 8b 44 24 20 mov 0x20(%esp),%eax
> 4486a0: 8b 40 24 mov 0x24(%eax),%eax
> 4486a3: 83 c4 18 add $0x18,%esp
> 4486a6: 5b pop %ebx
> 4486a7: c3 ret
>
>
> patched, works
>
> 00448620 <_qemu_coroutine_switch>:
> 448620: 83 ec 1c sub $0x1c,%esp
> 448623: c7 04 24 a8 62 6d 00 movl $0x6d62a8,(%esp)
> 44862a: 89 5c 24 14 mov %ebx,0x14(%esp)
> 44862e: 8b 5c 24 24 mov 0x24(%esp),%ebx
> 448632: 89 74 24 18 mov %esi,0x18(%esp)
> 448636: e8 25 9f 27 00 call 6c2560
> <___emutls_get_address>
> 44863b: 8b 30 mov (%eax),%esi
> 44863d: 89 18 mov %ebx,(%eax)
> 44863f: 8b 44 24 28 mov 0x28(%esp),%eax
> 448643: 89 43 24 mov %eax,0x24(%ebx)
> 448646: 8b 43 20 mov 0x20(%ebx),%eax
> 448649: 89 04 24 mov %eax,(%esp)
> 44864c: ff 15 c0 5f 83 00 call *0x835fc0
> 448652: 8b 46 24 mov 0x24(%esi),%eax
> 448655: 83 ec 04 sub $0x4,%esp
> 448658: 8b 5c 24 14 mov 0x14(%esp),%ebx
> 44865c: 8b 74 24 18 mov 0x18(%esp),%esi
> 448660: 83 c4 1c add $0x1c,%esp
> 448663: c3 ret
The only difference here is basically that "from" is being saved in
%esi across the calls to __emutls_get_address and SwitchToFiber. But
as Peter found out, the fix really happens because now the compiler
will not inline qemu_coroutine_switch anymore.
Peter provided another dump, this time from Win64. It is the
coroutine_trampoline with inlined qemu_coroutine_switch:
0: push %rdi
1: push %rsi
2: push %rbx
3: sub $0x30,%rsp
7: mov %rcx,%rbx
a: lea ...(%rip),%rcx
11: mov ...(%rip),%rax
18: mov %rax,0x28(%rsp)
1d: xor %eax,%eax
1f: callq ___emutls_get_address
24: mov ...(%rip),%rsi
2b: mov %rax,%rdi
2e: nop2
30: mov 0x8(%rbx),%rcx # ecx = co->entry_arg
34: callq *(%rbx) # co->entry(co->entry_arg);
36: mov 0x10(%rbx),%rax # load co->caller
3a: mov %rax,(%rdi) # "current" = co->caller; (wrong current!)
3d: movl $0x2,0x48(%rax) # co->caller->action = COROUTINE_TERMINATE;
44: mov 0x40(%rax),%rcx # load co->caller->fiber
48: callq *%rsi
4a: jmp 30 <coroutine_trampoline+0x30>
4c: nopl 0x0(%rax)
Some offsets are incomplete because this is from a .o, so not
linked, but it's already enough to see that ___emutls_get_address
(from "current = to_;" in qemu_coroutine_switch) is being hoisted
outside the loop, which becomes:
Coroutine *co = co_;
Coroutine **p_current = ¤t;
while (true) {
co->entry(co->entry_arg);
CoroutineWin32 *from = DO_UPCAST(CoroutineWin32, base, co);
CoroutineWin32 *to = DO_UPCAST(CoroutineWin32, base, co->caller);
*p_current = to_;
to->action = action;
SwitchToFiber(to->fiber);
return from->action;
}
This is wrong if co->entry yields out of the coroutine which is then
restarted on another thread. This can happen quite often. Typically.
a coroutine is started when a VCPU thread does bdrv_aio_readv:
VCPU thread
main VCPU thread coroutine I/O coroutine
bdrv_aio_readv ----->
start I/O operation
thread_pool_submit_co
<------------ yields
back to emulation
Then I/O finishes and the thread-pool.c event notifier triggers in
the I/O thread. event_notifier_ready calls thread_pool_co_cb, and
the I/O coroutine now restarts *in another thread*:
iothread
main iothread coroutine I/O coroutine (formerly in VCPU thread)
event_notifier_ready
thread_pool_co_cb -----> current = I/O coroutine;
call AIO callback
But on Win32, because of the bug, the "current" being set here the
current coroutine of the VCPU thread, not the iothread.
noinline is a good-enough workaround, and quite unlikely to break in
the future.
Paolo
next prev parent reply other threads:[~2014-06-23 14:39 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-10-23 7:39 [Qemu-devel] qemu 1.6.1 Michael W. Bombardieri
2013-10-23 9:00 ` Paolo Bonzini
2013-10-23 20:26 ` Stefan Weil
2013-10-24 10:38 ` Paolo Bonzini
2013-10-24 16:37 ` Stefan Weil
2013-10-24 21:47 ` Paolo Bonzini
2013-10-26 9:51 ` Stefan Weil
2013-10-27 6:54 ` Paolo Bonzini
2013-10-27 10:44 ` Stefan Weil
2013-10-27 15:38 ` Stefan Weil
2014-06-23 14:39 ` Paolo Bonzini [this message]
2014-06-24 1:41 ` [Qemu-devel] broken win32 coroutines (was Re: qemu 1.6.1) Michael W. Bombardieri
2014-06-24 5:22 ` Paolo Bonzini
2014-06-25 6:48 ` Michael W. Bombardieri
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=53A83C22.6090208@redhat.com \
--to=pbonzini@redhat.com \
--cc=mb@ii.net \
--cc=qemu-devel@nongnu.org \
--cc=stefanha@redhat.com \
--cc=sw@weilnetz.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.