* [Qemu-devel] [PATCH v4 0/2] char-socket: Fix race condition @ 2019-08-12 15:58 Andrey Shinkevich 2019-08-12 15:58 ` [Qemu-devel] [PATCH v4 1/2] main-loop: Fix GSource leak in qio_task_thread_worker() Andrey Shinkevich ` (2 more replies) 0 siblings, 3 replies; 7+ messages in thread From: Andrey Shinkevich @ 2019-08-12 15:58 UTC (permalink / raw) To: qemu-devel Cc: vsementsov, berrange, berto, marcandre.lureau, andrey.shinkevich, pbonzini, den This fixes a race condition in which the tcp_chr_read() ioc handler can close a connection that is being written to from another thread. v4: The functions qemu_idle_add() and tcp_chr_be_event_closed() were removed because the callback is invoked after the call to object_property_del_all() so, the "chardev" object had been deleted and the segmentation fault occurs. Let's please apply the Alberto's simplified series to avoid the race condition. v3: See the email thread with the Message ID <cover.1550842915.git.berto@igalia.com> Alberto Garcia (2): main-loop: Fix GSource leak in qio_task_thread_worker() char-socket: Lock tcp_chr_disconnect() chardev/char-socket.c | 16 +++++++++++++--- io/task.c | 1 + 2 files changed, 14 insertions(+), 3 deletions(-) -- 1.8.3.1 ^ permalink raw reply [flat|nested] 7+ messages in thread
* [Qemu-devel] [PATCH v4 1/2] main-loop: Fix GSource leak in qio_task_thread_worker() 2019-08-12 15:58 [Qemu-devel] [PATCH v4 0/2] char-socket: Fix race condition Andrey Shinkevich @ 2019-08-12 15:58 ` Andrey Shinkevich 2019-08-12 15:58 ` [Qemu-devel] [PATCH v4 2/2] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout() Andrey Shinkevich 2019-08-21 8:18 ` [Qemu-devel] [PATCH v4 0/2] char-socket: Fix race condition Paolo Bonzini 2 siblings, 0 replies; 7+ messages in thread From: Andrey Shinkevich @ 2019-08-12 15:58 UTC (permalink / raw) To: qemu-devel Cc: vsementsov, berrange, berto, marcandre.lureau, andrey.shinkevich, pbonzini, den From: Alberto Garcia <berto@igalia.com> After g_source_attach() the GMainContext holds a reference to the GSource, so the caller does not need to keep it. qio_task_thread_worker() is not releasing its reference so the GSource is being leaked since a17536c594bfed94d05667b419f747b692f5fc7f. Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> --- io/task.c | 1 + 1 file changed, 1 insertion(+) diff --git a/io/task.c b/io/task.c index 64c4c71..1ae7b86 100644 --- a/io/task.c +++ b/io/task.c @@ -136,6 +136,7 @@ static gpointer qio_task_thread_worker(gpointer opaque) qio_task_thread_result, task, NULL); g_source_attach(task->thread->completion, task->thread->context); + g_source_unref(task->thread->completion); trace_qio_task_thread_source_attach(task, task->thread->completion); qemu_cond_signal(&task->thread_cond); -- 1.8.3.1 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* [Qemu-devel] [PATCH v4 2/2] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout() 2019-08-12 15:58 [Qemu-devel] [PATCH v4 0/2] char-socket: Fix race condition Andrey Shinkevich 2019-08-12 15:58 ` [Qemu-devel] [PATCH v4 1/2] main-loop: Fix GSource leak in qio_task_thread_worker() Andrey Shinkevich @ 2019-08-12 15:58 ` Andrey Shinkevich 2019-08-20 9:58 ` Daniel P. Berrangé 2019-08-21 8:18 ` [Qemu-devel] [PATCH v4 0/2] char-socket: Fix race condition Paolo Bonzini 2 siblings, 1 reply; 7+ messages in thread From: Andrey Shinkevich @ 2019-08-12 15:58 UTC (permalink / raw) To: qemu-devel Cc: vsementsov, berrange, berto, marcandre.lureau, andrey.shinkevich, pbonzini, den From: Alberto Garcia <berto@igalia.com> There's a race condition in which the tcp_chr_read() ioc handler can close a connection that is being written to from another thread. Running iotest 136 in a loop triggers this problem and crashes QEMU. (gdb) bt #0 0x00005558b842902d in object_get_class (obj=0x0) at qom/object.c:860 #1 0x00005558b84f92db in qio_channel_writev_full (ioc=0x0, iov=0x7ffc355decf0, niov=1, fds=0x0, nfds=0, errp=0x0) at io/channel.c:76 #2 0x00005558b84e0e9e in io_channel_send_full (ioc=0x0, buf=0x5558baf5beb0, len=138, fds=0x0, nfds=0) at chardev/char-io.c:123 #3 0x00005558b84e4a69 in tcp_chr_write (chr=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138) at chardev/char-socket.c:135 #4 0x00005558b84dca55 in qemu_chr_write_buffer (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, offset=0x7ffc355dedd0, write_all=false) at chardev/char.c:112 #5 0x00005558b84dcbc2 in qemu_chr_write (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, write_all=false) at chardev/char.c:147 #6 0x00005558b84dfb26 in qemu_chr_fe_write (be=0x5558ba476610, buf=0x5558baf5beb0 "...", len=138) at chardev/char-fe.c:42 #7 0x00005558b8088c86 in monitor_flush_locked (mon=0x5558ba476610) at monitor.c:406 #8 0x00005558b8088e8c in monitor_puts (mon=0x5558ba476610, str=0x5558ba921e49 "") at monitor.c:449 #9 0x00005558b8089178 in qmp_send_response (mon=0x5558ba476610, rsp=0x5558bb161600) at monitor.c:498 #10 0x00005558b808920c in monitor_qapi_event_emit (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:526 #11 0x00005558b8089307 in monitor_qapi_event_queue_no_reenter (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:551 #12 0x00005558b80896c0 in qapi_event_emit (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:626 #13 0x00005558b855f23b in qapi_event_send_shutdown (guest=false, reason=SHUTDOWN_CAUSE_HOST_QMP_QUIT) at qapi/qapi-events-run-state.c:43 #14 0x00005558b81911ef in qemu_system_shutdown (cause=SHUTDOWN_CAUSE_HOST_QMP_QUIT) at vl.c:1837 #15 0x00005558b8191308 in main_loop_should_exit () at vl.c:1885 #16 0x00005558b819140d in main_loop () at vl.c:1924 #17 0x00005558b8198c84 in main (argc=18, argv=0x7ffc355df3f8, envp=0x7ffc355df490) at vl.c:4665 This patch adds a lock to protect tcp_chr_disconnect() and socket_reconnect_timeout() Signed-off-by: Alberto Garcia <berto@igalia.com> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> --- chardev/char-socket.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/chardev/char-socket.c b/chardev/char-socket.c index 7ca5d97..03f0340 100644 --- a/chardev/char-socket.c +++ b/chardev/char-socket.c @@ -150,7 +150,7 @@ static void tcp_chr_accept(QIONetListener *listener, void *opaque); static int tcp_chr_read_poll(void *opaque); -static void tcp_chr_disconnect(Chardev *chr); +static void tcp_chr_disconnect_locked(Chardev *chr); /* Called with chr_write_lock held. */ static int tcp_chr_write(Chardev *chr, const uint8_t *buf, int len) @@ -174,7 +174,7 @@ static int tcp_chr_write(Chardev *chr, const uint8_t *buf, int len) if (ret < 0 && errno != EAGAIN) { if (tcp_chr_read_poll(chr) <= 0) { - tcp_chr_disconnect(chr); + tcp_chr_disconnect_locked(chr); return len; } /* else let the read handler finish it properly */ } @@ -469,8 +469,9 @@ static void update_disconnected_filename(SocketChardev *s) /* NB may be called even if tcp_chr_connect has not been * reached, due to TLS or telnet initialization failure, * so can *not* assume s->state == TCP_CHARDEV_STATE_CONNECTED + * This must be called with chr->chr_write_lock held. */ -static void tcp_chr_disconnect(Chardev *chr) +static void tcp_chr_disconnect_locked(Chardev *chr) { SocketChardev *s = SOCKET_CHARDEV(chr); bool emit_close = s->state == TCP_CHARDEV_STATE_CONNECTED; @@ -490,6 +491,13 @@ static void tcp_chr_disconnect(Chardev *chr) } } +static void tcp_chr_disconnect(Chardev *chr) +{ + qemu_mutex_lock(&chr->chr_write_lock); + tcp_chr_disconnect_locked(chr); + qemu_mutex_unlock(&chr->chr_write_lock); +} + static gboolean tcp_chr_read(QIOChannel *chan, GIOCondition cond, void *opaque) { Chardev *chr = CHARDEV(opaque); @@ -1131,8 +1139,10 @@ static gboolean socket_reconnect_timeout(gpointer opaque) Chardev *chr = CHARDEV(opaque); SocketChardev *s = SOCKET_CHARDEV(opaque); + qemu_mutex_lock(&chr->chr_write_lock); g_source_unref(s->reconnect_timer); s->reconnect_timer = NULL; + qemu_mutex_unlock(&chr->chr_write_lock); if (chr->be_open) { return false; -- 1.8.3.1 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] [PATCH v4 2/2] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout() 2019-08-12 15:58 ` [Qemu-devel] [PATCH v4 2/2] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout() Andrey Shinkevich @ 2019-08-20 9:58 ` Daniel P. Berrangé 2019-08-20 11:10 ` Andrey Shinkevich 0 siblings, 1 reply; 7+ messages in thread From: Daniel P. Berrangé @ 2019-08-20 9:58 UTC (permalink / raw) To: Andrey Shinkevich Cc: vsementsov, berto, qemu-devel, marcandre.lureau, pbonzini, den On Mon, Aug 12, 2019 at 06:58:29PM +0300, Andrey Shinkevich wrote: > From: Alberto Garcia <berto@igalia.com> > > There's a race condition in which the tcp_chr_read() ioc handler can > close a connection that is being written to from another thread. > > Running iotest 136 in a loop triggers this problem and crashes QEMU. > > (gdb) bt > #0 0x00005558b842902d in object_get_class (obj=0x0) at qom/object.c:860 > #1 0x00005558b84f92db in qio_channel_writev_full (ioc=0x0, iov=0x7ffc355decf0, niov=1, fds=0x0, nfds=0, errp=0x0) at io/channel.c:76 > #2 0x00005558b84e0e9e in io_channel_send_full (ioc=0x0, buf=0x5558baf5beb0, len=138, fds=0x0, nfds=0) at chardev/char-io.c:123 > #3 0x00005558b84e4a69 in tcp_chr_write (chr=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138) at chardev/char-socket.c:135 > #4 0x00005558b84dca55 in qemu_chr_write_buffer (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, offset=0x7ffc355dedd0, write_all=false) at chardev/char.c:112 > #5 0x00005558b84dcbc2 in qemu_chr_write (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, write_all=false) at chardev/char.c:147 > #6 0x00005558b84dfb26 in qemu_chr_fe_write (be=0x5558ba476610, buf=0x5558baf5beb0 "...", len=138) at chardev/char-fe.c:42 > #7 0x00005558b8088c86 in monitor_flush_locked (mon=0x5558ba476610) at monitor.c:406 > #8 0x00005558b8088e8c in monitor_puts (mon=0x5558ba476610, str=0x5558ba921e49 "") at monitor.c:449 > #9 0x00005558b8089178 in qmp_send_response (mon=0x5558ba476610, rsp=0x5558bb161600) at monitor.c:498 > #10 0x00005558b808920c in monitor_qapi_event_emit (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:526 > #11 0x00005558b8089307 in monitor_qapi_event_queue_no_reenter (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:551 > #12 0x00005558b80896c0 in qapi_event_emit (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:626 > #13 0x00005558b855f23b in qapi_event_send_shutdown (guest=false, reason=SHUTDOWN_CAUSE_HOST_QMP_QUIT) at qapi/qapi-events-run-state.c:43 > #14 0x00005558b81911ef in qemu_system_shutdown (cause=SHUTDOWN_CAUSE_HOST_QMP_QUIT) at vl.c:1837 > #15 0x00005558b8191308 in main_loop_should_exit () at vl.c:1885 > #16 0x00005558b819140d in main_loop () at vl.c:1924 > #17 0x00005558b8198c84 in main (argc=18, argv=0x7ffc355df3f8, envp=0x7ffc355df490) at vl.c:4665 > > This patch adds a lock to protect tcp_chr_disconnect() and > socket_reconnect_timeout() > > Signed-off-by: Alberto Garcia <berto@igalia.com> > Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] [PATCH v4 2/2] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout() 2019-08-20 9:58 ` Daniel P. Berrangé @ 2019-08-20 11:10 ` Andrey Shinkevich 0 siblings, 0 replies; 7+ messages in thread From: Andrey Shinkevich @ 2019-08-20 11:10 UTC (permalink / raw) To: Daniel P. Berrangé Cc: Vladimir Sementsov-Ogievskiy, berto, Denis Lunev, qemu-devel, marcandre.lureau, pbonzini On 20/08/2019 12:58, Daniel P. Berrangé wrote: > On Mon, Aug 12, 2019 at 06:58:29PM +0300, Andrey Shinkevich wrote: >> From: Alberto Garcia <berto@igalia.com> >> >> There's a race condition in which the tcp_chr_read() ioc handler can >> close a connection that is being written to from another thread. >> >> Running iotest 136 in a loop triggers this problem and crashes QEMU. >> >> (gdb) bt >> #0 0x00005558b842902d in object_get_class (obj=0x0) at qom/object.c:860 >> #1 0x00005558b84f92db in qio_channel_writev_full (ioc=0x0, iov=0x7ffc355decf0, niov=1, fds=0x0, nfds=0, errp=0x0) at io/channel.c:76 >> #2 0x00005558b84e0e9e in io_channel_send_full (ioc=0x0, buf=0x5558baf5beb0, len=138, fds=0x0, nfds=0) at chardev/char-io.c:123 >> #3 0x00005558b84e4a69 in tcp_chr_write (chr=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138) at chardev/char-socket.c:135 >> #4 0x00005558b84dca55 in qemu_chr_write_buffer (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, offset=0x7ffc355dedd0, write_all=false) at chardev/char.c:112 >> #5 0x00005558b84dcbc2 in qemu_chr_write (s=0x5558ba460380, buf=0x5558baf5beb0 "...", len=138, write_all=false) at chardev/char.c:147 >> #6 0x00005558b84dfb26 in qemu_chr_fe_write (be=0x5558ba476610, buf=0x5558baf5beb0 "...", len=138) at chardev/char-fe.c:42 >> #7 0x00005558b8088c86 in monitor_flush_locked (mon=0x5558ba476610) at monitor.c:406 >> #8 0x00005558b8088e8c in monitor_puts (mon=0x5558ba476610, str=0x5558ba921e49 "") at monitor.c:449 >> #9 0x00005558b8089178 in qmp_send_response (mon=0x5558ba476610, rsp=0x5558bb161600) at monitor.c:498 >> #10 0x00005558b808920c in monitor_qapi_event_emit (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:526 >> #11 0x00005558b8089307 in monitor_qapi_event_queue_no_reenter (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:551 >> #12 0x00005558b80896c0 in qapi_event_emit (event=QAPI_EVENT_SHUTDOWN, qdict=0x5558bb161600) at monitor.c:626 >> #13 0x00005558b855f23b in qapi_event_send_shutdown (guest=false, reason=SHUTDOWN_CAUSE_HOST_QMP_QUIT) at qapi/qapi-events-run-state.c:43 >> #14 0x00005558b81911ef in qemu_system_shutdown (cause=SHUTDOWN_CAUSE_HOST_QMP_QUIT) at vl.c:1837 >> #15 0x00005558b8191308 in main_loop_should_exit () at vl.c:1885 >> #16 0x00005558b819140d in main_loop () at vl.c:1924 >> #17 0x00005558b8198c84 in main (argc=18, argv=0x7ffc355df3f8, envp=0x7ffc355df490) at vl.c:4665 >> >> This patch adds a lock to protect tcp_chr_disconnect() and >> socket_reconnect_timeout() >> >> Signed-off-by: Alberto Garcia <berto@igalia.com> >> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> > > Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> > > > Regards, > Daniel > Thank you Daniel Andrey -- With the best regards, Andrey Shinkevich ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] [PATCH v4 0/2] char-socket: Fix race condition 2019-08-12 15:58 [Qemu-devel] [PATCH v4 0/2] char-socket: Fix race condition Andrey Shinkevich 2019-08-12 15:58 ` [Qemu-devel] [PATCH v4 1/2] main-loop: Fix GSource leak in qio_task_thread_worker() Andrey Shinkevich 2019-08-12 15:58 ` [Qemu-devel] [PATCH v4 2/2] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout() Andrey Shinkevich @ 2019-08-21 8:18 ` Paolo Bonzini 2019-08-21 9:51 ` Andrey Shinkevich 2 siblings, 1 reply; 7+ messages in thread From: Paolo Bonzini @ 2019-08-21 8:18 UTC (permalink / raw) To: Andrey Shinkevich, qemu-devel Cc: den, marcandre.lureau, vsementsov, berrange, berto On 12/08/19 17:58, Andrey Shinkevich wrote: > This fixes a race condition in which the tcp_chr_read() ioc handler > can close a connection that is being written to from another thread. > > v4: > The functions qemu_idle_add() and tcp_chr_be_event_closed() were removed > because the callback is invoked after the call to object_property_del_all() > so, the "chardev" object had been deleted and the segmentation fault occurs. > Let's please apply the Alberto's simplified series to avoid the race condition. > > v3: > See the email thread with the Message ID > <cover.1550842915.git.berto@igalia.com> > > Alberto Garcia (2): > main-loop: Fix GSource leak in qio_task_thread_worker() > char-socket: Lock tcp_chr_disconnect() > > chardev/char-socket.c | 16 +++++++++++++--- > io/task.c | 1 + > 2 files changed, 14 insertions(+), 3 deletions(-) > Queued, thanks. Paolo ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] [PATCH v4 0/2] char-socket: Fix race condition 2019-08-21 8:18 ` [Qemu-devel] [PATCH v4 0/2] char-socket: Fix race condition Paolo Bonzini @ 2019-08-21 9:51 ` Andrey Shinkevich 0 siblings, 0 replies; 7+ messages in thread From: Andrey Shinkevich @ 2019-08-21 9:51 UTC (permalink / raw) To: Paolo Bonzini, qemu-devel Cc: marcandre.lureau, Vladimir Sementsov-Ogievskiy, berrange, Denis Lunev, berto On 21/08/2019 11:18, Paolo Bonzini wrote: > On 12/08/19 17:58, Andrey Shinkevich wrote: >> This fixes a race condition in which the tcp_chr_read() ioc handler >> can close a connection that is being written to from another thread. >> >> v4: >> The functions qemu_idle_add() and tcp_chr_be_event_closed() were removed >> because the callback is invoked after the call to object_property_del_all() >> so, the "chardev" object had been deleted and the segmentation fault occurs. >> Let's please apply the Alberto's simplified series to avoid the race condition. >> >> v3: >> See the email thread with the Message ID >> <cover.1550842915.git.berto@igalia.com> >> >> Alberto Garcia (2): >> main-loop: Fix GSource leak in qio_task_thread_worker() >> char-socket: Lock tcp_chr_disconnect() >> >> chardev/char-socket.c | 16 +++++++++++++--- >> io/task.c | 1 + >> 2 files changed, 14 insertions(+), 3 deletions(-) >> > > Queued, thanks. > > Paolo > Thanks you all too. Andrey -- With the best regards, Andrey Shinkevich ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2019-08-21 9:51 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-08-12 15:58 [Qemu-devel] [PATCH v4 0/2] char-socket: Fix race condition Andrey Shinkevich 2019-08-12 15:58 ` [Qemu-devel] [PATCH v4 1/2] main-loop: Fix GSource leak in qio_task_thread_worker() Andrey Shinkevich 2019-08-12 15:58 ` [Qemu-devel] [PATCH v4 2/2] char-socket: Lock tcp_chr_disconnect() and socket_reconnect_timeout() Andrey Shinkevich 2019-08-20 9:58 ` Daniel P. Berrangé 2019-08-20 11:10 ` Andrey Shinkevich 2019-08-21 8:18 ` [Qemu-devel] [PATCH v4 0/2] char-socket: Fix race condition Paolo Bonzini 2019-08-21 9:51 ` Andrey Shinkevich
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.