* [PATCH v2 0/1] migration: multifd live migration improvement
@ 2021-12-03 11:55 Li Zhang
2021-12-03 11:55 ` [PATCH v2 1/1] multifd: Shut down the QIO channels to avoid blocking the send threads when they are terminated Li Zhang
2021-12-06 19:54 ` [PATCH v2 0/1] migration: multifd live migration improvement Dr. David Alan Gilbert
0 siblings, 2 replies; 11+ messages in thread
From: Li Zhang @ 2021-12-03 11:55 UTC (permalink / raw)
To: dgilbert, quintela, berrange, cfontana, qemu-devel; +Cc: Li Zhang
When testing live migration with multifd channels (8, 16, or a bigger number)
and using qemu -incoming (without "defer"), if a network error occurs
(for example, triggering the kernel SYN flooding detection),
the migration fails and the guest hangs forever.
The test environment and the command line is as the following:
QEMU verions: QEMU emulator version 6.2.91 (v6.2.0-rc1-47-gc5fbdd60cf)
Host OS: SLE 15 with kernel: 5.14.5-1-default
Network Card: mlx5 100Gbps
Network card: Intel Corporation I350 Gigabit (1Gbps)
Source:
qemu-system-x86_64 -M q35 -smp 32 -nographic \
-serial telnet:10.156.208.153:4321,server,nowait \
-m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
-monitor stdio
Dest:
qemu-system-x86_64 -M q35 -smp 32 -nographic \
-serial telnet:10.156.208.154:4321,server,nowait \
-m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
-monitor stdio \
-incoming tcp:1.0.8.154:4000
(qemu) migrate_set_parameter max-bandwidth 100G
(qemu) migrate_set_capability multifd on
(qemu) migrate_set_parameter multifd-channels 16
The guest hangs when executing the command: migrate -d tcp:1.0.8.154:4000.
If a network problem happens, TCP ACK is not received by destination
and the destination resets the connection with RST.
No. Time Source Destination Protocol Length Info
119 1.021169 1.0.8.153 1.0.8.154 TCP 1410 60166 → 4000 [PSH, ACK] Seq=65 Ack=1 Win=62720 Len=1344 TSval=1338662881 TSecr=1399531897
No. Time Source Destination Protocol Length Info
125 1.021181 1.0.8.154 1.0.8.153 TCP 54 4000 → 60166 [RST] Seq=1 Win=0 Len=0
kernel log:
[334520.229445] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
[334562.994919] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
[334695.519927] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
[334734.689511] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
[335687.740415] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
[335730.013598] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
There are two problems here:
1. On the send side, the main thread is blocked on qemu_thread_join and
send threads are blocked on sendmsg
2. On receive side, the receive threads are blocked on qemu_sem_wait to
wait for a semaphore.
The patch is to fix the first problem, and the guest doesn't hang any more.
But there is no better solution to fix the second problem yet.
Li Zhang (1):
multifd: Shut down the QIO channels to avoid blocking the send threads
when they are terminated.
migration/multifd.c | 3 +++
1 file changed, 3 insertions(+)
--
2.31.1
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v2 1/1] multifd: Shut down the QIO channels to avoid blocking the send threads when they are terminated.
2021-12-03 11:55 [PATCH v2 0/1] migration: multifd live migration improvement Li Zhang
@ 2021-12-03 11:55 ` Li Zhang
2021-12-03 14:00 ` Daniel P. Berrangé
` (2 more replies)
2021-12-06 19:54 ` [PATCH v2 0/1] migration: multifd live migration improvement Dr. David Alan Gilbert
1 sibling, 3 replies; 11+ messages in thread
From: Li Zhang @ 2021-12-03 11:55 UTC (permalink / raw)
To: dgilbert, quintela, berrange, cfontana, qemu-devel; +Cc: Li Zhang
When doing live migration with multifd channels 8, 16 or larger number,
the guest hangs in the presence of the network errors such as missing TCP ACKs.
At sender's side:
The main thread is blocked on qemu_thread_join, migration_fd_cleanup
is called because one thread fails on qio_channel_write_all when
the network problem happens and other send threads are blocked on sendmsg.
They could not be terminated. So the main thread is blocked on qemu_thread_join
to wait for the threads terminated.
(gdb) bt
0 0x00007f30c8dcffc0 in __pthread_clockjoin_ex () at /lib64/libpthread.so.0
1 0x000055cbb716084b in qemu_thread_join (thread=0x55cbb881f418) at ../util/qemu-thread-posix.c:627
2 0x000055cbb6b54e40 in multifd_save_cleanup () at ../migration/multifd.c:542
3 0x000055cbb6b4de06 in migrate_fd_cleanup (s=0x55cbb8024000) at ../migration/migration.c:1808
4 0x000055cbb6b4dfb4 in migrate_fd_cleanup_bh (opaque=0x55cbb8024000) at ../migration/migration.c:1850
5 0x000055cbb7173ac1 in aio_bh_call (bh=0x55cbb7eb98e0) at ../util/async.c:141
6 0x000055cbb7173bcb in aio_bh_poll (ctx=0x55cbb7ebba80) at ../util/async.c:169
7 0x000055cbb715ba4b in aio_dispatch (ctx=0x55cbb7ebba80) at ../util/aio-posix.c:381
8 0x000055cbb7173ffe in aio_ctx_dispatch (source=0x55cbb7ebba80, callback=0x0, user_data=0x0) at ../util/async.c:311
9 0x00007f30c9c8cdf4 in g_main_context_dispatch () at /usr/lib64/libglib-2.0.so.0
10 0x000055cbb71851a2 in glib_pollfds_poll () at ../util/main-loop.c:232
11 0x000055cbb718521c in os_host_main_loop_wait (timeout=42251070366) at ../util/main-loop.c:255
12 0x000055cbb7185321 in main_loop_wait (nonblocking=0) at ../util/main-loop.c:531
13 0x000055cbb6e6ba27 in qemu_main_loop () at ../softmmu/runstate.c:726
14 0x000055cbb6ad6fd7 in main (argc=68, argv=0x7ffc0c578888, envp=0x7ffc0c578ab0) at ../softmmu/main.c:50
To make sure that the send threads could be terminated, IO channels should be
shut down to avoid waiting IO.
Signed-off-by: Li Zhang <lizhang@suse.de>
---
migration/multifd.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/migration/multifd.c b/migration/multifd.c
index 7c9deb1921..33f8287969 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -523,6 +523,9 @@ static void multifd_send_terminate_threads(Error *err)
qemu_mutex_lock(&p->mutex);
p->quit = true;
qemu_sem_post(&p->sem);
+ if (p->c) {
+ qio_channel_shutdown(p->c, QIO_CHANNEL_SHUTDOWN_BOTH, NULL);
+ }
qemu_mutex_unlock(&p->mutex);
}
}
--
2.31.1
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH v2 1/1] multifd: Shut down the QIO channels to avoid blocking the send threads when they are terminated.
2021-12-03 11:55 ` [PATCH v2 1/1] multifd: Shut down the QIO channels to avoid blocking the send threads when they are terminated Li Zhang
@ 2021-12-03 14:00 ` Daniel P. Berrangé
2021-12-06 8:55 ` Li Zhang
2021-12-09 10:52 ` Juan Quintela
2 siblings, 0 replies; 11+ messages in thread
From: Daniel P. Berrangé @ 2021-12-03 14:00 UTC (permalink / raw)
To: Li Zhang; +Cc: qemu-devel, cfontana, dgilbert, quintela
On Fri, Dec 03, 2021 at 12:55:33PM +0100, Li Zhang wrote:
> When doing live migration with multifd channels 8, 16 or larger number,
> the guest hangs in the presence of the network errors such as missing TCP ACKs.
>
> At sender's side:
> The main thread is blocked on qemu_thread_join, migration_fd_cleanup
> is called because one thread fails on qio_channel_write_all when
> the network problem happens and other send threads are blocked on sendmsg.
> They could not be terminated. So the main thread is blocked on qemu_thread_join
> to wait for the threads terminated.
>
> (gdb) bt
> 0 0x00007f30c8dcffc0 in __pthread_clockjoin_ex () at /lib64/libpthread.so.0
> 1 0x000055cbb716084b in qemu_thread_join (thread=0x55cbb881f418) at ../util/qemu-thread-posix.c:627
> 2 0x000055cbb6b54e40 in multifd_save_cleanup () at ../migration/multifd.c:542
> 3 0x000055cbb6b4de06 in migrate_fd_cleanup (s=0x55cbb8024000) at ../migration/migration.c:1808
> 4 0x000055cbb6b4dfb4 in migrate_fd_cleanup_bh (opaque=0x55cbb8024000) at ../migration/migration.c:1850
> 5 0x000055cbb7173ac1 in aio_bh_call (bh=0x55cbb7eb98e0) at ../util/async.c:141
> 6 0x000055cbb7173bcb in aio_bh_poll (ctx=0x55cbb7ebba80) at ../util/async.c:169
> 7 0x000055cbb715ba4b in aio_dispatch (ctx=0x55cbb7ebba80) at ../util/aio-posix.c:381
> 8 0x000055cbb7173ffe in aio_ctx_dispatch (source=0x55cbb7ebba80, callback=0x0, user_data=0x0) at ../util/async.c:311
> 9 0x00007f30c9c8cdf4 in g_main_context_dispatch () at /usr/lib64/libglib-2.0.so.0
> 10 0x000055cbb71851a2 in glib_pollfds_poll () at ../util/main-loop.c:232
> 11 0x000055cbb718521c in os_host_main_loop_wait (timeout=42251070366) at ../util/main-loop.c:255
> 12 0x000055cbb7185321 in main_loop_wait (nonblocking=0) at ../util/main-loop.c:531
> 13 0x000055cbb6e6ba27 in qemu_main_loop () at ../softmmu/runstate.c:726
> 14 0x000055cbb6ad6fd7 in main (argc=68, argv=0x7ffc0c578888, envp=0x7ffc0c578ab0) at ../softmmu/main.c:50
>
> To make sure that the send threads could be terminated, IO channels should be
> shut down to avoid waiting IO.
>
> Signed-off-by: Li Zhang <lizhang@suse.de>
> ---
> migration/multifd.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/migration/multifd.c b/migration/multifd.c
> index 7c9deb1921..33f8287969 100644
> --- a/migration/multifd.c
> +++ b/migration/multifd.c
> @@ -523,6 +523,9 @@ static void multifd_send_terminate_threads(Error *err)
> qemu_mutex_lock(&p->mutex);
> p->quit = true;
> qemu_sem_post(&p->sem);
> + if (p->c) {
> + qio_channel_shutdown(p->c, QIO_CHANNEL_SHUTDOWN_BOTH, NULL);
> + }
> qemu_mutex_unlock(&p->mutex);
> }
> }
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 1/1] multifd: Shut down the QIO channels to avoid blocking the send threads when they are terminated.
2021-12-03 11:55 ` [PATCH v2 1/1] multifd: Shut down the QIO channels to avoid blocking the send threads when they are terminated Li Zhang
2021-12-03 14:00 ` Daniel P. Berrangé
@ 2021-12-06 8:55 ` Li Zhang
2021-12-06 19:50 ` Dr. David Alan Gilbert
2021-12-09 10:52 ` Juan Quintela
2 siblings, 1 reply; 11+ messages in thread
From: Li Zhang @ 2021-12-06 8:55 UTC (permalink / raw)
To: dgilbert, quintela, berrange, cfontana, qemu-devel
Thanks for Daniel's review.
Hi David and Juan,
Any comments for this patch?
Thanks
Li
On 12/3/21 12:55 PM, Li Zhang wrote:
> When doing live migration with multifd channels 8, 16 or larger number,
> the guest hangs in the presence of the network errors such as missing TCP ACKs.
>
> At sender's side:
> The main thread is blocked on qemu_thread_join, migration_fd_cleanup
> is called because one thread fails on qio_channel_write_all when
> the network problem happens and other send threads are blocked on sendmsg.
> They could not be terminated. So the main thread is blocked on qemu_thread_join
> to wait for the threads terminated.
>
> (gdb) bt
> 0 0x00007f30c8dcffc0 in __pthread_clockjoin_ex () at /lib64/libpthread.so.0
> 1 0x000055cbb716084b in qemu_thread_join (thread=0x55cbb881f418) at ../util/qemu-thread-posix.c:627
> 2 0x000055cbb6b54e40 in multifd_save_cleanup () at ../migration/multifd.c:542
> 3 0x000055cbb6b4de06 in migrate_fd_cleanup (s=0x55cbb8024000) at ../migration/migration.c:1808
> 4 0x000055cbb6b4dfb4 in migrate_fd_cleanup_bh (opaque=0x55cbb8024000) at ../migration/migration.c:1850
> 5 0x000055cbb7173ac1 in aio_bh_call (bh=0x55cbb7eb98e0) at ../util/async.c:141
> 6 0x000055cbb7173bcb in aio_bh_poll (ctx=0x55cbb7ebba80) at ../util/async.c:169
> 7 0x000055cbb715ba4b in aio_dispatch (ctx=0x55cbb7ebba80) at ../util/aio-posix.c:381
> 8 0x000055cbb7173ffe in aio_ctx_dispatch (source=0x55cbb7ebba80, callback=0x0, user_data=0x0) at ../util/async.c:311
> 9 0x00007f30c9c8cdf4 in g_main_context_dispatch () at /usr/lib64/libglib-2.0.so.0
> 10 0x000055cbb71851a2 in glib_pollfds_poll () at ../util/main-loop.c:232
> 11 0x000055cbb718521c in os_host_main_loop_wait (timeout=42251070366) at ../util/main-loop.c:255
> 12 0x000055cbb7185321 in main_loop_wait (nonblocking=0) at ../util/main-loop.c:531
> 13 0x000055cbb6e6ba27 in qemu_main_loop () at ../softmmu/runstate.c:726
> 14 0x000055cbb6ad6fd7 in main (argc=68, argv=0x7ffc0c578888, envp=0x7ffc0c578ab0) at ../softmmu/main.c:50
>
> To make sure that the send threads could be terminated, IO channels should be
> shut down to avoid waiting IO.
>
> Signed-off-by: Li Zhang <lizhang@suse.de>
> ---
> migration/multifd.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/migration/multifd.c b/migration/multifd.c
> index 7c9deb1921..33f8287969 100644
> --- a/migration/multifd.c
> +++ b/migration/multifd.c
> @@ -523,6 +523,9 @@ static void multifd_send_terminate_threads(Error *err)
> qemu_mutex_lock(&p->mutex);
> p->quit = true;
> qemu_sem_post(&p->sem);
> + if (p->c) {
> + qio_channel_shutdown(p->c, QIO_CHANNEL_SHUTDOWN_BOTH, NULL);
> + }
> qemu_mutex_unlock(&p->mutex);
> }
> }
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 1/1] multifd: Shut down the QIO channels to avoid blocking the send threads when they are terminated.
2021-12-06 8:55 ` Li Zhang
@ 2021-12-06 19:50 ` Dr. David Alan Gilbert
2021-12-07 13:49 ` Li Zhang
0 siblings, 1 reply; 11+ messages in thread
From: Dr. David Alan Gilbert @ 2021-12-06 19:50 UTC (permalink / raw)
To: Li Zhang; +Cc: qemu-devel, berrange, cfontana, quintela
* Li Zhang (lizhang@suse.de) wrote:
>
> Thanks for Daniel's review.
>
> Hi David and Juan,
>
> Any comments for this patch?
>
Yeh I think that's OK, so
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
I'd have a slight preference for it being before the post I think.
Dave
>
> Thanks
>
> Li
>
> On 12/3/21 12:55 PM, Li Zhang wrote:
> > When doing live migration with multifd channels 8, 16 or larger number,
> > the guest hangs in the presence of the network errors such as missing TCP ACKs.
> >
> > At sender's side:
> > The main thread is blocked on qemu_thread_join, migration_fd_cleanup
> > is called because one thread fails on qio_channel_write_all when
> > the network problem happens and other send threads are blocked on sendmsg.
> > They could not be terminated. So the main thread is blocked on qemu_thread_join
> > to wait for the threads terminated.
> >
> > (gdb) bt
> > 0 0x00007f30c8dcffc0 in __pthread_clockjoin_ex () at /lib64/libpthread.so.0
> > 1 0x000055cbb716084b in qemu_thread_join (thread=0x55cbb881f418) at ../util/qemu-thread-posix.c:627
> > 2 0x000055cbb6b54e40 in multifd_save_cleanup () at ../migration/multifd.c:542
> > 3 0x000055cbb6b4de06 in migrate_fd_cleanup (s=0x55cbb8024000) at ../migration/migration.c:1808
> > 4 0x000055cbb6b4dfb4 in migrate_fd_cleanup_bh (opaque=0x55cbb8024000) at ../migration/migration.c:1850
> > 5 0x000055cbb7173ac1 in aio_bh_call (bh=0x55cbb7eb98e0) at ../util/async.c:141
> > 6 0x000055cbb7173bcb in aio_bh_poll (ctx=0x55cbb7ebba80) at ../util/async.c:169
> > 7 0x000055cbb715ba4b in aio_dispatch (ctx=0x55cbb7ebba80) at ../util/aio-posix.c:381
> > 8 0x000055cbb7173ffe in aio_ctx_dispatch (source=0x55cbb7ebba80, callback=0x0, user_data=0x0) at ../util/async.c:311
> > 9 0x00007f30c9c8cdf4 in g_main_context_dispatch () at /usr/lib64/libglib-2.0.so.0
> > 10 0x000055cbb71851a2 in glib_pollfds_poll () at ../util/main-loop.c:232
> > 11 0x000055cbb718521c in os_host_main_loop_wait (timeout=42251070366) at ../util/main-loop.c:255
> > 12 0x000055cbb7185321 in main_loop_wait (nonblocking=0) at ../util/main-loop.c:531
> > 13 0x000055cbb6e6ba27 in qemu_main_loop () at ../softmmu/runstate.c:726
> > 14 0x000055cbb6ad6fd7 in main (argc=68, argv=0x7ffc0c578888, envp=0x7ffc0c578ab0) at ../softmmu/main.c:50
> >
> > To make sure that the send threads could be terminated, IO channels should be
> > shut down to avoid waiting IO.
> >
> > Signed-off-by: Li Zhang <lizhang@suse.de>
> > ---
> > migration/multifd.c | 3 +++
> > 1 file changed, 3 insertions(+)
> >
> > diff --git a/migration/multifd.c b/migration/multifd.c
> > index 7c9deb1921..33f8287969 100644
> > --- a/migration/multifd.c
> > +++ b/migration/multifd.c
> > @@ -523,6 +523,9 @@ static void multifd_send_terminate_threads(Error *err)
> > qemu_mutex_lock(&p->mutex);
> > p->quit = true;
> > qemu_sem_post(&p->sem);
> > + if (p->c) {
> > + qio_channel_shutdown(p->c, QIO_CHANNEL_SHUTDOWN_BOTH, NULL);
> > + }
> > qemu_mutex_unlock(&p->mutex);
> > }
> > }
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/1] migration: multifd live migration improvement
2021-12-03 11:55 [PATCH v2 0/1] migration: multifd live migration improvement Li Zhang
2021-12-03 11:55 ` [PATCH v2 1/1] multifd: Shut down the QIO channels to avoid blocking the send threads when they are terminated Li Zhang
@ 2021-12-06 19:54 ` Dr. David Alan Gilbert
2021-12-07 13:45 ` Li Zhang
1 sibling, 1 reply; 11+ messages in thread
From: Dr. David Alan Gilbert @ 2021-12-06 19:54 UTC (permalink / raw)
To: Li Zhang; +Cc: qemu-devel, berrange, cfontana, quintela
* Li Zhang (lizhang@suse.de) wrote:
> When testing live migration with multifd channels (8, 16, or a bigger number)
> and using qemu -incoming (without "defer"), if a network error occurs
> (for example, triggering the kernel SYN flooding detection),
> the migration fails and the guest hangs forever.
>
> The test environment and the command line is as the following:
>
> QEMU verions: QEMU emulator version 6.2.91 (v6.2.0-rc1-47-gc5fbdd60cf)
> Host OS: SLE 15 with kernel: 5.14.5-1-default
> Network Card: mlx5 100Gbps
> Network card: Intel Corporation I350 Gigabit (1Gbps)
>
> Source:
> qemu-system-x86_64 -M q35 -smp 32 -nographic \
> -serial telnet:10.156.208.153:4321,server,nowait \
> -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
> -monitor stdio
> Dest:
> qemu-system-x86_64 -M q35 -smp 32 -nographic \
> -serial telnet:10.156.208.154:4321,server,nowait \
> -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
> -monitor stdio \
> -incoming tcp:1.0.8.154:4000
>
> (qemu) migrate_set_parameter max-bandwidth 100G
> (qemu) migrate_set_capability multifd on
> (qemu) migrate_set_parameter multifd-channels 16
>
> The guest hangs when executing the command: migrate -d tcp:1.0.8.154:4000.
>
> If a network problem happens, TCP ACK is not received by destination
> and the destination resets the connection with RST.
>
> No. Time Source Destination Protocol Length Info
> 119 1.021169 1.0.8.153 1.0.8.154 TCP 1410 60166 → 4000 [PSH, ACK] Seq=65 Ack=1 Win=62720 Len=1344 TSval=1338662881 TSecr=1399531897
> No. Time Source Destination Protocol Length Info
> 125 1.021181 1.0.8.154 1.0.8.153 TCP 54 4000 → 60166 [RST] Seq=1 Win=0 Len=0
>
> kernel log:
> [334520.229445] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
> [334562.994919] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
> [334695.519927] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
> [334734.689511] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
> [335687.740415] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
> [335730.013598] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
Should we document somewhere how to avoid that? Is there something we
should be doing in the connection code to avoid it?
Dave
> There are two problems here:
> 1. On the send side, the main thread is blocked on qemu_thread_join and
> send threads are blocked on sendmsg
> 2. On receive side, the receive threads are blocked on qemu_sem_wait to
> wait for a semaphore.
>
> The patch is to fix the first problem, and the guest doesn't hang any more.
> But there is no better solution to fix the second problem yet.
>
> Li Zhang (1):
> multifd: Shut down the QIO channels to avoid blocking the send threads
> when they are terminated.
>
> migration/multifd.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> --
> 2.31.1
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/1] migration: multifd live migration improvement
2021-12-06 19:54 ` [PATCH v2 0/1] migration: multifd live migration improvement Dr. David Alan Gilbert
@ 2021-12-07 13:45 ` Li Zhang
2021-12-07 14:16 ` Daniel P. Berrangé
0 siblings, 1 reply; 11+ messages in thread
From: Li Zhang @ 2021-12-07 13:45 UTC (permalink / raw)
To: Dr. David Alan Gilbert; +Cc: quintela, berrange, qemu-devel, cfontana
On 12/6/21 8:54 PM, Dr. David Alan Gilbert wrote:
> * Li Zhang (lizhang@suse.de) wrote:
>> When testing live migration with multifd channels (8, 16, or a bigger number)
>> and using qemu -incoming (without "defer"), if a network error occurs
>> (for example, triggering the kernel SYN flooding detection),
>> the migration fails and the guest hangs forever.
>>
>> The test environment and the command line is as the following:
>>
>> QEMU verions: QEMU emulator version 6.2.91 (v6.2.0-rc1-47-gc5fbdd60cf)
>> Host OS: SLE 15 with kernel: 5.14.5-1-default
>> Network Card: mlx5 100Gbps
>> Network card: Intel Corporation I350 Gigabit (1Gbps)
>>
>> Source:
>> qemu-system-x86_64 -M q35 -smp 32 -nographic \
>> -serial telnet:10.156.208.153:4321,server,nowait \
>> -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
>> -monitor stdio
>> Dest:
>> qemu-system-x86_64 -M q35 -smp 32 -nographic \
>> -serial telnet:10.156.208.154:4321,server,nowait \
>> -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
>> -monitor stdio \
>> -incoming tcp:1.0.8.154:4000
>>
>> (qemu) migrate_set_parameter max-bandwidth 100G
>> (qemu) migrate_set_capability multifd on
>> (qemu) migrate_set_parameter multifd-channels 16
>>
>> The guest hangs when executing the command: migrate -d tcp:1.0.8.154:4000.
>>
>> If a network problem happens, TCP ACK is not received by destination
>> and the destination resets the connection with RST.
>>
>> No. Time Source Destination Protocol Length Info
>> 119 1.021169 1.0.8.153 1.0.8.154 TCP 1410 60166 → 4000 [PSH, ACK] Seq=65 Ack=1 Win=62720 Len=1344 TSval=1338662881 TSecr=1399531897
>> No. Time Source Destination Protocol Length Info
>> 125 1.021181 1.0.8.154 1.0.8.153 TCP 54 4000 → 60166 [RST] Seq=1 Win=0 Len=0
>>
>> kernel log:
>> [334520.229445] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
>> [334562.994919] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
>> [334695.519927] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
>> [334734.689511] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
>> [335687.740415] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
>> [335730.013598] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
> Should we document somewhere how to avoid that? Is there something we
> should be doing in the connection code to avoid it?
We should use the command line -incoming defer in QEMU command line
instead of -incoming ip:port.
And the backlog of the socket will be set as the same as multifd
channels, this problem doesn't happen as far as I test.
If we use --incoming ip:port in the QEMU command line, the backlog of
the socket is always 1, it will cause the SYN flooding.
>
> Dave
>
>> There are two problems here:
>> 1. On the send side, the main thread is blocked on qemu_thread_join and
>> send threads are blocked on sendmsg
>> 2. On receive side, the receive threads are blocked on qemu_sem_wait to
>> wait for a semaphore.
>>
>> The patch is to fix the first problem, and the guest doesn't hang any more.
>> But there is no better solution to fix the second problem yet.
>>
>> Li Zhang (1):
>> multifd: Shut down the QIO channels to avoid blocking the send threads
>> when they are terminated.
>>
>> migration/multifd.c | 3 +++
>> 1 file changed, 3 insertions(+)
>>
>> --
>> 2.31.1
>>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 1/1] multifd: Shut down the QIO channels to avoid blocking the send threads when they are terminated.
2021-12-06 19:50 ` Dr. David Alan Gilbert
@ 2021-12-07 13:49 ` Li Zhang
0 siblings, 0 replies; 11+ messages in thread
From: Li Zhang @ 2021-12-07 13:49 UTC (permalink / raw)
To: Dr. David Alan Gilbert; +Cc: qemu-devel, berrange, cfontana, quintela
On 12/6/21 8:50 PM, Dr. David Alan Gilbert wrote:
> * Li Zhang (lizhang@suse.de) wrote:
>> Thanks for Daniel's review.
>>
>> Hi David and Juan,
>>
>> Any comments for this patch?
>>
> Yeh I think that's OK, so
>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>
> I'd have a slight preference for it being before the post I think.
Thanks.
>
> Dave
>
>> Thanks
>>
>> Li
>>
>> On 12/3/21 12:55 PM, Li Zhang wrote:
>>> When doing live migration with multifd channels 8, 16 or larger number,
>>> the guest hangs in the presence of the network errors such as missing TCP ACKs.
>>>
>>> At sender's side:
>>> The main thread is blocked on qemu_thread_join, migration_fd_cleanup
>>> is called because one thread fails on qio_channel_write_all when
>>> the network problem happens and other send threads are blocked on sendmsg.
>>> They could not be terminated. So the main thread is blocked on qemu_thread_join
>>> to wait for the threads terminated.
>>>
>>> (gdb) bt
>>> 0 0x00007f30c8dcffc0 in __pthread_clockjoin_ex () at /lib64/libpthread.so.0
>>> 1 0x000055cbb716084b in qemu_thread_join (thread=0x55cbb881f418) at ../util/qemu-thread-posix.c:627
>>> 2 0x000055cbb6b54e40 in multifd_save_cleanup () at ../migration/multifd.c:542
>>> 3 0x000055cbb6b4de06 in migrate_fd_cleanup (s=0x55cbb8024000) at ../migration/migration.c:1808
>>> 4 0x000055cbb6b4dfb4 in migrate_fd_cleanup_bh (opaque=0x55cbb8024000) at ../migration/migration.c:1850
>>> 5 0x000055cbb7173ac1 in aio_bh_call (bh=0x55cbb7eb98e0) at ../util/async.c:141
>>> 6 0x000055cbb7173bcb in aio_bh_poll (ctx=0x55cbb7ebba80) at ../util/async.c:169
>>> 7 0x000055cbb715ba4b in aio_dispatch (ctx=0x55cbb7ebba80) at ../util/aio-posix.c:381
>>> 8 0x000055cbb7173ffe in aio_ctx_dispatch (source=0x55cbb7ebba80, callback=0x0, user_data=0x0) at ../util/async.c:311
>>> 9 0x00007f30c9c8cdf4 in g_main_context_dispatch () at /usr/lib64/libglib-2.0.so.0
>>> 10 0x000055cbb71851a2 in glib_pollfds_poll () at ../util/main-loop.c:232
>>> 11 0x000055cbb718521c in os_host_main_loop_wait (timeout=42251070366) at ../util/main-loop.c:255
>>> 12 0x000055cbb7185321 in main_loop_wait (nonblocking=0) at ../util/main-loop.c:531
>>> 13 0x000055cbb6e6ba27 in qemu_main_loop () at ../softmmu/runstate.c:726
>>> 14 0x000055cbb6ad6fd7 in main (argc=68, argv=0x7ffc0c578888, envp=0x7ffc0c578ab0) at ../softmmu/main.c:50
>>>
>>> To make sure that the send threads could be terminated, IO channels should be
>>> shut down to avoid waiting IO.
>>>
>>> Signed-off-by: Li Zhang <lizhang@suse.de>
>>> ---
>>> migration/multifd.c | 3 +++
>>> 1 file changed, 3 insertions(+)
>>>
>>> diff --git a/migration/multifd.c b/migration/multifd.c
>>> index 7c9deb1921..33f8287969 100644
>>> --- a/migration/multifd.c
>>> +++ b/migration/multifd.c
>>> @@ -523,6 +523,9 @@ static void multifd_send_terminate_threads(Error *err)
>>> qemu_mutex_lock(&p->mutex);
>>> p->quit = true;
>>> qemu_sem_post(&p->sem);
>>> + if (p->c) {
>>> + qio_channel_shutdown(p->c, QIO_CHANNEL_SHUTDOWN_BOTH, NULL);
>>> + }
>>> qemu_mutex_unlock(&p->mutex);
>>> }
>>> }
>>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/1] migration: multifd live migration improvement
2021-12-07 13:45 ` Li Zhang
@ 2021-12-07 14:16 ` Daniel P. Berrangé
2021-12-07 15:32 ` Li Zhang
0 siblings, 1 reply; 11+ messages in thread
From: Daniel P. Berrangé @ 2021-12-07 14:16 UTC (permalink / raw)
To: Li Zhang; +Cc: cfontana, quintela, Dr. David Alan Gilbert, qemu-devel
On Tue, Dec 07, 2021 at 02:45:10PM +0100, Li Zhang wrote:
>
> On 12/6/21 8:54 PM, Dr. David Alan Gilbert wrote:
> > * Li Zhang (lizhang@suse.de) wrote:
> > > When testing live migration with multifd channels (8, 16, or a bigger number)
> > > and using qemu -incoming (without "defer"), if a network error occurs
> > > (for example, triggering the kernel SYN flooding detection),
> > > the migration fails and the guest hangs forever.
> > >
> > > The test environment and the command line is as the following:
> > >
> > > QEMU verions: QEMU emulator version 6.2.91 (v6.2.0-rc1-47-gc5fbdd60cf)
> > > Host OS: SLE 15 with kernel: 5.14.5-1-default
> > > Network Card: mlx5 100Gbps
> > > Network card: Intel Corporation I350 Gigabit (1Gbps)
> > >
> > > Source:
> > > qemu-system-x86_64 -M q35 -smp 32 -nographic \
> > > -serial telnet:10.156.208.153:4321,server,nowait \
> > > -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
> > > -monitor stdio
> > > Dest:
> > > qemu-system-x86_64 -M q35 -smp 32 -nographic \
> > > -serial telnet:10.156.208.154:4321,server,nowait \
> > > -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
> > > -monitor stdio \
> > > -incoming tcp:1.0.8.154:4000
> > >
> > > (qemu) migrate_set_parameter max-bandwidth 100G
> > > (qemu) migrate_set_capability multifd on
> > > (qemu) migrate_set_parameter multifd-channels 16
> > >
> > > The guest hangs when executing the command: migrate -d tcp:1.0.8.154:4000.
> > >
> > > If a network problem happens, TCP ACK is not received by destination
> > > and the destination resets the connection with RST.
> > >
> > > No. Time Source Destination Protocol Length Info
> > > 119 1.021169 1.0.8.153 1.0.8.154 TCP 1410 60166 → 4000 [PSH, ACK] Seq=65 Ack=1 Win=62720 Len=1344 TSval=1338662881 TSecr=1399531897
> > > No. Time Source Destination Protocol Length Info
> > > 125 1.021181 1.0.8.154 1.0.8.153 TCP 54 4000 → 60166 [RST] Seq=1 Win=0 Len=0
> > >
> > > kernel log:
> > > [334520.229445] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
> > > [334562.994919] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
> > > [334695.519927] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
> > > [334734.689511] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
> > > [335687.740415] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
> > > [335730.013598] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
> > Should we document somewhere how to avoid that? Is there something we
> > should be doing in the connection code to avoid it?
>
> We should use the command line -incoming defer in QEMU command line instead
> of -incoming ip:port.
>
> And the backlog of the socket will be set as the same as multifd channels,
> this problem doesn't happen as far as I test.
>
> If we use --incoming ip:port in the QEMU command line, the backlog of the
> socket is always 1, it will cause the SYN flooding.
Do we send migration parameters from the src to the dst QEMU ?
There are a bunch of things that we need to set to the same
value on the src and dst. If we sent any relevant MigrationParameters
fields to the dst, when the first/main migration chanel is opened, it
could validate that it is configured in a way that is compatible with
the src. If it isn't, it can drop the main channel immediately. This
would trigger the src to fail the migration and we couldn't get stuck
setting up the secondary data channels for multifd.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 0/1] migration: multifd live migration improvement
2021-12-07 14:16 ` Daniel P. Berrangé
@ 2021-12-07 15:32 ` Li Zhang
0 siblings, 0 replies; 11+ messages in thread
From: Li Zhang @ 2021-12-07 15:32 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: cfontana, quintela, Dr. David Alan Gilbert, qemu-devel
On 12/7/21 3:16 PM, Daniel P. Berrangé wrote:
> On Tue, Dec 07, 2021 at 02:45:10PM +0100, Li Zhang wrote:
>> On 12/6/21 8:54 PM, Dr. David Alan Gilbert wrote:
>>> * Li Zhang (lizhang@suse.de) wrote:
>>>> When testing live migration with multifd channels (8, 16, or a bigger number)
>>>> and using qemu -incoming (without "defer"), if a network error occurs
>>>> (for example, triggering the kernel SYN flooding detection),
>>>> the migration fails and the guest hangs forever.
>>>>
>>>> The test environment and the command line is as the following:
>>>>
>>>> QEMU verions: QEMU emulator version 6.2.91 (v6.2.0-rc1-47-gc5fbdd60cf)
>>>> Host OS: SLE 15 with kernel: 5.14.5-1-default
>>>> Network Card: mlx5 100Gbps
>>>> Network card: Intel Corporation I350 Gigabit (1Gbps)
>>>>
>>>> Source:
>>>> qemu-system-x86_64 -M q35 -smp 32 -nographic \
>>>> -serial telnet:10.156.208.153:4321,server,nowait \
>>>> -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
>>>> -monitor stdio
>>>> Dest:
>>>> qemu-system-x86_64 -M q35 -smp 32 -nographic \
>>>> -serial telnet:10.156.208.154:4321,server,nowait \
>>>> -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
>>>> -monitor stdio \
>>>> -incoming tcp:1.0.8.154:4000
>>>>
>>>> (qemu) migrate_set_parameter max-bandwidth 100G
>>>> (qemu) migrate_set_capability multifd on
>>>> (qemu) migrate_set_parameter multifd-channels 16
>>>>
>>>> The guest hangs when executing the command: migrate -d tcp:1.0.8.154:4000.
>>>>
>>>> If a network problem happens, TCP ACK is not received by destination
>>>> and the destination resets the connection with RST.
>>>>
>>>> No. Time Source Destination Protocol Length Info
>>>> 119 1.021169 1.0.8.153 1.0.8.154 TCP 1410 60166 → 4000 [PSH, ACK] Seq=65 Ack=1 Win=62720 Len=1344 TSval=1338662881 TSecr=1399531897
>>>> No. Time Source Destination Protocol Length Info
>>>> 125 1.021181 1.0.8.154 1.0.8.153 TCP 54 4000 → 60166 [RST] Seq=1 Win=0 Len=0
>>>>
>>>> kernel log:
>>>> [334520.229445] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
>>>> [334562.994919] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
>>>> [334695.519927] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
>>>> [334734.689511] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
>>>> [335687.740415] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
>>>> [335730.013598] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies. Check SNMP counters.
>>> Should we document somewhere how to avoid that? Is there something we
>>> should be doing in the connection code to avoid it?
>> We should use the command line -incoming defer in QEMU command line instead
>> of -incoming ip:port.
>>
>> And the backlog of the socket will be set as the same as multifd channels,
>> this problem doesn't happen as far as I test.
>>
>> If we use --incoming ip:port in the QEMU command line, the backlog of the
>> socket is always 1, it will cause the SYN flooding.
> Do we send migration parameters from the src to the dst QEMU ?
No, I don't think we send migration parameters from the src to the dest
QEMU.
I set migration parameters on both sides from qemu monitor seperately.
> There are a bunch of things that we need to set to the same
> value on the src and dst. If we sent any relevant MigrationParameters
> fields to the dst, when the first/main migration chanel is opened, it
> could validate that it is configured in a way that is compatible with
> the src. If it isn't, it can drop the main channel immediately. This
> would trigger the src to fail the migration and we couldn't get stuck
> setting up the secondary data channels for multifd.
OK, currently, we have same parameters on both sides if we set them the
same parameters.
If we use -incoming tcp:ip:port because the multifd is disabled by
default and backlog is 1 when the socket is created.
Here is the function which set the backlog:
static void
socket_start_incoming_migration_internal(SocketAddress *saddr,
Error **errp)
{
QIONetListener *listener = qio_net_listener_new();
MigrationIncomingState *mis = migration_incoming_get_current();
size_t i;
int num = 1;
qio_net_listener_set_name(listener, "migration-socket-listener");
if (migrate_use_multifd()) {
num = migrate_multifd_channels();
}
...
}
The process with -incoming tcp:ip:port is as the following:
1. Create qemu process with command line -incoming tcp:ip:port
2. socket_start_incoming_migration_internal is called and backlog is:
num=1, multifd is disabled, num = migrate_multifd_channels() is not called
3. Enable multifd and set multifd parameters, but the backlog is still
1, because the it couldn't be changed anymore.
4. Run migration
The process with -incoming defer is as the following:
1. Create qemu process with command line -incoming defer
2. Enable multifd and set multifd parameters
3. Execute the command (qemu) migrate_incoming tcp:ip:port
4. Call socket_start_incoming_migration_internal then the backlog is
set: num = migrate_multifd_channels();
5. Run migration
>
> Regards,
> Daniel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2 1/1] multifd: Shut down the QIO channels to avoid blocking the send threads when they are terminated.
2021-12-03 11:55 ` [PATCH v2 1/1] multifd: Shut down the QIO channels to avoid blocking the send threads when they are terminated Li Zhang
2021-12-03 14:00 ` Daniel P. Berrangé
2021-12-06 8:55 ` Li Zhang
@ 2021-12-09 10:52 ` Juan Quintela
2 siblings, 0 replies; 11+ messages in thread
From: Juan Quintela @ 2021-12-09 10:52 UTC (permalink / raw)
To: Li Zhang; +Cc: qemu-devel, berrange, dgilbert, cfontana
Li Zhang <lizhang@suse.de> wrote:
> When doing live migration with multifd channels 8, 16 or larger number,
> the guest hangs in the presence of the network errors such as missing TCP ACKs.
>
> At sender's side:
> The main thread is blocked on qemu_thread_join, migration_fd_cleanup
> is called because one thread fails on qio_channel_write_all when
> the network problem happens and other send threads are blocked on sendmsg.
> They could not be terminated. So the main thread is blocked on qemu_thread_join
> to wait for the threads terminated.
>
> (gdb) bt
> 0 0x00007f30c8dcffc0 in __pthread_clockjoin_ex () at /lib64/libpthread.so.0
> 1 0x000055cbb716084b in qemu_thread_join (thread=0x55cbb881f418) at ../util/qemu-thread-posix.c:627
> 2 0x000055cbb6b54e40 in multifd_save_cleanup () at ../migration/multifd.c:542
> 3 0x000055cbb6b4de06 in migrate_fd_cleanup (s=0x55cbb8024000) at ../migration/migration.c:1808
> 4 0x000055cbb6b4dfb4 in migrate_fd_cleanup_bh (opaque=0x55cbb8024000) at ../migration/migration.c:1850
> 5 0x000055cbb7173ac1 in aio_bh_call (bh=0x55cbb7eb98e0) at ../util/async.c:141
> 6 0x000055cbb7173bcb in aio_bh_poll (ctx=0x55cbb7ebba80) at ../util/async.c:169
> 7 0x000055cbb715ba4b in aio_dispatch (ctx=0x55cbb7ebba80) at ../util/aio-posix.c:381
> 8 0x000055cbb7173ffe in aio_ctx_dispatch (source=0x55cbb7ebba80, callback=0x0, user_data=0x0) at ../util/async.c:311
> 9 0x00007f30c9c8cdf4 in g_main_context_dispatch () at /usr/lib64/libglib-2.0.so.0
> 10 0x000055cbb71851a2 in glib_pollfds_poll () at ../util/main-loop.c:232
> 11 0x000055cbb718521c in os_host_main_loop_wait (timeout=42251070366) at ../util/main-loop.c:255
> 12 0x000055cbb7185321 in main_loop_wait (nonblocking=0) at ../util/main-loop.c:531
> 13 0x000055cbb6e6ba27 in qemu_main_loop () at ../softmmu/runstate.c:726
> 14 0x000055cbb6ad6fd7 in main (argc=68, argv=0x7ffc0c578888, envp=0x7ffc0c578ab0) at ../softmmu/main.c:50
>
> To make sure that the send threads could be terminated, IO channels should be
> shut down to avoid waiting IO.
>
> Signed-off-by: Li Zhang <lizhang@suse.de>
Reviewed-by: Juan Quintela <quintela@redhat.com>
queued for 7.0
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-12-09 10:54 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-12-03 11:55 [PATCH v2 0/1] migration: multifd live migration improvement Li Zhang
2021-12-03 11:55 ` [PATCH v2 1/1] multifd: Shut down the QIO channels to avoid blocking the send threads when they are terminated Li Zhang
2021-12-03 14:00 ` Daniel P. Berrangé
2021-12-06 8:55 ` Li Zhang
2021-12-06 19:50 ` Dr. David Alan Gilbert
2021-12-07 13:49 ` Li Zhang
2021-12-09 10:52 ` Juan Quintela
2021-12-06 19:54 ` [PATCH v2 0/1] migration: multifd live migration improvement Dr. David Alan Gilbert
2021-12-07 13:45 ` Li Zhang
2021-12-07 14:16 ` Daniel P. Berrangé
2021-12-07 15:32 ` Li Zhang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.