* [Qemu-devel] [PATCH QEMU 0/5] spee up live migration by skipping free pages
@ 2016-04-19 14:20 ` Liang Li
0 siblings, 0 replies; 19+ messages in thread
From: Liang Li @ 2016-04-19 14:20 UTC (permalink / raw)
To: mst, quintela, amit.shah, pbonzini, lcapitulino
Cc: armbru, peter.maydell, rth, ehabkost, james.hogan, aurelien,
leon.alrae, agraf, borntraeger, cornelia.huck, qemu-devel, kvm,
Liang Li
Current QEMU live migration implementation marks all guest's RAM pages
as dirtied in the ram bulk stage, all these pages will be processed
and it consumes quite a lot of CPU cycles and network bandwidth.
>From guest's point of view, it doesn't care about the content in free
page. We can make use of this fact and skip processing the free
pages, it can save a lot CPU cycles and reduce the network traffic
significantly while speed up the live migration process obviously.
This patch set is the QEMU side implementation.
The virtio-balloon is extended so that QEMU can get the free pages
information from the guest.
After getting the free page bitmap, QEMU can use it to filter out
guest's free pages. This make the live migration process much more
efficient.
In order to skip more free pages, we add an interface to let the user
decide whether dropping the cache in guest during live migration.
Performance data
================
Test environment:
CPU: Intel (R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
Host RAM: 64GB
Host Linux Kernel: 4.4.0 Host OS: CentOS 7.1
Guest Linux Kernel: 4.5.rc6 Guest OS: CentOS 6.6
Network: Intel X710 with 10 Gigabit connection
Guest RAM: 8GB
Case 1: Idle guest just boots:
==================================================
| original | skip free page
--------------------------------------------------
total time(ms) | 1505 | 573
--------------------------------------------------
transferred ram(KB) | 399792 | 379057
==================================================
Case 2: The guest has ever run some memory consuming workload, the
workload is terminated before live migration.
==================================================
| original | skip free page
--------------------------------------------------
total time(ms) | 10641 | 597
--------------------------------------------------
transferred ram(KB) | 8350829 | 389900
==================================================
Case 3: The guest has ever built the linux kernel, the building is
completed before live migration. The page cache is dropped
during live migration in this case.
==================================================
| original | skip free page
--------------------------------------------------
total time(ms) | 2636 | 914
--------------------------------------------------
transferred ram(KB) | 1316747 | 421980
==================================================
Liang Li (5):
bitmap: Add a new bitmap_move function
kvm: Add two new arch specific functions
virtio-balloon: Add a new feature to balloon device
migration: filter out free pages during live migration
migration: Add the interface for cache drop control
balloon.c | 29 +++++++-
hmp.c | 8 +++
hw/virtio/virtio-balloon.c | 92 +++++++++++++++++++++++-
include/hw/virtio/virtio-balloon.h | 30 +++++++-
include/migration/migration.h | 1 +
include/qemu/bitmap.h | 13 ++++
include/standard-headers/linux/virtio_balloon.h | 1 +
include/sysemu/balloon.h | 12 +++-
include/sysemu/kvm.h | 2 +
migration/migration.c | 31 +++++++-
migration/ram.c | 95 +++++++++++++++++++++++++
qapi-schema.json | 25 ++++++-
target-arm/kvm.c | 14 ++++
target-i386/kvm.c | 35 +++++++++
target-mips/kvm.c | 14 ++++
target-ppc/kvm.c | 14 ++++
target-s390x/kvm.c | 14 ++++
17 files changed, 421 insertions(+), 9 deletions(-)
--
1.8.3.1
^ permalink raw reply [flat|nested] 19+ messages in thread* [PATCH QEMU 1/5] bitmap: Add a new bitmap_move function
2016-04-19 14:20 ` [Qemu-devel] " Liang Li
@ 2016-04-19 14:20 ` Liang Li
-1 siblings, 0 replies; 19+ messages in thread
From: Liang Li @ 2016-04-19 14:20 UTC (permalink / raw)
To: mst, quintela, amit.shah, pbonzini, lcapitulino
Cc: armbru, peter.maydell, rth, ehabkost, james.hogan, aurelien,
leon.alrae, agraf, borntraeger, cornelia.huck, qemu-devel, kvm,
Liang Li
Sometimes, it is need to move a portion of bitmap to another place
in a large bitmap, if overlap happens, the bitmap_copy can't not
work correctly, we need a new function to do this work.
Signed-off-by: Liang Li <liang.z.li@intel.com>
---
include/qemu/bitmap.h | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/include/qemu/bitmap.h b/include/qemu/bitmap.h
index 0e33fa5..ce07444 100644
--- a/include/qemu/bitmap.h
+++ b/include/qemu/bitmap.h
@@ -38,6 +38,7 @@
* bitmap_set(dst, pos, nbits) Set specified bit area
* bitmap_set_atomic(dst, pos, nbits) Set specified bit area with atomic ops
* bitmap_clear(dst, pos, nbits) Clear specified bit area
+ * bitmap_move(dst, src, nbits) Move *src to *dst
* bitmap_test_and_clear_atomic(dst, pos, nbits) Test and clear area
* bitmap_find_next_zero_area(buf, len, pos, n, mask) Find bit free area
*/
@@ -137,6 +138,18 @@ static inline void bitmap_copy(unsigned long *dst, const unsigned long *src,
}
}
+static inline void bitmap_move(unsigned long *dst, const unsigned long *src,
+ long nbits)
+{
+ if (small_nbits(nbits)) {
+ unsigned long tmp = *src;
+ *dst = tmp;
+ } else {
+ long len = BITS_TO_LONGS(nbits) * sizeof(unsigned long);
+ memmove(dst, src, len);
+ }
+}
+
static inline int bitmap_and(unsigned long *dst, const unsigned long *src1,
const unsigned long *src2, long nbits)
{
--
1.8.3.1
^ permalink raw reply related [flat|nested] 19+ messages in thread* [Qemu-devel] [PATCH QEMU 1/5] bitmap: Add a new bitmap_move function
@ 2016-04-19 14:20 ` Liang Li
0 siblings, 0 replies; 19+ messages in thread
From: Liang Li @ 2016-04-19 14:20 UTC (permalink / raw)
To: mst, quintela, amit.shah, pbonzini, lcapitulino
Cc: armbru, peter.maydell, rth, ehabkost, james.hogan, aurelien,
leon.alrae, agraf, borntraeger, cornelia.huck, qemu-devel, kvm,
Liang Li
Sometimes, it is need to move a portion of bitmap to another place
in a large bitmap, if overlap happens, the bitmap_copy can't not
work correctly, we need a new function to do this work.
Signed-off-by: Liang Li <liang.z.li@intel.com>
---
include/qemu/bitmap.h | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/include/qemu/bitmap.h b/include/qemu/bitmap.h
index 0e33fa5..ce07444 100644
--- a/include/qemu/bitmap.h
+++ b/include/qemu/bitmap.h
@@ -38,6 +38,7 @@
* bitmap_set(dst, pos, nbits) Set specified bit area
* bitmap_set_atomic(dst, pos, nbits) Set specified bit area with atomic ops
* bitmap_clear(dst, pos, nbits) Clear specified bit area
+ * bitmap_move(dst, src, nbits) Move *src to *dst
* bitmap_test_and_clear_atomic(dst, pos, nbits) Test and clear area
* bitmap_find_next_zero_area(buf, len, pos, n, mask) Find bit free area
*/
@@ -137,6 +138,18 @@ static inline void bitmap_copy(unsigned long *dst, const unsigned long *src,
}
}
+static inline void bitmap_move(unsigned long *dst, const unsigned long *src,
+ long nbits)
+{
+ if (small_nbits(nbits)) {
+ unsigned long tmp = *src;
+ *dst = tmp;
+ } else {
+ long len = BITS_TO_LONGS(nbits) * sizeof(unsigned long);
+ memmove(dst, src, len);
+ }
+}
+
static inline int bitmap_and(unsigned long *dst, const unsigned long *src1,
const unsigned long *src2, long nbits)
{
--
1.8.3.1
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH QEMU 2/5] kvm: Add two new arch specific functions
2016-04-19 14:20 ` [Qemu-devel] " Liang Li
@ 2016-04-19 14:20 ` Liang Li
-1 siblings, 0 replies; 19+ messages in thread
From: Liang Li @ 2016-04-19 14:20 UTC (permalink / raw)
To: mst, quintela, amit.shah, pbonzini, lcapitulino
Cc: armbru, peter.maydell, rth, ehabkost, james.hogan, aurelien,
leon.alrae, agraf, borntraeger, cornelia.huck, qemu-devel, kvm,
Liang Li
Add a new function to get the vm's max pfn and a new function
to filter out the holes to get a tight free page bitmap.
They are implemented on X86, and all the arches should implement
them for live migration optimization.
Signed-off-by: Liang Li <liang.z.li@intel.com>
---
include/sysemu/kvm.h | 2 ++
target-arm/kvm.c | 14 ++++++++++++++
target-i386/kvm.c | 35 +++++++++++++++++++++++++++++++++++
target-mips/kvm.c | 14 ++++++++++++++
target-ppc/kvm.c | 14 ++++++++++++++
target-s390x/kvm.c | 14 ++++++++++++++
6 files changed, 93 insertions(+)
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index 0e18f15..5263304 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -228,6 +228,8 @@ int kvm_remove_breakpoint(CPUState *cpu, target_ulong addr,
target_ulong len, int type);
void kvm_remove_all_breakpoints(CPUState *cpu);
int kvm_update_guest_debug(CPUState *cpu, unsigned long reinject_trap);
+unsigned long *tighten_guest_free_page_bmap(unsigned long *bmap);
+unsigned long get_guest_max_pfn(void);
#ifndef _WIN32
int kvm_set_signal_mask(CPUState *cpu, const sigset_t *sigset);
#endif
diff --git a/target-arm/kvm.c b/target-arm/kvm.c
index 3671032..59e9417 100644
--- a/target-arm/kvm.c
+++ b/target-arm/kvm.c
@@ -626,3 +626,17 @@ int kvm_arch_msi_data_to_gsi(uint32_t data)
{
return (data - 32) & 0xffff;
}
+
+unsigned long get_guest_max_pfn(void)
+{
+ /* To be done */
+
+ return 0;
+}
+
+unsigned long *tighten_guest_free_page_bmap(unsigned long *bmap)
+{
+ /* To be done */
+
+ return bmap;
+}
diff --git a/target-i386/kvm.c b/target-i386/kvm.c
index 799fdfa..76a33bd 100644
--- a/target-i386/kvm.c
+++ b/target-i386/kvm.c
@@ -3334,3 +3334,38 @@ int kvm_arch_msi_data_to_gsi(uint32_t data)
{
abort();
}
+
+unsigned long get_guest_max_pfn(void)
+{
+ PCMachineState *pcms = PC_MACHINE(current_machine);
+ ram_addr_t above_4g_mem = pcms->above_4g_mem_size;
+ unsigned long max_pfn;
+
+ if (above_4g_mem) {
+ max_pfn = ((1ULL << 32) + above_4g_mem) >> TARGET_PAGE_BITS;
+ } else {
+ max_pfn = pcms->below_4g_mem_size >> TARGET_PAGE_BITS;
+ }
+
+ return max_pfn;
+}
+
+unsigned long *tighten_guest_free_page_bmap(unsigned long *bmap)
+{
+ PCMachineState *pcms = PC_MACHINE(current_machine);
+ ram_addr_t above_4g_mem = pcms->above_4g_mem_size;
+
+ if (above_4g_mem) {
+ unsigned long *src, *dst, len, pos;
+ ram_addr_t below_4g_mem = pcms->below_4g_mem_size;
+ src = bmap + ((1ULL << 32) >> TARGET_PAGE_BITS) / BITS_PER_LONG;
+ dst = bmap + (below_4g_mem >> TARGET_PAGE_BITS) / BITS_PER_LONG;
+ bitmap_move(dst, src, above_4g_mem >> TARGET_PAGE_BITS);
+
+ pos = (above_4g_mem + below_4g_mem) >> TARGET_PAGE_BITS;
+ len = ((1ULL << 32) - below_4g_mem) >> TARGET_PAGE_BITS;
+ bitmap_clear(bmap, pos, len);
+ }
+
+ return bmap;
+}
diff --git a/target-mips/kvm.c b/target-mips/kvm.c
index 950bc05..23fdc50 100644
--- a/target-mips/kvm.c
+++ b/target-mips/kvm.c
@@ -1048,3 +1048,17 @@ int kvm_arch_msi_data_to_gsi(uint32_t data)
{
abort();
}
+
+unsigned long get_guest_max_pfn(void)
+{
+ /* To be done */
+
+ return 0;
+}
+
+unsigned long *tighten_guest_free_page_bmap(unsigned long *bmap)
+{
+ /* To be done */
+
+ return bmap;
+}
diff --git a/target-ppc/kvm.c b/target-ppc/kvm.c
index c4c8146..bf4c60f 100644
--- a/target-ppc/kvm.c
+++ b/target-ppc/kvm.c
@@ -2579,3 +2579,17 @@ int kvmppc_enable_hwrng(void)
return kvmppc_enable_hcall(kvm_state, H_RANDOM);
}
+
+unsigned long get_guest_max_pfn(void)
+{
+ /* To be done */
+
+ return 0;
+}
+
+unsigned long *tighten_guest_free_page_bmap(unsigned long *bmap)
+{
+ /* To be done */
+
+ return bmap;
+}
diff --git a/target-s390x/kvm.c b/target-s390x/kvm.c
index e1859ca..7f5e1b8 100644
--- a/target-s390x/kvm.c
+++ b/target-s390x/kvm.c
@@ -2250,3 +2250,17 @@ int kvm_arch_msi_data_to_gsi(uint32_t data)
{
abort();
}
+
+unsigned long get_guest_max_pfn(void)
+{
+ /* To be done */
+
+ return 0;
+}
+
+unsigned long *tighten_guest_free_page_bmap(unsigned long *bmap)
+{
+ /* To be done */
+
+ return bmap;
+}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 19+ messages in thread* [Qemu-devel] [PATCH QEMU 2/5] kvm: Add two new arch specific functions
@ 2016-04-19 14:20 ` Liang Li
0 siblings, 0 replies; 19+ messages in thread
From: Liang Li @ 2016-04-19 14:20 UTC (permalink / raw)
To: mst, quintela, amit.shah, pbonzini, lcapitulino
Cc: armbru, peter.maydell, rth, ehabkost, james.hogan, aurelien,
leon.alrae, agraf, borntraeger, cornelia.huck, qemu-devel, kvm,
Liang Li
Add a new function to get the vm's max pfn and a new function
to filter out the holes to get a tight free page bitmap.
They are implemented on X86, and all the arches should implement
them for live migration optimization.
Signed-off-by: Liang Li <liang.z.li@intel.com>
---
include/sysemu/kvm.h | 2 ++
target-arm/kvm.c | 14 ++++++++++++++
target-i386/kvm.c | 35 +++++++++++++++++++++++++++++++++++
target-mips/kvm.c | 14 ++++++++++++++
target-ppc/kvm.c | 14 ++++++++++++++
target-s390x/kvm.c | 14 ++++++++++++++
6 files changed, 93 insertions(+)
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index 0e18f15..5263304 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -228,6 +228,8 @@ int kvm_remove_breakpoint(CPUState *cpu, target_ulong addr,
target_ulong len, int type);
void kvm_remove_all_breakpoints(CPUState *cpu);
int kvm_update_guest_debug(CPUState *cpu, unsigned long reinject_trap);
+unsigned long *tighten_guest_free_page_bmap(unsigned long *bmap);
+unsigned long get_guest_max_pfn(void);
#ifndef _WIN32
int kvm_set_signal_mask(CPUState *cpu, const sigset_t *sigset);
#endif
diff --git a/target-arm/kvm.c b/target-arm/kvm.c
index 3671032..59e9417 100644
--- a/target-arm/kvm.c
+++ b/target-arm/kvm.c
@@ -626,3 +626,17 @@ int kvm_arch_msi_data_to_gsi(uint32_t data)
{
return (data - 32) & 0xffff;
}
+
+unsigned long get_guest_max_pfn(void)
+{
+ /* To be done */
+
+ return 0;
+}
+
+unsigned long *tighten_guest_free_page_bmap(unsigned long *bmap)
+{
+ /* To be done */
+
+ return bmap;
+}
diff --git a/target-i386/kvm.c b/target-i386/kvm.c
index 799fdfa..76a33bd 100644
--- a/target-i386/kvm.c
+++ b/target-i386/kvm.c
@@ -3334,3 +3334,38 @@ int kvm_arch_msi_data_to_gsi(uint32_t data)
{
abort();
}
+
+unsigned long get_guest_max_pfn(void)
+{
+ PCMachineState *pcms = PC_MACHINE(current_machine);
+ ram_addr_t above_4g_mem = pcms->above_4g_mem_size;
+ unsigned long max_pfn;
+
+ if (above_4g_mem) {
+ max_pfn = ((1ULL << 32) + above_4g_mem) >> TARGET_PAGE_BITS;
+ } else {
+ max_pfn = pcms->below_4g_mem_size >> TARGET_PAGE_BITS;
+ }
+
+ return max_pfn;
+}
+
+unsigned long *tighten_guest_free_page_bmap(unsigned long *bmap)
+{
+ PCMachineState *pcms = PC_MACHINE(current_machine);
+ ram_addr_t above_4g_mem = pcms->above_4g_mem_size;
+
+ if (above_4g_mem) {
+ unsigned long *src, *dst, len, pos;
+ ram_addr_t below_4g_mem = pcms->below_4g_mem_size;
+ src = bmap + ((1ULL << 32) >> TARGET_PAGE_BITS) / BITS_PER_LONG;
+ dst = bmap + (below_4g_mem >> TARGET_PAGE_BITS) / BITS_PER_LONG;
+ bitmap_move(dst, src, above_4g_mem >> TARGET_PAGE_BITS);
+
+ pos = (above_4g_mem + below_4g_mem) >> TARGET_PAGE_BITS;
+ len = ((1ULL << 32) - below_4g_mem) >> TARGET_PAGE_BITS;
+ bitmap_clear(bmap, pos, len);
+ }
+
+ return bmap;
+}
diff --git a/target-mips/kvm.c b/target-mips/kvm.c
index 950bc05..23fdc50 100644
--- a/target-mips/kvm.c
+++ b/target-mips/kvm.c
@@ -1048,3 +1048,17 @@ int kvm_arch_msi_data_to_gsi(uint32_t data)
{
abort();
}
+
+unsigned long get_guest_max_pfn(void)
+{
+ /* To be done */
+
+ return 0;
+}
+
+unsigned long *tighten_guest_free_page_bmap(unsigned long *bmap)
+{
+ /* To be done */
+
+ return bmap;
+}
diff --git a/target-ppc/kvm.c b/target-ppc/kvm.c
index c4c8146..bf4c60f 100644
--- a/target-ppc/kvm.c
+++ b/target-ppc/kvm.c
@@ -2579,3 +2579,17 @@ int kvmppc_enable_hwrng(void)
return kvmppc_enable_hcall(kvm_state, H_RANDOM);
}
+
+unsigned long get_guest_max_pfn(void)
+{
+ /* To be done */
+
+ return 0;
+}
+
+unsigned long *tighten_guest_free_page_bmap(unsigned long *bmap)
+{
+ /* To be done */
+
+ return bmap;
+}
diff --git a/target-s390x/kvm.c b/target-s390x/kvm.c
index e1859ca..7f5e1b8 100644
--- a/target-s390x/kvm.c
+++ b/target-s390x/kvm.c
@@ -2250,3 +2250,17 @@ int kvm_arch_msi_data_to_gsi(uint32_t data)
{
abort();
}
+
+unsigned long get_guest_max_pfn(void)
+{
+ /* To be done */
+
+ return 0;
+}
+
+unsigned long *tighten_guest_free_page_bmap(unsigned long *bmap)
+{
+ /* To be done */
+
+ return bmap;
+}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH QEMU 3/5] virtio-balloon: Add a new feature to balloon device
2016-04-19 14:20 ` [Qemu-devel] " Liang Li
@ 2016-04-19 14:20 ` Liang Li
-1 siblings, 0 replies; 19+ messages in thread
From: Liang Li @ 2016-04-19 14:20 UTC (permalink / raw)
To: mst, quintela, amit.shah, pbonzini, lcapitulino
Cc: armbru, peter.maydell, rth, ehabkost, james.hogan, aurelien,
leon.alrae, agraf, borntraeger, cornelia.huck, qemu-devel, kvm,
Liang Li
Extend the virtio balloon device to support a new feature, this
new feature can help to get guest's free pages information, which
can be used for live migration optimzation.
Signed-off-by: Liang Li <liang.z.li@intel.com>
---
balloon.c | 29 +++++++-
hw/virtio/virtio-balloon.c | 92 ++++++++++++++++++++++++-
include/hw/virtio/virtio-balloon.h | 30 +++++++-
include/standard-headers/linux/virtio_balloon.h | 1 +
include/sysemu/balloon.h | 12 +++-
5 files changed, 159 insertions(+), 5 deletions(-)
diff --git a/balloon.c b/balloon.c
index f2ef50c..346e215 100644
--- a/balloon.c
+++ b/balloon.c
@@ -36,6 +36,7 @@
static QEMUBalloonEvent *balloon_event_fn;
static QEMUBalloonStatus *balloon_stat_fn;
+static QEMUBalloonGetFreePage *balloon_get_free_page_fn;
static void *balloon_opaque;
static bool balloon_inhibited;
@@ -65,9 +66,12 @@ static bool have_balloon(Error **errp)
}
int qemu_add_balloon_handler(QEMUBalloonEvent *event_func,
- QEMUBalloonStatus *stat_func, void *opaque)
+ QEMUBalloonStatus *stat_func,
+ QEMUBalloonGetFreePage *get_free_page_func,
+ void *opaque)
{
- if (balloon_event_fn || balloon_stat_fn || balloon_opaque) {
+ if (balloon_event_fn || balloon_stat_fn || balloon_get_free_page_fn
+ || balloon_opaque) {
/* We're already registered one balloon handler. How many can
* a guest really have?
*/
@@ -75,6 +79,7 @@ int qemu_add_balloon_handler(QEMUBalloonEvent *event_func,
}
balloon_event_fn = event_func;
balloon_stat_fn = stat_func;
+ balloon_get_free_page_fn = get_free_page_func;
balloon_opaque = opaque;
return 0;
}
@@ -86,6 +91,7 @@ void qemu_remove_balloon_handler(void *opaque)
}
balloon_event_fn = NULL;
balloon_stat_fn = NULL;
+ balloon_get_free_page_fn = NULL;
balloon_opaque = NULL;
}
@@ -116,3 +122,22 @@ void qmp_balloon(int64_t target, Error **errp)
trace_balloon_event(balloon_opaque, target);
balloon_event_fn(balloon_opaque, target);
}
+
+bool balloon_free_pages_support(void)
+{
+ return balloon_get_free_page_fn ? true : false;
+}
+
+FreePageStatus balloon_get_free_pages(unsigned long *bitmap,
+ unsigned long len, int drop_cache)
+{
+ if (!balloon_get_free_page_fn) {
+ return FREE_PAGE_UNSUPPORT;
+ }
+
+ if (!bitmap) {
+ return FREE_PAGE_INVALID_PARAM;
+ }
+
+ return balloon_get_free_page_fn(balloon_opaque, bitmap, len, drop_cache);
+}
diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c
index 9dbe681..0abf375 100644
--- a/hw/virtio/virtio-balloon.c
+++ b/hw/virtio/virtio-balloon.c
@@ -78,6 +78,12 @@ static bool balloon_stats_supported(const VirtIOBalloon *s)
return virtio_vdev_has_feature(vdev, VIRTIO_BALLOON_F_STATS_VQ);
}
+static bool balloon_free_pages_supported(const VirtIOBalloon *s)
+{
+ VirtIODevice *vdev = VIRTIO_DEVICE(s);
+ return virtio_vdev_has_feature(vdev, VIRTIO_BALLOON_F_GET_FREE_PAGE);
+}
+
static bool balloon_stats_enabled(const VirtIOBalloon *s)
{
return s->stats_poll_interval > 0;
@@ -304,6 +310,38 @@ out:
}
}
+static void virtio_balloon_get_free_pages(VirtIODevice *vdev, VirtQueue *vq)
+{
+ VirtIOBalloon *s = VIRTIO_BALLOON(vdev);
+ VirtQueueElement *elem;
+ size_t offset = 0;
+ uint64_t bitmap_bytes = 0;
+
+ elem = virtqueue_pop(vq, sizeof(VirtQueueElement));
+ if (!elem) {
+ s->req_status = REQ_ERROR;
+ return;
+ }
+
+ s->free_page_vq_elem = elem;
+
+ if (!elem->out_num) {
+ return;
+ }
+
+ iov_to_buf(elem->out_sg, elem->out_num, offset,
+ &bitmap_bytes, sizeof(uint64_t));
+
+ if (s->bmap_len < bitmap_bytes) {
+ s->req_status = REQ_INVALID_PARAM;
+ return;
+ }
+ offset += sizeof(uint64_t);
+ iov_to_buf(elem->out_sg, elem->out_num, offset,
+ s->free_page_bmap, bitmap_bytes);
+ s->req_status = REQ_DONE;
+}
+
static void virtio_balloon_get_config(VirtIODevice *vdev, uint8_t *config_data)
{
VirtIOBalloon *dev = VIRTIO_BALLOON(vdev);
@@ -373,6 +411,7 @@ static uint64_t virtio_balloon_get_features(VirtIODevice *vdev, uint64_t f,
VirtIOBalloon *dev = VIRTIO_BALLOON(vdev);
f |= dev->host_features;
virtio_add_feature(&f, VIRTIO_BALLOON_F_STATS_VQ);
+ virtio_add_feature(&f, VIRTIO_BALLOON_F_GET_FREE_PAGE);
return f;
}
@@ -383,6 +422,48 @@ static void virtio_balloon_stat(void *opaque, BalloonInfo *info)
VIRTIO_BALLOON_PFN_SHIFT);
}
+static FreePageStatus virtio_balloon_free_pages(void *opaque,
+ unsigned long *bitmap,
+ unsigned long bmap_len,
+ int drop_cache_ctl)
+{
+ VirtIOBalloon *s = opaque;
+ VirtIODevice *vdev = VIRTIO_DEVICE(s);
+ VirtQueueElement *elem = s->free_page_vq_elem;
+ int len;
+
+ if (!balloon_free_pages_supported(s)) {
+ return FREE_PAGE_UNSUPPORT;
+ }
+
+ if (s->req_status == REQ_INIT) {
+ s->free_page_bmap = bitmap;
+ if (elem == NULL || !elem->in_num) {
+ elem = virtqueue_pop(s->fvq, sizeof(VirtQueueElement));
+ if (!elem) {
+ return FREE_PAGE_ERROR;
+ }
+ s->free_page_vq_elem = elem;
+ }
+ s->free_page_req.param = drop_cache_ctl;
+ s->bmap_len = bmap_len;
+ len = iov_from_buf(elem->in_sg, elem->in_num, 0, &s->free_page_req,
+ sizeof(s->free_page_req));
+ virtqueue_push(s->fvq, elem, len);
+ virtio_notify(vdev, s->fvq);
+ g_free(s->free_page_vq_elem);
+ s->free_page_vq_elem = NULL;
+ s->req_status = REQ_ON_GOING;
+ return FREE_PAGE_REQ;
+ } else if (s->req_status == REQ_ON_GOING) {
+ return FREE_PAGE_WAITING_FOR_RSP;
+ } else if (s->req_status == REQ_DONE) {
+ s->req_status = REQ_INIT;
+ }
+
+ return FREE_PAGE_READY;
+}
+
static void virtio_balloon_to_target(void *opaque, ram_addr_t target)
{
VirtIOBalloon *dev = VIRTIO_BALLOON(opaque);
@@ -444,7 +525,8 @@ static void virtio_balloon_device_realize(DeviceState *dev, Error **errp)
sizeof(struct virtio_balloon_config));
ret = qemu_add_balloon_handler(virtio_balloon_to_target,
- virtio_balloon_stat, s);
+ virtio_balloon_stat,
+ virtio_balloon_free_pages, s);
if (ret < 0) {
error_setg(errp, "Only one balloon device is supported");
@@ -455,8 +537,10 @@ static void virtio_balloon_device_realize(DeviceState *dev, Error **errp)
s->ivq = virtio_add_queue(vdev, 128, virtio_balloon_handle_output);
s->dvq = virtio_add_queue(vdev, 128, virtio_balloon_handle_output);
s->svq = virtio_add_queue(vdev, 128, virtio_balloon_receive_stats);
+ s->fvq = virtio_add_queue(vdev, 128, virtio_balloon_get_free_pages);
reset_stats(s);
+ s->req_status = REQ_INIT;
register_savevm(dev, "virtio-balloon", -1, 1,
virtio_balloon_save, virtio_balloon_load, s);
@@ -481,6 +565,12 @@ static void virtio_balloon_device_reset(VirtIODevice *vdev)
g_free(s->stats_vq_elem);
s->stats_vq_elem = NULL;
}
+
+ if (s->free_page_vq_elem != NULL) {
+ g_free(s->free_page_vq_elem);
+ s->free_page_vq_elem = NULL;
+ }
+ s->req_status = REQ_INIT;
}
static void virtio_balloon_instance_init(Object *obj)
diff --git a/include/hw/virtio/virtio-balloon.h b/include/hw/virtio/virtio-balloon.h
index 35f62ac..fbed81c 100644
--- a/include/hw/virtio/virtio-balloon.h
+++ b/include/hw/virtio/virtio-balloon.h
@@ -23,6 +23,27 @@
#define VIRTIO_BALLOON(obj) \
OBJECT_CHECK(VirtIOBalloon, (obj), TYPE_VIRTIO_BALLOON)
+typedef enum {
+ REQ_INIT,
+ REQ_ON_GOING,
+ REQ_DONE,
+ REQ_ERROR,
+ REQ_INVALID_PARAM,
+} VirtIoReqStatus;
+
+typedef enum {
+ FREE_PAGE_REQ,
+ FREE_PAGE_WAITING_FOR_RSP,
+ FREE_PAGE_READY,
+ FREE_PAGE_INVALID_PARAM,
+ FREE_PAGE_UNSUPPORT,
+ FREE_PAGE_ERROR,
+} FreePageStatus;
+
+typedef struct GetFreePageReq {
+ uint64_t param;
+} GetFreePageReq;
+
typedef struct virtio_balloon_stat VirtIOBalloonStat;
typedef struct virtio_balloon_stat_modern {
@@ -33,16 +54,23 @@ typedef struct virtio_balloon_stat_modern {
typedef struct VirtIOBalloon {
VirtIODevice parent_obj;
- VirtQueue *ivq, *dvq, *svq;
+ VirtQueue *ivq, *dvq, *svq, *fvq;
uint32_t num_pages;
uint32_t actual;
uint64_t stats[VIRTIO_BALLOON_S_NR];
VirtQueueElement *stats_vq_elem;
+ VirtQueueElement *free_page_vq_elem;
size_t stats_vq_offset;
QEMUTimer *stats_timer;
int64_t stats_last_update;
int64_t stats_poll_interval;
uint32_t host_features;
+ uint64_t *free_page_bmap;
+ uint64_t bmap_len;
+ uint64_t free_pages;
+ uint64_t cache_pages;
+ GetFreePageReq free_page_req;
+ VirtIoReqStatus req_status;
} VirtIOBalloon;
#endif
diff --git a/include/standard-headers/linux/virtio_balloon.h b/include/standard-headers/linux/virtio_balloon.h
index 9d06ccd..09211b0 100644
--- a/include/standard-headers/linux/virtio_balloon.h
+++ b/include/standard-headers/linux/virtio_balloon.h
@@ -34,6 +34,7 @@
#define VIRTIO_BALLOON_F_MUST_TELL_HOST 0 /* Tell before reclaiming pages */
#define VIRTIO_BALLOON_F_STATS_VQ 1 /* Memory Stats virtqueue */
#define VIRTIO_BALLOON_F_DEFLATE_ON_OOM 2 /* Deflate balloon on OOM */
+#define VIRTIO_BALLOON_F_GET_FREE_PAGE 3 /* Get the free page bitmap */
/* Size of a PFN in the balloon interface. */
#define VIRTIO_BALLOON_PFN_SHIFT 12
diff --git a/include/sysemu/balloon.h b/include/sysemu/balloon.h
index 3f976b4..81a1b9f 100644
--- a/include/sysemu/balloon.h
+++ b/include/sysemu/balloon.h
@@ -15,14 +15,24 @@
#define _QEMU_BALLOON_H
#include "qapi-types.h"
+#include "hw/virtio/virtio-balloon.h"
typedef void (QEMUBalloonEvent)(void *opaque, ram_addr_t target);
typedef void (QEMUBalloonStatus)(void *opaque, BalloonInfo *info);
+typedef FreePageStatus (QEMUBalloonGetFreePage)(void *opaque,
+ unsigned long *bitmap,
+ unsigned long len,
+ int drop_cache_ctl);
int qemu_add_balloon_handler(QEMUBalloonEvent *event_func,
- QEMUBalloonStatus *stat_func, void *opaque);
+ QEMUBalloonStatus *stat_func,
+ QEMUBalloonGetFreePage *get_free_page_func,
+ void *opaque);
void qemu_remove_balloon_handler(void *opaque);
bool qemu_balloon_is_inhibited(void);
void qemu_balloon_inhibit(bool state);
+bool balloon_free_pages_support(void);
+FreePageStatus balloon_get_free_pages(unsigned long *bitmap,
+ unsigned long len, int drop_cache_ctl);
#endif
--
1.8.3.1
^ permalink raw reply related [flat|nested] 19+ messages in thread* [Qemu-devel] [PATCH QEMU 3/5] virtio-balloon: Add a new feature to balloon device
@ 2016-04-19 14:20 ` Liang Li
0 siblings, 0 replies; 19+ messages in thread
From: Liang Li @ 2016-04-19 14:20 UTC (permalink / raw)
To: mst, quintela, amit.shah, pbonzini, lcapitulino
Cc: armbru, peter.maydell, rth, ehabkost, james.hogan, aurelien,
leon.alrae, agraf, borntraeger, cornelia.huck, qemu-devel, kvm,
Liang Li
Extend the virtio balloon device to support a new feature, this
new feature can help to get guest's free pages information, which
can be used for live migration optimzation.
Signed-off-by: Liang Li <liang.z.li@intel.com>
---
balloon.c | 29 +++++++-
hw/virtio/virtio-balloon.c | 92 ++++++++++++++++++++++++-
include/hw/virtio/virtio-balloon.h | 30 +++++++-
include/standard-headers/linux/virtio_balloon.h | 1 +
include/sysemu/balloon.h | 12 +++-
5 files changed, 159 insertions(+), 5 deletions(-)
diff --git a/balloon.c b/balloon.c
index f2ef50c..346e215 100644
--- a/balloon.c
+++ b/balloon.c
@@ -36,6 +36,7 @@
static QEMUBalloonEvent *balloon_event_fn;
static QEMUBalloonStatus *balloon_stat_fn;
+static QEMUBalloonGetFreePage *balloon_get_free_page_fn;
static void *balloon_opaque;
static bool balloon_inhibited;
@@ -65,9 +66,12 @@ static bool have_balloon(Error **errp)
}
int qemu_add_balloon_handler(QEMUBalloonEvent *event_func,
- QEMUBalloonStatus *stat_func, void *opaque)
+ QEMUBalloonStatus *stat_func,
+ QEMUBalloonGetFreePage *get_free_page_func,
+ void *opaque)
{
- if (balloon_event_fn || balloon_stat_fn || balloon_opaque) {
+ if (balloon_event_fn || balloon_stat_fn || balloon_get_free_page_fn
+ || balloon_opaque) {
/* We're already registered one balloon handler. How many can
* a guest really have?
*/
@@ -75,6 +79,7 @@ int qemu_add_balloon_handler(QEMUBalloonEvent *event_func,
}
balloon_event_fn = event_func;
balloon_stat_fn = stat_func;
+ balloon_get_free_page_fn = get_free_page_func;
balloon_opaque = opaque;
return 0;
}
@@ -86,6 +91,7 @@ void qemu_remove_balloon_handler(void *opaque)
}
balloon_event_fn = NULL;
balloon_stat_fn = NULL;
+ balloon_get_free_page_fn = NULL;
balloon_opaque = NULL;
}
@@ -116,3 +122,22 @@ void qmp_balloon(int64_t target, Error **errp)
trace_balloon_event(balloon_opaque, target);
balloon_event_fn(balloon_opaque, target);
}
+
+bool balloon_free_pages_support(void)
+{
+ return balloon_get_free_page_fn ? true : false;
+}
+
+FreePageStatus balloon_get_free_pages(unsigned long *bitmap,
+ unsigned long len, int drop_cache)
+{
+ if (!balloon_get_free_page_fn) {
+ return FREE_PAGE_UNSUPPORT;
+ }
+
+ if (!bitmap) {
+ return FREE_PAGE_INVALID_PARAM;
+ }
+
+ return balloon_get_free_page_fn(balloon_opaque, bitmap, len, drop_cache);
+}
diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c
index 9dbe681..0abf375 100644
--- a/hw/virtio/virtio-balloon.c
+++ b/hw/virtio/virtio-balloon.c
@@ -78,6 +78,12 @@ static bool balloon_stats_supported(const VirtIOBalloon *s)
return virtio_vdev_has_feature(vdev, VIRTIO_BALLOON_F_STATS_VQ);
}
+static bool balloon_free_pages_supported(const VirtIOBalloon *s)
+{
+ VirtIODevice *vdev = VIRTIO_DEVICE(s);
+ return virtio_vdev_has_feature(vdev, VIRTIO_BALLOON_F_GET_FREE_PAGE);
+}
+
static bool balloon_stats_enabled(const VirtIOBalloon *s)
{
return s->stats_poll_interval > 0;
@@ -304,6 +310,38 @@ out:
}
}
+static void virtio_balloon_get_free_pages(VirtIODevice *vdev, VirtQueue *vq)
+{
+ VirtIOBalloon *s = VIRTIO_BALLOON(vdev);
+ VirtQueueElement *elem;
+ size_t offset = 0;
+ uint64_t bitmap_bytes = 0;
+
+ elem = virtqueue_pop(vq, sizeof(VirtQueueElement));
+ if (!elem) {
+ s->req_status = REQ_ERROR;
+ return;
+ }
+
+ s->free_page_vq_elem = elem;
+
+ if (!elem->out_num) {
+ return;
+ }
+
+ iov_to_buf(elem->out_sg, elem->out_num, offset,
+ &bitmap_bytes, sizeof(uint64_t));
+
+ if (s->bmap_len < bitmap_bytes) {
+ s->req_status = REQ_INVALID_PARAM;
+ return;
+ }
+ offset += sizeof(uint64_t);
+ iov_to_buf(elem->out_sg, elem->out_num, offset,
+ s->free_page_bmap, bitmap_bytes);
+ s->req_status = REQ_DONE;
+}
+
static void virtio_balloon_get_config(VirtIODevice *vdev, uint8_t *config_data)
{
VirtIOBalloon *dev = VIRTIO_BALLOON(vdev);
@@ -373,6 +411,7 @@ static uint64_t virtio_balloon_get_features(VirtIODevice *vdev, uint64_t f,
VirtIOBalloon *dev = VIRTIO_BALLOON(vdev);
f |= dev->host_features;
virtio_add_feature(&f, VIRTIO_BALLOON_F_STATS_VQ);
+ virtio_add_feature(&f, VIRTIO_BALLOON_F_GET_FREE_PAGE);
return f;
}
@@ -383,6 +422,48 @@ static void virtio_balloon_stat(void *opaque, BalloonInfo *info)
VIRTIO_BALLOON_PFN_SHIFT);
}
+static FreePageStatus virtio_balloon_free_pages(void *opaque,
+ unsigned long *bitmap,
+ unsigned long bmap_len,
+ int drop_cache_ctl)
+{
+ VirtIOBalloon *s = opaque;
+ VirtIODevice *vdev = VIRTIO_DEVICE(s);
+ VirtQueueElement *elem = s->free_page_vq_elem;
+ int len;
+
+ if (!balloon_free_pages_supported(s)) {
+ return FREE_PAGE_UNSUPPORT;
+ }
+
+ if (s->req_status == REQ_INIT) {
+ s->free_page_bmap = bitmap;
+ if (elem == NULL || !elem->in_num) {
+ elem = virtqueue_pop(s->fvq, sizeof(VirtQueueElement));
+ if (!elem) {
+ return FREE_PAGE_ERROR;
+ }
+ s->free_page_vq_elem = elem;
+ }
+ s->free_page_req.param = drop_cache_ctl;
+ s->bmap_len = bmap_len;
+ len = iov_from_buf(elem->in_sg, elem->in_num, 0, &s->free_page_req,
+ sizeof(s->free_page_req));
+ virtqueue_push(s->fvq, elem, len);
+ virtio_notify(vdev, s->fvq);
+ g_free(s->free_page_vq_elem);
+ s->free_page_vq_elem = NULL;
+ s->req_status = REQ_ON_GOING;
+ return FREE_PAGE_REQ;
+ } else if (s->req_status == REQ_ON_GOING) {
+ return FREE_PAGE_WAITING_FOR_RSP;
+ } else if (s->req_status == REQ_DONE) {
+ s->req_status = REQ_INIT;
+ }
+
+ return FREE_PAGE_READY;
+}
+
static void virtio_balloon_to_target(void *opaque, ram_addr_t target)
{
VirtIOBalloon *dev = VIRTIO_BALLOON(opaque);
@@ -444,7 +525,8 @@ static void virtio_balloon_device_realize(DeviceState *dev, Error **errp)
sizeof(struct virtio_balloon_config));
ret = qemu_add_balloon_handler(virtio_balloon_to_target,
- virtio_balloon_stat, s);
+ virtio_balloon_stat,
+ virtio_balloon_free_pages, s);
if (ret < 0) {
error_setg(errp, "Only one balloon device is supported");
@@ -455,8 +537,10 @@ static void virtio_balloon_device_realize(DeviceState *dev, Error **errp)
s->ivq = virtio_add_queue(vdev, 128, virtio_balloon_handle_output);
s->dvq = virtio_add_queue(vdev, 128, virtio_balloon_handle_output);
s->svq = virtio_add_queue(vdev, 128, virtio_balloon_receive_stats);
+ s->fvq = virtio_add_queue(vdev, 128, virtio_balloon_get_free_pages);
reset_stats(s);
+ s->req_status = REQ_INIT;
register_savevm(dev, "virtio-balloon", -1, 1,
virtio_balloon_save, virtio_balloon_load, s);
@@ -481,6 +565,12 @@ static void virtio_balloon_device_reset(VirtIODevice *vdev)
g_free(s->stats_vq_elem);
s->stats_vq_elem = NULL;
}
+
+ if (s->free_page_vq_elem != NULL) {
+ g_free(s->free_page_vq_elem);
+ s->free_page_vq_elem = NULL;
+ }
+ s->req_status = REQ_INIT;
}
static void virtio_balloon_instance_init(Object *obj)
diff --git a/include/hw/virtio/virtio-balloon.h b/include/hw/virtio/virtio-balloon.h
index 35f62ac..fbed81c 100644
--- a/include/hw/virtio/virtio-balloon.h
+++ b/include/hw/virtio/virtio-balloon.h
@@ -23,6 +23,27 @@
#define VIRTIO_BALLOON(obj) \
OBJECT_CHECK(VirtIOBalloon, (obj), TYPE_VIRTIO_BALLOON)
+typedef enum {
+ REQ_INIT,
+ REQ_ON_GOING,
+ REQ_DONE,
+ REQ_ERROR,
+ REQ_INVALID_PARAM,
+} VirtIoReqStatus;
+
+typedef enum {
+ FREE_PAGE_REQ,
+ FREE_PAGE_WAITING_FOR_RSP,
+ FREE_PAGE_READY,
+ FREE_PAGE_INVALID_PARAM,
+ FREE_PAGE_UNSUPPORT,
+ FREE_PAGE_ERROR,
+} FreePageStatus;
+
+typedef struct GetFreePageReq {
+ uint64_t param;
+} GetFreePageReq;
+
typedef struct virtio_balloon_stat VirtIOBalloonStat;
typedef struct virtio_balloon_stat_modern {
@@ -33,16 +54,23 @@ typedef struct virtio_balloon_stat_modern {
typedef struct VirtIOBalloon {
VirtIODevice parent_obj;
- VirtQueue *ivq, *dvq, *svq;
+ VirtQueue *ivq, *dvq, *svq, *fvq;
uint32_t num_pages;
uint32_t actual;
uint64_t stats[VIRTIO_BALLOON_S_NR];
VirtQueueElement *stats_vq_elem;
+ VirtQueueElement *free_page_vq_elem;
size_t stats_vq_offset;
QEMUTimer *stats_timer;
int64_t stats_last_update;
int64_t stats_poll_interval;
uint32_t host_features;
+ uint64_t *free_page_bmap;
+ uint64_t bmap_len;
+ uint64_t free_pages;
+ uint64_t cache_pages;
+ GetFreePageReq free_page_req;
+ VirtIoReqStatus req_status;
} VirtIOBalloon;
#endif
diff --git a/include/standard-headers/linux/virtio_balloon.h b/include/standard-headers/linux/virtio_balloon.h
index 9d06ccd..09211b0 100644
--- a/include/standard-headers/linux/virtio_balloon.h
+++ b/include/standard-headers/linux/virtio_balloon.h
@@ -34,6 +34,7 @@
#define VIRTIO_BALLOON_F_MUST_TELL_HOST 0 /* Tell before reclaiming pages */
#define VIRTIO_BALLOON_F_STATS_VQ 1 /* Memory Stats virtqueue */
#define VIRTIO_BALLOON_F_DEFLATE_ON_OOM 2 /* Deflate balloon on OOM */
+#define VIRTIO_BALLOON_F_GET_FREE_PAGE 3 /* Get the free page bitmap */
/* Size of a PFN in the balloon interface. */
#define VIRTIO_BALLOON_PFN_SHIFT 12
diff --git a/include/sysemu/balloon.h b/include/sysemu/balloon.h
index 3f976b4..81a1b9f 100644
--- a/include/sysemu/balloon.h
+++ b/include/sysemu/balloon.h
@@ -15,14 +15,24 @@
#define _QEMU_BALLOON_H
#include "qapi-types.h"
+#include "hw/virtio/virtio-balloon.h"
typedef void (QEMUBalloonEvent)(void *opaque, ram_addr_t target);
typedef void (QEMUBalloonStatus)(void *opaque, BalloonInfo *info);
+typedef FreePageStatus (QEMUBalloonGetFreePage)(void *opaque,
+ unsigned long *bitmap,
+ unsigned long len,
+ int drop_cache_ctl);
int qemu_add_balloon_handler(QEMUBalloonEvent *event_func,
- QEMUBalloonStatus *stat_func, void *opaque);
+ QEMUBalloonStatus *stat_func,
+ QEMUBalloonGetFreePage *get_free_page_func,
+ void *opaque);
void qemu_remove_balloon_handler(void *opaque);
bool qemu_balloon_is_inhibited(void);
void qemu_balloon_inhibit(bool state);
+bool balloon_free_pages_support(void);
+FreePageStatus balloon_get_free_pages(unsigned long *bitmap,
+ unsigned long len, int drop_cache_ctl);
#endif
--
1.8.3.1
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH QEMU 3/5] virtio-balloon: Add a new feature to balloon device
2016-04-19 14:20 ` [Qemu-devel] " Liang Li
@ 2016-04-19 16:32 ` Michael S. Tsirkin
-1 siblings, 0 replies; 19+ messages in thread
From: Michael S. Tsirkin @ 2016-04-19 16:32 UTC (permalink / raw)
To: Liang Li
Cc: quintela, amit.shah, pbonzini, lcapitulino, armbru,
peter.maydell, rth, ehabkost, james.hogan, aurelien, leon.alrae,
agraf, borntraeger, cornelia.huck, qemu-devel, kvm
On Tue, Apr 19, 2016 at 10:20:41PM +0800, Liang Li wrote:
> + if (s->req_status == REQ_INIT) {
> + s->free_page_bmap = bitmap;
> + if (elem == NULL || !elem->in_num) {
> + elem = virtqueue_pop(s->fvq, sizeof(VirtQueueElement));
> + if (!elem) {
> + return FREE_PAGE_ERROR;
> + }
> + s->free_page_vq_elem = elem;
> + }
> + s->free_page_req.param = drop_cache_ctl;
Dropping cache should probably be a separate request -
seems independently useful.
--
MST
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [Qemu-devel] [PATCH QEMU 3/5] virtio-balloon: Add a new feature to balloon device
@ 2016-04-19 16:32 ` Michael S. Tsirkin
0 siblings, 0 replies; 19+ messages in thread
From: Michael S. Tsirkin @ 2016-04-19 16:32 UTC (permalink / raw)
To: Liang Li
Cc: quintela, amit.shah, pbonzini, lcapitulino, armbru,
peter.maydell, rth, ehabkost, james.hogan, aurelien, leon.alrae,
agraf, borntraeger, cornelia.huck, qemu-devel, kvm
On Tue, Apr 19, 2016 at 10:20:41PM +0800, Liang Li wrote:
> + if (s->req_status == REQ_INIT) {
> + s->free_page_bmap = bitmap;
> + if (elem == NULL || !elem->in_num) {
> + elem = virtqueue_pop(s->fvq, sizeof(VirtQueueElement));
> + if (!elem) {
> + return FREE_PAGE_ERROR;
> + }
> + s->free_page_vq_elem = elem;
> + }
> + s->free_page_req.param = drop_cache_ctl;
Dropping cache should probably be a separate request -
seems independently useful.
--
MST
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH QEMU 4/5] migration: filter out free pages during live migration
2016-04-19 14:20 ` [Qemu-devel] " Liang Li
@ 2016-04-19 14:20 ` Liang Li
-1 siblings, 0 replies; 19+ messages in thread
From: Liang Li @ 2016-04-19 14:20 UTC (permalink / raw)
To: mst, quintela, amit.shah, pbonzini, lcapitulino
Cc: armbru, peter.maydell, rth, ehabkost, james.hogan, aurelien,
leon.alrae, agraf, borntraeger, cornelia.huck, qemu-devel, kvm,
Liang Li
After sending out the request for free pages, live migration
process will start without waiting for the free page bitmap is
ready. If the free page bitmap is not ready when doing the 1st
migration_bitmap_sync() after ram_save_setup(), the free page
bitmap will be ignored, this means the free pages will not be
filtered out in this case.
The current implementation can not work with post copy, if post
copy is enabled, we simply ignore the free pages. Will make it
work later.
Signed-off-by: Liang Li <liang.z.li@intel.com>
---
migration/ram.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 101 insertions(+)
diff --git a/migration/ram.c b/migration/ram.c
index 3f05738..3944426 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -41,6 +41,8 @@
#include "trace.h"
#include "exec/ram_addr.h"
#include "qemu/rcu_queue.h"
+#include "sysemu/balloon.h"
+#include "sysemu/kvm.h"
#ifdef DEBUG_MIGRATION_RAM
#define DPRINTF(fmt, ...) \
@@ -226,6 +228,8 @@ static QemuMutex migration_bitmap_mutex;
static uint64_t migration_dirty_pages;
static uint32_t last_version;
static bool ram_bulk_stage;
+static bool ignore_freepage_rsp;
+static bool drop_page_cache;
/* used by the search for pages to send */
struct PageSearchStatus {
@@ -242,6 +246,7 @@ static struct BitmapRcu {
struct rcu_head rcu;
/* Main migration bitmap */
unsigned long *bmap;
+ unsigned long *free_page_bmap;
/* bitmap of pages that haven't been sent even once
* only maintained and used in postcopy at the moment
* where it's used to send the dirtymap at the start
@@ -639,6 +644,7 @@ static void migration_bitmap_sync(void)
rcu_read_unlock();
qemu_mutex_unlock(&migration_bitmap_mutex);
+ ignore_freepage_rsp = true;
trace_migration_bitmap_sync_end(migration_dirty_pages
- num_dirty_pages_init);
num_dirty_pages_period += migration_dirty_pages - num_dirty_pages_init;
@@ -1417,6 +1423,9 @@ static void migration_bitmap_free(struct BitmapRcu *bmap)
{
g_free(bmap->bmap);
g_free(bmap->unsentmap);
+ if (balloon_free_pages_support() && !migrate_postcopy_ram()) {
+ g_free(bmap->free_page_bmap);
+ }
g_free(bmap);
}
@@ -1487,6 +1496,90 @@ void migration_bitmap_extend(ram_addr_t old, ram_addr_t new)
}
}
+static void filter_out_guest_free_page(unsigned long *free_page_bmap,
+ long nbits)
+{
+ long i, page_count = 0, len;
+ unsigned long *bitmap;
+
+ tighten_guest_free_page_bmap(free_page_bmap);
+ qemu_mutex_lock(&migration_bitmap_mutex);
+ bitmap = atomic_rcu_read(&migration_bitmap_rcu)->bmap;
+ slow_bitmap_complement(bitmap, free_page_bmap, nbits);
+
+ len = (last_ram_offset() >> TARGET_PAGE_BITS) / BITS_PER_LONG;
+ for (i = 0; i < len; i++) {
+ page_count += hweight_long(bitmap[i]);
+ }
+
+ migration_dirty_pages = page_count;
+ qemu_mutex_unlock(&migration_bitmap_mutex);
+}
+
+static void ram_request_free_page(unsigned long *bmap, unsigned long max_pfn)
+{
+ FreePageStatus status;
+
+ /* drop_page_cache should be set by user, the related code will be
+ * added later, set it to ture temporarily.
+ */
+ drop_page_cache = true;
+
+ status = balloon_get_free_pages(bmap, max_pfn, drop_page_cache);
+ switch (status) {
+ case FREE_PAGE_REQ:
+ ignore_freepage_rsp = false;
+ break;
+ case FREE_PAGE_ERROR:
+ error_report("Errro happend when request free page");
+ break;
+ default:
+ error_report("unexpected response status: %d", status);
+ break;
+ }
+}
+
+static void ram_handle_free_page(void)
+{
+ unsigned long nbits;
+ RAMBlock *pc_ram_block;
+ FreePageStatus status;
+
+ status = balloon_get_free_pages(migration_bitmap_rcu->free_page_bmap,
+ get_guest_max_pfn(), drop_page_cache);
+ switch (status) {
+ case FREE_PAGE_READY:
+ rcu_read_lock();
+ pc_ram_block = QLIST_FIRST_RCU(&ram_list.blocks);
+ nbits = pc_ram_block->used_length >> TARGET_PAGE_BITS;
+ filter_out_guest_free_page(migration_bitmap_rcu->free_page_bmap, nbits);
+ rcu_read_unlock();
+
+ qemu_mutex_lock_iothread();
+ migration_bitmap_sync();
+ qemu_mutex_unlock_iothread();
+ /*
+ * bulk stage assumes in (migration_bitmap_find_and_reset_dirty) that
+ * every page is dirty, that's no longer ture at this point.
+ */
+ ram_bulk_stage = false;
+ last_seen_block = NULL;
+ last_sent_block = NULL;
+ last_offset = 0;
+ break;
+ case FREE_PAGE_ERROR:
+ ignore_freepage_rsp = true;
+ error_report("failed to get free page");
+ break;
+ case FREE_PAGE_INVALID_PARAM:
+ ignore_freepage_rsp = true;
+ error_report("buffer overflow");
+ break;
+ default:
+ break;
+ }
+}
+
/*
* 'expected' is the value you expect the bitmap mostly to be full
* of; it won't bother printing lines that are all this value.
@@ -1950,6 +2043,11 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
qemu_mutex_unlock_ramlist();
qemu_mutex_unlock_iothread();
+ if (balloon_free_pages_support() && !migrate_postcopy_ram()) {
+ unsigned long max_pfn = get_guest_max_pfn();
+ migration_bitmap_rcu->free_page_bmap = bitmap_new(max_pfn);
+ ram_request_free_page(migration_bitmap_rcu->free_page_bmap, max_pfn);
+ }
qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE);
QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
@@ -1990,6 +2088,9 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
while ((ret = qemu_file_rate_limit(f)) == 0) {
int pages;
+ if (!ignore_freepage_rsp) {
+ ram_handle_free_page();
+ }
pages = ram_find_and_save_block(f, false, &bytes_transferred);
/* no more pages to sent */
if (pages == 0) {
--
1.8.3.1
^ permalink raw reply related [flat|nested] 19+ messages in thread* [Qemu-devel] [PATCH QEMU 4/5] migration: filter out free pages during live migration
@ 2016-04-19 14:20 ` Liang Li
0 siblings, 0 replies; 19+ messages in thread
From: Liang Li @ 2016-04-19 14:20 UTC (permalink / raw)
To: mst, quintela, amit.shah, pbonzini, lcapitulino
Cc: armbru, peter.maydell, rth, ehabkost, james.hogan, aurelien,
leon.alrae, agraf, borntraeger, cornelia.huck, qemu-devel, kvm,
Liang Li
After sending out the request for free pages, live migration
process will start without waiting for the free page bitmap is
ready. If the free page bitmap is not ready when doing the 1st
migration_bitmap_sync() after ram_save_setup(), the free page
bitmap will be ignored, this means the free pages will not be
filtered out in this case.
The current implementation can not work with post copy, if post
copy is enabled, we simply ignore the free pages. Will make it
work later.
Signed-off-by: Liang Li <liang.z.li@intel.com>
---
migration/ram.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 101 insertions(+)
diff --git a/migration/ram.c b/migration/ram.c
index 3f05738..3944426 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -41,6 +41,8 @@
#include "trace.h"
#include "exec/ram_addr.h"
#include "qemu/rcu_queue.h"
+#include "sysemu/balloon.h"
+#include "sysemu/kvm.h"
#ifdef DEBUG_MIGRATION_RAM
#define DPRINTF(fmt, ...) \
@@ -226,6 +228,8 @@ static QemuMutex migration_bitmap_mutex;
static uint64_t migration_dirty_pages;
static uint32_t last_version;
static bool ram_bulk_stage;
+static bool ignore_freepage_rsp;
+static bool drop_page_cache;
/* used by the search for pages to send */
struct PageSearchStatus {
@@ -242,6 +246,7 @@ static struct BitmapRcu {
struct rcu_head rcu;
/* Main migration bitmap */
unsigned long *bmap;
+ unsigned long *free_page_bmap;
/* bitmap of pages that haven't been sent even once
* only maintained and used in postcopy at the moment
* where it's used to send the dirtymap at the start
@@ -639,6 +644,7 @@ static void migration_bitmap_sync(void)
rcu_read_unlock();
qemu_mutex_unlock(&migration_bitmap_mutex);
+ ignore_freepage_rsp = true;
trace_migration_bitmap_sync_end(migration_dirty_pages
- num_dirty_pages_init);
num_dirty_pages_period += migration_dirty_pages - num_dirty_pages_init;
@@ -1417,6 +1423,9 @@ static void migration_bitmap_free(struct BitmapRcu *bmap)
{
g_free(bmap->bmap);
g_free(bmap->unsentmap);
+ if (balloon_free_pages_support() && !migrate_postcopy_ram()) {
+ g_free(bmap->free_page_bmap);
+ }
g_free(bmap);
}
@@ -1487,6 +1496,90 @@ void migration_bitmap_extend(ram_addr_t old, ram_addr_t new)
}
}
+static void filter_out_guest_free_page(unsigned long *free_page_bmap,
+ long nbits)
+{
+ long i, page_count = 0, len;
+ unsigned long *bitmap;
+
+ tighten_guest_free_page_bmap(free_page_bmap);
+ qemu_mutex_lock(&migration_bitmap_mutex);
+ bitmap = atomic_rcu_read(&migration_bitmap_rcu)->bmap;
+ slow_bitmap_complement(bitmap, free_page_bmap, nbits);
+
+ len = (last_ram_offset() >> TARGET_PAGE_BITS) / BITS_PER_LONG;
+ for (i = 0; i < len; i++) {
+ page_count += hweight_long(bitmap[i]);
+ }
+
+ migration_dirty_pages = page_count;
+ qemu_mutex_unlock(&migration_bitmap_mutex);
+}
+
+static void ram_request_free_page(unsigned long *bmap, unsigned long max_pfn)
+{
+ FreePageStatus status;
+
+ /* drop_page_cache should be set by user, the related code will be
+ * added later, set it to ture temporarily.
+ */
+ drop_page_cache = true;
+
+ status = balloon_get_free_pages(bmap, max_pfn, drop_page_cache);
+ switch (status) {
+ case FREE_PAGE_REQ:
+ ignore_freepage_rsp = false;
+ break;
+ case FREE_PAGE_ERROR:
+ error_report("Errro happend when request free page");
+ break;
+ default:
+ error_report("unexpected response status: %d", status);
+ break;
+ }
+}
+
+static void ram_handle_free_page(void)
+{
+ unsigned long nbits;
+ RAMBlock *pc_ram_block;
+ FreePageStatus status;
+
+ status = balloon_get_free_pages(migration_bitmap_rcu->free_page_bmap,
+ get_guest_max_pfn(), drop_page_cache);
+ switch (status) {
+ case FREE_PAGE_READY:
+ rcu_read_lock();
+ pc_ram_block = QLIST_FIRST_RCU(&ram_list.blocks);
+ nbits = pc_ram_block->used_length >> TARGET_PAGE_BITS;
+ filter_out_guest_free_page(migration_bitmap_rcu->free_page_bmap, nbits);
+ rcu_read_unlock();
+
+ qemu_mutex_lock_iothread();
+ migration_bitmap_sync();
+ qemu_mutex_unlock_iothread();
+ /*
+ * bulk stage assumes in (migration_bitmap_find_and_reset_dirty) that
+ * every page is dirty, that's no longer ture at this point.
+ */
+ ram_bulk_stage = false;
+ last_seen_block = NULL;
+ last_sent_block = NULL;
+ last_offset = 0;
+ break;
+ case FREE_PAGE_ERROR:
+ ignore_freepage_rsp = true;
+ error_report("failed to get free page");
+ break;
+ case FREE_PAGE_INVALID_PARAM:
+ ignore_freepage_rsp = true;
+ error_report("buffer overflow");
+ break;
+ default:
+ break;
+ }
+}
+
/*
* 'expected' is the value you expect the bitmap mostly to be full
* of; it won't bother printing lines that are all this value.
@@ -1950,6 +2043,11 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
qemu_mutex_unlock_ramlist();
qemu_mutex_unlock_iothread();
+ if (balloon_free_pages_support() && !migrate_postcopy_ram()) {
+ unsigned long max_pfn = get_guest_max_pfn();
+ migration_bitmap_rcu->free_page_bmap = bitmap_new(max_pfn);
+ ram_request_free_page(migration_bitmap_rcu->free_page_bmap, max_pfn);
+ }
qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE);
QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
@@ -1990,6 +2088,9 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
while ((ret = qemu_file_rate_limit(f)) == 0) {
int pages;
+ if (!ignore_freepage_rsp) {
+ ram_handle_free_page();
+ }
pages = ram_find_and_save_block(f, false, &bytes_transferred);
/* no more pages to sent */
if (pages == 0) {
--
1.8.3.1
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH QEMU 5/5] migration: Add the interface for cache drop control
2016-04-19 14:20 ` [Qemu-devel] " Liang Li
@ 2016-04-19 14:20 ` Liang Li
-1 siblings, 0 replies; 19+ messages in thread
From: Liang Li @ 2016-04-19 14:20 UTC (permalink / raw)
To: mst, quintela, amit.shah, pbonzini, lcapitulino
Cc: armbru, peter.maydell, rth, ehabkost, james.hogan, aurelien,
leon.alrae, agraf, borntraeger, cornelia.huck, qemu-devel, kvm,
Liang Li
Whether drop the cache and drop what kind of cache depend on the
user, add the related qmp and hmp interface to query and set the
cache control value.
Signed-off-by: Liang Li <liang.z.li@intel.com>
---
hmp.c | 8 ++++++++
include/migration/migration.h | 1 +
migration/migration.c | 31 ++++++++++++++++++++++++++++++-
migration/ram.c | 10 ++--------
qapi-schema.json | 25 ++++++++++++++++++++++---
5 files changed, 63 insertions(+), 12 deletions(-)
diff --git a/hmp.c b/hmp.c
index d510236..17f418e 100644
--- a/hmp.c
+++ b/hmp.c
@@ -286,6 +286,9 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
monitor_printf(mon, " %s: %" PRId64,
MigrationParameter_lookup[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT],
params->x_cpu_throttle_increment);
+ monitor_printf(mon, " %s: %" PRId64,
+ MigrationParameter_lookup[MIGRATION_PARAMETER_X_DROP_CACHE],
+ params->x_drop_cache);
monitor_printf(mon, "\n");
}
@@ -1242,6 +1245,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
bool has_decompress_threads = false;
bool has_x_cpu_throttle_initial = false;
bool has_x_cpu_throttle_increment = false;
+ bool has_x_drop_cache = false;
int i;
for (i = 0; i < MIGRATION_PARAMETER__MAX; i++) {
@@ -1262,12 +1266,16 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
case MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT:
has_x_cpu_throttle_increment = true;
break;
+ case MIGRATION_PARAMETER_X_DROP_CACHE:
+ has_x_drop_cache = true;
+ break;
}
qmp_migrate_set_parameters(has_compress_level, value,
has_compress_threads, value,
has_decompress_threads, value,
has_x_cpu_throttle_initial, value,
has_x_cpu_throttle_increment, value,
+ has_x_drop_cache, value,
&err);
break;
}
diff --git a/include/migration/migration.h b/include/migration/migration.h
index ac2c12c..873e3bc 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -283,6 +283,7 @@ bool migrate_use_compression(void);
int migrate_compress_level(void);
int migrate_compress_threads(void);
int migrate_decompress_threads(void);
+int migrate_drop_cache(void);
bool migrate_use_events(void);
/* Sending on the return path - generic and then for each message type */
diff --git a/migration/migration.c b/migration/migration.c
index 991313a..ecd07b8 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -52,6 +52,13 @@
/* Define default autoconverge cpu throttle migration parameters */
#define DEFAULT_MIGRATE_X_CPU_THROTTLE_INITIAL 20
#define DEFAULT_MIGRATE_X_CPU_THROTTLE_INCREMENT 10
+/* Default cache drop control
+ * 0: no drop
+ * 1: drop clean page cache
+ * 2: drop slab cache
+ * 3: drop both clean and slab cache
+ */
+#define DEFAULT_MIGRATE_X_DROP_CACHE 0
/* Migration XBZRLE default cache size */
#define DEFAULT_MIGRATE_CACHE_SIZE (64 * 1024 * 1024)
@@ -91,6 +98,8 @@ MigrationState *migrate_get_current(void)
DEFAULT_MIGRATE_X_CPU_THROTTLE_INITIAL,
.parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT] =
DEFAULT_MIGRATE_X_CPU_THROTTLE_INCREMENT,
+ .parameters[MIGRATION_PARAMETER_X_DROP_CACHE] =
+ DEFAULT_MIGRATE_X_DROP_CACHE,
};
if (!once) {
@@ -525,6 +534,7 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL];
params->x_cpu_throttle_increment =
s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT];
+ params->x_drop_cache = s->parameters[MIGRATION_PARAMETER_X_DROP_CACHE];
return params;
}
@@ -721,7 +731,9 @@ void qmp_migrate_set_parameters(bool has_compress_level,
bool has_x_cpu_throttle_initial,
int64_t x_cpu_throttle_initial,
bool has_x_cpu_throttle_increment,
- int64_t x_cpu_throttle_increment, Error **errp)
+ int64_t x_cpu_throttle_increment,
+ bool has_x_drop_cache,
+ int64_t x_drop_cache, Error **errp)
{
MigrationState *s = migrate_get_current();
@@ -756,6 +768,11 @@ void qmp_migrate_set_parameters(bool has_compress_level,
"x_cpu_throttle_increment",
"an integer in the range of 1 to 99");
}
+ if (has_x_drop_cache && (x_drop_cache < 0 || x_drop_cache > 3)) {
+ error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
+ "x_drop_cache",
+ "an integer in the range of 0 to 3");
+ }
if (has_compress_level) {
s->parameters[MIGRATION_PARAMETER_COMPRESS_LEVEL] = compress_level;
@@ -776,6 +793,9 @@ void qmp_migrate_set_parameters(bool has_compress_level,
s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT] =
x_cpu_throttle_increment;
}
+ if (has_x_drop_cache) {
+ s->parameters[MIGRATION_PARAMETER_X_DROP_CACHE] = x_drop_cache;
+ }
}
void qmp_migrate_start_postcopy(Error **errp)
@@ -1182,6 +1202,15 @@ int migrate_decompress_threads(void)
return s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS];
}
+int migrate_drop_cache(void)
+{
+ MigrationState *s;
+
+ s = migrate_get_current();
+
+ return s->parameters[MIGRATION_PARAMETER_X_DROP_CACHE];
+}
+
bool migrate_use_events(void)
{
MigrationState *s;
diff --git a/migration/ram.c b/migration/ram.c
index 3944426..0d5fc97 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -229,7 +229,6 @@ static uint64_t migration_dirty_pages;
static uint32_t last_version;
static bool ram_bulk_stage;
static bool ignore_freepage_rsp;
-static bool drop_page_cache;
/* used by the search for pages to send */
struct PageSearchStatus {
@@ -1520,12 +1519,7 @@ static void ram_request_free_page(unsigned long *bmap, unsigned long max_pfn)
{
FreePageStatus status;
- /* drop_page_cache should be set by user, the related code will be
- * added later, set it to ture temporarily.
- */
- drop_page_cache = true;
-
- status = balloon_get_free_pages(bmap, max_pfn, drop_page_cache);
+ status = balloon_get_free_pages(bmap, max_pfn, migrate_drop_cache());
switch (status) {
case FREE_PAGE_REQ:
ignore_freepage_rsp = false;
@@ -1546,7 +1540,7 @@ static void ram_handle_free_page(void)
FreePageStatus status;
status = balloon_get_free_pages(migration_bitmap_rcu->free_page_bmap,
- get_guest_max_pfn(), drop_page_cache);
+ get_guest_max_pfn(), migrate_drop_cache());
switch (status) {
case FREE_PAGE_READY:
rcu_read_lock();
diff --git a/qapi-schema.json b/qapi-schema.json
index 54634c4..8674d35 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -612,11 +612,21 @@
# @x-cpu-throttle-increment: throttle percentage increase each time
# auto-converge detects that migration is not making
# progress. The default value is 10. (Since 2.5)
+# @x-drop-cache: drop guest's page cache during live migration, the value can
+# be set to 0, 1, 2, 3. 1 means drop the clean page cache, 2 means
+# drop the slab cache, 3 means to drop both clean page cache, 0 means
+# do not drop any cache. Drop cache can speed up live migration
+# process and reduce the network traffic, the side affect is the
+# performance impact to the guest, 1 is the recommended value for
+# proper balance between speed and performance. The value only takes
+# affect when the virtio-balloon device are enabled and the guest
+# has the related driver. The default value is 0. (Since 2.7)
# Since: 2.4
##
{ 'enum': 'MigrationParameter',
'data': ['compress-level', 'compress-threads', 'decompress-threads',
- 'x-cpu-throttle-initial', 'x-cpu-throttle-increment'] }
+ 'x-cpu-throttle-initial', 'x-cpu-throttle-increment',
+ 'x-drop-cache'] }
#
# @migrate-set-parameters
@@ -636,6 +646,10 @@
# @x-cpu-throttle-increment: throttle percentage increase each time
# auto-converge detects that migration is not making
# progress. The default value is 10. (Since 2.5)
+#
+# @x-drop-cache: guest cache drop control, determine what kind of guest cache
+# to drop during live migration. The default value is 0. (Since 2.7)
+#
# Since: 2.4
##
{ 'command': 'migrate-set-parameters',
@@ -643,7 +657,8 @@
'*compress-threads': 'int',
'*decompress-threads': 'int',
'*x-cpu-throttle-initial': 'int',
- '*x-cpu-throttle-increment': 'int'} }
+ '*x-cpu-throttle-increment': 'int',
+ '*x-drop-cache': 'int'} }
#
# @MigrationParameters
@@ -662,6 +677,9 @@
# auto-converge detects that migration is not making
# progress. The default value is 10. (Since 2.5)
#
+# @x-drop-cache: guest cache drop control, determine what kind of guest cache
+# to drop during live migration. The default value is 0. (Since 2.7)
+#
# Since: 2.4
##
{ 'struct': 'MigrationParameters',
@@ -669,7 +687,8 @@
'compress-threads': 'int',
'decompress-threads': 'int',
'x-cpu-throttle-initial': 'int',
- 'x-cpu-throttle-increment': 'int'} }
+ 'x-cpu-throttle-increment': 'int',
+ 'x-drop-cache': 'int'} }
##
# @query-migrate-parameters
#
--
1.8.3.1
^ permalink raw reply related [flat|nested] 19+ messages in thread* [Qemu-devel] [PATCH QEMU 5/5] migration: Add the interface for cache drop control
@ 2016-04-19 14:20 ` Liang Li
0 siblings, 0 replies; 19+ messages in thread
From: Liang Li @ 2016-04-19 14:20 UTC (permalink / raw)
To: mst, quintela, amit.shah, pbonzini, lcapitulino
Cc: armbru, peter.maydell, rth, ehabkost, james.hogan, aurelien,
leon.alrae, agraf, borntraeger, cornelia.huck, qemu-devel, kvm,
Liang Li
Whether drop the cache and drop what kind of cache depend on the
user, add the related qmp and hmp interface to query and set the
cache control value.
Signed-off-by: Liang Li <liang.z.li@intel.com>
---
hmp.c | 8 ++++++++
include/migration/migration.h | 1 +
migration/migration.c | 31 ++++++++++++++++++++++++++++++-
migration/ram.c | 10 ++--------
qapi-schema.json | 25 ++++++++++++++++++++++---
5 files changed, 63 insertions(+), 12 deletions(-)
diff --git a/hmp.c b/hmp.c
index d510236..17f418e 100644
--- a/hmp.c
+++ b/hmp.c
@@ -286,6 +286,9 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
monitor_printf(mon, " %s: %" PRId64,
MigrationParameter_lookup[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT],
params->x_cpu_throttle_increment);
+ monitor_printf(mon, " %s: %" PRId64,
+ MigrationParameter_lookup[MIGRATION_PARAMETER_X_DROP_CACHE],
+ params->x_drop_cache);
monitor_printf(mon, "\n");
}
@@ -1242,6 +1245,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
bool has_decompress_threads = false;
bool has_x_cpu_throttle_initial = false;
bool has_x_cpu_throttle_increment = false;
+ bool has_x_drop_cache = false;
int i;
for (i = 0; i < MIGRATION_PARAMETER__MAX; i++) {
@@ -1262,12 +1266,16 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
case MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT:
has_x_cpu_throttle_increment = true;
break;
+ case MIGRATION_PARAMETER_X_DROP_CACHE:
+ has_x_drop_cache = true;
+ break;
}
qmp_migrate_set_parameters(has_compress_level, value,
has_compress_threads, value,
has_decompress_threads, value,
has_x_cpu_throttle_initial, value,
has_x_cpu_throttle_increment, value,
+ has_x_drop_cache, value,
&err);
break;
}
diff --git a/include/migration/migration.h b/include/migration/migration.h
index ac2c12c..873e3bc 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -283,6 +283,7 @@ bool migrate_use_compression(void);
int migrate_compress_level(void);
int migrate_compress_threads(void);
int migrate_decompress_threads(void);
+int migrate_drop_cache(void);
bool migrate_use_events(void);
/* Sending on the return path - generic and then for each message type */
diff --git a/migration/migration.c b/migration/migration.c
index 991313a..ecd07b8 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -52,6 +52,13 @@
/* Define default autoconverge cpu throttle migration parameters */
#define DEFAULT_MIGRATE_X_CPU_THROTTLE_INITIAL 20
#define DEFAULT_MIGRATE_X_CPU_THROTTLE_INCREMENT 10
+/* Default cache drop control
+ * 0: no drop
+ * 1: drop clean page cache
+ * 2: drop slab cache
+ * 3: drop both clean and slab cache
+ */
+#define DEFAULT_MIGRATE_X_DROP_CACHE 0
/* Migration XBZRLE default cache size */
#define DEFAULT_MIGRATE_CACHE_SIZE (64 * 1024 * 1024)
@@ -91,6 +98,8 @@ MigrationState *migrate_get_current(void)
DEFAULT_MIGRATE_X_CPU_THROTTLE_INITIAL,
.parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT] =
DEFAULT_MIGRATE_X_CPU_THROTTLE_INCREMENT,
+ .parameters[MIGRATION_PARAMETER_X_DROP_CACHE] =
+ DEFAULT_MIGRATE_X_DROP_CACHE,
};
if (!once) {
@@ -525,6 +534,7 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL];
params->x_cpu_throttle_increment =
s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT];
+ params->x_drop_cache = s->parameters[MIGRATION_PARAMETER_X_DROP_CACHE];
return params;
}
@@ -721,7 +731,9 @@ void qmp_migrate_set_parameters(bool has_compress_level,
bool has_x_cpu_throttle_initial,
int64_t x_cpu_throttle_initial,
bool has_x_cpu_throttle_increment,
- int64_t x_cpu_throttle_increment, Error **errp)
+ int64_t x_cpu_throttle_increment,
+ bool has_x_drop_cache,
+ int64_t x_drop_cache, Error **errp)
{
MigrationState *s = migrate_get_current();
@@ -756,6 +768,11 @@ void qmp_migrate_set_parameters(bool has_compress_level,
"x_cpu_throttle_increment",
"an integer in the range of 1 to 99");
}
+ if (has_x_drop_cache && (x_drop_cache < 0 || x_drop_cache > 3)) {
+ error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
+ "x_drop_cache",
+ "an integer in the range of 0 to 3");
+ }
if (has_compress_level) {
s->parameters[MIGRATION_PARAMETER_COMPRESS_LEVEL] = compress_level;
@@ -776,6 +793,9 @@ void qmp_migrate_set_parameters(bool has_compress_level,
s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT] =
x_cpu_throttle_increment;
}
+ if (has_x_drop_cache) {
+ s->parameters[MIGRATION_PARAMETER_X_DROP_CACHE] = x_drop_cache;
+ }
}
void qmp_migrate_start_postcopy(Error **errp)
@@ -1182,6 +1202,15 @@ int migrate_decompress_threads(void)
return s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS];
}
+int migrate_drop_cache(void)
+{
+ MigrationState *s;
+
+ s = migrate_get_current();
+
+ return s->parameters[MIGRATION_PARAMETER_X_DROP_CACHE];
+}
+
bool migrate_use_events(void)
{
MigrationState *s;
diff --git a/migration/ram.c b/migration/ram.c
index 3944426..0d5fc97 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -229,7 +229,6 @@ static uint64_t migration_dirty_pages;
static uint32_t last_version;
static bool ram_bulk_stage;
static bool ignore_freepage_rsp;
-static bool drop_page_cache;
/* used by the search for pages to send */
struct PageSearchStatus {
@@ -1520,12 +1519,7 @@ static void ram_request_free_page(unsigned long *bmap, unsigned long max_pfn)
{
FreePageStatus status;
- /* drop_page_cache should be set by user, the related code will be
- * added later, set it to ture temporarily.
- */
- drop_page_cache = true;
-
- status = balloon_get_free_pages(bmap, max_pfn, drop_page_cache);
+ status = balloon_get_free_pages(bmap, max_pfn, migrate_drop_cache());
switch (status) {
case FREE_PAGE_REQ:
ignore_freepage_rsp = false;
@@ -1546,7 +1540,7 @@ static void ram_handle_free_page(void)
FreePageStatus status;
status = balloon_get_free_pages(migration_bitmap_rcu->free_page_bmap,
- get_guest_max_pfn(), drop_page_cache);
+ get_guest_max_pfn(), migrate_drop_cache());
switch (status) {
case FREE_PAGE_READY:
rcu_read_lock();
diff --git a/qapi-schema.json b/qapi-schema.json
index 54634c4..8674d35 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -612,11 +612,21 @@
# @x-cpu-throttle-increment: throttle percentage increase each time
# auto-converge detects that migration is not making
# progress. The default value is 10. (Since 2.5)
+# @x-drop-cache: drop guest's page cache during live migration, the value can
+# be set to 0, 1, 2, 3. 1 means drop the clean page cache, 2 means
+# drop the slab cache, 3 means to drop both clean page cache, 0 means
+# do not drop any cache. Drop cache can speed up live migration
+# process and reduce the network traffic, the side affect is the
+# performance impact to the guest, 1 is the recommended value for
+# proper balance between speed and performance. The value only takes
+# affect when the virtio-balloon device are enabled and the guest
+# has the related driver. The default value is 0. (Since 2.7)
# Since: 2.4
##
{ 'enum': 'MigrationParameter',
'data': ['compress-level', 'compress-threads', 'decompress-threads',
- 'x-cpu-throttle-initial', 'x-cpu-throttle-increment'] }
+ 'x-cpu-throttle-initial', 'x-cpu-throttle-increment',
+ 'x-drop-cache'] }
#
# @migrate-set-parameters
@@ -636,6 +646,10 @@
# @x-cpu-throttle-increment: throttle percentage increase each time
# auto-converge detects that migration is not making
# progress. The default value is 10. (Since 2.5)
+#
+# @x-drop-cache: guest cache drop control, determine what kind of guest cache
+# to drop during live migration. The default value is 0. (Since 2.7)
+#
# Since: 2.4
##
{ 'command': 'migrate-set-parameters',
@@ -643,7 +657,8 @@
'*compress-threads': 'int',
'*decompress-threads': 'int',
'*x-cpu-throttle-initial': 'int',
- '*x-cpu-throttle-increment': 'int'} }
+ '*x-cpu-throttle-increment': 'int',
+ '*x-drop-cache': 'int'} }
#
# @MigrationParameters
@@ -662,6 +677,9 @@
# auto-converge detects that migration is not making
# progress. The default value is 10. (Since 2.5)
#
+# @x-drop-cache: guest cache drop control, determine what kind of guest cache
+# to drop during live migration. The default value is 0. (Since 2.7)
+#
# Since: 2.4
##
{ 'struct': 'MigrationParameters',
@@ -669,7 +687,8 @@
'compress-threads': 'int',
'decompress-threads': 'int',
'x-cpu-throttle-initial': 'int',
- 'x-cpu-throttle-increment': 'int'} }
+ 'x-cpu-throttle-increment': 'int',
+ 'x-drop-cache': 'int'} }
##
# @query-migrate-parameters
#
--
1.8.3.1
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [Qemu-devel] [PATCH QEMU 5/5] migration: Add the interface for cache drop control
2016-04-19 14:20 ` [Qemu-devel] " Liang Li
(?)
@ 2016-04-19 14:52 ` Eric Blake
2016-04-19 14:59 ` Li, Liang Z
-1 siblings, 1 reply; 19+ messages in thread
From: Eric Blake @ 2016-04-19 14:52 UTC (permalink / raw)
To: Liang Li, mst, quintela, amit.shah, pbonzini, lcapitulino
Cc: peter.maydell, james.hogan, ehabkost, kvm, qemu-devel, armbru,
agraf, borntraeger, cornelia.huck, leon.alrae, aurelien, rth
[-- Attachment #1: Type: text/plain, Size: 1633 bytes --]
On 04/19/2016 08:20 AM, Liang Li wrote:
> Whether drop the cache and drop what kind of cache depend on the
Whether to drop the cache, and what kind of cache drop to perform,
depend ...
> user, add the related qmp and hmp interface to query and set the
> cache control value.
>
> Signed-off-by: Liang Li <liang.z.li@intel.com>
> ---
> +++ b/qapi-schema.json
> @@ -612,11 +612,21 @@
> # @x-cpu-throttle-increment: throttle percentage increase each time
> # auto-converge detects that migration is not making
> # progress. The default value is 10. (Since 2.5)
> +# @x-drop-cache: drop guest's page cache during live migration, the value can
> +# be set to 0, 1, 2, 3. 1 means drop the clean page cache, 2 means
> +# drop the slab cache, 3 means to drop both clean page cache, 0 means
> +# do not drop any cache. Drop cache can speed up live migration
> +# process and reduce the network traffic, the side affect is the
> +# performance impact to the guest, 1 is the recommended value for
> +# proper balance between speed and performance. The value only takes
> +# affect when the virtio-balloon device are enabled and the guest
> +# has the related driver. The default value is 0. (Since 2.7)
Absolutely not this interface. Make this an enum type, not a raw
magic-number integer type, where the user requests
"x-drop-cache":"clean", "x-drop-cache":"slab", and so on.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* RE: [Qemu-devel] [PATCH QEMU 5/5] migration: Add the interface for cache drop control
2016-04-19 14:52 ` Eric Blake
@ 2016-04-19 14:59 ` Li, Liang Z
0 siblings, 0 replies; 19+ messages in thread
From: Li, Liang Z @ 2016-04-19 14:59 UTC (permalink / raw)
To: Eric Blake, mst, quintela, amit.shah, pbonzini, lcapitulino
Cc: peter.maydell, james.hogan, ehabkost, kvm, qemu-devel, armbru,
agraf, borntraeger, cornelia.huck, leon.alrae, aurelien, rth
> Sent: Tuesday, April 19, 2016 10:52 PM
> To: Li, Liang Z; mst@redhat.com; quintela@redhat.com;
> amit.shah@redhat.com; pbonzini@redhat.com; lcapitulino@redhat.com
> Cc: peter.maydell@linaro.org; james.hogan@imgtec.com;
> ehabkost@redhat.com; kvm@vger.kernel.org; qemu-devel@nongnu.org;
> armbru@redhat.com; agraf@suse.de; borntraeger@de.ibm.com;
> cornelia.huck@de.ibm.com; leon.alrae@imgtec.com; aurelien@aurel32.net;
> rth@twiddle.net
> Subject: Re: [Qemu-devel] [PATCH QEMU 5/5] migration: Add the interface
> for cache drop control
>
> On 04/19/2016 08:20 AM, Liang Li wrote:
> > Whether drop the cache and drop what kind of cache depend on the
>
> Whether to drop the cache, and what kind of cache drop to perform,
> depend ...
>
> > user, add the related qmp and hmp interface to query and set the cache
> > control value.
> >
> > Signed-off-by: Liang Li <liang.z.li@intel.com>
> > ---
>
> > +++ b/qapi-schema.json
> > @@ -612,11 +612,21 @@
> > # @x-cpu-throttle-increment: throttle percentage increase each time
> > # auto-converge detects that migration is not making
> > # progress. The default value is 10. (Since 2.5)
> > +# @x-drop-cache: drop guest's page cache during live migration, the value
> can
> > +# be set to 0, 1, 2, 3. 1 means drop the clean page cache, 2 means
> > +# drop the slab cache, 3 means to drop both clean page cache, 0
> means
> > +# do not drop any cache. Drop cache can speed up live migration
> > +# process and reduce the network traffic, the side affect is the
> > +# performance impact to the guest, 1 is the recommended value for
> > +# proper balance between speed and performance. The value only
> takes
> > +# affect when the virtio-balloon device are enabled and the guest
> > +# has the related driver. The default value is 0. (Since 2.7)
>
> Absolutely not this interface. Make this an enum type, not a raw magic-
> number integer type, where the user requests "x-drop-cache":"clean", "x-
> drop-cache":"slab", and so on.
>
The enum type looks better, thanks!
Liang
> --
> Eric Blake eblake redhat com +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] [PATCH QEMU 5/5] migration: Add the interface for cache drop control
@ 2016-04-19 14:59 ` Li, Liang Z
0 siblings, 0 replies; 19+ messages in thread
From: Li, Liang Z @ 2016-04-19 14:59 UTC (permalink / raw)
To: Eric Blake, mst, quintela, amit.shah, pbonzini, lcapitulino
Cc: peter.maydell, james.hogan, ehabkost, kvm, qemu-devel, armbru,
agraf, borntraeger, cornelia.huck, leon.alrae, aurelien, rth
> Sent: Tuesday, April 19, 2016 10:52 PM
> To: Li, Liang Z; mst@redhat.com; quintela@redhat.com;
> amit.shah@redhat.com; pbonzini@redhat.com; lcapitulino@redhat.com
> Cc: peter.maydell@linaro.org; james.hogan@imgtec.com;
> ehabkost@redhat.com; kvm@vger.kernel.org; qemu-devel@nongnu.org;
> armbru@redhat.com; agraf@suse.de; borntraeger@de.ibm.com;
> cornelia.huck@de.ibm.com; leon.alrae@imgtec.com; aurelien@aurel32.net;
> rth@twiddle.net
> Subject: Re: [Qemu-devel] [PATCH QEMU 5/5] migration: Add the interface
> for cache drop control
>
> On 04/19/2016 08:20 AM, Liang Li wrote:
> > Whether drop the cache and drop what kind of cache depend on the
>
> Whether to drop the cache, and what kind of cache drop to perform,
> depend ...
>
> > user, add the related qmp and hmp interface to query and set the cache
> > control value.
> >
> > Signed-off-by: Liang Li <liang.z.li@intel.com>
> > ---
>
> > +++ b/qapi-schema.json
> > @@ -612,11 +612,21 @@
> > # @x-cpu-throttle-increment: throttle percentage increase each time
> > # auto-converge detects that migration is not making
> > # progress. The default value is 10. (Since 2.5)
> > +# @x-drop-cache: drop guest's page cache during live migration, the value
> can
> > +# be set to 0, 1, 2, 3. 1 means drop the clean page cache, 2 means
> > +# drop the slab cache, 3 means to drop both clean page cache, 0
> means
> > +# do not drop any cache. Drop cache can speed up live migration
> > +# process and reduce the network traffic, the side affect is the
> > +# performance impact to the guest, 1 is the recommended value for
> > +# proper balance between speed and performance. The value only
> takes
> > +# affect when the virtio-balloon device are enabled and the guest
> > +# has the related driver. The default value is 0. (Since 2.7)
>
> Absolutely not this interface. Make this an enum type, not a raw magic-
> number integer type, where the user requests "x-drop-cache":"clean", "x-
> drop-cache":"slab", and so on.
>
The enum type looks better, thanks!
Liang
> --
> Eric Blake eblake redhat com +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH QEMU 0/5] spee up live migration by skipping free pages
2016-04-19 14:20 ` [Qemu-devel] " Liang Li
@ 2016-04-19 14:37 ` Alexander Graf
-1 siblings, 0 replies; 19+ messages in thread
From: Alexander Graf @ 2016-04-19 14:37 UTC (permalink / raw)
To: Liang Li, mst, quintela, amit.shah, pbonzini, lcapitulino
Cc: peter.maydell, james.hogan, ehabkost, kvm, armbru, qemu-devel,
borntraeger, cornelia.huck, leon.alrae, aurelien, rth
On 04/19/2016 04:20 PM, Liang Li wrote:
> Current QEMU live migration implementation marks all guest's RAM pages
> as dirtied in the ram bulk stage, all these pages will be processed
> and it consumes quite a lot of CPU cycles and network bandwidth.
>
> From guest's point of view, it doesn't care about the content in free
> page. We can make use of this fact and skip processing the free
> pages, it can save a lot CPU cycles and reduce the network traffic
> significantly while speed up the live migration process obviously.
>
> This patch set is the QEMU side implementation.
>
> The virtio-balloon is extended so that QEMU can get the free pages
> information from the guest.
>
> After getting the free page bitmap, QEMU can use it to filter out
> guest's free pages. This make the live migration process much more
> efficient.
>
> In order to skip more free pages, we add an interface to let the user
> decide whether dropping the cache in guest during live migration.
Wouldn't that potentially break things like CoLo? Which side is the slow
one? We already do condense zero pages into small metadata in the
migration stream. Is writing them out the slow part? If so, just don't
write them in you're on a clean state.
Alex
>
> Performance data
> ================
>
> Test environment:
>
> CPU: Intel (R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
> Host RAM: 64GB
> Host Linux Kernel: 4.4.0 Host OS: CentOS 7.1
> Guest Linux Kernel: 4.5.rc6 Guest OS: CentOS 6.6
> Network: Intel X710 with 10 Gigabit connection
> Guest RAM: 8GB
>
> Case 1: Idle guest just boots:
> ==================================================
> | original | skip free page
> --------------------------------------------------
> total time(ms) | 1505 | 573
> --------------------------------------------------
> transferred ram(KB) | 399792 | 379057
> ==================================================
>
> Case 2: The guest has ever run some memory consuming workload, the
> workload is terminated before live migration.
> ==================================================
> | original | skip free page
> --------------------------------------------------
> total time(ms) | 10641 | 597
> --------------------------------------------------
> transferred ram(KB) | 8350829 | 389900
> ==================================================
>
> Case 3: The guest has ever built the linux kernel, the building is
> completed before live migration. The page cache is dropped
> during live migration in this case.
> ==================================================
> | original | skip free page
> --------------------------------------------------
> total time(ms) | 2636 | 914
> --------------------------------------------------
> transferred ram(KB) | 1316747 | 421980
> ==================================================
>
> Liang Li (5):
> bitmap: Add a new bitmap_move function
> kvm: Add two new arch specific functions
> virtio-balloon: Add a new feature to balloon device
> migration: filter out free pages during live migration
> migration: Add the interface for cache drop control
>
> balloon.c | 29 +++++++-
> hmp.c | 8 +++
> hw/virtio/virtio-balloon.c | 92 +++++++++++++++++++++++-
> include/hw/virtio/virtio-balloon.h | 30 +++++++-
> include/migration/migration.h | 1 +
> include/qemu/bitmap.h | 13 ++++
> include/standard-headers/linux/virtio_balloon.h | 1 +
> include/sysemu/balloon.h | 12 +++-
> include/sysemu/kvm.h | 2 +
> migration/migration.c | 31 +++++++-
> migration/ram.c | 95 +++++++++++++++++++++++++
> qapi-schema.json | 25 ++++++-
> target-arm/kvm.c | 14 ++++
> target-i386/kvm.c | 35 +++++++++
> target-mips/kvm.c | 14 ++++
> target-ppc/kvm.c | 14 ++++
> target-s390x/kvm.c | 14 ++++
> 17 files changed, 421 insertions(+), 9 deletions(-)
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Qemu-devel] [PATCH QEMU 0/5] spee up live migration by skipping free pages
@ 2016-04-19 14:37 ` Alexander Graf
0 siblings, 0 replies; 19+ messages in thread
From: Alexander Graf @ 2016-04-19 14:37 UTC (permalink / raw)
To: Liang Li, mst, quintela, amit.shah, pbonzini, lcapitulino
Cc: armbru, peter.maydell, rth, ehabkost, james.hogan, aurelien,
leon.alrae, borntraeger, cornelia.huck, qemu-devel, kvm
On 04/19/2016 04:20 PM, Liang Li wrote:
> Current QEMU live migration implementation marks all guest's RAM pages
> as dirtied in the ram bulk stage, all these pages will be processed
> and it consumes quite a lot of CPU cycles and network bandwidth.
>
> From guest's point of view, it doesn't care about the content in free
> page. We can make use of this fact and skip processing the free
> pages, it can save a lot CPU cycles and reduce the network traffic
> significantly while speed up the live migration process obviously.
>
> This patch set is the QEMU side implementation.
>
> The virtio-balloon is extended so that QEMU can get the free pages
> information from the guest.
>
> After getting the free page bitmap, QEMU can use it to filter out
> guest's free pages. This make the live migration process much more
> efficient.
>
> In order to skip more free pages, we add an interface to let the user
> decide whether dropping the cache in guest during live migration.
Wouldn't that potentially break things like CoLo? Which side is the slow
one? We already do condense zero pages into small metadata in the
migration stream. Is writing them out the slow part? If so, just don't
write them in you're on a clean state.
Alex
>
> Performance data
> ================
>
> Test environment:
>
> CPU: Intel (R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
> Host RAM: 64GB
> Host Linux Kernel: 4.4.0 Host OS: CentOS 7.1
> Guest Linux Kernel: 4.5.rc6 Guest OS: CentOS 6.6
> Network: Intel X710 with 10 Gigabit connection
> Guest RAM: 8GB
>
> Case 1: Idle guest just boots:
> ==================================================
> | original | skip free page
> --------------------------------------------------
> total time(ms) | 1505 | 573
> --------------------------------------------------
> transferred ram(KB) | 399792 | 379057
> ==================================================
>
> Case 2: The guest has ever run some memory consuming workload, the
> workload is terminated before live migration.
> ==================================================
> | original | skip free page
> --------------------------------------------------
> total time(ms) | 10641 | 597
> --------------------------------------------------
> transferred ram(KB) | 8350829 | 389900
> ==================================================
>
> Case 3: The guest has ever built the linux kernel, the building is
> completed before live migration. The page cache is dropped
> during live migration in this case.
> ==================================================
> | original | skip free page
> --------------------------------------------------
> total time(ms) | 2636 | 914
> --------------------------------------------------
> transferred ram(KB) | 1316747 | 421980
> ==================================================
>
> Liang Li (5):
> bitmap: Add a new bitmap_move function
> kvm: Add two new arch specific functions
> virtio-balloon: Add a new feature to balloon device
> migration: filter out free pages during live migration
> migration: Add the interface for cache drop control
>
> balloon.c | 29 +++++++-
> hmp.c | 8 +++
> hw/virtio/virtio-balloon.c | 92 +++++++++++++++++++++++-
> include/hw/virtio/virtio-balloon.h | 30 +++++++-
> include/migration/migration.h | 1 +
> include/qemu/bitmap.h | 13 ++++
> include/standard-headers/linux/virtio_balloon.h | 1 +
> include/sysemu/balloon.h | 12 +++-
> include/sysemu/kvm.h | 2 +
> migration/migration.c | 31 +++++++-
> migration/ram.c | 95 +++++++++++++++++++++++++
> qapi-schema.json | 25 ++++++-
> target-arm/kvm.c | 14 ++++
> target-i386/kvm.c | 35 +++++++++
> target-mips/kvm.c | 14 ++++
> target-ppc/kvm.c | 14 ++++
> target-s390x/kvm.c | 14 ++++
> 17 files changed, 421 insertions(+), 9 deletions(-)
>
^ permalink raw reply [flat|nested] 19+ messages in thread