* [Qemu-devel] [PATCH 0/2] use blk_pwrite_zeroes for each zero cluster
@ 2017-04-06 13:15 jemmy858585
2017-04-06 13:15 ` [Qemu-devel] [PATCH 1/2] block: make bdrv_get_cluster_size public jemmy858585
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: jemmy858585 @ 2017-04-06 13:15 UTC (permalink / raw)
To: qemu-devel
Cc: stefanha, famz, kwolf, mreitz, qemu-block, quintela, dgilbert,
Lidong Chen
From: Lidong Chen <lidongchen@tencent.com>
BLOCK_SIZE is (1 << 20), qcow2 cluster size is 65536 by default,
this maybe cause the qcow2 file size is bigger after migration.
This patch check each cluster, use blk_pwrite_zeroes for each
zero cluster.
Lidong Chen (2):
block: make bdrv_get_cluster_size public
migration/block: use blk_pwrite_zeroes for each zero cluster
block/io.c | 2 +-
include/block/block.h | 1 +
migration/block.c | 34 ++++++++++++++++++++++++++++++++--
3 files changed, 34 insertions(+), 3 deletions(-)
--
1.8.3.1
^ permalink raw reply [flat|nested] 7+ messages in thread
* [Qemu-devel] [PATCH 1/2] block: make bdrv_get_cluster_size public
2017-04-06 13:15 [Qemu-devel] [PATCH 0/2] use blk_pwrite_zeroes for each zero cluster jemmy858585
@ 2017-04-06 13:15 ` jemmy858585
2017-04-06 13:15 ` [Qemu-devel] [PATCH 2/2] migration/block: use blk_pwrite_zeroes for each zero cluster jemmy858585
2017-04-07 5:22 ` [Qemu-devel] [PATCH 0/2] " 858585 jemmy
2 siblings, 0 replies; 7+ messages in thread
From: jemmy858585 @ 2017-04-06 13:15 UTC (permalink / raw)
To: qemu-devel
Cc: stefanha, famz, kwolf, mreitz, qemu-block, quintela, dgilbert,
Lidong Chen
From: Lidong Chen <lidongchen@tencent.com>
The bdrv_get_cluster_size function is need by block migration
to optimize zero cluster.
Signed-off-by: Lidong Chen <lidongchen@tencent.com>
---
block/io.c | 2 +-
include/block/block.h | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/block/io.c b/block/io.c
index 2709a70..1c31c2b 100644
--- a/block/io.c
+++ b/block/io.c
@@ -459,7 +459,7 @@ void bdrv_round_to_clusters(BlockDriverState *bs,
}
}
-static int bdrv_get_cluster_size(BlockDriverState *bs)
+int bdrv_get_cluster_size(BlockDriverState *bs)
{
BlockDriverInfo bdi;
int ret;
diff --git a/include/block/block.h b/include/block/block.h
index 5149260..a0d78fc 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -473,6 +473,7 @@ void bdrv_round_to_clusters(BlockDriverState *bs,
int64_t offset, unsigned int bytes,
int64_t *cluster_offset,
unsigned int *cluster_bytes);
+int bdrv_get_cluster_size(BlockDriverState *bs);
const char *bdrv_get_encrypted_filename(BlockDriverState *bs);
void bdrv_get_backing_filename(BlockDriverState *bs,
--
1.8.3.1
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [Qemu-devel] [PATCH 2/2] migration/block: use blk_pwrite_zeroes for each zero cluster
2017-04-06 13:15 [Qemu-devel] [PATCH 0/2] use blk_pwrite_zeroes for each zero cluster jemmy858585
2017-04-06 13:15 ` [Qemu-devel] [PATCH 1/2] block: make bdrv_get_cluster_size public jemmy858585
@ 2017-04-06 13:15 ` jemmy858585
2017-04-07 7:08 ` Fam Zheng
2017-04-07 5:22 ` [Qemu-devel] [PATCH 0/2] " 858585 jemmy
2 siblings, 1 reply; 7+ messages in thread
From: jemmy858585 @ 2017-04-06 13:15 UTC (permalink / raw)
To: qemu-devel
Cc: stefanha, famz, kwolf, mreitz, qemu-block, quintela, dgilbert,
Lidong Chen
From: Lidong Chen <lidongchen@tencent.com>
BLOCK_SIZE is (1 << 20), qcow2 cluster size is 65536 by default,
this maybe cause the qcow2 file size is bigger after migration.
This patch check each cluster, use blk_pwrite_zeroes for each
zero cluster.
Signed-off-by: Lidong Chen <lidongchen@tencent.com>
---
migration/block.c | 34 ++++++++++++++++++++++++++++++++--
1 file changed, 32 insertions(+), 2 deletions(-)
diff --git a/migration/block.c b/migration/block.c
index 7734ff7..1fce9b9 100644
--- a/migration/block.c
+++ b/migration/block.c
@@ -885,6 +885,10 @@ static int block_load(QEMUFile *f, void *opaque, int version_id)
int64_t total_sectors = 0;
int nr_sectors;
int ret;
+ int i;
+ int cluster_size;
+ int64_t addr_offset;
+ uint8_t *buf_offset;
do {
addr = qemu_get_be64(f);
@@ -934,8 +938,34 @@ static int block_load(QEMUFile *f, void *opaque, int version_id)
} else {
buf = g_malloc(BLOCK_SIZE);
qemu_get_buffer(f, buf, BLOCK_SIZE);
- ret = blk_pwrite(blk, addr * BDRV_SECTOR_SIZE, buf,
- nr_sectors * BDRV_SECTOR_SIZE, 0);
+
+ cluster_size = bdrv_get_cluster_size(blk_bs(blk));
+
+ if (cluster_size > 0) {
+ for (i = 0; i < BLOCK_SIZE / cluster_size; i++) {
+ addr_offset = addr * BDRV_SECTOR_SIZE
+ + i * cluster_size;
+ buf_offset = buf + i * cluster_size;
+
+ if (buffer_is_zero(buf_offset, cluster_size)) {
+ ret = blk_pwrite_zeroes(blk, addr_offset,
+ cluster_size,
+ BDRV_REQ_MAY_UNMAP);
+ } else {
+ ret = blk_pwrite(blk, addr_offset,
+ buf_offset, cluster_size, 0);
+ }
+
+ if (ret < 0) {
+ g_free(buf);
+ return ret;
+ }
+ }
+ } else {
+ ret = blk_pwrite(blk, addr * BDRV_SECTOR_SIZE, buf,
+ nr_sectors * BDRV_SECTOR_SIZE, 0);
+ }
+
g_free(buf);
}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] [PATCH 0/2] use blk_pwrite_zeroes for each zero cluster
2017-04-06 13:15 [Qemu-devel] [PATCH 0/2] use blk_pwrite_zeroes for each zero cluster jemmy858585
2017-04-06 13:15 ` [Qemu-devel] [PATCH 1/2] block: make bdrv_get_cluster_size public jemmy858585
2017-04-06 13:15 ` [Qemu-devel] [PATCH 2/2] migration/block: use blk_pwrite_zeroes for each zero cluster jemmy858585
@ 2017-04-07 5:22 ` 858585 jemmy
2017-04-07 6:55 ` 858585 jemmy
2 siblings, 1 reply; 7+ messages in thread
From: 858585 jemmy @ 2017-04-07 5:22 UTC (permalink / raw)
To: qemu-devel
Cc: Stefan Hajnoczi, Fam Zheng, kwolf, mreitz, qemu-block, quintela,
dgilbert, Lidong Chen
the test result for this patch:
the migration command :
virsh migrate --live 165cf436-312f-47e7-90f2-f8aa63f34893
--copy-storage-all qemu+ssh://10.59.163.38/system
the qemu-img info on source host:
qemu-img info /instanceimage/165cf436-312f-47e7-90f2-f8aa63f34893/165cf436-312f-47e7-90f2-f8aa63f34893_vda.qcow2
image: /instanceimage/165cf436-312f-47e7-90f2-f8aa63f34893/165cf436-312f-47e7-90f2-f8aa63f34893_vda.qcow2
file format: qcow2
virtual size: 1.0T (1095216660480 bytes)
disk size: 1.5G (1638989824 bytes)
cluster_size: 65536
backing file: /baseimage/img2016042213665396/img2016042213665396.qcow2
the qemu-img info on dest host(before apply patch):
qemu-img info /instanceimage/165cf436-312f-47e7-90f2-f8aa63f34893/165cf436-312f-47e7-90f2-f8aa63f34893_vda.qcow2
image: /instanceimage/165cf436-312f-47e7-90f2-f8aa63f34893/165cf436-312f-47e7-90f2-f8aa63f34893_vda.qcow2
file format: qcow2
virtual size: 40G (42949672960 bytes)
disk size: 4.1G (4423286784 bytes)
cluster_size: 65536
backing file: /baseimage/img2016042213665396/img2016042213665396.qcow2
the qemu-img info on dest host(after apply patch):
qemu-img info /instanceimage/165cf436-312f-47e7-90f2-f8aa63f34893/165cf436-312f-47e7-90f2-f8aa63f34893_vda.qcow2
image: /instanceimage/165cf436-312f-47e7-90f2-f8aa63f34893/165cf436-312f-47e7-90f2-f8aa63f34893_vda.qcow2
file format: qcow2
virtual size: 40G (42949672960 bytes)
disk size: 2.3G (2496200704 bytes)
cluster_size: 65536
backing file: /baseimage/img2016042213665396/img2016042213665396.qcow2
the disk size reduce from 4.1G to 2.3G.
On Thu, Apr 6, 2017 at 9:15 PM, <jemmy858585@gmail.com> wrote:
> From: Lidong Chen <lidongchen@tencent.com>
>
> BLOCK_SIZE is (1 << 20), qcow2 cluster size is 65536 by default,
> this maybe cause the qcow2 file size is bigger after migration.
> This patch check each cluster, use blk_pwrite_zeroes for each
> zero cluster.
>
> Lidong Chen (2):
> block: make bdrv_get_cluster_size public
> migration/block: use blk_pwrite_zeroes for each zero cluster
>
> block/io.c | 2 +-
> include/block/block.h | 1 +
> migration/block.c | 34 ++++++++++++++++++++++++++++++++--
> 3 files changed, 34 insertions(+), 3 deletions(-)
>
> --
> 1.8.3.1
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] [PATCH 0/2] use blk_pwrite_zeroes for each zero cluster
2017-04-07 5:22 ` [Qemu-devel] [PATCH 0/2] " 858585 jemmy
@ 2017-04-07 6:55 ` 858585 jemmy
0 siblings, 0 replies; 7+ messages in thread
From: 858585 jemmy @ 2017-04-07 6:55 UTC (permalink / raw)
To: qemu-devel
Cc: Stefan Hajnoczi, Fam Zheng, kwolf, mreitz, qemu-block, quintela,
dgilbert, Lidong Chen
On Fri, Apr 7, 2017 at 1:22 PM, 858585 jemmy <jemmy858585@gmail.com> wrote:
> the test result for this patch:
>
> the migration command :
> virsh migrate --live 165cf436-312f-47e7-90f2-f8aa63f34893
> --copy-storage-all qemu+ssh://10.59.163.38/system
>
> the qemu-img info on source host:
> qemu-img info /instanceimage/165cf436-312f-47e7-90f2-f8aa63f34893/165cf436-312f-47e7-90f2-f8aa63f34893_vda.qcow2
> image: /instanceimage/165cf436-312f-47e7-90f2-f8aa63f34893/165cf436-312f-47e7-90f2-f8aa63f34893_vda.qcow2
> file format: qcow2
> virtual size: 1.0T (1095216660480 bytes)
> disk size: 1.5G (1638989824 bytes)
> cluster_size: 65536
> backing file: /baseimage/img2016042213665396/img2016042213665396.qcow2
>
> the qemu-img info on dest host(before apply patch):
> qemu-img info /instanceimage/165cf436-312f-47e7-90f2-f8aa63f34893/165cf436-312f-47e7-90f2-f8aa63f34893_vda.qcow2
> image: /instanceimage/165cf436-312f-47e7-90f2-f8aa63f34893/165cf436-312f-47e7-90f2-f8aa63f34893_vda.qcow2
> file format: qcow2
> virtual size: 40G (42949672960 bytes)
> disk size: 4.1G (4423286784 bytes)
> cluster_size: 65536
> backing file: /baseimage/img2016042213665396/img2016042213665396.qcow2
>
> the qemu-img info on dest host(after apply patch):
> qemu-img info /instanceimage/165cf436-312f-47e7-90f2-f8aa63f34893/165cf436-312f-47e7-90f2-f8aa63f34893_vda.qcow2
> image: /instanceimage/165cf436-312f-47e7-90f2-f8aa63f34893/165cf436-312f-47e7-90f2-f8aa63f34893_vda.qcow2
> file format: qcow2
> virtual size: 40G (42949672960 bytes)
> disk size: 2.3G (2496200704 bytes)
> cluster_size: 65536
> backing file: /baseimage/img2016042213665396/img2016042213665396.qcow2
>
> the disk size reduce from 4.1G to 2.3G.
>
I find a bug for my patch.
unfortunately, when use raw format , bdrv_get_cluster_size return 1.
the raw format cluster size is 1.
and will reduce migration speed.
bdrv_get_cluster_size return bs->bl.request_alignment when cluster_size is zero.
and bs->bl.request_alignment is 1 when use raw format.
static int bdrv_get_cluster_size(BlockDriverState *bs)
{
BlockDriverInfo bdi;
int ret;
ret = bdrv_get_info(bs, &bdi);
if (ret < 0 || bdi.cluster_size == 0) {
return bs->bl.request_alignment;
} else {
return bdi.cluster_size;
}
}
so i will change to use bdrv_get_info function to get cluster_size.
and bdrv_get_cluster_size is not need to be public.
>
> On Thu, Apr 6, 2017 at 9:15 PM, <jemmy858585@gmail.com> wrote:
>> From: Lidong Chen <lidongchen@tencent.com>
>>
>> BLOCK_SIZE is (1 << 20), qcow2 cluster size is 65536 by default,
>> this maybe cause the qcow2 file size is bigger after migration.
>> This patch check each cluster, use blk_pwrite_zeroes for each
>> zero cluster.
>>
>> Lidong Chen (2):
>> block: make bdrv_get_cluster_size public
>> migration/block: use blk_pwrite_zeroes for each zero cluster
>>
>> block/io.c | 2 +-
>> include/block/block.h | 1 +
>> migration/block.c | 34 ++++++++++++++++++++++++++++++++--
>> 3 files changed, 34 insertions(+), 3 deletions(-)
>>
>> --
>> 1.8.3.1
>>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] [PATCH 2/2] migration/block: use blk_pwrite_zeroes for each zero cluster
2017-04-06 13:15 ` [Qemu-devel] [PATCH 2/2] migration/block: use blk_pwrite_zeroes for each zero cluster jemmy858585
@ 2017-04-07 7:08 ` Fam Zheng
2017-04-07 7:15 ` 858585 jemmy
0 siblings, 1 reply; 7+ messages in thread
From: Fam Zheng @ 2017-04-07 7:08 UTC (permalink / raw)
To: jemmy858585
Cc: qemu-devel, stefanha, kwolf, mreitz, qemu-block, quintela,
dgilbert, Lidong Chen
On Thu, 04/06 21:15, jemmy858585@gmail.com wrote:
> From: Lidong Chen <lidongchen@tencent.com>
>
> BLOCK_SIZE is (1 << 20), qcow2 cluster size is 65536 by default,
> this maybe cause the qcow2 file size is bigger after migration.
> This patch check each cluster, use blk_pwrite_zeroes for each
> zero cluster.
>
> Signed-off-by: Lidong Chen <lidongchen@tencent.com>
> ---
> migration/block.c | 34 ++++++++++++++++++++++++++++++++--
> 1 file changed, 32 insertions(+), 2 deletions(-)
>
> diff --git a/migration/block.c b/migration/block.c
> index 7734ff7..1fce9b9 100644
> --- a/migration/block.c
> +++ b/migration/block.c
> @@ -885,6 +885,10 @@ static int block_load(QEMUFile *f, void *opaque, int version_id)
> int64_t total_sectors = 0;
> int nr_sectors;
> int ret;
> + int i;
> + int cluster_size;
> + int64_t addr_offset;
> + uint8_t *buf_offset;
>
> do {
> addr = qemu_get_be64(f);
> @@ -934,8 +938,34 @@ static int block_load(QEMUFile *f, void *opaque, int version_id)
> } else {
> buf = g_malloc(BLOCK_SIZE);
> qemu_get_buffer(f, buf, BLOCK_SIZE);
> - ret = blk_pwrite(blk, addr * BDRV_SECTOR_SIZE, buf,
> - nr_sectors * BDRV_SECTOR_SIZE, 0);
> +
> + cluster_size = bdrv_get_cluster_size(blk_bs(blk));
> +
> + if (cluster_size > 0) {
> + for (i = 0; i < BLOCK_SIZE / cluster_size; i++) {
Should we check that cluster_size < BLOCK_SIZE and (BLOCK_SIZE % cluster_size ==
0)?
Fam
> + addr_offset = addr * BDRV_SECTOR_SIZE
> + + i * cluster_size;
> + buf_offset = buf + i * cluster_size;
> +
> + if (buffer_is_zero(buf_offset, cluster_size)) {
> + ret = blk_pwrite_zeroes(blk, addr_offset,
> + cluster_size,
> + BDRV_REQ_MAY_UNMAP);
> + } else {
> + ret = blk_pwrite(blk, addr_offset,
> + buf_offset, cluster_size, 0);
> + }
> +
> + if (ret < 0) {
> + g_free(buf);
> + return ret;
> + }
> + }
> + } else {
> + ret = blk_pwrite(blk, addr * BDRV_SECTOR_SIZE, buf,
> + nr_sectors * BDRV_SECTOR_SIZE, 0);
> + }
> +
> g_free(buf);
> }
>
> --
> 1.8.3.1
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] [PATCH 2/2] migration/block: use blk_pwrite_zeroes for each zero cluster
2017-04-07 7:08 ` Fam Zheng
@ 2017-04-07 7:15 ` 858585 jemmy
0 siblings, 0 replies; 7+ messages in thread
From: 858585 jemmy @ 2017-04-07 7:15 UTC (permalink / raw)
To: Fam Zheng
Cc: qemu-devel, Stefan Hajnoczi, kwolf, mreitz, qemu-block, quintela,
dgilbert, Lidong Chen
On Fri, Apr 7, 2017 at 3:08 PM, Fam Zheng <famz@redhat.com> wrote:
> On Thu, 04/06 21:15, jemmy858585@gmail.com wrote:
>> From: Lidong Chen <lidongchen@tencent.com>
>>
>> BLOCK_SIZE is (1 << 20), qcow2 cluster size is 65536 by default,
>> this maybe cause the qcow2 file size is bigger after migration.
>> This patch check each cluster, use blk_pwrite_zeroes for each
>> zero cluster.
>>
>> Signed-off-by: Lidong Chen <lidongchen@tencent.com>
>> ---
>> migration/block.c | 34 ++++++++++++++++++++++++++++++++--
>> 1 file changed, 32 insertions(+), 2 deletions(-)
>>
>> diff --git a/migration/block.c b/migration/block.c
>> index 7734ff7..1fce9b9 100644
>> --- a/migration/block.c
>> +++ b/migration/block.c
>> @@ -885,6 +885,10 @@ static int block_load(QEMUFile *f, void *opaque, int version_id)
>> int64_t total_sectors = 0;
>> int nr_sectors;
>> int ret;
>> + int i;
>> + int cluster_size;
>> + int64_t addr_offset;
>> + uint8_t *buf_offset;
>>
>> do {
>> addr = qemu_get_be64(f);
>> @@ -934,8 +938,34 @@ static int block_load(QEMUFile *f, void *opaque, int version_id)
>> } else {
>> buf = g_malloc(BLOCK_SIZE);
>> qemu_get_buffer(f, buf, BLOCK_SIZE);
>> - ret = blk_pwrite(blk, addr * BDRV_SECTOR_SIZE, buf,
>> - nr_sectors * BDRV_SECTOR_SIZE, 0);
>> +
>> + cluster_size = bdrv_get_cluster_size(blk_bs(blk));
>> +
>> + if (cluster_size > 0) {
>> + for (i = 0; i < BLOCK_SIZE / cluster_size; i++) {
>
> Should we check that cluster_size < BLOCK_SIZE and (BLOCK_SIZE % cluster_size ==
> 0)?
I think this is necessary.
Thanks.
>
> Fam
>
>> + addr_offset = addr * BDRV_SECTOR_SIZE
>> + + i * cluster_size;
>> + buf_offset = buf + i * cluster_size;
>> +
>> + if (buffer_is_zero(buf_offset, cluster_size)) {
>> + ret = blk_pwrite_zeroes(blk, addr_offset,
>> + cluster_size,
>> + BDRV_REQ_MAY_UNMAP);
>> + } else {
>> + ret = blk_pwrite(blk, addr_offset,
>> + buf_offset, cluster_size, 0);
>> + }
>> +
>> + if (ret < 0) {
>> + g_free(buf);
>> + return ret;
>> + }
>> + }
>> + } else {
>> + ret = blk_pwrite(blk, addr * BDRV_SECTOR_SIZE, buf,
>> + nr_sectors * BDRV_SECTOR_SIZE, 0);
>> + }
>> +
>> g_free(buf);
>> }
>>
>> --
>> 1.8.3.1
>>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-04-07 7:15 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-04-06 13:15 [Qemu-devel] [PATCH 0/2] use blk_pwrite_zeroes for each zero cluster jemmy858585
2017-04-06 13:15 ` [Qemu-devel] [PATCH 1/2] block: make bdrv_get_cluster_size public jemmy858585
2017-04-06 13:15 ` [Qemu-devel] [PATCH 2/2] migration/block: use blk_pwrite_zeroes for each zero cluster jemmy858585
2017-04-07 7:08 ` Fam Zheng
2017-04-07 7:15 ` 858585 jemmy
2017-04-07 5:22 ` [Qemu-devel] [PATCH 0/2] " 858585 jemmy
2017-04-07 6:55 ` 858585 jemmy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.