qemu drive-mirror to rbd storage : no sparse rbd image

qemu drive-mirror to rbd storage : no sparse rbd image

[Alexandre DERUMIER]

Hi,

I'm currently planning to migrate our storage to ceph/rbd through qemu drive-mirror

and It seem that drive-mirror with rbd block driver, don't create a sparse image. (all zeros are copied to the target rbd).

Also note, that it's working fine with "qemu-img convert" , the rbd volume is sparse after conversion.


Could it be related to the "bdrv_co_write_zeroes" missing features in block/rbd.c ?

(It's available in other block drivers (scsi,gluster,raw-aio) , and I don't have this problem with theses block drivers).



Regards,

Alexandre Derumier

[Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Alexandre DERUMIER]

Hi,

I'm currently planning to migrate our storage to ceph/rbd through qemu drive-mirror

and It seem that drive-mirror with rbd block driver, don't create a sparse image. (all zeros are copied to the target rbd).

Also note, that it's working fine with "qemu-img convert" , the rbd volume is sparse after conversion.


Could it be related to the "bdrv_co_write_zeroes" missing features in block/rbd.c ?

(It's available in other block drivers (scsi,gluster,raw-aio) , and I don't have this problem with theses block drivers).



Regards,

Alexandre Derumier

Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Fam Zheng]

On Wed, 10/08 13:15, Alexandre DERUMIER wrote:
> Hi,
> 
> I'm currently planning to migrate our storage to ceph/rbd through qemu drive-mirror
> 
> and It seem that drive-mirror with rbd block driver, don't create a sparse image. (all zeros are copied to the target rbd).
> 
> Also note, that it's working fine with "qemu-img convert" , the rbd volume is sparse after conversion.

What is the source format? If the zero clusters are actually unallocated in the
source image, drive-mirror will not write those clusters either. I.e. with
"drive-mirror sync=top", both source and target should have the same "qemu-img
map" output.

Fam

> 
> 
> Could it be related to the "bdrv_co_write_zeroes" missing features in block/rbd.c ?
> 
> (It's available in other block drivers (scsi,gluster,raw-aio) , and I don't have this problem with theses block drivers).
> 
> 
> 
> Regards,
> 
> Alexandre Derumier
> 
> 
> 

Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Fam Zheng]

On Wed, 10/08 13:15, Alexandre DERUMIER wrote:
> Hi,
> 
> I'm currently planning to migrate our storage to ceph/rbd through qemu drive-mirror
> 
> and It seem that drive-mirror with rbd block driver, don't create a sparse image. (all zeros are copied to the target rbd).
> 
> Also note, that it's working fine with "qemu-img convert" , the rbd volume is sparse after conversion.

What is the source format? If the zero clusters are actually unallocated in the
source image, drive-mirror will not write those clusters either. I.e. with
"drive-mirror sync=top", both source and target should have the same "qemu-img
map" output.

Fam

> 
> 
> Could it be related to the "bdrv_co_write_zeroes" missing features in block/rbd.c ?
> 
> (It's available in other block drivers (scsi,gluster,raw-aio) , and I don't have this problem with theses block drivers).
> 
> 
> 
> Regards,
> 
> Alexandre Derumier
> 
> 
> 

Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Alexandre DERUMIER]

>>What is the source format? If the zero clusters are actually unallocated in the
>>source image, drive-mirror will not write those clusters either. I.e. with
>>"drive-mirror sync=top", both source and target should have the same "qemu-img
>>map" output.

Thanks for your reply,

I had tried drive mirror (sync=full) with

raw file (sparse) -> rbd  (no sparse)
rbd (sparse) -> rbd (no sparse)
raw file (sparse) -> qcow2 on ext4  (sparse)
rbd (sparse) -> raw on ext4 (sparse)

Also I see that I have the same problem with target file format on xfs.

raw file (sparse) -> qcow2 on xfs  (no sparse)
rbd (sparse) -> raw on xfs (no sparse)


I only have this problem with drive-mirror, qemu-img convert seem to simply skip zero blocks.


Or maybe this is because I'm using sync=full ?

What is the difference between full and top ?

""sync": what parts of the disk image should be copied to the destination;
  possibilities include "full" for all the disk, "top" for only the sectors
  allocated in the topmost image".

(what is topmost image ?)


----- Mail original ----- 

De: "Fam Zheng" <famz@redhat.com> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: "qemu-devel" <qemu-devel@nongnu.org>, "Ceph Devel" <ceph-devel@vger.kernel.org> 
Envoyé: Samedi 11 Octobre 2014 09:01:18 
Objet: Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image 

On Wed, 10/08 13:15, Alexandre DERUMIER wrote: 
> Hi, 
> 
> I'm currently planning to migrate our storage to ceph/rbd through qemu drive-mirror 
> 
> and It seem that drive-mirror with rbd block driver, don't create a sparse image. (all zeros are copied to the target rbd). 
> 
> Also note, that it's working fine with "qemu-img convert" , the rbd volume is sparse after conversion. 

What is the source format? If the zero clusters are actually unallocated in the 
source image, drive-mirror will not write those clusters either. I.e. with 
"drive-mirror sync=top", both source and target should have the same "qemu-img 
map" output. 

Fam 

> 
> 
> Could it be related to the "bdrv_co_write_zeroes" missing features in block/rbd.c ? 
> 
> (It's available in other block drivers (scsi,gluster,raw-aio) , and I don't have this problem with theses block drivers). 
> 
> 
> 
> Regards, 
> 
> Alexandre Derumier 
> 
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Alexandre DERUMIER]

>>What is the source format? If the zero clusters are actually unallocated in the
>>source image, drive-mirror will not write those clusters either. I.e. with
>>"drive-mirror sync=top", both source and target should have the same "qemu-img
>>map" output.

Thanks for your reply,

I had tried drive mirror (sync=full) with

raw file (sparse) -> rbd  (no sparse)
rbd (sparse) -> rbd (no sparse)
raw file (sparse) -> qcow2 on ext4  (sparse)
rbd (sparse) -> raw on ext4 (sparse)

Also I see that I have the same problem with target file format on xfs.

raw file (sparse) -> qcow2 on xfs  (no sparse)
rbd (sparse) -> raw on xfs (no sparse)


I only have this problem with drive-mirror, qemu-img convert seem to simply skip zero blocks.


Or maybe this is because I'm using sync=full ?

What is the difference between full and top ?

""sync": what parts of the disk image should be copied to the destination;
  possibilities include "full" for all the disk, "top" for only the sectors
  allocated in the topmost image".

(what is topmost image ?)


----- Mail original -----

De: "Fam Zheng" <famz@redhat.com>
À: "Alexandre DERUMIER" <aderumier@odiso.com>
Cc: "qemu-devel" <qemu-devel@nongnu.org>, "Ceph Devel" <ceph-devel@vger.kernel.org>
Envoyé: Samedi 11 Octobre 2014 09:01:18
Objet: Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

On Wed, 10/08 13:15, Alexandre DERUMIER wrote:
> Hi,
>
> I'm currently planning to migrate our storage to ceph/rbd through qemu drive-mirror
>
> and It seem that drive-mirror with rbd block driver, don't create a sparse image. (all zeros are copied to the target rbd).
>
> Also note, that it's working fine with "qemu-img convert" , the rbd volume is sparse after conversion.

What is the source format? If the zero clusters are actually unallocated in the
source image, drive-mirror will not write those clusters either. I.e. with 
"drive-mirror sync=top", both source and target should have the same "qemu-img
map" output.

Fam

>
>
> Could it be related to the "bdrv_co_write_zeroes" missing features in block/rbd.c ?
>
> (It's available in other block drivers (scsi,gluster,raw-aio) , and I don't have this problem with theses block drivers).
>
>
>
> Regards,
>
> Alexandre Derumier
>
>
>

Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Fam Zheng]

On Sat, 10/11 10:00, Alexandre DERUMIER wrote:
> >>What is the source format? If the zero clusters are actually unallocated in the
> >>source image, drive-mirror will not write those clusters either. I.e. with
> >>"drive-mirror sync=top", both source and target should have the same "qemu-img
> >>map" output.
> 
> Thanks for your reply,
> 
> I had tried drive mirror (sync=full) with
> 
> raw file (sparse) -> rbd  (no sparse)
> rbd (sparse) -> rbd (no sparse)
> raw file (sparse) -> qcow2 on ext4  (sparse)
> rbd (sparse) -> raw on ext4 (sparse)
> 
> Also I see that I have the same problem with target file format on xfs.
> 
> raw file (sparse) -> qcow2 on xfs  (no sparse)
> rbd (sparse) -> raw on xfs (no sparse)
> 

These don't tell me much. Maybe it's better to show the actual commands and how
you tell sparse from no sparse?

Does "qcow2 -> qcow2" work for you on xfs?

> 
> I only have this problem with drive-mirror, qemu-img convert seem to simply skip zero blocks.
> 
> 
> Or maybe this is because I'm using sync=full ?
> 
> What is the difference between full and top ?
> 
> ""sync": what parts of the disk image should be copied to the destination;
>   possibilities include "full" for all the disk, "top" for only the sectors
>   allocated in the topmost image".
> 
> (what is topmost image ?)

For "sync=top", only the clusters allocated in the image itself is copied; for
"full", all those clusters allocated in the image itself, and its backing
image, and it's backing's backing image, ..., are copied.

The image itself, having a backing image or not, is called the topmost image.

Fam

Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Fam Zheng]

On Sat, 10/11 10:00, Alexandre DERUMIER wrote:
> >>What is the source format? If the zero clusters are actually unallocated in the
> >>source image, drive-mirror will not write those clusters either. I.e. with
> >>"drive-mirror sync=top", both source and target should have the same "qemu-img
> >>map" output.
> 
> Thanks for your reply,
> 
> I had tried drive mirror (sync=full) with
> 
> raw file (sparse) -> rbd  (no sparse)
> rbd (sparse) -> rbd (no sparse)
> raw file (sparse) -> qcow2 on ext4  (sparse)
> rbd (sparse) -> raw on ext4 (sparse)
> 
> Also I see that I have the same problem with target file format on xfs.
> 
> raw file (sparse) -> qcow2 on xfs  (no sparse)
> rbd (sparse) -> raw on xfs (no sparse)
> 

These don't tell me much. Maybe it's better to show the actual commands and how
you tell sparse from no sparse?

Does "qcow2 -> qcow2" work for you on xfs?

> 
> I only have this problem with drive-mirror, qemu-img convert seem to simply skip zero blocks.
> 
> 
> Or maybe this is because I'm using sync=full ?
> 
> What is the difference between full and top ?
> 
> ""sync": what parts of the disk image should be copied to the destination;
>   possibilities include "full" for all the disk, "top" for only the sectors
>   allocated in the topmost image".
> 
> (what is topmost image ?)

For "sync=top", only the clusters allocated in the image itself is copied; for
"full", all those clusters allocated in the image itself, and its backing
image, and it's backing's backing image, ..., are copied.

The image itself, having a backing image or not, is called the topmost image.

Fam

Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Andrey Korolyov]

On Sat, Oct 11, 2014 at 12:25 PM, Fam Zheng <famz@redhat.com> wrote:
> On Sat, 10/11 10:00, Alexandre DERUMIER wrote:
>> >>What is the source format? If the zero clusters are actually unallocated in the
>> >>source image, drive-mirror will not write those clusters either. I.e. with
>> >>"drive-mirror sync=top", both source and target should have the same "qemu-img
>> >>map" output.
>>
>> Thanks for your reply,
>>
>> I had tried drive mirror (sync=full) with
>>
>> raw file (sparse) -> rbd  (no sparse)
>> rbd (sparse) -> rbd (no sparse)
>> raw file (sparse) -> qcow2 on ext4  (sparse)
>> rbd (sparse) -> raw on ext4 (sparse)
>>
>> Also I see that I have the same problem with target file format on xfs.
>>
>> raw file (sparse) -> qcow2 on xfs  (no sparse)
>> rbd (sparse) -> raw on xfs (no sparse)
>>
>
> These don't tell me much. Maybe it's better to show the actual commands and how
> you tell sparse from no sparse?
>
> Does "qcow2 -> qcow2" work for you on xfs?
>
>>
>> I only have this problem with drive-mirror, qemu-img convert seem to simply skip zero blocks.
>>
>>
>> Or maybe this is because I'm using sync=full ?
>>
>> What is the difference between full and top ?
>>
>> ""sync": what parts of the disk image should be copied to the destination;
>>   possibilities include "full" for all the disk, "top" for only the sectors
>>   allocated in the topmost image".
>>
>> (what is topmost image ?)
>
> For "sync=top", only the clusters allocated in the image itself is copied; for
> "full", all those clusters allocated in the image itself, and its backing
> image, and it's backing's backing image, ..., are copied.
>
> The image itself, having a backing image or not, is called the topmost image.
>
> Fam
> --

Just a wild guess - Alexandre, did you tried detect-zeroes blk option
for mirroring targets?

Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Andrey Korolyov]

On Sat, Oct 11, 2014 at 12:25 PM, Fam Zheng <famz@redhat.com> wrote:
> On Sat, 10/11 10:00, Alexandre DERUMIER wrote:
>> >>What is the source format? If the zero clusters are actually unallocated in the
>> >>source image, drive-mirror will not write those clusters either. I.e. with
>> >>"drive-mirror sync=top", both source and target should have the same "qemu-img
>> >>map" output.
>>
>> Thanks for your reply,
>>
>> I had tried drive mirror (sync=full) with
>>
>> raw file (sparse) -> rbd  (no sparse)
>> rbd (sparse) -> rbd (no sparse)
>> raw file (sparse) -> qcow2 on ext4  (sparse)
>> rbd (sparse) -> raw on ext4 (sparse)
>>
>> Also I see that I have the same problem with target file format on xfs.
>>
>> raw file (sparse) -> qcow2 on xfs  (no sparse)
>> rbd (sparse) -> raw on xfs (no sparse)
>>
>
> These don't tell me much. Maybe it's better to show the actual commands and how
> you tell sparse from no sparse?
>
> Does "qcow2 -> qcow2" work for you on xfs?
>
>>
>> I only have this problem with drive-mirror, qemu-img convert seem to simply skip zero blocks.
>>
>>
>> Or maybe this is because I'm using sync=full ?
>>
>> What is the difference between full and top ?
>>
>> ""sync": what parts of the disk image should be copied to the destination;
>>   possibilities include "full" for all the disk, "top" for only the sectors
>>   allocated in the topmost image".
>>
>> (what is topmost image ?)
>
> For "sync=top", only the clusters allocated in the image itself is copied; for
> "full", all those clusters allocated in the image itself, and its backing
> image, and it's backing's backing image, ..., are copied.
>
> The image itself, having a backing image or not, is called the topmost image.
>
> Fam
> --

Just a wild guess - Alexandre, did you tried detect-zeroes blk option
for mirroring targets?

Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Alexandre DERUMIER]

>>Just a wild guess - Alexandre, did you tried detect-zeroes blk option 
>>for mirroring targets? 

Hi, yes, I have also tried with detect-zeroes (on or discard) with virtio and virtio-scsi, doesn't help. (I'm not sure that is implemtend in drive-mirror).

As workaround currently, after drive-mirror, I doing fstrim inside the guest (with virtio-scsi + discard), 
and like this I can free space on rbd storage.



----- Mail original ----- 

De: "Andrey Korolyov" <andrey@xdel.ru> 
À: "Fam Zheng" <famz@redhat.com> 
Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, "qemu-devel" <qemu-devel@nongnu.org>, "Ceph Devel" <ceph-devel@vger.kernel.org> 
Envoyé: Samedi 11 Octobre 2014 10:30:40 
Objet: Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image 

On Sat, Oct 11, 2014 at 12:25 PM, Fam Zheng <famz@redhat.com> wrote: 
> On Sat, 10/11 10:00, Alexandre DERUMIER wrote: 
>> >>What is the source format? If the zero clusters are actually unallocated in the 
>> >>source image, drive-mirror will not write those clusters either. I.e. with 
>> >>"drive-mirror sync=top", both source and target should have the same "qemu-img 
>> >>map" output. 
>> 
>> Thanks for your reply, 
>> 
>> I had tried drive mirror (sync=full) with 
>> 
>> raw file (sparse) -> rbd (no sparse) 
>> rbd (sparse) -> rbd (no sparse) 
>> raw file (sparse) -> qcow2 on ext4 (sparse) 
>> rbd (sparse) -> raw on ext4 (sparse) 
>> 
>> Also I see that I have the same problem with target file format on xfs. 
>> 
>> raw file (sparse) -> qcow2 on xfs (no sparse) 
>> rbd (sparse) -> raw on xfs (no sparse) 
>> 
> 
> These don't tell me much. Maybe it's better to show the actual commands and how 
> you tell sparse from no sparse? 
> 
> Does "qcow2 -> qcow2" work for you on xfs? 
> 
>> 
>> I only have this problem with drive-mirror, qemu-img convert seem to simply skip zero blocks. 
>> 
>> 
>> Or maybe this is because I'm using sync=full ? 
>> 
>> What is the difference between full and top ? 
>> 
>> ""sync": what parts of the disk image should be copied to the destination; 
>> possibilities include "full" for all the disk, "top" for only the sectors 
>> allocated in the topmost image". 
>> 
>> (what is topmost image ?) 
> 
> For "sync=top", only the clusters allocated in the image itself is copied; for 
> "full", all those clusters allocated in the image itself, and its backing 
> image, and it's backing's backing image, ..., are copied. 
> 
> The image itself, having a backing image or not, is called the topmost image. 
> 
> Fam 
> -- 

Just a wild guess - Alexandre, did you tried detect-zeroes blk option 
for mirroring targets? 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Alexandre DERUMIER]

>>Just a wild guess - Alexandre, did you tried detect-zeroes blk option
>>for mirroring targets?

Hi, yes, I have also tried with detect-zeroes (on or discard) with virtio and virtio-scsi, doesn't help. (I'm not sure that is implemtend in drive-mirror).

As workaround currently, after drive-mirror, I doing fstrim inside the guest (with virtio-scsi + discard),
and like this I can free space on rbd storage.



----- Mail original -----

De: "Andrey Korolyov" <andrey@xdel.ru>
À: "Fam Zheng" <famz@redhat.com>
Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, "qemu-devel" <qemu-devel@nongnu.org>, "Ceph Devel" <ceph-devel@vger.kernel.org>
Envoyé: Samedi 11 Octobre 2014 10:30:40
Objet: Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

On Sat, Oct 11, 2014 at 12:25 PM, Fam Zheng <famz@redhat.com> wrote:
> On Sat, 10/11 10:00, Alexandre DERUMIER wrote:
>> >>What is the source format? If the zero clusters are actually unallocated in the
>> >>source image, drive-mirror will not write those clusters either. I.e. with
>> >>"drive-mirror sync=top", both source and target should have the same "qemu-img
>> >>map" output.
>>
>> Thanks for your reply,
>>
>> I had tried drive mirror (sync=full) with
>>
>> raw file (sparse) -> rbd (no sparse)
>> rbd (sparse) -> rbd (no sparse)
>> raw file (sparse) -> qcow2 on ext4 (sparse)
>> rbd (sparse) -> raw on ext4 (sparse)
>>
>> Also I see that I have the same problem with target file format on xfs. 
>>
>> raw file (sparse) -> qcow2 on xfs (no sparse)
>> rbd (sparse) -> raw on xfs (no sparse)
>>
>
> These don't tell me much. Maybe it's better to show the actual commands and how
> you tell sparse from no sparse?
>
> Does "qcow2 -> qcow2" work for you on xfs?
>
>>
>> I only have this problem with drive-mirror, qemu-img convert seem to simply skip zero blocks.
>>
>>
>> Or maybe this is because I'm using sync=full ?
>>
>> What is the difference between full and top ?
>>
>> ""sync": what parts of the disk image should be copied to the destination;
>> possibilities include "full" for all the disk, "top" for only the sectors
>> allocated in the topmost image".
>>
>> (what is topmost image ?)
>
> For "sync=top", only the clusters allocated in the image itself is copied; for
> "full", all those clusters allocated in the image itself, and its backing
> image, and it's backing's backing image, ..., are copied.
>
> The image itself, having a backing image or not, is called the topmost image.
>
> Fam
> --

Just a wild guess - Alexandre, did you tried detect-zeroes blk option
for mirroring targets?

Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Alexandre DERUMIER]

>>These don't tell me much. Maybe it's better to show the actual commands and how 
>>you tell sparse from no sparse? 


Well,

I create 2 empty source images files of 10G. (source.qcow2 and source.raw)

then:

du -sh source.qcow2  : 2M
du -sh source.raw  : 0M


then I convert them with qemu-img convert : (source.qcow2 -> target.qcow2 , source.raw -> target.raw)

du -sh target.qcow2  : 2M
du -sh target.raw  : 0M

(So it's ok here)

But If I convert them with drive-mirror:

du -sh target.qcow2  : 11G
du -sh target.raw  : 11G



I have also double check with #df, and I see space allocated on filesystem when using drive-mirror.

I have the same behavior if target is a rbd storage.

Also I have done test again ext3,ext4, and I see the same problem than with xfs. 


I'm pretty sure that drive-mirror copy zero block, qemu-img convert take around 2s to convert the empty file
(because it's skipping zero block), and drive mirror take around 5min.

----- Mail original ----- 

De: "Fam Zheng" <famz@redhat.com> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: "qemu-devel" <qemu-devel@nongnu.org>, "Ceph Devel" <ceph-devel@vger.kernel.org> 
Envoyé: Samedi 11 Octobre 2014 10:25:35 
Objet: Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image 

On Sat, 10/11 10:00, Alexandre DERUMIER wrote: 
> >>What is the source format? If the zero clusters are actually unallocated in the 
> >>source image, drive-mirror will not write those clusters either. I.e. with 
> >>"drive-mirror sync=top", both source and target should have the same "qemu-img 
> >>map" output. 
> 
> Thanks for your reply, 
> 
> I had tried drive mirror (sync=full) with 
> 
> raw file (sparse) -> rbd (no sparse) 
> rbd (sparse) -> rbd (no sparse) 
> raw file (sparse) -> qcow2 on ext4 (sparse) 
> rbd (sparse) -> raw on ext4 (sparse) 
> 
> Also I see that I have the same problem with target file format on xfs. 
> 
> raw file (sparse) -> qcow2 on xfs (no sparse) 
> rbd (sparse) -> raw on xfs (no sparse) 
> 

These don't tell me much. Maybe it's better to show the actual commands and how 
you tell sparse from no sparse? 

Does "qcow2 -> qcow2" work for you on xfs? 

> 
> I only have this problem with drive-mirror, qemu-img convert seem to simply skip zero blocks. 
> 
> 
> Or maybe this is because I'm using sync=full ? 
> 
> What is the difference between full and top ? 
> 
> ""sync": what parts of the disk image should be copied to the destination; 
> possibilities include "full" for all the disk, "top" for only the sectors 
> allocated in the topmost image". 
> 
> (what is topmost image ?) 

For "sync=top", only the clusters allocated in the image itself is copied; for 
"full", all those clusters allocated in the image itself, and its backing 
image, and it's backing's backing image, ..., are copied. 

The image itself, having a backing image or not, is called the topmost image. 

Fam 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Alexandre DERUMIER]

>>These don't tell me much. Maybe it's better to show the actual commands and how
>>you tell sparse from no sparse?


Well,

I create 2 empty source images files of 10G. (source.qcow2 and source.raw)

then:

du -sh source.qcow2  : 2M
du -sh source.raw  : 0M


then I convert them with qemu-img convert : (source.qcow2 -> target.qcow2 , source.raw -> target.raw)

du -sh target.qcow2  : 2M
du -sh target.raw  : 0M

(So it's ok here)

But If I convert them with drive-mirror:

du -sh target.qcow2  : 11G
du -sh target.raw  : 11G



I have also double check with #df, and I see space allocated on filesystem when using drive-mirror.

I have the same behavior if target is a rbd storage.

Also I have done test again ext3,ext4, and I see the same problem than with xfs.


I'm pretty sure that drive-mirror copy zero block, qemu-img convert take around 2s to convert the empty file
(because it's skipping zero block), and drive mirror take around 5min.

----- Mail original -----

De: "Fam Zheng" <famz@redhat.com>
À: "Alexandre DERUMIER" <aderumier@odiso.com>
Cc: "qemu-devel" <qemu-devel@nongnu.org>, "Ceph Devel" <ceph-devel@vger.kernel.org>
Envoyé: Samedi 11 Octobre 2014 10:25:35
Objet: Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

On Sat, 10/11 10:00, Alexandre DERUMIER wrote:
> >>What is the source format? If the zero clusters are actually unallocated in the
> >>source image, drive-mirror will not write those clusters either. I.e. with
> >>"drive-mirror sync=top", both source and target should have the same "qemu-img
> >>map" output.
>
> Thanks for your reply,
>
> I had tried drive mirror (sync=full) with
>
> raw file (sparse) -> rbd (no sparse)
> rbd (sparse) -> rbd (no sparse)
> raw file (sparse) -> qcow2 on ext4 (sparse)
> rbd (sparse) -> raw on ext4 (sparse)
>
> Also I see that I have the same problem with target file format on xfs.
>
> raw file (sparse) -> qcow2 on xfs (no sparse)
> rbd (sparse) -> raw on xfs (no sparse)
>

These don't tell me much. Maybe it's better to show the actual commands and how
you tell sparse from no sparse?

Does "qcow2 -> qcow2" work for you on xfs?

>
> I only have this problem with drive-mirror, qemu-img convert seem to simply skip zero blocks.
>
>
> Or maybe this is because I'm using sync=full ?
>
> What is the difference between full and top ?
>
> ""sync": what parts of the disk image should be copied to the destination;
> possibilities include "full" for all the disk, "top" for only the sectors
> allocated in the topmost image".
>
> (what is topmost image ?)

For "sync=top", only the clusters allocated in the image itself is copied; for
"full", all those clusters allocated in the image itself, and its backing
image, and it's backing's backing image, ..., are copied.

The image itself, having a backing image or not, is called the topmost image.

Fam

Re: qemu drive-mirror to rbd storage : no sparse rbd image

[Paolo Bonzini]

Il 08/10/2014 13:15, Alexandre DERUMIER ha scritto:
> Hi,
> 
> I'm currently planning to migrate our storage to ceph/rbd through qemu drive-mirror
> 
> and It seem that drive-mirror with rbd block driver, don't create a sparse image. (all zeros are copied to the target rbd).
> 
> Also note, that it's working fine with "qemu-img convert" , the rbd volume is sparse after conversion.
> 
> 
> Could it be related to the "bdrv_co_write_zeroes" missing features in block/rbd.c ?
> 
> (It's available in other block drivers (scsi,gluster,raw-aio) , and I don't have this problem with theses block drivers).

Lack of bdrv_co_write_zeroes is why detect-zeroes does not work.

Lack of bdrv_get_block_status is why sparse->sparse does not work
without detect-zeroes.

Paolo

Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Paolo Bonzini]

Il 08/10/2014 13:15, Alexandre DERUMIER ha scritto:
> Hi,
> 
> I'm currently planning to migrate our storage to ceph/rbd through qemu drive-mirror
> 
> and It seem that drive-mirror with rbd block driver, don't create a sparse image. (all zeros are copied to the target rbd).
> 
> Also note, that it's working fine with "qemu-img convert" , the rbd volume is sparse after conversion.
> 
> 
> Could it be related to the "bdrv_co_write_zeroes" missing features in block/rbd.c ?
> 
> (It's available in other block drivers (scsi,gluster,raw-aio) , and I don't have this problem with theses block drivers).

Lack of bdrv_co_write_zeroes is why detect-zeroes does not work.

Lack of bdrv_get_block_status is why sparse->sparse does not work
without detect-zeroes.

Paolo

Re: qemu drive-mirror to rbd storage : no sparse rbd image

[Alexandre DERUMIER]

>>Lack of bdrv_co_write_zeroes is why detect-zeroes does not work. 
>>
>>Lack of bdrv_get_block_status is why sparse->sparse does not work 
>>without detect-zeroes.

Ok, thanks Paolo !

Both are missing in rbd block driver. @ceph-devel . Could it be possible to implement them ?



Also, about drive-mirror, I had tried with detect-zeroes with simple qcow2 file, 
and It don't seem to help.
I'm not sure that detect-zeroes is implement in drive-mirror.

also, the target mirrored volume don't seem to have the detect-zeroes option


# info block
drive-virtio1: /source.qcow2 (qcow2)
    Detect zeroes:    on

#du -sh source.qcow2 : 2M

drive-mirror  source.qcow2 -> target.qcow2

# info block
drive-virtio1: /target.qcow2 (qcow2)

#du -sh target.qcow2 : 11G




----- Mail original ----- 

De: "Paolo Bonzini" <pbonzini@redhat.com> 
À: "Alexandre DERUMIER" <aderumier@odiso.com>, "qemu-devel" <qemu-devel@nongnu.org> 
Cc: "Ceph Devel" <ceph-devel@vger.kernel.org> 
Envoyé: Dimanche 12 Octobre 2014 15:02:12 
Objet: Re: qemu drive-mirror to rbd storage : no sparse rbd image 

Il 08/10/2014 13:15, Alexandre DERUMIER ha scritto: 
> Hi, 
> 
> I'm currently planning to migrate our storage to ceph/rbd through qemu drive-mirror 
> 
> and It seem that drive-mirror with rbd block driver, don't create a sparse image. (all zeros are copied to the target rbd). 
> 
> Also note, that it's working fine with "qemu-img convert" , the rbd volume is sparse after conversion. 
> 
> 
> Could it be related to the "bdrv_co_write_zeroes" missing features in block/rbd.c ? 
> 
> (It's available in other block drivers (scsi,gluster,raw-aio) , and I don't have this problem with theses block drivers). 

Lack of bdrv_co_write_zeroes is why detect-zeroes does not work. 

Lack of bdrv_get_block_status is why sparse->sparse does not work 
without detect-zeroes. 

Paolo 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Alexandre DERUMIER]

>>Lack of bdrv_co_write_zeroes is why detect-zeroes does not work.
>>
>>Lack of bdrv_get_block_status is why sparse->sparse does not work
>>without detect-zeroes.

Ok, thanks Paolo !

Both are missing in rbd block driver. @ceph-devel . Could it be possible to implement them ?



Also, about drive-mirror, I had tried with detect-zeroes with simple qcow2 file,
and It don't seem to help.
I'm not sure that detect-zeroes is implement in drive-mirror.

also, the target mirrored volume don't seem to have the detect-zeroes option


# info block
drive-virtio1: /source.qcow2 (qcow2)
    Detect zeroes:    on

#du -sh source.qcow2 : 2M

drive-mirror  source.qcow2 -> target.qcow2

# info block
drive-virtio1: /target.qcow2 (qcow2)

#du -sh target.qcow2 : 11G




----- Mail original -----

De: "Paolo Bonzini" <pbonzini@redhat.com>
À: "Alexandre DERUMIER" <aderumier@odiso.com>, "qemu-devel" <qemu-devel@nongnu.org>
Cc: "Ceph Devel" <ceph-devel@vger.kernel.org>
Envoyé: Dimanche 12 Octobre 2014 15:02:12
Objet: Re: qemu drive-mirror to rbd storage : no sparse rbd image

Il 08/10/2014 13:15, Alexandre DERUMIER ha scritto:
> Hi,
>
> I'm currently planning to migrate our storage to ceph/rbd through qemu drive-mirror
>
> and It seem that drive-mirror with rbd block driver, don't create a sparse image. (all zeros are copied to the target rbd).
>
> Also note, that it's working fine with "qemu-img convert" , the rbd volume is sparse after conversion.
>
>
> Could it be related to the "bdrv_co_write_zeroes" missing features in block/rbd.c ?
>
> (It's available in other block drivers (scsi,gluster,raw-aio) , and I don't have this problem with theses block drivers).

Lack of bdrv_co_write_zeroes is why detect-zeroes does not work.

Lack of bdrv_get_block_status is why sparse->sparse does not work
without detect-zeroes.

Paolo

Re: qemu drive-mirror to rbd storage : no sparse rbd image

[Paolo Bonzini]

Il 13/10/2014 08:06, Alexandre DERUMIER ha scritto:
> 
> Also, about drive-mirror, I had tried with detect-zeroes with simple qcow2 file, 
> and It don't seem to help.
> I'm not sure that detect-zeroes is implement in drive-mirror.
> 
> also, the target mirrored volume don't seem to have the detect-zeroes option
> 
> 
> # info block
> drive-virtio1: /source.qcow2 (qcow2)
>     Detect zeroes:    on
> 
> #du -sh source.qcow2 : 2M
> 
> drive-mirror  source.qcow2 -> target.qcow2
> 
> # info block
> drive-virtio1: /target.qcow2 (qcow2)
> 
> #du -sh target.qcow2 : 11G
> 

Ah, you're right.  We need to add an options field, or use a new
blockdev-mirror command.

Paolo

Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Paolo Bonzini]

Il 13/10/2014 08:06, Alexandre DERUMIER ha scritto:
> 
> Also, about drive-mirror, I had tried with detect-zeroes with simple qcow2 file, 
> and It don't seem to help.
> I'm not sure that detect-zeroes is implement in drive-mirror.
> 
> also, the target mirrored volume don't seem to have the detect-zeroes option
> 
> 
> # info block
> drive-virtio1: /source.qcow2 (qcow2)
>     Detect zeroes:    on
> 
> #du -sh source.qcow2 : 2M
> 
> drive-mirror  source.qcow2 -> target.qcow2
> 
> # info block
> drive-virtio1: /target.qcow2 (qcow2)
> 
> #du -sh target.qcow2 : 11G
> 

Ah, you're right.  We need to add an options field, or use a new
blockdev-mirror command.

Paolo

Re: qemu drive-mirror to rbd storage : no sparse rbd image

[Alexandre DERUMIER]

>>Ah, you're right.  We need to add an options field, or use a new
>>blockdev-mirror command.

Ok, thanks. Can't help to implement this, but I'll glad to help for testing.


----- Mail original ----- 

De: "Paolo Bonzini" <pbonzini@redhat.com> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: "Ceph Devel" <ceph-devel@vger.kernel.org>, "qemu-devel" <qemu-devel@nongnu.org> 
Envoyé: Lundi 13 Octobre 2014 09:06:01 
Objet: Re: qemu drive-mirror to rbd storage : no sparse rbd image 

Il 13/10/2014 08:06, Alexandre DERUMIER ha scritto: 
> 
> Also, about drive-mirror, I had tried with detect-zeroes with simple qcow2 file, 
> and It don't seem to help. 
> I'm not sure that detect-zeroes is implement in drive-mirror. 
> 
> also, the target mirrored volume don't seem to have the detect-zeroes option 
> 
> 
> # info block 
> drive-virtio1: /source.qcow2 (qcow2) 
> Detect zeroes: on 
> 
> #du -sh source.qcow2 : 2M 
> 
> drive-mirror source.qcow2 -> target.qcow2 
> 
> # info block 
> drive-virtio1: /target.qcow2 (qcow2) 
> 
> #du -sh target.qcow2 : 11G 
> 

Ah, you're right. We need to add an options field, or use a new 
blockdev-mirror command. 

Paolo 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image

[Alexandre DERUMIER]

>>Ah, you're right.  We need to add an options field, or use a new
>>blockdev-mirror command.

Ok, thanks. Can't help to implement this, but I'll glad to help for testing.


----- Mail original -----

De: "Paolo Bonzini" <pbonzini@redhat.com>
À: "Alexandre DERUMIER" <aderumier@odiso.com>
Cc: "Ceph Devel" <ceph-devel@vger.kernel.org>, "qemu-devel" <qemu-devel@nongnu.org>
Envoyé: Lundi 13 Octobre 2014 09:06:01
Objet: Re: qemu drive-mirror to rbd storage : no sparse rbd image

Il 13/10/2014 08:06, Alexandre DERUMIER ha scritto:
>
> Also, about drive-mirror, I had tried with detect-zeroes with simple qcow2 file,
> and It don't seem to help.
> I'm not sure that detect-zeroes is implement in drive-mirror.
>
> also, the target mirrored volume don't seem to have the detect-zeroes option
>
>
> # info block
> drive-virtio1: /source.qcow2 (qcow2)
> Detect zeroes: on
>
> #du -sh source.qcow2 : 2M
>
> drive-mirror source.qcow2 -> target.qcow2
>
> # info block
> drive-virtio1: /target.qcow2 (qcow2)
>
> #du -sh target.qcow2 : 11G
>

Ah, you're right. We need to add an options field, or use a new
blockdev-mirror command.

Paolo