Sequential write performance on cloning images

Sequential write performance on cloning images

[Ridge Chen]

Hi Experts,

I was trying to test the performance on rbd images. I find the
sequential write throughput on cloning image is only harf of that on
origin one. I have enabled object-map feature on both images and the
cloning image is fully over-written before the performance test.

Here is my steps:
1. create two images called "ori-vol1", "ori-vol2"
2. run a "dd" on them to make the images full of data
3. create a snap on "ori-vol1", called "snap-vol1"
4. clone a image from "snap-vol1", called "clone-vol1"
5. run a "dd" on "clone-vol1" to make the image full "copy-up"
6. finally test performance on "ori-vol2" and "clone-vol1". --
sequential write, 64K iosize, fio

I have tested on hammer and jewel, the results are similar. Is it expected?

I know there may be performance drop on cloning image since the issued
object may be at parent image and it required a extra "copy-up"
operation. However in my case the objects are all at own image and
librbd should have knowledge about this from object-map.

Regards
Ridge

Re: Sequential write performance on cloning images

[Jason Dillaman]

On Sun, Aug 14, 2016 at 10:41 PM, Ridge Chen <ridge.chen@gmail.com> wrote:
> I have tested on hammer and jewel, the results are similar. Is it expected?

In this case, there actually would not be an extra copy-up since you
already performed the necessary copy-ups during the "dd" against
"clone-vol1". The only difference, on the librbd side, is that the
write operation would have an additional "assert object exists"
operation before the write operation.

Do you see similar differences with the librbd cache disabled and the
object map disabled?



-- 
Jason

Re: Sequential write performance on cloning images

[Ridge Chen]

Hi

Thanks for your quick response.

Yes, the result is similar with object-map disabled. And in all my
tests the librbd cache is disabled.

Could you explain more about the additional "assert object exists"
operation? what it does in osd side?

And can we bypass the operation in this very common case based on the
object-map feature?

Regards,
Ridge

2016-08-15 20:16 GMT+08:00 Jason Dillaman <jdillama@redhat.com>:
> On Sun, Aug 14, 2016 at 10:41 PM, Ridge Chen <ridge.chen@gmail.com> wrote:
>> I have tested on hammer and jewel, the results are similar. Is it expected?
>
> In this case, there actually would not be an extra copy-up since you
> already performed the necessary copy-ups during the "dd" against
> "clone-vol1". The only difference, on the librbd side, is that the
> write operation would have an additional "assert object exists"
> operation before the write operation.
>
> Do you see similar differences with the librbd cache disabled and the
> object map disabled?
>
>
>
> --
> Jason

Re: Sequential write performance on cloning images

[Jason Dillaman]

The "assert object exists" check is just a "stat" op inserted before
the write op in the same transaction.  If the object doesn't exist
(i.e. a clone copy-on-write is required), the transaction will fail
within the OSD and librbd will know it needs to perform a copy-up.
However, in your case, this stat op should be essentially a no-op so I
would expect zero performance penalty.

Do you rebuild your images after every test run? I ask because one big
difference is that when you are writing to the parent image, it has a
snapshot so the first write to a backing object would require a
copy-on-write operation within the OSD.



On Mon, Aug 15, 2016 at 10:05 PM, Ridge Chen <ridge.chen@gmail.com> wrote:
> Hi
>
> Thanks for your quick response.
>
> Yes, the result is similar with object-map disabled. And in all my
> tests the librbd cache is disabled.
>
> Could you explain more about the additional "assert object exists"
> operation? what it does in osd side?
>
> And can we bypass the operation in this very common case based on the
> object-map feature?
>
> Regards,
> Ridge
>
> 2016-08-15 20:16 GMT+08:00 Jason Dillaman <jdillama@redhat.com>:
>> On Sun, Aug 14, 2016 at 10:41 PM, Ridge Chen <ridge.chen@gmail.com> wrote:
>>> I have tested on hammer and jewel, the results are similar. Is it expected?
>>
>> In this case, there actually would not be an extra copy-up since you
>> already performed the necessary copy-ups during the "dd" against
>> "clone-vol1". The only difference, on the librbd side, is that the
>> write operation would have an additional "assert object exists"
>> operation before the write operation.
>>
>> Do you see similar differences with the librbd cache disabled and the
>> object map disabled?
>>
>>
>>
>> --
>> Jason



-- 
Jason

Re: Sequential write performance on cloning images

[Ridge Chen]

Hi

The parent image of "clone-vol1", which is "ori-vol1", is not touched
during the test.

Regards
Ridge

2016-08-16 10:22 GMT+08:00 Jason Dillaman <jdillama@redhat.com>:
> The "assert object exists" check is just a "stat" op inserted before
> the write op in the same transaction.  If the object doesn't exist
> (i.e. a clone copy-on-write is required), the transaction will fail
> within the OSD and librbd will know it needs to perform a copy-up.
> However, in your case, this stat op should be essentially a no-op so I
> would expect zero performance penalty.
>
> Do you rebuild your images after every test run? I ask because one big
> difference is that when you are writing to the parent image, it has a
> snapshot so the first write to a backing object would require a
> copy-on-write operation within the OSD.
>
>
>
> On Mon, Aug 15, 2016 at 10:05 PM, Ridge Chen <ridge.chen@gmail.com> wrote:
>> Hi
>>
>> Thanks for your quick response.
>>
>> Yes, the result is similar with object-map disabled. And in all my
>> tests the librbd cache is disabled.
>>
>> Could you explain more about the additional "assert object exists"
>> operation? what it does in osd side?
>>
>> And can we bypass the operation in this very common case based on the
>> object-map feature?
>>
>> Regards,
>> Ridge
>>
>> 2016-08-15 20:16 GMT+08:00 Jason Dillaman <jdillama@redhat.com>:
>>> On Sun, Aug 14, 2016 at 10:41 PM, Ridge Chen <ridge.chen@gmail.com> wrote:
>>>> I have tested on hammer and jewel, the results are similar. Is it expected?
>>>
>>> In this case, there actually would not be an extra copy-up since you
>>> already performed the necessary copy-ups during the "dd" against
>>> "clone-vol1". The only difference, on the librbd side, is that the
>>> write operation would have an additional "assert object exists"
>>> operation before the write operation.
>>>
>>> Do you see similar differences with the librbd cache disabled and the
>>> object map disabled?
>>>
>>>
>>>
>>> --
>>> Jason
>
>
>
> --
> Jason

Re: Sequential write performance on cloning images

[Jason Dillaman]

How and when did you compare the IOPS from the clone against the IOPS
from the parent?

On Mon, Aug 15, 2016 at 11:58 PM, Ridge Chen <ridge.chen@gmail.com> wrote:
> Hi
>
> The parent image of "clone-vol1", which is "ori-vol1", is not touched
> during the test.
>
> Regards
> Ridge
>
> 2016-08-16 10:22 GMT+08:00 Jason Dillaman <jdillama@redhat.com>:
>> The "assert object exists" check is just a "stat" op inserted before
>> the write op in the same transaction.  If the object doesn't exist
>> (i.e. a clone copy-on-write is required), the transaction will fail
>> within the OSD and librbd will know it needs to perform a copy-up.
>> However, in your case, this stat op should be essentially a no-op so I
>> would expect zero performance penalty.
>>
>> Do you rebuild your images after every test run? I ask because one big
>> difference is that when you are writing to the parent image, it has a
>> snapshot so the first write to a backing object would require a
>> copy-on-write operation within the OSD.
>>
>>
>>
>> On Mon, Aug 15, 2016 at 10:05 PM, Ridge Chen <ridge.chen@gmail.com> wrote:
>>> Hi
>>>
>>> Thanks for your quick response.
>>>
>>> Yes, the result is similar with object-map disabled. And in all my
>>> tests the librbd cache is disabled.
>>>
>>> Could you explain more about the additional "assert object exists"
>>> operation? what it does in osd side?
>>>
>>> And can we bypass the operation in this very common case based on the
>>> object-map feature?
>>>
>>> Regards,
>>> Ridge
>>>
>>> 2016-08-15 20:16 GMT+08:00 Jason Dillaman <jdillama@redhat.com>:
>>>> On Sun, Aug 14, 2016 at 10:41 PM, Ridge Chen <ridge.chen@gmail.com> wrote:
>>>>> I have tested on hammer and jewel, the results are similar. Is it expected?
>>>>
>>>> In this case, there actually would not be an extra copy-up since you
>>>> already performed the necessary copy-ups during the "dd" against
>>>> "clone-vol1". The only difference, on the librbd side, is that the
>>>> write operation would have an additional "assert object exists"
>>>> operation before the write operation.
>>>>
>>>> Do you see similar differences with the librbd cache disabled and the
>>>> object map disabled?
>>>>
>>>>
>>>>
>>>> --
>>>> Jason
>>
>>
>>
>> --
>> Jason



-- 
Jason