Speed up 'rbd rm'

Speed up 'rbd rm'

[Chris Dunlop]

Hi,

I know 'rbd rm' is notoriously slow, even on never-written devices:

http://thread.gmane.org/gmane.comp.file-systems.ceph.devel/6740
http://tracker.ceph.com/issues/2256

I fat fingered an 'rbd create' and accidentally created a 1.5 PB device,
and it's going to take some considerable time to remove this!

I see there's a new commit to speed up an 'rbd rm':

http://tracker.ceph.com/projects/ceph/repository/revisions/40956410169709c32a282d9b872cb5f618a48926

Is it safe to cherry-pick this commit on top of 0.56.6 (or, if not, v0.61.2) to speed up the remove?

Cheers,

Chris

Re: Speed up 'rbd rm'

[Josh Durgin]

On 05/28/2013 10:59 PM, Chris Dunlop wrote:
> I see there's a new commit to speed up an 'rbd rm':
>
> http://tracker.ceph.com/projects/ceph/repository/revisions/40956410169709c32a282d9b872cb5f618a48926
>
> Is it safe to cherry-pick this commit on top of 0.56.6 (or, if not, v0.61.2) to speed up the remove?

You'll need 537386d906b8c0e395433461dcb03a82eb33f34f as well. It should
apply cleanly to 0.61.2, and probably 0.56.6 too.

Re: Speed up 'rbd rm'

[Chris Dunlop]

On Wed, May 29, 2013 at 12:21:07PM -0700, Josh Durgin wrote:
> On 05/28/2013 10:59 PM, Chris Dunlop wrote:
>> I see there's a new commit to speed up an 'rbd rm':
>>
>> http://tracker.ceph.com/projects/ceph/repository/revisions/40956410169709c32a282d9b872cb5f618a48926
>>
>> Is it safe to cherry-pick this commit on top of 0.56.6 (or, if not, v0.61.2) to speed up the remove?
> 
> You'll need 537386d906b8c0e395433461dcb03a82eb33f34f as well. It should
> apply cleanly to 0.61.2, and probably 0.56.6 too.

Thanks. I'll see how I go, I may just leave the 'rm' running all
weekend rather than futzing around recompiling ceph and getting
off the mainline track.

Perhaps as a low-churn high-impact (for those that need it) change,
this is a candidate to be back-patched into 0.61.3 and perhaps 0.56.7 ?

Cheers,

Chris.

Re: Speed up 'rbd rm'

[Josh Durgin]

On 05/29/2013 07:23 PM, Chris Dunlop wrote:
> On Wed, May 29, 2013 at 12:21:07PM -0700, Josh Durgin wrote:
>> On 05/28/2013 10:59 PM, Chris Dunlop wrote:
>>> I see there's a new commit to speed up an 'rbd rm':
>>>
>>> http://tracker.ceph.com/projects/ceph/repository/revisions/40956410169709c32a282d9b872cb5f618a48926
>>>
>>> Is it safe to cherry-pick this commit on top of 0.56.6 (or, if not, v0.61.2) to speed up the remove?
>>
>> You'll need 537386d906b8c0e395433461dcb03a82eb33f34f as well. It should
>> apply cleanly to 0.61.2, and probably 0.56.6 too.
>
> Thanks. I'll see how I go, I may just leave the 'rm' running all
> weekend rather than futzing around recompiling ceph and getting
> off the mainline track.

If you're mainly interested in getting rid of the accidentally 1.5PB
image, you can just delete the header (and id object if it's format 2)
and then 'rbd rm' will just remove it from the rbd_directory index, and
not try to delete all the non-existent data objects.

> Perhaps as a low-churn high-impact (for those that need it) change,
> this is a candidate to be back-patched into 0.61.3 and perhaps 0.56.7 ?
>
> Cheers,
>
> Chris.

Re: Speed up 'rbd rm'

[Smart Weblications GmbH - Florian Wiessner]

Am 30.05.2013 22:50, schrieb Josh Durgin:

> 
>> Perhaps as a low-churn high-impact (for those that need it) change,
>> this is a candidate to be back-patched into 0.61.3 and perhaps 0.56.7 ?
>>

+1


-- 

Mit freundlichen Grüßen,

Florian Wiessner

Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 9282 9638 200
fax.: +49 9282 9638 205
24/7: +49 900 144 000 00 - 0,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Speed up 'rbd rm'

[Chris Dunlop]

On Thu, May 30, 2013 at 01:50:14PM -0700, Josh Durgin wrote:
> On 05/29/2013 07:23 PM, Chris Dunlop wrote:
>> On Wed, May 29, 2013 at 12:21:07PM -0700, Josh Durgin wrote:
>>> On 05/28/2013 10:59 PM, Chris Dunlop wrote:
>>>> I see there's a new commit to speed up an 'rbd rm':
>>>>
>>>> http://tracker.ceph.com/projects/ceph/repository/revisions/40956410169709c32a282d9b872cb5f618a48926
>>>>
>>>> Is it safe to cherry-pick this commit on top of 0.56.6 (or, if not, v0.61.2) to speed up the remove?
>>>
>>> You'll need 537386d906b8c0e395433461dcb03a82eb33f34f as well. It should
>>> apply cleanly to 0.61.2, and probably 0.56.6 too.
>>
>> Thanks. I'll see how I go, I may just leave the 'rm' running all
>> weekend rather than futzing around recompiling ceph and getting
>> off the mainline track.
> 
> If you're mainly interested in getting rid of the accidentally 1.5PB
> image, you can just delete the header (and id object if it's format 2)
> and then 'rbd rm' will just remove it from the rbd_directory index, and
> not try to delete all the non-existent data objects.

Yes, that's my main interest. Sorry, I haven't yet delved far
into the details of how the rbd stuff hangs together: can you
give me a hint or point me towards any docs regarding what
"delete the header (and id object" would look like?


Cheers,

Chris.

Re: Speed up 'rbd rm'

[Josh Durgin]

On 05/30/2013 06:40 PM, Chris Dunlop wrote:
> On Thu, May 30, 2013 at 01:50:14PM -0700, Josh Durgin wrote:
>> On 05/29/2013 07:23 PM, Chris Dunlop wrote:
>>> On Wed, May 29, 2013 at 12:21:07PM -0700, Josh Durgin wrote:
>>>> On 05/28/2013 10:59 PM, Chris Dunlop wrote:
>>>>> I see there's a new commit to speed up an 'rbd rm':
>>>>>
>>>>> http://tracker.ceph.com/projects/ceph/repository/revisions/40956410169709c32a282d9b872cb5f618a48926
>>>>>
>>>>> Is it safe to cherry-pick this commit on top of 0.56.6 (or, if not, v0.61.2) to speed up the remove?
>>>>
>>>> You'll need 537386d906b8c0e395433461dcb03a82eb33f34f as well. It should
>>>> apply cleanly to 0.61.2, and probably 0.56.6 too.
>>>
>>> Thanks. I'll see how I go, I may just leave the 'rm' running all
>>> weekend rather than futzing around recompiling ceph and getting
>>> off the mainline track.
>>
>> If you're mainly interested in getting rid of the accidentally 1.5PB
>> image, you can just delete the header (and id object if it's format 2)
>> and then 'rbd rm' will just remove it from the rbd_directory index, and
>> not try to delete all the non-existent data objects.
>
> Yes, that's my main interest. Sorry, I haven't yet delved far
> into the details of how the rbd stuff hangs together: can you
> give me a hint or point me towards any docs regarding what
> "delete the header (and id object" would look like?

For a format 2 image, 'rbd info imagename' will show a block_prefix like 
'rbd_data.101574b0dc51'.

The random suffix after the '.' is the id of the image.
For format 2, the header is named after this id, so you'd do:

rados -p poolname rm rbd_header.101574b0dc51

For format 1 images, the header object is named after the image name, 
like 'imagename.rbd'.

After removing the header object manually, rbd rm will clean up the
rest.

Re: Speed up 'rbd rm'

[Stefan Priebe - Profihost AG]

Am 31.05.2013 03:40, schrieb Chris Dunlop:
> On Thu, May 30, 2013 at 01:50:14PM -0700, Josh Durgin wrote:
>> On 05/29/2013 07:23 PM, Chris Dunlop wrote:
>>> On Wed, May 29, 2013 at 12:21:07PM -0700, Josh Durgin wrote:
>>>> On 05/28/2013 10:59 PM, Chris Dunlop wrote:
>>>>> I see there's a new commit to speed up an 'rbd rm':
>>>>>
>>>>> http://tracker.ceph.com/projects/ceph/repository/revisions/40956410169709c32a282d9b872cb5f618a48926
>>>>>
>>>>> Is it safe to cherry-pick this commit on top of 0.56.6 (or, if not, v0.61.2) to speed up the remove?
>>>>
>>>> You'll need 537386d906b8c0e395433461dcb03a82eb33f34f as well. It should
>>>> apply cleanly to 0.61.2, and probably 0.56.6 too.

I cherry picked both on top of cuttlefish it works but not very good.
rbd command finishes but rbd image is still in rbd ls and just gets
deleted later. I also saw a variant, where rbd rm outputs errors even
though the image was there but it still got deleted...

Stefan

Re: Speed up 'rbd rm'

[Josh Durgin]

On 05/30/2013 11:40 PM, Stefan Priebe - Profihost AG wrote:
> Am 31.05.2013 03:40, schrieb Chris Dunlop:
>> On Thu, May 30, 2013 at 01:50:14PM -0700, Josh Durgin wrote:
>>> On 05/29/2013 07:23 PM, Chris Dunlop wrote:
>>>> On Wed, May 29, 2013 at 12:21:07PM -0700, Josh Durgin wrote:
>>>>> On 05/28/2013 10:59 PM, Chris Dunlop wrote:
>>>>>> I see there's a new commit to speed up an 'rbd rm':
>>>>>>
>>>>>> http://tracker.ceph.com/projects/ceph/repository/revisions/40956410169709c32a282d9b872cb5f618a48926
>>>>>>
>>>>>> Is it safe to cherry-pick this commit on top of 0.56.6 (or, if not, v0.61.2) to speed up the remove?
>>>>>
>>>>> You'll need 537386d906b8c0e395433461dcb03a82eb33f34f as well. It should
>>>>> apply cleanly to 0.61.2, and probably 0.56.6 too.
>
> I cherry picked both on top of cuttlefish it works but not very good.
> rbd command finishes but rbd image is still in rbd ls and just gets
> deleted later. I also saw a variant, where rbd rm outputs errors even
> though the image was there but it still got deleted...

I forgot that there was a bug in the original version. You'll need
395a775a8c87b0ee1ce314257e25240214f4081c too.

Re: Speed up 'rbd rm'

[Chris Dunlop]

On Thu, May 30, 2013 at 07:04:28PM -0700, Josh Durgin wrote:
> On 05/30/2013 06:40 PM, Chris Dunlop wrote:
>> On Thu, May 30, 2013 at 01:50:14PM -0700, Josh Durgin wrote:
>>> On 05/29/2013 07:23 PM, Chris Dunlop wrote:
>>>> On Wed, May 29, 2013 at 12:21:07PM -0700, Josh Durgin wrote:
>>>>> On 05/28/2013 10:59 PM, Chris Dunlop wrote:
>>>>>> I see there's a new commit to speed up an 'rbd rm':
>>>>>>
>>>>>> http://tracker.ceph.com/projects/ceph/repository/revisions/40956410169709c32a282d9b872cb5f618a48926
>>>>>>
>>>>>> Is it safe to cherry-pick this commit on top of 0.56.6 (or, if not, v0.61.2) to speed up the remove?
>>>>>
>>>>> You'll need 537386d906b8c0e395433461dcb03a82eb33f34f as well. It should
>>>>> apply cleanly to 0.61.2, and probably 0.56.6 too.
>>>>
>>>> Thanks. I'll see how I go, I may just leave the 'rm' running all
>>>> weekend rather than futzing around recompiling ceph and getting
>>>> off the mainline track.

# time rbd rm rbd/large-image
Removing image: 36% complete...Terminated
real    2819m37.117s

I.e. 47 hours and only 36% complete before I gave up (I wanted
to restart that server). At that rate it would take 5.5 days to
remove!

>>> If you're mainly interested in getting rid of the accidentally 1.5PB
>>> image, you can just delete the header (and id object if it's format 2)
>>> and then 'rbd rm' will just remove it from the rbd_directory index, and
>>> not try to delete all the non-existent data objects.
>>
>> Yes, that's my main interest. Sorry, I haven't yet delved far
>> into the details of how the rbd stuff hangs together: can you
>> give me a hint or point me towards any docs regarding what
>> "delete the header (and id object" would look like?
> 
> For a format 2 image, 'rbd info imagename' will show a block_prefix
> like 'rbd_data.101574b0dc51'.
> 
> The random suffix after the '.' is the id of the image.
> For format 2, the header is named after this id, so you'd do:
> 
> rados -p poolname rm rbd_header.101574b0dc51
> 
> For format 1 images, the header object is named after the image name,
> like 'imagename.rbd'.
>
> After removing the header object manually, rbd rm will clean up the
> rest.

The problematical image is format 2.

If it's tricky to manually remove, it's not doing any harm just
sitting there (it it??) so I guess I can just wait until the
parallelized delete is available in a stable release, i.e.
dumpling, or backported to bobtail or cuttlefish.

Cheers,

Chris.