trimming rbd image does not reclaim free space

trimming rbd image does not reclaim free space

[Edgaras Lukoševičius]

Hello,

I have ceph pool (size=2) and three RBD images (2TB each) and I have noticed, that deleting data does not reclaim data on ceph.
What I have done is mount RBD device as a loopback device loop0, mounted it to a filesystem, ran “fstrim” and got response:

/mnt/fstrim/: 1.1 TiB (1167710830592 bytes) trimmed
/mnt/fstrim/: 1 TiB (1125871489024 bytes) trimmed
/mnt/fstrim/: 1.1 TiB (1186485440512 bytes) trimmed

Now that would be almost perfect if not the fact, that space was not reclaimed on ceph itself. Using “ceph osd df” I still see same usage even after waiting few hours and I don’t see any action.

I’m using ceph v0.94-2 and kernel 3.10

What is the proper way to reclaim unused space? I had pool size 3, but had to lower it to two, to avoid getting my OSDs full.

Thanks.--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: trimming rbd image does not reclaim free space

[Ilya Dryomov]

On Mon, May 11, 2015 at 2:26 PM, Edgaras Lukoševičius
<edgaras.lukosevicius@gmail.com> wrote:
> Hello,
>
> I have ceph pool (size=2) and three RBD images (2TB each) and I have noticed, that deleting data does not reclaim data on ceph.
> What I have done is mount RBD device as a loopback device loop0, mounted it to a filesystem, ran “fstrim” and got response:

What do you mean by that?  Can you elaborate - paste the commands you
used?

Thanks,

                Ilya
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: trimming rbd image does not reclaim free space

[Edgaras Lukoševičius]

Resending commands I’m using to everyone.

# rbd-fuse -p pool -r image /mnt/trim/
# losetup /dev/loop0 /mnt/trim/image
# kpartx -av /dev/loop0
# mount /dev/mapper/loop0p1 /mnt/fstrim/
# fstrim -v /mnt/fstrim/
/mnt/fstrim/: 1.1 TiB (1167710830592 bytes) trimmed

I don’t use qemu, kvm.

Anyone can help with this?


On 11 May 2015, at 14:26, Edgaras Lukoševičius <edgaras.lukosevicius@gmail.com> wrote:

> Hello,
> 
> I have ceph pool (size=2) and three RBD images (2TB each) and I have noticed, that deleting data does not reclaim data on ceph.
> What I have done is mount RBD device as a loopback device loop0, mounted it to a filesystem, ran “fstrim” and got response:
> 
> /mnt/fstrim/: 1.1 TiB (1167710830592 bytes) trimmed
> /mnt/fstrim/: 1 TiB (1125871489024 bytes) trimmed
> /mnt/fstrim/: 1.1 TiB (1186485440512 bytes) trimmed
> 
> Now that would be almost perfect if not the fact, that space was not reclaimed on ceph itself. Using “ceph osd df” I still see same usage even after waiting few hours and I don’t see any action.
> 
> I’m using ceph v0.94-2 and kernel 3.10
> 
> What is the proper way to reclaim unused space? I had pool size 3, but had to lower it to two, to avoid getting my OSDs full.
> 
> Thanks.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: trimming rbd image does not reclaim free space

[Wido den Hollander]

On 05/12/2015 09:48 AM, Edgaras Lukoševičius wrote:
> Resending commands I’m using to everyone.
> 
> # rbd-fuse -p pool -r image /mnt/trim/
> # losetup /dev/loop0 /mnt/trim/image
> # kpartx -av /dev/loop0
> # mount /dev/mapper/loop0p1 /mnt/fstrim/
> # fstrim -v /mnt/fstrim/
> /mnt/fstrim/: 1.1 TiB (1167710830592 bytes) trimmed
> 
> I don’t use qemu, kvm.
> 

I think that kernel 3.10 doesn't support TRIM yet. Afaik it was
implemented in kernel 3.18 of 3.19

Wido

> Anyone can help with this?
> 
> 
> On 11 May 2015, at 14:26, Edgaras Lukoševičius <edgaras.lukosevicius@gmail.com> wrote:
> 
>> Hello,
>>
>> I have ceph pool (size=2) and three RBD images (2TB each) and I have noticed, that deleting data does not reclaim data on ceph.
>> What I have done is mount RBD device as a loopback device loop0, mounted it to a filesystem, ran “fstrim” and got response:
>>
>> /mnt/fstrim/: 1.1 TiB (1167710830592 bytes) trimmed
>> /mnt/fstrim/: 1 TiB (1125871489024 bytes) trimmed
>> /mnt/fstrim/: 1.1 TiB (1186485440512 bytes) trimmed
>>
>> Now that would be almost perfect if not the fact, that space was not reclaimed on ceph itself. Using “ceph osd df” I still see same usage even after waiting few hours and I don’t see any action.
>>
>> I’m using ceph v0.94-2 and kernel 3.10
>>
>> What is the proper way to reclaim unused space? I had pool size 3, but had to lower it to two, to avoid getting my OSDs full.
>>
>> Thanks.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: trimming rbd image does not reclaim free space

[Ilya Dryomov]

On Tue, May 12, 2015 at 11:09 AM, Wido den Hollander <wido@42on.com> wrote:
> On 05/12/2015 09:48 AM, Edgaras Lukoševičius wrote:
>> Resending commands I’m using to everyone.
>>
>> # rbd-fuse -p pool -r image /mnt/trim/
>> # losetup /dev/loop0 /mnt/trim/image
>> # kpartx -av /dev/loop0
>> # mount /dev/mapper/loop0p1 /mnt/fstrim/
>> # fstrim -v /mnt/fstrim/
>> /mnt/fstrim/: 1.1 TiB (1167710830592 bytes) trimmed
>>
>> I don’t use qemu, kvm.
>>
>
> I think that kernel 3.10 doesn't support TRIM yet. Afaik it was
> implemented in kernel 3.18 of 3.19
>
> Wido
>
>> Anyone can help with this?

This is rbd-fuse, not kernel client.  I wanted to take a look, but
haven't had a chance yet.  rbd-fuse probably just doesn't support
trimming.  AFAIR discard requests to a loop device are translated into
hole punches on the underlying file, and rbd-fuse doesn't seem to
provide ->fallocate() callback.  It wouldn't be hard to add though,
I can take a shot at it later.

Thanks,

                Ilya
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html