Deleting files from radosgw-bucket doesn't free up space in ceph?

Deleting files from radosgw-bucket doesn't free up space in ceph?

[John Axel Eriksson]

I'm worried that data deleted in radosgw wasn't actually deleted from
disk/cluster.

Here's the output using df:

/dev/xvdf      1000G  779G  185G  81% /var/lib/ceph/osd/ceph-0

Quite full that disk. Now for ceph -s I get

health HEALTH_OK
  <lines removed>
    pgmap v256604: 2304 pgs: 2304 active+clean; 1103 GB data, 1547 GB
used, 378 GB / 2000 GB avail
  <lines removed>

Still looks pretty full here.

And here's finally the output when checking the only bucket we have:

{ "bucket": "<bucket-name-removed>",
  "pool": ".rgw.buckets",
  "id": "4122.1",
  "marker": "4122.1",
  "owner": "<owner-removed>",
  "usage": { "rgw.main": { "size_kb": 247104513,
          "size_kb_actual": 247345748,
          "num_objects": 108889}}}

This translates to around 236GB which is FAR from the around 770GB
that df and ceph -s reports. The thing is - the only way we're storing
data in ceph is through radosgw and the only bucket we have is the one
shown above (yes a pretty simple deployment). How can the stats
be so very different? Was data not actually deleted from disk? The
deletion took place yesterday so the cluster has had some time to do
any
delayed deletion if that's how it's done.

Any ideas? Thanks!

John

Re: Deleting files from radosgw-bucket doesn't free up space in ceph?

[Tommi Virtanen]

On Tue, Oct 9, 2012 at 9:31 AM, John Axel Eriksson <john@insane.se> wrote:
> I'm worried that data deleted in radosgw wasn't actually deleted from
> disk/cluster.

Are you aware of "radosgw-admin temp remove"?

I was trying to point you to docs, but couldn't find any, so I filed
http://tracker.newdream.net/issues/3278

> Here's the output using df:
> /dev/xvdf      1000G  779G  185G  81% /var/lib/ceph/osd/ceph-0
...
> And here's finally the output when checking the only bucket we have:
>
> { "bucket": "<bucket-name-removed>",
>   "pool": ".rgw.buckets",
>   "id": "4122.1",
>   "marker": "4122.1",
>   "owner": "<owner-removed>",
>   "usage": { "rgw.main": { "size_kb": 247104513,
>           "size_kb_actual": 247345748,
>           "num_objects": 108889}}}
>
> This translates to around 236GB which is FAR from the around 770GB
> that df and ceph -s reports. The thing is - the only way we're storing
> data in ceph is through radosgw and the only bucket we have is the one
> shown above (yes a pretty simple deployment). How can the stats
> be so very different? Was data not actually deleted from disk? The
> deletion took place yesterday so the cluster has had some time to do
> any
> delayed deletion if that's how it's done.

The delayed deletion is done with "radosgw-admin temp remove".

Also, be aware that df etc free space can be confusing in the presence
of 3x replication. So for example, seeing 1TB available across all
your OSDs means you actually have 0.33TB writable, because of the 3x
replication. (Ceph does not try to estimate this factor for you, as
the replica count depends on which pool the data actually gets store
in, and that's not generally predictable.)

Re: Deleting files from radosgw-bucket doesn't free up space in ceph?

[Yehuda Sadeh]

On Tue, Oct 9, 2012 at 10:07 AM, John Axel Eriksson <john@insane.se> wrote:
> I'm running 0.48.1. Wow I had no idea that was the case. I guess
> everything that's been deleted up until today can be removed
> using "radosgw-admin temp remove --date=2012-10-09"... am I correct in
> assuming this only removes "garbage" (e.g. deleted objects)
> and not anything else (such as the stored data in a bucket)?
>
It completes the final stage in removing objects (mostly for objects
with more than 512k in data), either when they were deleted explicitly
or overwritten. If you don't have any pending readers on large objects
that were removed then you can put today's date.

Yehuda

Re: Deleting files from radosgw-bucket doesn't free up space in ceph?

[John Axel Eriksson]

Thanks, seems to have done the trick. I simply didn't realize it was
necessary and since we've done quite a bit of deleting these past
two days I naturally looked at the numbers and they didn't add up.

On Tue, Oct 9, 2012 at 7:17 PM, Yehuda Sadeh <yehuda@inktank.com> wrote:
> On Tue, Oct 9, 2012 at 10:07 AM, John Axel Eriksson <john@insane.se> wrote:
>> I'm running 0.48.1. Wow I had no idea that was the case. I guess
>> everything that's been deleted up until today can be removed
>> using "radosgw-admin temp remove --date=2012-10-09"... am I correct in
>> assuming this only removes "garbage" (e.g. deleted objects)
>> and not anything else (such as the stored data in a bucket)?
>>
> It completes the final stage in removing objects (mostly for objects
> with more than 512k in data), either when they were deleted explicitly
> or overwritten. If you don't have any pending readers on large objects
> that were removed then you can put today's date.
>
> Yehuda