the question of way to modify bucket metadata

the question of way to modify bucket metadata

[Scott Jing]

Hi,

I'm Wenjun Jing from China Mobile(suzhou). Recently ceph multisite is
applied to our production onest, short for Object-Orient Storage
System,
which is a object storage production based on ceph. Needless to say,
the feature of active-active is awesome! But there are also some
features
which may not be supported very well and the community is improving
them constantly. So we want to discuss with you to comfirm the roadmap
of them.

This is the fact that if creating bucket on the secondary
zone/zonegroup. The request would send to meta master and the bucket
will be created
on meta master firstly, then it will be created locally.

Some metadata/feature of bucket in the process upper is not synced to
secondary zone or zonegroup. For instance, the bucket acl and bucket
versionning cannot be modified on the slave zone/zonegroup, which has
been fixed in the jewel. Now, we want to ack you to get the
development
direction(way) of the bucket metadata to modify.

The below are some ways which I've thought out currently.

1. All the bucket metadata will be modified in the meta master firstly
and then modify the metadata
locally according to the status retured from meta master. Considering
this way, some metadata (eg. creation time)
may not consistent with the meta master, which doesn't improve user experience.

2. All the bucket metadata will be modified in the meta master firstly
and sync the changed bucket metadata to slave zone or zonegroup.

3. The other alternative way to handle this situation? Could you
discribe to me detailedly?

Once the upper way could be comfirmed Definitely, we can make some
contribution to the community.

Look forward to your kind reply.
Thanks!

Re: the question of way to modify bucket metadata

[Casey Bodley]

On 12/16/2016 09:20 AM, Scott Jing wrote:
> Hi,
>
> I'm Wenjun Jing from China Mobile(suzhou). Recently ceph multisite is
> applied to our production onest, short for Object-Orient Storage
> System,
> which is a object storage production based on ceph. Needless to say,
> the feature of active-active is awesome! But there are also some
> features
> which may not be supported very well and the community is improving
> them constantly. So we want to discuss with you to comfirm the roadmap
> of them.
>
> This is the fact that if creating bucket on the secondary
> zone/zonegroup. The request would send to meta master and the bucket
> will be created
> on meta master firstly, then it will be created locally.
>
> Some metadata/feature of bucket in the process upper is not synced to
> secondary zone or zonegroup. For instance, the bucket acl and bucket
> versionning cannot be modified on the slave zone/zonegroup, which has
> been fixed in the jewel. Now, we want to ack you to get the
> development
> direction(way) of the bucket metadata to modify.
>
> The below are some ways which I've thought out currently.
>
> 1. All the bucket metadata will be modified in the meta master firstly
> and then modify the metadata
> locally according to the status retured from meta master. Considering
> this way, some metadata (eg. creation time)
> may not consistent with the meta master, which doesn't improve user experience.
>
> 2. All the bucket metadata will be modified in the meta master firstly
> and sync the changed bucket metadata to slave zone or zonegroup.
>
> 3. The other alternative way to handle this situation? Could you
> discribe to me detailedly?
>
> Once the upper way could be comfirmed Definitely, we can make some
> contribution to the community.
>
> Look forward to your kind reply.
> Thanks!
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi,

#2 is the current strategy for updates, but there's been some discussion 
about this related to the fix for bucket versioning at 
https://github.com/ceph/ceph/pull/12444. The issue here is that if you 
enable bucket versioning on a non-master zone, the change is not 
reflected on that zone until the background sync process reads it from 
the metadata master.

That pull request changes RGWSetBucketVersioning to use strategy #1, but 
as you point out, this can lead to inconsistent metadata. It also 
introduces the potential to overwrite the results of other operations 
applied on the metadata master.

I think the ideal solution would involve getting the updated metadata 
from the master (either in the reply to our forwarded request, or with a 
separate request to /admin/metadata) and writing that locally. We'd just 
have to make sure that this write respects the same object versioning 
requirements as other updates from metadata sync - I think 
RGWBucketInstanceMetadataHandler::put() takes care of for us, but it 
needs to be tested.

I created an issue to track this at http://tracker.ceph.com/issues/18292.

Thanks,
Casey