Fwd: S3 API Compatibility support

Fwd: S3 API Compatibility support

[M Ranga Swami Reddy]

Hi Team: As per the ceph document a few S3 APIs compatibility not supported.

Link: http://ceph.com/docs/master/radosgw/s3/

Is there plan to support the ün supported item in the above table?
or
Any working on this?

Thanks
Swami
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: S3 API Compatibility support

[Sage Weil]

On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
> 
> Link: http://ceph.com/docs/master/radosgw/s3/
> 
> Is there plan to support the ?n supported item in the above table?
> or
> Any working on this?

Yes.  Unfortunately this table isn't particularly detailed or accurate or 
up to date.   The main gap, I think, is versioned objects.

Are there specfiic parts of the S3 API that are missing that you need?  
That sort of info is very helpful for prioritizing effort...

sage

Re: Fwd: S3 API Compatibility support

[M Ranga Swami Reddy]

Thanks for quick reply.
Yes,  versioned object - missing in ceph ATM
Iam looking for: bucket lifecylce (get/put/delete), bucket location,
put object notification and object restore (ie versioned object) S3
API support.
Please let me now any of the above work is in progress or some one
planned to work on.

Thanks
Swami


On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>>
>> Link: http://ceph.com/docs/master/radosgw/s3/
>>
>> Is there plan to support the ?n supported item in the above table?
>> or
>> Any working on this?
>
> Yes.  Unfortunately this table isn't particularly detailed or accurate or
> up to date.   The main gap, I think, is versioned objects.
>
> Are there specfiic parts of the S3 API that are missing that you need?
> That sort of info is very helpful for prioritizing effort...
>
> sage

Re: Fwd: S3 API Compatibility support

[Yehuda Sadeh]

On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
<swamireddy@gmail.com> wrote:
> Thanks for quick reply.
> Yes,  versioned object - missing in ceph ATM
> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
> put object notification and object restore (ie versioned object) S3
> API support.
> Please let me now any of the above work is in progress or some one
> planned to work on.


I opened an issue for bucket lifecycle (we already had an issue open
for object expiration though). We do have bucket location already
(part of the multi-region feature). Object versioning is definitely on
our backlog and one that we'll hopefully implement sooner rather
later.
With regard to object notification, it'll require having a
notification service which is a bit out of the scope. Integrating the
gateway with such a service whouldn't be hard, but we'll need to have
that first.

Yehuda

>
> Thanks
> Swami
>
>
> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>>>
>>> Link: http://ceph.com/docs/master/radosgw/s3/
>>>
>>> Is there plan to support the ?n supported item in the above table?
>>> or
>>> Any working on this?
>>
>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
>> up to date.   The main gap, I think, is versioned objects.
>>
>> Are there specfiic parts of the S3 API that are missing that you need?
>> That sort of info is very helpful for prioritizing effort...
>>
>> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: S3 API Compatibility support

[Robin H. Johnson]

On Fri, Jul 25, 2014 at 08:49:25AM -0700,  Sage Weil wrote:
> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
> > Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
> > 
> > Link: http://ceph.com/docs/master/radosgw/s3/
> > 
> > Is there plan to support the ?n supported item in the above table?
> > or
> > Any working on this?
> 
> Yes.  Unfortunately this table isn't particularly detailed or accurate or 
> up to date.   The main gap, I think, is versioned objects.
> 
> Are there specfiic parts of the S3 API that are missing that you need?  
> That sort of info is very helpful for prioritizing effort...
Fine-grained object policies would be useful here, w/ users+groups
brought in from keystone.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: Fwd: S3 API Compatibility support

[M Ranga Swami Reddy]

Good no know the details. Can you please share the issue ID for bucket
lifecycle? My team also could start help here.
Regarding the notification - Do we have the issue ID?
Yes, the object versioning will be backlog one - I strongly feel we
start working on this asap.

Thanks
Swami

On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
> <swamireddy@gmail.com> wrote:
>> Thanks for quick reply.
>> Yes,  versioned object - missing in ceph ATM
>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
>> put object notification and object restore (ie versioned object) S3
>> API support.
>> Please let me now any of the above work is in progress or some one
>> planned to work on.
>
>
> I opened an issue for bucket lifecycle (we already had an issue open
> for object expiration though). We do have bucket location already
> (part of the multi-region feature). Object versioning is definitely on
> our backlog and one that we'll hopefully implement sooner rather
> later.
> With regard to object notification, it'll require having a
> notification service which is a bit out of the scope. Integrating the
> gateway with such a service whouldn't be hard, but we'll need to have
> that first.
>
> Yehuda
>
>>
>> Thanks
>> Swami
>>
>>
>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>>>>
>>>> Link: http://ceph.com/docs/master/radosgw/s3/
>>>>
>>>> Is there plan to support the ?n supported item in the above table?
>>>> or
>>>> Any working on this?
>>>
>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
>>> up to date.   The main gap, I think, is versioned objects.
>>>
>>> Are there specfiic parts of the S3 API that are missing that you need?
>>> That sort of info is very helpful for prioritizing effort...
>>>
>>> sage
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: S3 API Compatibility support

[M Ranga Swami Reddy]

Can you please share the details on coverage of  the ceph/s3-tests suite?
(mean how many S3 compatibility APIs tests performed with s3-tests suite)

Thanks
Swami

On Sun, Jul 27, 2014 at 1:24 PM, M Ranga Swami Reddy
<swamireddy@gmail.com> wrote:
> Good no know the details. Can you please share the issue ID for bucket
> lifecycle? My team also could start help here.
> Regarding the notification - Do we have the issue ID?
> Yes, the object versioning will be backlog one - I strongly feel we
> start working on this asap.
>
> Thanks
> Swami
>
> On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
>> <swamireddy@gmail.com> wrote:
>>> Thanks for quick reply.
>>> Yes,  versioned object - missing in ceph ATM
>>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
>>> put object notification and object restore (ie versioned object) S3
>>> API support.
>>> Please let me now any of the above work is in progress or some one
>>> planned to work on.
>>
>>
>> I opened an issue for bucket lifecycle (we already had an issue open
>> for object expiration though). We do have bucket location already
>> (part of the multi-region feature). Object versioning is definitely on
>> our backlog and one that we'll hopefully implement sooner rather
>> later.
>> With regard to object notification, it'll require having a
>> notification service which is a bit out of the scope. Integrating the
>> gateway with such a service whouldn't be hard, but we'll need to have
>> that first.
>>
>> Yehuda
>>
>>>
>>> Thanks
>>> Swami
>>>
>>>
>>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
>>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>>>>>
>>>>> Link: http://ceph.com/docs/master/radosgw/s3/
>>>>>
>>>>> Is there plan to support the ?n supported item in the above table?
>>>>> or
>>>>> Any working on this?
>>>>
>>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
>>>> up to date.   The main gap, I think, is versioned objects.
>>>>
>>>> Are there specfiic parts of the S3 API that are missing that you need?
>>>> That sort of info is very helpful for prioritizing effort...
>>>>
>>>> sage
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: S3 API Compatibility support

[Yehuda Sadeh]

Bucket lifecycle:
http://tracker.ceph.com/issues/8929

Bucket notification:
http://tracker.ceph.com/issues/8956

On Sun, Jul 27, 2014 at 12:54 AM, M Ranga Swami Reddy
<swamireddy@gmail.com> wrote:
> Good no know the details. Can you please share the issue ID for bucket
> lifecycle? My team also could start help here.
> Regarding the notification - Do we have the issue ID?
> Yes, the object versioning will be backlog one - I strongly feel we
> start working on this asap.
>
> Thanks
> Swami
>
> On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
>> <swamireddy@gmail.com> wrote:
>>> Thanks for quick reply.
>>> Yes,  versioned object - missing in ceph ATM
>>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
>>> put object notification and object restore (ie versioned object) S3
>>> API support.
>>> Please let me now any of the above work is in progress or some one
>>> planned to work on.
>>
>>
>> I opened an issue for bucket lifecycle (we already had an issue open
>> for object expiration though). We do have bucket location already
>> (part of the multi-region feature). Object versioning is definitely on
>> our backlog and one that we'll hopefully implement sooner rather
>> later.
>> With regard to object notification, it'll require having a
>> notification service which is a bit out of the scope. Integrating the
>> gateway with such a service whouldn't be hard, but we'll need to have
>> that first.
>>
>> Yehuda
>>
>>>
>>> Thanks
>>> Swami
>>>
>>>
>>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
>>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>>>>>
>>>>> Link: http://ceph.com/docs/master/radosgw/s3/
>>>>>
>>>>> Is there plan to support the ?n supported item in the above table?
>>>>> or
>>>>> Any working on this?
>>>>
>>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
>>>> up to date.   The main gap, I think, is versioned objects.
>>>>
>>>> Are there specfiic parts of the S3 API that are missing that you need?
>>>> That sort of info is very helpful for prioritizing effort...
>>>>
>>>> sage
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: S3 API Compatibility support

[Yehuda Sadeh]

There's no good enumeration of the actual apis and features that are
covered by this test. It's definitely a worthy cause to pursue though,
and one that could be done with the help of a willing community
member. The s3 test suite can be found here:

https://github.com/ceph/s3-tests

Thanks,
Yehuda

On Mon, Jul 28, 2014 at 6:05 AM, M Ranga Swami Reddy
<swamireddy@gmail.com> wrote:
> Can you please share the details on coverage of  the ceph/s3-tests suite?
> (mean how many S3 compatibility APIs tests performed with s3-tests suite)
>
> Thanks
> Swami
>
> On Sun, Jul 27, 2014 at 1:24 PM, M Ranga Swami Reddy
> <swamireddy@gmail.com> wrote:
>> Good no know the details. Can you please share the issue ID for bucket
>> lifecycle? My team also could start help here.
>> Regarding the notification - Do we have the issue ID?
>> Yes, the object versioning will be backlog one - I strongly feel we
>> start working on this asap.
>>
>> Thanks
>> Swami
>>
>> On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>>> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
>>> <swamireddy@gmail.com> wrote:
>>>> Thanks for quick reply.
>>>> Yes,  versioned object - missing in ceph ATM
>>>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
>>>> put object notification and object restore (ie versioned object) S3
>>>> API support.
>>>> Please let me now any of the above work is in progress or some one
>>>> planned to work on.
>>>
>>>
>>> I opened an issue for bucket lifecycle (we already had an issue open
>>> for object expiration though). We do have bucket location already
>>> (part of the multi-region feature). Object versioning is definitely on
>>> our backlog and one that we'll hopefully implement sooner rather
>>> later.
>>> With regard to object notification, it'll require having a
>>> notification service which is a bit out of the scope. Integrating the
>>> gateway with such a service whouldn't be hard, but we'll need to have
>>> that first.
>>>
>>> Yehuda
>>>
>>>>
>>>> Thanks
>>>> Swami
>>>>
>>>>
>>>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
>>>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>>>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>>>>>>
>>>>>> Link: http://ceph.com/docs/master/radosgw/s3/
>>>>>>
>>>>>> Is there plan to support the ?n supported item in the above table?
>>>>>> or
>>>>>> Any working on this?
>>>>>
>>>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
>>>>> up to date.   The main gap, I think, is versioned objects.
>>>>>
>>>>> Are there specfiic parts of the S3 API that are missing that you need?
>>>>> That sort of info is very helpful for prioritizing effort...
>>>>>
>>>>> sage
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: S3 API Compatibility support

[M Ranga Swami Reddy]

Hi Yehuda,
I tried to test the get bucket location API, but got an error as
below. Is this known issue? Could you please confirm.  Thanks,Swami
====
In [14]: b.get_location()
send: 'GET /some-bucket-from-libs3/?
location HTTP/1.1\r\nHost: 192.168.122.61\r\nAccept-Encoding:
identity\r\nDate: Fri, 01 Aug 2014 12:02:22 GMT\r\nContent-Length:
0\r\nAuthorization: AWS
cec157b52253463b91547e0a73369960:sT5aw9yXantfAafRn/6x+IgI1pk=\r\nUser-Agent:
Boto/2.20.1 Python/2.7.6 Linux/3.13.0-29-generic\r\n\r\n'
reply: ''
send: 'GET /some-bucket-from-libs3/?location HTTP/1.1\r\nHost:
192.168.122.61\r\nAccept-Encoding: identity\r\nDate: Fri, 01 Aug 2014
12:02:22 GMT\r\nContent-Length: 0\r\nAuthorization: AWS
cec157b52253463b91547e0a73369960:sT5aw9yXantfAafRn/6x+IgI1pk=\r\nUser-Agent:
Boto/2.20.1 Python/2.7.6 Linux/3.13.0-29-generic\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Fri, 01 Aug 2014 12:02:23 GMT
header: Server: Apache/2.4.7 (Ubuntu)
header: Vary: Accept-Encoding
header: Transfer-Encoding: chunked
header: Content-Type: application/xml
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-14-7c597523523d> in <module>()
----> 1 b.get_location()

/usr/lib/python2.7/dist-packages/boto/s3/bucket.pyc in get_location(self)
   1005             h = handler.XmlHandler(rs, self)
   1006             xml.sax.parseString(body, h)
-> 1007             return rs.LocationConstraint
   1008         else:
   1009             raise self.connection.provider.storage_response_error(

AttributeError: 'ResultSet' object has no attribute 'LocationConstraint'
===

On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
> <swamireddy@gmail.com> wrote:
>> Thanks for quick reply.
>> Yes,  versioned object - missing in ceph ATM
>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
>> put object notification and object restore (ie versioned object) S3
>> API support.
>> Please let me now any of the above work is in progress or some one
>> planned to work on.
>
>
> I opened an issue for bucket lifecycle (we already had an issue open
> for object expiration though). We do have bucket location already
> (part of the multi-region feature). Object versioning is definitely on
> our backlog and one that we'll hopefully implement sooner rather
> later.
> With regard to object notification, it'll require having a
> notification service which is a bit out of the scope. Integrating the
> gateway with such a service whouldn't be hard, but we'll need to have
> that first.
>
> Yehuda
>
>>
>> Thanks
>> Swami
>>
>>
>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>>>>
>>>> Link: http://ceph.com/docs/master/radosgw/s3/
>>>>
>>>> Is there plan to support the ?n supported item in the above table?
>>>> or
>>>> Any working on this?
>>>
>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
>>> up to date.   The main gap, I think, is versioned objects.
>>>
>>> Are there specfiic parts of the S3 API that are missing that you need?
>>> That sort of info is very helpful for prioritizing effort...
>>>
>>> sage
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: S3 API Compatibility support

[M Ranga Swami Reddy]

Hi Yehuda,
Can I please get any suggestion on the "get bucket_location" API
error?  Please share any info, which can help us to debug more here.

Thanks
Swami

On Mon, Aug 4, 2014 at 4:44 PM, M Ranga Swami Reddy
<swamireddy@gmail.com> wrote:
> Hi Yehuda,
> I tried to test the get bucket location API, but got an error as
> below. Is this known issue? Could you please confirm.  Thanks,Swami
> ====
> In [14]: b.get_location()
> send: 'GET /some-bucket-from-libs3/?
> location HTTP/1.1\r\nHost: 192.168.122.61\r\nAccept-Encoding:
> identity\r\nDate: Fri, 01 Aug 2014 12:02:22 GMT\r\nContent-Length:
> 0\r\nAuthorization: AWS
> cec157b52253463b91547e0a73369960:sT5aw9yXantfAafRn/6x+IgI1pk=\r\nUser-Agent:
> Boto/2.20.1 Python/2.7.6 Linux/3.13.0-29-generic\r\n\r\n'
> reply: ''
> send: 'GET /some-bucket-from-libs3/?location HTTP/1.1\r\nHost:
> 192.168.122.61\r\nAccept-Encoding: identity\r\nDate: Fri, 01 Aug 2014
> 12:02:22 GMT\r\nContent-Length: 0\r\nAuthorization: AWS
> cec157b52253463b91547e0a73369960:sT5aw9yXantfAafRn/6x+IgI1pk=\r\nUser-Agent:
> Boto/2.20.1 Python/2.7.6 Linux/3.13.0-29-generic\r\n\r\n'
> reply: 'HTTP/1.1 200 OK\r\n'
> header: Date: Fri, 01 Aug 2014 12:02:23 GMT
> header: Server: Apache/2.4.7 (Ubuntu)
> header: Vary: Accept-Encoding
> header: Transfer-Encoding: chunked
> header: Content-Type: application/xml
> ---------------------------------------------------------------------------
> AttributeError                            Traceback (most recent call last)
> <ipython-input-14-7c597523523d> in <module>()
> ----> 1 b.get_location()
>
> /usr/lib/python2.7/dist-packages/boto/s3/bucket.pyc in get_location(self)
>    1005             h = handler.XmlHandler(rs, self)
>    1006             xml.sax.parseString(body, h)
> -> 1007             return rs.LocationConstraint
>    1008         else:
>    1009             raise self.connection.provider.storage_response_error(
>
> AttributeError: 'ResultSet' object has no attribute 'LocationConstraint'
> ===
>
> On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
>> <swamireddy@gmail.com> wrote:
>>> Thanks for quick reply.
>>> Yes,  versioned object - missing in ceph ATM
>>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
>>> put object notification and object restore (ie versioned object) S3
>>> API support.
>>> Please let me now any of the above work is in progress or some one
>>> planned to work on.
>>
>>
>> I opened an issue for bucket lifecycle (we already had an issue open
>> for object expiration though). We do have bucket location already
>> (part of the multi-region feature). Object versioning is definitely on
>> our backlog and one that we'll hopefully implement sooner rather
>> later.
>> With regard to object notification, it'll require having a
>> notification service which is a bit out of the scope. Integrating the
>> gateway with such a service whouldn't be hard, but we'll need to have
>> that first.
>>
>> Yehuda
>>
>>>
>>> Thanks
>>> Swami
>>>
>>>
>>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
>>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>>>>>
>>>>> Link: http://ceph.com/docs/master/radosgw/s3/
>>>>>
>>>>> Is there plan to support the ?n supported item in the above table?
>>>>> or
>>>>> Any working on this?
>>>>
>>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
>>>> up to date.   The main gap, I think, is versioned objects.
>>>>
>>>> Are there specfiic parts of the S3 API that are missing that you need?
>>>> That sort of info is very helpful for prioritizing effort...
>>>>
>>>> sage
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: S3 API Compatibility support

[Yehuda Sadeh]

At the moment we don't support this api call. Adding it should be
pretty easy, and it's definitely something that could be done with the
help of a willing community member.

Yehuda

On Tue, Aug 12, 2014 at 9:58 AM, M Ranga Swami Reddy
<swamireddy@gmail.com> wrote:
> Hi Yehuda,
> Can I please get any suggestion on the "get bucket_location" API
> error?  Please share any info, which can help us to debug more here.
>
> Thanks
> Swami
>
> On Mon, Aug 4, 2014 at 4:44 PM, M Ranga Swami Reddy
> <swamireddy@gmail.com> wrote:
>> Hi Yehuda,
>> I tried to test the get bucket location API, but got an error as
>> below. Is this known issue? Could you please confirm.  Thanks,Swami
>> ====
>> In [14]: b.get_location()
>> send: 'GET /some-bucket-from-libs3/?
>> location HTTP/1.1\r\nHost: 192.168.122.61\r\nAccept-Encoding:
>> identity\r\nDate: Fri, 01 Aug 2014 12:02:22 GMT\r\nContent-Length:
>> 0\r\nAuthorization: AWS
>> cec157b52253463b91547e0a73369960:sT5aw9yXantfAafRn/6x+IgI1pk=\r\nUser-Agent:
>> Boto/2.20.1 Python/2.7.6 Linux/3.13.0-29-generic\r\n\r\n'
>> reply: ''
>> send: 'GET /some-bucket-from-libs3/?location HTTP/1.1\r\nHost:
>> 192.168.122.61\r\nAccept-Encoding: identity\r\nDate: Fri, 01 Aug 2014
>> 12:02:22 GMT\r\nContent-Length: 0\r\nAuthorization: AWS
>> cec157b52253463b91547e0a73369960:sT5aw9yXantfAafRn/6x+IgI1pk=\r\nUser-Agent:
>> Boto/2.20.1 Python/2.7.6 Linux/3.13.0-29-generic\r\n\r\n'
>> reply: 'HTTP/1.1 200 OK\r\n'
>> header: Date: Fri, 01 Aug 2014 12:02:23 GMT
>> header: Server: Apache/2.4.7 (Ubuntu)
>> header: Vary: Accept-Encoding
>> header: Transfer-Encoding: chunked
>> header: Content-Type: application/xml
>> ---------------------------------------------------------------------------
>> AttributeError                            Traceback (most recent call last)
>> <ipython-input-14-7c597523523d> in <module>()
>> ----> 1 b.get_location()
>>
>> /usr/lib/python2.7/dist-packages/boto/s3/bucket.pyc in get_location(self)
>>    1005             h = handler.XmlHandler(rs, self)
>>    1006             xml.sax.parseString(body, h)
>> -> 1007             return rs.LocationConstraint
>>    1008         else:
>>    1009             raise self.connection.provider.storage_response_error(
>>
>> AttributeError: 'ResultSet' object has no attribute 'LocationConstraint'
>> ===
>>
>> On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>>> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
>>> <swamireddy@gmail.com> wrote:
>>>> Thanks for quick reply.
>>>> Yes,  versioned object - missing in ceph ATM
>>>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
>>>> put object notification and object restore (ie versioned object) S3
>>>> API support.
>>>> Please let me now any of the above work is in progress or some one
>>>> planned to work on.
>>>
>>>
>>> I opened an issue for bucket lifecycle (we already had an issue open
>>> for object expiration though). We do have bucket location already
>>> (part of the multi-region feature). Object versioning is definitely on
>>> our backlog and one that we'll hopefully implement sooner rather
>>> later.
>>> With regard to object notification, it'll require having a
>>> notification service which is a bit out of the scope. Integrating the
>>> gateway with such a service whouldn't be hard, but we'll need to have
>>> that first.
>>>
>>> Yehuda
>>>
>>>>
>>>> Thanks
>>>> Swami
>>>>
>>>>
>>>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
>>>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>>>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>>>>>>
>>>>>> Link: http://ceph.com/docs/master/radosgw/s3/
>>>>>>
>>>>>> Is there plan to support the ?n supported item in the above table?
>>>>>> or
>>>>>> Any working on this?
>>>>>
>>>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
>>>>> up to date.   The main gap, I think, is versioned objects.
>>>>>
>>>>> Are there specfiic parts of the S3 API that are missing that you need?
>>>>> That sort of info is very helpful for prioritizing effort...
>>>>>
>>>>> sage
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: S3 API Compatibility support

[M Ranga Swami Reddy]

Thanks Yehuda. I am testing the S3 APIs suing the openstack+ceph.
Could able test a couple of APIs like put/get/delete bucket (using the
boto python libraries). When I tried for get bucket location I got an
error mentioned in my previous email.

And explored the ceph source code and not able see the appropriate
code/function for "get_bucket_location". But as per the previous email
discuss,
the bucket_location supported as part multi-region support.
Let me know, if I miss something here.

Thanks
Swami

Re: Fwd: S3 API Compatibility support

[Abhishek L]

[-- Attachment #1: Type: text/plain, Size: 779 bytes --]


Yehuda Sadeh writes:

> At the moment we don't support this api call. Adding it should be
> pretty easy, and it's definitely something that could be done with the
> help of a willing community member.
>
> Yehuda
>
Hi,

I have a basic implementation for S3 get bucket location, which returns
the region of the bucket.[1] For the "default" region it returns the
empty string similiar to s3 (which returns empty string for the
"default" us-east region). 

Please let me know whether this is acceptable (or if I have missed
something obvious), and what else needs to be done in order to get this
in. Once I have a review on code I'll try to update the docs & examples
regarding this call.


[1] https://github.com/ceph/ceph/pull/2329

Thanks
-- 
Abhishek

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: Fwd: S3 API Compatibility support

[M Ranga Swami Reddy]

Hi Yehuda,

Could you please check and clarify the below question on object
lifecycle and notification S3 APIs support:

1. To support the bucket lifecycle - we need to support the
moving/deleting the objects/buckets based lifecycle settings.
For ex: If an object lifecyle set as below:
          1. Archive it after 10 days - means move this object to low
cost object storage after 10 days of the creation date.
           2. Remove this object after 90days - mean remove this
object from the low cost object after 90days of creation date.

Q1- Does the ceph support the above concept like moving to low cost
storage and delete from that storage?

2. To support the object notifications:
      - First there should be low cost and high availability storage
with single replica only. If an object created with this type of
object storage,
        There could be chances that object could lose, so if an object
of this type of storage lost, set the notifications.

Q2- Does Ceph support low cost and high availability storage type?

Thanks
Swami







On Tue, Jul 29, 2014 at 1:35 AM, Yehuda Sadeh <yehuda@redhat.com> wrote:
> Bucket lifecycle:
> http://tracker.ceph.com/issues/8929
>
> Bucket notification:
> http://tracker.ceph.com/issues/8956
>
> On Sun, Jul 27, 2014 at 12:54 AM, M Ranga Swami Reddy
> <swamireddy@gmail.com> wrote:
>> Good no know the details. Can you please share the issue ID for bucket
>> lifecycle? My team also could start help here.
>> Regarding the notification - Do we have the issue ID?
>> Yes, the object versioning will be backlog one - I strongly feel we
>> start working on this asap.
>>
>> Thanks
>> Swami
>>
>> On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>>> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
>>> <swamireddy@gmail.com> wrote:
>>>> Thanks for quick reply.
>>>> Yes,  versioned object - missing in ceph ATM
>>>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
>>>> put object notification and object restore (ie versioned object) S3
>>>> API support.
>>>> Please let me now any of the above work is in progress or some one
>>>> planned to work on.
>>>
>>>
>>> I opened an issue for bucket lifecycle (we already had an issue open
>>> for object expiration though). We do have bucket location already
>>> (part of the multi-region feature). Object versioning is definitely on
>>> our backlog and one that we'll hopefully implement sooner rather
>>> later.
>>> With regard to object notification, it'll require having a
>>> notification service which is a bit out of the scope. Integrating the
>>> gateway with such a service whouldn't be hard, but we'll need to have
>>> that first.
>>>
>>> Yehuda
>>>
>>>>
>>>> Thanks
>>>> Swami
>>>>
>>>>
>>>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
>>>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>>>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>>>>>>
>>>>>> Link: http://ceph.com/docs/master/radosgw/s3/
>>>>>>
>>>>>> Is there plan to support the ?n supported item in the above table?
>>>>>> or
>>>>>> Any working on this?
>>>>>
>>>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
>>>>> up to date.   The main gap, I think, is versioned objects.
>>>>>
>>>>> Are there specfiic parts of the S3 API that are missing that you need?
>>>>> That sort of info is very helpful for prioritizing effort...
>>>>>
>>>>> sage
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: S3 API Compatibility support

[M Ranga Swami Reddy]

Hi ,

Could you please check and clarify the below question on object
lifecycle and notification S3 APIs support:

1. To support the bucket lifecycle - we need to support the
moving/deleting the objects/buckets based lifecycle settings.
For ex: If an object lifecyle set as below:
          1. Archive it after 10 days - means move this object to low
cost object storage after 10 days of the creation date.
           2. Remove this object after 90days - mean remove this
object from the low cost object after 90days of creation date.

Q1- Does the ceph support the above concept like moving to low cost
storage and delete from that storage?

2. To support the object notifications:
      - First there should be low cost and high availability storage
with single replica only. If an object created with this type of
object storage,
        There could be chances that object could lose, so if an object
of this type of storage lost, set the notifications.

Q2- Does Ceph support low cost and high availability storage type?

Thanks

On Fri, Sep 12, 2014 at 8:00 PM, M Ranga Swami Reddy
<swamireddy@gmail.com> wrote:
> Hi Yehuda,
>
> Could you please check and clarify the below question on object
> lifecycle and notification S3 APIs support:
>
> 1. To support the bucket lifecycle - we need to support the
> moving/deleting the objects/buckets based lifecycle settings.
> For ex: If an object lifecyle set as below:
>           1. Archive it after 10 days - means move this object to low
> cost object storage after 10 days of the creation date.
>            2. Remove this object after 90days - mean remove this
> object from the low cost object after 90days of creation date.
>
> Q1- Does the ceph support the above concept like moving to low cost
> storage and delete from that storage?
>
> 2. To support the object notifications:
>       - First there should be low cost and high availability storage
> with single replica only. If an object created with this type of
> object storage,
>         There could be chances that object could lose, so if an object
> of this type of storage lost, set the notifications.
>
> Q2- Does Ceph support low cost and high availability storage type?
>
> Thanks
> Swami
>
>
>
>
>
>
>
> On Tue, Jul 29, 2014 at 1:35 AM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>> Bucket lifecycle:
>> http://tracker.ceph.com/issues/8929
>>
>> Bucket notification:
>> http://tracker.ceph.com/issues/8956
>>
>> On Sun, Jul 27, 2014 at 12:54 AM, M Ranga Swami Reddy
>> <swamireddy@gmail.com> wrote:
>>> Good no know the details. Can you please share the issue ID for bucket
>>> lifecycle? My team also could start help here.
>>> Regarding the notification - Do we have the issue ID?
>>> Yes, the object versioning will be backlog one - I strongly feel we
>>> start working on this asap.
>>>
>>> Thanks
>>> Swami
>>>
>>> On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>>>> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
>>>> <swamireddy@gmail.com> wrote:
>>>>> Thanks for quick reply.
>>>>> Yes,  versioned object - missing in ceph ATM
>>>>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
>>>>> put object notification and object restore (ie versioned object) S3
>>>>> API support.
>>>>> Please let me now any of the above work is in progress or some one
>>>>> planned to work on.
>>>>
>>>>
>>>> I opened an issue for bucket lifecycle (we already had an issue open
>>>> for object expiration though). We do have bucket location already
>>>> (part of the multi-region feature). Object versioning is definitely on
>>>> our backlog and one that we'll hopefully implement sooner rather
>>>> later.
>>>> With regard to object notification, it'll require having a
>>>> notification service which is a bit out of the scope. Integrating the
>>>> gateway with such a service whouldn't be hard, but we'll need to have
>>>> that first.
>>>>
>>>> Yehuda
>>>>
>>>>>
>>>>> Thanks
>>>>> Swami
>>>>>
>>>>>
>>>>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
>>>>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>>>>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>>>>>>>
>>>>>>> Link: http://ceph.com/docs/master/radosgw/s3/
>>>>>>>
>>>>>>> Is there plan to support the ?n supported item in the above table?
>>>>>>> or
>>>>>>> Any working on this?
>>>>>>
>>>>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
>>>>>> up to date.   The main gap, I think, is versioned objects.
>>>>>>
>>>>>> Are there specfiic parts of the S3 API that are missing that you need?
>>>>>> That sort of info is very helpful for prioritizing effort...
>>>>>>
>>>>>> sage
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: S3 API Compatibility support

[M Ranga Swami Reddy]

Hi Sage,
Could you please advise, if Ceph support the low cost object
storages(like Amazon Glacier or RRS) for archiving objects like log
file etc.?

Thanks
Swami

On Thu, Sep 18, 2014 at 6:20 PM, M Ranga Swami Reddy
<swamireddy@gmail.com> wrote:
> Hi ,
>
> Could you please check and clarify the below question on object
> lifecycle and notification S3 APIs support:
>
> 1. To support the bucket lifecycle - we need to support the
> moving/deleting the objects/buckets based lifecycle settings.
> For ex: If an object lifecyle set as below:
>           1. Archive it after 10 days - means move this object to low
> cost object storage after 10 days of the creation date.
>            2. Remove this object after 90days - mean remove this
> object from the low cost object after 90days of creation date.
>
> Q1- Does the ceph support the above concept like moving to low cost
> storage and delete from that storage?
>
> 2. To support the object notifications:
>       - First there should be low cost and high availability storage
> with single replica only. If an object created with this type of
> object storage,
>         There could be chances that object could lose, so if an object
> of this type of storage lost, set the notifications.
>
> Q2- Does Ceph support low cost and high availability storage type?
>
> Thanks
>
> On Fri, Sep 12, 2014 at 8:00 PM, M Ranga Swami Reddy
> <swamireddy@gmail.com> wrote:
>> Hi Yehuda,
>>
>> Could you please check and clarify the below question on object
>> lifecycle and notification S3 APIs support:
>>
>> 1. To support the bucket lifecycle - we need to support the
>> moving/deleting the objects/buckets based lifecycle settings.
>> For ex: If an object lifecyle set as below:
>>           1. Archive it after 10 days - means move this object to low
>> cost object storage after 10 days of the creation date.
>>            2. Remove this object after 90days - mean remove this
>> object from the low cost object after 90days of creation date.
>>
>> Q1- Does the ceph support the above concept like moving to low cost
>> storage and delete from that storage?
>>
>> 2. To support the object notifications:
>>       - First there should be low cost and high availability storage
>> with single replica only. If an object created with this type of
>> object storage,
>>         There could be chances that object could lose, so if an object
>> of this type of storage lost, set the notifications.
>>
>> Q2- Does Ceph support low cost and high availability storage type?
>>
>> Thanks
>> Swami
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Jul 29, 2014 at 1:35 AM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>>> Bucket lifecycle:
>>> http://tracker.ceph.com/issues/8929
>>>
>>> Bucket notification:
>>> http://tracker.ceph.com/issues/8956
>>>
>>> On Sun, Jul 27, 2014 at 12:54 AM, M Ranga Swami Reddy
>>> <swamireddy@gmail.com> wrote:
>>>> Good no know the details. Can you please share the issue ID for bucket
>>>> lifecycle? My team also could start help here.
>>>> Regarding the notification - Do we have the issue ID?
>>>> Yes, the object versioning will be backlog one - I strongly feel we
>>>> start working on this asap.
>>>>
>>>> Thanks
>>>> Swami
>>>>
>>>> On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>>>>> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
>>>>> <swamireddy@gmail.com> wrote:
>>>>>> Thanks for quick reply.
>>>>>> Yes,  versioned object - missing in ceph ATM
>>>>>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
>>>>>> put object notification and object restore (ie versioned object) S3
>>>>>> API support.
>>>>>> Please let me now any of the above work is in progress or some one
>>>>>> planned to work on.
>>>>>
>>>>>
>>>>> I opened an issue for bucket lifecycle (we already had an issue open
>>>>> for object expiration though). We do have bucket location already
>>>>> (part of the multi-region feature). Object versioning is definitely on
>>>>> our backlog and one that we'll hopefully implement sooner rather
>>>>> later.
>>>>> With regard to object notification, it'll require having a
>>>>> notification service which is a bit out of the scope. Integrating the
>>>>> gateway with such a service whouldn't be hard, but we'll need to have
>>>>> that first.
>>>>>
>>>>> Yehuda
>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Swami
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
>>>>>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>>>>>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>>>>>>>>
>>>>>>>> Link: http://ceph.com/docs/master/radosgw/s3/
>>>>>>>>
>>>>>>>> Is there plan to support the ?n supported item in the above table?
>>>>>>>> or
>>>>>>>> Any working on this?
>>>>>>>
>>>>>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
>>>>>>> up to date.   The main gap, I think, is versioned objects.
>>>>>>>
>>>>>>> Are there specfiic parts of the S3 API that are missing that you need?
>>>>>>> That sort of info is very helpful for prioritizing effort...
>>>>>>>
>>>>>>> sage
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: S3 API Compatibility support

[Sage Weil]

On Fri, 19 Sep 2014, M Ranga Swami Reddy wrote:
> Hi Sage,
> Could you please advise, if Ceph support the low cost object
> storages(like Amazon Glacier or RRS) for archiving objects like log
> file etc.?

Ceph doesn't interact at all with AWS services like Glacier, if that's 
what you mean.

For RRS, though, I assume you mean the ability to create buckets with 
reduced redundancy with radosgw?  That is supported, although not quite 
the way AWS does it.  You can create different pools that back RGW 
buckets, and each bucket is stored in one of those pools.  So you could 
make one of them 2x instead of 3x, or use an erasure code of your choice.

What isn't currently supported is the ability to reduce the redundancy of 
individual objects in a bucket.  I don't think there is anything 
architecturally preventing that, but it is not implemented or supported.  

When we look at the S3 archival features in more detail (soon!) I'm sure 
this will come up!  The current plan is to address object versioning 
first.  That is, unless a developer surfaces who wants to start hacking on 
this right away...

sage



> 
> Thanks
> Swami
> 
> On Thu, Sep 18, 2014 at 6:20 PM, M Ranga Swami Reddy
> <swamireddy@gmail.com> wrote:
> > Hi ,
> >
> > Could you please check and clarify the below question on object
> > lifecycle and notification S3 APIs support:
> >
> > 1. To support the bucket lifecycle - we need to support the
> > moving/deleting the objects/buckets based lifecycle settings.
> > For ex: If an object lifecyle set as below:
> >           1. Archive it after 10 days - means move this object to low
> > cost object storage after 10 days of the creation date.
> >            2. Remove this object after 90days - mean remove this
> > object from the low cost object after 90days of creation date.
> >
> > Q1- Does the ceph support the above concept like moving to low cost
> > storage and delete from that storage?
> >
> > 2. To support the object notifications:
> >       - First there should be low cost and high availability storage
> > with single replica only. If an object created with this type of
> > object storage,
> >         There could be chances that object could lose, so if an object
> > of this type of storage lost, set the notifications.
> >
> > Q2- Does Ceph support low cost and high availability storage type?
> >
> > Thanks
> >
> > On Fri, Sep 12, 2014 at 8:00 PM, M Ranga Swami Reddy
> > <swamireddy@gmail.com> wrote:
> >> Hi Yehuda,
> >>
> >> Could you please check and clarify the below question on object
> >> lifecycle and notification S3 APIs support:
> >>
> >> 1. To support the bucket lifecycle - we need to support the
> >> moving/deleting the objects/buckets based lifecycle settings.
> >> For ex: If an object lifecyle set as below:
> >>           1. Archive it after 10 days - means move this object to low
> >> cost object storage after 10 days of the creation date.
> >>            2. Remove this object after 90days - mean remove this
> >> object from the low cost object after 90days of creation date.
> >>
> >> Q1- Does the ceph support the above concept like moving to low cost
> >> storage and delete from that storage?
> >>
> >> 2. To support the object notifications:
> >>       - First there should be low cost and high availability storage
> >> with single replica only. If an object created with this type of
> >> object storage,
> >>         There could be chances that object could lose, so if an object
> >> of this type of storage lost, set the notifications.
> >>
> >> Q2- Does Ceph support low cost and high availability storage type?
> >>
> >> Thanks
> >> Swami
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Tue, Jul 29, 2014 at 1:35 AM, Yehuda Sadeh <yehuda@redhat.com> wrote:
> >>> Bucket lifecycle:
> >>> http://tracker.ceph.com/issues/8929
> >>>
> >>> Bucket notification:
> >>> http://tracker.ceph.com/issues/8956
> >>>
> >>> On Sun, Jul 27, 2014 at 12:54 AM, M Ranga Swami Reddy
> >>> <swamireddy@gmail.com> wrote:
> >>>> Good no know the details. Can you please share the issue ID for bucket
> >>>> lifecycle? My team also could start help here.
> >>>> Regarding the notification - Do we have the issue ID?
> >>>> Yes, the object versioning will be backlog one - I strongly feel we
> >>>> start working on this asap.
> >>>>
> >>>> Thanks
> >>>> Swami
> >>>>
> >>>> On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
> >>>>> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
> >>>>> <swamireddy@gmail.com> wrote:
> >>>>>> Thanks for quick reply.
> >>>>>> Yes,  versioned object - missing in ceph ATM
> >>>>>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
> >>>>>> put object notification and object restore (ie versioned object) S3
> >>>>>> API support.
> >>>>>> Please let me now any of the above work is in progress or some one
> >>>>>> planned to work on.
> >>>>>
> >>>>>
> >>>>> I opened an issue for bucket lifecycle (we already had an issue open
> >>>>> for object expiration though). We do have bucket location already
> >>>>> (part of the multi-region feature). Object versioning is definitely on
> >>>>> our backlog and one that we'll hopefully implement sooner rather
> >>>>> later.
> >>>>> With regard to object notification, it'll require having a
> >>>>> notification service which is a bit out of the scope. Integrating the
> >>>>> gateway with such a service whouldn't be hard, but we'll need to have
> >>>>> that first.
> >>>>>
> >>>>> Yehuda
> >>>>>
> >>>>>>
> >>>>>> Thanks
> >>>>>> Swami
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
> >>>>>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
> >>>>>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
> >>>>>>>>
> >>>>>>>> Link: http://ceph.com/docs/master/radosgw/s3/
> >>>>>>>>
> >>>>>>>> Is there plan to support the ?n supported item in the above table?
> >>>>>>>> or
> >>>>>>>> Any working on this?
> >>>>>>>
> >>>>>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
> >>>>>>> up to date.   The main gap, I think, is versioned objects.
> >>>>>>>
> >>>>>>> Are there specfiic parts of the S3 API that are missing that you need?
> >>>>>>> That sort of info is very helpful for prioritizing effort...
> >>>>>>>
> >>>>>>> sage
> >>>>>> --
> >>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >>>>>> the body of a message to majordomo@vger.kernel.org
> >>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Fwd: S3 API Compatibility support

[M Ranga Swami Reddy]

Hi Sage,
Thanks for quick reply.

>Ceph doesn't interact at all with AWS services like Glacier, if that's

No. I meant that - Ceph interaction with a glacier and RRS type of
storages along with currently used OSD (or standard storage).

>what you mean.
>For RRS, though, I assume you mean the ability to create buckets with
>reduced redundancy with radosgw?  That is supported, although not quite
>the way AWS does it.  You can create different pools that back RGW
>buckets, and each bucket is stored in one of those pools.  So you could
>make one of them 2x instead of 3x, or use an erasure code of your choice.

Yes, we can confiure ceph to use 2x replicas, which will look like
reduced redundancy, but AWS uses a separate RRS storage-low cost
(instead of
standard) storage for this purpose. I am checking, if we could
similarly in ceph too.

>What isn't currently supported is the ability to reduce the redundancy of
>individual objects in a bucket.  I don't think there is anything
>architecturally preventing that, but it is not implemented or supported.

OK. Do we have the issue id for the above? Else, we can file one. Please advise.

>When we look at the S3 archival features in more detail (soon!) I'm sure
>this will come up!  The current plan is to address object versioning
>first.  That is, unless a developer surfaces who wants to start hacking on
>this right away...

Great to know this. Even we are keen with S3 support in Ceph and we
are happy support you here.

Thanks
Swami

On Fri, Sep 19, 2014 at 11:08 AM, Sage Weil <sweil@redhat.com> wrote:
> On Fri, 19 Sep 2014, M Ranga Swami Reddy wrote:
>> Hi Sage,
>> Could you please advise, if Ceph support the low cost object
>> storages(like Amazon Glacier or RRS) for archiving objects like log
>> file etc.?
>
> Ceph doesn't interact at all with AWS services like Glacier, if that's
> what you mean.
>
> For RRS, though, I assume you mean the ability to create buckets with
> reduced redundancy with radosgw?  That is supported, although not quite
> the way AWS does it.  You can create different pools that back RGW
> buckets, and each bucket is stored in one of those pools.  So you could
> make one of them 2x instead of 3x, or use an erasure code of your choice.
>
> What isn't currently supported is the ability to reduce the redundancy of
> individual objects in a bucket.  I don't think there is anything
> architecturally preventing that, but it is not implemented or supported.
>
> When we look at the S3 archival features in more detail (soon!) I'm sure
> this will come up!  The current plan is to address object versioning
> first.  That is, unless a developer surfaces who wants to start hacking on
> this right away...
>
> sage
>
>
>
>>
>> Thanks
>> Swami
>>
>> On Thu, Sep 18, 2014 at 6:20 PM, M Ranga Swami Reddy
>> <swamireddy@gmail.com> wrote:
>> > Hi ,
>> >
>> > Could you please check and clarify the below question on object
>> > lifecycle and notification S3 APIs support:
>> >
>> > 1. To support the bucket lifecycle - we need to support the
>> > moving/deleting the objects/buckets based lifecycle settings.
>> > For ex: If an object lifecyle set as below:
>> >           1. Archive it after 10 days - means move this object to low
>> > cost object storage after 10 days of the creation date.
>> >            2. Remove this object after 90days - mean remove this
>> > object from the low cost object after 90days of creation date.
>> >
>> > Q1- Does the ceph support the above concept like moving to low cost
>> > storage and delete from that storage?
>> >
>> > 2. To support the object notifications:
>> >       - First there should be low cost and high availability storage
>> > with single replica only. If an object created with this type of
>> > object storage,
>> >         There could be chances that object could lose, so if an object
>> > of this type of storage lost, set the notifications.
>> >
>> > Q2- Does Ceph support low cost and high availability storage type?
>> >
>> > Thanks
>> >
>> > On Fri, Sep 12, 2014 at 8:00 PM, M Ranga Swami Reddy
>> > <swamireddy@gmail.com> wrote:
>> >> Hi Yehuda,
>> >>
>> >> Could you please check and clarify the below question on object
>> >> lifecycle and notification S3 APIs support:
>> >>
>> >> 1. To support the bucket lifecycle - we need to support the
>> >> moving/deleting the objects/buckets based lifecycle settings.
>> >> For ex: If an object lifecyle set as below:
>> >>           1. Archive it after 10 days - means move this object to low
>> >> cost object storage after 10 days of the creation date.
>> >>            2. Remove this object after 90days - mean remove this
>> >> object from the low cost object after 90days of creation date.
>> >>
>> >> Q1- Does the ceph support the above concept like moving to low cost
>> >> storage and delete from that storage?
>> >>
>> >> 2. To support the object notifications:
>> >>       - First there should be low cost and high availability storage
>> >> with single replica only. If an object created with this type of
>> >> object storage,
>> >>         There could be chances that object could lose, so if an object
>> >> of this type of storage lost, set the notifications.
>> >>
>> >> Q2- Does Ceph support low cost and high availability storage type?
>> >>
>> >> Thanks
>> >> Swami
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Tue, Jul 29, 2014 at 1:35 AM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>> >>> Bucket lifecycle:
>> >>> http://tracker.ceph.com/issues/8929
>> >>>
>> >>> Bucket notification:
>> >>> http://tracker.ceph.com/issues/8956
>> >>>
>> >>> On Sun, Jul 27, 2014 at 12:54 AM, M Ranga Swami Reddy
>> >>> <swamireddy@gmail.com> wrote:
>> >>>> Good no know the details. Can you please share the issue ID for bucket
>> >>>> lifecycle? My team also could start help here.
>> >>>> Regarding the notification - Do we have the issue ID?
>> >>>> Yes, the object versioning will be backlog one - I strongly feel we
>> >>>> start working on this asap.
>> >>>>
>> >>>> Thanks
>> >>>> Swami
>> >>>>
>> >>>> On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>> >>>>> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
>> >>>>> <swamireddy@gmail.com> wrote:
>> >>>>>> Thanks for quick reply.
>> >>>>>> Yes,  versioned object - missing in ceph ATM
>> >>>>>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
>> >>>>>> put object notification and object restore (ie versioned object) S3
>> >>>>>> API support.
>> >>>>>> Please let me now any of the above work is in progress or some one
>> >>>>>> planned to work on.
>> >>>>>
>> >>>>>
>> >>>>> I opened an issue for bucket lifecycle (we already had an issue open
>> >>>>> for object expiration though). We do have bucket location already
>> >>>>> (part of the multi-region feature). Object versioning is definitely on
>> >>>>> our backlog and one that we'll hopefully implement sooner rather
>> >>>>> later.
>> >>>>> With regard to object notification, it'll require having a
>> >>>>> notification service which is a bit out of the scope. Integrating the
>> >>>>> gateway with such a service whouldn't be hard, but we'll need to have
>> >>>>> that first.
>> >>>>>
>> >>>>> Yehuda
>> >>>>>
>> >>>>>>
>> >>>>>> Thanks
>> >>>>>> Swami
>> >>>>>>
>> >>>>>>
>> >>>>>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
>> >>>>>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>> >>>>>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>> >>>>>>>>
>> >>>>>>>> Link: http://ceph.com/docs/master/radosgw/s3/
>> >>>>>>>>
>> >>>>>>>> Is there plan to support the ?n supported item in the above table?
>> >>>>>>>> or
>> >>>>>>>> Any working on this?
>> >>>>>>>
>> >>>>>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
>> >>>>>>> up to date.   The main gap, I think, is versioned objects.
>> >>>>>>>
>> >>>>>>> Are there specfiic parts of the S3 API that are missing that you need?
>> >>>>>>> That sort of info is very helpful for prioritizing effort...
>> >>>>>>>
>> >>>>>>> sage
>> >>>>>> --
>> >>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> >>>>>> the body of a message to majordomo@vger.kernel.org
>> >>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>

Re: Fwd: S3 API Compatibility support

[Sage Weil]

On Fri, 19 Sep 2014, M Ranga Swami Reddy wrote:
> Hi Sage,
> Thanks for quick reply.
> 
> >what you mean.
> >For RRS, though, I assume you mean the ability to create buckets with
> >reduced redundancy with radosgw?  That is supported, although not quite
> >the way AWS does it.  You can create different pools that back RGW
> >buckets, and each bucket is stored in one of those pools.  So you could
> >make one of them 2x instead of 3x, or use an erasure code of your choice.
> 
> Yes, we can confiure ceph to use 2x replicas, which will look like
> reduced redundancy, but AWS uses a separate RRS storage-low cost
> (instead of
> standard) storage for this purpose. I am checking, if we could
> similarly in ceph too.

What do you mean by "RRS storage-low cost storage"?  My read of the RRS 
numbers is that they simply have a different tier of S3 that runs fewer 
replicas and (probably) cheaper disks.  In radosgw-land, this would just 
be a different rados pool with 2x replicas and (probably) a CRUSH rule 
mapping it to different hardware (with bigger and/or cheaper disks).

> >What isn't currently supported is the ability to reduce the redundancy of
> >individual objects in a bucket.  I don't think there is anything
> >architecturally preventing that, but it is not implemented or supported.
> 
> OK. Do we have the issue id for the above? Else, we can file one. Please advise.

There is the main #4099 issue for object expiration, but there is no real 
detail there.  The plan is (as always) to have equivalent functionality to 
S3.

Do you mind creating a new feature ticket that specifically references the 
ability to move objects to a second storage tier based on policy?  Any 
references to AWS docs about the API or functionality would be helpful in 
the ticket.

> >When we look at the S3 archival features in more detail (soon!) I'm sure
> >this will come up!  The current plan is to address object versioning
> >first.  That is, unless a developer surfaces who wants to start hacking on
> >this right away...
> 
> Great to know this. Even we are keen with S3 support in Ceph and we
> are happy support you here.

Great to hear!

Thanks-
sage


> 
> Thanks
> Swami
> 
> On Fri, Sep 19, 2014 at 11:08 AM, Sage Weil <sweil@redhat.com> wrote:
> > On Fri, 19 Sep 2014, M Ranga Swami Reddy wrote:
> >> Hi Sage,
> >> Could you please advise, if Ceph support the low cost object
> >> storages(like Amazon Glacier or RRS) for archiving objects like log
> >> file etc.?
> >
> > Ceph doesn't interact at all with AWS services like Glacier, if that's
> > what you mean.
> >
> > For RRS, though, I assume you mean the ability to create buckets with
> > reduced redundancy with radosgw?  That is supported, although not quite
> > the way AWS does it.  You can create different pools that back RGW
> > buckets, and each bucket is stored in one of those pools.  So you could
> > make one of them 2x instead of 3x, or use an erasure code of your choice.
> >
> > What isn't currently supported is the ability to reduce the redundancy of
> > individual objects in a bucket.  I don't think there is anything
> > architecturally preventing that, but it is not implemented or supported.
> >
> > When we look at the S3 archival features in more detail (soon!) I'm sure
> > this will come up!  The current plan is to address object versioning
> > first.  That is, unless a developer surfaces who wants to start hacking on
> > this right away...
> >
> > sage
> >
> >
> >
> >>
> >> Thanks
> >> Swami
> >>
> >> On Thu, Sep 18, 2014 at 6:20 PM, M Ranga Swami Reddy
> >> <swamireddy@gmail.com> wrote:
> >> > Hi ,
> >> >
> >> > Could you please check and clarify the below question on object
> >> > lifecycle and notification S3 APIs support:
> >> >
> >> > 1. To support the bucket lifecycle - we need to support the
> >> > moving/deleting the objects/buckets based lifecycle settings.
> >> > For ex: If an object lifecyle set as below:
> >> >           1. Archive it after 10 days - means move this object to low
> >> > cost object storage after 10 days of the creation date.
> >> >            2. Remove this object after 90days - mean remove this
> >> > object from the low cost object after 90days of creation date.
> >> >
> >> > Q1- Does the ceph support the above concept like moving to low cost
> >> > storage and delete from that storage?
> >> >
> >> > 2. To support the object notifications:
> >> >       - First there should be low cost and high availability storage
> >> > with single replica only. If an object created with this type of
> >> > object storage,
> >> >         There could be chances that object could lose, so if an object
> >> > of this type of storage lost, set the notifications.
> >> >
> >> > Q2- Does Ceph support low cost and high availability storage type?
> >> >
> >> > Thanks
> >> >
> >> > On Fri, Sep 12, 2014 at 8:00 PM, M Ranga Swami Reddy
> >> > <swamireddy@gmail.com> wrote:
> >> >> Hi Yehuda,
> >> >>
> >> >> Could you please check and clarify the below question on object
> >> >> lifecycle and notification S3 APIs support:
> >> >>
> >> >> 1. To support the bucket lifecycle - we need to support the
> >> >> moving/deleting the objects/buckets based lifecycle settings.
> >> >> For ex: If an object lifecyle set as below:
> >> >>           1. Archive it after 10 days - means move this object to low
> >> >> cost object storage after 10 days of the creation date.
> >> >>            2. Remove this object after 90days - mean remove this
> >> >> object from the low cost object after 90days of creation date.
> >> >>
> >> >> Q1- Does the ceph support the above concept like moving to low cost
> >> >> storage and delete from that storage?
> >> >>
> >> >> 2. To support the object notifications:
> >> >>       - First there should be low cost and high availability storage
> >> >> with single replica only. If an object created with this type of
> >> >> object storage,
> >> >>         There could be chances that object could lose, so if an object
> >> >> of this type of storage lost, set the notifications.
> >> >>
> >> >> Q2- Does Ceph support low cost and high availability storage type?
> >> >>
> >> >> Thanks
> >> >> Swami
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Tue, Jul 29, 2014 at 1:35 AM, Yehuda Sadeh <yehuda@redhat.com> wrote:
> >> >>> Bucket lifecycle:
> >> >>> http://tracker.ceph.com/issues/8929
> >> >>>
> >> >>> Bucket notification:
> >> >>> http://tracker.ceph.com/issues/8956
> >> >>>
> >> >>> On Sun, Jul 27, 2014 at 12:54 AM, M Ranga Swami Reddy
> >> >>> <swamireddy@gmail.com> wrote:
> >> >>>> Good no know the details. Can you please share the issue ID for bucket
> >> >>>> lifecycle? My team also could start help here.
> >> >>>> Regarding the notification - Do we have the issue ID?
> >> >>>> Yes, the object versioning will be backlog one - I strongly feel we
> >> >>>> start working on this asap.
> >> >>>>
> >> >>>> Thanks
> >> >>>> Swami
> >> >>>>
> >> >>>> On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
> >> >>>>> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
> >> >>>>> <swamireddy@gmail.com> wrote:
> >> >>>>>> Thanks for quick reply.
> >> >>>>>> Yes,  versioned object - missing in ceph ATM
> >> >>>>>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
> >> >>>>>> put object notification and object restore (ie versioned object) S3
> >> >>>>>> API support.
> >> >>>>>> Please let me now any of the above work is in progress or some one
> >> >>>>>> planned to work on.
> >> >>>>>
> >> >>>>>
> >> >>>>> I opened an issue for bucket lifecycle (we already had an issue open
> >> >>>>> for object expiration though). We do have bucket location already
> >> >>>>> (part of the multi-region feature). Object versioning is definitely on
> >> >>>>> our backlog and one that we'll hopefully implement sooner rather
> >> >>>>> later.
> >> >>>>> With regard to object notification, it'll require having a
> >> >>>>> notification service which is a bit out of the scope. Integrating the
> >> >>>>> gateway with such a service whouldn't be hard, but we'll need to have
> >> >>>>> that first.
> >> >>>>>
> >> >>>>> Yehuda
> >> >>>>>
> >> >>>>>>
> >> >>>>>> Thanks
> >> >>>>>> Swami
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
> >> >>>>>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
> >> >>>>>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
> >> >>>>>>>>
> >> >>>>>>>> Link: http://ceph.com/docs/master/radosgw/s3/
> >> >>>>>>>>
> >> >>>>>>>> Is there plan to support the ?n supported item in the above table?
> >> >>>>>>>> or
> >> >>>>>>>> Any working on this?
> >> >>>>>>>
> >> >>>>>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
> >> >>>>>>> up to date.   The main gap, I think, is versioned objects.
> >> >>>>>>>
> >> >>>>>>> Are there specfiic parts of the S3 API that are missing that you need?
> >> >>>>>>> That sort of info is very helpful for prioritizing effort...
> >> >>>>>>>
> >> >>>>>>> sage
> >> >>>>>> --
> >> >>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> >>>>>> the body of a message to majordomo@vger.kernel.org
> >> >>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> >>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Fwd: S3 API Compatibility support

[M Ranga Swami Reddy]

>What do you mean by "RRS storage-low cost storage"?  My read of the RRS
>numbers is that they simply have a different tier of S3 that runs fewer
>replicas and (probably) cheaper disks.  In radosgw-land, this would just
>be a different rados pool with 2x replicas and (probably) a CRUSH rule
>mapping it to different hardware (with bigger and/or cheaper disks).

Thats correct. If we could do the with a different rados pool  using
2x replicas along with CURSH
mapping it to different h/w (with bigger and cheaper disks) , then its
same as RRS support in AWS.


>> >What isn't currently supported is the ability to reduce the redundancy of
>> >individual objects in a bucket.  I don't think there is anything
>> >architecturally preventing that, but it is not implemented or supported.
>>
>> OK. Do we have the issue id for the above? Else, we can file one. Please advise.

>There is the main #4099 issue for object expiration, but there is no real
>detail there.  The plan is (as always) to have equivalent functionality to S3.

>Do you mind creating a new feature ticket that specifically references the
>ability to move objects to a second storage tier based on policy?  Any
>references to AWS docs about the API or functionality would be helpful in
>the ticket.


Sure, I will create a new feature ticket and add the needful information  there.

Thanks
Swami

On Fri, Sep 19, 2014 at 9:08 PM, Sage Weil <sweil@redhat.com> wrote:
> On Fri, 19 Sep 2014, M Ranga Swami Reddy wrote:
>> Hi Sage,
>> Thanks for quick reply.
>>
>> >what you mean.
>> >For RRS, though, I assume you mean the ability to create buckets with
>> >reduced redundancy with radosgw?  That is supported, although not quite
>> >the way AWS does it.  You can create different pools that back RGW
>> >buckets, and each bucket is stored in one of those pools.  So you could
>> >make one of them 2x instead of 3x, or use an erasure code of your choice.
>>
>> Yes, we can confiure ceph to use 2x replicas, which will look like
>> reduced redundancy, but AWS uses a separate RRS storage-low cost
>> (instead of
>> standard) storage for this purpose. I am checking, if we could
>> similarly in ceph too.
>
> What do you mean by "RRS storage-low cost storage"?  My read of the RRS
> numbers is that they simply have a different tier of S3 that runs fewer
> replicas and (probably) cheaper disks.  In radosgw-land, this would just
> be a different rados pool with 2x replicas and (probably) a CRUSH rule
> mapping it to different hardware (with bigger and/or cheaper disks).
>
>> >What isn't currently supported is the ability to reduce the redundancy of
>> >individual objects in a bucket.  I don't think there is anything
>> >architecturally preventing that, but it is not implemented or supported.
>>
>> OK. Do we have the issue id for the above? Else, we can file one. Please advise.
>
> There is the main #4099 issue for object expiration, but there is no real
> detail there.  The plan is (as always) to have equivalent functionality to
> S3.
>
> Do you mind creating a new feature ticket that specifically references the
> ability to move objects to a second storage tier based on policy?  Any
> references to AWS docs about the API or functionality would be helpful in
> the ticket.
>
>> >When we look at the S3 archival features in more detail (soon!) I'm sure
>> >this will come up!  The current plan is to address object versioning
>> >first.  That is, unless a developer surfaces who wants to start hacking on
>> >this right away...
>>
>> Great to know this. Even we are keen with S3 support in Ceph and we
>> are happy support you here.
>
> Great to hear!
>
> Thanks-
> sage
>
>
>>
>> Thanks
>> Swami
>>
>> On Fri, Sep 19, 2014 at 11:08 AM, Sage Weil <sweil@redhat.com> wrote:
>> > On Fri, 19 Sep 2014, M Ranga Swami Reddy wrote:
>> >> Hi Sage,
>> >> Could you please advise, if Ceph support the low cost object
>> >> storages(like Amazon Glacier or RRS) for archiving objects like log
>> >> file etc.?
>> >
>> > Ceph doesn't interact at all with AWS services like Glacier, if that's
>> > what you mean.
>> >
>> > For RRS, though, I assume you mean the ability to create buckets with
>> > reduced redundancy with radosgw?  That is supported, although not quite
>> > the way AWS does it.  You can create different pools that back RGW
>> > buckets, and each bucket is stored in one of those pools.  So you could
>> > make one of them 2x instead of 3x, or use an erasure code of your choice.
>> >
>> > What isn't currently supported is the ability to reduce the redundancy of
>> > individual objects in a bucket.  I don't think there is anything
>> > architecturally preventing that, but it is not implemented or supported.
>> >
>> > When we look at the S3 archival features in more detail (soon!) I'm sure
>> > this will come up!  The current plan is to address object versioning
>> > first.  That is, unless a developer surfaces who wants to start hacking on
>> > this right away...
>> >
>> > sage
>> >
>> >
>> >
>> >>
>> >> Thanks
>> >> Swami
>> >>
>> >> On Thu, Sep 18, 2014 at 6:20 PM, M Ranga Swami Reddy
>> >> <swamireddy@gmail.com> wrote:
>> >> > Hi ,
>> >> >
>> >> > Could you please check and clarify the below question on object
>> >> > lifecycle and notification S3 APIs support:
>> >> >
>> >> > 1. To support the bucket lifecycle - we need to support the
>> >> > moving/deleting the objects/buckets based lifecycle settings.
>> >> > For ex: If an object lifecyle set as below:
>> >> >           1. Archive it after 10 days - means move this object to low
>> >> > cost object storage after 10 days of the creation date.
>> >> >            2. Remove this object after 90days - mean remove this
>> >> > object from the low cost object after 90days of creation date.
>> >> >
>> >> > Q1- Does the ceph support the above concept like moving to low cost
>> >> > storage and delete from that storage?
>> >> >
>> >> > 2. To support the object notifications:
>> >> >       - First there should be low cost and high availability storage
>> >> > with single replica only. If an object created with this type of
>> >> > object storage,
>> >> >         There could be chances that object could lose, so if an object
>> >> > of this type of storage lost, set the notifications.
>> >> >
>> >> > Q2- Does Ceph support low cost and high availability storage type?
>> >> >
>> >> > Thanks
>> >> >
>> >> > On Fri, Sep 12, 2014 at 8:00 PM, M Ranga Swami Reddy
>> >> > <swamireddy@gmail.com> wrote:
>> >> >> Hi Yehuda,
>> >> >>
>> >> >> Could you please check and clarify the below question on object
>> >> >> lifecycle and notification S3 APIs support:
>> >> >>
>> >> >> 1. To support the bucket lifecycle - we need to support the
>> >> >> moving/deleting the objects/buckets based lifecycle settings.
>> >> >> For ex: If an object lifecyle set as below:
>> >> >>           1. Archive it after 10 days - means move this object to low
>> >> >> cost object storage after 10 days of the creation date.
>> >> >>            2. Remove this object after 90days - mean remove this
>> >> >> object from the low cost object after 90days of creation date.
>> >> >>
>> >> >> Q1- Does the ceph support the above concept like moving to low cost
>> >> >> storage and delete from that storage?
>> >> >>
>> >> >> 2. To support the object notifications:
>> >> >>       - First there should be low cost and high availability storage
>> >> >> with single replica only. If an object created with this type of
>> >> >> object storage,
>> >> >>         There could be chances that object could lose, so if an object
>> >> >> of this type of storage lost, set the notifications.
>> >> >>
>> >> >> Q2- Does Ceph support low cost and high availability storage type?
>> >> >>
>> >> >> Thanks
>> >> >> Swami
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Tue, Jul 29, 2014 at 1:35 AM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>> >> >>> Bucket lifecycle:
>> >> >>> http://tracker.ceph.com/issues/8929
>> >> >>>
>> >> >>> Bucket notification:
>> >> >>> http://tracker.ceph.com/issues/8956
>> >> >>>
>> >> >>> On Sun, Jul 27, 2014 at 12:54 AM, M Ranga Swami Reddy
>> >> >>> <swamireddy@gmail.com> wrote:
>> >> >>>> Good no know the details. Can you please share the issue ID for bucket
>> >> >>>> lifecycle? My team also could start help here.
>> >> >>>> Regarding the notification - Do we have the issue ID?
>> >> >>>> Yes, the object versioning will be backlog one - I strongly feel we
>> >> >>>> start working on this asap.
>> >> >>>>
>> >> >>>> Thanks
>> >> >>>> Swami
>> >> >>>>
>> >> >>>> On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>> >> >>>>> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
>> >> >>>>> <swamireddy@gmail.com> wrote:
>> >> >>>>>> Thanks for quick reply.
>> >> >>>>>> Yes,  versioned object - missing in ceph ATM
>> >> >>>>>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
>> >> >>>>>> put object notification and object restore (ie versioned object) S3
>> >> >>>>>> API support.
>> >> >>>>>> Please let me now any of the above work is in progress or some one
>> >> >>>>>> planned to work on.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> I opened an issue for bucket lifecycle (we already had an issue open
>> >> >>>>> for object expiration though). We do have bucket location already
>> >> >>>>> (part of the multi-region feature). Object versioning is definitely on
>> >> >>>>> our backlog and one that we'll hopefully implement sooner rather
>> >> >>>>> later.
>> >> >>>>> With regard to object notification, it'll require having a
>> >> >>>>> notification service which is a bit out of the scope. Integrating the
>> >> >>>>> gateway with such a service whouldn't be hard, but we'll need to have
>> >> >>>>> that first.
>> >> >>>>>
>> >> >>>>> Yehuda
>> >> >>>>>
>> >> >>>>>>
>> >> >>>>>> Thanks
>> >> >>>>>> Swami
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
>> >> >>>>>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>> >> >>>>>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>> >> >>>>>>>>
>> >> >>>>>>>> Link: http://ceph.com/docs/master/radosgw/s3/
>> >> >>>>>>>>
>> >> >>>>>>>> Is there plan to support the ?n supported item in the above table?
>> >> >>>>>>>> or
>> >> >>>>>>>> Any working on this?
>> >> >>>>>>>
>> >> >>>>>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
>> >> >>>>>>> up to date.   The main gap, I think, is versioned objects.
>> >> >>>>>>>
>> >> >>>>>>> Are there specfiic parts of the S3 API that are missing that you need?
>> >> >>>>>>> That sort of info is very helpful for prioritizing effort...
>> >> >>>>>>>
>> >> >>>>>>> sage
>> >> >>>>>> --
>> >> >>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> >> >>>>>> the body of a message to majordomo@vger.kernel.org
>> >> >>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >>
>> >>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>

Re: Fwd: S3 API Compatibility support

[Yehuda Sadeh]

On Fri, Sep 19, 2014 at 8:32 AM, M Ranga Swami Reddy
<swamireddy@gmail.com> wrote:
> Hi Sage,
> Thanks for quick reply.
>
>>Ceph doesn't interact at all with AWS services like Glacier, if that's
>
> No. I meant that - Ceph interaction with a glacier and RRS type of
> storages along with currently used OSD (or standard storage).
>
>>what you mean.
>>For RRS, though, I assume you mean the ability to create buckets with
>>reduced redundancy with radosgw?  That is supported, although not quite
>>the way AWS does it.  You can create different pools that back RGW
>>buckets, and each bucket is stored in one of those pools.  So you could
>>make one of them 2x instead of 3x, or use an erasure code of your choice.
>
> Yes, we can confiure ceph to use 2x replicas, which will look like
> reduced redundancy, but AWS uses a separate RRS storage-low cost
> (instead of
> standard) storage for this purpose. I am checking, if we could
> similarly in ceph too.

You can use multiple placement targets and can specify on bucket
creation which placement target to use. At this time we don't support
the exact S3 reduced redundancy fields, although it should be pretty
easy to add.

>
>>What isn't currently supported is the ability to reduce the redundancy of
>>individual objects in a bucket.  I don't think there is anything
>>architecturally preventing that, but it is not implemented or supported.
>
> OK. Do we have the issue id for the above? Else, we can file one. Please advise.

I think #8929 would cover it.

Yehuda

>
>>When we look at the S3 archival features in more detail (soon!) I'm sure
>>this will come up!  The current plan is to address object versioning
>>first.  That is, unless a developer surfaces who wants to start hacking on
>>this right away...
>
> Great to know this. Even we are keen with S3 support in Ceph and we
> are happy support you here.
>
> Thanks
> Swami
>
> On Fri, Sep 19, 2014 at 11:08 AM, Sage Weil <sweil@redhat.com> wrote:
>> On Fri, 19 Sep 2014, M Ranga Swami Reddy wrote:
>>> Hi Sage,
>>> Could you please advise, if Ceph support the low cost object
>>> storages(like Amazon Glacier or RRS) for archiving objects like log
>>> file etc.?
>>
>> Ceph doesn't interact at all with AWS services like Glacier, if that's
>> what you mean.
>>
>> For RRS, though, I assume you mean the ability to create buckets with
>> reduced redundancy with radosgw?  That is supported, although not quite
>> the way AWS does it.  You can create different pools that back RGW
>> buckets, and each bucket is stored in one of those pools.  So you could
>> make one of them 2x instead of 3x, or use an erasure code of your choice.
>>
>> What isn't currently supported is the ability to reduce the redundancy of
>> individual objects in a bucket.  I don't think there is anything
>> architecturally preventing that, but it is not implemented or supported.
>>
>> When we look at the S3 archival features in more detail (soon!) I'm sure
>> this will come up!  The current plan is to address object versioning
>> first.  That is, unless a developer surfaces who wants to start hacking on
>> this right away...
>>
>> sage
>>
>>
>>
>>>
>>> Thanks
>>> Swami
>>>
>>> On Thu, Sep 18, 2014 at 6:20 PM, M Ranga Swami Reddy
>>> <swamireddy@gmail.com> wrote:
>>> > Hi ,
>>> >
>>> > Could you please check and clarify the below question on object
>>> > lifecycle and notification S3 APIs support:
>>> >
>>> > 1. To support the bucket lifecycle - we need to support the
>>> > moving/deleting the objects/buckets based lifecycle settings.
>>> > For ex: If an object lifecyle set as below:
>>> >           1. Archive it after 10 days - means move this object to low
>>> > cost object storage after 10 days of the creation date.
>>> >            2. Remove this object after 90days - mean remove this
>>> > object from the low cost object after 90days of creation date.
>>> >
>>> > Q1- Does the ceph support the above concept like moving to low cost
>>> > storage and delete from that storage?
>>> >
>>> > 2. To support the object notifications:
>>> >       - First there should be low cost and high availability storage
>>> > with single replica only. If an object created with this type of
>>> > object storage,
>>> >         There could be chances that object could lose, so if an object
>>> > of this type of storage lost, set the notifications.
>>> >
>>> > Q2- Does Ceph support low cost and high availability storage type?
>>> >
>>> > Thanks
>>> >
>>> > On Fri, Sep 12, 2014 at 8:00 PM, M Ranga Swami Reddy
>>> > <swamireddy@gmail.com> wrote:
>>> >> Hi Yehuda,
>>> >>
>>> >> Could you please check and clarify the below question on object
>>> >> lifecycle and notification S3 APIs support:
>>> >>
>>> >> 1. To support the bucket lifecycle - we need to support the
>>> >> moving/deleting the objects/buckets based lifecycle settings.
>>> >> For ex: If an object lifecyle set as below:
>>> >>           1. Archive it after 10 days - means move this object to low
>>> >> cost object storage after 10 days of the creation date.
>>> >>            2. Remove this object after 90days - mean remove this
>>> >> object from the low cost object after 90days of creation date.
>>> >>
>>> >> Q1- Does the ceph support the above concept like moving to low cost
>>> >> storage and delete from that storage?
>>> >>
>>> >> 2. To support the object notifications:
>>> >>       - First there should be low cost and high availability storage
>>> >> with single replica only. If an object created with this type of
>>> >> object storage,
>>> >>         There could be chances that object could lose, so if an object
>>> >> of this type of storage lost, set the notifications.
>>> >>
>>> >> Q2- Does Ceph support low cost and high availability storage type?
>>> >>
>>> >> Thanks
>>> >> Swami
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Tue, Jul 29, 2014 at 1:35 AM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>>> >>> Bucket lifecycle:
>>> >>> http://tracker.ceph.com/issues/8929
>>> >>>
>>> >>> Bucket notification:
>>> >>> http://tracker.ceph.com/issues/8956
>>> >>>
>>> >>> On Sun, Jul 27, 2014 at 12:54 AM, M Ranga Swami Reddy
>>> >>> <swamireddy@gmail.com> wrote:
>>> >>>> Good no know the details. Can you please share the issue ID for bucket
>>> >>>> lifecycle? My team also could start help here.
>>> >>>> Regarding the notification - Do we have the issue ID?
>>> >>>> Yes, the object versioning will be backlog one - I strongly feel we
>>> >>>> start working on this asap.
>>> >>>>
>>> >>>> Thanks
>>> >>>> Swami
>>> >>>>
>>> >>>> On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>>> >>>>> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
>>> >>>>> <swamireddy@gmail.com> wrote:
>>> >>>>>> Thanks for quick reply.
>>> >>>>>> Yes,  versioned object - missing in ceph ATM
>>> >>>>>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
>>> >>>>>> put object notification and object restore (ie versioned object) S3
>>> >>>>>> API support.
>>> >>>>>> Please let me now any of the above work is in progress or some one
>>> >>>>>> planned to work on.
>>> >>>>>
>>> >>>>>
>>> >>>>> I opened an issue for bucket lifecycle (we already had an issue open
>>> >>>>> for object expiration though). We do have bucket location already
>>> >>>>> (part of the multi-region feature). Object versioning is definitely on
>>> >>>>> our backlog and one that we'll hopefully implement sooner rather
>>> >>>>> later.
>>> >>>>> With regard to object notification, it'll require having a
>>> >>>>> notification service which is a bit out of the scope. Integrating the
>>> >>>>> gateway with such a service whouldn't be hard, but we'll need to have
>>> >>>>> that first.
>>> >>>>>
>>> >>>>> Yehuda
>>> >>>>>
>>> >>>>>>
>>> >>>>>> Thanks
>>> >>>>>> Swami
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
>>> >>>>>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>>> >>>>>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>>> >>>>>>>>
>>> >>>>>>>> Link: http://ceph.com/docs/master/radosgw/s3/
>>> >>>>>>>>
>>> >>>>>>>> Is there plan to support the ?n supported item in the above table?
>>> >>>>>>>> or
>>> >>>>>>>> Any working on this?
>>> >>>>>>>
>>> >>>>>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
>>> >>>>>>> up to date.   The main gap, I think, is versioned objects.
>>> >>>>>>>
>>> >>>>>>> Are there specfiic parts of the S3 API that are missing that you need?
>>> >>>>>>> That sort of info is very helpful for prioritizing effort...
>>> >>>>>>>
>>> >>>>>>> sage
>>> >>>>>> --
>>> >>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> >>>>>> the body of a message to majordomo@vger.kernel.org
>>> >>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>

Re: Fwd: S3 API Compatibility support

[M Ranga Swami Reddy]

>> Yes, we can confiure ceph to use 2x replicas, which will look like
>> reduced redundancy, but AWS uses a separate RRS storage-low cost
>> (instead of
>> standard) storage for this purpose. I am checking, if we could
>> similarly in ceph too.

>You can use multiple placement targets and can specify on bucket
>creation which placement target to use. At this time we don't support
>the exact S3 reduced redundancy fields, although it should be pretty
>easy to add.

OK.  Could you please guide us to implement the above.

>>>What isn't currently supported is the ability to reduce the redundancy of
>>>individual objects in a bucket.  I don't think there is anything
>>>architecturally preventing that, but it is not implemented or supported.
>>
>> OK. Do we have the issue id for the above? Else, we can file one. Please advise.

>I think #8929 would cover it.

The above issue for bucket lifecyle, but we are storing the
object/buckets on a separate
disk like RRS type. I think, its needed to support an object to be
stored based on the
option (default on standard storage).


Thanks
Swami

On Fri, Sep 19, 2014 at 10:15 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
> On Fri, Sep 19, 2014 at 8:32 AM, M Ranga Swami Reddy
> <swamireddy@gmail.com> wrote:
>> Hi Sage,
>> Thanks for quick reply.
>>
>>>Ceph doesn't interact at all with AWS services like Glacier, if that's
>>
>> No. I meant that - Ceph interaction with a glacier and RRS type of
>> storages along with currently used OSD (or standard storage).
>>
>>>what you mean.
>>>For RRS, though, I assume you mean the ability to create buckets with
>>>reduced redundancy with radosgw?  That is supported, although not quite
>>>the way AWS does it.  You can create different pools that back RGW
>>>buckets, and each bucket is stored in one of those pools.  So you could
>>>make one of them 2x instead of 3x, or use an erasure code of your choice.
>>
>> Yes, we can confiure ceph to use 2x replicas, which will look like
>> reduced redundancy, but AWS uses a separate RRS storage-low cost
>> (instead of
>> standard) storage for this purpose. I am checking, if we could
>> similarly in ceph too.
>
> You can use multiple placement targets and can specify on bucket
> creation which placement target to use. At this time we don't support
> the exact S3 reduced redundancy fields, although it should be pretty
> easy to add.
>
>>
>>>What isn't currently supported is the ability to reduce the redundancy of
>>>individual objects in a bucket.  I don't think there is anything
>>>architecturally preventing that, but it is not implemented or supported.
>>
>> OK. Do we have the issue id for the above? Else, we can file one. Please advise.
>
> I think #8929 would cover it.
>
> Yehuda
>
>>
>>>When we look at the S3 archival features in more detail (soon!) I'm sure
>>>this will come up!  The current plan is to address object versioning
>>>first.  That is, unless a developer surfaces who wants to start hacking on
>>>this right away...
>>
>> Great to know this. Even we are keen with S3 support in Ceph and we
>> are happy support you here.
>>
>> Thanks
>> Swami
>>
>> On Fri, Sep 19, 2014 at 11:08 AM, Sage Weil <sweil@redhat.com> wrote:
>>> On Fri, 19 Sep 2014, M Ranga Swami Reddy wrote:
>>>> Hi Sage,
>>>> Could you please advise, if Ceph support the low cost object
>>>> storages(like Amazon Glacier or RRS) for archiving objects like log
>>>> file etc.?
>>>
>>> Ceph doesn't interact at all with AWS services like Glacier, if that's
>>> what you mean.
>>>
>>> For RRS, though, I assume you mean the ability to create buckets with
>>> reduced redundancy with radosgw?  That is supported, although not quite
>>> the way AWS does it.  You can create different pools that back RGW
>>> buckets, and each bucket is stored in one of those pools.  So you could
>>> make one of them 2x instead of 3x, or use an erasure code of your choice.
>>>
>>> What isn't currently supported is the ability to reduce the redundancy of
>>> individual objects in a bucket.  I don't think there is anything
>>> architecturally preventing that, but it is not implemented or supported.
>>>
>>> When we look at the S3 archival features in more detail (soon!) I'm sure
>>> this will come up!  The current plan is to address object versioning
>>> first.  That is, unless a developer surfaces who wants to start hacking on
>>> this right away...
>>>
>>> sage
>>>
>>>
>>>
>>>>
>>>> Thanks
>>>> Swami
>>>>
>>>> On Thu, Sep 18, 2014 at 6:20 PM, M Ranga Swami Reddy
>>>> <swamireddy@gmail.com> wrote:
>>>> > Hi ,
>>>> >
>>>> > Could you please check and clarify the below question on object
>>>> > lifecycle and notification S3 APIs support:
>>>> >
>>>> > 1. To support the bucket lifecycle - we need to support the
>>>> > moving/deleting the objects/buckets based lifecycle settings.
>>>> > For ex: If an object lifecyle set as below:
>>>> >           1. Archive it after 10 days - means move this object to low
>>>> > cost object storage after 10 days of the creation date.
>>>> >            2. Remove this object after 90days - mean remove this
>>>> > object from the low cost object after 90days of creation date.
>>>> >
>>>> > Q1- Does the ceph support the above concept like moving to low cost
>>>> > storage and delete from that storage?
>>>> >
>>>> > 2. To support the object notifications:
>>>> >       - First there should be low cost and high availability storage
>>>> > with single replica only. If an object created with this type of
>>>> > object storage,
>>>> >         There could be chances that object could lose, so if an object
>>>> > of this type of storage lost, set the notifications.
>>>> >
>>>> > Q2- Does Ceph support low cost and high availability storage type?
>>>> >
>>>> > Thanks
>>>> >
>>>> > On Fri, Sep 12, 2014 at 8:00 PM, M Ranga Swami Reddy
>>>> > <swamireddy@gmail.com> wrote:
>>>> >> Hi Yehuda,
>>>> >>
>>>> >> Could you please check and clarify the below question on object
>>>> >> lifecycle and notification S3 APIs support:
>>>> >>
>>>> >> 1. To support the bucket lifecycle - we need to support the
>>>> >> moving/deleting the objects/buckets based lifecycle settings.
>>>> >> For ex: If an object lifecyle set as below:
>>>> >>           1. Archive it after 10 days - means move this object to low
>>>> >> cost object storage after 10 days of the creation date.
>>>> >>            2. Remove this object after 90days - mean remove this
>>>> >> object from the low cost object after 90days of creation date.
>>>> >>
>>>> >> Q1- Does the ceph support the above concept like moving to low cost
>>>> >> storage and delete from that storage?
>>>> >>
>>>> >> 2. To support the object notifications:
>>>> >>       - First there should be low cost and high availability storage
>>>> >> with single replica only. If an object created with this type of
>>>> >> object storage,
>>>> >>         There could be chances that object could lose, so if an object
>>>> >> of this type of storage lost, set the notifications.
>>>> >>
>>>> >> Q2- Does Ceph support low cost and high availability storage type?
>>>> >>
>>>> >> Thanks
>>>> >> Swami
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Tue, Jul 29, 2014 at 1:35 AM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>>>> >>> Bucket lifecycle:
>>>> >>> http://tracker.ceph.com/issues/8929
>>>> >>>
>>>> >>> Bucket notification:
>>>> >>> http://tracker.ceph.com/issues/8956
>>>> >>>
>>>> >>> On Sun, Jul 27, 2014 at 12:54 AM, M Ranga Swami Reddy
>>>> >>> <swamireddy@gmail.com> wrote:
>>>> >>>> Good no know the details. Can you please share the issue ID for bucket
>>>> >>>> lifecycle? My team also could start help here.
>>>> >>>> Regarding the notification - Do we have the issue ID?
>>>> >>>> Yes, the object versioning will be backlog one - I strongly feel we
>>>> >>>> start working on this asap.
>>>> >>>>
>>>> >>>> Thanks
>>>> >>>> Swami
>>>> >>>>
>>>> >>>> On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>>>> >>>>> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
>>>> >>>>> <swamireddy@gmail.com> wrote:
>>>> >>>>>> Thanks for quick reply.
>>>> >>>>>> Yes,  versioned object - missing in ceph ATM
>>>> >>>>>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
>>>> >>>>>> put object notification and object restore (ie versioned object) S3
>>>> >>>>>> API support.
>>>> >>>>>> Please let me now any of the above work is in progress or some one
>>>> >>>>>> planned to work on.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> I opened an issue for bucket lifecycle (we already had an issue open
>>>> >>>>> for object expiration though). We do have bucket location already
>>>> >>>>> (part of the multi-region feature). Object versioning is definitely on
>>>> >>>>> our backlog and one that we'll hopefully implement sooner rather
>>>> >>>>> later.
>>>> >>>>> With regard to object notification, it'll require having a
>>>> >>>>> notification service which is a bit out of the scope. Integrating the
>>>> >>>>> gateway with such a service whouldn't be hard, but we'll need to have
>>>> >>>>> that first.
>>>> >>>>>
>>>> >>>>> Yehuda
>>>> >>>>>
>>>> >>>>>>
>>>> >>>>>> Thanks
>>>> >>>>>> Swami
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
>>>> >>>>>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>>>> >>>>>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>>>> >>>>>>>>
>>>> >>>>>>>> Link: http://ceph.com/docs/master/radosgw/s3/
>>>> >>>>>>>>
>>>> >>>>>>>> Is there plan to support the ?n supported item in the above table?
>>>> >>>>>>>> or
>>>> >>>>>>>> Any working on this?
>>>> >>>>>>>
>>>> >>>>>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
>>>> >>>>>>> up to date.   The main gap, I think, is versioned objects.
>>>> >>>>>>>
>>>> >>>>>>> Are there specfiic parts of the S3 API that are missing that you need?
>>>> >>>>>>> That sort of info is very helpful for prioritizing effort...
>>>> >>>>>>>
>>>> >>>>>>> sage
>>>> >>>>>> --
>>>> >>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> >>>>>> the body of a message to majordomo@vger.kernel.org
>>>> >>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>>

Re: Fwd: S3 API Compatibility support

[M Ranga Swami Reddy]

>>There is the main #4099 issue for object expiration, but there is no real
>>detail there.  The plan is (as always) to have equivalent functionality to S3.

>>Do you mind creating a new feature ticket that specifically references the
>>ability to move objects to a second storage tier based on policy?  Any
>>references to AWS docs about the API or functionality would be helpful in
>>the ticket.


>Sure, I will create a new feature ticket and add the needful information  there.

Created a new ticket: http://tracker.ceph.com/issues/9581


Thanks
Swami

On Fri, Sep 19, 2014 at 9:23 PM, M Ranga Swami Reddy
<swamireddy@gmail.com> wrote:
>>What do you mean by "RRS storage-low cost storage"?  My read of the RRS
>>numbers is that they simply have a different tier of S3 that runs fewer
>>replicas and (probably) cheaper disks.  In radosgw-land, this would just
>>be a different rados pool with 2x replicas and (probably) a CRUSH rule
>>mapping it to different hardware (with bigger and/or cheaper disks).
>
> Thats correct. If we could do the with a different rados pool  using
> 2x replicas along with CURSH
> mapping it to different h/w (with bigger and cheaper disks) , then its
> same as RRS support in AWS.
>
>
>>> >What isn't currently supported is the ability to reduce the redundancy of
>>> >individual objects in a bucket.  I don't think there is anything
>>> >architecturally preventing that, but it is not implemented or supported.
>>>
>>> OK. Do we have the issue id for the above? Else, we can file one. Please advise.
>
>>There is the main #4099 issue for object expiration, but there is no real
>>detail there.  The plan is (as always) to have equivalent functionality to S3.
>
>>Do you mind creating a new feature ticket that specifically references the
>>ability to move objects to a second storage tier based on policy?  Any
>>references to AWS docs about the API or functionality would be helpful in
>>the ticket.
>
>
> Sure, I will create a new feature ticket and add the needful information  there.
>
> Thanks
> Swami
>
> On Fri, Sep 19, 2014 at 9:08 PM, Sage Weil <sweil@redhat.com> wrote:
>> On Fri, 19 Sep 2014, M Ranga Swami Reddy wrote:
>>> Hi Sage,
>>> Thanks for quick reply.
>>>
>>> >what you mean.
>>> >For RRS, though, I assume you mean the ability to create buckets with
>>> >reduced redundancy with radosgw?  That is supported, although not quite
>>> >the way AWS does it.  You can create different pools that back RGW
>>> >buckets, and each bucket is stored in one of those pools.  So you could
>>> >make one of them 2x instead of 3x, or use an erasure code of your choice.
>>>
>>> Yes, we can confiure ceph to use 2x replicas, which will look like
>>> reduced redundancy, but AWS uses a separate RRS storage-low cost
>>> (instead of
>>> standard) storage for this purpose. I am checking, if we could
>>> similarly in ceph too.
>>
>> What do you mean by "RRS storage-low cost storage"?  My read of the RRS
>> numbers is that they simply have a different tier of S3 that runs fewer
>> replicas and (probably) cheaper disks.  In radosgw-land, this would just
>> be a different rados pool with 2x replicas and (probably) a CRUSH rule
>> mapping it to different hardware (with bigger and/or cheaper disks).
>>
>>> >What isn't currently supported is the ability to reduce the redundancy of
>>> >individual objects in a bucket.  I don't think there is anything
>>> >architecturally preventing that, but it is not implemented or supported.
>>>
>>> OK. Do we have the issue id for the above? Else, we can file one. Please advise.
>>
>> There is the main #4099 issue for object expiration, but there is no real
>> detail there.  The plan is (as always) to have equivalent functionality to
>> S3.
>>
>> Do you mind creating a new feature ticket that specifically references the
>> ability to move objects to a second storage tier based on policy?  Any
>> references to AWS docs about the API or functionality would be helpful in
>> the ticket.
>>
>>> >When we look at the S3 archival features in more detail (soon!) I'm sure
>>> >this will come up!  The current plan is to address object versioning
>>> >first.  That is, unless a developer surfaces who wants to start hacking on
>>> >this right away...
>>>
>>> Great to know this. Even we are keen with S3 support in Ceph and we
>>> are happy support you here.
>>
>> Great to hear!
>>
>> Thanks-
>> sage
>>
>>
>>>
>>> Thanks
>>> Swami
>>>
>>> On Fri, Sep 19, 2014 at 11:08 AM, Sage Weil <sweil@redhat.com> wrote:
>>> > On Fri, 19 Sep 2014, M Ranga Swami Reddy wrote:
>>> >> Hi Sage,
>>> >> Could you please advise, if Ceph support the low cost object
>>> >> storages(like Amazon Glacier or RRS) for archiving objects like log
>>> >> file etc.?
>>> >
>>> > Ceph doesn't interact at all with AWS services like Glacier, if that's
>>> > what you mean.
>>> >
>>> > For RRS, though, I assume you mean the ability to create buckets with
>>> > reduced redundancy with radosgw?  That is supported, although not quite
>>> > the way AWS does it.  You can create different pools that back RGW
>>> > buckets, and each bucket is stored in one of those pools.  So you could
>>> > make one of them 2x instead of 3x, or use an erasure code of your choice.
>>> >
>>> > What isn't currently supported is the ability to reduce the redundancy of
>>> > individual objects in a bucket.  I don't think there is anything
>>> > architecturally preventing that, but it is not implemented or supported.
>>> >
>>> > When we look at the S3 archival features in more detail (soon!) I'm sure
>>> > this will come up!  The current plan is to address object versioning
>>> > first.  That is, unless a developer surfaces who wants to start hacking on
>>> > this right away...
>>> >
>>> > sage
>>> >
>>> >
>>> >
>>> >>
>>> >> Thanks
>>> >> Swami
>>> >>
>>> >> On Thu, Sep 18, 2014 at 6:20 PM, M Ranga Swami Reddy
>>> >> <swamireddy@gmail.com> wrote:
>>> >> > Hi ,
>>> >> >
>>> >> > Could you please check and clarify the below question on object
>>> >> > lifecycle and notification S3 APIs support:
>>> >> >
>>> >> > 1. To support the bucket lifecycle - we need to support the
>>> >> > moving/deleting the objects/buckets based lifecycle settings.
>>> >> > For ex: If an object lifecyle set as below:
>>> >> >           1. Archive it after 10 days - means move this object to low
>>> >> > cost object storage after 10 days of the creation date.
>>> >> >            2. Remove this object after 90days - mean remove this
>>> >> > object from the low cost object after 90days of creation date.
>>> >> >
>>> >> > Q1- Does the ceph support the above concept like moving to low cost
>>> >> > storage and delete from that storage?
>>> >> >
>>> >> > 2. To support the object notifications:
>>> >> >       - First there should be low cost and high availability storage
>>> >> > with single replica only. If an object created with this type of
>>> >> > object storage,
>>> >> >         There could be chances that object could lose, so if an object
>>> >> > of this type of storage lost, set the notifications.
>>> >> >
>>> >> > Q2- Does Ceph support low cost and high availability storage type?
>>> >> >
>>> >> > Thanks
>>> >> >
>>> >> > On Fri, Sep 12, 2014 at 8:00 PM, M Ranga Swami Reddy
>>> >> > <swamireddy@gmail.com> wrote:
>>> >> >> Hi Yehuda,
>>> >> >>
>>> >> >> Could you please check and clarify the below question on object
>>> >> >> lifecycle and notification S3 APIs support:
>>> >> >>
>>> >> >> 1. To support the bucket lifecycle - we need to support the
>>> >> >> moving/deleting the objects/buckets based lifecycle settings.
>>> >> >> For ex: If an object lifecyle set as below:
>>> >> >>           1. Archive it after 10 days - means move this object to low
>>> >> >> cost object storage after 10 days of the creation date.
>>> >> >>            2. Remove this object after 90days - mean remove this
>>> >> >> object from the low cost object after 90days of creation date.
>>> >> >>
>>> >> >> Q1- Does the ceph support the above concept like moving to low cost
>>> >> >> storage and delete from that storage?
>>> >> >>
>>> >> >> 2. To support the object notifications:
>>> >> >>       - First there should be low cost and high availability storage
>>> >> >> with single replica only. If an object created with this type of
>>> >> >> object storage,
>>> >> >>         There could be chances that object could lose, so if an object
>>> >> >> of this type of storage lost, set the notifications.
>>> >> >>
>>> >> >> Q2- Does Ceph support low cost and high availability storage type?
>>> >> >>
>>> >> >> Thanks
>>> >> >> Swami
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> On Tue, Jul 29, 2014 at 1:35 AM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>>> >> >>> Bucket lifecycle:
>>> >> >>> http://tracker.ceph.com/issues/8929
>>> >> >>>
>>> >> >>> Bucket notification:
>>> >> >>> http://tracker.ceph.com/issues/8956
>>> >> >>>
>>> >> >>> On Sun, Jul 27, 2014 at 12:54 AM, M Ranga Swami Reddy
>>> >> >>> <swamireddy@gmail.com> wrote:
>>> >> >>>> Good no know the details. Can you please share the issue ID for bucket
>>> >> >>>> lifecycle? My team also could start help here.
>>> >> >>>> Regarding the notification - Do we have the issue ID?
>>> >> >>>> Yes, the object versioning will be backlog one - I strongly feel we
>>> >> >>>> start working on this asap.
>>> >> >>>>
>>> >> >>>> Thanks
>>> >> >>>> Swami
>>> >> >>>>
>>> >> >>>> On Fri, Jul 25, 2014 at 11:31 PM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>>> >> >>>>> On Fri, Jul 25, 2014 at 10:14 AM, M Ranga Swami Reddy
>>> >> >>>>> <swamireddy@gmail.com> wrote:
>>> >> >>>>>> Thanks for quick reply.
>>> >> >>>>>> Yes,  versioned object - missing in ceph ATM
>>> >> >>>>>> Iam looking for: bucket lifecylce (get/put/delete), bucket location,
>>> >> >>>>>> put object notification and object restore (ie versioned object) S3
>>> >> >>>>>> API support.
>>> >> >>>>>> Please let me now any of the above work is in progress or some one
>>> >> >>>>>> planned to work on.
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> I opened an issue for bucket lifecycle (we already had an issue open
>>> >> >>>>> for object expiration though). We do have bucket location already
>>> >> >>>>> (part of the multi-region feature). Object versioning is definitely on
>>> >> >>>>> our backlog and one that we'll hopefully implement sooner rather
>>> >> >>>>> later.
>>> >> >>>>> With regard to object notification, it'll require having a
>>> >> >>>>> notification service which is a bit out of the scope. Integrating the
>>> >> >>>>> gateway with such a service whouldn't be hard, but we'll need to have
>>> >> >>>>> that first.
>>> >> >>>>>
>>> >> >>>>> Yehuda
>>> >> >>>>>
>>> >> >>>>>>
>>> >> >>>>>> Thanks
>>> >> >>>>>> Swami
>>> >> >>>>>>
>>> >> >>>>>>
>>> >> >>>>>> On Fri, Jul 25, 2014 at 9:19 PM, Sage Weil <sweil@redhat.com> wrote:
>>> >> >>>>>>> On Fri, 25 Jul 2014, M Ranga Swami Reddy wrote:
>>> >> >>>>>>>> Hi Team: As per the ceph document a few S3 APIs compatibility not supported.
>>> >> >>>>>>>>
>>> >> >>>>>>>> Link: http://ceph.com/docs/master/radosgw/s3/
>>> >> >>>>>>>>
>>> >> >>>>>>>> Is there plan to support the ?n supported item in the above table?
>>> >> >>>>>>>> or
>>> >> >>>>>>>> Any working on this?
>>> >> >>>>>>>
>>> >> >>>>>>> Yes.  Unfortunately this table isn't particularly detailed or accurate or
>>> >> >>>>>>> up to date.   The main gap, I think, is versioned objects.
>>> >> >>>>>>>
>>> >> >>>>>>> Are there specfiic parts of the S3 API that are missing that you need?
>>> >> >>>>>>> That sort of info is very helpful for prioritizing effort...
>>> >> >>>>>>>
>>> >> >>>>>>> sage
>>> >> >>>>>> --
>>> >> >>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> >> >>>>>> the body of a message to majordomo@vger.kernel.org
>>> >> >>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> >> --
>>> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> >> the body of a message to majordomo@vger.kernel.org
>>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> >>
>>> >>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>