storlets

storlets

[Sage Weil]

http://domino.research.ibm.com/library/cyberdig.nsf/papers/7233ABCCEC84F0BF85257D3100559FD6/$File/H-0320.pdf
http://researchweb.watson.ibm.com/haifa/projects/storage/datastores/papers/StorletEnginePaper_CameraReady.pdf

This is similar in concept to rados classes for the zerovm stuff, done in 
the context of swift.  It would be great to create a model for doing 
this sort of computation with RGW that takes advantage of rados classes 
where possible.  It's tricky because we're striping data...

sage

Re: storlets

[Yehuda Sadeh]

On Fri, Sep 26, 2014 at 6:43 AM, Sage Weil <sweil@redhat.com> wrote:
> http://domino.research.ibm.com/library/cyberdig.nsf/papers/7233ABCCEC84F0BF85257D3100559FD6/$File/H-0320.pdf
> http://researchweb.watson.ibm.com/haifa/projects/storage/datastores/papers/StorletEnginePaper_CameraReady.pdf
>
> This is similar in concept to rados classes for the zerovm stuff, done in
> the context of swift.  It would be great to create a model for doing
> this sort of computation with RGW that takes advantage of rados classes
> where possible.  It's tricky because we're striping data...
>

These can still run on the rgw nodes (similar to the storlets running
on the proxy nodes in the swift case). Ones that run on the data nodes
will need to be stripe-aware obviously. Another possibility is to not
stripe, and have a large object size limit. Swift has a 5GB object
size limit iirc, so it's not much different. Of course, there are
other issues in ceph with larger objects (e.g., deep scrubbing), so
these will need to be addressed too.

Yehuda