ceph, slow (cheap) and fast (expensive) storage?

ceph, slow (cheap) and fast (expensive) storage?

[Tomasz Chmielewski]

I'd like to build a cluster for storing large amounts of data.

The cluster would consist of slow (cheap) storage and fast (expensive) 
storage.

I would like to migrate data from fast (expensive) storage to slow 
(cheap) storage if it wasn't accessed for, say, 6 months.

Would ceph somehow help me achieve that?

If not, do you know anything which comes to mind?



-- 
Tomasz Chmielewski
http://wpkg.org

Re: ceph, slow (cheap) and fast (expensive) storage?

[Wido den Hollander]

Hi,

I'm not aware of any of those features in Ceph and i don't think that it
will be possible in the near future.

But while thinking about it, is ZFS an option for you? You could then
use a lot of SATA disks and add some SAS disks for your caching. ZFS
will then place the most used blocks on the SAS disks.

-- 
Met vriendelijke groet,

Wido den Hollander
Hoofd Systeembeheer / CSO
Telefoon Support Nederland: 0900 9633 (45 cpm)
Telefoon Support België: 0900 70312 (45 cpm)
Telefoon Direct: (+31) (0)20 50 60 104
Fax: +31 (0)20 50 60 111
E-mail: support@pcextreme.nl
Website: http://www.pcextreme.nl
Kennisbank: http://support.pcextreme.nl/
Netwerkstatus: http://nmc.pcextreme.nl


On Tue, 2010-06-08 at 10:30 +0000, Tomasz Chmielewski wrote:
> I'd like to build a cluster for storing large amounts of data.
> 
> The cluster would consist of slow (cheap) storage and fast (expensive) 
> storage.
> 
> I would like to migrate data from fast (expensive) storage to slow 
> (cheap) storage if it wasn't accessed for, say, 6 months.
> 
> Would ceph somehow help me achieve that?
> 
> If not, do you know anything which comes to mind?
> 
> 
> 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph, slow (cheap) and fast (expensive) storage?

[Tomasz Chmielewski]

Am 08.06.2010 12:53, Wido den Hollander wrote:
> Hi,
>
> I'm not aware of any of those features in Ceph and i don't think that it
> will be possible in the near future.
>
> But while thinking about it, is ZFS an option for you? You could then
> use a lot of SATA disks and add some SAS disks for your caching. ZFS
> will then place the most used blocks on the SAS disks.

Not sure, perhaps not.

I was thinking of something in the line of:

http://en.wikipedia.org/wiki/Hierarchical_storage_management


-- 
Tomasz Chmielewski
http://wpkg.org

Re: ceph, slow (cheap) and fast (expensive) storage?

[Thomas Mueller]

Am Tue, 08 Jun 2010 12:30:04 +0200 schrieb Tomasz Chmielewski:

> I'd like to build a cluster for storing large amounts of data.
> 
> The cluster would consist of slow (cheap) storage and fast (expensive)
> storage.
> 
> I would like to migrate data from fast (expensive) storage to slow
> (cheap) storage if it wasn't accessed for, say, 6 months.
> 
> Would ceph somehow help me achieve that?


I would not expect to be the fast (expensive) storage to be that faster 
in a ceph cluster. if you get enough cheap storage nodes it will scale 
with the amount of disks/nics as the date is striped among all the nodes 
(osd's). But this is just my speculation.

if you have some amount storage nodes and you are on GigE you will hit 
the ethernet limit on the client faster than anything else. 

"cheap" not in sens of crap hardware. cheap as in SATA 24x7 enterprise 
drives.

and remember:

 ** WARNING: Ceph is still under heavy development, and is only suitable for **
 **          testing and review.  Do not trust it with important data.       **

- Thomas

PS: "cheap" storage selfmade: 
http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-
cheap-cloud-storage/

Re: ceph, slow (cheap) and fast (expensive) storage?

[ales-76]

Hello,

During the last year Ottawa Linux Symposium there was a presentation of open-souce implementation of Hierarchical Storage Management framework for Linux. See: http://www.kernel.org/doc/ols/2009/ols2009-pages-263-274.pdf AFAK it is only a prototype, works only with ext2/ext3 and no other open-source storage system offers anything like this just yet. So I guess you have to wait a little. But it sounds like useful feature and it would be nice if Ceph can do something like that. Would it be possible for Ceph to change data placement depending on by example file popularity?

Ales Blaha

> ------------ Původní zpráva ------------
> Od: Tomasz Chmielewski <mangoo@wpkg.org>
> Předmět: Re: ceph, slow (cheap) and fast (expensive) storage?
> Datum: 08.6.2010 14:04:17
> ----------------------------------------
> Am 08.06.2010 12:53, Wido den Hollander wrote:
> > Hi,
> >
> > I'm not aware of any of those features in Ceph and i don't think that it
> > will be possible in the near future.
> >
> > But while thinking about it, is ZFS an option for you? You could then
> > use a lot of SATA disks and add some SAS disks for your caching. ZFS
> > will then place the most used blocks on the SAS disks.
> 
> Not sure, perhaps not.
> 
> I was thinking of something in the line of:
> 
> http://en.wikipedia.org/wiki/Hierarchical_storage_management
> 
> 
> -- 
> Tomasz Chmielewski
> http://wpkg.org
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph, slow (cheap) and fast (expensive) storage?

[Sage Weil]

On Tue, 8 Jun 2010, Tomasz Chmielewski wrote:
> I'd like to build a cluster for storing large amounts of data.
> 
> The cluster would consist of slow (cheap) storage and fast (expensive)
> storage.
> 
> I would like to migrate data from fast (expensive) storage to slow (cheap)
> storage if it wasn't accessed for, say, 6 months.
> 
> Would ceph somehow help me achieve that?

We don't currently have that functionality.  In principle it's certainly 
something the system could do, and the underlying storage architecture 
lends itself to that: you can create different object storage pools, with 
different data distribution (crush) rules, mapped to different devices, 
etc.  What's missing is the HSM process that goes and finds old, idle data 
and migrates it.  And atime tracking.

You can also use a mix of small+fast and big+slow disks/ssds with a single 
pool of storage by writing a crush rule that puts the primary replica on 
the fast disks and additional replicas on slow disks.  (The primary 
replicas handle the read workload.)  That approach fixes the ratio between 
fast and slow capacity, though.  You could also build each osd volume with 
a mix of fast and slow disks/ssds, and use btrfs or some other local fs to 
manage the distribution of objects across disk types.  (Btrfs doesn't do 
that yet, but presumably will at some point.)

sage

Re: ceph, slow (cheap) and fast (expensive) storage?

[Sage Weil]

On Tue, 8 Jun 2010, Thomas Mueller wrote:
> I would not expect to be the fast (expensive) storage to be that faster 
> in a ceph cluster. if you get enough cheap storage nodes it will scale 
> with the amount of disks/nics as the date is striped among all the nodes 
> (osd's). But this is just my speculation.
> 
> if you have some amount storage nodes and you are on GigE you will hit 
> the ethernet limit on the client faster than anything else. 

You can also think of "fast" and "slow" in terms of the ratio of io 
bandwidth to capacity on each individual device.  For example you can use 
lots of SATA disks in parallel to get more IOPs and throughput, but the 
platters will be mostly empty.  It's often a question of how much total 
data you have and how much IO to it you need to serve. 

Of course, there are also absolute performance considerations (SSD read 
latencies are lower than a disk), so both factors should be considered.

sage

Re: ceph, slow (cheap) and fast (expensive) storage?

[Tomasz Chmielewski]

Am 08.06.2010 19:51, Sage Weil wrote:
> On Tue, 8 Jun 2010, Tomasz Chmielewski wrote:
>> I'd like to build a cluster for storing large amounts of data.
>>
>> The cluster would consist of slow (cheap) storage and fast (expensive)
>> storage.
>>
>> I would like to migrate data from fast (expensive) storage to slow (cheap)
>> storage if it wasn't accessed for, say, 6 months.
>>
>> Would ceph somehow help me achieve that?
>
> We don't currently have that functionality.  In principle it's certainly
> something the system could do, and the underlying storage architecture
> lends itself to that:

Hmm, since quite recently, Linux kernel has FS-Cache, which:

	This facility is a general purpose cache for network
	filesystems, though it could be used for caching other things
	such as ISO9660 filesystems too.


I wonder if it could work with ceph and/or traditional filesystems, too?


-- 
Tomasz Chmielewski
http://wpkg.org

Re: ceph, slow (cheap) and fast (expensive) storage?

[Wido den Hollander]

Hi Thomas,

For every backend (like NFS, Ceph, CIFS, etc, etc) FS-Cache needs a
custom module. So in theory it should work with Ceph, but someone would
have to write a backend for it.

I don't really know what the benefit would be for Ceph, sometimes
caching can slow things down rather then speeding up.

-- 
Met vriendelijke groet,

Wido den Hollander
Hoofd Systeembeheer / CSO
Telefoon Support Nederland: 0900 9633 (45 cpm)
Telefoon Support België: 0900 70312 (45 cpm)
Telefoon Direct: (+31) (0)20 50 60 104
Fax: +31 (0)20 50 60 111
E-mail: support@pcextreme.nl
Website: http://www.pcextreme.nl
Kennisbank: http://support.pcextreme.nl/
Netwerkstatus: http://nmc.pcextreme.nl


On Wed, 2010-06-09 at 14:55 +0000, Tomasz Chmielewski wrote:
> Am 08.06.2010 19:51, Sage Weil wrote:
> > On Tue, 8 Jun 2010, Tomasz Chmielewski wrote:
> >> I'd like to build a cluster for storing large amounts of data.
> >>
> >> The cluster would consist of slow (cheap) storage and fast (expensive)
> >> storage.
> >>
> >> I would like to migrate data from fast (expensive) storage to slow (cheap)
> >> storage if it wasn't accessed for, say, 6 months.
> >>
> >> Would ceph somehow help me achieve that?
> >
> > We don't currently have that functionality.  In principle it's certainly
> > something the system could do, and the underlying storage architecture
> > lends itself to that:
> 
> Hmm, since quite recently, Linux kernel has FS-Cache, which:
> 
> 	This facility is a general purpose cache for network
> 	filesystems, though it could be used for caching other things
> 	such as ISO9660 filesystems too.
> 
> 
> I wonder if it could work with ceph and/or traditional filesystems, too?
> 
> 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html