workload balance

workload balance

[huang jun]

hi，all
Here is a problem confused me a lot. Some OSD's workload will be very high
when the client read a file continuously. So i think the cluster must
have a strategy to balance workload
there are two classes "MovingAverager and  IATAverager ",but i can not
figure out what the class IATAverager mean here?
and futhermore, can we bring tiered storage theory in ceph?

thanks!
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: workload balance

[Josh Durgin]

On 06/27/2011 02:34 AM, huang jun wrote:
> hi，all
> Here is a problem confused me a lot. Some OSD's workload will be very high
> when the client read a file continuously. So i think the cluster must
> have a strategy to balance workload
> there are two classes "MovingAverager and  IATAverager ",but i can not
> figure out what the class IATAverager mean here?
> and futhermore, can we bring tiered storage theory in ceph?
> 
> thanks!

Those classes were attempts to use some statistics to discard some
reads, but they did not work so they've been removed from master.
Our current strategy is simply striping files across objects.

Josh
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: workload balance

[huang jun]

thanks，Josh
By default，we set two replicas for each PG,so if we use ceph
as back-end storage of a website, you know, some files will be frequently read,
if then of thousands clients do this, some osd's workload will be very high.
so in this circumstance, how to balance the whole  cluster's workload?

在 2011年6月28日 上午5:06，Josh Durgin <josh.durgin@dreamhost.com> 写道：
> On 06/27/2011 02:34 AM, huang jun wrote:
>> hi，all
>> Here is a problem confused me a lot. Some OSD's workload will be very high
>> when the client read a file continuously. So i think the cluster must
>> have a strategy to balance workload
>> there are two classes "MovingAverager and  IATAverager ",but i can not
>> figure out what the class IATAverager mean here?
>> and futhermore, can we bring tiered storage theory in ceph?
>>
>> thanks!
>
> Those classes were attempts to use some statistics to discard some
> reads, but they did not work so they've been removed from master.
> Our current strategy is simply striping files across objects.
>
> Josh
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: workload balance

[Josh Durgin]

On 06/27/2011 05:25 PM, huang jun wrote:
> thanks，Josh
> By default，we set two replicas for each PG,so if we use ceph
> as back-end storage of a website, you know, some files will be frequently read,
> if then of thousands clients do this, some osd's workload will be very high.
> so in this circumstance, how to balance the whole  cluster's workload?

If the files don't change often, they can be cached by the clients. If
there really is one object that is being updated and read frequently,
there's not much you can do currently. To reduce the load on the primary
OSD, we could add a flag to the MDS to tell clients to read from
replicas based on the usage.

Josh
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: workload balance

[huang jun]

thank you Josh
Currently,we can reduce the primary workload by distributing read
requests to replicas,
but once a object becomes a extremly hot point, this stratey does not
make much difference, so can we bring Tiered Storage into ceph, just
like to use HSM or ILM? In this way, client can got data more
effeciently.

best regards!

在 2011年7月1日 上午2:04，Josh Durgin <josh.durgin@dreamhost.com> 写道：
> On 06/27/2011 05:25 PM, huang jun wrote:
>> thanks，Josh
>> By default，we set two replicas for each PG,so if we use ceph
>> as back-end storage of a website, you know, some files will be frequently read,
>> if then of thousands clients do this, some osd's workload will be very high.
>> so in this circumstance, how to balance the whole  cluster's workload?
>
> If the files don't change often, they can be cached by the clients. If
> there really is one object that is being updated and read frequently,
> there's not much you can do currently. To reduce the load on the primary
> OSD, we could add a flag to the MDS to tell clients to read from
> replicas based on the usage.
>
> Josh
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: workload balance

[srimugunthan dhandapani]

2011/6/30 Josh Durgin <josh.durgin@dreamhost.com>
>
> On 06/27/2011 05:25 PM, huang jun wrote:
> > thanks，Josh
> > By default，we set two replicas for each PG,so if we use ceph
> > as back-end storage of a website, you know, some files will be frequently read,
> > if then of thousands clients do this, some osd's workload will be very high.
> > so in this circumstance, how to balance the whole  cluster's workload?
>
> If the files don't change often, they can be cached by the clients. If
> there really is one object that is being updated and read frequently,
> there's not much you can do currently. To reduce the load on the primary
> OSD, we could add a flag to the MDS to tell clients to read from
> replicas based on the usage.


If a particular file is updated heavily, if we can change the inode
number of the heavily updated file, then the objects will be remapped
to new locations and can result in balancing.
Will that be a good solution to implement?

>
> Josh
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: workload balance

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1808 bytes --]

On Sun, 17 Jul 2011, srimugunthan dhandapani wrote:
> 2011/6/30 Josh Durgin <josh.durgin@dreamhost.com>
> >
> > On 06/27/2011 05:25 PM, huang jun wrote:
> > > thanks，Josh
> > > By default，we set two replicas for each PG,so if we use ceph
> > > as back-end storage of a website, you know, some files will be frequently read,
> > > if then of thousands clients do this, some osd's workload will be very high.
> > > so in this circumstance, how to balance the whole  cluster's workload?
> >
> > If the files don't change often, they can be cached by the clients. If
> > there really is one object that is being updated and read frequently,
> > there's not much you can do currently. To reduce the load on the primary
> > OSD, we could add a flag to the MDS to tell clients to read from
> > replicas based on the usage.
> 
> 
> If a particular file is updated heavily, if we can change the inode
> number of the heavily updated file, then the objects will be remapped
> to new locations and can result in balancing.
> Will that be a good solution to implement?

I'm not sure that would help.  If the inode changes (a big if), then the 
existing data has to move too, and you probably don't win anything.

The challenge with many writers in general is keeping the writes atomic 
and (logically) serialized.  That's simple enough if they all go through a 
single node.  The second problem is that, even with some clever way to 
distribute that work (some tree hierarchy aggregating writes in front of 
the final object, say), the clients have to know when to do that (vs the 
simple approach in the general case).

Do you really have thousands of clients writing to the same 4MB range of a 
file?  (Remember the file striping parameters can be adjusted to change 
that.)

sage