On Sun, 17 Jul 2011, srimugunthan dhandapani wrote:
> 2011/6/30 Josh Durgin <josh.durgin@dreamhost.com>
> >
> > On 06/27/2011 05:25 PM, huang jun wrote:
> > > thanksŁ¬Josh
> > > By defaultŁ¬we set two replicas for each PG,so if we use ceph
> > > as back-end storage of a website, you know, some files will be frequently read,
> > > if then of thousands clients do this, some osd's workload will be very high.
> > > so in this circumstance, how to balance the whole  cluster's workload?
> >
> > If the files don't change often, they can be cached by the clients. If
> > there really is one object that is being updated and read frequently,
> > there's not much you can do currently. To reduce the load on the primary
> > OSD, we could add a flag to the MDS to tell clients to read from
> > replicas based on the usage.
> 
> 
> If a particular file is updated heavily, if we can change the inode
> number of the heavily updated file, then the objects will be remapped
> to new locations and can result in balancing.
> Will that be a good solution to implement?

I'm not sure that would help.  If the inode changes (a big if), then the 
existing data has to move too, and you probably don't win anything.

The challenge with many writers in general is keeping the writes atomic 
and (logically) serialized.  That's simple enough if they all go through a 
single node.  The second problem is that, even with some clever way to 
distribute that work (some tree hierarchy aggregating writes in front of 
the final object, say), the clients have to know when to do that (vs the 
simple approach in the general case).

Do you really have thousands of clients writing to the same 4MB range of a 
file?  (Remember the file striping parameters can be adjusted to change 
that.)

sage