writes-starving-reads problem

writes-starving-reads problem

[Henry C Chang]

Hi,

I ran into "writes-starving-reads" problem on a machine with 48G RAM.

I have two processes running concurrently:
Process 1 copies an 8 GB file from local disk to CEPH.
Process 2 copies an 8 GB file from CEPH to local disk.

Process 1 finished quickly since the data were cached in memory first.
Process 2 was blocked until all the cached data have been written to OSD.
(Maybe not all the cached data, but I waited for a long time.)

I don't have this problem on a machine with 8G RAM.

Has anyone experienced this before or any idea why?

Thanks,
Henry

Re: writes-starving-reads problem

[Gregory Farnum]

On Fri, Jul 30, 2010 at 2:40 AM, Henry C Chang
<henry_c_chang@tcloudcomputing.com> wrote:
> Hi,
>
> I ran into "writes-starving-reads" problem on a machine with 48G RAM.
>
> I have two processes running concurrently:
> Process 1 copies an 8 GB file from local disk to CEPH.
> Process 2 copies an 8 GB file from CEPH to local disk.
>
> Process 1 finished quickly since the data were cached in memory first.
> Process 2 was blocked until all the cached data have been written to OSD.
> (Maybe not all the cached data, but I waited for a long time.)
>
> I don't have this problem on a machine with 8G RAM.
>
> Has anyone experienced this before or any idea why?
>
> Thanks,
> Henry
We haven't heard of anything like this ourselves, although locally
we're not using such beefy machines yet.
Can you give us more information on your cluster configuration and how
the problem presented itself? Did the processes have progress bars and
you watched the ceph->local have no progress until the local->ceph
completed, or did the ceph->local copy just take much longer?
-Greg