Some Question Regarding to RBD performance

Some Question Regarding to RBD performance

[Leander Yu]

Hi all,
I have a question regarding to RBD performance. Recently I test RBD in
2 cluster, using dd if=/dev/zero of=/dev/rbd5 bs=4M count=1024
oflag=direct. I got 33.4MB/s in one cluster , however got only 6MB/s
in the other cluster.

I have confirmed that the network bandwidth is ok. and osd bench shows
around 32 ~ 40 MB/s on all OSD. Is this performance normal? if not ,
how can I troubleshoot it?
I have move journal to ramdisk for testing and get performance boost
to 7x MB/s. Does this means the HDD is the bottleneck ?

The other cluster that got 33.4 MB/s , it's osd bench shows around
44MB/s. The major difference between those two cluster is the better
performed one use SAS with RAID controller the other use SATA with no
RAID control.

Thanks.

Regards,
Leander Yu.

Re: Some Question Regarding to RBD performance

[Sage Weil]

Hi Leander,

On Mon, 26 Dec 2011, Leander Yu wrote:
> Hi all,
> I have a question regarding to RBD performance. Recently I test RBD in
> 2 cluster, using dd if=/dev/zero of=/dev/rbd5 bs=4M count=1024
> oflag=direct. I got 33.4MB/s in one cluster , however got only 6MB/s
> in the other cluster.
> 
> I have confirmed that the network bandwidth is ok. and osd bench shows
> around 32 ~ 40 MB/s on all OSD. Is this performance normal? if not ,
> how can I troubleshoot it?
> I have move journal to ramdisk for testing and get performance boost
> to 7x MB/s. Does this means the HDD is the bottleneck ?
> 
> The other cluster that got 33.4 MB/s , it's osd bench shows around
> 44MB/s. The major difference between those two cluster is the better
> performed one use SAS with RAID controller the other use SATA with no
> RAID control.

If you are using the kernel client, each 4M write is fully synchronous, 
which means (among other things) that the client is completely idle while 
the write is being replicated by the OSDs.  This makes the performance 
very sensitive to the write latency; it's not really testing write 
throughput.  Your SAS RAID controller is probably doing writeback caching 
of some kind that keeps the write latency low, while the SATA disks 
clearly are not.

'dd oflag=direct' is probably not the benchmark you want; no application 
should do a single synchronous IO at a time and expect good throughput.  
The more interesting number would be steady-state buffered write 
throughput when you have a file system on top, or something along those 
lines.

sage

Re: Some Question Regarding to RBD performance

[Leander Yu]

Hi Sage,
Thanks for the feedback. We found the performance issue actually from
the journal writing. even we separate journal into the other SATA HDD,
it didn't shows any performance difference.
At the end , we use SSD to store the journal file which give us the
performance 52.3MB/s with oflag=direct. compare SATA 6MB/s
I didn't have chance to trace the code for journal writing , however I
am wondering that is there a way to improve the journal writing so we
can get better performance in SATA HDD.

Thanks.

Regards,
Leander Yu.

On Tue, Dec 27, 2011 at 12:27 AM, Sage Weil <sage@newdream.net> wrote:
> Hi Leander,
>
> On Mon, 26 Dec 2011, Leander Yu wrote:
>> Hi all,
>> I have a question regarding to RBD performance. Recently I test RBD in
>> 2 cluster, using dd if=/dev/zero of=/dev/rbd5 bs=4M count=1024
>> oflag=direct. I got 33.4MB/s in one cluster , however got only 6MB/s
>> in the other cluster.
>>
>> I have confirmed that the network bandwidth is ok. and osd bench shows
>> around 32 ~ 40 MB/s on all OSD. Is this performance normal? if not ,
>> how can I troubleshoot it?
>> I have move journal to ramdisk for testing and get performance boost
>> to 7x MB/s. Does this means the HDD is the bottleneck ?
>>
>> The other cluster that got 33.4 MB/s , it's osd bench shows around
>> 44MB/s. The major difference between those two cluster is the better
>> performed one use SAS with RAID controller the other use SATA with no
>> RAID control.
>
> If you are using the kernel client, each 4M write is fully synchronous,
> which means (among other things) that the client is completely idle while
> the write is being replicated by the OSDs.  This makes the performance
> very sensitive to the write latency; it's not really testing write
> throughput.  Your SAS RAID controller is probably doing writeback caching
> of some kind that keeps the write latency low, while the SATA disks
> clearly are not.
>
> 'dd oflag=direct' is probably not the benchmark you want; no application
> should do a single synchronous IO at a time and expect good throughput.
> The more interesting number would be steady-state buffered write
> throughput when you have a file system on top, or something along those
> lines.
>
> sage
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Some Question Regarding to RBD performance

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3037 bytes --]

On Tue, 27 Dec 2011, Leander Yu wrote:
> Thanks for the feedback. We found the performance issue actually from
> the journal writing. even we separate journal into the other SATA HDD,
> it didn't shows any performance difference.
> At the end , we use SSD to store the journal file which give us the
> performance 52.3MB/s with oflag=direct. compare SATA 6MB/s
> I didn't have chance to trace the code for journal writing , however I
> am wondering that is there a way to improve the journal writing so we
> can get better performance in SATA HDD.

We can.  The FileJournal code is currently doing a single sync direct-io 
write to the device, which means that you usually do a full disk rotation 
between each write.  It needs to do aio so that we can keep multiple 
writes in flight.

http://tracker.newdream.net/issues/1836

This would be a great thing for an external contributor to work on, as 
it's not dependent on anything else!

Thanks-
sage


> 
> Thanks.
> 
> Regards,
> Leander Yu.
> 
> On Tue, Dec 27, 2011 at 12:27 AM, Sage Weil <sage@newdream.net> wrote:
> > Hi Leander,
> >
> > On Mon, 26 Dec 2011, Leander Yu wrote:
> >> Hi all,
> >> I have a question regarding to RBD performance. Recently I test RBD in
> >> 2 cluster, using dd if=/dev/zero of=/dev/rbd5 bs=4M count=1024
> >> oflag=direct. I got 33.4MB/s in one cluster , however got only 6MB/s
> >> in the other cluster.
> >>
> >> I have confirmed that the network bandwidth is ok. and osd bench shows
> >> around 32 ~ 40 MB/s on all OSD. Is this performance normal? if not ,
> >> how can I troubleshoot it?
> >> I have move journal to ramdisk for testing and get performance boost
> >> to 7x MB/s. Does this means the HDD is the bottleneck ?
> >>
> >> The other cluster that got 33.4 MB/s , it's osd bench shows around
> >> 44MB/s. The major difference between those two cluster is the better
> >> performed one use SAS with RAID controller the other use SATA with no
> >> RAID control.
> >
> > If you are using the kernel client, each 4M write is fully synchronous,
> > which means (among other things) that the client is completely idle while
> > the write is being replicated by the OSDs.  This makes the performance
> > very sensitive to the write latency; it's not really testing write
> > throughput.  Your SAS RAID controller is probably doing writeback caching
> > of some kind that keeps the write latency low, while the SATA disks
> > clearly are not.
> >
> > 'dd oflag=direct' is probably not the benchmark you want; no application
> > should do a single synchronous IO at a time and expect good throughput.
> > The more interesting number would be steady-state buffered write
> > throughput when you have a file system on top, or something along those
> > lines.
> >
> > sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
>