ceph/rbd benchmarks

ceph/rbd benchmarks

[Marcus Sorensen]

Just thought I'd share this basic testing I did, comparing cephfs 0.32
on 3.1-rc1 to nfs as well as rbd to iscsi. I'm sure you guys see a lot
of this. Any feedback would be appreciated.

The data is here:

http://learnitwithme.com/wp-content/uploads/2011/08/ceph-nfs-iscsi-benchmarks.ods

and the writeup is here:

http://learnitwithme.com/?p=303

Re: ceph/rbd benchmarks

[Gregory Farnum]

On Wed, Aug 24, 2011 at 8:29 AM, Marcus Sorensen <shadowsor@gmail.com> wrote:
> Just thought I'd share this basic testing I did, comparing cephfs 0.32
> on 3.1-rc1 to nfs as well as rbd to iscsi. I'm sure you guys see a lot
> of this. Any feedback would be appreciated.
>
> The data is here:
>
> http://learnitwithme.com/wp-content/uploads/2011/08/ceph-nfs-iscsi-benchmarks.ods
>
> and the writeup is here:
>
> http://learnitwithme.com/?p=303

We see less of it than you'd think, actually. Thanks!

To address a few things specifically
Ceph is both the name of the project and of the POSIX-compliant
filesystem. RADOS stands for Reliable Autonomous Distributed Object
Store. Apparently we should publish this a bit more. :)

Looks like most of the differences in your tests have to do with our
relatively lousy read performance -- this is probably due to lousy
readahead, which nobody's spent a lot of time optimizing as we focus
on stability. Sage made some improvements a few weeks ago but I don't
remember what version of stuff they ended up in. :) (Optimizing
cross-server reads is hard!)

Re: ceph/rbd benchmarks

[Sage Weil]

On Wed, 24 Aug 2011, Gregory Farnum wrote:
> On Wed, Aug 24, 2011 at 8:29 AM, Marcus Sorensen <shadowsor@gmail.com> wrote:
> > Just thought I'd share this basic testing I did, comparing cephfs 0.32
> > on 3.1-rc1 to nfs as well as rbd to iscsi. I'm sure you guys see a lot
> > of this. Any feedback would be appreciated.
> >
> > The data is here:
> >
> > http://learnitwithme.com/wp-content/uploads/2011/08/ceph-nfs-iscsi-benchmarks.ods
> >
> > and the writeup is here:
> >
> > http://learnitwithme.com/?p=303
> 
> We see less of it than you'd think, actually. Thanks!
> 
> To address a few things specifically
> Ceph is both the name of the project and of the POSIX-compliant
> filesystem. RADOS stands for Reliable Autonomous Distributed Object
> Store. Apparently we should publish this a bit more. :)
> 
> Looks like most of the differences in your tests have to do with our
> relatively lousy read performance -- this is probably due to lousy
> readahead, which nobody's spent a lot of time optimizing as we focus
> on stability. Sage made some improvements a few weeks ago but I don't
> remember what version of stuff they ended up in. :) (Optimizing
> cross-server reads is hard!)

The readahead improvements are in the 'master' branch of ceph-client.git, 
and will go upstream for Linux 3.2-rc1 (I just missed the 3.1-rc1 cutoff).  
In my tests I was limited by the wire speed with these patches.  I'm 
guessing you were using 3.0 or earlier kernel?

The file copy test was also surprising.  I think there is a regression 
there somewhere, taking a look.

sage

Re: ceph/rbd benchmarks

[Marcus Sorensen]

I knew I read the acronym in a pdf somewhere (something from '08 I
think), but I couldn't find it when I needed it, thanks.

Everything was running 3.1-rc1, I checked the kernel source before
building and it already included the following patches, so I assumed I
was good on the readahead thing.

https://patchwork.kernel.org/patch/1001462/
https://patchwork.kernel.org/patch/1001432/

On Wed, Aug 24, 2011 at 10:46 AM, Sage Weil <sage@newdream.net> wrote:
> On Wed, 24 Aug 2011, Gregory Farnum wrote:
>> On Wed, Aug 24, 2011 at 8:29 AM, Marcus Sorensen <shadowsor@gmail.com> wrote:
>> > Just thought I'd share this basic testing I did, comparing cephfs 0.32
>> > on 3.1-rc1 to nfs as well as rbd to iscsi. I'm sure you guys see a lot
>> > of this. Any feedback would be appreciated.
>> >
>> > The data is here:
>> >
>> > http://learnitwithme.com/wp-content/uploads/2011/08/ceph-nfs-iscsi-benchmarks.ods
>> >
>> > and the writeup is here:
>> >
>> > http://learnitwithme.com/?p=303
>>
>> We see less of it than you'd think, actually. Thanks!
>>
>> To address a few things specifically
>> Ceph is both the name of the project and of the POSIX-compliant
>> filesystem. RADOS stands for Reliable Autonomous Distributed Object
>> Store. Apparently we should publish this a bit more. :)
>>
>> Looks like most of the differences in your tests have to do with our
>> relatively lousy read performance -- this is probably due to lousy
>> readahead, which nobody's spent a lot of time optimizing as we focus
>> on stability. Sage made some improvements a few weeks ago but I don't
>> remember what version of stuff they ended up in. :) (Optimizing
>> cross-server reads is hard!)
>
> The readahead improvements are in the 'master' branch of ceph-client.git,
> and will go upstream for Linux 3.2-rc1 (I just missed the 3.1-rc1 cutoff).
> In my tests I was limited by the wire speed with these patches.  I'm
> guessing you were using 3.0 or earlier kernel?
>
> The file copy test was also surprising.  I think there is a regression
> there somewhere, taking a look.
>
> sage
>
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph/rbd benchmarks

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3000 bytes --]

On Wed, 24 Aug 2011, Marcus Sorensen wrote:
> I knew I read the acronym in a pdf somewhere (something from '08 I
> think), but I couldn't find it when I needed it, thanks.
> 
> Everything was running 3.1-rc1, I checked the kernel source before
> building and it already included the following patches, so I assumed I
> was good on the readahead thing.
> 
> https://patchwork.kernel.org/patch/1001462/
> https://patchwork.kernel.org/patch/1001432/

Yeah, those two help marginally, but the big fix is 

http://ceph.newdream.net/git/?p=ceph-client.git;a=commit;h=78e669966f994964581167c6e25c83d22ebb26c6

and you'd probably also want

http://ceph.newdream.net/git/?p=ceph-client.git;a=commitdiff;h=6468bfe33c8e674509e39e43fad6bc833398fee2

Those are in linux-next and will be sent upstream for 3.2-rc1.

Not sure if it's worth rerunning your tests just yet (I want to look at 
the MDS stuff still), but it should fix the sequential read performance.

sage


> 
> On Wed, Aug 24, 2011 at 10:46 AM, Sage Weil <sage@newdream.net> wrote:
> > On Wed, 24 Aug 2011, Gregory Farnum wrote:
> >> On Wed, Aug 24, 2011 at 8:29 AM, Marcus Sorensen <shadowsor@gmail.com> wrote:
> >> > Just thought I'd share this basic testing I did, comparing cephfs 0.32
> >> > on 3.1-rc1 to nfs as well as rbd to iscsi. I'm sure you guys see a lot
> >> > of this. Any feedback would be appreciated.
> >> >
> >> > The data is here:
> >> >
> >> > http://learnitwithme.com/wp-content/uploads/2011/08/ceph-nfs-iscsi-benchmarks.ods
> >> >
> >> > and the writeup is here:
> >> >
> >> > http://learnitwithme.com/?p=303
> >>
> >> We see less of it than you'd think, actually. Thanks!
> >>
> >> To address a few things specifically
> >> Ceph is both the name of the project and of the POSIX-compliant
> >> filesystem. RADOS stands for Reliable Autonomous Distributed Object
> >> Store. Apparently we should publish this a bit more. :)
> >>
> >> Looks like most of the differences in your tests have to do with our
> >> relatively lousy read performance -- this is probably due to lousy
> >> readahead, which nobody's spent a lot of time optimizing as we focus
> >> on stability. Sage made some improvements a few weeks ago but I don't
> >> remember what version of stuff they ended up in. :) (Optimizing
> >> cross-server reads is hard!)
> >
> > The readahead improvements are in the 'master' branch of ceph-client.git,
> > and will go upstream for Linux 3.2-rc1 (I just missed the 3.1-rc1 cutoff).
> > In my tests I was limited by the wire speed with these patches.  I'm
> > guessing you were using 3.0 or earlier kernel?
> >
> > The file copy test was also surprising.  I think there is a regression
> > there somewhere, taking a look.
> >
> > sage
> >
> >
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: ceph/rbd benchmarks

[Marcus Sorensen]

I applied the patches to the client's existing kernel version, here
are the results with respect to the pre-patch:

seq char write	100.60%
seq blk write	97.18%
seq blk rewrite	120.07%
seq char read	161.52%
seq blk read	162.30%
seeks	141.93%
seq create	76.02%
seq read	186.78%
seq delete	103.18%
rand create	91.50%
rand read	86.34%
rand delete	84.94%


seq reads are now limited by the NIC like the other filesystems.
Rewrite also got a nice bump, as well as the other sequential reads,
but strangely the sequential create is down a non-trivial amount. I
expected the possibility of a small hit to random.

On Wed, Aug 24, 2011 at 11:51 AM, Sage Weil <sage@newdream.net> wrote:
> On Wed, 24 Aug 2011, Marcus Sorensen wrote:
>> I knew I read the acronym in a pdf somewhere (something from '08 I
>> think), but I couldn't find it when I needed it, thanks.
>>
>> Everything was running 3.1-rc1, I checked the kernel source before
>> building and it already included the following patches, so I assumed I
>> was good on the readahead thing.
>>
>> https://patchwork.kernel.org/patch/1001462/
>> https://patchwork.kernel.org/patch/1001432/
>
> Yeah, those two help marginally, but the big fix is
>
> http://ceph.newdream.net/git/?p=ceph-client.git;a=commit;h=78e669966f994964581167c6e25c83d22ebb26c6
>
> and you'd probably also want
>
> http://ceph.newdream.net/git/?p=ceph-client.git;a=commitdiff;h=6468bfe33c8e674509e39e43fad6bc833398fee2
>
> Those are in linux-next and will be sent upstream for 3.2-rc1.
>
> Not sure if it's worth rerunning your tests just yet (I want to look at
> the MDS stuff still), but it should fix the sequential read performance.
>
> sage
>
>
>>
>> On Wed, Aug 24, 2011 at 10:46 AM, Sage Weil <sage@newdream.net> wrote:
>> > On Wed, 24 Aug 2011, Gregory Farnum wrote:
>> >> On Wed, Aug 24, 2011 at 8:29 AM, Marcus Sorensen <shadowsor@gmail.com> wrote:
>> >> > Just thought I'd share this basic testing I did, comparing cephfs 0.32
>> >> > on 3.1-rc1 to nfs as well as rbd to iscsi. I'm sure you guys see a lot
>> >> > of this. Any feedback would be appreciated.
>> >> >
>> >> > The data is here:
>> >> >
>> >> > http://learnitwithme.com/wp-content/uploads/2011/08/ceph-nfs-iscsi-benchmarks.ods
>> >> >
>> >> > and the writeup is here:
>> >> >
>> >> > http://learnitwithme.com/?p=303
>> >>
>> >> We see less of it than you'd think, actually. Thanks!
>> >>
>> >> To address a few things specifically
>> >> Ceph is both the name of the project and of the POSIX-compliant
>> >> filesystem. RADOS stands for Reliable Autonomous Distributed Object
>> >> Store. Apparently we should publish this a bit more. :)
>> >>
>> >> Looks like most of the differences in your tests have to do with our
>> >> relatively lousy read performance -- this is probably due to lousy
>> >> readahead, which nobody's spent a lot of time optimizing as we focus
>> >> on stability. Sage made some improvements a few weeks ago but I don't
>> >> remember what version of stuff they ended up in. :) (Optimizing
>> >> cross-server reads is hard!)
>> >
>> > The readahead improvements are in the 'master' branch of ceph-client.git,
>> > and will go upstream for Linux 3.2-rc1 (I just missed the 3.1-rc1 cutoff).
>> > In my tests I was limited by the wire speed with these patches.  I'm
>> > guessing you were using 3.0 or earlier kernel?
>> >
>> > The file copy test was also surprising.  I think there is a regression
>> > there somewhere, taking a look.
>> >
>> > sage
>> >
>> >
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html