code stability (production readiness) and kernel versions

code stability (production readiness) and kernel versions

[Alexandru Ionica]

Hello,

I have been doing several benchmarks (Phoronix test suite, disk suite) and
I'm really impressed with the performance when doing writeback caching. For
the benchmarks I did a git checkout and compiled the 3.1.0+ kernel. As the
wiki is outdated I am wondering if the patches have also been applied on a
kernel version which is used by server oriented distributions, meaning
kernel version 2.6.32 for example or if there is a way to apply the patches
(if they exist separately) to kernel version 2.6.32 .
I am interested in this specific version of the kernel as other constrains
impose it.

Also ... do you think that your code is production ready when using bcache
to do writeback caching ? Of course I will keep testing but I'd like to
know if you think the code by now is production ready.
Basically I plan to run a setup like: bcache device assembled from software
raid10 or raid0 (4 disk) + ssd ; on top of this a volume group ; on top of
logical devices drbd setup . We are running this for a long while without
bcache so the setup is stable and the new part here would be bcache.

P.S. during the benchmarks bcache outperformed in every way flashcache (I
tried two different sequential size settings with flash cache, both
underperformed)

Regards,

Alexandru Ionica

Re: code stability (production readiness) and kernel versions

[Kent Overstreet]

Hey, sorry for the delay. Was travelling and I've been slow to catch
up on email...

On Fri, Apr 6, 2012 at 6:21 AM, Alexandru Ionica
<alexandru.ionica-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hello,
>
> I have been doing several benchmarks (Phoronix test suite, disk suite) and
> I'm really impressed with the performance when doing writeback caching. For
> the benchmarks I did a git checkout and compiled the 3.1.0+ kernel. As the
> wiki is outdated I am wondering if the patches have also been applied on a
> kernel version which is used by server oriented distributions, meaning
> kernel version 2.6.32 for example or if there is a way to apply the patches
> (if they exist separately) to kernel version 2.6.32 .
> I am interested in this specific version of the kernel as other constrains
> impose it.

Any chance you could share those benchmarks? I'll post them on the
wiki (or give you an account). I could really use some benchmarks that
are suitable for sharing, all the benchmarking I've done has been just
focused on optimizing stuff.

> Also ... do you think that your code is production ready when using bcache
> to do writeback caching ? Of course I will keep testing but I'd like to
> know if you think the code by now is production ready.

Yeah, it is. Test it on your configuration, etc. etc. but writeback is
pretty mature and well tested at this point.

> Basically I plan to run a setup like: bcache device assembled from software
> raid10 or raid0 (4 disk) + ssd ; on top of this a volume group ; on top of
> logical devices drbd setup . We are running this for a long while without
> bcache so the setup is stable and the new part here would be bcache.

Sounds pretty reasonable.

> P.S. during the benchmarks bcache outperformed in every way flashcache (I
> tried two different sequential size settings with flash cache, both
> underperformed)

Cool! Would love to see the numbers :)

> Regards,
>
> Alexandru Ionica
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: code stability (production readiness) and kernel versions

[Alexandru Ionica]

Hello,

My employer allowed me to publish the data, so here you go:
http://www.accelcloud.com/2012/04/18/linux-flashcache-and-bcache-performance-testing/

On Tue, Apr 17, 2012 at 10:30 AM, Kent Overstreet
<koverstreet-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> wrote:
> Hey, sorry for the delay. Was travelling and I've been slow to catch
> up on email...
>
> On Fri, Apr 6, 2012 at 6:21 AM, Alexandru Ionica
> <alexandru.ionica-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Hello,
>>
>> I have been doing several benchmarks (Phoronix test suite, disk suite) and
>> I'm really impressed with the performance when doing writeback caching. For
>> the benchmarks I did a git checkout and compiled the 3.1.0+ kernel. As the
>> wiki is outdated I am wondering if the patches have also been applied on a
>> kernel version which is used by server oriented distributions, meaning
>> kernel version 2.6.32 for example or if there is a way to apply the patches
>> (if they exist separately) to kernel version 2.6.32 .
>> I am interested in this specific version of the kernel as other constrains
>> impose it.
>
> Any chance you could share those benchmarks? I'll post them on the
> wiki (or give you an account). I could really use some benchmarks that
> are suitable for sharing, all the benchmarking I've done has been just
> focused on optimizing stuff.
>
>> Also ... do you think that your code is production ready when using bcache
>> to do writeback caching ? Of course I will keep testing but I'd like to
>> know if you think the code by now is production ready.
>
> Yeah, it is. Test it on your configuration, etc. etc. but writeback is
> pretty mature and well tested at this point.
>
>> Basically I plan to run a setup like: bcache device assembled from software
>> raid10 or raid0 (4 disk) + ssd ; on top of this a volume group ; on top of
>> logical devices drbd setup . We are running this for a long while without
>> bcache so the setup is stable and the new part here would be bcache.
>
> Sounds pretty reasonable.
>
>> P.S. during the benchmarks bcache outperformed in every way flashcache (I
>> tried two different sequential size settings with flash cache, both
>> underperformed)
>
> Cool! Would love to see the numbers :)
>
>> Regards,
>>
>> Alexandru Ionica
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: code stability (production readiness) and kernel versions

[Gerrit Tamboer]

Hi Alexandru,

Very interesting results, thanks for sharing!

Regards,
Gerrit

-----Original Message-----
From: linux-bcache-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
[mailto:linux-bcache-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Alexandru Ionica
Sent: woensdag 18 april 2012 16:23
To: Kent Overstreet
Cc: linux-bcache-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Re: code stability (production readiness) and kernel versions

Hello,

My employer allowed me to publish the data, so here you go:
http://www.accelcloud.com/2012/04/18/linux-flashcache-and-bcache-performance
-testing/

On Tue, Apr 17, 2012 at 10:30 AM, Kent Overstreet <koverstreet@google.com>
wrote:
> Hey, sorry for the delay. Was travelling and I've been slow to catch 
> up on email...
>
> On Fri, Apr 6, 2012 at 6:21 AM, Alexandru Ionica 
> <alexandru.ionica-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Hello,
>>
>> I have been doing several benchmarks (Phoronix test suite, disk 
>> suite) and I'm really impressed with the performance when doing 
>> writeback caching. For the benchmarks I did a git checkout and 
>> compiled the 3.1.0+ kernel. As the wiki is outdated I am wondering if 
>> the patches have also been applied on a kernel version which is used 
>> by server oriented distributions, meaning kernel version 2.6.32 for 
>> example or if there is a way to apply the patches (if they exist
separately) to kernel version 2.6.32 .
>> I am interested in this specific version of the kernel as other 
>> constrains impose it.
>
> Any chance you could share those benchmarks? I'll post them on the 
> wiki (or give you an account). I could really use some benchmarks that 
> are suitable for sharing, all the benchmarking I've done has been just 
> focused on optimizing stuff.
>
>> Also ... do you think that your code is production ready when using 
>> bcache to do writeback caching ? Of course I will keep testing but 
>> I'd like to know if you think the code by now is production ready.
>
> Yeah, it is. Test it on your configuration, etc. etc. but writeback is 
> pretty mature and well tested at this point.
>
>> Basically I plan to run a setup like: bcache device assembled from 
>> software
>> raid10 or raid0 (4 disk) + ssd ; on top of this a volume group ; on 
>> top of logical devices drbd setup . We are running this for a long 
>> while without bcache so the setup is stable and the new part here would
be bcache.
>
> Sounds pretty reasonable.
>
>> P.S. during the benchmarks bcache outperformed in every way 
>> flashcache (I tried two different sequential size settings with flash 
>> cache, both
>> underperformed)
>
> Cool! Would love to see the numbers :)
>
>> Regards,
>>
>> Alexandru Ionica
>> --
>> To unsubscribe from this list: send the line "unsubscribe 
>> linux-bcache" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org 
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at
http://vger.kernel.org/majordomo-info.html

Re: code stability (production readiness) and kernel versions

[Brad Campbell]

> 
On 17/04/2012, at 4:30 PM, Kent Overstreet <koverstreet-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> wrote:

> 
> 
>> Also ... do you think that your code is production ready when using bcache
>> to do writeback caching ? Of course I will keep testing but I'd like to
>> know if you think the code by now is production ready.
> 
> Yeah, it is. Test it on your configuration, etc. etc. but writeback is
> pretty mature and well tested at this point.
> 
I've been giving serious thought to using this in a production environment. It performs well in my staging system, but my biggest concern revolves around the ultimate reliability of the ssd. 

How much testing have you performed with regard to cache device failures in a writeback scenario? I like the idea of mirroring the cache devices to replicate dirty data, however I know that is not implemented yet. 

We run a raid10 of SAS cheetahs. I'd love to mount a cache in there, but ultimately we have little information about what happens if the ssd starts to flake out. I guess more than that, there is little real information out there detailing common or potential ssd failure modes.

I'd assume you've performed plenty of testing over the development of bcache. Can you fill us in a bit as to what to expect when things go south?

Regards,
Brad--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: code stability (production readiness) and kernel versions

[Adam Berkan]

If you completely lose a flash device that contains significant
quantities of dirty data, you'll have a hard time recovering any data
from the backing device.  All of the writeback testing we've done so
far has assumed device failures are limited to small regions of the
device, and bcache tries to recover what it can when it encounters
such errors.  It doesn't do anything reasonable in the case of full
device loss.

I would not recommend using bcache in writeback mode anywhere where
you can't afford to lose the whole device.

Adam

On Fri, Apr 20, 2012 at 8:51 AM, Brad Campbell <brad-+nnirC7rrGZibQn6LdNjmg@public.gmane.org> wrote:
>>
> On 17/04/2012, at 4:30 PM, Kent Overstreet <koverstreet-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> wrote:
>
>>
>>
>>> Also ... do you think that your code is production ready when using bcache
>>> to do writeback caching ? Of course I will keep testing but I'd like to
>>> know if you think the code by now is production ready.
>>
>> Yeah, it is. Test it on your configuration, etc. etc. but writeback is
>> pretty mature and well tested at this point.
>>
> I've been giving serious thought to using this in a production environment. It performs well in my staging system, but my biggest concern revolves around the ultimate reliability of the ssd.
>
> How much testing have you performed with regard to cache device failures in a writeback scenario? I like the idea of mirroring the cache devices to replicate dirty data, however I know that is not implemented yet.
>
> We run a raid10 of SAS cheetahs. I'd love to mount a cache in there, but ultimately we have little information about what happens if the ssd starts to flake out. I guess more than that, there is little real information out there detailing common or potential ssd failure modes.
>
> I'd assume you've performed plenty of testing over the development of bcache. Can you fill us in a bit as to what to expect when things go south?
>
> Regards,
> Brad--
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: code stability (production readiness) and kernel versions

[Roberto Alcântara]

Adam,

What about use bache in raid1 flash devices?

         - Roberto


On Fri, Apr 20, 2012 at 1:53 PM, Adam Berkan <aberkan-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> wrote:
> If you completely lose a flash device that contains significant
> quantities of dirty data, you'll have a hard time recovering any data
> from the backing device.  All of the writeback testing we've done so
> far has assumed device failures are limited to small regions of the
> device, and bcache tries to recover what it can when it encounters
> such errors.  It doesn't do anything reasonable in the case of full
> device loss.
>
> I would not recommend using bcache in writeback mode anywhere where
> you can't afford to lose the whole device.
>
> Adam
>
> On Fri, Apr 20, 2012 at 8:51 AM, Brad Campbell <brad-+nnirC7rrGZibQn6LdNjmg@public.gmane.org> wrote:
>>>
>> On 17/04/2012, at 4:30 PM, Kent Overstreet <koverstreet-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> wrote:
>>
>>>
>>>
>>>> Also ... do you think that your code is production ready when using bcache
>>>> to do writeback caching ? Of course I will keep testing but I'd like to
>>>> know if you think the code by now is production ready.
>>>
>>> Yeah, it is. Test it on your configuration, etc. etc. but writeback is
>>> pretty mature and well tested at this point.
>>>
>> I've been giving serious thought to using this in a production environment. It performs well in my staging system, but my biggest concern revolves around the ultimate reliability of the ssd.
>>
>> How much testing have you performed with regard to cache device failures in a writeback scenario? I like the idea of mirroring the cache devices to replicate dirty data, however I know that is not implemented yet.
>>
>> We run a raid10 of SAS cheetahs. I'd love to mount a cache in there, but ultimately we have little information about what happens if the ssd starts to flake out. I guess more than that, there is little real information out there detailing common or potential ssd failure modes.
>>
>> I'd assume you've performed plenty of testing over the development of bcache. Can you fill us in a bit as to what to expect when things go south?
>>
>> Regards,
>> Brad--
>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: code stability (production readiness) and kernel versions

[Adam Berkan]

Raid 1 is going to slow down the writes, but it's probably still a big
win over write-through mode.  Of course raid 1 isn't invincible, and
you can still get silent data corruption on the devices.

I'm sure there's still some nasty bugs somewhere in bcache writeback
mode that we haven't found yet.  They should be rare, and you might
not encounter them, but I'd be leary of putting anything too valuable
on writeback devices.  If things go wrong, recovery can be extremely
difficult.

Caveat emptor, but if you do try it please let everyone know how it
works for you.

Adam

On Fri, Apr 20, 2012 at 10:05 AM, Roberto Alcântara
<roberto-ek926an30Vg66dlzjLgC1g@public.gmane.org> wrote:
> Adam,
>
> What about use bache in raid1 flash devices?
>
>         - Roberto
>
>
> On Fri, Apr 20, 2012 at 1:53 PM, Adam Berkan <aberkan-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> wrote:
>> If you completely lose a flash device that contains significant
>> quantities of dirty data, you'll have a hard time recovering any data
>> from the backing device.  All of the writeback testing we've done so
>> far has assumed device failures are limited to small regions of the
>> device, and bcache tries to recover what it can when it encounters
>> such errors.  It doesn't do anything reasonable in the case of full
>> device loss.
>>
>> I would not recommend using bcache in writeback mode anywhere where
>> you can't afford to lose the whole device.
>>
>> Adam
>>
>> On Fri, Apr 20, 2012 at 8:51 AM, Brad Campbell <brad-+nnirC7rrGZibQn6LdNjmg@public.gmane.org> wrote:
>>>>
>>> On 17/04/2012, at 4:30 PM, Kent Overstreet <koverstreet-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> wrote:
>>>
>>>>
>>>>
>>>>> Also ... do you think that your code is production ready when using bcache
>>>>> to do writeback caching ? Of course I will keep testing but I'd like to
>>>>> know if you think the code by now is production ready.
>>>>
>>>> Yeah, it is. Test it on your configuration, etc. etc. but writeback is
>>>> pretty mature and well tested at this point.
>>>>
>>> I've been giving serious thought to using this in a production environment. It performs well in my staging system, but my biggest concern revolves around the ultimate reliability of the ssd.
>>>
>>> How much testing have you performed with regard to cache device failures in a writeback scenario? I like the idea of mirroring the cache devices to replicate dirty data, however I know that is not implemented yet.
>>>
>>> We run a raid10 of SAS cheetahs. I'd love to mount a cache in there, but ultimately we have little information about what happens if the ssd starts to flake out. I guess more than that, there is little real information out there detailing common or potential ssd failure modes.
>>>
>>> I'd assume you've performed plenty of testing over the development of bcache. Can you fill us in a bit as to what to expect when things go south?
>>>
>>> Regards,
>>> Brad--
>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: code stability (production readiness) and kernel versions

[Kent Overstreet]

Thanks for publishing these, very much appreciated :)

If it's not too much to ask, do you think you could rerun with the
latest code? There've been some performance improvements since the
version you tested, and in particular I think fix to make inserting
cache misses reliable might make a big difference on the apache
benchmark - I suspect that was a pathalogical case for the old code.

Also if you'd be willing to do more benchmarking, I'll definitely be
looking at your data for performance bugs and things that could be
improved. I expect there's performance bugs still - just have to
identify the workloads that tickle them.

On Wed, Apr 18, 2012 at 7:15 AM, Alexandru Ionica
<alexandru.ionica-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hello,
>
> My employer allowed me to publish the data, so here you go:
> http://www.accelcloud.com/2012/04/18/linux-flashcache-and-bcache-performance-testing/
>
> :)
>
>
> On Tue, Apr 17, 2012 at 10:30 AM, Kent Overstreet <koverstreet@google.com>
> wrote:
>>
>> Hey, sorry for the delay. Was travelling and I've been slow to catch
>> up on email...
>>
>> On Fri, Apr 6, 2012 at 6:21 AM, Alexandru Ionica
>> <alexandru.ionica-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> > Hello,
>> >
>> > I have been doing several benchmarks (Phoronix test suite, disk suite)
>> > and
>> > I'm really impressed with the performance when doing writeback caching.
>> > For
>> > the benchmarks I did a git checkout and compiled the 3.1.0+ kernel. As
>> > the
>> > wiki is outdated I am wondering if the patches have also been applied on
>> > a
>> > kernel version which is used by server oriented distributions, meaning
>> > kernel version 2.6.32 for example or if there is a way to apply the
>> > patches
>> > (if they exist separately) to kernel version 2.6.32 .
>> > I am interested in this specific version of the kernel as other
>> > constrains
>> > impose it.
>>
>> Any chance you could share those benchmarks? I'll post them on the
>> wiki (or give you an account). I could really use some benchmarks that
>> are suitable for sharing, all the benchmarking I've done has been just
>> focused on optimizing stuff.
>>
>> > Also ... do you think that your code is production ready when using
>> > bcache
>> > to do writeback caching ? Of course I will keep testing but I'd like to
>> > know if you think the code by now is production ready.
>>
>> Yeah, it is. Test it on your configuration, etc. etc. but writeback is
>> pretty mature and well tested at this point.
>>
>> > Basically I plan to run a setup like: bcache device assembled from
>> > software
>> > raid10 or raid0 (4 disk) + ssd ; on top of this a volume group ; on top
>> > of
>> > logical devices drbd setup . We are running this for a long while
>> > without
>> > bcache so the setup is stable and the new part here would be bcache.
>>
>> Sounds pretty reasonable.
>>
>> > P.S. during the benchmarks bcache outperformed in every way flashcache
>> > (I
>> > tried two different sequential size settings with flash cache, both
>> > underperformed)
>>
>> Cool! Would love to see the numbers :)
>>
>> > Regards,
>> >
>> > Alexandru Ionica
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe linux-bcache"
>> > in
>> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>