Best option for SSD caching on a md raid array?

Best option for SSD caching on a md raid array?

[Thomas Fjellstrom]

Hi,

I'm planning on setting up a dual raid5 array for hosting vm's off LVM, I would 
like to add an ssd cache to help with typical server system loads, and some 
mixed read/write loads for game servers.

I'm wondering what the "best" route for me to go is. There's a few options 
that I know of, like flashcache, dm-cache, and bcache. The question is which of 
those is better suited to my use case?

Also wondering which ones people here use, and what they use it for.

Thanks!

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: Best option for SSD caching on a md raid array?

[David Brown]

Hi,

If the server is going to have reasonably long uptimes, then lots of ram
could easily be a better choice than an SSD caching system.  I don't
know how you are planning your "dual raid5 array", nor what sort of VM's
you are dealing with.  But the key point of an SSD cache is to get fast
access to common data with random access patterns, since a raid array
will give you plenty of bandwidth for large serial accesses.  And while
an SSD is fast for random reads, having the data in the server's cache
is even faster.


On 08/02/16 06:14, Thomas Fjellstrom wrote:
> Hi,
> 
> I'm planning on setting up a dual raid5 array for hosting vm's off LVM, I would 
> like to add an ssd cache to help with typical server system loads, and some 
> mixed read/write loads for game servers.
> 
> I'm wondering what the "best" route for me to go is. There's a few options 
> that I know of, like flashcache, dm-cache, and bcache. The question is which of 
> those is better suited to my use case?
> 
> Also wondering which ones people here use, and what they use it for.
> 
> Thanks!
> 

Re: Best option for SSD caching on a md raid array?

[Jens-U. Mozdzen]

Hi *,

Zitat von David Brown <david.brown@hesbynett.no>:
> Hi,
>
> If the server is going to have reasonably long uptimes, then lots of ram
> could easily be a better choice than an SSD caching system.
> [...] But the key point of an SSD cache is to get fast
> access to common data with random access patterns, since a raid array
> will give you plenty of bandwidth for large serial accesses.  And while
> an SSD is fast for random reads, having the data in the server's cache
> is even faster.

to me, the key point of SSD caching was fast (persistent) writes and  
optimization of expensive partial writes for RAID. Using memory  
caching helped, but keeping that much dirty data in RAM wasn't my cup  
of tea (it's what I had before moving to bcache).

Thomas, I decided to go the bcache route and am running two SAN/NAS  
servers for months, serving a mix of NFS (i.e. for home directories  
and large software compilation scenarios across multiple servers),  
Samba (for a few SMB-dependent machines), iSCSI and Fibre Channel  
(both via SCST) to serve various virtual disks to about 40 VMs, and a  
vanishing amount of other services.

I have not further explored the other options you name - but up to  
now, I've been sufficiently happy with bcache to see no need to run  
intensive compares.

Mateusz named pros and cons - for bcache, the lack of maintainer got  
somewhat resolved by others jumping in and i.e. SUSE putting an  
up-to-date (patched) bcache version in their kernels. What I feel  
missing in Mateusz' list is bcache's current lack of a live resizing  
feature: Even if you live-resize the back-end storage (i.e. RAID),  
you'll currently have to re-init the bcache device (i.e. by rebooting  
the node) for it to recognize the changed back-end size. Not good for  
your uptime.

Regards,
Jens

Re: Best option for SSD caching on a md raid array?

[Thomas Fjellstrom]

On Mon Feb 8 2016 09:36:32 AM you wrote:
> Hi,
> 
> If the server is going to have reasonably long uptimes, then lots of ram
> could easily be a better choice than an SSD caching system.  I don't
> know how you are planning your "dual raid5 array", nor what sort of VM's
> you are dealing with.  But the key point of an SSD cache is to get fast
> access to common data with random access patterns, since a raid array
> will give you plenty of bandwidth for large serial accesses.  And while
> an SSD is fast for random reads, having the data in the server's cache
> is even faster.

Basically I have two servers, one is an older single socket xeon e3 with 16GB 
ram, that tops out at 32GB IIRC (if i can find reasonably priced non buffered 
8GB ddr3 ecc DIMMs). And my other box is a dual socket 32 core (2x16) Opteron 
box with about 120GB ram.

I'm planning on having each machine have a storage array that are synced using 
DRBD or something similar. The idea is to allow online migration with little 
fuss, not having to access storage over the network, and have what is 
essentially a raid 1 on top of the raid5 on each machine.

The Xeon box has a fair bit less resources (except per core perf is probably 
better), and may have issues caching things in ram.

The main use case of both machines is to host various services, like a couple 
game servers, plex, gitlab, unifi controller, testing/staging instances for 
various services for work. Also a couple windows vms.

So its a mix of things.

> On 08/02/16 06:14, Thomas Fjellstrom wrote:
> > Hi,
> > 
> > I'm planning on setting up a dual raid5 array for hosting vm's off LVM, I
> > would like to add an ssd cache to help with typical server system loads,
> > and some mixed read/write loads for game servers.
> > 
> > I'm wondering what the "best" route for me to go is. There's a few options
> > that I know of, like flashcache, dm-cache, and bcache. The question is
> > which of those is better suited to my use case?
> > 
> > Also wondering which ones people here use, and what they use it for.
> > 
> > Thanks!
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best option for SSD caching on a md raid array?

[Mateusz Korniak]

On Sunday 07 February 2016 22:14:08 Thomas Fjellstrom wrote:
> I'm wondering what the "best" route for me to go is. There's a few options
> that I know of, like flashcache, dm-cache, and bcache. The question is which
> of those is better suited to my use case?

bcache cons:
- no maintainer, bugs living years
- partition have to be formatted as bcache

lvmcache cons:
- in case of failure of caching device, whole cache volume stops working (data 
is not lost though).
- cumbersome way of removing cache from LV when caching device is not 
accessible/failed.

lvmcache pros:
- One can turn on/off caching for active/mounted LV.

We moved from bcache to lvmcache (knowing it's limitations) using write-
through mode and for more-reads-than-writes DB access patterns [1].


[1]: Examble postgresql load (sdc is caching SSD, over sda/b RAID10).
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz 
avgqu-sz   await r_await w_await  svctm  %util
sda               0.00    33.93    0.07  179.13     0.00     3.38    38.59     
6.23   34.75   23.50   34.76   2.63  47.07
sdb               0.00    35.97    0.03  177.17     0.00     3.38    39.04     
9.57   53.99  337.00   53.94   2.97  52.66
sdc               8.03     2.63  631.07  201.13     6.04     3.35    23.10     
0.19    0.23    0.19    0.35   0.15  12.34



Regards,
-- 
Mateusz Korniak
"(...) mam brata - poważny, domator, liczykrupa, hipokryta, pobożniś,
 	krótko mówiąc - podpora społeczeństwa."
				Nikos Kazantzakis - "Grek Zorba"

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best option for SSD caching on a md raid array?

[John Stoffel]

Thomas> I'm planning on setting up a dual raid5 array for hosting vm's
Thomas> off LVM, I would like to add an ssd cache to help with typical
Thomas> server system loads, and some mixed read/write loads for game
Thomas> servers.

Thomas> I'm wondering what the "best" route for me to go is. There's a
Thomas> few options that I know of, like flashcache, dm-cache, and
Thomas> bcache. The question is which of those is better suited to my
Thomas> use case?

I looked into both bcache and lvmcache and went with lvmcache because
it made more sense to me.  But one restriction is that the cache LVs
must be in the same VGs as the volume to cache.  So you might need to
partition your SSD(s) and put the partitions into seperate VGs so that
you can cache one or more LVs.

I'm using it at home to cache a RAID1 4tb VG, with a pair of mirrored
SSDs.  I'm a big believer in mirroring, esp for critical stuff like
this.

Unfortunately, I don't have a good performance measurements of how
much help this actually gives me.  It *feels* faster, but since I
don't have before/after measurements, I could just have wasted a bunch
of money.

So I'd recommend lvmcache overall, since it does seem to help, but
push for better performance metrics and measurement.  Heh, maybe it's
time for me to start writing up lvmcachetop to help measure things.

John

Re: Best option for SSD caching on a md raid array?

[Pasi Kärkkäinen]

On Mon, Feb 08, 2016 at 10:34:09AM -0500, John Stoffel wrote:
> 
> Thomas> I'm planning on setting up a dual raid5 array for hosting vm's
> Thomas> off LVM, I would like to add an ssd cache to help with typical
> Thomas> server system loads, and some mixed read/write loads for game
> Thomas> servers.
> 
> Thomas> I'm wondering what the "best" route for me to go is. There's a
> Thomas> few options that I know of, like flashcache, dm-cache, and
> Thomas> bcache. The question is which of those is better suited to my
> Thomas> use case?
> 
> I looked into both bcache and lvmcache and went with lvmcache because
> it made more sense to me.  But one restriction is that the cache LVs
> must be in the same VGs as the volume to cache.  So you might need to
> partition your SSD(s) and put the partitions into seperate VGs so that
> you can cache one or more LVs.
> 
> I'm using it at home to cache a RAID1 4tb VG, with a pair of mirrored
> SSDs.  I'm a big believer in mirroring, esp for critical stuff like
> this.
> 
> Unfortunately, I don't have a good performance measurements of how
> much help this actually gives me.  It *feels* faster, but since I
> don't have before/after measurements, I could just have wasted a bunch
> of money.
> 
> So I'd recommend lvmcache overall, since it does seem to help, but
> push for better performance metrics and measurement.  Heh, maybe it's
> time for me to start writing up lvmcachetop to help measure things.
> 

The other day I was benchmarking lvmcache (on CentOS 7.2),
but I couldn't get it to give much advantage for writeback caching use case.. 
even when I was using fast nvme/pcie cache device.

It just wouldn't cache writes properly for me. I tried both the smq and mq, 
and played with all of the parameters.. it just wouldn't help much for my use case.

YMMV.

> John


-- Pasi

Re: Best option for SSD caching on a md raid array?

[John Stoffel]

>>>>> "Pasi" == Pasi Kärkkäinen <pasik@iki.fi> writes:

Pasi> On Mon, Feb 08, 2016 at 10:34:09AM -0500, John Stoffel wrote:
>> 
Thomas> I'm planning on setting up a dual raid5 array for hosting vm's
Thomas> off LVM, I would like to add an ssd cache to help with typical
Thomas> server system loads, and some mixed read/write loads for game
Thomas> servers.
>> 
Thomas> I'm wondering what the "best" route for me to go is. There's a
Thomas> few options that I know of, like flashcache, dm-cache, and
Thomas> bcache. The question is which of those is better suited to my
Thomas> use case?
>> 
>> I looked into both bcache and lvmcache and went with lvmcache because
>> it made more sense to me.  But one restriction is that the cache LVs
>> must be in the same VGs as the volume to cache.  So you might need to
>> partition your SSD(s) and put the partitions into seperate VGs so that
>> you can cache one or more LVs.
>> 
>> I'm using it at home to cache a RAID1 4tb VG, with a pair of mirrored
>> SSDs.  I'm a big believer in mirroring, esp for critical stuff like
>> this.
>> 
>> Unfortunately, I don't have a good performance measurements of how
>> much help this actually gives me.  It *feels* faster, but since I
>> don't have before/after measurements, I could just have wasted a bunch
>> of money.
>> 
>> So I'd recommend lvmcache overall, since it does seem to help, but
>> push for better performance metrics and measurement.  Heh, maybe it's
>> time for me to start writing up lvmcachetop to help measure things.

Pasi> The other day I was benchmarking lvmcache (on CentOS 7.2), but I
Pasi> couldn't get it to give much advantage for writeback caching use
Pasi> case..  even when I was using fast nvme/pcie cache device.

What were you using?  I should spin up fio and see what I can
find... or maybe even just some kernel compiles, image manipulation,
etc.

Let's see what we can come up with for benchmarking, since it would be
a big help!  I'm running kernel 4.4-rc7 right now on a Debian Jessie system.

Pasi> It just wouldn't cache writes properly for me. I tried both the
Pasi> smq and mq, and played with all of the parameters.. it just
Pasi> wouldn't help much for my use case.

I have both mq and smq compiled into the kernel, so I can't figure out
how to change between them to test.

John
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Best option for SSD caching on a md raid array?

[Pasi Kärkkäinen]

On Wed, Feb 10, 2016 at 05:21:51PM -0500, John Stoffel wrote:
> >>>>> "Pasi" == Pasi Kärkkäinen <pasik@iki.fi> writes:
> 
> Pasi> On Mon, Feb 08, 2016 at 10:34:09AM -0500, John Stoffel wrote:
> >> 
> Thomas> I'm planning on setting up a dual raid5 array for hosting vm's
> Thomas> off LVM, I would like to add an ssd cache to help with typical
> Thomas> server system loads, and some mixed read/write loads for game
> Thomas> servers.
> >> 
> Thomas> I'm wondering what the "best" route for me to go is. There's a
> Thomas> few options that I know of, like flashcache, dm-cache, and
> Thomas> bcache. The question is which of those is better suited to my
> Thomas> use case?
> >> 
> >> I looked into both bcache and lvmcache and went with lvmcache because
> >> it made more sense to me.  But one restriction is that the cache LVs
> >> must be in the same VGs as the volume to cache.  So you might need to
> >> partition your SSD(s) and put the partitions into seperate VGs so that
> >> you can cache one or more LVs.
> >> 
> >> I'm using it at home to cache a RAID1 4tb VG, with a pair of mirrored
> >> SSDs.  I'm a big believer in mirroring, esp for critical stuff like
> >> this.
> >> 
> >> Unfortunately, I don't have a good performance measurements of how
> >> much help this actually gives me.  It *feels* faster, but since I
> >> don't have before/after measurements, I could just have wasted a bunch
> >> of money.
> >> 
> >> So I'd recommend lvmcache overall, since it does seem to help, but
> >> push for better performance metrics and measurement.  Heh, maybe it's
> >> time for me to start writing up lvmcachetop to help measure things.
> 
> Pasi> The other day I was benchmarking lvmcache (on CentOS 7.2), but I
> Pasi> couldn't get it to give much advantage for writeback caching use
> Pasi> case..  even when I was using fast nvme/pcie cache device.
> 
> What were you using?  I should spin up fio and see what I can
> find... or maybe even just some kernel compiles, image manipulation,
> etc.
>

Yeah I was benchmarking small random 4 kB writes, using fio. 
iodepth=64 and jobs=12.


> Let's see what we can come up with for benchmarking, since it would be
> a big help!  I'm running kernel 4.4-rc7 right now on a Debian Jessie system.
> 
> Pasi> It just wouldn't cache writes properly for me. I tried both the
> Pasi> smq and mq, and played with all of the parameters.. it just
> Pasi> wouldn't help much for my use case.
> 
> I have both mq and smq compiled into the kernel, so I can't figure out
> how to change between them to test.
>

Hmm, yeah, I think it was documented on the lvmcache man page. 
I can't check my notes right now..

 
> John


-- Pasi

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html