* Best option for SSD caching on a md raid array?
@ 2016-02-08 5:14 Thomas Fjellstrom
2016-02-08 8:36 ` David Brown
` (2 more replies)
0 siblings, 3 replies; 9+ messages in thread
From: Thomas Fjellstrom @ 2016-02-08 5:14 UTC (permalink / raw)
To: linux-raid
Hi,
I'm planning on setting up a dual raid5 array for hosting vm's off LVM, I would
like to add an ssd cache to help with typical server system loads, and some
mixed read/write loads for game servers.
I'm wondering what the "best" route for me to go is. There's a few options
that I know of, like flashcache, dm-cache, and bcache. The question is which of
those is better suited to my use case?
Also wondering which ones people here use, and what they use it for.
Thanks!
--
Thomas Fjellstrom
thomas@fjellstrom.ca
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Best option for SSD caching on a md raid array?
2016-02-08 5:14 Best option for SSD caching on a md raid array? Thomas Fjellstrom
@ 2016-02-08 8:36 ` David Brown
2016-02-08 15:45 ` Jens-U. Mozdzen
2016-02-08 20:21 ` Thomas Fjellstrom
2016-02-08 9:38 ` Mateusz Korniak
2016-02-08 15:34 ` John Stoffel
2 siblings, 2 replies; 9+ messages in thread
From: David Brown @ 2016-02-08 8:36 UTC (permalink / raw)
To: Thomas Fjellstrom, linux-raid
Hi,
If the server is going to have reasonably long uptimes, then lots of ram
could easily be a better choice than an SSD caching system. I don't
know how you are planning your "dual raid5 array", nor what sort of VM's
you are dealing with. But the key point of an SSD cache is to get fast
access to common data with random access patterns, since a raid array
will give you plenty of bandwidth for large serial accesses. And while
an SSD is fast for random reads, having the data in the server's cache
is even faster.
On 08/02/16 06:14, Thomas Fjellstrom wrote:
> Hi,
>
> I'm planning on setting up a dual raid5 array for hosting vm's off LVM, I would
> like to add an ssd cache to help with typical server system loads, and some
> mixed read/write loads for game servers.
>
> I'm wondering what the "best" route for me to go is. There's a few options
> that I know of, like flashcache, dm-cache, and bcache. The question is which of
> those is better suited to my use case?
>
> Also wondering which ones people here use, and what they use it for.
>
> Thanks!
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Best option for SSD caching on a md raid array?
2016-02-08 5:14 Best option for SSD caching on a md raid array? Thomas Fjellstrom
2016-02-08 8:36 ` David Brown
@ 2016-02-08 9:38 ` Mateusz Korniak
2016-02-08 15:34 ` John Stoffel
2 siblings, 0 replies; 9+ messages in thread
From: Mateusz Korniak @ 2016-02-08 9:38 UTC (permalink / raw)
To: Thomas Fjellstrom; +Cc: linux-raid
On Sunday 07 February 2016 22:14:08 Thomas Fjellstrom wrote:
> I'm wondering what the "best" route for me to go is. There's a few options
> that I know of, like flashcache, dm-cache, and bcache. The question is which
> of those is better suited to my use case?
bcache cons:
- no maintainer, bugs living years
- partition have to be formatted as bcache
lvmcache cons:
- in case of failure of caching device, whole cache volume stops working (data
is not lost though).
- cumbersome way of removing cache from LV when caching device is not
accessible/failed.
lvmcache pros:
- One can turn on/off caching for active/mounted LV.
We moved from bcache to lvmcache (knowing it's limitations) using write-
through mode and for more-reads-than-writes DB access patterns [1].
[1]: Examble postgresql load (sdc is caching SSD, over sda/b RAID10).
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
sda 0.00 33.93 0.07 179.13 0.00 3.38 38.59
6.23 34.75 23.50 34.76 2.63 47.07
sdb 0.00 35.97 0.03 177.17 0.00 3.38 39.04
9.57 53.99 337.00 53.94 2.97 52.66
sdc 8.03 2.63 631.07 201.13 6.04 3.35 23.10
0.19 0.23 0.19 0.35 0.15 12.34
Regards,
--
Mateusz Korniak
"(...) mam brata - poważny, domator, liczykrupa, hipokryta, pobożniś,
krótko mówiąc - podpora społeczeństwa."
Nikos Kazantzakis - "Grek Zorba"
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Best option for SSD caching on a md raid array?
2016-02-08 5:14 Best option for SSD caching on a md raid array? Thomas Fjellstrom
2016-02-08 8:36 ` David Brown
2016-02-08 9:38 ` Mateusz Korniak
@ 2016-02-08 15:34 ` John Stoffel
2016-02-10 22:13 ` Pasi Kärkkäinen
2 siblings, 1 reply; 9+ messages in thread
From: John Stoffel @ 2016-02-08 15:34 UTC (permalink / raw)
To: Thomas Fjellstrom; +Cc: linux-raid
Thomas> I'm planning on setting up a dual raid5 array for hosting vm's
Thomas> off LVM, I would like to add an ssd cache to help with typical
Thomas> server system loads, and some mixed read/write loads for game
Thomas> servers.
Thomas> I'm wondering what the "best" route for me to go is. There's a
Thomas> few options that I know of, like flashcache, dm-cache, and
Thomas> bcache. The question is which of those is better suited to my
Thomas> use case?
I looked into both bcache and lvmcache and went with lvmcache because
it made more sense to me. But one restriction is that the cache LVs
must be in the same VGs as the volume to cache. So you might need to
partition your SSD(s) and put the partitions into seperate VGs so that
you can cache one or more LVs.
I'm using it at home to cache a RAID1 4tb VG, with a pair of mirrored
SSDs. I'm a big believer in mirroring, esp for critical stuff like
this.
Unfortunately, I don't have a good performance measurements of how
much help this actually gives me. It *feels* faster, but since I
don't have before/after measurements, I could just have wasted a bunch
of money.
So I'd recommend lvmcache overall, since it does seem to help, but
push for better performance metrics and measurement. Heh, maybe it's
time for me to start writing up lvmcachetop to help measure things.
John
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Best option for SSD caching on a md raid array?
2016-02-08 8:36 ` David Brown
@ 2016-02-08 15:45 ` Jens-U. Mozdzen
2016-02-08 20:21 ` Thomas Fjellstrom
1 sibling, 0 replies; 9+ messages in thread
From: Jens-U. Mozdzen @ 2016-02-08 15:45 UTC (permalink / raw)
To: David Brown; +Cc: Thomas Fjellstrom, linux-raid
Hi *,
Zitat von David Brown <david.brown@hesbynett.no>:
> Hi,
>
> If the server is going to have reasonably long uptimes, then lots of ram
> could easily be a better choice than an SSD caching system.
> [...] But the key point of an SSD cache is to get fast
> access to common data with random access patterns, since a raid array
> will give you plenty of bandwidth for large serial accesses. And while
> an SSD is fast for random reads, having the data in the server's cache
> is even faster.
to me, the key point of SSD caching was fast (persistent) writes and
optimization of expensive partial writes for RAID. Using memory
caching helped, but keeping that much dirty data in RAM wasn't my cup
of tea (it's what I had before moving to bcache).
Thomas, I decided to go the bcache route and am running two SAN/NAS
servers for months, serving a mix of NFS (i.e. for home directories
and large software compilation scenarios across multiple servers),
Samba (for a few SMB-dependent machines), iSCSI and Fibre Channel
(both via SCST) to serve various virtual disks to about 40 VMs, and a
vanishing amount of other services.
I have not further explored the other options you name - but up to
now, I've been sufficiently happy with bcache to see no need to run
intensive compares.
Mateusz named pros and cons - for bcache, the lack of maintainer got
somewhat resolved by others jumping in and i.e. SUSE putting an
up-to-date (patched) bcache version in their kernels. What I feel
missing in Mateusz' list is bcache's current lack of a live resizing
feature: Even if you live-resize the back-end storage (i.e. RAID),
you'll currently have to re-init the bcache device (i.e. by rebooting
the node) for it to recognize the changed back-end size. Not good for
your uptime.
Regards,
Jens
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Best option for SSD caching on a md raid array?
2016-02-08 8:36 ` David Brown
2016-02-08 15:45 ` Jens-U. Mozdzen
@ 2016-02-08 20:21 ` Thomas Fjellstrom
1 sibling, 0 replies; 9+ messages in thread
From: Thomas Fjellstrom @ 2016-02-08 20:21 UTC (permalink / raw)
To: David Brown; +Cc: linux-raid
On Mon Feb 8 2016 09:36:32 AM you wrote:
> Hi,
>
> If the server is going to have reasonably long uptimes, then lots of ram
> could easily be a better choice than an SSD caching system. I don't
> know how you are planning your "dual raid5 array", nor what sort of VM's
> you are dealing with. But the key point of an SSD cache is to get fast
> access to common data with random access patterns, since a raid array
> will give you plenty of bandwidth for large serial accesses. And while
> an SSD is fast for random reads, having the data in the server's cache
> is even faster.
Basically I have two servers, one is an older single socket xeon e3 with 16GB
ram, that tops out at 32GB IIRC (if i can find reasonably priced non buffered
8GB ddr3 ecc DIMMs). And my other box is a dual socket 32 core (2x16) Opteron
box with about 120GB ram.
I'm planning on having each machine have a storage array that are synced using
DRBD or something similar. The idea is to allow online migration with little
fuss, not having to access storage over the network, and have what is
essentially a raid 1 on top of the raid5 on each machine.
The Xeon box has a fair bit less resources (except per core perf is probably
better), and may have issues caching things in ram.
The main use case of both machines is to host various services, like a couple
game servers, plex, gitlab, unifi controller, testing/staging instances for
various services for work. Also a couple windows vms.
So its a mix of things.
> On 08/02/16 06:14, Thomas Fjellstrom wrote:
> > Hi,
> >
> > I'm planning on setting up a dual raid5 array for hosting vm's off LVM, I
> > would like to add an ssd cache to help with typical server system loads,
> > and some mixed read/write loads for game servers.
> >
> > I'm wondering what the "best" route for me to go is. There's a few options
> > that I know of, like flashcache, dm-cache, and bcache. The question is
> > which of those is better suited to my use case?
> >
> > Also wondering which ones people here use, and what they use it for.
> >
> > Thanks!
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Best option for SSD caching on a md raid array?
2016-02-08 15:34 ` John Stoffel
@ 2016-02-10 22:13 ` Pasi Kärkkäinen
2016-02-10 22:21 ` John Stoffel
0 siblings, 1 reply; 9+ messages in thread
From: Pasi Kärkkäinen @ 2016-02-10 22:13 UTC (permalink / raw)
To: John Stoffel; +Cc: Thomas Fjellstrom, linux-raid
On Mon, Feb 08, 2016 at 10:34:09AM -0500, John Stoffel wrote:
>
> Thomas> I'm planning on setting up a dual raid5 array for hosting vm's
> Thomas> off LVM, I would like to add an ssd cache to help with typical
> Thomas> server system loads, and some mixed read/write loads for game
> Thomas> servers.
>
> Thomas> I'm wondering what the "best" route for me to go is. There's a
> Thomas> few options that I know of, like flashcache, dm-cache, and
> Thomas> bcache. The question is which of those is better suited to my
> Thomas> use case?
>
> I looked into both bcache and lvmcache and went with lvmcache because
> it made more sense to me. But one restriction is that the cache LVs
> must be in the same VGs as the volume to cache. So you might need to
> partition your SSD(s) and put the partitions into seperate VGs so that
> you can cache one or more LVs.
>
> I'm using it at home to cache a RAID1 4tb VG, with a pair of mirrored
> SSDs. I'm a big believer in mirroring, esp for critical stuff like
> this.
>
> Unfortunately, I don't have a good performance measurements of how
> much help this actually gives me. It *feels* faster, but since I
> don't have before/after measurements, I could just have wasted a bunch
> of money.
>
> So I'd recommend lvmcache overall, since it does seem to help, but
> push for better performance metrics and measurement. Heh, maybe it's
> time for me to start writing up lvmcachetop to help measure things.
>
The other day I was benchmarking lvmcache (on CentOS 7.2),
but I couldn't get it to give much advantage for writeback caching use case..
even when I was using fast nvme/pcie cache device.
It just wouldn't cache writes properly for me. I tried both the smq and mq,
and played with all of the parameters.. it just wouldn't help much for my use case.
YMMV.
> John
-- Pasi
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Best option for SSD caching on a md raid array?
2016-02-10 22:13 ` Pasi Kärkkäinen
@ 2016-02-10 22:21 ` John Stoffel
2016-02-10 22:54 ` Pasi Kärkkäinen
0 siblings, 1 reply; 9+ messages in thread
From: John Stoffel @ 2016-02-10 22:21 UTC (permalink / raw)
To: Pasi Kärkkäinen; +Cc: John Stoffel, Thomas Fjellstrom, linux-raid
>>>>> "Pasi" == Pasi Kärkkäinen <pasik@iki.fi> writes:
Pasi> On Mon, Feb 08, 2016 at 10:34:09AM -0500, John Stoffel wrote:
>>
Thomas> I'm planning on setting up a dual raid5 array for hosting vm's
Thomas> off LVM, I would like to add an ssd cache to help with typical
Thomas> server system loads, and some mixed read/write loads for game
Thomas> servers.
>>
Thomas> I'm wondering what the "best" route for me to go is. There's a
Thomas> few options that I know of, like flashcache, dm-cache, and
Thomas> bcache. The question is which of those is better suited to my
Thomas> use case?
>>
>> I looked into both bcache and lvmcache and went with lvmcache because
>> it made more sense to me. But one restriction is that the cache LVs
>> must be in the same VGs as the volume to cache. So you might need to
>> partition your SSD(s) and put the partitions into seperate VGs so that
>> you can cache one or more LVs.
>>
>> I'm using it at home to cache a RAID1 4tb VG, with a pair of mirrored
>> SSDs. I'm a big believer in mirroring, esp for critical stuff like
>> this.
>>
>> Unfortunately, I don't have a good performance measurements of how
>> much help this actually gives me. It *feels* faster, but since I
>> don't have before/after measurements, I could just have wasted a bunch
>> of money.
>>
>> So I'd recommend lvmcache overall, since it does seem to help, but
>> push for better performance metrics and measurement. Heh, maybe it's
>> time for me to start writing up lvmcachetop to help measure things.
Pasi> The other day I was benchmarking lvmcache (on CentOS 7.2), but I
Pasi> couldn't get it to give much advantage for writeback caching use
Pasi> case.. even when I was using fast nvme/pcie cache device.
What were you using? I should spin up fio and see what I can
find... or maybe even just some kernel compiles, image manipulation,
etc.
Let's see what we can come up with for benchmarking, since it would be
a big help! I'm running kernel 4.4-rc7 right now on a Debian Jessie system.
Pasi> It just wouldn't cache writes properly for me. I tried both the
Pasi> smq and mq, and played with all of the parameters.. it just
Pasi> wouldn't help much for my use case.
I have both mq and smq compiled into the kernel, so I can't figure out
how to change between them to test.
John
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Best option for SSD caching on a md raid array?
2016-02-10 22:21 ` John Stoffel
@ 2016-02-10 22:54 ` Pasi Kärkkäinen
0 siblings, 0 replies; 9+ messages in thread
From: Pasi Kärkkäinen @ 2016-02-10 22:54 UTC (permalink / raw)
To: John Stoffel; +Cc: Thomas Fjellstrom, linux-raid
On Wed, Feb 10, 2016 at 05:21:51PM -0500, John Stoffel wrote:
> >>>>> "Pasi" == Pasi Kärkkäinen <pasik@iki.fi> writes:
>
> Pasi> On Mon, Feb 08, 2016 at 10:34:09AM -0500, John Stoffel wrote:
> >>
> Thomas> I'm planning on setting up a dual raid5 array for hosting vm's
> Thomas> off LVM, I would like to add an ssd cache to help with typical
> Thomas> server system loads, and some mixed read/write loads for game
> Thomas> servers.
> >>
> Thomas> I'm wondering what the "best" route for me to go is. There's a
> Thomas> few options that I know of, like flashcache, dm-cache, and
> Thomas> bcache. The question is which of those is better suited to my
> Thomas> use case?
> >>
> >> I looked into both bcache and lvmcache and went with lvmcache because
> >> it made more sense to me. But one restriction is that the cache LVs
> >> must be in the same VGs as the volume to cache. So you might need to
> >> partition your SSD(s) and put the partitions into seperate VGs so that
> >> you can cache one or more LVs.
> >>
> >> I'm using it at home to cache a RAID1 4tb VG, with a pair of mirrored
> >> SSDs. I'm a big believer in mirroring, esp for critical stuff like
> >> this.
> >>
> >> Unfortunately, I don't have a good performance measurements of how
> >> much help this actually gives me. It *feels* faster, but since I
> >> don't have before/after measurements, I could just have wasted a bunch
> >> of money.
> >>
> >> So I'd recommend lvmcache overall, since it does seem to help, but
> >> push for better performance metrics and measurement. Heh, maybe it's
> >> time for me to start writing up lvmcachetop to help measure things.
>
> Pasi> The other day I was benchmarking lvmcache (on CentOS 7.2), but I
> Pasi> couldn't get it to give much advantage for writeback caching use
> Pasi> case.. even when I was using fast nvme/pcie cache device.
>
> What were you using? I should spin up fio and see what I can
> find... or maybe even just some kernel compiles, image manipulation,
> etc.
>
Yeah I was benchmarking small random 4 kB writes, using fio.
iodepth=64 and jobs=12.
> Let's see what we can come up with for benchmarking, since it would be
> a big help! I'm running kernel 4.4-rc7 right now on a Debian Jessie system.
>
> Pasi> It just wouldn't cache writes properly for me. I tried both the
> Pasi> smq and mq, and played with all of the parameters.. it just
> Pasi> wouldn't help much for my use case.
>
> I have both mq and smq compiled into the kernel, so I can't figure out
> how to change between them to test.
>
Hmm, yeah, I think it was documented on the lvmcache man page.
I can't check my notes right now..
> John
-- Pasi
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2016-02-10 22:54 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-02-08 5:14 Best option for SSD caching on a md raid array? Thomas Fjellstrom
2016-02-08 8:36 ` David Brown
2016-02-08 15:45 ` Jens-U. Mozdzen
2016-02-08 20:21 ` Thomas Fjellstrom
2016-02-08 9:38 ` Mateusz Korniak
2016-02-08 15:34 ` John Stoffel
2016-02-10 22:13 ` Pasi Kärkkäinen
2016-02-10 22:21 ` John Stoffel
2016-02-10 22:54 ` Pasi Kärkkäinen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.