Best strategy for caching VMs storage

Best strategy for caching VMs storage

[Santiago Castillo Oli]

Hi there.


I have a host running 4 VMs using qcow2 storage on a ext4 fs over HDD. 
Each VM has 3 qcow files (system, data and swap). I know I have an I/O 
bottleneck.

I want to use bcache with an SSD to accelerate disk access but I´m not 
sure where should I put bcache on storage stack.


Should I use bcache on host or in guests?

Just one bcache backing device for a single (ext4) filesystem with all 
qcow files there, or different bcache and backing devices for each qcow2 
file?


Right know, I prefer qcow2 over thin-lvm for storage, but i could change 
my mind if thin-lvm is a much better combination for bcache.


What would be the best strategy for caching VMs storage ?

Any recommendation, please?


Regards and thank you


-- 
___________________________________________________________

Re: Best strategy for caching VMs storage

[Matthias Ferdinand]

On Fri, May 21, 2021 at 01:56:16PM +0200, Santiago Castillo Oli wrote:
> Hi there.
> 
> 
> I have a host running 4 VMs using qcow2 storage on a ext4 fs over HDD. Each
> VM has 3 qcow files (system, data and swap). I know I have an I/O
> bottleneck.
> 
> I want to use bcache with an SSD to accelerate disk access but I´m not sure
> where should I put bcache on storage stack.
> 
> 
> Should I use bcache on host or in guests?
> 
> Just one bcache backing device for a single (ext4) filesystem with all qcow
> files there, or different bcache and backing devices for each qcow2 file?
> 
> 
> Right know, I prefer qcow2 over thin-lvm for storage, but i could change my
> mind if thin-lvm is a much better combination for bcache.
> 
> 
> What would be the best strategy for caching VMs storage ?
> 
> Any recommendation, please?


Hi,

not claiming to know "the best" strategy, but I would recommend

  - use a single bcache device on the host

  - either use LVM (thick provisioned) to provide block devices
    to VMs, or put a filesystem on it and store qcow2 files there as
    you did before

With lvm-thin you have all metadata activity for all your VMs in one
place. Any error there and you might lose all your VM storage at once.
Of course you should do regular backups of your VMs anyway, but I would
not start using lvm-thin unless I can have the relevant metadata volume
on redundant storage because of the blast radius.
Just my paranoic 2c :-)

Speaking of blast radius: adding an SSD to the stack will make your VMs
storage performance and availability depend on two devices, not just
one, so this might increase your failure rate. Choose a high-quality
SSD, preferably datacenter-grade equipment. And of course, do your own
performance tests to see if there is enough performance improvement to
justify the higher risk of failure.

Matthias