KVM on top of BTRFS

KVM on top of BTRFS

[Matthew Hawn]

What are the recommendations for running KVM images on BTRFS systems using kernel 3.4?  I saw older posts on the web complaining about poor performance, but I know a lot of work has gone into btrfs since then.  There also seemed to be the nocow option, but I didn't find anything that said it actualy helped.

Anybody have ideas?

Thanks,
Matt

Re: KVM on top of BTRFS

[Alex]

Matthew Hawn <steamraven <at> yahoo.com> writes:

> 
> What are the recommendations for running KVM images on BTRFS systems using
kernel 3.4?  I saw older
> posts on the web complaining about poor performance, but I know a lot of work
has gone into btrfs since then.
>  There also seemed to be the nocow option, but I didn't find anything that
said it actualy helped.
> 

Running KVM image files on btrfs as of yesterday.
Used mkfs.btrfs -l 32k -n 32k to create and default options only (no mount
options bar defaults, so no compression (just asking for trouble!), or
autodefrag). 3.4.1 official Debian AMD64 kernel. Multi-subvol including root
with set-default enacted. 32k's adopted per Chris's post.

Install:
virt-install --connect qemu:///system -n china -r 256 --disk
path=/var/lib/libvirt/images/china.img,size=4 -c
/home/alex/debian-testing-amd64-CD-1.iso --vnc --noautoconsole --os-type linux
--os-variant debianwheezy --accelerate --network=bridge:br0 --hvm 

This runs much slower than expected - have done many debian bare minimum
installs like this before. Can't hear any disk thrashing. Doesn't appear to be
CPU or memory bound - will double check.

Want to add autodefrag option but saw a comment about it on the wiki gotchas
page or here (~ need to make sure it doesn't multi invoke or sommat?).

In contrast, VirtualBox 60GB WindowsXP guests quite happy running and very quick
on standard btrfs 3.3.7 kernel create in ArchLinux - with absolutely no flash
stuff. Didn't create them in btrfs though (XFS).

That may be the difference: level of churn in the file.
Saw a post about using filefrag program on btrfs and want to try that.

Questions?

Re: KVM on top of BTRFS

[dima]

On 06/12/2012 08:53 AM, Alex wrote:
> Matthew Hawn <steamraven <at> yahoo.com> writes:
>> What are the recommendations for running KVM images on BTRFS systems using
> kernel 3.4?  I saw older
>> posts on the web complaining about poor performance, but I know a lot of work
> has gone into btrfs since then.
>>   There also seemed to be the nocow option, but I didn't find anything that
> said it actualy helped.

I don't think it is about the kernel version, but rather about choosing 
the right option for KVM.
I am running KVM via libvirt on btrfs with lzo compression, autodefrag, 
inode and space cache for quite some time. Yes, I did set nocow for the 
directory with images. I have no proof that it actually helps with disk 
images, but being set for the build directory my build time is down 5 
minutes.
I used raw virtio images with no caching. I guess this is the key if we 
talk about disk I/O. The performance looks fine, though I only use it 
for testing.
The last time I tried to install WinXP on KVM it was a disaster. But I 
guess I did not choose the right options, nor did I install virtio 
drivers inside the guest.
VirtualBox still outperforms KVM on btrfs in my view.

best

Re: KVM on top of BTRFS

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1538 bytes --]

On Mon, 11 Jun 2012 23:53:34 +0000 (UTC)
Alex <alex@bpmit.com> wrote:

> Running KVM image files on btrfs as of yesterday.
> Used mkfs.btrfs -l 32k -n 32k to create and default options only (no mount
> options bar defaults, so no compression (just asking for trouble!), or
> autodefrag). 3.4.1 official Debian AMD64 kernel. Multi-subvol including root
> with set-default enacted. 32k's adopted per Chris's post.
> 
> Install:
> virt-install --connect qemu:///system -n china -r 256 --disk
> path=/var/lib/libvirt/images/china.img,size=4 -c
> /home/alex/debian-testing-amd64-CD-1.iso --vnc --noautoconsole --os-type linux
> --os-variant debianwheezy --accelerate --network=bridge:br0 --hvm 
> 
> This runs much slower than expected - have done many debian bare minimum
> installs like this before. Can't hear any disk thrashing. Doesn't appear to be
> CPU or memory bound - will double check.


From man kvm:

           cache=cache
               cache is "none", "writeback", "unsafe", "directsync" or
               "writethrough" and controls how the host cache is used to
               access block data.

The default setting is writethrough, can you try writeback, then unsafe, to
see if they help increase performance in your case? AFAIK the default setting
may cause KVM to call sync a bit too often, and sync can be slow on btrfs.

-- 
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: KVM on top of BTRFS

[Hubert Kario]

On Monday 11 of June 2012 23:53:34 Alex wrote:
> Matthew Hawn <steamraven <at> yahoo.com> writes:
> > What are the recommendations for running KVM images on BTRFS systems using
> 
> Install:
> virt-install --connect qemu:///system -n china -r 256 --disk
> path=/var/lib/libvirt/images/china.img,size=4 -c
> /home/alex/debian-testing-amd64-CD-1.iso --vnc --noautoconsole --os-type
> linux --os-variant debianwheezy --accelerate --network=bridge:br0 --hvm
> 
> This runs much slower than expected - have done many debian bare minimum
> installs like this before. Can't hear any disk thrashing. Doesn't appear to
> be CPU or memory bound - will double check.

>From what I heard, this is caused by slow KVM CD virtualisation.

Try to install it and do some tests then.

Regards,
--
Hubert Kario
QBS - Quality Business Software
02-656 Warszawa, ul. Ksawerów 30/85
tel. +48 (22) 646-61-51, 646-74-24
www.qbs.com.pl

Re: KVM on top of BTRFS

[Alex]

Roman Mamedov <rm <at> romanrm.ru> writes:

(Machine has no other load at all)

No material difference between cache types (none, writeback and writethrough)
when I tried.
I had been using partition based KVM which is obviously going to
be much faster.

Seems a little unfair on btrfs to just to look at absolutes in this context.
Prior reports said that the fs ground to a halt,
it isn't doing that by any stretch.

The installs are very much diskbound iotop/atop.
More than happy to answer questions and try out things.

I haven't let any of these installs complete and used it as intended.
So that's what I intend to do next; after all one doesn't install every day.

Re: KVM on top of BTRFS

[Alex]

Hubert Kario <hka <at> qbs.com.pl> writes:

> 

> 
> From what I heard, this is caused by slow KVM CD virtualisation.
> 
> Try to install it and do some tests then.
> 

You read my mind :-)

Re: KVM on top of BTRFS

[Alex]

Alex <alex <at> bpmit.com> writes:

> 
> Roman Mamedov <rm <at> romanrm.ru> writes:
> 
> (Machine has no other load at all)

> 

> 

Roman Mamedov didn't write it, I did, I had huge trouble getting past the 80
char limit thing! Sorry Roman.

Re: KVM on top of BTRFS

[steamraven]

> 
> Seems a little unfair on btrfs to just to look at absolutes in this context.
> Prior reports said that the fs ground to a halt,
> it isn't doing that by any stretch.


I agree.  What I am mostly looking for is the best setup 
for using KVM snapshots: 

KVM qcow2 on top of something like ext4  or
raw on top of btrfs


> 
> I haven't let any of these installs complete and used it as intended.
> So that's what I intend to do next; after all one doesn't install every day.
> 

I am going to try to benchmark a couple variations and flags
qcow2 on ext4  (noatime)
raw on btrfs (defaults)
raw on btrfs (noatime,space_cache)
raw on btrfs (noatime,nospace_cache)
raw on btrfs (noatime,nodatacow)

Any other options that might be good to try?

Re: KVM on top of BTRFS

[Ernst Sjöstrand]

Hi,

you can't beat the benchmarks that Serge Hallyn did, really thorough!

http://s3hh.wordpress.com/2012/05/02/first-round-of-kvm-performance-tests/

Regards
//Ernst

2012/6/12 steamraven <steamraven@yahoo.com>:
>>
>> Seems a little unfair on btrfs to just to look at absolutes in this context.
>> Prior reports said that the fs ground to a halt,
>> it isn't doing that by any stretch.
>
>
> I agree.  What I am mostly looking for is the best setup
> for using KVM snapshots:
>
> KVM qcow2 on top of something like ext4  or
> raw on top of btrfs
>
>
>>
>> I haven't let any of these installs complete and used it as intended.
>> So that's what I intend to do next; after all one doesn't install every day.
>>
>
> I am going to try to benchmark a couple variations and flags
> qcow2 on ext4  (noatime)
> raw on btrfs (defaults)
> raw on btrfs (noatime,space_cache)
> raw on btrfs (noatime,nospace_cache)
> raw on btrfs (noatime,nodatacow)
>
> Any other options that might be good to try?
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: KVM on top of BTRFS

[steamraven]

Ernst Sjöstrand <ernstp <at> gmail.com> writes:

> 
> Hi,
> 
> you can't beat the benchmarks that Serge Hallyn did, really thorough!
> 
> http://s3hh.wordpress.com/2012/05/02/first-round-of-kvm-performance-tests/

They do seem very thorough. Unfortunately, they are kvm on top of ext4 and
 he was mainly checking caching parameters and storage formats.  I am 
looking at BTRFS options and comparing them against qcow2 on ext4.

Matt  

Re: KVM on top of BTRFS

[steamraven]

A couple tests have been uploaded to http://openbenchmarking.org/result/1206152-
STEA-VMTESTS05.


Note for the nodatacow, the subvolume did have a snapshot, since the point was 
for VM's that could be snapshoted (either through virsh or btrfs).  Thus the 
first write to a block was cow, but subsequent ones where not.

FY, the tests are not very thorough yet