[linux-lvm] LVM Read/Write speed <10% drive's normal speed

[linux-lvm] LVM Read/Write speed <10% drive's normal speed

[Stephen Hamer]

[-- Attachment #1: Type: text/plain, Size: 2609 bytes --]

Hello all,

I've been searching through the archives and the internet for a while
now, and can't seem to find anything that helps me out. I hope you don't
my posting this to both xen-users and linux-lvm simultaneously, but i
figured it'd help keep the solution in one place... even though it'll be
the same across two places... Anyways:

My setup is this:
        Xen 3.3 hypervisor (from the ubuntu repositories)
        Ubuntu 9.04 server
        Kernel and module listed here:
http://www.infohit.net/blog/post/running-xen-on-ubuntu-intrepid-and-jaunty.html

The dom0 and domUs in my Xen setup are all Ubuntu 9.04, using that
kernel. (I'm not sure if the kernel has something to do with it, so i
included it just in case.) All the domUs are set up to use LVM. The dom0
uses a straight ext3 fs on /dev/sda1. The domUs use ext3 fs's within
LVMs (using LVM2) set up on /dev/sda5.

I have noticed a massive file I/O problem on all of my domUs. While I
can peak file operations around 100 MB/s within the dom0, I can't get
anything more than 3 MB/s read OR write out of a file operation on the
domUs. I am using values reported by "iotop" to make this distinction.
I'm using both the deployment of a Jboss server and the "dd" command to
benchmark this. The Jboss slowdown is how I found this. On the dom0, the
server takes 56s to come up. On a domU, the same operation takes 15
minutes.

It should also be noted that I attempted to change the filesystems to
ext2, because I noticed that 'kjournald' was chugging away taking up
most of the i/o percentage, but only writing at KB/s! Actually, when I
removed the journaling is when the process slowed from 7 minutes to 15.
Even without the kjournald there, the java process took forever to load
and never wrote more than 3 MB/s.

I have messed around with the 'blockdev' and 'hdparm' commands (the
second of which doesn't even interact with any partition on my computer,
lvm or no, instead simply failing with an "input/output error" message),
but have had no performance increases after messing around with the read
ahead speed of the devices. I also tried the "lvchange -r" command to
set the readahead higher. I think there's something else missing here,
though.

I'm fairly fresh to xen, this being only my second installation, and I'm
extremely fresh to lvm. I've played with what seems like every option
within "xm" and the 'lv*' commands. The LVMs themselves I haven't played
with all too much yet, as I don't want to nuke a certain domU before it
needs to be used heavily tomorrow. I've already killed a test domU that
I brought up just for this.

-Stephen

[-- Attachment #2: Type: text/html, Size: 3163 bytes --]

Re: [linux-lvm] LVM Read/Write speed <10% drive's normal speed

[André Gillibert]

Stephen Hamer <Stephen.Hamer@jhuapl.edu> wrote:
> Hello all,
> 
> I've been searching through the archives and the internet for a while
> now, and can't seem to find anything that helps me out. I hope you don't
> my posting this to both xen-users and linux-lvm simultaneously, but i
> figured it'd help keep the solution in one place... even though it'll be
> the same across two places... Anyways:
> 
> My setup is this:
>         Xen 3.3 hypervisor (from the ubuntu repositories)
>         Ubuntu 9.04 server
>         Kernel and module listed here:
> http://www.infohit.net/blog/post/running-xen-on-ubuntu-intrepid-and-jaunty.html
> 
> The dom0 and domUs in my Xen setup are all Ubuntu 9.04, using that
> kernel. (I'm not sure if the kernel has something to do with it, so i
> included it just in case.) All the domUs are set up to use LVM. The dom0
> uses a straight ext3 fs on /dev/sda1. The domUs use ext3 fs's within
> LVMs (using LVM2) set up on /dev/sda5.
> 
> I have noticed a massive file I/O problem on all of my domUs. While I
> can peak file operations around 100 MB/s within the dom0, I can't get
> anything more than 3 MB/s read OR write out of a file operation on the
> domUs. I am using values reported by "iotop" to make this distinction.
> I'm using both the deployment of a Jboss server and the "dd" command to
> benchmark this. The Jboss slowdown is how I found this. On the dom0, the
> server takes 56s to come up. On a domU, the same operation takes 15
> minutes.
> 
> It should also be noted that I attempted to change the filesystems to
> ext2, because I noticed that 'kjournald' was chugging away taking up
> most of the i/o percentage, but only writing at KB/s! Actually, when I
> removed the journaling is when the process slowed from 7 minutes to 15.
> Even without the kjournald there, the java process took forever to load
> and never wrote more than 3 MB/s.
> 
> I have messed around with the 'blockdev' and 'hdparm' commands (the
> second of which doesn't even interact with any partition on my computer,
> lvm or no, instead simply failing with an "input/output error" message),
> but have had no performance increases after messing around with the read
> ahead speed of the devices. I also tried the "lvchange -r" command to
> set the readahead higher. I think there's something else missing here,
> though.
> 
> I'm fairly fresh to xen, this being only my second installation, and I'm
> extremely fresh to lvm. I've played with what seems like every option
> within "xm" and the 'lv*' commands. The LVMs themselves I haven't played
> with all too much yet, as I don't want to nuke a certain domU before it
> needs to be used heavily tomorrow. I've already killed a test domU that
> I brought up just for this.
> 
> -Stephen

Are you sure the file systems in DomUs are not mounted with the sync option, either in their fstab entry or in their default mount options as set with tune2fs? That could explain poor write performance, but not poor read performance. :|
When using dd to create a file, does the block size change performances significantly?

i.e. compare:
dd if=/dev/zero of=testfile bs=1M count=10
to:
dd if=/dev/zero of=testfile bs=1K count=10240

You should individually test the performance (in a test DomU) of the file system on the LVM partition in the DomU, the performance of the LVM logical volume and the performance of the physical volume containing the LVM logical volume, so that you see which layer is responsible.

That's general advice, but I hope it may help.

-- 
André Gillibert

Re: [linux-lvm] LVM Read/Write speed <10% drive's normal speed

[Bryn M. Reeves]

On Wed, 2009-09-16 at 21:49 +0000, Stephen Hamer wrote:
> Hello all,
> 
> I've been searching through the archives and the internet for a while
> now, and can't seem to find anything that helps me out. I hope you
> don't my posting this to both xen-users and linux-lvm simultaneously,
> but i figured it'd help keep the solution in one place... even though
> it'll be the same across two places... Anyways:
> 
> My setup is this:
>         Xen 3.3 hypervisor (from the ubuntu repositories)
>         Ubuntu 9.04 server
>         Kernel and module listed here:
> http://www.infohit.net/blog/post/running-xen-on-ubuntu-intrepid-and-jaunty.html
> 
> The dom0 and domUs in my Xen setup are all Ubuntu 9.04, using that
> kernel. (I'm not sure if the kernel has something to do with it, so i
> included it just in case.) All the domUs are set up to use LVM. The
> dom0 uses a straight ext3 fs on /dev/sda1. The domUs use ext3 fs's
> within LVMs (using LVM2) set up on /dev/sda5.

What is the configuration of the guest's storage? I.e. what type of virt
driver are you using to expose the dom0 storage to the domUs? This can
have a massive impact on the performance levels you can achieve.

You should also test the performance of these underlying devices in the
domUs - i.e. benchmark read (and if possible write) performance of
the /dev/sd*, /dev/hd* or /dev/xvd* devices seen in domU (or whatever
other type of virtual disks you are using).

Regards,
Bryn.

[linux-lvm] Re: [Xen-users] LVM Read/Write speed <10% drive's normal speed: SOLVED

[Stephen Hamer]

[-- Attachment #1: Type: text/plain, Size: 4290 bytes --]

WOW. Thank you all for the overwhelming response!!! I'm going to answer
you all in order of who replied first. 
(NOTE: the solution is in the 2nd section. Thanks Oliver!)
-Stephen



==================================
On Thu, 2009-09-17 at 03:41 -0400, Robert Dunkley wrote: 

> >Hi Stephen,
> >Can you please post your DomU config file to the list. 
> >Rob


Sure, without any of the comments, here it is (without the true names or
IPs, of course):

kernel      = '/boot/vmlinuz-2.6.26-2-xen-686'
ramdisk     = '/boot/initrd.img-2.6.26-2-xen-686'
memory      = '1024'
root        = '/dev/hda2 ro'
disk        = [ 'phy:/dev/xenvg/testVM1-swap,hda1,w',
                  'phy:/dev/xenvg/testVM1-root,hda2,w', ]

name        = 'testVM1'

vif = [ 'ip=192.168.0.100' ]

on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

# Needs to be inserted to allow console to attach to VM
extra="console=hvc0 xencons=tty"


This was generated by xen-tools (after a little modification to
xen-tools xm.tmpl). I like how easy xen-tools is to expand upon. For
example, I used 'xen-update-image' as a model to create my own
'xen-image-cmd' which can pass a single command to multiple offline
domUs. Works just like 'xen-update-image', except for the fact that you
can pass ANY command that the domU can execute AND you can force the
output to be redirected to a file INSIDE the chrooted image. I was sort
of proud of myself for that scripting voodoo. =D


==================================
On Thu, 2009-09-17 at 03:45 -0400, Olivier B. wrote:

> >Hi,
> 
> >what mount options do you use on the DomU ?
> >Under Debian there is xen-tools example with the "sync" mount options : 
> >if you use it, writes will be really slow.
> 
> >Olivier


THIS is where the money is. Once I removed the 'sync' option from the
fstab for my root partition, the I/O speed sailed. Makes sense, really.
This got the VM's performance to actually BEAT the old bare metal
machine that we were running this server on before. 
Thanks for your help!


==================================
On Thu, 2009-09-17 at 06:21 -0400, Bryn M. Reeves wrote: 

> What is the configuration of the guest's storage? I.e. what type of virt
> driver are you using to expose the dom0 storage to the domUs? This can
> have a massive impact on the performance levels you can achieve.
> 
> You should also test the performance of these underlying devices in the
> domUs - i.e. benchmark read (and if possible write) performance of
> the /dev/sd*, /dev/hd* or /dev/xvd* devices seen in domU (or whatever
> other type of virtual disks you are using).
> 
> Regards,
> Bryn.


I'm not PRECISELY sure what you mean here, but I think you're referring
to the xen driver used to pass the storage in as a block dev. to the
domU. If that is what you mean, I used the 'phy:' command. The
performance of the /dev/hda* inside the domU is what I was mentioning
trying to benchmark simplistically with 'dd'. 
- A 'dd' command in my dom0 on the true physical /dev/sda1 got 100
MB/s. 
- Before I turned off 'sync', a 'dd' command on /dev/hda2 inside my domU
(physically a 10GB LVM on the physical device /dev/sda5) got 3 MB/s. 
I noted later that running a DD command directly to another LVM on that
same VG as the others got comparable performance to the 100 MB/s. That
was my first indicator that the domUs were doing something nasty. 
- After I turned off 'sync', a 'dd' command on that same /dev/hda2
inside the same domU got 100 MB/s!




==================================
On Thu, 2009-09-17 at 10:59 -0400, Jeff Sturm wrote: 

> Your domU's--are they paravirt, or HVM?

I believe they're paravirtualized, unless I'm doing something silly.
They're running off the modified kernel, and I'm not doing any hardware
forwarding magic. See config file at the top.

==================================
On Thu, 2009-09-17 at 11:16 -0400, Fajar A. Nugraha wrote:

> I suggest you try using official xen.org's 2.6.18 kernel, or other
> vendor-supported 2.6.18 kernel-xen (like RHEL5's kernel-xen, or even
> Etch's kernel) for both domU and dom0 and see if the problem persists.

I've noticed a couple of quirks in the startup sequence of my domUs, I
figured it had something interesting to do with the kernel. I'll look
into using one of those kernels. Thanks!

[-- Attachment #2: Type: text/html, Size: 5259 bytes --]