How to Allocate Disk Bandwidth among VMs?

How to Allocate Disk Bandwidth among VMs?

[Tom Creck]

Hello, everyone:

      I want to do disk I/O rate control over VMs. Therefore, I want to
allocate different disk I/O bandwidth for different Xen VMs on my host
machine. All domainUs use file-backed VBDs stored in domain0's file system.
      Do you have any idea to do that? Hopefully it can be done by modifying
Xend in domain0.
-- 
View this message in context: http://www.nabble.com/How-to-Allocate-Disk-Bandwidth-among-VMs--tp23993255p23993255.html
Sent from the Xen - Dev mailing list archive at Nabble.com.

Re: How to Allocate Disk Bandwidth among VMs?

[Ryo Tsuruta]

Hi Tom,

# I'm sorry, I sent an empty e-mail a while ago.

>       I want to do disk I/O rate control over VMs. Therefore, I want to
> allocate different disk I/O bandwidth for different Xen VMs on my host
> machine. All domainUs use file-backed VBDs stored in domain0's file system.
>       Do you have any idea to do that? Hopefully it can be done by modifying
> Xend in domain0.

You can use dm-ioband for this purpose. dm-ioband is an I/O bandwidth
controller implemented as a device-mapper driver and can control
bandwidth on per partition, per user, per process basis.

In this case, install dm-ioband to the host OS, create a
dm-ioband device on the disk which stores domainU's VBD files,
and then assign bandwidth(determined proportional to the weight of
each disk) to each virtual machine.

There is an example configuration available at:
"Example #5: Bandwidth control for Xen blktap devices"
http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband/man/examples

Please see the following URL for more information, kernel patch files
and binary packages for RHEL5 and CentOS5 are availble.
http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband

Please feel free to ask me if you have any questions.

Thanks,
Ryo Tsuruta

Re: How to Allocate Disk Bandwidth among VMs?

[Tom Creck]

Thank you a lot for your reply. I will try dm-ioband for Xen disk I/O QoS
ASAP.

Actually, I'm also surveying ways of Xen disk I/O QoS. There are two kinds
of solution as I see.
One is at the frontend/backend driver level. It avoids replying on any
specific guest OS.
There are two patches for this same purpose.
The first is Xen I/O manager.  
(http://lists.xensource.com/archives/html/xen-devel/2007-07/msg00863.html)

The second is a token-based resource limitation in backend driver 
(http://www.nabble.com/-PATCH-0-2--blkback:-Token-based-QoS-resource-limiting-for-VBD-I-O.-td22808160.html)

However, my lastest Xen 3.3.0 doesn't include either. I wonder does Xen
3.3.0 incorporate any disk I/O QoS mechanism inside?

The second kind of solution, as the one you suggest, bases on Domain0 OS
kernel's scheduling of backend driver's kernel thread, such as dm-ioband and
ionice for per-process scheduling.

I tried ionice, but got no effect. I don't know why.

Is there any other idea or implementation for disk I/O QoS in Xen?



Ryo Tsuruta wrote:
> 
> Hi Tom,
> 
> # I'm sorry, I sent an empty e-mail a while ago.
> 
>>       I want to do disk I/O rate control over VMs. Therefore, I want to
>> allocate different disk I/O bandwidth for different Xen VMs on my host
>> machine. All domainUs use file-backed VBDs stored in domain0's file
>> system.
>>       Do you have any idea to do that? Hopefully it can be done by
>> modifying
>> Xend in domain0.
> 
> You can use dm-ioband for this purpose. dm-ioband is an I/O bandwidth
> controller implemented as a device-mapper driver and can control
> bandwidth on per partition, per user, per process basis.
> 
> In this case, install dm-ioband to the host OS, create a
> dm-ioband device on the disk which stores domainU's VBD files,
> and then assign bandwidth(determined proportional to the weight of
> each disk) to each virtual machine.
> 
> There is an example configuration available at:
> "Example #5: Bandwidth control for Xen blktap devices"
> http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband/man/examples
> 
> Please see the following URL for more information, kernel patch files
> and binary packages for RHEL5 and CentOS5 are availble.
> http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband
> 
> Please feel free to ask me if you have any questions.
> 
> Thanks,
> Ryo Tsuruta
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 

-- 
View this message in context: http://www.nabble.com/How-to-Allocate-Disk-Bandwidth-among-VMs--tp23993255p24049939.html
Sent from the Xen - Dev mailing list archive at Nabble.com.

Re: How to Allocate Disk Bandwidth among VMs?

[Tom Creck]

Thank you a lot for your reply. I will try dm-ioband for Xen disk I/O QoS
ASAP.

Actually, I'm also surveying ways of Xen disk I/O QoS. There are two kinds
of solution as I see.
One is at the frontend/backend driver level. It avoids replying on any
specific guest OS.
There are two patches for this same purpose.
The first is Xen I/O manager.  
(http://lists.xensource.com/archives/html/xen-devel/2007-07/msg00863.html)

The second is a token-based resource limitation in backend driver 
(http://www.nabble.com/-PATCH-0-2--blkback:-Token-based-QoS-resource-limiting-for-VBD-I-O.-td22808160.html)

However, my lastest Xen 3.3.0 doesn't include either. I wonder does Xen
3.3.0 incorporate any disk I/O QoS mechanism inside?

The second kind of solution, as the one you suggest, bases on Domain0 OS
kernel's scheduling of backend driver's kernel thread, such as dm-ioband and
ionice for per-process scheduling.

I tried ionice, but got no effect. I don't know why.

Is there any other idea or implementation for disk I/O QoS in Xen?



Ryo Tsuruta wrote:
> 
> Hi Tom,
> 
> # I'm sorry, I sent an empty e-mail a while ago.
> 
>>       I want to do disk I/O rate control over VMs. Therefore, I want to
>> allocate different disk I/O bandwidth for different Xen VMs on my host
>> machine. All domainUs use file-backed VBDs stored in domain0's file
>> system.
>>       Do you have any idea to do that? Hopefully it can be done by
>> modifying
>> Xend in domain0.
> 
> You can use dm-ioband for this purpose. dm-ioband is an I/O bandwidth
> controller implemented as a device-mapper driver and can control
> bandwidth on per partition, per user, per process basis.
> 
> In this case, install dm-ioband to the host OS, create a
> dm-ioband device on the disk which stores domainU's VBD files,
> and then assign bandwidth(determined proportional to the weight of
> each disk) to each virtual machine.
> 
> There is an example configuration available at:
> "Example #5: Bandwidth control for Xen blktap devices"
> http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband/man/examples
> 
> Please see the following URL for more information, kernel patch files
> and binary packages for RHEL5 and CentOS5 are availble.
> http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband
> 
> Please feel free to ask me if you have any questions.
> 
> Thanks,
> Ryo Tsuruta
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 

-- 
View this message in context: http://www.nabble.com/How-to-Allocate-Disk-Bandwidth-among-VMs--tp23993255p24049942.html
Sent from the Xen - Dev mailing list archive at Nabble.com.

Re: How to Allocate Disk Bandwidth among VMs?

[Tom Creck]

The example at
"http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband/man/examples" uses
host partition-backed VBD for domainUs.

However, I use file-backed VBD for domainU. Does dm-ioband work for
file-backed VBD? Is there any example how to configure for this case?


Ryo Tsuruta wrote:
> 
> Hi Tom,
> 
> # I'm sorry, I sent an empty e-mail a while ago.
> 
>>       I want to do disk I/O rate control over VMs. Therefore, I want to
>> allocate different disk I/O bandwidth for different Xen VMs on my host
>> machine. All domainUs use file-backed VBDs stored in domain0's file
>> system.
>>       Do you have any idea to do that? Hopefully it can be done by
>> modifying
>> Xend in domain0.
> 
> You can use dm-ioband for this purpose. dm-ioband is an I/O bandwidth
> controller implemented as a device-mapper driver and can control
> bandwidth on per partition, per user, per process basis.
> 
> In this case, install dm-ioband to the host OS, create a
> dm-ioband device on the disk which stores domainU's VBD files,
> and then assign bandwidth(determined proportional to the weight of
> each disk) to each virtual machine.
> 
> There is an example configuration available at:
> "Example #5: Bandwidth control for Xen blktap devices"
> http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband/man/examples
> 
> Please see the following URL for more information, kernel patch files
> and binary packages for RHEL5 and CentOS5 are availble.
> http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband
> 
> Please feel free to ask me if you have any questions.
> 
> Thanks,
> Ryo Tsuruta
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 

-- 
View this message in context: http://www.nabble.com/How-to-Allocate-Disk-Bandwidth-among-VMs--tp23993255p24053119.html
Sent from the Xen - Dev mailing list archive at Nabble.com.

Re: How to Allocate Disk Bandwidth among VMs?

[Ryo Tsuruta]

Hi Tom,

Tom Creck <tom-xen@hotmail.com> wrote:
> 
> The example at
> "http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband/man/examples" uses
> host partition-backed VBD for domainUs.
> 
> However, I use file-backed VBD for domainU. Does dm-ioband work for
> file-backed VBD? Is there any example how to configure for this case?

Of course, dm-ioband can work for file-backed VBD.
Please see "Example #5: Bandwidth control for Xen blktap devices" on
that page. That example shows configurations for file-backed VBD.

Thanks,
Ryo Tsuruta

> Ryo Tsuruta wrote:
> > 
> > Hi Tom,
> > 
> > # I'm sorry, I sent an empty e-mail a while ago.
> > 
> >>       I want to do disk I/O rate control over VMs. Therefore, I want to
> >> allocate different disk I/O bandwidth for different Xen VMs on my host
> >> machine. All domainUs use file-backed VBDs stored in domain0's file
> >> system.
> >>       Do you have any idea to do that? Hopefully it can be done by
> >> modifying
> >> Xend in domain0.
> > 
> > You can use dm-ioband for this purpose. dm-ioband is an I/O bandwidth
> > controller implemented as a device-mapper driver and can control
> > bandwidth on per partition, per user, per process basis.
> > 
> > In this case, install dm-ioband to the host OS, create a
> > dm-ioband device on the disk which stores domainU's VBD files,
> > and then assign bandwidth(determined proportional to the weight of
> > each disk) to each virtual machine.
> > 
> > There is an example configuration available at:
> > "Example #5: Bandwidth control for Xen blktap devices"
> > http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband/man/examples
> > 
> > Please see the following URL for more information, kernel patch files
> > and binary packages for RHEL5 and CentOS5 are availble.
> > http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband
> > 
> > Please feel free to ask me if you have any questions.
> > 
> > Thanks,
> > Ryo Tsuruta
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> > 
> > 
> 
> -- 
> View this message in context: http://www.nabble.com/How-to-Allocate-Disk-Bandwidth-among-VMs--tp23993255p24053119.html
> Sent from the Xen - Dev mailing list archive at Nabble.com.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel