xen 4.3.2 rc1 memory ballooning

xen 4.3.2 rc1 memory ballooning

[Vasiliy Tolstov]

Hi all.
Today i'm compile xen 4.3.2 rc1 from stable-4.3 branch.
Also i'm using upstream qemu from debian jessie (1.7.0).

Domain created and works fine.
memory=512
maxmemory=1024

Problem:
>From dom0 xl mem-set works fine, it modifies xenstore and do hipercall to xen.
>From domU side memory ballooned up.

Now i'm try to balloon up memory from guest (write to
/sys/devices/system/xen_memory/xen_memory0/target) and memory
ballooned up only to 1Mb. For example before 512, after 513.

Main problems that i cant upgrade guest kernel (i have many vps with
2.6.32.26 from old xen kernel git tree).

How can i deal with it? Whats wrong from xen side?


-- 
Vasiliy Tolstov,
e-mail: v.tolstov@selfip.ru
jabber: vase@selfip.ru

Re: xen 4.3.2 rc1 memory ballooning

[Vasiliy Tolstov]

As workaround i can use xl mem-set xxx maxmemory, after that guest vm
can balloon up and down memory inside range 0 - maxmemory.

2014-02-14 18:46 GMT+04:00 Vasiliy Tolstov <v.tolstov@selfip.ru>:
> Hi all.
> Today i'm compile xen 4.3.2 rc1 from stable-4.3 branch.
> Also i'm using upstream qemu from debian jessie (1.7.0).
>
> Domain created and works fine.
> memory=512
> maxmemory=1024
>
> Problem:
> From dom0 xl mem-set works fine, it modifies xenstore and do hipercall to xen.
> From domU side memory ballooned up.
>
> Now i'm try to balloon up memory from guest (write to
> /sys/devices/system/xen_memory/xen_memory0/target) and memory
> ballooned up only to 1Mb. For example before 512, after 513.
>
> Main problems that i cant upgrade guest kernel (i have many vps with
> 2.6.32.26 from old xen kernel git tree).
>
> How can i deal with it? Whats wrong from xen side?
>
>
> --
> Vasiliy Tolstov,
> e-mail: v.tolstov@selfip.ru
> jabber: vase@selfip.ru



-- 
Vasiliy Tolstov,
e-mail: v.tolstov@selfip.ru
jabber: vase@selfip.ru

Re: xen 4.3.2 rc1 memory ballooning

[Wei Liu]

On Fri, Feb 14, 2014 at 06:46:02PM +0400, Vasiliy Tolstov wrote:
> Hi all.
> Today i'm compile xen 4.3.2 rc1 from stable-4.3 branch.
> Also i'm using upstream qemu from debian jessie (1.7.0).
> 
> Domain created and works fine.
> memory=512
> maxmemory=1024
> 

This should be "maxmem"? I could not find "maxmemory" in documents.

> Problem:
> >From dom0 xl mem-set works fine, it modifies xenstore and do hipercall to xen.
> >From domU side memory ballooned up.
> 
> Now i'm try to balloon up memory from guest (write to
> /sys/devices/system/xen_memory/xen_memory0/target) and memory
> ballooned up only to 1Mb. For example before 512, after 513.
> 
> Main problems that i cant upgrade guest kernel (i have many vps with
> 2.6.32.26 from old xen kernel git tree).
> 
> How can i deal with it? Whats wrong from xen side?
> 

My gut feeling is nothing is wrong with Xen. It's just that you have a
typo in your config file. :-)

Wei.

> 
> -- 
> Vasiliy Tolstov,
> e-mail: v.tolstov@selfip.ru
> jabber: vase@selfip.ru
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: xen 4.3.2 rc1 memory ballooning

[Vasiliy Tolstov]

2014-02-14 20:34 GMT+04:00 Wei Liu <wei.liu2@citrix.com>:
> My gut feeling is nothing is wrong with Xen. It's just that you have a
> typo in your config file. :-)
>
> Wei.

No, if i have error in config domain can't start. You known that.
my config is:

name="21-10918"
kernel="/var/storage/kernel/debian/6/kernel-64"
ramdisk="/var/storage/kernel/ramdisk-64"
vif=["mac=00:16:3e:00:2c:63,ip=62.76.185.64"]
disk=["phy:/dev/disk/vbd/21-916,xvda,w"]
memory=512
maxmem=1024
vcpus=3
maxvcpus=3
cpu_cap=300
cpu_weight=256
vfb=["type=vnc,vnclisten=0.0.0.0,vncpasswd=7QOG1885y3"]
extra="root=/dev/xvda1 selinux=1 enforcing=0 iommu=off swiotlb=off
earlyprintk=xen console=hvc0"
on_restart="destroy"
cpuid="host,x2apic=0,aes=0,xsave=0,avx=0"
device_model_version="qemu-xen"
device_model_override="/usr/bin/qemu-system-x86_64"

-- 
Vasiliy Tolstov,
e-mail: v.tolstov@selfip.ru
jabber: vase@selfip.ru

Re: xen 4.3.2 rc1 memory ballooning

[Vasiliy Tolstov]

Update info:
If i from dom0 do xl mem-set XXX maxmem, domU can balloon up/down fine.
What is the difference between 4.1 and 4.3 in this case?

2014-02-14 22:12 GMT+04:00 Vasiliy Tolstov <v.tolstov@selfip.ru>:
> 2014-02-14 20:34 GMT+04:00 Wei Liu <wei.liu2@citrix.com>:
>> My gut feeling is nothing is wrong with Xen. It's just that you have a
>> typo in your config file. :-)
>>
>> Wei.
>
> No, if i have error in config domain can't start. You known that.
> my config is:
>
> name="21-10918"
> kernel="/var/storage/kernel/debian/6/kernel-64"
> ramdisk="/var/storage/kernel/ramdisk-64"
> vif=["mac=00:16:3e:00:2c:63,ip=62.76.185.64"]
> disk=["phy:/dev/disk/vbd/21-916,xvda,w"]
> memory=512
> maxmem=1024
> vcpus=3
> maxvcpus=3
> cpu_cap=300
> cpu_weight=256
> vfb=["type=vnc,vnclisten=0.0.0.0,vncpasswd=7QOG1885y3"]
> extra="root=/dev/xvda1 selinux=1 enforcing=0 iommu=off swiotlb=off
> earlyprintk=xen console=hvc0"
> on_restart="destroy"
> cpuid="host,x2apic=0,aes=0,xsave=0,avx=0"
> device_model_version="qemu-xen"
> device_model_override="/usr/bin/qemu-system-x86_64"
>
> --
> Vasiliy Tolstov,
> e-mail: v.tolstov@selfip.ru
> jabber: vase@selfip.ru



-- 
Vasiliy Tolstov,
e-mail: v.tolstov@selfip.ru
jabber: vase@selfip.ru

Re: xen 4.3.2 rc1 memory ballooning

[Daniel Kiper]

On Mon, Feb 17, 2014 at 01:27:57PM +0400, Vasiliy Tolstov wrote:
> Update info:
> If i from dom0 do xl mem-set XXX maxmem, domU can balloon up/down fine.
> What is the difference between 4.1 and 4.3 in this case?

Have you moved from xm to xl? If yes this is the change. You could
find more info here: http://lists.xen.org/archives/html/xen-devel/2013-04/msg03072.html
Those patches are not applied. However, I am going to back to this
work one day.

Daniel

Re: xen 4.3.2 rc1 memory ballooning

[Vasiliy Tolstov]

2014-02-17 14:19 GMT+04:00 Daniel Kiper <dkiper@net-space.pl>:
> Have you moved from xm to xl? If yes this is the change. You could
> find more info here: http://lists.xen.org/archives/html/xen-devel/2013-04/msg03072.html
> Those patches are not applied. However, I am going to back to this
> work one day.


Thanks! Yes, i moved from xm to xl.  Does this patches can work with xen 4.3?

-- 
Vasiliy Tolstov,
e-mail: v.tolstov@selfip.ru
jabber: vase@selfip.ru

Re: xen 4.3.2 rc1 memory ballooning

[Wei Liu]

On Mon, Feb 17, 2014 at 01:27:57PM +0400, Vasiliy Tolstov wrote:
> Update info:
> If i from dom0 do xl mem-set XXX maxmem, domU can balloon up/down fine.
> What is the difference between 4.1 and 4.3 in this case?
> 

I don't quite know the difference between 4.1 and 4.3 off the top of my
head. 4.1 is too old for me.

I think I've seen this before. The root cause is that xl (now) always
caps maxmem to memory when building a domain but never raises it to the
real maxmem set in config file. It's a matter of turning a parameter
from "true" to "false" in source code, but it's not exposed to users at
the moment.

If you're interested, have a look at libxl/xl_cmdimpl.c:2683. The
"enforce" parameter to libxl_set_memory_target is always set to 1.

Daniel posted a series of patch to fix that behavior but it wasn't
merged. It's too late to fix that for 4.5.  We can revisit that series
during next development cycle if we have time.

http://lists.xenproject.org/archives/html/xen-devel/2013-04/msg03072.html

Wei.

> 2014-02-14 22:12 GMT+04:00 Vasiliy Tolstov <v.tolstov@selfip.ru>:
> > 2014-02-14 20:34 GMT+04:00 Wei Liu <wei.liu2@citrix.com>:
> >> My gut feeling is nothing is wrong with Xen. It's just that you have a
> >> typo in your config file. :-)
> >>
> >> Wei.
> >
> > No, if i have error in config domain can't start. You known that.
> > my config is:
> >
> > name="21-10918"
> > kernel="/var/storage/kernel/debian/6/kernel-64"
> > ramdisk="/var/storage/kernel/ramdisk-64"
> > vif=["mac=00:16:3e:00:2c:63,ip=62.76.185.64"]
> > disk=["phy:/dev/disk/vbd/21-916,xvda,w"]
> > memory=512
> > maxmem=1024
> > vcpus=3
> > maxvcpus=3
> > cpu_cap=300
> > cpu_weight=256
> > vfb=["type=vnc,vnclisten=0.0.0.0,vncpasswd=7QOG1885y3"]
> > extra="root=/dev/xvda1 selinux=1 enforcing=0 iommu=off swiotlb=off
> > earlyprintk=xen console=hvc0"
> > on_restart="destroy"
> > cpuid="host,x2apic=0,aes=0,xsave=0,avx=0"
> > device_model_version="qemu-xen"
> > device_model_override="/usr/bin/qemu-system-x86_64"
> >
> > --
> > Vasiliy Tolstov,
> > e-mail: v.tolstov@selfip.ru
> > jabber: vase@selfip.ru
> 
> 
> 
> -- 
> Vasiliy Tolstov,
> e-mail: v.tolstov@selfip.ru
> jabber: vase@selfip.ru

Re: xen 4.3.2 rc1 memory ballooning

[Daniel Kiper]

On Mon, Feb 17, 2014 at 02:25:53PM +0400, Vasiliy Tolstov wrote:
> 2014-02-17 14:19 GMT+04:00 Daniel Kiper <dkiper@net-space.pl>:
> > Have you moved from xm to xl? If yes this is the change. You could
> > find more info here: http://lists.xen.org/archives/html/xen-devel/2013-04/msg03072.html
> > Those patches are not applied. However, I am going to back to this
> > work one day.
>
>
> Thanks! Yes, i moved from xm to xl.  Does this patches can work with xen 4.3?

They should. IIRC they were targeted for 4.3.

Daniel

Re: xen 4.3.2 rc1 memory ballooning

[Vasiliy Tolstov]

2014-02-17 14:56 GMT+04:00 Daniel Kiper <dkiper@net-space.pl>:
> They should. IIRC they were targeted for 4.3.


Daniel, can you check this
https://gist.github.com/vtolstov/9090413/raw/dc457631fe0b6df793552494d059eac62fd962e0/gistfile1.txt
this is patch that merges 4 patches from email (because i can't get
attaches and after copy/paste i have errors).
As i see nothing changed , i'm use enforce=0 in xl.conf and start
domain with memory=512 and maxmem=1024.


-- 
Vasiliy Tolstov,
e-mail: v.tolstov@selfip.ru
jabber: vase@selfip.ru

Re: xen 4.3.2 rc1 memory ballooning

[Daniel Kiper]

Hey,

On Wed, Feb 19, 2014 at 03:47:10PM +0400, Vasiliy Tolstov wrote:
> 2014-02-17 14:56 GMT+04:00 Daniel Kiper <dkiper@net-space.pl>:
> > They should. IIRC they were targeted for 4.3.
>
>
> Daniel, can you check this
> https://gist.github.com/vtolstov/9090413/raw/dc457631fe0b6df793552494d059eac62fd962e0/gistfile1.txt
> this is patch that merges 4 patches from email (because i can't get
> attaches and after copy/paste i have errors).
> As i see nothing changed , i'm use enforce=0 in xl.conf and start
> domain with memory=512 and maxmem=1024.

Use "mem_set_enforce_limit=0" in xl.conf instead of "enforce=0".

Daniel

Re: xen 4.3.2 rc1 memory ballooning

[Vasiliy Tolstov]

2014-02-20 23:07 GMT+04:00 Daniel Kiper <dkiper@net-space.pl>:
> Use "mem_set_enforce_limit=0" in xl.conf instead of "enforce=0".


I'm already do that, but nothing changed =(

-- 
Vasiliy Tolstov,
e-mail: v.tolstov@selfip.ru
jabber: vase@selfip.ru

Re: xen 4.3.2 rc1 memory ballooning

[Wei Liu]

On Fri, Feb 21, 2014 at 06:52:30AM +0400, Vasiliy Tolstov wrote:
> 2014-02-20 23:07 GMT+04:00 Daniel Kiper <dkiper@net-space.pl>:
> > Use "mem_set_enforce_limit=0" in xl.conf instead of "enforce=0".
> 
> 
> I'm already do that, but nothing changed =(
> 

Just to confirm, you added that in /etc/xen/xl.conf, right? My vague
memory tells me that it's a global thing. I could be wrong though...

Wei.

> -- 
> Vasiliy Tolstov,
> e-mail: v.tolstov@selfip.ru
> jabber: vase@selfip.ru

Re: xen 4.3.2 rc1 memory ballooning

[Vasiliy Tolstov]

2014-02-21 13:53 GMT+04:00 Wei Liu <wei.liu2@citrix.com>:
> Just to confirm, you added that in /etc/xen/xl.conf, right? My vague
> memory tells me that it's a global thing. I could be wrong though...


Yes,
autoballoon = 0
lockfile = "/var/lock/xl"
vifscript = "/etc/xen/scripts/vif-ospf"
mem_set_enforce_limit = 0
claim_mode = 1
vif.default.script = "/etc/xen/scripts/vif-ospf"


-- 
Vasiliy Tolstov,
e-mail: v.tolstov@selfip.ru
jabber: vase@selfip.ru

Re: xen 4.3.2 rc1 memory ballooning

[Wei Liu]

On Fri, Feb 21, 2014 at 01:59:39PM +0400, Vasiliy Tolstov wrote:
> 2014-02-21 13:53 GMT+04:00 Wei Liu <wei.liu2@citrix.com>:
> > Just to confirm, you added that in /etc/xen/xl.conf, right? My vague
> > memory tells me that it's a global thing. I could be wrong though...
> 
> 
> Yes,
> autoballoon = 0
> lockfile = "/var/lock/xl"
> vifscript = "/etc/xen/scripts/vif-ospf"
> mem_set_enforce_limit = 0
> claim_mode = 1
> vif.default.script = "/etc/xen/scripts/vif-ospf"
> 

Ah, so Daniel's change only affects "xl mem-set" command. When building a
domain the maxmem is still capped to target_memory (which is memory= in
you domain config file).

Unfortunately Daniel's patch won't help you much with that.

Wei.

> 
> -- 
> Vasiliy Tolstov,
> e-mail: v.tolstov@selfip.ru
> jabber: vase@selfip.ru

Re: xen 4.3.2 rc1 memory ballooning

[Vasiliy Tolstov]

2014-02-21 14:39 GMT+04:00 Wei Liu <wei.liu2@citrix.com>:
> Ah, so Daniel's change only affects "xl mem-set" command. When building a
> domain the maxmem is still capped to target_memory (which is memory= in
> you domain config file).
>
> Unfortunately Daniel's patch won't help you much with that


=( Very bad. How much changes needed to fix that?

-- 
Vasiliy Tolstov,
e-mail: v.tolstov@selfip.ru
jabber: vase@selfip.ru

Re: xen 4.3.2 rc1 memory ballooning

[Wei Liu]

On Fri, Feb 21, 2014 at 02:44:41PM +0400, Vasiliy Tolstov wrote:
> 2014-02-21 14:39 GMT+04:00 Wei Liu <wei.liu2@citrix.com>:
> > Ah, so Daniel's change only affects "xl mem-set" command. When building a
> > domain the maxmem is still capped to target_memory (which is memory= in
> > you domain config file).
> >
> > Unfortunately Daniel's patch won't help you much with that
> 
> 
> =( Very bad. How much changes needed to fix that?
> 

DISCLAIMER: xl is designed like that for a reason, I don't think I know
this particular reason. I just know how it is implemented. So you need
to evaluate the risk before changing it. I wouldn't claim the following
paragraphs as a "fix" to this issue.

If you don't care much about altering the behavior, the change is only
oneliner. Find libxl/libxl_dom.c:libxl__build_pre, then look for
xc_domain_setmaxmem. There's a parameter "info->target_memkb". Change it 
to "info->max_memkb".

With that change, not the cap is set to maxmem= by default when building
a domain.

Wei.

> -- 
> Vasiliy Tolstov,
> e-mail: v.tolstov@selfip.ru
> jabber: vase@selfip.ru

Re: xen 4.3.2 rc1 memory ballooning

[Vasiliy Tolstov]

2014-02-21 14:56 GMT+04:00 Wei Liu <wei.liu2@citrix.com>:
> DISCLAIMER: xl is designed like that for a reason, I don't think I know
> this particular reason. I just know how it is implemented. So you need
> to evaluate the risk before changing it. I wouldn't claim the following
> paragraphs as a "fix" to this issue.
>
> If you don't care much about altering the behavior, the change is only
> oneliner. Find libxl/libxl_dom.c:libxl__build_pre, then look for
> xc_domain_setmaxmem. There's a parameter "info->target_memkb". Change it
> to "info->max_memkb".
>
> With that change, not the cap is set to maxmem= by default when building
> a domain.


Thanks! =) May be alter current setting based on enforce param from
Daniel patch?

-- 
Vasiliy Tolstov,
e-mail: v.tolstov@selfip.ru
jabber: vase@selfip.ru

Re: xen 4.3.2 rc1 memory ballooning

[Wei Liu]

On Fri, Feb 21, 2014 at 03:30:21PM +0400, Vasiliy Tolstov wrote:
> 2014-02-21 14:56 GMT+04:00 Wei Liu <wei.liu2@citrix.com>:
> > DISCLAIMER: xl is designed like that for a reason, I don't think I know
> > this particular reason. I just know how it is implemented. So you need
> > to evaluate the risk before changing it. I wouldn't claim the following
> > paragraphs as a "fix" to this issue.
> >
> > If you don't care much about altering the behavior, the change is only
> > oneliner. Find libxl/libxl_dom.c:libxl__build_pre, then look for
> > xc_domain_setmaxmem. There's a parameter "info->target_memkb". Change it
> > to "info->max_memkb".
> >
> > With that change, not the cap is set to maxmem= by default when building
> > a domain.
> 
> 
> Thanks! =) May be alter current setting based on enforce param from
> Daniel patch?
> 

That would require a fair amount of plumbing. Plus the paramater Daniel
introduced might not fit this purpose. I don't think we should / can do
anything before reaching a concensus on the list, not without toolstack
maintainer's input. Probably something for 4.5 cycle.

Wei.

> -- 
> Vasiliy Tolstov,
> e-mail: v.tolstov@selfip.ru
> jabber: vase@selfip.ru

Re: xen 4.3.2 rc1 memory ballooning

[Vasiliy Tolstov]

2014-02-21 15:35 GMT+04:00 Wei Liu <wei.liu2@citrix.com>:
> That would require a fair amount of plumbing. Plus the paramater Daniel
> introduced might not fit this purpose. I don't think we should / can do
> anything before reaching a concensus on the list, not without toolstack
> maintainer's input. Probably something for 4.5 cycle.


Ok. Anyway - big thanks for help.

-- 
Vasiliy Tolstov,
e-mail: v.tolstov@selfip.ru
jabber: vase@selfip.ru

Re: xen 4.3.2 rc1 memory ballooning

[Daniel Kiper]

On Fri, Feb 21, 2014 at 10:39:25AM +0000, Wei Liu wrote:
> On Fri, Feb 21, 2014 at 01:59:39PM +0400, Vasiliy Tolstov wrote:
> > 2014-02-21 13:53 GMT+04:00 Wei Liu <wei.liu2@citrix.com>:
> > > Just to confirm, you added that in /etc/xen/xl.conf, right? My vague
> > > memory tells me that it's a global thing. I could be wrong though...
> >
> >
> > Yes,
> > autoballoon = 0
> > lockfile = "/var/lock/xl"
> > vifscript = "/etc/xen/scripts/vif-ospf"
> > mem_set_enforce_limit = 0
> > claim_mode = 1
> > vif.default.script = "/etc/xen/scripts/vif-ospf"
> >
>
> Ah, so Daniel's change only affects "xl mem-set" command. When building a
> domain the maxmem is still capped to target_memory (which is memory= in
> you domain config file).

Ugh... I forgot about that. I will take this into accunt in next patch
series release. Thanks.

Daniel