[Qemu-devel] Memory: how to determine the max memory size of one VM?

[Qemu-devel] Memory: how to determine the max memory size of one VM?

[Zhi Yong Wu]

HI,

Today i tried to create one VM with the option "-m 4000", and found it
failed with the following errors:

Failed to allocate 4194304000 B: Cannot allocate memory
Aborted (core dumped)

I checked the qemu code, and found that in qemu_memalign() function,
posix_memalign() failed to allocate requested memory. Perhaps due to
No enough memory.

But when i tried with "-m 3000", it can work.

So i have one question about this. How to determine the supported max
memory size of one VM on one specifc host?

Can anyone elaborate it for me? thanks.



-- 
Regards,

Zhi Yong Wu

Re: [Qemu-devel] Memory: how to determine the max memory size of one VM?

[Stefan Hajnoczi]

On Fri, Feb 10, 2012 at 9:47 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
> Today i tried to create one VM with the option "-m 4000", and found it
> failed with the following errors:
>
> Failed to allocate 4194304000 B: Cannot allocate memory
> Aborted (core dumped)

Did you run on a 32-bit host?

> I checked the qemu code, and found that in qemu_memalign() function,
> posix_memalign() failed to allocate requested memory. Perhaps due to
> No enough memory.
>
> But when i tried with "-m 3000", it can work.
>
> So i have one question about this. How to determine the supported max
> memory size of one VM on one specifc host?

Since QEMU may use host virtual memory for guest RAM the limit isn't a
hard limit that is easy to describe.  If the host OS performs memory
overcommit by default (i.e. will allow processes to map memory without
dedicating that amount of swap), then you can theoretically allocate
up to your address space limit (32-bit or 64-bit) and won't get error
until you begin dirtying more host pages than the host has resources
for.

But normally for KVM you want sum(vms) < host_ram, unless you are
betting on KSM to deduplicate guest memory.

Stefan

Re: [Qemu-devel] Memory: how to determine the max memory size of one VM?

[Stefan Hajnoczi]

On Fri, Feb 10, 2012 at 10:35 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Fri, Feb 10, 2012 at 9:47 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
>> Today i tried to create one VM with the option "-m 4000", and found it
>> failed with the following errors:
>>
>> Failed to allocate 4194304000 B: Cannot allocate memory
>> Aborted (core dumped)
>
> Did you run on a 32-bit host?

BTW we shouldn't abort(3).  If you have time please send a patch to do
a clean failure exit.

Stefan

Re: [Qemu-devel] Memory: how to determine the max memory size of one VM?

[Zhi Yong Wu]

On Fri, Feb 10, 2012 at 6:35 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Fri, Feb 10, 2012 at 9:47 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
>> Today i tried to create one VM with the option "-m 4000", and found it
>> failed with the following errors:
>>
>> Failed to allocate 4194304000 B: Cannot allocate memory
>> Aborted (core dumped)
>
> Did you run on a 32-bit host?
No, it is one x86_64 host.
>
>> I checked the qemu code, and found that in qemu_memalign() function,
>> posix_memalign() failed to allocate requested memory. Perhaps due to
>> No enough memory.
>>
>> But when i tried with "-m 3000", it can work.
>>
>> So i have one question about this. How to determine the supported max
>> memory size of one VM on one specifc host?
>
> Since QEMU may use host virtual memory for guest RAM the limit isn't a
> hard limit that is easy to describe.  If the host OS performs memory
> overcommit by default (i.e. will allow processes to map memory without
> dedicating that amount of swap), then you can theoretically allocate
> up to your address space limit (32-bit or 64-bit) and won't get error
Since the host is 64 bits, it should not fail with "-m 4000". I am
very surprised at why posix_memalign failed.

> until you begin dirtying more host pages than the host has resources
> for.
>
> But normally for KVM you want sum(vms) < host_ram, unless you are
here host_ram is the free ram of host?
> betting on KSM to deduplicate guest memory.
>
> Stefan



-- 
Regards,

Zhi Yong Wu

Re: [Qemu-devel] Memory: how to determine the max memory size of one VM?

[Stefan Hajnoczi]

On Fri, Feb 10, 2012 at 11:00 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
> On Fri, Feb 10, 2012 at 6:35 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Fri, Feb 10, 2012 at 9:47 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
>>> Today i tried to create one VM with the option "-m 4000", and found it
>>> failed with the following errors:
>>>
>>> Failed to allocate 4194304000 B: Cannot allocate memory
>>> Aborted (core dumped)
>>
>> Did you run on a 32-bit host?
> No, it is one x86_64 host.

That is weird.  Have you tried strace(1) to find out which system call
is failing and why?

>> until you begin dirtying more host pages than the host has resources
>> for.
>>
>> But normally for KVM you want sum(vms) < host_ram, unless you are
> here host_ram is the free ram of host?

I meant that for predictable, good performance it is normal to size
hosts so they have more physical RAM than the sum of guest RAM.

For example, 2 GB VM and 4 GB VM => make sure you have more than 6 GB
RAM.  If you go under that threshold you will hit swap when the guests
utilize their memory.

Stefan

Re: [Qemu-devel] Memory: how to determine the max memory size of one VM?

[Zhi Yong Wu]

On Fri, Feb 10, 2012 at 7:10 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Fri, Feb 10, 2012 at 11:00 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
>> On Fri, Feb 10, 2012 at 6:35 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>> On Fri, Feb 10, 2012 at 9:47 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
>>>> Today i tried to create one VM with the option "-m 4000", and found it
>>>> failed with the following errors:
>>>>
>>>> Failed to allocate 4194304000 B: Cannot allocate memory
>>>> Aborted (core dumped)
>>>
>>> Did you run on a 32-bit host?
>> No, it is one x86_64 host.
>
> That is weird.  Have you tried strace(1) to find out which system call
> is failing and why?
It seems that it failed to call mmap().  ENOMEM

mprotect(0x7fdd5f973000, 4096, PROT_NONE) = 0
clone(child_stack=0x7fdd60172eb0,
flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
parent_tidptr=0x7fdd601739d0, tls=0x7fdd60173700,
child_tidptr=0x7fdd601739d0) = 25488
rt_sigprocmask(SIG_SETMASK, [BUS ALRM IO RT_6], NULL, 8) = 0
select(11, [10], [], NULL, NULL)        = 1 (in [10])
read(10, "\0", 16)                      = 1
close(9)                                = 0
open("/home/zwu/work/misc/rh6.img", O_RDONLY|O_NONBLOCK) = 9
fstat(9, {st_mode=S_IFREG|0644, st_size=1396244480, ...}) = 0
close(9)                                = 0
open("/home/zwu/work/misc/rh6.img", O_RDONLY|O_NONBLOCK) = 9
fstat(9, {st_mode=S_IFREG|0644, st_size=1396244480, ...}) = 0
close(9)                                = 0
stat("/home/zwu/work/misc/rh6.img", {st_mode=S_IFREG|0644,
st_size=1396244480, ...}) = 0
open("/home/zwu/work/misc/rh6.img", O_RDWR|O_DSYNC|O_CLOEXEC) = 9
lseek(9, 0, SEEK_END)                   = 1396244480
futex(0x7fdd67c6e724, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fdd67c6e720,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
select(11, [10], [], NULL, NULL)        = 1 (in [10])
read(10, "\0", 16)                      = 1
lseek(9, 0, SEEK_END)                   = 1396244480
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x7fdd670d1000
write(7, "\1\0\0\0\0\0\0\0", 8)         = 8
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [BUS ALRM IO RT_6], 8) = 0
mmap(NULL, 8392704, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fdd5f172000
mprotect(0x7fdd5f172000, 4096, PROT_NONE) = 0
clone(child_stack=0x7fdd5f971eb0,
flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
parent_tidptr=0x7fdd5f9729d0, tls=0x7fdd5f972700,
child_tidptr=0x7fdd5f9729d0) = 25489
rt_sigprocmask(SIG_SETMASK, [BUS ALRM IO RT_6], NULL, 8) = 0
select(11, [10], [], NULL, NULL)        = 1 (in [10])
read(10, "\0", 16)                      = 1
write(7, "\1\0\0\0\0\0\0\0", 8)         = 8
brk(0)                                  = 0x7fdd68f1a000
brk(0x7fdd68f3b000)                     = 0x7fdd68f3b000
open("/proc/meminfo", O_RDONLY)         = 12
fstat(12, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7fdd67243000
read(12, "MemTotal:        2019156 kB\nMemF"..., 1024) = 1024
close(12)                               = 0
munmap(0x7fdd67243000, 4096)            = 0
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0
gettid()                                = 25462
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [BUS ALRM IO RT_6], 8) = 0
mmap(NULL, 8392704, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fdd5e971000
mprotect(0x7fdd5e971000, 4096, PROT_NONE) = 0
clone(child_stack=0x7fdd5f170eb0,
flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
parent_tidptr=0x7fdd5f1719d0, tls=0x7fdd5f171700,
child_tidptr=0x7fdd5f1719d0) = 25490
rt_sigprocmask(SIG_SETMASK, [BUS ALRM IO RT_6], NULL, 8) = 0
futex(0x7fdd67c8efc4, FUTEX_WAIT_PRIVATE, 1, NULL) = 0
mmap(NULL, 4196405248, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
brk(0)                                  = 0x7fdd68f3b000
brk(0x7fde63158000)                     = 0x7fdd68f3b000
mmap(NULL, 4196536320, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 134217728, PROT_NONE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fdd50000000
munmap(0x7fdd54000000, 67108864)        = 0
mprotect(0x7fdd50000000, 135168, PROT_READ|PROT_WRITE) = 0
mmap(NULL, 4196405248, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
write(2, "Failed to allocate 4194304000 B:"..., 56Failed to allocate
4194304000 B: Cannot allocate memory
) = 56
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
tgkill(25462, 25462, SIGABRT)           = 0
--- {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=25462, si_uid=0,
si_value={int=1667424088, ptr=0xffff88006362df58}} (Aborted) ---
+++ killed by SIGABRT (core dumped) +++
Aborted (core dumped)

>
>>> until you begin dirtying more host pages than the host has resources
>>> for.
>>>
>>> But normally for KVM you want sum(vms) < host_ram, unless you are
>> here host_ram is the free ram of host?
>
> I meant that for predictable, good performance it is normal to size
> hosts so they have more physical RAM than the sum of guest RAM.
>
> For example, 2 GB VM and 4 GB VM => make sure you have more than 6 GB
> RAM.  If you go under that threshold you will hit swap when the guests
> utilize their memory.
>
> Stefan



-- 
Regards,

Zhi Yong Wu

Re: [Qemu-devel] Memory: how to determine the max memory size of one VM?

[Stefan Hajnoczi]

On Fri, Feb 10, 2012 at 11:23 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
> On Fri, Feb 10, 2012 at 7:10 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Fri, Feb 10, 2012 at 11:00 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
>>> On Fri, Feb 10, 2012 at 6:35 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>>> On Fri, Feb 10, 2012 at 9:47 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
>>>>> Today i tried to create one VM with the option "-m 4000", and found it
>>>>> failed with the following errors:
>>>>>
>>>>> Failed to allocate 4194304000 B: Cannot allocate memory
>>>>> Aborted (core dumped)
>>>>
>>>> Did you run on a 32-bit host?
>>> No, it is one x86_64 host.
>>
>> That is weird.  Have you tried strace(1) to find out which system call
>> is failing and why?
> It seems that it failed to call mmap().  ENOMEM
>
> mprotect(0x7fdd5f973000, 4096, PROT_NONE) = 0
> clone(child_stack=0x7fdd60172eb0,
> flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
> parent_tidptr=0x7fdd601739d0, tls=0x7fdd60173700,
> child_tidptr=0x7fdd601739d0) = 25488
> rt_sigprocmask(SIG_SETMASK, [BUS ALRM IO RT_6], NULL, 8) = 0
> select(11, [10], [], NULL, NULL)        = 1 (in [10])
> read(10, "\0", 16)                      = 1
> close(9)                                = 0
> open("/home/zwu/work/misc/rh6.img", O_RDONLY|O_NONBLOCK) = 9
> fstat(9, {st_mode=S_IFREG|0644, st_size=1396244480, ...}) = 0
> close(9)                                = 0
> open("/home/zwu/work/misc/rh6.img", O_RDONLY|O_NONBLOCK) = 9
> fstat(9, {st_mode=S_IFREG|0644, st_size=1396244480, ...}) = 0
> close(9)                                = 0
> stat("/home/zwu/work/misc/rh6.img", {st_mode=S_IFREG|0644,
> st_size=1396244480, ...}) = 0
> open("/home/zwu/work/misc/rh6.img", O_RDWR|O_DSYNC|O_CLOEXEC) = 9
> lseek(9, 0, SEEK_END)                   = 1396244480
> futex(0x7fdd67c6e724, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fdd67c6e720,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> select(11, [10], [], NULL, NULL)        = 1 (in [10])
> read(10, "\0", 16)                      = 1
> lseek(9, 0, SEEK_END)                   = 1396244480
> mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
> -1, 0) = 0x7fdd670d1000
> write(7, "\1\0\0\0\0\0\0\0", 8)         = 8
> rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [BUS ALRM IO RT_6], 8) = 0
> mmap(NULL, 8392704, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fdd5f172000
> mprotect(0x7fdd5f172000, 4096, PROT_NONE) = 0
> clone(child_stack=0x7fdd5f971eb0,
> flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
> parent_tidptr=0x7fdd5f9729d0, tls=0x7fdd5f972700,
> child_tidptr=0x7fdd5f9729d0) = 25489
> rt_sigprocmask(SIG_SETMASK, [BUS ALRM IO RT_6], NULL, 8) = 0
> select(11, [10], [], NULL, NULL)        = 1 (in [10])
> read(10, "\0", 16)                      = 1
> write(7, "\1\0\0\0\0\0\0\0", 8)         = 8
> brk(0)                                  = 0x7fdd68f1a000
> brk(0x7fdd68f3b000)                     = 0x7fdd68f3b000
> open("/proc/meminfo", O_RDONLY)         = 12
> fstat(12, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0x7fdd67243000
> read(12, "MemTotal:        2019156 kB\nMemF"..., 1024) = 1024
> close(12)                               = 0
> munmap(0x7fdd67243000, 4096)            = 0
> getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0
> gettid()                                = 25462
> rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [BUS ALRM IO RT_6], 8) = 0
> mmap(NULL, 8392704, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fdd5e971000
> mprotect(0x7fdd5e971000, 4096, PROT_NONE) = 0
> clone(child_stack=0x7fdd5f170eb0,
> flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
> parent_tidptr=0x7fdd5f1719d0, tls=0x7fdd5f171700,
> child_tidptr=0x7fdd5f1719d0) = 25490
> rt_sigprocmask(SIG_SETMASK, [BUS ALRM IO RT_6], NULL, 8) = 0
> futex(0x7fdd67c8efc4, FUTEX_WAIT_PRIVATE, 1, NULL) = 0
> mmap(NULL, 4196405248, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
> brk(0)                                  = 0x7fdd68f3b000
> brk(0x7fde63158000)                     = 0x7fdd68f3b000
> mmap(NULL, 4196536320, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
> mmap(NULL, 134217728, PROT_NONE,
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fdd50000000
> munmap(0x7fdd54000000, 67108864)        = 0
> mprotect(0x7fdd50000000, 135168, PROT_READ|PROT_WRITE) = 0
> mmap(NULL, 4196405248, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)

I suggest checking the following:
$ free -m
$ cat /proc/sys/vm/overcommit_memory

See also http://www.kernel.org/doc/Documentation/sysctl/vm.txt.

Stefan

Re: [Qemu-devel] Memory: how to determine the max memory size of one VM?

[Zhi Yong Wu]

On Fri, Feb 10, 2012 at 7:31 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Fri, Feb 10, 2012 at 11:23 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
>> On Fri, Feb 10, 2012 at 7:10 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>> On Fri, Feb 10, 2012 at 11:00 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
>>>> On Fri, Feb 10, 2012 at 6:35 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>>>> On Fri, Feb 10, 2012 at 9:47 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
>>>>>> Today i tried to create one VM with the option "-m 4000", and found it
>>>>>> failed with the following errors:
>>>>>>
>>>>>> Failed to allocate 4194304000 B: Cannot allocate memory
>>>>>> Aborted (core dumped)
>>>>>
>>>>> Did you run on a 32-bit host?
>>>> No, it is one x86_64 host.
>>>
>>> That is weird.  Have you tried strace(1) to find out which system call
>>> is failing and why?
>> It seems that it failed to call mmap().  ENOMEM
>>
>> mprotect(0x7fdd5f973000, 4096, PROT_NONE) = 0
>> clone(child_stack=0x7fdd60172eb0,
>> flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
>> parent_tidptr=0x7fdd601739d0, tls=0x7fdd60173700,
>> child_tidptr=0x7fdd601739d0) = 25488
>> rt_sigprocmask(SIG_SETMASK, [BUS ALRM IO RT_6], NULL, 8) = 0
>> select(11, [10], [], NULL, NULL)        = 1 (in [10])
>> read(10, "\0", 16)                      = 1
>> close(9)                                = 0
>> open("/home/zwu/work/misc/rh6.img", O_RDONLY|O_NONBLOCK) = 9
>> fstat(9, {st_mode=S_IFREG|0644, st_size=1396244480, ...}) = 0
>> close(9)                                = 0
>> open("/home/zwu/work/misc/rh6.img", O_RDONLY|O_NONBLOCK) = 9
>> fstat(9, {st_mode=S_IFREG|0644, st_size=1396244480, ...}) = 0
>> close(9)                                = 0
>> stat("/home/zwu/work/misc/rh6.img", {st_mode=S_IFREG|0644,
>> st_size=1396244480, ...}) = 0
>> open("/home/zwu/work/misc/rh6.img", O_RDWR|O_DSYNC|O_CLOEXEC) = 9
>> lseek(9, 0, SEEK_END)                   = 1396244480
>> futex(0x7fdd67c6e724, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fdd67c6e720,
>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
>> select(11, [10], [], NULL, NULL)        = 1 (in [10])
>> read(10, "\0", 16)                      = 1
>> lseek(9, 0, SEEK_END)                   = 1396244480
>> mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
>> -1, 0) = 0x7fdd670d1000
>> write(7, "\1\0\0\0\0\0\0\0", 8)         = 8
>> rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [BUS ALRM IO RT_6], 8) = 0
>> mmap(NULL, 8392704, PROT_READ|PROT_WRITE,
>> MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fdd5f172000
>> mprotect(0x7fdd5f172000, 4096, PROT_NONE) = 0
>> clone(child_stack=0x7fdd5f971eb0,
>> flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
>> parent_tidptr=0x7fdd5f9729d0, tls=0x7fdd5f972700,
>> child_tidptr=0x7fdd5f9729d0) = 25489
>> rt_sigprocmask(SIG_SETMASK, [BUS ALRM IO RT_6], NULL, 8) = 0
>> select(11, [10], [], NULL, NULL)        = 1 (in [10])
>> read(10, "\0", 16)                      = 1
>> write(7, "\1\0\0\0\0\0\0\0", 8)         = 8
>> brk(0)                                  = 0x7fdd68f1a000
>> brk(0x7fdd68f3b000)                     = 0x7fdd68f3b000
>> open("/proc/meminfo", O_RDONLY)         = 12
>> fstat(12, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
>> 0) = 0x7fdd67243000
>> read(12, "MemTotal:        2019156 kB\nMemF"..., 1024) = 1024
>> close(12)                               = 0
>> munmap(0x7fdd67243000, 4096)            = 0
>> getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0
>> gettid()                                = 25462
>> rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [BUS ALRM IO RT_6], 8) = 0
>> mmap(NULL, 8392704, PROT_READ|PROT_WRITE,
>> MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fdd5e971000
>> mprotect(0x7fdd5e971000, 4096, PROT_NONE) = 0
>> clone(child_stack=0x7fdd5f170eb0,
>> flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
>> parent_tidptr=0x7fdd5f1719d0, tls=0x7fdd5f171700,
>> child_tidptr=0x7fdd5f1719d0) = 25490
>> rt_sigprocmask(SIG_SETMASK, [BUS ALRM IO RT_6], NULL, 8) = 0
>> futex(0x7fdd67c8efc4, FUTEX_WAIT_PRIVATE, 1, NULL) = 0
>> mmap(NULL, 4196405248, PROT_READ|PROT_WRITE,
>> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
>> brk(0)                                  = 0x7fdd68f3b000
>> brk(0x7fde63158000)                     = 0x7fdd68f3b000
>> mmap(NULL, 4196536320, PROT_READ|PROT_WRITE,
>> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
>> mmap(NULL, 134217728, PROT_NONE,
>> MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fdd50000000
>> munmap(0x7fdd54000000, 67108864)        = 0
>> mprotect(0x7fdd50000000, 135168, PROT_READ|PROT_WRITE) = 0
>> mmap(NULL, 4196405248, PROT_READ|PROT_WRITE,
>> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
>
> I suggest checking the following:
> $ free -m
Actually i had checked this, the host only has <200m free memory.

> $ cat /proc/sys/vm/overcommit_memory
>
> See also http://www.kernel.org/doc/Documentation/sysctl/vm.txt.
After i change it to 1, qemu can work now. thanks.

By the way, need we still to do one clean failure exit?

>
> Stefan



-- 
Regards,

Zhi Yong Wu

Re: [Qemu-devel] Memory: how to determine the max memory size of one VM?

[Stefan Hajnoczi]

On Fri, Feb 10, 2012 at 11:53 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
> By the way, need we still to do one clean failure exit?

I find the abort(3) usage odd but it's not critical to change it.
Since we printed an error message I would have expected
exit(EXIT_FAILURE), not abort(3) which is usually used for strange
cases that you want to debug if hit.

Stefan

Re: [Qemu-devel] Memory: how to determine the max memory size of one VM?

[Zhi Yong Wu]

On Fri, Feb 10, 2012 at 10:08 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Fri, Feb 10, 2012 at 11:53 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
>> By the way, need we still to do one clean failure exit?
>
> I find the abort(3) usage odd but it's not critical to change it.
> Since we printed an error message I would have expected
> exit(EXIT_FAILURE), not abort(3) which is usually used for strange
> cases that you want to debug if hit.
OK. i will send out one patch since it is on hand. thanks

>
> Stefan



-- 
Regards,

Zhi Yong Wu

Re: [Qemu-devel] Memory: how to determine the max memory size of one VM?

[Andreas Färber]

Am 10.02.2012 12:23, schrieb Zhi Yong Wu:
> On Fri, Feb 10, 2012 at 7:10 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Fri, Feb 10, 2012 at 11:00 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
>>> On Fri, Feb 10, 2012 at 6:35 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>>> On Fri, Feb 10, 2012 at 9:47 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
>>>>> Today i tried to create one VM with the option "-m 4000", and found it
>>>>> failed with the following errors:
>>>>>
>>>>> Failed to allocate 4194304000 B: Cannot allocate memory
>>>>> Aborted (core dumped)
>>>>
>>>> Did you run on a 32-bit host?
>>> No, it is one x86_64 host.
>>
>> That is weird.  Have you tried strace(1) to find out which system call
>> is failing and why?
> It seems that it failed to call mmap().  ENOMEM

/etc/sysconfig/ulimit on SUSE has a default SOFTVIRTUALLIMIT value of
"80", i.e. 80% of the available physical + swap memory can be allocated
by one process.
For our SLES use cases we applied a simple patch to raise this limit via
RLIMIT_AS (so that, e.g., a 24 GB RAM host can create VMs > -m 20G).

ulimit -Sv can change this value, too.

For upstream QEMU we felt setting proper limits is the responsibility of
a management tool.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] Memory: how to determine the max memory size of one VM?

[Andreas Färber]

Am 10.02.2012 11:36, schrieb Stefan Hajnoczi:
> On Fri, Feb 10, 2012 at 10:35 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Fri, Feb 10, 2012 at 9:47 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
>>> Today i tried to create one VM with the option "-m 4000", and found it
>>> failed with the following errors:
>>>
>>> Failed to allocate 4194304000 B: Cannot allocate memory
>>> Aborted (core dumped)
>>
>> Did you run on a 32-bit host?
> 
> BTW we shouldn't abort(3).

There's two slightly different scenarios to consider here:

i) User specifies command line options that cannot possibly work.
=> Ideally, yeah, we should just provide an understandable error message
and exit with error code.

ii) Some tracing of mine indicates QEMU has a highly dynamic memory
usage during runtime, be it due to network layer, block layer or
whatever exactly. Any memory allocation of those may fail due to soft or
hard limits, including pthread_create.
=> We should not abort because the previously running guest is gone and
it's data may be incomplete or corrupted. Not to mention that libvirt
shows an odd error message.

Problem is that there is no distinction at this point, so if we remove
the abort(3) for use case i) we loose it for debugging ii) as well.

What would be cool is finding some way to avoid dynamic allocations
after guest startup (i.e., preallocated memory pools) and dealing with
running out of space there in a non-fatal way.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] Memory: how to determine the max memory size of one VM?

[Paul Brook]

> There's two slightly different scenarios to consider here:
> 
> i) User specifies command line options that cannot possibly work.
> => Ideally, yeah, we should just provide an understandable error message
> and exit with error code.
> 
> ii) Some tracing of mine indicates QEMU has a highly dynamic memory
> usage during runtime, be it due to network layer, block layer or
> whatever exactly.

We do? Significant compared to the size of guest ram?  That sounds like a bug.
I know I've nacked patches before (particularly SCSI ones) because they 
allocarted memory proportional to the size of the guest block device request.

Paul

Re: [Qemu-devel] Memory: how to determine the max memory size of one VM?

[Andreas Färber]

[-- Attachment #1: Type: text/plain, Size: 1297 bytes --]

Am 11.02.2012 00:25, schrieb Paul Brook:
>> ii) Some tracing of mine indicates QEMU has a highly dynamic memory
>> usage during runtime, be it due to network layer, block layer or
>> whatever exactly.
> 
> We do? Significant compared to the size of guest ram?  That sounds like a bug.

Attached is a gnuplot from a simpletrace trace file while installing a
SLES 11 SP2 Release Candidate over slirp to virtio with -m 8G on an 8
GiB host (post-1.0 master).

It's not fully scientifically correct (it doesn't take into account
memory allocations not traced by QEMU itself, like pthreads) but it
suggests that after the initial surge to ~8.7 GB we have a fluctuation
of ~0.2 GB for 8 GiB guest RAM. It's larger than I'd expect at least.

Since the user was close to the limit, this lead to an abort. In their
case it was a pthread_create() that failed, and we used tap + virtio.

Didn't investigate the exact origin of the fluctuations (yet), but the
combination of the choice to have virtually all allocations abort (which
is okay at startup) and of doing allocations during guest runtime turns
out Bad(tm) in practice.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

[-- Attachment #2: memorytrace.png --]
[-- Type: image/png, Size: 15075 bytes --]

Re: [Qemu-devel] Memory: how to determine the max memory size of one VM?

[Paul Brook]

> Am 11.02.2012 00:25, schrieb Paul Brook:
> >> ii) Some tracing of mine indicates QEMU has a highly dynamic memory
> >> usage during runtime, be it due to network layer, block layer or
> >> whatever exactly.
> > 
> > We do? Significant compared to the size of guest ram?  That sounds like a
> > bug.
> 
> Attached is a gnuplot from a simpletrace trace file while installing a
> SLES 11 SP2 Release Candidate over slirp to virtio with -m 8G on an 8
> GiB host (post-1.0 master).
> 
> It's not fully scientifically correct (it doesn't take into account
> memory allocations not traced by QEMU itself, like pthreads) but it
> suggests that after the initial surge to ~8.7 GB we have a fluctuation
> of ~0.2 GB for 8 GiB guest RAM. It's larger than I'd expect at least.
> 
> Since the user was close to the limit, this lead to an abort. In their
> case it was a pthread_create() that failed, and we used tap + virtio.

Hmm, yes, we're clearly allocating some very large buffers somewhere.

Paul