* Swap usage with KVM
@ 2010-07-11 15:12 Daniel Bareiro
2010-07-11 19:12 ` Daniel Bareiro
2010-07-11 21:05 ` Swap usage with KVM Freddie Cash
0 siblings, 2 replies; 15+ messages in thread
From: Daniel Bareiro @ 2010-07-11 15:12 UTC (permalink / raw)
To: KVM General
[-- Attachment #1: Type: text/plain, Size: 5390 bytes --]
Hi all!
I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm
0.12.3 compiled with the source code obtained from the official site of
KVM and Linux 2.6.32.12 compiled from source code of kernel.org. All
this is installed on an HP Proliant DL380 G6 with two Xeon E5530
quadcore processors and 16 GiB of RAM which has two VMs with the
following configuration of memory:
Hostname | RAM
===============+===============
Aps4 | 7 GiB
Leela | 7 GiB
===============+===============
TOTAL | 14 GiB
Initially the host was created with a swap partition of 1 GiB, but today
we found that the use of swap quickly began to grow increasingly.
Therefore, as a contingency, we had to hot-add a logical volume of 1 GB
of swap on the VMHost. Is 'normal' this use of memory?
I copy the Nagios Service Log Entries for the VMHost:
Event Start Time Event End Time Event Duration Event/State Type Event/State
Information
---------------------------------------------------------------------------------------------------
06-07-2010 00:00:00 07-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
100% free (956 MB out of 956 MB)
07-07-2010 00:00:00 08-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
100% free (956 MB out of 956 MB)
08-07-2010 00:00:00 08-07-2010 16:41:43 0d 16h 41m 43s SERVICE OK (HARD) SWAP OK -
100% free (956 MB out of 956 MB)
09-07-2010 00:00:00 10-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
99% free (939 MB out of 956 MB)
10-07-2010 00:00:00 11-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
79% free (754 MB out of 956 MB)
11-07-2010 00:00:00 11-07-2010 07:08:17 0d 7h 8m 17s SERVICE OK (HARD) SWAP OK -
51% free (482 MB out of 956 MB)
11-07-2010 07:08:17 11-07-2010 10:41:07 0d 3h 32m 50s SERVICE WARNING (HARD) SWAP WARNING
- 29% free (272 MB out of 956 MB)
11-07-2010 10:41:07 11-07-2010 10:45:57 0d 0h 4m 50s SERVICE CRITICAL (HARD) SWAP
CRITICAL - 9% free (83 MB out of 956 MB)
---------------------------------------------------------------------------------------------------
I'm not using qcow2 files. The /dev/cciss/c0d0p3 partition is a physical
volume that maintains the logical volumes that are used for VM's disks.
The Nagios Service Log Entries for VMs shows no excessive use of swap in
the window of time when the problem occurred in the VMHost:
Aps4:
Event Start Time Event End Time Event Duration Event/State Type Event/State
Information
---------------------------------------------------------------------------------------------------
06-07-2010 00:00:00 07-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
100% free (2850 MB out of 2863 MB)
07-07-2010 00:00:00 08-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
98% free (2797 MB out of 2863 MB)
08-07-2010 00:00:00 08-07-2010 16:41:43 0d 16h 41m 43s SERVICE OK (HARD) SWAP OK -
99% free (2812 MB out of 2863 MB)
09-07-2010 00:00:00 10-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
98% free (2784 MB out of 2863 MB)
10-07-2010 00:00:00 11-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
97% free (2754 MB out of 2863 MB)
11-07-2010 00:00:00 11-07-2010 11:53:38 0d 11h 53m 38s+ SERVICE OK (HARD) SWAP OK -
100% free (2839 MB out of 2863 MB)
---------------------------------------------------------------------------------------------------
Leela:
Event Start Time Event End Time Event Duration Event/State Type Event/State
Information
---------------------------------------------------------------------------------------------------
06-07-2010 00:00:00 07-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
97% free (919 MB out of 956 MB)
07-07-2010 00:00:00 08-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
97% free (920 MB out of 956 MB)
08-07-2010 00:00:00 08-07-2010 16:41:43 0d 16h 41m 43s SERVICE OK (HARD) SWAP OK -
97% free (920 MB out of 956 MB)
08-07-2010 17:01:36 08-07-2010 17:04:16 0d 0h 2m 40s SERVICE CRITICAL (HARD) Connection
refused by host
08-07-2010 17:04:16 09-07-2010 00:00:00 0d 6h 55m 44s SERVICE OK (HARD) SWAP OK -
97% free (921 MB out of 956 MB)
09-07-2010 00:00:00 10-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
97% free (921 MB out of 956 MB)
10-07-2010 00:00:00 11-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
97% free (921 MB out of 956 MB)
11-07-2010 00:00:00 11-07-2010 11:58:29 0d 11h 58m 29s+ SERVICE OK (HARD) SWAP OK -
97% free (921 MB out of 956 MB)
---------------------------------------------------------------------------------------------------
Unfortunately I could not take much more data because we had to act
quickly, but if you need any additional information, please feel free to
ask.
Thanks in advance for your replies.
Regards,
Daniel
--
Fingerprint: BFB3 08D6 B4D1 31B2 72B9 29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Swap usage with KVM
2010-07-11 15:12 Swap usage with KVM Daniel Bareiro
@ 2010-07-11 19:12 ` Daniel Bareiro
2010-07-11 21:49 ` Rik van Riel
2010-07-11 21:05 ` Swap usage with KVM Freddie Cash
1 sibling, 1 reply; 15+ messages in thread
From: Daniel Bareiro @ 2010-07-11 19:12 UTC (permalink / raw)
To: KVM General; +Cc: hugh.dickins, Rik van Riel
[-- Attachment #1: Type: text/plain, Size: 5820 bytes --]
On Sunday, 11 July 2010 12:12:57 -0300,
Daniel Bareiro wrote:
> I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm
> 0.12.3 compiled with the source code obtained from the official site
> of KVM and Linux 2.6.32.12 compiled from source code of kernel.org.
> All this is installed on an HP Proliant DL380 G6 with two Xeon E5530
> quadcore processors and 16 GiB of RAM which has two VMs with the
> following configuration of memory:
>
> Hostname | RAM
> ===============+===============
> Aps4 | 7 GiB
> Leela | 7 GiB
> ===============+===============
> TOTAL | 14 GiB
>
> Initially the host was created with a swap partition of 1 GiB, but
> today we found that the use of swap quickly began to grow
> increasingly. Therefore, as a contingency, we had to hot-add a
> logical volume of 1 GB of swap on the VMHost. Is 'normal' this use of
> memory?
>
> I copy the Nagios Service Log Entries for the VMHost:
>
> Event Start Time Event End Time Event Duration Event/State Type Event/State
> Information
> ---------------------------------------------------------------------------------------------------
> 06-07-2010 00:00:00 07-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
> 100% free (956 MB out of 956 MB)
> 07-07-2010 00:00:00 08-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
> 100% free (956 MB out of 956 MB)
> 08-07-2010 00:00:00 08-07-2010 16:41:43 0d 16h 41m 43s SERVICE OK (HARD) SWAP OK -
> 100% free (956 MB out of 956 MB)
> 09-07-2010 00:00:00 10-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
> 99% free (939 MB out of 956 MB)
> 10-07-2010 00:00:00 11-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
> 79% free (754 MB out of 956 MB)
> 11-07-2010 00:00:00 11-07-2010 07:08:17 0d 7h 8m 17s SERVICE OK (HARD) SWAP OK -
> 51% free (482 MB out of 956 MB)
> 11-07-2010 07:08:17 11-07-2010 10:41:07 0d 3h 32m 50s SERVICE WARNING (HARD) SWAP WARNING
> - 29% free (272 MB out of 956 MB)
> 11-07-2010 10:41:07 11-07-2010 10:45:57 0d 0h 4m 50s SERVICE CRITICAL (HARD) SWAP
> CRITICAL - 9% free (83 MB out of 956 MB)
> ---------------------------------------------------------------------------------------------------
>
> I'm not using qcow2 files. The /dev/cciss/c0d0p3 partition is a
> physical volume that maintains the logical volumes that are used for
> VM's disks.
>
> The Nagios Service Log Entries for VMs shows no excessive use of swap
> in the window of time when the problem occurred in the VMHost:
>
> Aps4:
>
> Event Start Time Event End Time Event Duration Event/State Type Event/State
> Information
> ---------------------------------------------------------------------------------------------------
> 06-07-2010 00:00:00 07-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
> 100% free (2850 MB out of 2863 MB)
> 07-07-2010 00:00:00 08-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
> 98% free (2797 MB out of 2863 MB)
> 08-07-2010 00:00:00 08-07-2010 16:41:43 0d 16h 41m 43s SERVICE OK (HARD) SWAP OK -
> 99% free (2812 MB out of 2863 MB)
> 09-07-2010 00:00:00 10-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
> 98% free (2784 MB out of 2863 MB)
> 10-07-2010 00:00:00 11-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
> 97% free (2754 MB out of 2863 MB)
> 11-07-2010 00:00:00 11-07-2010 11:53:38 0d 11h 53m 38s+ SERVICE OK (HARD) SWAP OK -
> 100% free (2839 MB out of 2863 MB)
> ---------------------------------------------------------------------------------------------------
>
> Leela:
>
> Event Start Time Event End Time Event Duration Event/State Type Event/State
> Information
> ---------------------------------------------------------------------------------------------------
> 06-07-2010 00:00:00 07-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
> 97% free (919 MB out of 956 MB)
> 07-07-2010 00:00:00 08-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
> 97% free (920 MB out of 956 MB)
> 08-07-2010 00:00:00 08-07-2010 16:41:43 0d 16h 41m 43s SERVICE OK (HARD) SWAP OK -
> 97% free (920 MB out of 956 MB)
> 08-07-2010 17:01:36 08-07-2010 17:04:16 0d 0h 2m 40s SERVICE CRITICAL (HARD) Connection
> refused by host
> 08-07-2010 17:04:16 09-07-2010 00:00:00 0d 6h 55m 44s SERVICE OK (HARD) SWAP OK -
> 97% free (921 MB out of 956 MB)
> 09-07-2010 00:00:00 10-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
> 97% free (921 MB out of 956 MB)
> 10-07-2010 00:00:00 11-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
> 97% free (921 MB out of 956 MB)
> 11-07-2010 00:00:00 11-07-2010 11:58:29 0d 11h 58m 29s+ SERVICE OK (HARD) SWAP OK -
> 97% free (921 MB out of 956 MB)
> ---------------------------------------------------------------------------------------------------
>
>
>
> Unfortunately I could not take much more data because we had to act
> quickly, but if you need any additional information, please feel free to
> ask.
Has anyone experienced something like this? Avi? I remember late last
year there was a regression in Linux swapping and Rik and Hugh were
working on it. Are you aware of any?
Thanks in advance for your reply.
Regards,
Daniel
--
Fingerprint: BFB3 08D6 B4D1 31B2 72B9 29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Swap usage with KVM
2010-07-11 15:12 Swap usage with KVM Daniel Bareiro
2010-07-11 19:12 ` Daniel Bareiro
@ 2010-07-11 21:05 ` Freddie Cash
2010-07-11 22:08 ` Daniel Bareiro
1 sibling, 1 reply; 15+ messages in thread
From: Freddie Cash @ 2010-07-11 21:05 UTC (permalink / raw)
To: dbareiro, KVM General
On Sun, Jul 11, 2010 at 8:12 AM, Daniel Bareiro <daniel-listas@gmx.net> wrote:
> I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm
> 0.12.3 compiled with the source code obtained from the official site of
> KVM and Linux 2.6.32.12 compiled from source code of kernel.org. All
> this is installed on an HP Proliant DL380 G6 with two Xeon E5530
> quadcore processors and 16 GiB of RAM which has two VMs with the
> following configuration of memory:
Are you using virtio drivers in the VMs?
There was an issue with KVM-72 and virtio that leaks memory in the
host until all RAM and swap is used (inside the VMs, no swap is used).
It was supposed to be fixed in KVM-80-something, though.
Perhaps something similar is happening again? If you switch the disks
to scsi instead of virtio, does the problem go away?
We are running KVM-72 on Debian 5.0 and have run into this issue.
We'll be upgrading our hosts this month to fix this.
--
Freddie Cash
fjwcash@gmail.com
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Swap usage with KVM
2010-07-11 19:12 ` Daniel Bareiro
@ 2010-07-11 21:49 ` Rik van Riel
2010-08-02 18:57 ` Daniel Bareiro
0 siblings, 1 reply; 15+ messages in thread
From: Rik van Riel @ 2010-07-11 21:49 UTC (permalink / raw)
To: dbareiro, KVM General, hugh.dickins
On 07/11/2010 03:12 PM, Daniel Bareiro wrote:
> On Sunday, 11 July 2010 12:12:57 -0300,
> Daniel Bareiro wrote:
>
>> I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm
>> 0.12.3 compiled with the source code obtained from the official site
>> of KVM and Linux 2.6.32.12 compiled from source code of kernel.org.
>> All this is installed on an HP Proliant DL380 G6 with two Xeon E5530
>> quadcore processors and 16 GiB of RAM which has two VMs with the
>> following configuration of memory:
>>
>> Hostname | RAM
>> ===============+===============
>> Aps4 | 7 GiB
>> Leela | 7 GiB
>> ===============+===============
>> TOTAL | 14 GiB
>>
>> Initially the host was created with a swap partition of 1 GiB, but
>> today we found that the use of swap quickly began to grow
>> increasingly. Therefore, as a contingency, we had to hot-add a
>> logical volume of 1 GB of swap on the VMHost. Is 'normal' this use of
>> memory?
That depends on what is going on in the host.
Did you notice any performance issues in the guest when
you started using swap?
> Has anyone experienced something like this? Avi? I remember late last
> year there was a regression in Linux swapping and Rik and Hugh were
> working on it. Are you aware of any?
That one should have been fixed.
--
All rights reversed
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Swap usage with KVM
2010-07-11 21:05 ` Swap usage with KVM Freddie Cash
@ 2010-07-11 22:08 ` Daniel Bareiro
2010-07-20 15:04 ` Daniel Bareiro
2010-07-20 19:18 ` David Weber
0 siblings, 2 replies; 15+ messages in thread
From: Daniel Bareiro @ 2010-07-11 22:08 UTC (permalink / raw)
To: KVM General
[-- Attachment #1: Type: text/plain, Size: 3665 bytes --]
Hi, Freddie.
On Sunday, 11 July 2010 14:05:38 -0700,
Freddie Cash wrote:
> > I have an installation with Debian GNU/Linux 5.0.4 amd64 with
> > qemu-kvm 0.12.3 compiled with the source code obtained from the
> > official site of KVM and Linux 2.6.32.12 compiled from source code
> > of kernel.org. All this is installed on an HP Proliant DL380 G6 with
> > two Xeon E5530 quadcore processors and 16 GiB of RAM which has two
> > VMs with the following configuration of memory:
> Are you using virtio drivers in the VMs?
>
> There was an issue with KVM-72 and virtio that leaks memory in the
> host until all RAM and swap is used (inside the VMs, no swap is used).
> It was supposed to be fixed in KVM-80-something, though.
>
> Perhaps something similar is happening again? If you switch the disks
> to scsi instead of virtio, does the problem go away?
>
> We are running KVM-72 on Debian 5.0 and have run into this issue.
> We'll be upgrading our hosts this month to fix this.
Yes, we are using Virtio drivers for networking and storage in both VMs
with cache=none. Both VMs are running Linux 2.6.32-bpo.5-amd64 from
Lenny Backports repositories. For VMHost, we are using a stable version
of KVM with Linux 2.6.32.12 compiled from source code of kernel.org and
qemu-kvm 0.12.3 compiled with the source code obtained from the official
site of KVM.
This is the syntax I'm using to boot the virtual machines:
8587 ? Sl 6515:25 /usr/local/qemu-kvm/bin/qemu-system-x86_64 -drive
file=/dev/vm/aps4-raiz,cache=none,if=virtio,boot=on -drive
file=/dev/vm/aps4-cache,cache=none,if=virtio -drive file=/dev/vm/aps4-index,cache=none,if=virtio
-drive file=/dev/vm/aps4-space,cache=none,if=virtio -m 7168 -smp 4 -net
nic,model=virtio,macaddr=00:16:3e:00:00:95 -net tap -daemonize -vnc :3 -k es -localtime -monitor
telnet:localhost:4003,server,nowait -serial telnet:localhost:4043,server,nowait
9769 ? Rl 11968:47 /usr/local/qemu-kvm/bin/qemu-system-x86_64 -drive
file=/dev/vm/leela-raiz,cache=none,if=virtio,boot=on -drive
file=/dev/vm/leela-u01,cache=none,if=virtio -drive file=/dev/vm/leela-u02,cache=none,if=virtio
-drive file=/dev/vm/leela-u03,cache=none,if=virtio -drive
file=/dev/vm/leela-u04,cache=none,if=virtio -drive file=/dev/vm/leela-u05,cache=none,if=virtio
-drive file=/dev/vm/leela-u06,cache=none,if=virtio -drive
file=/dev/vm/leela-u07,cache=none,if=virtio -drive file=/dev/vm/leela-u08,cache=none,if=virtio
-drive file=/dev/vm/leela-u09,cache=none,if=virtio -drive
file=/dev/vm/leela-space,cache=none,if=virtio -m 7168 -smp 8 -net
nic,model=virtio,macaddr=00:16:3e:00:00:96 -net tap -daemonize -vnc :4 -k es -localtime -monitor
telnet:localhost:4004,server,nowait -serial telnet:localhost:4044,server,nowait
To make the switch from Virtio to SCSI I would have to shut down the
hosts, which would not be a good idea whereas are two productive
systems. At least, before doing so I would be sure of what might be the
problem.
Taking a current measurement in VMHost with free, I got the following:
ss04:~# free
total used free shared buffers cached
Mem: 16461588 16406504 55084 0 2920 21504
-/+ buffers/cache: 16382080 79508
Swap: 2028492 983140 1045352
It draws attention to me that thinking about initially leaving a margin
of 2 GB of RAM for the VMHost, already it has used almost half of swap.
Thanks for your reply.
Regards,
Daniel
--
Fingerprint: BFB3 08D6 B4D1 31B2 72B9 29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Swap usage with KVM
2010-07-11 22:08 ` Daniel Bareiro
@ 2010-07-20 15:04 ` Daniel Bareiro
2010-07-20 19:18 ` David Weber
1 sibling, 0 replies; 15+ messages in thread
From: Daniel Bareiro @ 2010-07-20 15:04 UTC (permalink / raw)
To: KVM General; +Cc: dbareiro
[-- Attachment #1: Type: text/plain, Size: 4755 bytes --]
On Sunday, 11 July 2010 19:08:58 -0300,
Daniel Bareiro wrote:
> > > I have an installation with Debian GNU/Linux 5.0.4 amd64 with
> > > qemu-kvm 0.12.3 compiled with the source code obtained from the
> > > official site of KVM and Linux 2.6.32.12 compiled from source code
> > > of kernel.org. All this is installed on an HP Proliant DL380 G6
> > > with two Xeon E5530 quadcore processors and 16 GiB of RAM which
> > > has two VMs with the following configuration of memory:
> > Are you using virtio drivers in the VMs?
> >
> > There was an issue with KVM-72 and virtio that leaks memory in the
> > host until all RAM and swap is used (inside the VMs, no swap is
> > used). It was supposed to be fixed in KVM-80-something, though.
> >
> > Perhaps something similar is happening again? If you switch the
> > disks to scsi instead of virtio, does the problem go away?
> >
> > We are running KVM-72 on Debian 5.0 and have run into this issue.
> > We'll be upgrading our hosts this month to fix this.
> Yes, we are using Virtio drivers for networking and storage in both
> VMs with cache=none. Both VMs are running Linux 2.6.32-bpo.5-amd64
> from Lenny Backports repositories. For VMHost, we are using a stable
> version of KVM with Linux 2.6.32.12 compiled from source code of
> kernel.org and qemu-kvm 0.12.3 compiled with the source code obtained
> from the official site of KVM.
>
> This is the syntax I'm using to boot the virtual machines:
>
>
> 8587 ? Sl 6515:25 /usr/local/qemu-kvm/bin/qemu-system-x86_64 -drive
> file=/dev/vm/aps4-raiz,cache=none,if=virtio,boot=on -drive
> file=/dev/vm/aps4-cache,cache=none,if=virtio -drive file=/dev/vm/aps4-index,cache=none,if=virtio
> -drive file=/dev/vm/aps4-space,cache=none,if=virtio -m 7168 -smp 4 -net
> nic,model=virtio,macaddr=00:16:3e:00:00:95 -net tap -daemonize -vnc :3 -k es -localtime -monitor
> telnet:localhost:4003,server,nowait -serial telnet:localhost:4043,server,nowait
>
> 9769 ? Rl 11968:47 /usr/local/qemu-kvm/bin/qemu-system-x86_64 -drive
> file=/dev/vm/leela-raiz,cache=none,if=virtio,boot=on -drive
> file=/dev/vm/leela-u01,cache=none,if=virtio -drive file=/dev/vm/leela-u02,cache=none,if=virtio
> -drive file=/dev/vm/leela-u03,cache=none,if=virtio -drive
> file=/dev/vm/leela-u04,cache=none,if=virtio -drive file=/dev/vm/leela-u05,cache=none,if=virtio
> -drive file=/dev/vm/leela-u06,cache=none,if=virtio -drive
> file=/dev/vm/leela-u07,cache=none,if=virtio -drive file=/dev/vm/leela-u08,cache=none,if=virtio
> -drive file=/dev/vm/leela-u09,cache=none,if=virtio -drive
> file=/dev/vm/leela-space,cache=none,if=virtio -m 7168 -smp 8 -net
> nic,model=virtio,macaddr=00:16:3e:00:00:96 -net tap -daemonize -vnc :4 -k es -localtime -monitor
> telnet:localhost:4004,server,nowait -serial telnet:localhost:4044,server,nowait
> To make the switch from Virtio to SCSI I would have to shut down the
> hosts, which would not be a good idea whereas are two productive
> systems. At least, before doing so I would be sure of what might be
> the problem.
>
> Taking a current measurement in VMHost with free, I got the following:
>
>
> ss04:~# free
> total used free shared buffers cached
> Mem: 16461588 16406504 55084 0 2920 21504
> -/+ buffers/cache: 16382080 79508
> Swap: 2028492 983140 1045352
>
>
> It draws attention to me that thinking about initially leaving a margin
> of 2 GB of RAM for the VMHost, already it has used almost half of swap.
This is a current measurement I've taken in both the VMs and in VMHost:
* VMHost:
ss04:~# free
total used free shared buffers cached
Mem: 16461588 16405140 56448 0 3496 18604
-/+ buffers/cache: 16383040 78548
Swap: 5174220 2401552 2772668
* Aps4:
aps4:~# free
total used free shared buffers cached
Mem: 7164300 7120192 44108 0 23108 239076
-/+ buffers/cache: 6858008 306292
Swap: 2931820 14084 2917736
* Leela:
leela:~# free
total used free shared buffers cached
Mem: 7163836 6905224 258612 0 123380 6282816
-/+ buffers/cache: 499028 6664808
Swap: 979924 35640 944284
As you can see, I added more swap in VMHost for more margin, but
currently only 54% is free.
Thanks in advance for your replies.
Regards,
Daniel
--
Fingerprint: BFB3 08D6 B4D1 31B2 72B9 29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Swap usage with KVM
2010-07-11 22:08 ` Daniel Bareiro
2010-07-20 15:04 ` Daniel Bareiro
@ 2010-07-20 19:18 ` David Weber
2010-07-22 10:09 ` Daniel Bareiro
1 sibling, 1 reply; 15+ messages in thread
From: David Weber @ 2010-07-20 19:18 UTC (permalink / raw)
To: dbareiro, KVM General
> Yes, we are using Virtio drivers for networking and storage in both VMs
> with cache=none. Both VMs are running Linux 2.6.32-bpo.5-amd64 from
> Lenny Backports repositories. For VMHost, we are using a stable version
> of KVM with Linux 2.6.32.12 compiled from source code of kernel.org and
> qemu-kvm 0.12.3 compiled with the source code obtained from the official
> site of KVM.
>
Afaik this should be this bug
http://sourceforge.net/tracker/?func=detail&atid=893831&aid=2989366&group_id=180599
try upgrading to 0.12.4 or backport this commit
http://git.kernel.org/?p=virt/kvm/qemu-
kvm.git;a=commit;h=012d4869c1eb195e83f159ed7b2bced33f37f960
David
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Swap usage with KVM
2010-07-20 19:18 ` David Weber
@ 2010-07-22 10:09 ` Daniel Bareiro
0 siblings, 0 replies; 15+ messages in thread
From: Daniel Bareiro @ 2010-07-22 10:09 UTC (permalink / raw)
To: KVM General
[-- Attachment #1.1: Type: text/plain, Size: 1138 bytes --]
Hi, David.
On Tuesday, 20 July 2010 21:18:09 +0200,
David Weber wrote:
> > Yes, we are using Virtio drivers for networking and storage in both VMs
> > with cache=none. Both VMs are running Linux 2.6.32-bpo.5-amd64 from
> > Lenny Backports repositories. For VMHost, we are using a stable version
> > of KVM with Linux 2.6.32.12 compiled from source code of kernel.org and
> > qemu-kvm 0.12.3 compiled with the source code obtained from the official
> > site of KVM.
> Afaik this should be this bug
> http://sourceforge.net/tracker/?func=detail&atid=893831&aid=2989366&group_id=180599
>
> try upgrading to 0.12.4 or backport this commit
> http://git.kernel.org/?p=virt/kvm/qemu-
> kvm.git;a=commit;h=012d4869c1eb195e83f159ed7b2bced33f37f960
Interesting... I'll try upgrading to qemu-kvm 0.12.4.
Marcelo Tosatti also recommended me to apply this patch I'm attaching.
Someone could confirm whether this is already included in Linux stable?
Thanks for your reply.
Regards,
Daniel
--
Fingerprint: BFB3 08D6 B4D1 31B2 72B9 29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598
[-- Attachment #1.2: kvm-vmx-accessed-bit.patch --]
[-- Type: text/x-diff, Size: 1691 bytes --]
commit 6316e1c8c6af6ccb55ff8564231710660608f46c
Author: Rik van Riel <riel@redhat.com>
Date: Wed Feb 3 16:11:03 2010 -0500
KVM: VMX: emulate accessed bit for EPT
Currently KVM pretends that pages with EPT mappings never got
accessed. This has some side effects in the VM, like swapping
out actively used guest pages and needlessly breaking up actively
used hugepages.
We can avoid those very costly side effects by emulating the
accessed bit for EPT PTEs, which should only be slightly costly
because pages pass through page_referenced infrequently.
TLB flushing is taken care of by kvm_mmu_notifier_clear_flush_young().
This seems to help prevent KVM guests from being swapped out when
they should not on my system.
Signed-off-by: Rik van Riel <riel@redhat.com>
Signed-off-by: Avi Kivity <avi@redhat.com>
diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c
index 913ef4b..b8da671 100644
--- a/arch/x86/kvm/mmu.c
+++ b/arch/x86/kvm/mmu.c
@@ -838,9 +838,15 @@ static int kvm_age_rmapp(struct kvm *kvm, unsigned long *rmapp,
u64 *spte;
int young = 0;
- /* always return old for EPT */
+ /*
+ * Emulate the accessed bit for EPT, by checking if this page has
+ * an EPT mapping, and clearing it if it does. On the next access,
+ * a new EPT mapping will be established.
+ * This has some overhead, but not as much as the cost of swapping
+ * out actively used pages or breaking up actively used hugepages.
+ */
if (!shadow_accessed_mask)
- return 0;
+ return kvm_unmap_rmapp(kvm, rmapp, data);
spte = rmap_next(kvm, rmapp, NULL);
while (spte) {
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: Swap usage with KVM
2010-07-11 21:49 ` Rik van Riel
@ 2010-08-02 18:57 ` Daniel Bareiro
2010-08-02 19:31 ` Rik van Riel
0 siblings, 1 reply; 15+ messages in thread
From: Daniel Bareiro @ 2010-08-02 18:57 UTC (permalink / raw)
To: Rik van Riel; +Cc: dbareiro, KVM General, hugh.dickins
[-- Attachment #1: Type: text/plain, Size: 2236 bytes --]
Hi, Rik.
On Sunday, 11 July 2010 17:49:43 -0400,
Rik van Riel wrote:
>>> I have an installation with Debian GNU/Linux 5.0.4 amd64 with
>>> qemu-kvm 0.12.3 compiled with the source code obtained from the
>>> official site of KVM and Linux 2.6.32.12 compiled from source code
>>> of kernel.org. All this is installed on an HP Proliant DL380 G6 with
>>> two Xeon E5530 quadcore processors and 16 GiB of RAM which has two
>>> VMs with the following configuration of memory:
>>>
>>> Hostname | RAM
>>> ===============+===============
>>> Aps4 | 7 GiB
>>> Leela | 7 GiB
>>> ===============+===============
>>> TOTAL | 14 GiB
>>>
>>> Initially the host was created with a swap partition of 1 GiB, but
>>> today we found that the use of swap quickly began to grow
>>> increasingly. Therefore, as a contingency, we had to hot-add a
>>> logical volume of 1 GB of swap on the VMHost. Is 'normal' this use
>>> of memory?
> That depends on what is going on in the host.
>
> Did you notice any performance issues in the guest when you started
> using swap?
After the logical volume of 1 GB that I added when I found this problem
(being the operating system with 2 GB), I added other 3 GB to have a
little more margin, but today I got again a new alert of Nagios:
Swap usage WARNING [...] SWAP WARNING - 30% free (1490 MB out of 5052 MB)
I don't see performance issue in the VMs.
Marcelo Tosatti recommended me to apply the EPT patch
(6316e1c8c6af6ccb55ff8564231710660608f46c). Should it be safe for use in
production? Are there plans that this patch is applied on some version
of Linux stable?
Also it can be advisable to update to qemu-kvm 0.12.4 considering what
David Weber said:
http://sourceforge.net/tracker/?func=detail&atid=893831&aid=2989366&group_id=180599
Thanks for your reply.
Regards,
Daniel
--
Mi frase del día:
Collaboration, n.:
A literary partnership based on the false assumption that the
other fellow can spell.
Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
15:28:17 up 23 days, 20:20, 12 users, load average: 0.16, 0.14, 0.09
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Swap usage with KVM
2010-08-02 18:57 ` Daniel Bareiro
@ 2010-08-02 19:31 ` Rik van Riel
2010-08-02 19:52 ` Daniel Bareiro
0 siblings, 1 reply; 15+ messages in thread
From: Rik van Riel @ 2010-08-02 19:31 UTC (permalink / raw)
To: dbareiro, KVM General, hugh.dickins
On 08/02/2010 02:57 PM, Daniel Bareiro wrote:
> Hi, Rik.
>
> On Sunday, 11 July 2010 17:49:43 -0400,
> Rik van Riel wrote:
>
>>>> I have an installation with Debian GNU/Linux 5.0.4 amd64 with
>>>> qemu-kvm 0.12.3 compiled with the source code obtained from the
>>>> official site of KVM and Linux 2.6.32.12 compiled from source code
>>>> of kernel.org. All this is installed on an HP Proliant DL380 G6 with
>>>> two Xeon E5530 quadcore processors and 16 GiB of RAM which has two
>>>> VMs with the following configuration of memory:
>>>>
>>>> Hostname | RAM
>>>> ===============+===============
>>>> Aps4 | 7 GiB
>>>> Leela | 7 GiB
>>>> ===============+===============
>>>> TOTAL | 14 GiB
>>>>
>>>> Initially the host was created with a swap partition of 1 GiB, but
>>>> today we found that the use of swap quickly began to grow
>>>> increasingly. Therefore, as a contingency, we had to hot-add a
>>>> logical volume of 1 GB of swap on the VMHost. Is 'normal' this use
>>>> of memory?
>
>> That depends on what is going on in the host.
>>
>> Did you notice any performance issues in the guest when you started
>> using swap?
>
> After the logical volume of 1 GB that I added when I found this problem
> (being the operating system with 2 GB), I added other 3 GB to have a
> little more margin, but today I got again a new alert of Nagios:
>
>
> Swap usage WARNING [...] SWAP WARNING - 30% free (1490 MB out of 5052 MB)
>
>
> I don't see performance issue in the VMs.
>
> Marcelo Tosatti recommended me to apply the EPT patch
> (6316e1c8c6af6ccb55ff8564231710660608f46c). Should it be safe for use in
> production? Are there plans that this patch is applied on some version
> of Linux stable?
Yes, that patch is safe and definitely recommended.
--
All rights reversed
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Swap usage with KVM
2010-08-02 19:31 ` Rik van Riel
@ 2010-08-02 19:52 ` Daniel Bareiro
2010-08-02 20:22 ` Rik van Riel
0 siblings, 1 reply; 15+ messages in thread
From: Daniel Bareiro @ 2010-08-02 19:52 UTC (permalink / raw)
To: Rik van Riel; +Cc: dbareiro, KVM General, hugh.dickins
[-- Attachment #1: Type: text/plain, Size: 2263 bytes --]
Hi, Rik.
On Monday, 02 August 2010 15:31:47 -0400,
Rik van Riel wrote:
>>>>> I have an installation with Debian GNU/Linux 5.0.4 amd64 with
>>>>> qemu-kvm 0.12.3 compiled with the source code obtained from the
>>>>> official site of KVM and Linux 2.6.32.12 compiled from source code
>>>>> of kernel.org. All this is installed on an HP Proliant DL380 G6
>>>>> with two Xeon E5530 quadcore processors and 16 GiB of RAM which
>>>>> has two VMs with the following configuration of memory:
>>>>>
>>>>> Hostname | RAM
>>>>> ===============+===============
>>>>> Aps4 | 7 GiB
>>>>> Leela | 7 GiB
>>>>> ===============+===============
>>>>> TOTAL | 14 GiB
>>>>>
>>>>> Initially the host was created with a swap partition of 1 GiB, but
>>>>> today we found that the use of swap quickly began to grow
>>>>> increasingly. Therefore, as a contingency, we had to hot-add a
>>>>> logical volume of 1 GB of swap on the VMHost. Is 'normal' this use
>>>>> of memory?
>>> That depends on what is going on in the host.
>>>
>>> Did you notice any performance issues in the guest when you started
>>> using swap?
>> After the logical volume of 1 GB that I added when I found this
>> problem (being the operating system with 2 GB), I added other 3 GB to
>> have a little more margin, but today I got again a new alert of
>> Nagios:
>>
>>
>> Swap usage WARNING [...] SWAP WARNING - 30% free (1490 MB out of 5052 MB)
>>
>>
>> I don't see performance issue in the VMs.
>>
>> Marcelo Tosatti recommended me to apply the EPT patch
>> (6316e1c8c6af6ccb55ff8564231710660608f46c). Should it be safe for use in
>> production? Are there plans that this patch is applied on some version
>> of Linux stable?
> Yes, that patch is safe and definitely recommended.
Perfect. Thanks for so fast answer.
And there are some estimates of when this patch is in Linux stable?
Regards,
Daniel
--
Mi frase del día:
BOFH excuse #281:
The co-locator cannot verify the frame-relay gateway to the ISDN server.
Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
16:44:53 up 23 days, 21:37, 11 users, load average: 0.02, 0.06, 0.08
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Swap usage with KVM
2010-08-02 19:52 ` Daniel Bareiro
@ 2010-08-02 20:22 ` Rik van Riel
2010-08-02 21:43 ` Daniel Bareiro
2010-08-27 10:04 ` Swap usage with KVM (and KSM) Daniel Bareiro
0 siblings, 2 replies; 15+ messages in thread
From: Rik van Riel @ 2010-08-02 20:22 UTC (permalink / raw)
To: dbareiro, KVM General, hugh.dickins
On 08/02/2010 03:52 PM, Daniel Bareiro wrote:
> And there are some estimates of when this patch is in Linux stable?
It should be there already in 2.6.33-stable and 2.6.34-stable.
--
All rights reversed
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Swap usage with KVM
2010-08-02 20:22 ` Rik van Riel
@ 2010-08-02 21:43 ` Daniel Bareiro
2010-08-27 10:04 ` Swap usage with KVM (and KSM) Daniel Bareiro
1 sibling, 0 replies; 15+ messages in thread
From: Daniel Bareiro @ 2010-08-02 21:43 UTC (permalink / raw)
To: Rik van Riel; +Cc: dbareiro, KVM General, hugh.dickins
[-- Attachment #1: Type: text/plain, Size: 606 bytes --]
El lunes 02 de agosto del 2010 a las 16:22:14,
Rik van Riel escribió:
>> And there are some estimates of when this patch is in Linux stable?
> It should be there already in 2.6.33-stable and 2.6.34-stable.
Great! Thanks for your reply.
Regards,
Daniel
--
Mi frase del día:
i dont even know if it makes sense at all :) This is an experimental patch
for an experimental kernel :))
-- Ingo Molnar on linux-kernel
Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
18:38:51 up 23 days, 23:31, 11 users, load average: 0.17, 0.16, 0.10
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Swap usage with KVM (and KSM)
2010-08-02 20:22 ` Rik van Riel
2010-08-02 21:43 ` Daniel Bareiro
@ 2010-08-27 10:04 ` Daniel Bareiro
2010-08-27 13:42 ` Rik van Riel
1 sibling, 1 reply; 15+ messages in thread
From: Daniel Bareiro @ 2010-08-27 10:04 UTC (permalink / raw)
To: Rik van Riel; +Cc: dbareiro, KVM General, hugh.dickins
[-- Attachment #1: Type: text/plain, Size: 1476 bytes --]
Hi, Rik.
On Monday, 02 August 2010 16:22:14 -0400,
Rik van Riel wrote:
>> And there are some estimates of when this patch is in Linux stable?
> It should be there already in 2.6.33-stable and 2.6.34-stable.
In a test environment I have Linux 2.6.34.5 compiled with the source
code from kernel.org and qemu-kvm 0.12.4 installed from the Backports
repository of Debian GNU/Linux.
In this environment I have running the following virtual machines:
* 2 x OpenBSD (2 x 512 MB) |
* 2 x Debian GNU/Linux (2 x 256 MB) |
* 2 x Debian GNU/Linux (2 x 512 MB) +--= 11008 MB
* 1 x Centos 5.5 (1 x 256 MB) |
* 8 x Debian GNU/Linux (8 x 1024 MB) |
I'm just starting to test KSM, so I was using as reference the read of
the tests made by RedHat about run as many as 52 Windows XP VMs with 1
GB of RAM each on a server with just 16 GB of RAM.
In the previous case the ratio would be 52/16 = 3.25. In my case the
VMHost has 4 GB of RAM, so the ratio would be 10.75/4 = 2.6875. In RH
tests do not talk about the amount of swap used in that case, so I
wonder if a distribution of VMs as I have, it is normal to use 290 MB of
swap.
Regards,
Daniel
--
Mi frase del día:
BOFH excuse #335:
the AA battery in the wallclock sends magnetic interference
Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
05:57:46 up 48 days, 10:50, 11 users, load average: 0.01, 0.04, 0.06
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: Swap usage with KVM (and KSM)
2010-08-27 10:04 ` Swap usage with KVM (and KSM) Daniel Bareiro
@ 2010-08-27 13:42 ` Rik van Riel
0 siblings, 0 replies; 15+ messages in thread
From: Rik van Riel @ 2010-08-27 13:42 UTC (permalink / raw)
To: dbareiro, KVM General, hugh.dickins
On 08/27/2010 06:04 AM, Daniel Bareiro wrote:
> In the previous case the ratio would be 52/16 = 3.25. In my case the
> VMHost has 4 GB of RAM, so the ratio would be 10.75/4 = 2.6875. In RH
> tests do not talk about the amount of swap used in that case, so I
> wonder if a distribution of VMs as I have, it is normal to use 290 MB of
> swap.
Given that you have 10.75GB worth of virtual machine on a
system with 4GB of RAM, I'd say that 290MB of swap is not
out of the ordinary.
--
All rights reversed
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2010-08-27 13:42 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-07-11 15:12 Swap usage with KVM Daniel Bareiro
2010-07-11 19:12 ` Daniel Bareiro
2010-07-11 21:49 ` Rik van Riel
2010-08-02 18:57 ` Daniel Bareiro
2010-08-02 19:31 ` Rik van Riel
2010-08-02 19:52 ` Daniel Bareiro
2010-08-02 20:22 ` Rik van Riel
2010-08-02 21:43 ` Daniel Bareiro
2010-08-27 10:04 ` Swap usage with KVM (and KSM) Daniel Bareiro
2010-08-27 13:42 ` Rik van Riel
2010-07-11 21:05 ` Swap usage with KVM Freddie Cash
2010-07-11 22:08 ` Daniel Bareiro
2010-07-20 15:04 ` Daniel Bareiro
2010-07-20 19:18 ` David Weber
2010-07-22 10:09 ` Daniel Bareiro
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).