From mboxrd@z Thu Jan  1 00:00:00 1970
From: Stefan Hajnoczi <stefanha@gmail.com>
Subject: Re: How to diagnose memory leak in kvm-qemu-0.14.0?
Date: Thu, 19 May 2011 09:40:41 +0100
Message-ID: <BANLkTimq2S3a=BbDJvxkeET9pK64ybKEiw@mail.gmail.com>
References: <20110518164429.GA20927@bytemark.co.uk>
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: QUOTED-PRINTABLE
Cc: kvm@vger.kernel.org
To: Steve Kemp <steve@bytemark.co.uk>
Return-path: <kvm-owner@vger.kernel.org>
Received: from mail-yi0-f46.google.com ([209.85.218.46]:42759 "EHLO
	mail-yi0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S932424Ab1ESIkn convert rfc822-to-8bit (ORCPT
	<rfc822;kvm@vger.kernel.org>); Thu, 19 May 2011 04:40:43 -0400
Received: by yia27 with SMTP id 27so827995yia.19
        for <kvm@vger.kernel.org>; Thu, 19 May 2011 01:40:42 -0700 (PDT)
In-Reply-To: <20110518164429.GA20927@bytemark.co.uk>
Sender: kvm-owner@vger.kernel.org
List-ID: <kvm.vger.kernel.org>

On Wed, May 18, 2011 at 5:44 PM, Steve Kemp <steve@bytemark.co.uk> wrot=
e:
>
> =A0I'm running the most recent release of KVM, version 0.14.0
> =A0on a host kernel 2.6.32.15, and seem to be able to trigger
> =A0a leak of memory pretty easily.
>
> =A0Inside a guest the following one-liner will cause the KVM
> =A0process on the host to gradually increase its memory
> =A0consumption:
>
> =A0 =A0while true; do
> =A0 =A0 =A0wget http://mirror.bytemark.co.uk/misc/test-files/500M; cp=
 500M new; rm 500M new; sleep 10 ;
> =A0 =A0done

You are exercising both networking and storage.  Have you cut the test
down to just wget vs cp/rm?  Also why the sleep 10?

If you are building qemu-kvm from source you might like to enable
tracing to track memory allocations in qemu-kvm.  For full information
see qemu-kvm/docs/tracing.txt.  There are several trace events of
interest:
$ cd qemu-kvm
$ $EDITOR trace-events
# qemu-malloc.c
disable qemu_malloc(size_t size, void *ptr) "size %zu ptr %p"
disable qemu_realloc(void *ptr, size_t size, void *newptr) "ptr %p
size %zu newptr %p"
disable qemu_free(void *ptr) "ptr %p"

# osdep.c
disable qemu_memalign(size_t alignment, size_t size, void *ptr)
"alignment %zu size %zu ptr %p"
disable qemu_vmalloc(size_t size, void *ptr) "size %zu ptr %p"
disable qemu_vfree(void *ptr) "ptr %p"
^--- remove the "disable" property from these memory allocation events
$ ./configure --enable-trace-backend=3Dsimple [...]
$ make
$ # run the VM, reproduce the leak, shut the VM down
$ scripts/simpletrace.py trace-events trace-<pid>  # where <pid> was
the process ID

It is fairly easy to write a script that correlates mallocs and frees,
printing out memory allocations that were never freed at the end.
There is a Python API for processing trace files, here is an
explanation of how ot use it:
http://blog.vmsplice.net/2011/03/how-to-write-trace-analysis-scripts-fo=
r.html

If you have SystemTap installed you may wish to use the "dtrace"
backend instead of "simple".  You can then use SystemTap scripts on
the probes.  SystemTap is more powerful, it should allow you to
extract call stacks when probes are fired but I'm not experienced with
it.

=46eel free to contact me on #qemu (oftc) or #kvm (freenode) IRC if you
want some pointers, my nick is stefanha.

Stefan