[Qemu-devel] [PATCH] configure: Add support for jemalloc

[Qemu-devel] [PATCH] configure: Add support for jemalloc

[Alexandre Derumier]

This adds "--enable-jemalloc" and "--disable-jemalloc" to allow linking
to jemalloc memory allocator.

We have already tcmalloc support,
but it seem to not working well with a lot of iothreads/disks.

The main problem is that tcmalloc use a shared thread cache of 16MB
by default.
With more threads, this cache is shared, and some bad garbage collections
can occur if the cache is too low.

It's possible to tcmalloc cache increase it with a env var:
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=256MB

With default 16MB, performances are  really bad with more than 2 disks.
Increasing to 256MB, it's helping but still have problem with 16 disks/iothreads.

Jemalloc don't have performance problem with default configuration.

Here the benchmark results in iops of 1 qemu vm randread 4K iodepth=32,
with rbd block backend (librbd is doing a lot of memory allocation),
1 iothread by disk

glibc malloc
------------

1 disk      29052
2 disks     55878
4 disks     127899
8 disks     240566
15 disks    269976

jemalloc
--------

1 disk      41278
2 disks     75781
4 disks     195351
8 disks     294241
15 disks    298199

tcmalloc 2.2.1 default 16M cache
--------------------------------

1 disk   37911
2 disks  67698
4 disks  41076
8 disks  43312
15 disks 37569

tcmalloc : 256M cache
---------------------------

1 disk     33914
2 disks    58839
4 disks    148205
8 disks    213298
15 disks   218383

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 configure | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/configure b/configure
index 222694f..2fe1e05 100755
--- a/configure
+++ b/configure
@@ -336,6 +336,7 @@ vhdx=""
 quorum=""
 numa=""
 tcmalloc="no"
+jemalloc="no"
 
 # parse CC options first
 for opt do
@@ -1147,6 +1148,10 @@ for opt do
   ;;
   --enable-tcmalloc) tcmalloc="yes"
   ;;
+  --disable-jemalloc) jemalloc="no"
+  ;;
+  --enable-jemalloc) jemalloc="yes"
+  ;;
   *)
       echo "ERROR: unknown option $opt"
       echo "Try '$0 --help' for more information"
@@ -1420,6 +1425,8 @@ Advanced options (experts only):
   --enable-numa            enable libnuma support
   --disable-tcmalloc       disable tcmalloc support
   --enable-tcmalloc        enable tcmalloc support
+  --disable-jemalloc       disable jemalloc support
+  --enable-jemalloc        enable jemalloc support
 
 NOTE: The object files are built at the place where configure is launched
 EOF
@@ -3344,6 +3351,11 @@ EOF
   fi
 fi
 
+if test "$tcmalloc" = "yes" && test "$jemalloc" = "yes" ; then
+    echo "ERROR: tcmalloc && jemalloc can't be used at the same time"
+    exit 1
+fi
+
 ##########################################
 # tcmalloc probe
 
@@ -3361,6 +3373,22 @@ EOF
 fi
 
 ##########################################
+# jemalloc probe
+
+if test "$jemalloc" = "yes" ; then
+  cat > $TMPC << EOF
+#include <stdlib.h>
+int main(void) { malloc(1); return 0; }
+EOF
+
+  if compile_prog "" "-ljemalloc" ; then
+    LIBS="-ljemalloc $LIBS"
+  else
+    feature_not_found "jemalloc" "install jemalloc devel"
+  fi
+fi
+
+##########################################
 # signalfd probe
 signalfd="no"
 cat > $TMPC << EOF
@@ -4499,6 +4527,7 @@ echo "snappy support    $snappy"
 echo "bzip2 support     $bzip2"
 echo "NUMA host support $numa"
 echo "tcmalloc support  $tcmalloc"
+echo "jemalloc support  $jemalloc"
 
 if test "$sdl_too_old" = "yes"; then
 echo "-> Your SDL version is too old - please upgrade to have SDL support"
-- 
2.1.4

Re: [Qemu-devel] [PATCH] configure: Add support for jemalloc

[Alexandre DERUMIER]

@cc Fam Zheng <famz@redhat.com>,

as he's the author of tcmalloc support patch

----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "qemu-devel" <qemu-devel@nongnu.org>
Cc: "aderumier" <aderumier@odiso.com>
Envoyé: Vendredi 19 Juin 2015 12:56:58
Objet: [PATCH] configure: Add support for jemalloc

This adds "--enable-jemalloc" and "--disable-jemalloc" to allow linking 
to jemalloc memory allocator. 

We have already tcmalloc support, 
but it seem to not working well with a lot of iothreads/disks. 

The main problem is that tcmalloc use a shared thread cache of 16MB 
by default. 
With more threads, this cache is shared, and some bad garbage collections 
can occur if the cache is too low. 

It's possible to tcmalloc cache increase it with a env var: 
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=256MB 

With default 16MB, performances are really bad with more than 2 disks. 
Increasing to 256MB, it's helping but still have problem with 16 disks/iothreads. 

Jemalloc don't have performance problem with default configuration. 

Here the benchmark results in iops of 1 qemu vm randread 4K iodepth=32, 
with rbd block backend (librbd is doing a lot of memory allocation), 
1 iothread by disk 

glibc malloc 
------------ 

1 disk 29052 
2 disks 55878 
4 disks 127899 
8 disks 240566 
15 disks 269976 

jemalloc 
-------- 

1 disk 41278 
2 disks 75781 
4 disks 195351 
8 disks 294241 
15 disks 298199 

tcmalloc 2.2.1 default 16M cache 
-------------------------------- 

1 disk 37911 
2 disks 67698 
4 disks 41076 
8 disks 43312 
15 disks 37569 

tcmalloc : 256M cache 
--------------------------- 

1 disk 33914 
2 disks 58839 
4 disks 148205 
8 disks 213298 
15 disks 218383 

Signed-off-by: Alexandre Derumier <aderumier@odiso.com> 
--- 
configure | 29 +++++++++++++++++++++++++++++ 
1 file changed, 29 insertions(+) 

diff --git a/configure b/configure 
index 222694f..2fe1e05 100755 
--- a/configure 
+++ b/configure 
@@ -336,6 +336,7 @@ vhdx="" 
quorum="" 
numa="" 
tcmalloc="no" 
+jemalloc="no" 

# parse CC options first 
for opt do 
@@ -1147,6 +1148,10 @@ for opt do 
;; 
--enable-tcmalloc) tcmalloc="yes" 
;; 
+ --disable-jemalloc) jemalloc="no" 
+ ;; 
+ --enable-jemalloc) jemalloc="yes" 
+ ;; 
*) 
echo "ERROR: unknown option $opt" 
echo "Try '$0 --help' for more information" 
@@ -1420,6 +1425,8 @@ Advanced options (experts only): 
--enable-numa enable libnuma support 
--disable-tcmalloc disable tcmalloc support 
--enable-tcmalloc enable tcmalloc support 
+ --disable-jemalloc disable jemalloc support 
+ --enable-jemalloc enable jemalloc support 

NOTE: The object files are built at the place where configure is launched 
EOF 
@@ -3344,6 +3351,11 @@ EOF 
fi 
fi 

+if test "$tcmalloc" = "yes" && test "$jemalloc" = "yes" ; then 
+ echo "ERROR: tcmalloc && jemalloc can't be used at the same time" 
+ exit 1 
+fi 
+ 
########################################## 
# tcmalloc probe 

@@ -3361,6 +3373,22 @@ EOF 
fi 

########################################## 
+# jemalloc probe 
+ 
+if test "$jemalloc" = "yes" ; then 
+ cat > $TMPC << EOF 
+#include <stdlib.h> 
+int main(void) { malloc(1); return 0; } 
+EOF 
+ 
+ if compile_prog "" "-ljemalloc" ; then 
+ LIBS="-ljemalloc $LIBS" 
+ else 
+ feature_not_found "jemalloc" "install jemalloc devel" 
+ fi 
+fi 
+ 
+########################################## 
# signalfd probe 
signalfd="no" 
cat > $TMPC << EOF 
@@ -4499,6 +4527,7 @@ echo "snappy support $snappy" 
echo "bzip2 support $bzip2" 
echo "NUMA host support $numa" 
echo "tcmalloc support $tcmalloc" 
+echo "jemalloc support $jemalloc" 

if test "$sdl_too_old" = "yes"; then 
echo "-> Your SDL version is too old - please upgrade to have SDL support" 
-- 
2.1.4 

Re: [Qemu-devel] [PATCH] configure: Add support for jemalloc

[Fam Zheng]

On Fri, 06/19 12:56, Alexandre Derumier wrote:
> This adds "--enable-jemalloc" and "--disable-jemalloc" to allow linking
> to jemalloc memory allocator.
> 
> We have already tcmalloc support,
> but it seem to not working well with a lot of iothreads/disks.
> 
> The main problem is that tcmalloc use a shared thread cache of 16MB
> by default.
> With more threads, this cache is shared, and some bad garbage collections
> can occur if the cache is too low.
> 
> It's possible to tcmalloc cache increase it with a env var:
> TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=256MB
> 
> With default 16MB, performances are  really bad with more than 2 disks.
> Increasing to 256MB, it's helping but still have problem with 16 disks/iothreads.
> 
> Jemalloc don't have performance problem with default configuration.
> 
> Here the benchmark results in iops of 1 qemu vm randread 4K iodepth=32,
> with rbd block backend (librbd is doing a lot of memory allocation),
> 1 iothread by disk
> 
> glibc malloc
> ------------
> 
> 1 disk      29052
> 2 disks     55878
> 4 disks     127899
> 8 disks     240566
> 15 disks    269976
> 
> jemalloc
> --------
> 
> 1 disk      41278
> 2 disks     75781
> 4 disks     195351
> 8 disks     294241
> 15 disks    298199
> 
> tcmalloc 2.2.1 default 16M cache
> --------------------------------
> 
> 1 disk   37911
> 2 disks  67698
> 4 disks  41076
> 8 disks  43312
> 15 disks 37569
> 
> tcmalloc : 256M cache
> ---------------------------
> 
> 1 disk     33914
> 2 disks    58839
> 4 disks    148205
> 8 disks    213298
> 15 disks   218383
> 
> Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
> ---
>  configure | 29 +++++++++++++++++++++++++++++
>  1 file changed, 29 insertions(+)
> 
> diff --git a/configure b/configure
> index 222694f..2fe1e05 100755
> --- a/configure
> +++ b/configure
> @@ -336,6 +336,7 @@ vhdx=""
>  quorum=""
>  numa=""
>  tcmalloc="no"
> +jemalloc="no"
>  
>  # parse CC options first
>  for opt do
> @@ -1147,6 +1148,10 @@ for opt do
>    ;;
>    --enable-tcmalloc) tcmalloc="yes"
>    ;;
> +  --disable-jemalloc) jemalloc="no"
> +  ;;
> +  --enable-jemalloc) jemalloc="yes"
> +  ;;
>    *)
>        echo "ERROR: unknown option $opt"
>        echo "Try '$0 --help' for more information"
> @@ -1420,6 +1425,8 @@ Advanced options (experts only):
>    --enable-numa            enable libnuma support
>    --disable-tcmalloc       disable tcmalloc support
>    --enable-tcmalloc        enable tcmalloc support
> +  --disable-jemalloc       disable jemalloc support
> +  --enable-jemalloc        enable jemalloc support
>  
>  NOTE: The object files are built at the place where configure is launched
>  EOF
> @@ -3344,6 +3351,11 @@ EOF
>    fi
>  fi
>  
> +if test "$tcmalloc" = "yes" && test "$jemalloc" = "yes" ; then
> +    echo "ERROR: tcmalloc && jemalloc can't be used at the same time"
> +    exit 1
> +fi
> +
>  ##########################################
>  # tcmalloc probe
>  
> @@ -3361,6 +3373,22 @@ EOF
>  fi
>  
>  ##########################################
> +# jemalloc probe
> +
> +if test "$jemalloc" = "yes" ; then
> +  cat > $TMPC << EOF
> +#include <stdlib.h>
> +int main(void) { malloc(1); return 0; }
> +EOF
> +
> +  if compile_prog "" "-ljemalloc" ; then
> +    LIBS="-ljemalloc $LIBS"
> +  else
> +    feature_not_found "jemalloc" "install jemalloc devel"
> +  fi
> +fi
> +
> +##########################################
>  # signalfd probe
>  signalfd="no"
>  cat > $TMPC << EOF
> @@ -4499,6 +4527,7 @@ echo "snappy support    $snappy"
>  echo "bzip2 support     $bzip2"
>  echo "NUMA host support $numa"
>  echo "tcmalloc support  $tcmalloc"
> +echo "jemalloc support  $jemalloc"
>  
>  if test "$sdl_too_old" = "yes"; then
>  echo "-> Your SDL version is too old - please upgrade to have SDL support"
> -- 
> 2.1.4
> 
> 

Reviewed-by: Fam Zheng <famz@redhat.com>

Re: [Qemu-devel] [PATCH] configure: Add support for jemalloc

[Paolo Bonzini]

On 19/06/2015 12:56, Alexandre Derumier wrote:
> This adds "--enable-jemalloc" and "--disable-jemalloc" to allow linking
> to jemalloc memory allocator.
> 
> We have already tcmalloc support,
> but it seem to not working well with a lot of iothreads/disks.
> 
> The main problem is that tcmalloc use a shared thread cache of 16MB
> by default.
> With more threads, this cache is shared, and some bad garbage collections
> can occur if the cache is too low.
> 
> It's possible to tcmalloc cache increase it with a env var:
> TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=256MB

It's also possible to do

MallocExtension_SetNumericProperty("tcmalloc.max_total_thread_cache_bytes",
				   num_io_threads << 24);

What is the peak memory usage of jemalloc and tcmalloc?

Paolo

> With default 16MB, performances are  really bad with more than 2 disks.
> Increasing to 256MB, it's helping but still have problem with 16 disks/iothreads.
> 
> Jemalloc don't have performance problem with default configuration.
> 
> Here the benchmark results in iops of 1 qemu vm randread 4K iodepth=32,
> with rbd block backend (librbd is doing a lot of memory allocation),
> 1 iothread by disk
> 
> glibc malloc
> ------------
> 
> 1 disk      29052
> 2 disks     55878
> 4 disks     127899
> 8 disks     240566
> 15 disks    269976
> 
> jemalloc
> --------
> 
> 1 disk      41278
> 2 disks     75781
> 4 disks     195351
> 8 disks     294241
> 15 disks    298199
> 
> tcmalloc 2.2.1 default 16M cache
> --------------------------------
> 
> 1 disk   37911
> 2 disks  67698
> 4 disks  41076
> 8 disks  43312
> 15 disks 37569
> 
> tcmalloc : 256M cache
> ---------------------------
> 
> 1 disk     33914
> 2 disks    58839
> 4 disks    148205
> 8 disks    213298
> 15 disks   218383
> 
> Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
> ---
>  configure | 29 +++++++++++++++++++++++++++++
>  1 file changed, 29 insertions(+)
> 
> diff --git a/configure b/configure
> index 222694f..2fe1e05 100755
> --- a/configure
> +++ b/configure
> @@ -336,6 +336,7 @@ vhdx=""
>  quorum=""
>  numa=""
>  tcmalloc="no"
> +jemalloc="no"
>  
>  # parse CC options first
>  for opt do
> @@ -1147,6 +1148,10 @@ for opt do
>    ;;
>    --enable-tcmalloc) tcmalloc="yes"
>    ;;
> +  --disable-jemalloc) jemalloc="no"
> +  ;;
> +  --enable-jemalloc) jemalloc="yes"
> +  ;;
>    *)
>        echo "ERROR: unknown option $opt"
>        echo "Try '$0 --help' for more information"
> @@ -1420,6 +1425,8 @@ Advanced options (experts only):
>    --enable-numa            enable libnuma support
>    --disable-tcmalloc       disable tcmalloc support
>    --enable-tcmalloc        enable tcmalloc support
> +  --disable-jemalloc       disable jemalloc support
> +  --enable-jemalloc        enable jemalloc support
>  
>  NOTE: The object files are built at the place where configure is launched
>  EOF
> @@ -3344,6 +3351,11 @@ EOF
>    fi
>  fi
>  
> +if test "$tcmalloc" = "yes" && test "$jemalloc" = "yes" ; then
> +    echo "ERROR: tcmalloc && jemalloc can't be used at the same time"
> +    exit 1
> +fi
> +
>  ##########################################
>  # tcmalloc probe
>  
> @@ -3361,6 +3373,22 @@ EOF
>  fi
>  
>  ##########################################
> +# jemalloc probe
> +
> +if test "$jemalloc" = "yes" ; then
> +  cat > $TMPC << EOF
> +#include <stdlib.h>
> +int main(void) { malloc(1); return 0; }
> +EOF
> +
> +  if compile_prog "" "-ljemalloc" ; then
> +    LIBS="-ljemalloc $LIBS"
> +  else
> +    feature_not_found "jemalloc" "install jemalloc devel"
> +  fi
> +fi
> +
> +##########################################
>  # signalfd probe
>  signalfd="no"
>  cat > $TMPC << EOF
> @@ -4499,6 +4527,7 @@ echo "snappy support    $snappy"
>  echo "bzip2 support     $bzip2"
>  echo "NUMA host support $numa"
>  echo "tcmalloc support  $tcmalloc"
> +echo "jemalloc support  $jemalloc"
>  
>  if test "$sdl_too_old" = "yes"; then
>  echo "-> Your SDL version is too old - please upgrade to have SDL support"
> 

Re: [Qemu-devel] [PATCH] configure: Add support for jemalloc

[Alexandre DERUMIER]

Hi Paolo,

>>What is the peak memory usage of jemalloc and tcmalloc?

I'll try to use an heap profiling to see.
Don't known if "perf" can give me the info easily without profiling ?



----- Mail original -----
De: "pbonzini" <pbonzini@redhat.com>
À: "aderumier" <aderumier@odiso.com>, "qemu-devel" <qemu-devel@nongnu.org>
Envoyé: Mardi 23 Juin 2015 09:57:19
Objet: Re: [PATCH] configure: Add support for jemalloc

On 19/06/2015 12:56, Alexandre Derumier wrote: 
> This adds "--enable-jemalloc" and "--disable-jemalloc" to allow linking 
> to jemalloc memory allocator. 
> 
> We have already tcmalloc support, 
> but it seem to not working well with a lot of iothreads/disks. 
> 
> The main problem is that tcmalloc use a shared thread cache of 16MB 
> by default. 
> With more threads, this cache is shared, and some bad garbage collections 
> can occur if the cache is too low. 
> 
> It's possible to tcmalloc cache increase it with a env var: 
> TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=256MB 

It's also possible to do 

MallocExtension_SetNumericProperty("tcmalloc.max_total_thread_cache_bytes", 
num_io_threads << 24); 

What is the peak memory usage of jemalloc and tcmalloc? 

Paolo 

> With default 16MB, performances are really bad with more than 2 disks. 
> Increasing to 256MB, it's helping but still have problem with 16 disks/iothreads. 
> 
> Jemalloc don't have performance problem with default configuration. 
> 
> Here the benchmark results in iops of 1 qemu vm randread 4K iodepth=32, 
> with rbd block backend (librbd is doing a lot of memory allocation), 
> 1 iothread by disk 
> 
> glibc malloc 
> ------------ 
> 
> 1 disk 29052 
> 2 disks 55878 
> 4 disks 127899 
> 8 disks 240566 
> 15 disks 269976 
> 
> jemalloc 
> -------- 
> 
> 1 disk 41278 
> 2 disks 75781 
> 4 disks 195351 
> 8 disks 294241 
> 15 disks 298199 
> 
> tcmalloc 2.2.1 default 16M cache 
> -------------------------------- 
> 
> 1 disk 37911 
> 2 disks 67698 
> 4 disks 41076 
> 8 disks 43312 
> 15 disks 37569 
> 
> tcmalloc : 256M cache 
> --------------------------- 
> 
> 1 disk 33914 
> 2 disks 58839 
> 4 disks 148205 
> 8 disks 213298 
> 15 disks 218383 
> 
> Signed-off-by: Alexandre Derumier <aderumier@odiso.com> 
> --- 
> configure | 29 +++++++++++++++++++++++++++++ 
> 1 file changed, 29 insertions(+) 
> 
> diff --git a/configure b/configure 
> index 222694f..2fe1e05 100755 
> --- a/configure 
> +++ b/configure 
> @@ -336,6 +336,7 @@ vhdx="" 
> quorum="" 
> numa="" 
> tcmalloc="no" 
> +jemalloc="no" 
> 
> # parse CC options first 
> for opt do 
> @@ -1147,6 +1148,10 @@ for opt do 
> ;; 
> --enable-tcmalloc) tcmalloc="yes" 
> ;; 
> + --disable-jemalloc) jemalloc="no" 
> + ;; 
> + --enable-jemalloc) jemalloc="yes" 
> + ;; 
> *) 
> echo "ERROR: unknown option $opt" 
> echo "Try '$0 --help' for more information" 
> @@ -1420,6 +1425,8 @@ Advanced options (experts only): 
> --enable-numa enable libnuma support 
> --disable-tcmalloc disable tcmalloc support 
> --enable-tcmalloc enable tcmalloc support 
> + --disable-jemalloc disable jemalloc support 
> + --enable-jemalloc enable jemalloc support 
> 
> NOTE: The object files are built at the place where configure is launched 
> EOF 
> @@ -3344,6 +3351,11 @@ EOF 
> fi 
> fi 
> 
> +if test "$tcmalloc" = "yes" && test "$jemalloc" = "yes" ; then 
> + echo "ERROR: tcmalloc && jemalloc can't be used at the same time" 
> + exit 1 
> +fi 
> + 
> ########################################## 
> # tcmalloc probe 
> 
> @@ -3361,6 +3373,22 @@ EOF 
> fi 
> 
> ########################################## 
> +# jemalloc probe 
> + 
> +if test "$jemalloc" = "yes" ; then 
> + cat > $TMPC << EOF 
> +#include <stdlib.h> 
> +int main(void) { malloc(1); return 0; } 
> +EOF 
> + 
> + if compile_prog "" "-ljemalloc" ; then 
> + LIBS="-ljemalloc $LIBS" 
> + else 
> + feature_not_found "jemalloc" "install jemalloc devel" 
> + fi 
> +fi 
> + 
> +########################################## 
> # signalfd probe 
> signalfd="no" 
> cat > $TMPC << EOF 
> @@ -4499,6 +4527,7 @@ echo "snappy support $snappy" 
> echo "bzip2 support $bzip2" 
> echo "NUMA host support $numa" 
> echo "tcmalloc support $tcmalloc" 
> +echo "jemalloc support $jemalloc" 
> 
> if test "$sdl_too_old" = "yes"; then 
> echo "-> Your SDL version is too old - please upgrade to have SDL support" 
> 

Re: [Qemu-devel] [PATCH] configure: Add support for jemalloc

[Paolo Bonzini]

On 23/06/2015 10:22, Alexandre DERUMIER wrote:
> > What is the peak memory usage of jemalloc and tcmalloc?
> 
> I'll try to use an heap profiling to see.
> Don't known if "perf" can give me the info easily without profiling ?

You can try using or modifying this systemtap script:

https://sourceware.org/systemtap/examples/process/proctop.stp

Paolo

Re: [Qemu-devel] [PATCH] configure: Add support for jemalloc

[Alexandre DERUMIER]

>>You can try using or modifying this systemtap script: 
Thanks, I'll check this.


If it can help, I have done a small top of qemu libc vs jemalloc vs tcmalloc



libc-2.19.so
-------------
started
-------
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND  
 5294 root      20   0 26.953g 596744  18468 S   1.0  0.2   0:16.81 qemu  

fio running (200kiops)
----------------------
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                  
  5294 root      20   0 30.373g 633300  18552 S  2053  0.2   9:37.03 qemu   


jemalloc
--------
started
--------
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                  
  1789 root      20   0 11.640g 609224  18812 S   1.0  0.2   0:21.03 qemu  

fio running (300kiops)
----------------------
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                  
  1789 root      20   0 12.868g 684924  19144 S  2689  0.3   4:57.96 qemu   




tcmalloc (16MB)
---------------
started
-------
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                  
  1789 root      20   0 11.640g 609224  18812 S   1.0  0.2   0:21.03 qemu  

fio running (65k iops)
-------------------
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                  
  6900 root      20   0 10.925g 643624  18432 S  1125  0.2   4:18.51 qemu    


tcmalloc (256MB)
---------------
started
-------
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND   
 10640 root      20   0 10.117g 584524  18236 S   0.7  0.2   0:16.28 qemu

fio running  (200k iops)
-----------------------
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                  
 10640 root      20   0 10.983g 685136  18984 S  2566  0.3  12:34.73 qemu  


----- Mail original -----
De: "pbonzini" <pbonzini@redhat.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "qemu-devel" <qemu-devel@nongnu.org>
Envoyé: Mardi 23 Juin 2015 10:57:03
Objet: Re: [PATCH] configure: Add support for jemalloc

On 23/06/2015 10:22, Alexandre DERUMIER wrote: 
> > What is the peak memory usage of jemalloc and tcmalloc? 
> 
> I'll try to use an heap profiling to see. 
> Don't known if "perf" can give me the info easily without profiling ? 

You can try using or modifying this systemtap script: 

https://sourceware.org/systemtap/examples/process/proctop.stp 

Paolo 

Re: [Qemu-devel] [PATCH] configure: Add support for jemalloc

[Paolo Bonzini]

On 19/06/2015 12:56, Alexandre Derumier wrote:
> This adds "--enable-jemalloc" and "--disable-jemalloc" to allow linking
> to jemalloc memory allocator.
> 
> We have already tcmalloc support,
> but it seem to not working well with a lot of iothreads/disks.
> 
> The main problem is that tcmalloc use a shared thread cache of 16MB
> by default.
> With more threads, this cache is shared, and some bad garbage collections
> can occur if the cache is too low.
> 
> It's possible to tcmalloc cache increase it with a env var:
> TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=256MB
> 
> With default 16MB, performances are  really bad with more than 2 disks.
> Increasing to 256MB, it's helping but still have problem with 16 disks/iothreads.
> 
> Jemalloc don't have performance problem with default configuration.
> 
> Here the benchmark results in iops of 1 qemu vm randread 4K iodepth=32,
> with rbd block backend (librbd is doing a lot of memory allocation),
> 1 iothread by disk
> 
> glibc malloc
> ------------
> 
> 1 disk      29052
> 2 disks     55878
> 4 disks     127899
> 8 disks     240566
> 15 disks    269976
> 
> jemalloc
> --------
> 
> 1 disk      41278
> 2 disks     75781
> 4 disks     195351
> 8 disks     294241
> 15 disks    298199
> 
> tcmalloc 2.2.1 default 16M cache
> --------------------------------
> 
> 1 disk   37911
> 2 disks  67698
> 4 disks  41076
> 8 disks  43312
> 15 disks 37569
> 
> tcmalloc : 256M cache
> ---------------------------
> 
> 1 disk     33914
> 2 disks    58839
> 4 disks    148205
> 8 disks    213298
> 15 disks   218383
> 
> Signed-off-by: Alexandre Derumier <aderumier@odiso.com>

Rebased and applied, thanks.

Paolo

> ---
>  configure | 29 +++++++++++++++++++++++++++++
>  1 file changed, 29 insertions(+)
> 
> diff --git a/configure b/configure
> index 222694f..2fe1e05 100755
> --- a/configure
> +++ b/configure
> @@ -336,6 +336,7 @@ vhdx=""
>  quorum=""
>  numa=""
>  tcmalloc="no"
> +jemalloc="no"
>  
>  # parse CC options first
>  for opt do
> @@ -1147,6 +1148,10 @@ for opt do
>    ;;
>    --enable-tcmalloc) tcmalloc="yes"
>    ;;
> +  --disable-jemalloc) jemalloc="no"
> +  ;;
> +  --enable-jemalloc) jemalloc="yes"
> +  ;;
>    *)
>        echo "ERROR: unknown option $opt"
>        echo "Try '$0 --help' for more information"
> @@ -1420,6 +1425,8 @@ Advanced options (experts only):
>    --enable-numa            enable libnuma support
>    --disable-tcmalloc       disable tcmalloc support
>    --enable-tcmalloc        enable tcmalloc support
> +  --disable-jemalloc       disable jemalloc support
> +  --enable-jemalloc        enable jemalloc support
>  
>  NOTE: The object files are built at the place where configure is launched
>  EOF
> @@ -3344,6 +3351,11 @@ EOF
>    fi
>  fi
>  
> +if test "$tcmalloc" = "yes" && test "$jemalloc" = "yes" ; then
> +    echo "ERROR: tcmalloc && jemalloc can't be used at the same time"
> +    exit 1
> +fi
> +
>  ##########################################
>  # tcmalloc probe
>  
> @@ -3361,6 +3373,22 @@ EOF
>  fi
>  
>  ##########################################
> +# jemalloc probe
> +
> +if test "$jemalloc" = "yes" ; then
> +  cat > $TMPC << EOF
> +#include <stdlib.h>
> +int main(void) { malloc(1); return 0; }
> +EOF
> +
> +  if compile_prog "" "-ljemalloc" ; then
> +    LIBS="-ljemalloc $LIBS"
> +  else
> +    feature_not_found "jemalloc" "install jemalloc devel"
> +  fi
> +fi
> +
> +##########################################
>  # signalfd probe
>  signalfd="no"
>  cat > $TMPC << EOF
> @@ -4499,6 +4527,7 @@ echo "snappy support    $snappy"
>  echo "bzip2 support     $bzip2"
>  echo "NUMA host support $numa"
>  echo "tcmalloc support  $tcmalloc"
> +echo "jemalloc support  $jemalloc"
>  
>  if test "$sdl_too_old" = "yes"; then
>  echo "-> Your SDL version is too old - please upgrade to have SDL support"
>