tcmalloc

tcmalloc

[Milosz Tanski]

I wanted to bring up an issue with Ceph's use of tcmalloc. I know that
in Ubuntu (12.04) Ceph uses the distro version of tcmalloc which
older.

I've personally ran into issues with tcmalloc for our application
where the amount of private space for our application (analytics
database) is listed in 10s of gigs but the heap profiler only reports
a couple hundreds megs of use.

It seams like tcmalloc while fast does a poor job with fragmentation.
In most cases I rather leave more memory to the OS to cache disk data
leave it in case the app needs to use it. I've observed issues with
the MDS when it's grown to a few gigs and it doesn't grow down even
after I've removed a 2/3 of the tree.

It's possible that this is fixed in newer versions of tcmalloc
(https://code.google.com/p/gperftools/issues/detail?id=368). I was
curious if anyone else audited this / ran into similar issues and
bring it up. For us, we moved to jemalloc which does a fine job
returning the memory back to the OS.

-- 
Milosz Tanski
CTO
10 East 53rd Street, 37th floor
New York, NY 10022

p: 646-253-9055
e: milosz@adfin.com

Re: tcmalloc

[Gregory Farnum]

On Tue, Dec 17, 2013 at 4:15 PM, Milosz Tanski <milosz@adfin.com> wrote:
> I wanted to bring up an issue with Ceph's use of tcmalloc. I know that
> in Ubuntu (12.04) Ceph uses the distro version of tcmalloc which
> older.
>
> I've personally ran into issues with tcmalloc for our application
> where the amount of private space for our application (analytics
> database) is listed in 10s of gigs but the heap profiler only reports
> a couple hundreds megs of use.
>
> It seams like tcmalloc while fast does a poor job with fragmentation.
> In most cases I rather leave more memory to the OS to cache disk data
> leave it in case the app needs to use it. I've observed issues with
> the MDS when it's grown to a few gigs and it doesn't grow down even
> after I've removed a 2/3 of the tree.
>
> It's possible that this is fixed in newer versions of tcmalloc
> (https://code.google.com/p/gperftools/issues/detail?id=368). I was
> curious if anyone else audited this / ran into similar issues and
> bring it up. For us, we moved to jemalloc which does a fine job
> returning the memory back to the OS.

We've seen scattered instances of this happening on systems run by
external users (generally on Gentoo or maybe a Fedora build), but have
never experienced it anywhere that we could reproduce. Helpfully, when
that's the case it's always been easy to resolve by running the "heap
release" command (or whatever it actually is; search the docs) to make
tcmalloc give up its free RAM.
I haven't evaluated jemalloc, but in all the systems we've run it on
tcmalloc behaves perfectly under our workloads and provides some
helpful features (heap analysis, etc); the default system malloc on
the other hand actually did grow without bound.

I'm not sure your MDS memory use test is actually valid — removing
inodes from the tree leaves them in cache until they've been cleaned
up out of RADOS, and we have in the past had bugs related to things
not being trimmed. You'd really want to look at the tcmalloc stats on
the MDS to see if tcmalloc is misbehaving or not.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html