Re: Memory Pooling and Containers

[Gregory Farnum <gfarnum@redhat.com>]

On Fri, Sep 30, 2016 at 7:22 AM, Sage Weil <sage@newdream.net> wrote:
> On Wed, 28 Sep 2016, Allen Samuels wrote:
>> Boost::pool works very well when you're allocated "same" sized objects.
>> That's not our situation, we're allocating lots of different sized
>> objects -- some small, some large. The only way that Boost::pool
>> supports that situation is to use the "ordered_free" operation to keep
>> the freelist sorted (if you don't use it then you'll get fragmentation
>> that'll prevent allocation of large objects -- even though there's
>> plenty of free memory). The implementation of the sorted freelist is
>> O(N). Which should work well for small pools, but that's the exact
>> opposite of the desired use for Ceph, we're targeting large pools (think
>> 1GB).
>>
>> I didn't word it very well, but my proposal doesn't actually change the
>> underlying malloc/free algorithm, rather it's intended to put some
>> statistics around memory usage so that we can self-trim our memory
>> pools.
>
> We were doing some heap profiling yesterday and one interesting thing is
> that the utilized heap reported by tcmalloc is about 1/2 the RSS.  We
> probably want to consider creating separate pools for the handful of
> objects that are consuming the bulk of the heap.

Off-hand I'd say this is almost certainly the result of bumping the
thread cache sizes way up; I don't think it used to be the case. What
settings were being used?

> We did this a few years back in the MDS and IIRC it helped significantly
> with memory utilization there.

More than a "few" years, but yes.
-Greg