From mboxrd@z Thu Jan 1 00:00:00 1970 From: Gregory Farnum Subject: Re: Memory Pooling and Containers Date: Fri, 30 Sep 2016 11:54:41 -0700 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Return-path: Received: from mail-it0-f53.google.com ([209.85.214.53]:34963 "EHLO mail-it0-f53.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933349AbcI3Syn (ORCPT ); Fri, 30 Sep 2016 14:54:43 -0400 Received: by mail-it0-f53.google.com with SMTP id r192so60632287ita.0 for ; Fri, 30 Sep 2016 11:54:43 -0700 (PDT) In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Sage Weil Cc: Allen Samuels , Jesse Williamson , Ceph Development On Fri, Sep 30, 2016 at 7:22 AM, Sage Weil wrote: > On Wed, 28 Sep 2016, Allen Samuels wrote: >> Boost::pool works very well when you're allocated "same" sized objects. >> That's not our situation, we're allocating lots of different sized >> objects -- some small, some large. The only way that Boost::pool >> supports that situation is to use the "ordered_free" operation to keep >> the freelist sorted (if you don't use it then you'll get fragmentation >> that'll prevent allocation of large objects -- even though there's >> plenty of free memory). The implementation of the sorted freelist is >> O(N). Which should work well for small pools, but that's the exact >> opposite of the desired use for Ceph, we're targeting large pools (think >> 1GB). >> >> I didn't word it very well, but my proposal doesn't actually change the >> underlying malloc/free algorithm, rather it's intended to put some >> statistics around memory usage so that we can self-trim our memory >> pools. > > We were doing some heap profiling yesterday and one interesting thing is > that the utilized heap reported by tcmalloc is about 1/2 the RSS. We > probably want to consider creating separate pools for the handful of > objects that are consuming the bulk of the heap. Off-hand I'd say this is almost certainly the result of bumping the thread cache sizes way up; I don't think it used to be the case. What settings were being used? > We did this a few years back in the MDS and IIRC it helped significantly > with memory utilization there. More than a "few" years, but yes. -Greg