From: Samuel Just <sjust@redhat.com>
To: "Marcin Gibuła" <m.gibula@beyond.pl>
Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Subject: Re: OSD memory usage during startup - advice needed
Date: Thu, 19 Nov 2015 14:26:24 -0800 [thread overview]
Message-ID: <CAN=+7FVRGm7Dh6uiL5G-giD32vw8XVP16bpi_8qgCJ+iC38atA@mail.gmail.com> (raw)
In-Reply-To: <564E316A.8050601@beyond.pl>
Actually, looks like Xiaoxi beat you to it for infernalis!
42a3ab95ec459042e92198fb061c8393146bd1b4
-Sam
On Thu, Nov 19, 2015 at 12:30 PM, Marcin Gibuła <m.gibula@beyond.pl> wrote:
>> Judging from debug output, the problem is in journal recovery, when it
>> tries to delete object with huge (several milion keys - it is radosgw
>> index* for bucket with over 50mln objects) amount of keys, using
>> leveldb's rmkeys_by_prefix() method.
>>
>> Looking at the source code, rmkeys_by_prefix() batches all operations
>> into one list and then submit_transaction() executes them all atomically.
>>
>> I'd love to write a patch for this issue, but it seems unfixable (or is
>> it?) with current API and method behaviour. Could you offer any advice
>> on how to proceed?
>
>
> Answering myself, could anyone verify if attached patch looks ok? Should
> reduce memory footprint a bit.
>
> When I first read this code, I assumed that data pointed by leveldb::Slice
> have to be reachable until db->Write is called.
>
> However, looking into leveldb and into its source code, there is no such
> requirement - leveldb makes its own copy of key, so we're effectivly
> doubling memory footprint for no reason.
>
> --
> mg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
prev parent reply other threads:[~2015-11-19 22:26 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-11-19 18:16 OSD memory usage during startup - advice needed Marcin Gibuła
2015-11-19 20:30 ` Marcin Gibuła
2015-11-19 22:26 ` Samuel Just [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAN=+7FVRGm7Dh6uiL5G-giD32vw8XVP16bpi_8qgCJ+iC38atA@mail.gmail.com' \
--to=sjust@redhat.com \
--cc=ceph-devel@vger.kernel.org \
--cc=m.gibula@beyond.pl \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.