OSD memory usage during startup - advice needed

From: "Marcin Gibuła" <m.gibula@beyond.pl>
To: ceph-devel@vger.kernel.org
Subject: OSD memory usage during startup - advice needed
Date: Thu, 19 Nov 2015 19:16:13 +0100	[thread overview]
Message-ID: <564E11ED.7080001@beyond.pl> (raw)

Hi Ceph devs!

I'm trying to track down and fix huge memory usage when OSD is starting 
after unclean shutdown. Recently, after editing crush map, when 
backfills started, one of our OSDs died (it hit suicide timeout). It was 
also refusing to start again, crashing due to memory allocation failure 
(over 15G used) shortly after start.

Judging from debug output, the problem is in journal recovery, when it 
tries to delete object with huge (several milion keys - it is radosgw 
index* for bucket with over 50mln objects) amount of keys, using 
leveldb's rmkeys_by_prefix() method.

Looking at the source code, rmkeys_by_prefix() batches all operations 
into one list and then submit_transaction() executes them all atomically.

I'd love to write a patch for this issue, but it seems unfixable (or is 
it?) with current API and method behaviour. Could you offer any advice 
on how to proceed?

Backtrace below:

  1: /usr/bin/ceph-osd() [0xacd7ba]
  2: (()+0x10340) [0x7f9713500340]
  3: (gsignal()+0x39) [0x7f971199fcc9]
  4: (abort()+0x148) [0x7f97119a30d8]
  5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f97122aa535]
  6: (()+0x5e6d6) [0x7f97122a86d6]
  7: (()+0x5e703) [0x7f97122a8703]
  8: (()+0x5e922) [0x7f97122a8922]
  9: (()+0x12b1e) [0x7f9713720b1e]
  10: (tc_new()+0x1e0) [0x7f9713740a00]
  11: (std::string::_Rep::_S_create(unsigned long, unsigned long, 
std::allocator<char> const&)+0x59) [0x7f9712304209]
  12: (std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned 
long)+0x1b) [0x7f9712304dcb]
  13: (std::string::reserve(unsigned long)+0x34) [0x7f9712304e64]
  14: (std::string::append(char const*, unsigned long)+0x4f) 
[0x7f97123050af]
  15: 
(LevelDBStore::LevelDBTransactionImpl::rmkeys_by_prefix(std::string 
const&)+0xcf) [0x97c44f]
  16: 
(DBObjectMap::clear_header(std::tr1::shared_ptr<DBObjectMap::_Header>, 
std::tr1::shared_ptr<KeyValueDB::TransactionImpl>)+0xc1) [0xa63171]
  17: (DBObjectMap::_clear(std::tr1::shared_ptr<DBObjectMap::_Header>, 
std::tr1::shared_ptr<KeyValueDB::TransactionImpl>)+0x91) [0xa682b1]
  18: (DBObjectMap::clear(ghobject_t const&, SequencerPosition 
const*)+0x202) [0xa6b292]
  19: (FileStore::lfn_unlink(coll_t, ghobject_t const&, 
SequencerPosition const&, bool)+0x16b) [0x9154fb]
  20: (FileStore::_remove(coll_t, ghobject_t const&, SequencerPosition 
const&)+0x8b) [0x915f6b]
  21: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned 
long, int, ThreadPool::TPHandle*)+0x3174) [0x926434]
  22: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, 
std::allocator<ObjectStore::Transaction*> >&, unsigned long, 
ThreadPool::TPHandle*)+0x64) [0x92a3a4]
  23: (JournalingObjectStore::journal_replay(unsigned long)+0x5cb) 
[0x94355b]
  24: (FileStore::mount()+0x3bb6) [0x9139f6]
  25: (OSD::init()+0x259) [0x6c59b9]
  26: (main()+0x2860) [0x6527e0]
  27: (__libc_start_main()+0xf5) [0x7f971198aec5]
  28: /usr/bin/ceph-osd() [0x66b887]

I also suspect that deleting this object was also somehow responsible 
for initial crash, when OSD hit suicide timeout. Any advices on how to 
debug it further?

* - yes, I am aware of shared indexes, but that bucket was created 
pre-hammer and I can't move migrate it

-- 
mg

P.S. Please CC me, as I'm not subscribed.