RE: Having issues trying to get the OSD up on a MIPS64!!!

[Prashanth Nednoor <Prashanth.Nednoor@taec.toshiba.com>]

[-- Attachment #1: Type: text/plain, Size: 6900 bytes --]

Hi Sage,

Thank you for the prompt response.
Is there anything in /dev/disk/by-partuuid/ or is it missing entirely?  
  Nothing , it was Missing Entirely. 
  GOOD NEWS:  I worked around  this issue, if I set my journal path in the /etc/ceph.conf.

My udev version is
udevd --version
164 

I still see the segfaults, I have attached details.
I put the osd debug logs(osd-output.txt) and the leveldb_bt(leveldb_bt.txt).
Looks like we have an issue in leveldb....

HERE IS THE BACK TRACE: I have attached the gdb before running it.
#0  0x77f68ee0 in leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const* const&, leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::Node**) const () from /usr/local/lib/libleveldb.so.1
#1  0x77f69054 in leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::Insert(char const* const&) () from /usr/local/lib/libleveldb.so.1
#2  0x77f68618 in leveldb::MemTable::Add(unsigned long long, leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) ()
   from /usr/local/lib/libleveldb.so.1
#3  0x77f7e434 in leveldb::(anonymous namespace)::MemTableInserter::Put(leveldb::Slice const&, leveldb::Slice const&) ()
   from /usr/local/lib/libleveldb.so.1
#4  0x77f7e93c in leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const () from /usr/local/lib/libleveldb.so.1
#5  0x77f7eb8c in leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch const*, leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1
#6  0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions const&, leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1
#7  0x00a5dda0 in LevelDBStore::submit_transaction_sync (this=0x1f77d10, t=<value optimized out>) at os/LevelDBStore.cc:146
#8  0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0, spos=0x72cfe3b8) at os/DBObjectMap.cc:1126
#9  0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, fd=17, spos=..., hoid=0x0, in_progress=false) at os/FileStore.cc:2070
#10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450, cid=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at os/FileStore.cc:2047
#11 0x009b2138 in FileStore::_create_collection (this=0x1f72450, c=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at os/FileStore.cc:4753
#12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, t=..., op_seq=<value optimized out>, trans_num=0, handle=0x72cfec3c) at os/FileStore.cc:2413
#13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450, tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952
#14 0x009eb858 in FileStore::_do_op (this=0x1f72450, osr=0x1f801b8, handle=...) at os/FileStore.cc:1761
#15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0, wt=0x1f7ea90) at common/WorkQueue.cc:128
#16 0x00c91b94 in ThreadPool::WorkThread::entry() ()
#17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0
#18 0x777c1738 in ?? () from /lib/libc.so.6

Do  I need to set any variable to set the cache size etcetc in ceph.conf.
I only have osd_leveldb_cache_size=5242880 for now.


Thanks
Prashanth







-----Original Message-----
From: Sage Weil [mailto:sage@newdream.net] 
Sent: Thursday, October 23, 2014 5:54 PM
To: Prashanth Nednoor
Cc: ceph-devel@vger.kernel.org
Subject: Re: Having issues trying to get the OSD up on a MIPS64!!!

Hi Prashanth,

On Thu, 23 Oct 2014, Prashanth Nednoor wrote:
> Hello Everyone,
> 
> We are using ceph-0.86, good news is we were able to compile and load 
> all the libraries and binaries needed to configure a CEPH-OSD on MIPS 
> 64 platform. The CEPH monitor is also able to detect the OSD, but not 
> up yet, as the osd activate failed.
> Since we don?t have the required CEPH deploy utility for MIPS64, we 
> are following the manual procedure to create and activate an OSD.
> We have disabled authentication between the clients and the OSD?s for 
> now.
> 
> Has any body tried CEPH on a MIPS64?
> /dev/sda is a 2TB local hard drive.
> 
> This is how my partition looks after ceph-disk-prepare 
> /home/prashan/ceph-0.86/src# parted GNU Parted 2.3 Using /dev/sda 
> Welcome to GNU Parted! Type 'help' to view a list of commands.
> (parted) p
> Model: ATA TOSHIBA MQ01ABB2 (scsi)
> Disk /dev/sda: 2000GB
> Sector size (logical/physical): 512B/4096B Partition Table: gpt
> 
> Number  Start   End     Size    File system  Name          Flags
>  2      1049kB  5369MB  5368MB               ceph journal
>  1      5370MB  2000GB  1995GB  xfs          ceph data
> 
> 
> 
> The following are the steps to create an OSD
> 1)	ceph-disk zap /dev/sda
> 2)	ceph-disk-prepare --cluster  f615496c-b40a-4905-bbcd-
> 2d3e181ff21a --fs-type xfs /dev/sda
> 3)	mount /dev/sda1 /var/lib/ceph/osd/ceph-0/
> 4)	ceph-osd -i 0 ?mkfs is giving an error , 
> filestore(/var/lib/ceph/osd/ceph-0) could not find
> 23c2fcde/osd_superblock/0//-1 in index: (2) No such file.
> After this it segfaults. We have analyzed this further with the help 
> of strace and root caused this as objectmap file reading issue.
> open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", O_RDONLY) = 
> 11, the first time it reads 32k, the read succeeds with 63 bytes and 
> it tries to read again with 27k and the read returns 0 bytes and the 
> CEPH osd segfaults.

Can you generate a full log with --debug-osd 20 --debug-filestore 20 --debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere?  It should tell us where things are going wrong.  In particular, we want to see if that file/object is being written properly.  It will also have a backtrace showing exactly where it crashed.

> Please note that ceph-disk prepare creates a journal in a path which 
> is not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb).
> So after step3 above I remove this journal below and manually create a 
> journal file before doing step4 above.
> 
> 
> ls -l /var/lib/ceph/osd/ceph-0/
> total 16
> -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid
> -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid lrwxrwxrwx 1 root root 58 
> Oct 22 21:40 journal -> /dev/disk/by- 
> partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb

Is there anything in /dev/disk/by-partuuid/ or is it missing entirely?  
Maybe you have an old udev.  What distro is this?

sage

> -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid
> -rw-r--r-- 1 root root 21 Oct 22 21:40 magic
> 
> Any pointers to move ahead will be greatly appreciated??
> 
> thanks
> Prashanth
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
> 
> 

[-- Attachment #2: leveldb_bt.txt --]
[-- Type: text/plain, Size: 2548 bytes --]


#0  0x77f68ee0 in leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const* const&, leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::Node**) const () from /usr/local/lib/libleveldb.so.1
#1  0x77f69054 in leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::Insert(char const* const&) () from /usr/local/lib/libleveldb.so.1
#2  0x77f68618 in leveldb::MemTable::Add(unsigned long long, leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) ()
   from /usr/local/lib/libleveldb.so.1
#3  0x77f7e434 in leveldb::(anonymous namespace)::MemTableInserter::Put(leveldb::Slice const&, leveldb::Slice const&) ()
   from /usr/local/lib/libleveldb.so.1
#4  0x77f7e93c in leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const () from /usr/local/lib/libleveldb.so.1
#5  0x77f7eb8c in leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch const*, leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1
#6  0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions const&, leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1
#7  0x00a5dda0 in LevelDBStore::submit_transaction_sync (this=0x1f77d10, t=<value optimized out>) at os/LevelDBStore.cc:146
#8  0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0, spos=0x72cfe3b8) at os/DBObjectMap.cc:1126
#9  0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, fd=17, spos=..., hoid=0x0, in_progress=false) at os/FileStore.cc:2070
#10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450, cid=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at os/FileStore.cc:2047
#11 0x009b2138 in FileStore::_create_collection (this=0x1f72450, c=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at os/FileStore.cc:4753
#12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, t=..., op_seq=<value optimized out>, trans_num=0, handle=0x72cfec3c) at os/FileStore.cc:2413
#13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450, tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952
#14 0x009eb858 in FileStore::_do_op (this=0x1f72450, osr=0x1f801b8, handle=...) at os/FileStore.cc:1761
#15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0, wt=0x1f7ea90) at common/WorkQueue.cc:128
#16 0x00c91b94 in ThreadPool::WorkThread::entry() ()
#17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0
#18 0x777c1738 in ?? () from /lib/libc.so.6

[-- Attachment #3: osd-output.txt --]
[-- Type: text/plain, Size: 12344 bytes --]

This is were my journal points to
/var/lib/ceph/osd/ceph-0/journal -> /dev/disk/by-partuuid/9a868ac8-5ec8-4047-aa4c-8e9e6ce0

In the above output, for my system The path "/dev/disk/by-partuuid/" is missing.

udevd --version
164


gdb --args ceph-osd -i=0 --mkfs --debug-osd 20 --debug-filestore 20 --debug-journal 20 --mkfs -d
root@octeon:/home/prashan/ceph-0.86/src# ./ceph-osd -i=0 --mkfs --debug-osd 20 --debug-filestore 20 --debug-journal 20 --mkfs -d
 ./ceph-osd -i=0 --mkfs --debug-osd 20 --debug-filestore 20 --debug-journal 20 --mkfs -d
2014-10-24 19:26:59.981239 7760b000  0 ceph version 0.86 (97dcc0539dfa7dac3de74852305d51580b7b1f82), process ceph-osd, pid 3563
2014-10-24 19:26:59.982021 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) dump_stop
2014-10-24 19:26:59.983418 7760b000  1 filestore(/var/lib/ceph/osd/ceph-0) mkfs in /var/lib/ceph/osd/ceph-0
2014-10-24 19:27:00.001999 7760b000  1 filestore(/var/lib/ceph/osd/ceph-0) mkfs fsid is already set to 536991b9-cc6d-4023-860d-53ad361dce7e
2014-10-24 19:27:00.126542 7760b000  0 filestore(/var/lib/ceph/osd/ceph-0) backend generic (magic 0x58465342)
2014-10-24 19:27:00.126590 7760b000  1 filestore(/var/lib/ceph/osd/ceph-0)  disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs
2014-10-24 19:27:00.338490 7760b000  1 filestore(/var/lib/ceph/osd/ceph-0) leveldb db exists/created
2014-10-24 19:27:00.338577 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) open_journal at /dev/sda2
2014-10-24 19:27:00.338752 7760b000 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056)
2014-10-24 19:27:00.345391 7760b000 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/dev/sda2)
2014-10-24 19:27:00.345793 7760b000  1 journal _open /dev/sda2 fd 8: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-10-24 19:27:00.345857 7760b000 10 journal read_header
2014-10-24 19:27:00.360413 7760b000 10 journal header: block_size 4096 alignment 4096 max_size 5367660544
2014-10-24 19:27:00.360442 7760b000 10 journal header: start 4096
2014-10-24 19:27:00.360448 7760b000 10 journal  write_pos 0
2014-10-24 19:27:00.360485 7760b000 -1 journal check: ondisk fsid 8b1f947b-0a47-4f4e-bcd7-57ed33d5d300 doesn't match expected 536991b9-cc6d-4023-860d-53ad361dce7e, invalid (someone else's?) journal
2014-10-24 19:27:00.360636 7760b000  2 journal create /dev/sda2 fsid 536991b9-cc6d-4023-860d-53ad361dce7e
2014-10-24 19:27:00.360726 7760b000 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056)
2014-10-24 19:27:00.372395 7760b000 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/dev/sda2)
2014-10-24 19:27:00.372554 7760b000  1 journal _open /dev/sda2 fd 8: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-10-24 19:27:00.372588 7760b000 10 journal header: block_size 4096 alignment 4096 max_size 5367660544
2014-10-24 19:27:00.372596 7760b000 10 journal header: start 4096
2014-10-24 19:27:00.372601 7760b000 10 journal  write_pos 0
2014-10-24 19:27:00.393775 7760b000  2 journal create done
2014-10-24 19:27:00.394579 7760b000  0 filestore(/var/lib/ceph/osd/ceph-0) mkjournal created journal on /dev/sda2
2014-10-24 19:27:00.394712 7760b000  1 filestore(/var/lib/ceph/osd/ceph-0) mkfs done in /var/lib/ceph/osd/ceph-0
2014-10-24 19:27:00.394774 7760b000  5 filestore(/var/lib/ceph/osd/ceph-0) basedir /var/lib/ceph/osd/ceph-0 journal /dev/sda2
2014-10-24 19:27:00.394897 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) mount fsid is 536991b9-cc6d-4023-860d-53ad361dce7e
2014-10-24 19:27:00.395107 7760b000  0 filestore(/var/lib/ceph/osd/ceph-0) backend generic (magic 0x58465342)
2014-10-24 19:27:00.526956 7760b000  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work
2014-10-24 19:27:00.526999 7760b000  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2014-10-24 19:27:00.527553 7760b000  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall not supported
2014-10-24 19:27:00.527595 7760b000  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: no syncfs(2), must use sync(2).
2014-10-24 19:27:00.527600 7760b000  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: WARNING: multiple ceph-osd daemons on the same host will be slow
2014-10-24 19:27:00.528461 7760b000  5 filestore(/var/lib/ceph/osd/ceph-0) mount op_seq is 1
2014-10-24 19:27:00.660719 7760b000 20 filestore (init)dbobjectmap: seq is 1
2014-10-24 19:27:00.660791 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) open_journal at /dev/sda2
2014-10-24 19:27:00.660883 7760b000  0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2014-10-24 19:27:00.660901 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) list_collections
2014-10-24 19:27:00.661284 7760b000 10 journal journal_replay fs op_seq 1
2014-10-24 19:27:00.661293 752ff460 20 filestore(/var/lib/ceph/osd/ceph-0) sync_entry waiting for max_interval 5.000000
2014-10-24 19:27:00.661318 7760b000  2 journal open /dev/sda2 fsid 536991b9-cc6d-4023-860d-53ad361dce7e fs_op_seq 1
2014-10-24 19:27:00.661377 7760b000 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056)
2014-10-24 19:27:00.668052 7760b000 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/dev/sda2)
2014-10-24 19:27:00.668387 7760b000  1 journal _open /dev/sda2 fd 14: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-10-24 19:27:00.668426 7760b000 10 journal read_header
2014-10-24 19:27:00.682659 7760b000 10 journal header: block_size 4096 alignment 4096 max_size 5367660544
2014-10-24 19:27:00.682686 7760b000 10 journal header: start 4096
2014-10-24 19:27:00.682692 7760b000 10 journal  write_pos 4096
2014-10-24 19:27:00.682705 7760b000 10 journal open header.fsid = 536991b9-cc6d-4023-860d-53ad361dce7e
2014-10-24 19:27:00.683010 7760b000  2 journal No further valid entries found, journal is most likely valid
2014-10-24 19:27:00.683024 7760b000 10 journal open reached end of journal.
2014-10-24 19:27:00.683070 7760b000  2 journal No further valid entries found, journal is most likely valid
2014-10-24 19:27:00.683078 7760b000  3 journal journal_replay: end of journal, done.
2014-10-24 19:27:00.683084 7760b000 10 journal make_writeable
2014-10-24 19:27:00.683372 7760b000 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056)
2014-10-24 19:27:00.694396 7760b000 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/dev/sda2)
2014-10-24 19:27:00.694542 7760b000  1 journal _open /dev/sda2 fd 14: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-10-24 19:27:00.694677 74aff460 10 journal write_thread_entry start
2014-10-24 19:27:00.694719 74aff460 20 journal prepare_multi_write queue_pos now 4096
2014-10-24 19:27:00.694753 73aff460 10 journal write_finish_thread_entry enter
2014-10-24 19:27:00.694759 7760b000 10 journal journal_start
2014-10-24 19:27:00.694773 73aff460 20 journal write_finish_thread_entry sleeping
2014-10-24 19:27:00.694867 74aff460 15 journal do_aio_write writing 4096~0 + header
2014-10-24 19:27:00.694890 74aff460 20 journal write_aio_bl 0~4096 seq 0
2014-10-24 19:27:00.694958 74aff460 20 journal write_aio_bl .. 0~4096 in 1
2014-10-24 19:27:00.695129 74aff460 20 journal write_aio_bl 4096~0 seq 0
2014-10-24 19:27:00.695306 73aff460 20 journal write_finish_thread_entry waiting for aio(s)
2014-10-24 19:27:00.695306 74aff460  5 journal put_throttle finished 0 ops and 0 bytes, now 0 ops and 0 bytes
2014-10-24 19:27:00.695333 74aff460 20 journal write_thread_entry going to sleep
2014-10-24 19:27:00.695357 73aff460 10 journal write_finish_thread_entry aio 0~4096 done
2014-10-24 19:27:00.695373 73aff460 20 journal check_aio_completion
2014-10-24 19:27:00.695377 73aff460 20 journal check_aio_completion completed seq 0 0~4096
2014-10-24 19:27:00.695392 73aff460 20 journal write_finish_thread_entry sleeping
2014-10-24 19:27:00.695822 7760b000 15 filestore(/var/lib/ceph/osd/ceph-0) read meta/23c2fcde/osd_superblock/0//-1 0~0
2014-10-24 19:27:00.696088 7760b000 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2014-10-24 19:27:00.696207 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) FileStore::read(meta/23c2fcde/osd_superblock/0//-1) open error: (2) No such file or directory
2014-10-24 19:27:00.696479 7760b000  5 filestore(/var/lib/ceph/osd/ceph-0) queue_transactions new osr(default 0x1f72ac8)/0x1f72ac8
2014-10-24 19:27:00.696612 7760b000 10 journal op_submit_start 2
2014-10-24 19:27:00.696692 7760b000  5 filestore(/var/lib/ceph/osd/ceph-0) queue_transactions (writeahead) 2 0x7f73ea18
2014-10-24 19:27:00.696766 7760b000 10 journal op_journal_transactions 2 0x7f73ea18
2014-10-24 19:27:00.696908 7760b000  5 journal submit_entry seq 2 len 534 (0x7630c3f8)
2014-10-24 19:27:00.697037 7760b000 10 journal op_submit_finish 2
2014-10-24 19:27:00.697046 74aff460 20 journal write_thread_entry woke up
2014-10-24 19:27:00.697088 74aff460 10 journal room 5367656447 max_size 5367660544 pos 4096 header.start 4096 top 4096
2014-10-24 19:27:00.697104 74aff460 10 journal check_for_full at 4096 : 4096 < 5367656447
2014-10-24 19:27:00.697115 74aff460 15 journal prepare_single_write 1 will write 4096 : seq 2 len 534 -> 4096 (head 40 pre_pad 0 ebl 534 post_pad 3482 tail 40) (ebl alignment -1)
2014-10-24 19:27:00.697166 74aff460 20 journal prepare_multi_write queue_pos now 8192
2014-10-24 19:27:00.697176 74aff460 15 journal do_aio_write writing 4096~4096
2014-10-24 19:27:00.697226 74aff460 20 journal write_aio_bl 4096~4096 seq 2
2014-10-24 19:27:00.697244 74aff460 20 journal write_aio_bl .. 4096~4096 in 1
2014-10-24 19:27:00.704964 73aff460 20 journal write_finish_thread_entry waiting for aio(s)
2014-10-24 19:27:00.704958 74aff460  5 journal put_throttle finished 1 ops and 534 bytes, now 0 ops and 0 bytes
2014-10-24 19:27:00.704995 74aff460 20 journal write_thread_entry going to sleep
2014-10-24 19:27:00.705062 73aff460 10 journal write_finish_thread_entry aio 4096~4096 done
2014-10-24 19:27:00.705078 73aff460 20 journal check_aio_completion
2014-10-24 19:27:00.705082 73aff460 20 journal check_aio_completion completed seq 2 4096~4096
2014-10-24 19:27:00.705094 73aff460 20 journal check_aio_completion queueing finishers through seq 2
2014-10-24 19:27:00.705104 73aff460 10 journal queue_completions_thru seq 2 queueing seq 2 0x7630c3f8 lat 0.008102
2014-10-24 19:27:00.705137 73aff460 20 journal write_finish_thread_entry sleeping
2014-10-24 19:27:00.705154 732ff460  5 filestore(/var/lib/ceph/osd/ceph-0) _journaled_ahead 0x7630c0e0 seq 2 osr(default 0x1f72ac8) 0x7f73ea18
2014-10-24 19:27:00.705174 732ff460  5 filestore(/var/lib/ceph/osd/ceph-0) queue_op 0x7630c0e0 seq 2 osr(default 0x1f72ac8) 528 bytes   (queue has 1 ops and 528 bytes)
2014-10-24 19:27:00.705220 72aff460 10 journal op_apply_start 2 open_ops 0 -> 1
2014-10-24 19:27:00.705229 72aff460  5 filestore(/var/lib/ceph/osd/ceph-0) _do_op 0x7630c0e0 seq 2 osr(default 0x1f72ac8)/0x1f72ac8 start
2014-10-24 19:27:00.705242 72aff460 10 filestore(/var/lib/ceph/osd/ceph-0) _do_transaction on 0x7f73ea18
2014-10-24 19:27:00.705283 72aff460 15 filestore(/var/lib/ceph/osd/ceph-0) create_collection /var/lib/ceph/osd/ceph-0/current/meta
2014-10-24 19:27:00.733384 72aff460 10 filestore(/var/lib/ceph/osd/ceph-0) create_collection /var/lib/ceph/osd/ceph-0/current/meta = 0
2014-10-24 19:27:00.733695 72aff460 10 filestore(/var/lib/ceph/osd/ceph-0) _set_replay_guard 2.0.0
2014-10-24 19:27:00.771441 72aff460 20 filestore dbobjectmap: seq is 1
*** Caught signal (Segmentation fault) **
 in thread 72aff460
Segmentation fault