* Having issues trying to get the OSD up on a MIPS64!!! @ 2014-10-23 1:28 Prashanth Nednoor 2014-10-24 0:53 ` Sage Weil 0 siblings, 1 reply; 12+ messages in thread From: Prashanth Nednoor @ 2014-10-23 1:28 UTC (permalink / raw) To: ceph-devel Hello Everyone, We are using ceph-0.86, good news is we were able to compile and load all the libraries and binaries needed to configure a CEPH-OSD on MIPS 64 platform. The CEPH monitor is also able to detect the OSD, but not up yet, as the osd activate failed. Since we don’t have the required CEPH deploy utility for MIPS64, we are following the manual procedure to create and activate an OSD. We have disabled authentication between the clients and the OSD’s for now. Has any body tried CEPH on a MIPS64? /dev/sda is a 2TB local hard drive. This is how my partition looks after ceph-disk-prepare /home/prashan/ceph-0.86/src# parted GNU Parted 2.3 Using /dev/sda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Model: ATA TOSHIBA MQ01ABB2 (scsi) Disk /dev/sda: 2000GB Sector size (logical/physical): 512B/4096B Partition Table: gpt Number Start End Size File system Name Flags 2 1049kB 5369MB 5368MB ceph journal 1 5370MB 2000GB 1995GB xfs ceph data The following are the steps to create an OSD 1) ceph-disk zap /dev/sda 2) ceph-disk-prepare --cluster f615496c-b40a-4905-bbcd- 2d3e181ff21a --fs-type xfs /dev/sda 3) mount /dev/sda1 /var/lib/ceph/osd/ceph-0/ 4) ceph-osd -i 0 –mkfs is giving an error , filestore(/var/lib/ceph/osd/ceph-0) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file. After this it segfaults. We have analyzed this further with the help of strace and root caused this as objectmap file reading issue. open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", O_RDONLY) = 11, the first time it reads 32k, the read succeeds with 63 bytes and it tries to read again with 27k and the read returns 0 bytes and the CEPH osd segfaults. Please note that ceph-disk prepare creates a journal in a path which is not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb). So after step3 above I remove this journal below and manually create a journal file before doing step4 above. ls -l /var/lib/ceph/osd/ceph-0/ total 16 -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid lrwxrwxrwx 1 root root 58 Oct 22 21:40 journal -> /dev/disk/by- partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid -rw-r--r-- 1 root root 21 Oct 22 21:40 magic Any pointers to move ahead will be greatly appreciated?? thanks Prashanth -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Having issues trying to get the OSD up on a MIPS64!!! 2014-10-23 1:28 Having issues trying to get the OSD up on a MIPS64!!! Prashanth Nednoor @ 2014-10-24 0:53 ` Sage Weil 2014-10-24 21:38 ` Prashanth Nednoor 0 siblings, 1 reply; 12+ messages in thread From: Sage Weil @ 2014-10-24 0:53 UTC (permalink / raw) To: Prashanth Nednoor; +Cc: ceph-devel Hi Prashanth, On Thu, 23 Oct 2014, Prashanth Nednoor wrote: > Hello Everyone, > > We are using ceph-0.86, good news is we were able to compile and load > all the libraries and binaries needed to configure a CEPH-OSD on MIPS 64 > platform. The CEPH monitor is also able to detect the OSD, but not up > yet, as the osd activate failed. > Since we don?t have the required CEPH deploy utility for MIPS64, we are > following the manual procedure to create and activate an OSD. > We have disabled authentication between the clients and the OSD?s for > now. > > Has any body tried CEPH on a MIPS64? > /dev/sda is a 2TB local hard drive. > > This is how my partition looks after ceph-disk-prepare > /home/prashan/ceph-0.86/src# parted > GNU Parted 2.3 > Using /dev/sda > Welcome to GNU Parted! Type 'help' to view a list of commands. > (parted) p > Model: ATA TOSHIBA MQ01ABB2 (scsi) > Disk /dev/sda: 2000GB > Sector size (logical/physical): 512B/4096B > Partition Table: gpt > > Number Start End Size File system Name Flags > 2 1049kB 5369MB 5368MB ceph journal > 1 5370MB 2000GB 1995GB xfs ceph data > > > > The following are the steps to create an OSD > 1) ceph-disk zap /dev/sda > 2) ceph-disk-prepare --cluster f615496c-b40a-4905-bbcd- > 2d3e181ff21a --fs-type xfs /dev/sda > 3) mount /dev/sda1 /var/lib/ceph/osd/ceph-0/ > 4) ceph-osd -i 0 ?mkfs is giving an error , > filestore(/var/lib/ceph/osd/ceph-0) could not find > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file. > After this it segfaults. We have analyzed this further with the help of > strace and root caused this as objectmap file reading issue. > open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", O_RDONLY) = 11, > the first time it reads 32k, the read succeeds with 63 bytes and it > tries to read again with 27k and the read returns 0 bytes and the CEPH > osd segfaults. Can you generate a full log with --debug-osd 20 --debug-filestore 20 --debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere? It should tell us where things are going wrong. In particular, we want to see if that file/object is being written properly. It will also have a backtrace showing exactly where it crashed. > Please note that ceph-disk prepare creates a journal in a path which is > not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb). > So after step3 above I remove this journal below and manually create a > journal file before doing step4 above. > > > ls -l /var/lib/ceph/osd/ceph-0/ > total 16 > -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid > -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid > lrwxrwxrwx 1 root root 58 Oct 22 21:40 journal -> /dev/disk/by- > partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb Is there anything in /dev/disk/by-partuuid/ or is it missing entirely? Maybe you have an old udev. What distro is this? sage > -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid > -rw-r--r-- 1 root root 21 Oct 22 21:40 magic > > Any pointers to move ahead will be greatly appreciated?? > > thanks > Prashanth > > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Having issues trying to get the OSD up on a MIPS64!!! 2014-10-24 0:53 ` Sage Weil @ 2014-10-24 21:38 ` Prashanth Nednoor 2014-10-25 0:47 ` Sage Weil 0 siblings, 1 reply; 12+ messages in thread From: Prashanth Nednoor @ 2014-10-24 21:38 UTC (permalink / raw) To: Sage Weil; +Cc: ceph-devel, Philip Kufeldt [-- Attachment #1: Type: text/plain, Size: 6900 bytes --] Hi Sage, Thank you for the prompt response. Is there anything in /dev/disk/by-partuuid/ or is it missing entirely? Nothing , it was Missing Entirely. GOOD NEWS: I worked around this issue, if I set my journal path in the /etc/ceph.conf. My udev version is udevd --version 164 I still see the segfaults, I have attached details. I put the osd debug logs(osd-output.txt) and the leveldb_bt(leveldb_bt.txt). Looks like we have an issue in leveldb.... HERE IS THE BACK TRACE: I have attached the gdb before running it. #0 0x77f68ee0 in leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const* const&, leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::Node**) const () from /usr/local/lib/libleveldb.so.1 #1 0x77f69054 in leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::Insert(char const* const&) () from /usr/local/lib/libleveldb.so.1 #2 0x77f68618 in leveldb::MemTable::Add(unsigned long long, leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) () from /usr/local/lib/libleveldb.so.1 #3 0x77f7e434 in leveldb::(anonymous namespace)::MemTableInserter::Put(leveldb::Slice const&, leveldb::Slice const&) () from /usr/local/lib/libleveldb.so.1 #4 0x77f7e93c in leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const () from /usr/local/lib/libleveldb.so.1 #5 0x77f7eb8c in leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch const*, leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1 #6 0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions const&, leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1 #7 0x00a5dda0 in LevelDBStore::submit_transaction_sync (this=0x1f77d10, t=<value optimized out>) at os/LevelDBStore.cc:146 #8 0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0, spos=0x72cfe3b8) at os/DBObjectMap.cc:1126 #9 0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, fd=17, spos=..., hoid=0x0, in_progress=false) at os/FileStore.cc:2070 #10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450, cid=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece. ) at os/FileStore.cc:2047 #11 0x009b2138 in FileStore::_create_collection (this=0x1f72450, c=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece. ) at os/FileStore.cc:4753 #12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, t=..., op_seq=<value optimized out>, trans_num=0, handle=0x72cfec3c) at os/FileStore.cc:2413 #13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450, tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952 #14 0x009eb858 in FileStore::_do_op (this=0x1f72450, osr=0x1f801b8, handle=...) at os/FileStore.cc:1761 #15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0, wt=0x1f7ea90) at common/WorkQueue.cc:128 #16 0x00c91b94 in ThreadPool::WorkThread::entry() () #17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0 #18 0x777c1738 in ?? () from /lib/libc.so.6 Do I need to set any variable to set the cache size etcetc in ceph.conf. I only have osd_leveldb_cache_size=5242880 for now. Thanks Prashanth -----Original Message----- From: Sage Weil [mailto:sage@newdream.net] Sent: Thursday, October 23, 2014 5:54 PM To: Prashanth Nednoor Cc: ceph-devel@vger.kernel.org Subject: Re: Having issues trying to get the OSD up on a MIPS64!!! Hi Prashanth, On Thu, 23 Oct 2014, Prashanth Nednoor wrote: > Hello Everyone, > > We are using ceph-0.86, good news is we were able to compile and load > all the libraries and binaries needed to configure a CEPH-OSD on MIPS > 64 platform. The CEPH monitor is also able to detect the OSD, but not > up yet, as the osd activate failed. > Since we don?t have the required CEPH deploy utility for MIPS64, we > are following the manual procedure to create and activate an OSD. > We have disabled authentication between the clients and the OSD?s for > now. > > Has any body tried CEPH on a MIPS64? > /dev/sda is a 2TB local hard drive. > > This is how my partition looks after ceph-disk-prepare > /home/prashan/ceph-0.86/src# parted GNU Parted 2.3 Using /dev/sda > Welcome to GNU Parted! Type 'help' to view a list of commands. > (parted) p > Model: ATA TOSHIBA MQ01ABB2 (scsi) > Disk /dev/sda: 2000GB > Sector size (logical/physical): 512B/4096B Partition Table: gpt > > Number Start End Size File system Name Flags > 2 1049kB 5369MB 5368MB ceph journal > 1 5370MB 2000GB 1995GB xfs ceph data > > > > The following are the steps to create an OSD > 1) ceph-disk zap /dev/sda > 2) ceph-disk-prepare --cluster f615496c-b40a-4905-bbcd- > 2d3e181ff21a --fs-type xfs /dev/sda > 3) mount /dev/sda1 /var/lib/ceph/osd/ceph-0/ > 4) ceph-osd -i 0 ?mkfs is giving an error , > filestore(/var/lib/ceph/osd/ceph-0) could not find > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file. > After this it segfaults. We have analyzed this further with the help > of strace and root caused this as objectmap file reading issue. > open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", O_RDONLY) = > 11, the first time it reads 32k, the read succeeds with 63 bytes and > it tries to read again with 27k and the read returns 0 bytes and the > CEPH osd segfaults. Can you generate a full log with --debug-osd 20 --debug-filestore 20 --debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere? It should tell us where things are going wrong. In particular, we want to see if that file/object is being written properly. It will also have a backtrace showing exactly where it crashed. > Please note that ceph-disk prepare creates a journal in a path which > is not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb). > So after step3 above I remove this journal below and manually create a > journal file before doing step4 above. > > > ls -l /var/lib/ceph/osd/ceph-0/ > total 16 > -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid > -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid lrwxrwxrwx 1 root root 58 > Oct 22 21:40 journal -> /dev/disk/by- > partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb Is there anything in /dev/disk/by-partuuid/ or is it missing entirely? Maybe you have an old udev. What distro is this? sage > -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid > -rw-r--r-- 1 root root 21 Oct 22 21:40 magic > > Any pointers to move ahead will be greatly appreciated?? > > thanks > Prashanth > > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to majordomo@vger.kernel.org More majordomo > info at http://vger.kernel.org/majordomo-info.html > > [-- Attachment #2: leveldb_bt.txt --] [-- Type: text/plain, Size: 2548 bytes --] #0 0x77f68ee0 in leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const* const&, leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::Node**) const () from /usr/local/lib/libleveldb.so.1 #1 0x77f69054 in leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::Insert(char const* const&) () from /usr/local/lib/libleveldb.so.1 #2 0x77f68618 in leveldb::MemTable::Add(unsigned long long, leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) () from /usr/local/lib/libleveldb.so.1 #3 0x77f7e434 in leveldb::(anonymous namespace)::MemTableInserter::Put(leveldb::Slice const&, leveldb::Slice const&) () from /usr/local/lib/libleveldb.so.1 #4 0x77f7e93c in leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const () from /usr/local/lib/libleveldb.so.1 #5 0x77f7eb8c in leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch const*, leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1 #6 0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions const&, leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1 #7 0x00a5dda0 in LevelDBStore::submit_transaction_sync (this=0x1f77d10, t=<value optimized out>) at os/LevelDBStore.cc:146 #8 0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0, spos=0x72cfe3b8) at os/DBObjectMap.cc:1126 #9 0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, fd=17, spos=..., hoid=0x0, in_progress=false) at os/FileStore.cc:2070 #10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450, cid=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece. ) at os/FileStore.cc:2047 #11 0x009b2138 in FileStore::_create_collection (this=0x1f72450, c=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece. ) at os/FileStore.cc:4753 #12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, t=..., op_seq=<value optimized out>, trans_num=0, handle=0x72cfec3c) at os/FileStore.cc:2413 #13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450, tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952 #14 0x009eb858 in FileStore::_do_op (this=0x1f72450, osr=0x1f801b8, handle=...) at os/FileStore.cc:1761 #15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0, wt=0x1f7ea90) at common/WorkQueue.cc:128 #16 0x00c91b94 in ThreadPool::WorkThread::entry() () #17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0 #18 0x777c1738 in ?? () from /lib/libc.so.6 [-- Attachment #3: osd-output.txt --] [-- Type: text/plain, Size: 12344 bytes --] This is were my journal points to /var/lib/ceph/osd/ceph-0/journal -> /dev/disk/by-partuuid/9a868ac8-5ec8-4047-aa4c-8e9e6ce0 In the above output, for my system The path "/dev/disk/by-partuuid/" is missing. udevd --version 164 gdb --args ceph-osd -i=0 --mkfs --debug-osd 20 --debug-filestore 20 --debug-journal 20 --mkfs -d root@octeon:/home/prashan/ceph-0.86/src# ./ceph-osd -i=0 --mkfs --debug-osd 20 --debug-filestore 20 --debug-journal 20 --mkfs -d ./ceph-osd -i=0 --mkfs --debug-osd 20 --debug-filestore 20 --debug-journal 20 --mkfs -d 2014-10-24 19:26:59.981239 7760b000 0 ceph version 0.86 (97dcc0539dfa7dac3de74852305d51580b7b1f82), process ceph-osd, pid 3563 2014-10-24 19:26:59.982021 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) dump_stop 2014-10-24 19:26:59.983418 7760b000 1 filestore(/var/lib/ceph/osd/ceph-0) mkfs in /var/lib/ceph/osd/ceph-0 2014-10-24 19:27:00.001999 7760b000 1 filestore(/var/lib/ceph/osd/ceph-0) mkfs fsid is already set to 536991b9-cc6d-4023-860d-53ad361dce7e 2014-10-24 19:27:00.126542 7760b000 0 filestore(/var/lib/ceph/osd/ceph-0) backend generic (magic 0x58465342) 2014-10-24 19:27:00.126590 7760b000 1 filestore(/var/lib/ceph/osd/ceph-0) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs 2014-10-24 19:27:00.338490 7760b000 1 filestore(/var/lib/ceph/osd/ceph-0) leveldb db exists/created 2014-10-24 19:27:00.338577 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) open_journal at /dev/sda2 2014-10-24 19:27:00.338752 7760b000 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056) 2014-10-24 19:27:00.345391 7760b000 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/dev/sda2) 2014-10-24 19:27:00.345793 7760b000 1 journal _open /dev/sda2 fd 8: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2014-10-24 19:27:00.345857 7760b000 10 journal read_header 2014-10-24 19:27:00.360413 7760b000 10 journal header: block_size 4096 alignment 4096 max_size 5367660544 2014-10-24 19:27:00.360442 7760b000 10 journal header: start 4096 2014-10-24 19:27:00.360448 7760b000 10 journal write_pos 0 2014-10-24 19:27:00.360485 7760b000 -1 journal check: ondisk fsid 8b1f947b-0a47-4f4e-bcd7-57ed33d5d300 doesn't match expected 536991b9-cc6d-4023-860d-53ad361dce7e, invalid (someone else's?) journal 2014-10-24 19:27:00.360636 7760b000 2 journal create /dev/sda2 fsid 536991b9-cc6d-4023-860d-53ad361dce7e 2014-10-24 19:27:00.360726 7760b000 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056) 2014-10-24 19:27:00.372395 7760b000 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/dev/sda2) 2014-10-24 19:27:00.372554 7760b000 1 journal _open /dev/sda2 fd 8: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2014-10-24 19:27:00.372588 7760b000 10 journal header: block_size 4096 alignment 4096 max_size 5367660544 2014-10-24 19:27:00.372596 7760b000 10 journal header: start 4096 2014-10-24 19:27:00.372601 7760b000 10 journal write_pos 0 2014-10-24 19:27:00.393775 7760b000 2 journal create done 2014-10-24 19:27:00.394579 7760b000 0 filestore(/var/lib/ceph/osd/ceph-0) mkjournal created journal on /dev/sda2 2014-10-24 19:27:00.394712 7760b000 1 filestore(/var/lib/ceph/osd/ceph-0) mkfs done in /var/lib/ceph/osd/ceph-0 2014-10-24 19:27:00.394774 7760b000 5 filestore(/var/lib/ceph/osd/ceph-0) basedir /var/lib/ceph/osd/ceph-0 journal /dev/sda2 2014-10-24 19:27:00.394897 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) mount fsid is 536991b9-cc6d-4023-860d-53ad361dce7e 2014-10-24 19:27:00.395107 7760b000 0 filestore(/var/lib/ceph/osd/ceph-0) backend generic (magic 0x58465342) 2014-10-24 19:27:00.526956 7760b000 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work 2014-10-24 19:27:00.526999 7760b000 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2014-10-24 19:27:00.527553 7760b000 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall not supported 2014-10-24 19:27:00.527595 7760b000 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: no syncfs(2), must use sync(2). 2014-10-24 19:27:00.527600 7760b000 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: WARNING: multiple ceph-osd daemons on the same host will be slow 2014-10-24 19:27:00.528461 7760b000 5 filestore(/var/lib/ceph/osd/ceph-0) mount op_seq is 1 2014-10-24 19:27:00.660719 7760b000 20 filestore (init)dbobjectmap: seq is 1 2014-10-24 19:27:00.660791 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) open_journal at /dev/sda2 2014-10-24 19:27:00.660883 7760b000 0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2014-10-24 19:27:00.660901 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) list_collections 2014-10-24 19:27:00.661284 7760b000 10 journal journal_replay fs op_seq 1 2014-10-24 19:27:00.661293 752ff460 20 filestore(/var/lib/ceph/osd/ceph-0) sync_entry waiting for max_interval 5.000000 2014-10-24 19:27:00.661318 7760b000 2 journal open /dev/sda2 fsid 536991b9-cc6d-4023-860d-53ad361dce7e fs_op_seq 1 2014-10-24 19:27:00.661377 7760b000 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056) 2014-10-24 19:27:00.668052 7760b000 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/dev/sda2) 2014-10-24 19:27:00.668387 7760b000 1 journal _open /dev/sda2 fd 14: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2014-10-24 19:27:00.668426 7760b000 10 journal read_header 2014-10-24 19:27:00.682659 7760b000 10 journal header: block_size 4096 alignment 4096 max_size 5367660544 2014-10-24 19:27:00.682686 7760b000 10 journal header: start 4096 2014-10-24 19:27:00.682692 7760b000 10 journal write_pos 4096 2014-10-24 19:27:00.682705 7760b000 10 journal open header.fsid = 536991b9-cc6d-4023-860d-53ad361dce7e 2014-10-24 19:27:00.683010 7760b000 2 journal No further valid entries found, journal is most likely valid 2014-10-24 19:27:00.683024 7760b000 10 journal open reached end of journal. 2014-10-24 19:27:00.683070 7760b000 2 journal No further valid entries found, journal is most likely valid 2014-10-24 19:27:00.683078 7760b000 3 journal journal_replay: end of journal, done. 2014-10-24 19:27:00.683084 7760b000 10 journal make_writeable 2014-10-24 19:27:00.683372 7760b000 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056) 2014-10-24 19:27:00.694396 7760b000 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/dev/sda2) 2014-10-24 19:27:00.694542 7760b000 1 journal _open /dev/sda2 fd 14: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2014-10-24 19:27:00.694677 74aff460 10 journal write_thread_entry start 2014-10-24 19:27:00.694719 74aff460 20 journal prepare_multi_write queue_pos now 4096 2014-10-24 19:27:00.694753 73aff460 10 journal write_finish_thread_entry enter 2014-10-24 19:27:00.694759 7760b000 10 journal journal_start 2014-10-24 19:27:00.694773 73aff460 20 journal write_finish_thread_entry sleeping 2014-10-24 19:27:00.694867 74aff460 15 journal do_aio_write writing 4096~0 + header 2014-10-24 19:27:00.694890 74aff460 20 journal write_aio_bl 0~4096 seq 0 2014-10-24 19:27:00.694958 74aff460 20 journal write_aio_bl .. 0~4096 in 1 2014-10-24 19:27:00.695129 74aff460 20 journal write_aio_bl 4096~0 seq 0 2014-10-24 19:27:00.695306 73aff460 20 journal write_finish_thread_entry waiting for aio(s) 2014-10-24 19:27:00.695306 74aff460 5 journal put_throttle finished 0 ops and 0 bytes, now 0 ops and 0 bytes 2014-10-24 19:27:00.695333 74aff460 20 journal write_thread_entry going to sleep 2014-10-24 19:27:00.695357 73aff460 10 journal write_finish_thread_entry aio 0~4096 done 2014-10-24 19:27:00.695373 73aff460 20 journal check_aio_completion 2014-10-24 19:27:00.695377 73aff460 20 journal check_aio_completion completed seq 0 0~4096 2014-10-24 19:27:00.695392 73aff460 20 journal write_finish_thread_entry sleeping 2014-10-24 19:27:00.695822 7760b000 15 filestore(/var/lib/ceph/osd/ceph-0) read meta/23c2fcde/osd_superblock/0//-1 0~0 2014-10-24 19:27:00.696088 7760b000 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2014-10-24 19:27:00.696207 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) FileStore::read(meta/23c2fcde/osd_superblock/0//-1) open error: (2) No such file or directory 2014-10-24 19:27:00.696479 7760b000 5 filestore(/var/lib/ceph/osd/ceph-0) queue_transactions new osr(default 0x1f72ac8)/0x1f72ac8 2014-10-24 19:27:00.696612 7760b000 10 journal op_submit_start 2 2014-10-24 19:27:00.696692 7760b000 5 filestore(/var/lib/ceph/osd/ceph-0) queue_transactions (writeahead) 2 0x7f73ea18 2014-10-24 19:27:00.696766 7760b000 10 journal op_journal_transactions 2 0x7f73ea18 2014-10-24 19:27:00.696908 7760b000 5 journal submit_entry seq 2 len 534 (0x7630c3f8) 2014-10-24 19:27:00.697037 7760b000 10 journal op_submit_finish 2 2014-10-24 19:27:00.697046 74aff460 20 journal write_thread_entry woke up 2014-10-24 19:27:00.697088 74aff460 10 journal room 5367656447 max_size 5367660544 pos 4096 header.start 4096 top 4096 2014-10-24 19:27:00.697104 74aff460 10 journal check_for_full at 4096 : 4096 < 5367656447 2014-10-24 19:27:00.697115 74aff460 15 journal prepare_single_write 1 will write 4096 : seq 2 len 534 -> 4096 (head 40 pre_pad 0 ebl 534 post_pad 3482 tail 40) (ebl alignment -1) 2014-10-24 19:27:00.697166 74aff460 20 journal prepare_multi_write queue_pos now 8192 2014-10-24 19:27:00.697176 74aff460 15 journal do_aio_write writing 4096~4096 2014-10-24 19:27:00.697226 74aff460 20 journal write_aio_bl 4096~4096 seq 2 2014-10-24 19:27:00.697244 74aff460 20 journal write_aio_bl .. 4096~4096 in 1 2014-10-24 19:27:00.704964 73aff460 20 journal write_finish_thread_entry waiting for aio(s) 2014-10-24 19:27:00.704958 74aff460 5 journal put_throttle finished 1 ops and 534 bytes, now 0 ops and 0 bytes 2014-10-24 19:27:00.704995 74aff460 20 journal write_thread_entry going to sleep 2014-10-24 19:27:00.705062 73aff460 10 journal write_finish_thread_entry aio 4096~4096 done 2014-10-24 19:27:00.705078 73aff460 20 journal check_aio_completion 2014-10-24 19:27:00.705082 73aff460 20 journal check_aio_completion completed seq 2 4096~4096 2014-10-24 19:27:00.705094 73aff460 20 journal check_aio_completion queueing finishers through seq 2 2014-10-24 19:27:00.705104 73aff460 10 journal queue_completions_thru seq 2 queueing seq 2 0x7630c3f8 lat 0.008102 2014-10-24 19:27:00.705137 73aff460 20 journal write_finish_thread_entry sleeping 2014-10-24 19:27:00.705154 732ff460 5 filestore(/var/lib/ceph/osd/ceph-0) _journaled_ahead 0x7630c0e0 seq 2 osr(default 0x1f72ac8) 0x7f73ea18 2014-10-24 19:27:00.705174 732ff460 5 filestore(/var/lib/ceph/osd/ceph-0) queue_op 0x7630c0e0 seq 2 osr(default 0x1f72ac8) 528 bytes (queue has 1 ops and 528 bytes) 2014-10-24 19:27:00.705220 72aff460 10 journal op_apply_start 2 open_ops 0 -> 1 2014-10-24 19:27:00.705229 72aff460 5 filestore(/var/lib/ceph/osd/ceph-0) _do_op 0x7630c0e0 seq 2 osr(default 0x1f72ac8)/0x1f72ac8 start 2014-10-24 19:27:00.705242 72aff460 10 filestore(/var/lib/ceph/osd/ceph-0) _do_transaction on 0x7f73ea18 2014-10-24 19:27:00.705283 72aff460 15 filestore(/var/lib/ceph/osd/ceph-0) create_collection /var/lib/ceph/osd/ceph-0/current/meta 2014-10-24 19:27:00.733384 72aff460 10 filestore(/var/lib/ceph/osd/ceph-0) create_collection /var/lib/ceph/osd/ceph-0/current/meta = 0 2014-10-24 19:27:00.733695 72aff460 10 filestore(/var/lib/ceph/osd/ceph-0) _set_replay_guard 2.0.0 2014-10-24 19:27:00.771441 72aff460 20 filestore dbobjectmap: seq is 1 *** Caught signal (Segmentation fault) ** in thread 72aff460 Segmentation fault ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Having issues trying to get the OSD up on a MIPS64!!! 2014-10-24 21:38 ` Prashanth Nednoor @ 2014-10-25 0:47 ` Sage Weil 2014-10-25 0:55 ` Philip Kufeldt 0 siblings, 1 reply; 12+ messages in thread From: Sage Weil @ 2014-10-25 0:47 UTC (permalink / raw) To: Prashanth Nednoor; +Cc: ceph-devel, Philip Kufeldt Hi Prashanth, On Fri, 24 Oct 2014, Prashanth Nednoor wrote: > Hi Sage, > > Thank you for the prompt response. > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely? > Nothing , it was Missing Entirely. > GOOD NEWS: I worked around this issue, if I set my journal path in the /etc/ceph.conf. > > My udev version is udevd --version 164 Hmm, that should be new enough, but it seems like it isn't setting up the links. What distro is it? On most systems it's /lib/udev/rules.d/60-persistent-storage.rules that does it. Maybe see if running partprobe /dev/sda or run 'udevadm monitor' and do 'udevadm trigger /dev/sda' in another terminal to see what happens. Or, work around it like you did. :) > I still see the segfaults, I have attached details. > I put the osd debug logs(osd-output.txt) and the leveldb_bt(leveldb_bt.txt). > Looks like we have an issue in leveldb.... Yeah, that looks like a problem with leveldb. What distro is this? What version leveldb? I don't actually know anything about MIPS.. what's teh wordsize and endianess? sage > > HERE IS THE BACK TRACE: I have attached the gdb before running it. > #0 0x77f68ee0 in leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const* const&, leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::Node**) const () from /usr/local/lib/libleveldb.so.1 > #1 0x77f69054 in leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::Insert(char const* const&) () from /usr/local/lib/libleveldb.so.1 > #2 0x77f68618 in leveldb::MemTable::Add(unsigned long long, leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) () > from /usr/local/lib/libleveldb.so.1 > #3 0x77f7e434 in leveldb::(anonymous namespace)::MemTableInserter::Put(leveldb::Slice const&, leveldb::Slice const&) () > from /usr/local/lib/libleveldb.so.1 > #4 0x77f7e93c in leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const () from /usr/local/lib/libleveldb.so.1 > #5 0x77f7eb8c in leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch const*, leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1 > #6 0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions const&, leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1 > #7 0x00a5dda0 in LevelDBStore::submit_transaction_sync (this=0x1f77d10, t=<value optimized out>) at os/LevelDBStore.cc:146 > #8 0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0, spos=0x72cfe3b8) at os/DBObjectMap.cc:1126 > #9 0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, fd=17, spos=..., hoid=0x0, in_progress=false) at os/FileStore.cc:2070 > #10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450, cid=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece. > ) at os/FileStore.cc:2047 > #11 0x009b2138 in FileStore::_create_collection (this=0x1f72450, c=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece. > ) at os/FileStore.cc:4753 > #12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, t=..., op_seq=<value optimized out>, trans_num=0, handle=0x72cfec3c) at os/FileStore.cc:2413 > #13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450, tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952 > #14 0x009eb858 in FileStore::_do_op (this=0x1f72450, osr=0x1f801b8, handle=...) at os/FileStore.cc:1761 > #15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0, wt=0x1f7ea90) at common/WorkQueue.cc:128 > #16 0x00c91b94 in ThreadPool::WorkThread::entry() () > #17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0 > #18 0x777c1738 in ?? () from /lib/libc.so.6 > > Do I need to set any variable to set the cache size etcetc in ceph.conf. > I only have osd_leveldb_cache_size=5242880 for now. > > > Thanks > Prashanth > > > > > > > > -----Original Message----- > From: Sage Weil [mailto:sage@newdream.net] > Sent: Thursday, October 23, 2014 5:54 PM > To: Prashanth Nednoor > Cc: ceph-devel@vger.kernel.org > Subject: Re: Having issues trying to get the OSD up on a MIPS64!!! > > Hi Prashanth, > > On Thu, 23 Oct 2014, Prashanth Nednoor wrote: > > Hello Everyone, > > > > We are using ceph-0.86, good news is we were able to compile and load > > all the libraries and binaries needed to configure a CEPH-OSD on MIPS > > 64 platform. The CEPH monitor is also able to detect the OSD, but not > > up yet, as the osd activate failed. > > Since we don?t have the required CEPH deploy utility for MIPS64, we > > are following the manual procedure to create and activate an OSD. > > We have disabled authentication between the clients and the OSD?s for > > now. > > > > Has any body tried CEPH on a MIPS64? > > /dev/sda is a 2TB local hard drive. > > > > This is how my partition looks after ceph-disk-prepare > > /home/prashan/ceph-0.86/src# parted GNU Parted 2.3 Using /dev/sda > > Welcome to GNU Parted! Type 'help' to view a list of commands. > > (parted) p > > Model: ATA TOSHIBA MQ01ABB2 (scsi) > > Disk /dev/sda: 2000GB > > Sector size (logical/physical): 512B/4096B Partition Table: gpt > > > > Number Start End Size File system Name Flags > > 2 1049kB 5369MB 5368MB ceph journal > > 1 5370MB 2000GB 1995GB xfs ceph data > > > > > > > > The following are the steps to create an OSD > > 1) ceph-disk zap /dev/sda > > 2) ceph-disk-prepare --cluster f615496c-b40a-4905-bbcd- > > 2d3e181ff21a --fs-type xfs /dev/sda > > 3) mount /dev/sda1 /var/lib/ceph/osd/ceph-0/ > > 4) ceph-osd -i 0 ?mkfs is giving an error , > > filestore(/var/lib/ceph/osd/ceph-0) could not find > > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file. > > After this it segfaults. We have analyzed this further with the help > > of strace and root caused this as objectmap file reading issue. > > open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", O_RDONLY) = > > 11, the first time it reads 32k, the read succeeds with 63 bytes and > > it tries to read again with 27k and the read returns 0 bytes and the > > CEPH osd segfaults. > > Can you generate a full log with --debug-osd 20 --debug-filestore 20 --debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere? It should tell us where things are going wrong. In particular, we want to see if that file/object is being written properly. It will also have a backtrace showing exactly where it crashed. > > > Please note that ceph-disk prepare creates a journal in a path which > > is not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb). > > So after step3 above I remove this journal below and manually create a > > journal file before doing step4 above. > > > > > > ls -l /var/lib/ceph/osd/ceph-0/ > > total 16 > > -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid > > -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid lrwxrwxrwx 1 root root 58 > > Oct 22 21:40 journal -> /dev/disk/by- > > partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely? > Maybe you have an old udev. What distro is this? > > sage > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid > > -rw-r--r-- 1 root root 21 Oct 22 21:40 magic > > > > Any pointers to move ahead will be greatly appreciated?? > > > > thanks > > Prashanth > > > > > > > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > > in the body of a message to majordomo@vger.kernel.org More majordomo > > info at http://vger.kernel.org/majordomo-info.html > > > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Having issues trying to get the OSD up on a MIPS64!!! 2014-10-25 0:47 ` Sage Weil @ 2014-10-25 0:55 ` Philip Kufeldt 2014-10-25 1:11 ` Sage Weil 0 siblings, 1 reply; 12+ messages in thread From: Philip Kufeldt @ 2014-10-25 0:55 UTC (permalink / raw) To: Sage Weil, Prashanth Nednoor; +Cc: ceph-devel 64 bit big endian > -----Original Message----- > From: Sage Weil [mailto:sage@newdream.net] > Sent: Friday, October 24, 2014 5:47 PM > To: Prashanth Nednoor > Cc: ceph-devel@vger.kernel.org; Philip Kufeldt > Subject: RE: Having issues trying to get the OSD up on a MIPS64!!! > > Hi Prashanth, > > On Fri, 24 Oct 2014, Prashanth Nednoor wrote: > > Hi Sage, > > > > Thank you for the prompt response. > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely? > > Nothing , it was Missing Entirely. > > GOOD NEWS: I worked around this issue, if I set my journal path in the > /etc/ceph.conf. > > > > My udev version is udevd --version 164 > > Hmm, that should be new enough, but it seems like it isn't setting up the > links. What distro is it? On most systems it's /lib/udev/rules.d/60-persistent- > storage.rules that does it. Maybe see if running partprobe /dev/sda or run > 'udevadm monitor' and do 'udevadm trigger /dev/sda' in another terminal to > see what happens. > > Or, work around it like you did. :) > > > I still see the segfaults, I have attached details. > > I put the osd debug logs(osd-output.txt) and the > leveldb_bt(leveldb_bt.txt). > > Looks like we have an issue in leveldb.... > > Yeah, that looks like a problem with leveldb. What distro is this? What > version leveldb? > > I don't actually know anything about MIPS.. what's teh wordsize and > endianess? > > sage > > > > > > HERE IS THE BACK TRACE: I have attached the gdb before running it. > > #0 0x77f68ee0 in leveldb::SkipList<char const*, > > leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const* > > const&, leveldb::SkipList<char const*, > > leveldb::MemTable::KeyComparator>::Node**) const () from > > /usr/local/lib/libleveldb.so.1 > > #1 0x77f69054 in leveldb::SkipList<char const*, > > leveldb::MemTable::KeyComparator>::Insert(char const* const&) () from > > /usr/local/lib/libleveldb.so.1 > > #2 0x77f68618 in leveldb::MemTable::Add(unsigned long long, > leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) () > > from /usr/local/lib/libleveldb.so.1 > > #3 0x77f7e434 in leveldb::(anonymous > namespace)::MemTableInserter::Put(leveldb::Slice const&, leveldb::Slice > const&) () > > from /usr/local/lib/libleveldb.so.1 > > #4 0x77f7e93c in > > leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const () > > from /usr/local/lib/libleveldb.so.1 > > #5 0x77f7eb8c in > > leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch const*, > > leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1 > > #6 0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions const&, > > leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1 > > #7 0x00a5dda0 in LevelDBStore::submit_transaction_sync > > (this=0x1f77d10, t=<value optimized out>) at os/LevelDBStore.cc:146 > > #8 0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0, > > spos=0x72cfe3b8) at os/DBObjectMap.cc:1126 > > #9 0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, fd=17, > > spos=..., hoid=0x0, in_progress=false) at os/FileStore.cc:2070 > > #10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450, > cid=DWARF-2 expression error: DW_OP_reg operations must be used either > alone or in conjuction with DW_OP_piece. > > ) at os/FileStore.cc:2047 > > #11 0x009b2138 in FileStore::_create_collection (this=0x1f72450, c=DWARF- > 2 expression error: DW_OP_reg operations must be used either alone or in > conjuction with DW_OP_piece. > > ) at os/FileStore.cc:4753 > > #12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, t=..., > > op_seq=<value optimized out>, trans_num=0, handle=0x72cfec3c) at > > os/FileStore.cc:2413 > > #13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450, > > tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952 > > #14 0x009eb858 in FileStore::_do_op (this=0x1f72450, osr=0x1f801b8, > > handle=...) at os/FileStore.cc:1761 > > #15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0, wt=0x1f7ea90) at > > common/WorkQueue.cc:128 > > #16 0x00c91b94 in ThreadPool::WorkThread::entry() () > > #17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0 > > #18 0x777c1738 in ?? () from /lib/libc.so.6 > > > > Do I need to set any variable to set the cache size etcetc in ceph.conf. > > I only have osd_leveldb_cache_size=5242880 for now. > > > > > > Thanks > > Prashanth > > > > > > > > > > > > > > > > -----Original Message----- > > From: Sage Weil [mailto:sage@newdream.net] > > Sent: Thursday, October 23, 2014 5:54 PM > > To: Prashanth Nednoor > > Cc: ceph-devel@vger.kernel.org > > Subject: Re: Having issues trying to get the OSD up on a MIPS64!!! > > > > Hi Prashanth, > > > > On Thu, 23 Oct 2014, Prashanth Nednoor wrote: > > > Hello Everyone, > > > > > > We are using ceph-0.86, good news is we were able to compile and > > > load all the libraries and binaries needed to configure a CEPH-OSD > > > on MIPS > > > 64 platform. The CEPH monitor is also able to detect the OSD, but > > > not up yet, as the osd activate failed. > > > Since we don?t have the required CEPH deploy utility for MIPS64, we > > > are following the manual procedure to create and activate an OSD. > > > We have disabled authentication between the clients and the OSD?s > > > for now. > > > > > > Has any body tried CEPH on a MIPS64? > > > /dev/sda is a 2TB local hard drive. > > > > > > This is how my partition looks after ceph-disk-prepare > > > /home/prashan/ceph-0.86/src# parted GNU Parted 2.3 Using /dev/sda > > > Welcome to GNU Parted! Type 'help' to view a list of commands. > > > (parted) p > > > Model: ATA TOSHIBA MQ01ABB2 (scsi) > > > Disk /dev/sda: 2000GB > > > Sector size (logical/physical): 512B/4096B Partition Table: gpt > > > > > > Number Start End Size File system Name Flags > > > 2 1049kB 5369MB 5368MB ceph journal > > > 1 5370MB 2000GB 1995GB xfs ceph data > > > > > > > > > > > > The following are the steps to create an OSD > > > 1) ceph-disk zap /dev/sda > > > 2) ceph-disk-prepare --cluster f615496c-b40a-4905-bbcd- > > > 2d3e181ff21a --fs-type xfs /dev/sda > > > 3) mount /dev/sda1 /var/lib/ceph/osd/ceph-0/ > > > 4) ceph-osd -i 0 ?mkfs is giving an error , > > > filestore(/var/lib/ceph/osd/ceph-0) could not find > > > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file. > > > After this it segfaults. We have analyzed this further with the help > > > of strace and root caused this as objectmap file reading issue. > > > open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", O_RDONLY) > = > > > 11, the first time it reads 32k, the read succeeds with 63 bytes and > > > it tries to read again with 27k and the read returns 0 bytes and the > > > CEPH osd segfaults. > > > > Can you generate a full log with --debug-osd 20 --debug-filestore 20 -- > debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere? It > should tell us where things are going wrong. In particular, we want to see if > that file/object is being written properly. It will also have a backtrace > showing exactly where it crashed. > > > > > Please note that ceph-disk prepare creates a journal in a path which > > > is not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492- > 080ad2a505cb). > > > So after step3 above I remove this journal below and manually create > > > a journal file before doing step4 above. > > > > > > > > > ls -l /var/lib/ceph/osd/ceph-0/ > > > total 16 > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid lrwxrwxrwx 1 root root > > > 58 Oct 22 21:40 journal -> /dev/disk/by- > > > partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb > > > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely? > > Maybe you have an old udev. What distro is this? > > > > sage > > > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid > > > -rw-r--r-- 1 root root 21 Oct 22 21:40 magic > > > > > > Any pointers to move ahead will be greatly appreciated?? > > > > > > thanks > > > Prashanth > > > > > > > > > > > > -- > > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > > > in the body of a message to majordomo@vger.kernel.org More > majordomo > > > info at http://vger.kernel.org/majordomo-info.html > > > > > > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Having issues trying to get the OSD up on a MIPS64!!! 2014-10-25 0:55 ` Philip Kufeldt @ 2014-10-25 1:11 ` Sage Weil 2014-10-27 2:31 ` Prashanth Nednoor 2014-10-27 4:14 ` Prashanth Nednoor 0 siblings, 2 replies; 12+ messages in thread From: Sage Weil @ 2014-10-25 1:11 UTC (permalink / raw) To: Philip Kufeldt; +Cc: Prashanth Nednoor, ceph-devel On Sat, 25 Oct 2014, Philip Kufeldt wrote: > 64 bit big endian My guess is that there is an endianness bug in leveldb then. I wonder who else has tried it on MIPS? sage > > > -----Original Message----- > > From: Sage Weil [mailto:sage@newdream.net] > > Sent: Friday, October 24, 2014 5:47 PM > > To: Prashanth Nednoor > > Cc: ceph-devel@vger.kernel.org; Philip Kufeldt > > Subject: RE: Having issues trying to get the OSD up on a MIPS64!!! > > > > Hi Prashanth, > > > > On Fri, 24 Oct 2014, Prashanth Nednoor wrote: > > > Hi Sage, > > > > > > Thank you for the prompt response. > > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely? > > > Nothing , it was Missing Entirely. > > > GOOD NEWS: I worked around this issue, if I set my journal path in the > > /etc/ceph.conf. > > > > > > My udev version is udevd --version 164 > > > > Hmm, that should be new enough, but it seems like it isn't setting up the > > links. What distro is it? On most systems it's /lib/udev/rules.d/60-persistent- > > storage.rules that does it. Maybe see if running partprobe /dev/sda or run > > 'udevadm monitor' and do 'udevadm trigger /dev/sda' in another terminal to > > see what happens. > > > > Or, work around it like you did. :) > > > > > I still see the segfaults, I have attached details. > > > I put the osd debug logs(osd-output.txt) and the > > leveldb_bt(leveldb_bt.txt). > > > Looks like we have an issue in leveldb.... > > > > Yeah, that looks like a problem with leveldb. What distro is this? What > > version leveldb? > > > > I don't actually know anything about MIPS.. what's teh wordsize and > > endianess? > > > > sage > > > > > > > > > > HERE IS THE BACK TRACE: I have attached the gdb before running it. > > > #0 0x77f68ee0 in leveldb::SkipList<char const*, > > > leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const* > > > const&, leveldb::SkipList<char const*, > > > leveldb::MemTable::KeyComparator>::Node**) const () from > > > /usr/local/lib/libleveldb.so.1 > > > #1 0x77f69054 in leveldb::SkipList<char const*, > > > leveldb::MemTable::KeyComparator>::Insert(char const* const&) () from > > > /usr/local/lib/libleveldb.so.1 > > > #2 0x77f68618 in leveldb::MemTable::Add(unsigned long long, > > leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) () > > > from /usr/local/lib/libleveldb.so.1 > > > #3 0x77f7e434 in leveldb::(anonymous > > namespace)::MemTableInserter::Put(leveldb::Slice const&, leveldb::Slice > > const&) () > > > from /usr/local/lib/libleveldb.so.1 > > > #4 0x77f7e93c in > > > leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const () > > > from /usr/local/lib/libleveldb.so.1 > > > #5 0x77f7eb8c in > > > leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch const*, > > > leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1 > > > #6 0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions const&, > > > leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1 > > > #7 0x00a5dda0 in LevelDBStore::submit_transaction_sync > > > (this=0x1f77d10, t=<value optimized out>) at os/LevelDBStore.cc:146 > > > #8 0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0, > > > spos=0x72cfe3b8) at os/DBObjectMap.cc:1126 > > > #9 0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, fd=17, > > > spos=..., hoid=0x0, in_progress=false) at os/FileStore.cc:2070 > > > #10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450, > > cid=DWARF-2 expression error: DW_OP_reg operations must be used either > > alone or in conjuction with DW_OP_piece. > > > ) at os/FileStore.cc:2047 > > > #11 0x009b2138 in FileStore::_create_collection (this=0x1f72450, c=DWARF- > > 2 expression error: DW_OP_reg operations must be used either alone or in > > conjuction with DW_OP_piece. > > > ) at os/FileStore.cc:4753 > > > #12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, t=..., > > > op_seq=<value optimized out>, trans_num=0, handle=0x72cfec3c) at > > > os/FileStore.cc:2413 > > > #13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450, > > > tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952 > > > #14 0x009eb858 in FileStore::_do_op (this=0x1f72450, osr=0x1f801b8, > > > handle=...) at os/FileStore.cc:1761 > > > #15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0, wt=0x1f7ea90) at > > > common/WorkQueue.cc:128 > > > #16 0x00c91b94 in ThreadPool::WorkThread::entry() () > > > #17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0 > > > #18 0x777c1738 in ?? () from /lib/libc.so.6 > > > > > > Do I need to set any variable to set the cache size etcetc in ceph.conf. > > > I only have osd_leveldb_cache_size=5242880 for now. > > > > > > > > > Thanks > > > Prashanth > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > From: Sage Weil [mailto:sage@newdream.net] > > > Sent: Thursday, October 23, 2014 5:54 PM > > > To: Prashanth Nednoor > > > Cc: ceph-devel@vger.kernel.org > > > Subject: Re: Having issues trying to get the OSD up on a MIPS64!!! > > > > > > Hi Prashanth, > > > > > > On Thu, 23 Oct 2014, Prashanth Nednoor wrote: > > > > Hello Everyone, > > > > > > > > We are using ceph-0.86, good news is we were able to compile and > > > > load all the libraries and binaries needed to configure a CEPH-OSD > > > > on MIPS > > > > 64 platform. The CEPH monitor is also able to detect the OSD, but > > > > not up yet, as the osd activate failed. > > > > Since we don?t have the required CEPH deploy utility for MIPS64, we > > > > are following the manual procedure to create and activate an OSD. > > > > We have disabled authentication between the clients and the OSD?s > > > > for now. > > > > > > > > Has any body tried CEPH on a MIPS64? > > > > /dev/sda is a 2TB local hard drive. > > > > > > > > This is how my partition looks after ceph-disk-prepare > > > > /home/prashan/ceph-0.86/src# parted GNU Parted 2.3 Using /dev/sda > > > > Welcome to GNU Parted! Type 'help' to view a list of commands. > > > > (parted) p > > > > Model: ATA TOSHIBA MQ01ABB2 (scsi) > > > > Disk /dev/sda: 2000GB > > > > Sector size (logical/physical): 512B/4096B Partition Table: gpt > > > > > > > > Number Start End Size File system Name Flags > > > > 2 1049kB 5369MB 5368MB ceph journal > > > > 1 5370MB 2000GB 1995GB xfs ceph data > > > > > > > > > > > > > > > > The following are the steps to create an OSD > > > > 1) ceph-disk zap /dev/sda > > > > 2) ceph-disk-prepare --cluster f615496c-b40a-4905-bbcd- > > > > 2d3e181ff21a --fs-type xfs /dev/sda > > > > 3) mount /dev/sda1 /var/lib/ceph/osd/ceph-0/ > > > > 4) ceph-osd -i 0 ?mkfs is giving an error , > > > > filestore(/var/lib/ceph/osd/ceph-0) could not find > > > > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file. > > > > After this it segfaults. We have analyzed this further with the help > > > > of strace and root caused this as objectmap file reading issue. > > > > open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", O_RDONLY) > > = > > > > 11, the first time it reads 32k, the read succeeds with 63 bytes and > > > > it tries to read again with 27k and the read returns 0 bytes and the > > > > CEPH osd segfaults. > > > > > > Can you generate a full log with --debug-osd 20 --debug-filestore 20 -- > > debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere? It > > should tell us where things are going wrong. In particular, we want to see if > > that file/object is being written properly. It will also have a backtrace > > showing exactly where it crashed. > > > > > > > Please note that ceph-disk prepare creates a journal in a path which > > > > is not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492- > > 080ad2a505cb). > > > > So after step3 above I remove this journal below and manually create > > > > a journal file before doing step4 above. > > > > > > > > > > > > ls -l /var/lib/ceph/osd/ceph-0/ > > > > total 16 > > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid > > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid lrwxrwxrwx 1 root root > > > > 58 Oct 22 21:40 journal -> /dev/disk/by- > > > > partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb > > > > > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely? > > > Maybe you have an old udev. What distro is this? > > > > > > sage > > > > > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid > > > > -rw-r--r-- 1 root root 21 Oct 22 21:40 magic > > > > > > > > Any pointers to move ahead will be greatly appreciated?? > > > > > > > > thanks > > > > Prashanth > > > > > > > > > > > > > > > > -- > > > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > > > > in the body of a message to majordomo@vger.kernel.org More > > majordomo > > > > info at http://vger.kernel.org/majordomo-info.html > > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Having issues trying to get the OSD up on a MIPS64!!! 2014-10-25 1:11 ` Sage Weil @ 2014-10-27 2:31 ` Prashanth Nednoor 2014-10-27 4:14 ` Prashanth Nednoor 1 sibling, 0 replies; 12+ messages in thread From: Prashanth Nednoor @ 2014-10-27 2:31 UTC (permalink / raw) To: Sage Weil, Philip Kufeldt; +Cc: ceph-devel Hi Sage, Leveldb version is 1.17. Thanks Prashanth -----Original Message----- From: Sage Weil [mailto:sage@newdream.net] Sent: Friday, October 24, 2014 6:11 PM To: Philip Kufeldt Cc: Prashanth Nednoor; ceph-devel@vger.kernel.org Subject: RE: Having issues trying to get the OSD up on a MIPS64!!! On Sat, 25 Oct 2014, Philip Kufeldt wrote: > 64 bit big endian My guess is that there is an endianness bug in leveldb then. I wonder who else has tried it on MIPS? sage > > > -----Original Message----- > > From: Sage Weil [mailto:sage@newdream.net] > > Sent: Friday, October 24, 2014 5:47 PM > > To: Prashanth Nednoor > > Cc: ceph-devel@vger.kernel.org; Philip Kufeldt > > Subject: RE: Having issues trying to get the OSD up on a MIPS64!!! > > > > Hi Prashanth, > > > > On Fri, 24 Oct 2014, Prashanth Nednoor wrote: > > > Hi Sage, > > > > > > Thank you for the prompt response. > > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely? > > > Nothing , it was Missing Entirely. > > > GOOD NEWS: I worked around this issue, if I set my journal > > > path in the > > /etc/ceph.conf. > > > > > > My udev version is udevd --version 164 > > > > Hmm, that should be new enough, but it seems like it isn't setting > > up the links. What distro is it? On most systems it's > > /lib/udev/rules.d/60-persistent- storage.rules that does it. Maybe > > see if running partprobe /dev/sda or run 'udevadm monitor' and do > > 'udevadm trigger /dev/sda' in another terminal to see what happens. > > > > Or, work around it like you did. :) > > > > > I still see the segfaults, I have attached details. > > > I put the osd debug logs(osd-output.txt) and the > > leveldb_bt(leveldb_bt.txt). > > > Looks like we have an issue in leveldb.... > > > > Yeah, that looks like a problem with leveldb. What distro is this? > > What version leveldb? > > > > I don't actually know anything about MIPS.. what's teh wordsize and > > endianess? > > > > sage > > > > > > > > > > HERE IS THE BACK TRACE: I have attached the gdb before running it. > > > #0 0x77f68ee0 in leveldb::SkipList<char const*, > > > leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const* > > > const&, leveldb::SkipList<char const*, > > > leveldb::MemTable::KeyComparator>::Node**) const () from > > > /usr/local/lib/libleveldb.so.1 > > > #1 0x77f69054 in leveldb::SkipList<char const*, > > > leveldb::MemTable::KeyComparator>::Insert(char const* const&) () > > > from > > > /usr/local/lib/libleveldb.so.1 > > > #2 0x77f68618 in leveldb::MemTable::Add(unsigned long long, > > leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) () > > > from /usr/local/lib/libleveldb.so.1 > > > #3 0x77f7e434 in leveldb::(anonymous > > namespace)::MemTableInserter::Put(leveldb::Slice const&, > > leveldb::Slice > > const&) () > > > from /usr/local/lib/libleveldb.so.1 > > > #4 0x77f7e93c in > > > leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const > > > () from /usr/local/lib/libleveldb.so.1 > > > #5 0x77f7eb8c in > > > leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch > > > const*, > > > leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1 > > > #6 0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions > > > const&, > > > leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1 > > > #7 0x00a5dda0 in LevelDBStore::submit_transaction_sync > > > (this=0x1f77d10, t=<value optimized out>) at > > > os/LevelDBStore.cc:146 > > > #8 0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0, > > > spos=0x72cfe3b8) at os/DBObjectMap.cc:1126 > > > #9 0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, > > > fd=17, spos=..., hoid=0x0, in_progress=false) at > > > os/FileStore.cc:2070 > > > #10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450, > > cid=DWARF-2 expression error: DW_OP_reg operations must be used > > either alone or in conjuction with DW_OP_piece. > > > ) at os/FileStore.cc:2047 > > > #11 0x009b2138 in FileStore::_create_collection (this=0x1f72450, > > > c=DWARF- > > 2 expression error: DW_OP_reg operations must be used either alone > > or in conjuction with DW_OP_piece. > > > ) at os/FileStore.cc:4753 > > > #12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, > > > t=..., op_seq=<value optimized out>, trans_num=0, > > > handle=0x72cfec3c) at > > > os/FileStore.cc:2413 > > > #13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450, > > > tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952 > > > #14 0x009eb858 in FileStore::_do_op (this=0x1f72450, > > > osr=0x1f801b8, > > > handle=...) at os/FileStore.cc:1761 > > > #15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0, > > > wt=0x1f7ea90) at > > > common/WorkQueue.cc:128 > > > #16 0x00c91b94 in ThreadPool::WorkThread::entry() () > > > #17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0 > > > #18 0x777c1738 in ?? () from /lib/libc.so.6 > > > > > > Do I need to set any variable to set the cache size etcetc in ceph.conf. > > > I only have osd_leveldb_cache_size=5242880 for now. > > > > > > > > > Thanks > > > Prashanth > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > From: Sage Weil [mailto:sage@newdream.net] > > > Sent: Thursday, October 23, 2014 5:54 PM > > > To: Prashanth Nednoor > > > Cc: ceph-devel@vger.kernel.org > > > Subject: Re: Having issues trying to get the OSD up on a MIPS64!!! > > > > > > Hi Prashanth, > > > > > > On Thu, 23 Oct 2014, Prashanth Nednoor wrote: > > > > Hello Everyone, > > > > > > > > We are using ceph-0.86, good news is we were able to compile and > > > > load all the libraries and binaries needed to configure a > > > > CEPH-OSD on MIPS > > > > 64 platform. The CEPH monitor is also able to detect the OSD, > > > > but not up yet, as the osd activate failed. > > > > Since we don?t have the required CEPH deploy utility for MIPS64, > > > > we are following the manual procedure to create and activate an OSD. > > > > We have disabled authentication between the clients and the > > > > OSD?s for now. > > > > > > > > Has any body tried CEPH on a MIPS64? > > > > /dev/sda is a 2TB local hard drive. > > > > > > > > This is how my partition looks after ceph-disk-prepare > > > > /home/prashan/ceph-0.86/src# parted GNU Parted 2.3 Using > > > > /dev/sda Welcome to GNU Parted! Type 'help' to view a list of commands. > > > > (parted) p > > > > Model: ATA TOSHIBA MQ01ABB2 (scsi) Disk /dev/sda: 2000GB Sector > > > > size (logical/physical): 512B/4096B Partition Table: gpt > > > > > > > > Number Start End Size File system Name Flags > > > > 2 1049kB 5369MB 5368MB ceph journal > > > > 1 5370MB 2000GB 1995GB xfs ceph data > > > > > > > > > > > > > > > > The following are the steps to create an OSD > > > > 1) ceph-disk zap /dev/sda > > > > 2) ceph-disk-prepare --cluster f615496c-b40a-4905-bbcd- > > > > 2d3e181ff21a --fs-type xfs /dev/sda > > > > 3) mount /dev/sda1 /var/lib/ceph/osd/ceph-0/ > > > > 4) ceph-osd -i 0 ?mkfs is giving an error , > > > > filestore(/var/lib/ceph/osd/ceph-0) could not find > > > > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file. > > > > After this it segfaults. We have analyzed this further with the > > > > help of strace and root caused this as objectmap file reading issue. > > > > open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", > > > > O_RDONLY) > > = > > > > 11, the first time it reads 32k, the read succeeds with 63 bytes > > > > and it tries to read again with 27k and the read returns 0 bytes > > > > and the CEPH osd segfaults. > > > > > > Can you generate a full log with --debug-osd 20 --debug-filestore > > > 20 -- > > debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere? > > It should tell us where things are going wrong. In particular, we > > want to see if that file/object is being written properly. It will > > also have a backtrace showing exactly where it crashed. > > > > > > > Please note that ceph-disk prepare creates a journal in a path > > > > which is not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492- > > 080ad2a505cb). > > > > So after step3 above I remove this journal below and manually > > > > create a journal file before doing step4 above. > > > > > > > > > > > > ls -l /var/lib/ceph/osd/ceph-0/ > > > > total 16 > > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid > > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid lrwxrwxrwx 1 root > > > > root > > > > 58 Oct 22 21:40 journal -> /dev/disk/by- > > > > partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb > > > > > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely? > > > Maybe you have an old udev. What distro is this? > > > > > > sage > > > > > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid > > > > -rw-r--r-- 1 root root 21 Oct 22 21:40 magic > > > > > > > > Any pointers to move ahead will be greatly appreciated?? > > > > > > > > thanks > > > > Prashanth > > > > > > > > > > > > > > > > -- > > > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > > > > in the body of a message to majordomo@vger.kernel.org More > > majordomo > > > > info at http://vger.kernel.org/majordomo-info.html > > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: Having issues trying to get the OSD up on a MIPS64!!! 2014-10-25 1:11 ` Sage Weil 2014-10-27 2:31 ` Prashanth Nednoor @ 2014-10-27 4:14 ` Prashanth Nednoor 2015-02-17 7:58 ` Bao Vu 1 sibling, 1 reply; 12+ messages in thread From: Prashanth Nednoor @ 2014-10-27 4:14 UTC (permalink / raw) To: Sage Weil, Philip Kufeldt; +Cc: ceph-devel Sage, Good news, I am able to create the OSD successfully, let's see what's in store next. It was an issue with leveldb1.17 not having either memory barrier or atomic operation support for DEBIAN MIPS??? Not even the latest version leveldb1.18 I pulled from https://github.com/google/leveldb. But this link talks about that https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=681945 So, I ported over the memory barrier/atomic fix for MIPS onto leveldb1.17... I had to look into the mips/barrier.h files on our eval board, to make sure We had the correct macros. Now, my osd creation is successful on the MIPS, created object store /var/lib/ceph/osd/ceph-0 journal /dev/sda2 for osd.0 fsid f615496c-b40a-4905-bbcd-2d3e181ff21a I have to start looking into the CLIENT/MONITOR side to make sure everything is good. Really thankful for your suggestions for this quick resolution, for now we are good, untill the next and then the next...... Thanks Prashanth -----Original Message----- From: Prashanth Nednoor Sent: Sunday, October 26, 2014 7:32 PM To: 'Sage Weil'; Philip Kufeldt Cc: ceph-devel@vger.kernel.org Subject: RE: Having issues trying to get the OSD up on a MIPS64!!! Hi Sage, Leveldb version is 1.17. Thanks Prashanth -----Original Message----- From: Sage Weil [mailto:sage@newdream.net] Sent: Friday, October 24, 2014 6:11 PM To: Philip Kufeldt Cc: Prashanth Nednoor; ceph-devel@vger.kernel.org Subject: RE: Having issues trying to get the OSD up on a MIPS64!!! On Sat, 25 Oct 2014, Philip Kufeldt wrote: > 64 bit big endian My guess is that there is an endianness bug in leveldb then. I wonder who else has tried it on MIPS? sage > > > -----Original Message----- > > From: Sage Weil [mailto:sage@newdream.net] > > Sent: Friday, October 24, 2014 5:47 PM > > To: Prashanth Nednoor > > Cc: ceph-devel@vger.kernel.org; Philip Kufeldt > > Subject: RE: Having issues trying to get the OSD up on a MIPS64!!! > > > > Hi Prashanth, > > > > On Fri, 24 Oct 2014, Prashanth Nednoor wrote: > > > Hi Sage, > > > > > > Thank you for the prompt response. > > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely? > > > Nothing , it was Missing Entirely. > > > GOOD NEWS: I worked around this issue, if I set my journal > > > path in the > > /etc/ceph.conf. > > > > > > My udev version is udevd --version 164 > > > > Hmm, that should be new enough, but it seems like it isn't setting > > up the links. What distro is it? On most systems it's > > /lib/udev/rules.d/60-persistent- storage.rules that does it. Maybe > > see if running partprobe /dev/sda or run 'udevadm monitor' and do > > 'udevadm trigger /dev/sda' in another terminal to see what happens. > > > > Or, work around it like you did. :) > > > > > I still see the segfaults, I have attached details. > > > I put the osd debug logs(osd-output.txt) and the > > leveldb_bt(leveldb_bt.txt). > > > Looks like we have an issue in leveldb.... > > > > Yeah, that looks like a problem with leveldb. What distro is this? > > What version leveldb? > > > > I don't actually know anything about MIPS.. what's teh wordsize and > > endianess? > > > > sage > > > > > > > > > > HERE IS THE BACK TRACE: I have attached the gdb before running it. > > > #0 0x77f68ee0 in leveldb::SkipList<char const*, > > > leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const* > > > const&, leveldb::SkipList<char const*, > > > leveldb::MemTable::KeyComparator>::Node**) const () from > > > /usr/local/lib/libleveldb.so.1 > > > #1 0x77f69054 in leveldb::SkipList<char const*, > > > leveldb::MemTable::KeyComparator>::Insert(char const* const&) () > > > from > > > /usr/local/lib/libleveldb.so.1 > > > #2 0x77f68618 in leveldb::MemTable::Add(unsigned long long, > > leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) () > > > from /usr/local/lib/libleveldb.so.1 > > > #3 0x77f7e434 in leveldb::(anonymous > > namespace)::MemTableInserter::Put(leveldb::Slice const&, > > leveldb::Slice > > const&) () > > > from /usr/local/lib/libleveldb.so.1 > > > #4 0x77f7e93c in > > > leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const > > > () from /usr/local/lib/libleveldb.so.1 > > > #5 0x77f7eb8c in > > > leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch > > > const*, > > > leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1 > > > #6 0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions > > > const&, > > > leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1 > > > #7 0x00a5dda0 in LevelDBStore::submit_transaction_sync > > > (this=0x1f77d10, t=<value optimized out>) at > > > os/LevelDBStore.cc:146 > > > #8 0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0, > > > spos=0x72cfe3b8) at os/DBObjectMap.cc:1126 > > > #9 0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, > > > fd=17, spos=..., hoid=0x0, in_progress=false) at > > > os/FileStore.cc:2070 > > > #10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450, > > cid=DWARF-2 expression error: DW_OP_reg operations must be used > > either alone or in conjuction with DW_OP_piece. > > > ) at os/FileStore.cc:2047 > > > #11 0x009b2138 in FileStore::_create_collection (this=0x1f72450, > > > c=DWARF- > > 2 expression error: DW_OP_reg operations must be used either alone > > or in conjuction with DW_OP_piece. > > > ) at os/FileStore.cc:4753 > > > #12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, > > > t=..., op_seq=<value optimized out>, trans_num=0, > > > handle=0x72cfec3c) at > > > os/FileStore.cc:2413 > > > #13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450, > > > tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952 > > > #14 0x009eb858 in FileStore::_do_op (this=0x1f72450, > > > osr=0x1f801b8, > > > handle=...) at os/FileStore.cc:1761 > > > #15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0, > > > wt=0x1f7ea90) at > > > common/WorkQueue.cc:128 > > > #16 0x00c91b94 in ThreadPool::WorkThread::entry() () > > > #17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0 > > > #18 0x777c1738 in ?? () from /lib/libc.so.6 > > > > > > Do I need to set any variable to set the cache size etcetc in ceph.conf. > > > I only have osd_leveldb_cache_size=5242880 for now. > > > > > > > > > Thanks > > > Prashanth > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > From: Sage Weil [mailto:sage@newdream.net] > > > Sent: Thursday, October 23, 2014 5:54 PM > > > To: Prashanth Nednoor > > > Cc: ceph-devel@vger.kernel.org > > > Subject: Re: Having issues trying to get the OSD up on a MIPS64!!! > > > > > > Hi Prashanth, > > > > > > On Thu, 23 Oct 2014, Prashanth Nednoor wrote: > > > > Hello Everyone, > > > > > > > > We are using ceph-0.86, good news is we were able to compile and > > > > load all the libraries and binaries needed to configure a > > > > CEPH-OSD on MIPS > > > > 64 platform. The CEPH monitor is also able to detect the OSD, > > > > but not up yet, as the osd activate failed. > > > > Since we don?t have the required CEPH deploy utility for MIPS64, > > > > we are following the manual procedure to create and activate an OSD. > > > > We have disabled authentication between the clients and the > > > > OSD?s for now. > > > > > > > > Has any body tried CEPH on a MIPS64? > > > > /dev/sda is a 2TB local hard drive. > > > > > > > > This is how my partition looks after ceph-disk-prepare > > > > /home/prashan/ceph-0.86/src# parted GNU Parted 2.3 Using > > > > /dev/sda Welcome to GNU Parted! Type 'help' to view a list of commands. > > > > (parted) p > > > > Model: ATA TOSHIBA MQ01ABB2 (scsi) Disk /dev/sda: 2000GB Sector > > > > size (logical/physical): 512B/4096B Partition Table: gpt > > > > > > > > Number Start End Size File system Name Flags > > > > 2 1049kB 5369MB 5368MB ceph journal > > > > 1 5370MB 2000GB 1995GB xfs ceph data > > > > > > > > > > > > > > > > The following are the steps to create an OSD > > > > 1) ceph-disk zap /dev/sda > > > > 2) ceph-disk-prepare --cluster f615496c-b40a-4905-bbcd- > > > > 2d3e181ff21a --fs-type xfs /dev/sda > > > > 3) mount /dev/sda1 /var/lib/ceph/osd/ceph-0/ > > > > 4) ceph-osd -i 0 ?mkfs is giving an error , > > > > filestore(/var/lib/ceph/osd/ceph-0) could not find > > > > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file. > > > > After this it segfaults. We have analyzed this further with the > > > > help of strace and root caused this as objectmap file reading issue. > > > > open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", > > > > O_RDONLY) > > = > > > > 11, the first time it reads 32k, the read succeeds with 63 bytes > > > > and it tries to read again with 27k and the read returns 0 bytes > > > > and the CEPH osd segfaults. > > > > > > Can you generate a full log with --debug-osd 20 --debug-filestore > > > 20 -- > > debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere? > > It should tell us where things are going wrong. In particular, we > > want to see if that file/object is being written properly. It will > > also have a backtrace showing exactly where it crashed. > > > > > > > Please note that ceph-disk prepare creates a journal in a path > > > > which is not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492- > > 080ad2a505cb). > > > > So after step3 above I remove this journal below and manually > > > > create a journal file before doing step4 above. > > > > > > > > > > > > ls -l /var/lib/ceph/osd/ceph-0/ > > > > total 16 > > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid > > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid lrwxrwxrwx 1 root > > > > root > > > > 58 Oct 22 21:40 journal -> /dev/disk/by- > > > > partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb > > > > > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely? > > > Maybe you have an old udev. What distro is this? > > > > > > sage > > > > > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid > > > > -rw-r--r-- 1 root root 21 Oct 22 21:40 magic > > > > > > > > Any pointers to move ahead will be greatly appreciated?? > > > > > > > > thanks > > > > Prashanth > > > > > > > > > > > > > > > > -- > > > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > > > > in the body of a message to majordomo@vger.kernel.org More > > majordomo > > > > info at http://vger.kernel.org/majordomo-info.html > > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Having issues trying to get the OSD up on a MIPS64!!! 2014-10-27 4:14 ` Prashanth Nednoor @ 2015-02-17 7:58 ` Bao Vu 2015-02-17 14:06 ` Sage Weil 0 siblings, 1 reply; 12+ messages in thread From: Bao Vu @ 2015-02-17 7:58 UTC (permalink / raw) To: ceph-devel Prashanth Nednoor <Prashanth.Nednoor <at> taec.toshiba.com> writes: > > Sage, > > Good news, I am able to create the OSD successfully, let's see what's in store next. > > It was an issue with leveldb1.17 not having either memory barrier or atomic operation support for DEBIAN MIPS??? > Not even the latest version leveldb1.18 I pulled from https://github.com/google/leveldb. > > But this link talks about that https://bugs.debian.org/cgi- bin/bugreport.cgi?bug=681945 > > So, I ported over the memory barrier/atomic fix for MIPS onto leveldb1.17... I had to look into the > mips/barrier.h files on our eval board, to make sure > We had the correct macros. > > Now, my osd creation is successful on the MIPS, created object store /var/lib/ceph/osd/ceph-0 journal > /dev/sda2 for osd.0 fsid f615496c-b40a-4905-bbcd-2d3e181ff21a > I have to start looking into the CLIENT/MONITOR side to make sure everything is good. > > Really thankful for your suggestions for this quick resolution, for now we are good, untill the next and > then the next...... > > Thanks > Prashanth > > -----Original Message----- > From: Prashanth Nednoor > Sent: Sunday, October 26, 2014 7:32 PM > To: 'Sage Weil'; Philip Kufeldt > Cc: ceph-devel <at> vger.kernel.org > Subject: RE: Having issues trying to get the OSD up on a MIPS64!!! > > Hi Sage, > > Leveldb version is 1.17. > > Thanks > Prashanth > > -----Original Message----- > From: Sage Weil [mailto:sage <at> newdream.net] > Sent: Friday, October 24, 2014 6:11 PM > To: Philip Kufeldt > Cc: Prashanth Nednoor; ceph-devel <at> vger.kernel.org > Subject: RE: Having issues trying to get the OSD up on a MIPS64!!! > > On Sat, 25 Oct 2014, Philip Kufeldt wrote: > > 64 bit big endian > > My guess is that there is an endianness bug in leveldb then. I wonder who else has tried it on MIPS? > > sage > > > > > > -----Original Message----- > > > From: Sage Weil [mailto:sage <at> newdream.net] > > > Sent: Friday, October 24, 2014 5:47 PM > > > To: Prashanth Nednoor > > > Cc: ceph-devel <at> vger.kernel.org; Philip Kufeldt > > > Subject: RE: Having issues trying to get the OSD up on a MIPS64!!! > > > > > > Hi Prashanth, > > > > > > On Fri, 24 Oct 2014, Prashanth Nednoor wrote: > > > > Hi Sage, > > > > > > > > Thank you for the prompt response. > > > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely? > > > > Nothing , it was Missing Entirely. > > > > GOOD NEWS: I worked around this issue, if I set my journal > > > > path in the > > > /etc/ceph.conf. > > > > > > > > My udev version is udevd --version 164 > > > > > > Hmm, that should be new enough, but it seems like it isn't setting > > > up the links. What distro is it? On most systems it's > > > /lib/udev/rules.d/60-persistent- storage.rules that does it. Maybe > > > see if running partprobe /dev/sda or run 'udevadm monitor' and do > > > 'udevadm trigger /dev/sda' in another terminal to see what happens. > > > > > > Or, work around it like you did. :) > > > > > > > I still see the segfaults, I have attached details. > > > > I put the osd debug logs(osd-output.txt) and the > > > leveldb_bt(leveldb_bt.txt). > > > > Looks like we have an issue in leveldb.... > > > > > > Yeah, that looks like a problem with leveldb. What distro is this? > > > What version leveldb? > > > > > > I don't actually know anything about MIPS.. what's teh wordsize and > > > endianess? > > > > > > sage > > > > > > > > > > > > > > HERE IS THE BACK TRACE: I have attached the gdb before running it. > > > > #0 0x77f68ee0 in leveldb::SkipList<char const*, > > > > leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const* > > > > const&, leveldb::SkipList<char const*, > > > > leveldb::MemTable::KeyComparator>::Node**) const () from > > > > /usr/local/lib/libleveldb.so.1 > > > > #1 0x77f69054 in leveldb::SkipList<char const*, > > > > leveldb::MemTable::KeyComparator>::Insert(char const* const&) () > > > > from > > > > /usr/local/lib/libleveldb.so.1 > > > > #2 0x77f68618 in leveldb::MemTable::Add(unsigned long long, > > > leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) () > > > > from /usr/local/lib/libleveldb.so.1 > > > > #3 0x77f7e434 in leveldb::(anonymous > > > namespace)::MemTableInserter::Put(leveldb::Slice const&, > > > leveldb::Slice > > > const&) () > > > > from /usr/local/lib/libleveldb.so.1 > > > > #4 0x77f7e93c in > > > > leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const > > > > () from /usr/local/lib/libleveldb.so.1 > > > > #5 0x77f7eb8c in > > > > leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch > > > > const*, > > > > leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1 > > > > #6 0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions > > > > const&, > > > > leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1 > > > > #7 0x00a5dda0 in LevelDBStore::submit_transaction_sync > > > > (this=0x1f77d10, t=<value optimized out>) at > > > > os/LevelDBStore.cc:146 > > > > #8 0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0, > > > > spos=0x72cfe3b8) at os/DBObjectMap.cc:1126 > > > > #9 0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, > > > > fd=17, spos=..., hoid=0x0, in_progress=false) at > > > > os/FileStore.cc:2070 > > > > #10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450, > > > cid=DWARF-2 expression error: DW_OP_reg operations must be used > > > either alone or in conjuction with DW_OP_piece. > > > > ) at os/FileStore.cc:2047 > > > > #11 0x009b2138 in FileStore::_create_collection (this=0x1f72450, > > > > c=DWARF- > > > 2 expression error: DW_OP_reg operations must be used either alone > > > or in conjuction with DW_OP_piece. > > > > ) at os/FileStore.cc:4753 > > > > #12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, > > > > t=..., op_seq=<value optimized out>, trans_num=0, > > > > handle=0x72cfec3c) at > > > > os/FileStore.cc:2413 > > > > #13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450, > > > > tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952 > > > > #14 0x009eb858 in FileStore::_do_op (this=0x1f72450, > > > > osr=0x1f801b8, > > > > handle=...) at os/FileStore.cc:1761 > > > > #15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0, > > > > wt=0x1f7ea90) at > > > > common/WorkQueue.cc:128 > > > > #16 0x00c91b94 in ThreadPool::WorkThread::entry() () > > > > #17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0 > > > > #18 0x777c1738 in ?? () from /lib/libc.so.6 > > > > > > > > Do I need to set any variable to set the cache size etcetc in ceph.conf. > > > > I only have osd_leveldb_cache_size=5242880 for now. > > > > > > > > > > > > Thanks > > > > Prashanth > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Sage Weil [mailto:sage <at> newdream.net] > > > > Sent: Thursday, October 23, 2014 5:54 PM > > > > To: Prashanth Nednoor > > > > Cc: ceph-devel <at> vger.kernel.org > > > > Subject: Re: Having issues trying to get the OSD up on a MIPS64!!! > > > > > > > > Hi Prashanth, > > > > > > > > On Thu, 23 Oct 2014, Prashanth Nednoor wrote: > > > > > Hello Everyone, > > > > > > > > > > We are using ceph-0.86, good news is we were able to compile and > > > > > load all the libraries and binaries needed to configure a > > > > > CEPH-OSD on MIPS > > > > > 64 platform. The CEPH monitor is also able to detect the OSD, > > > > > but not up yet, as the osd activate failed. > > > > > Since we don?t have the required CEPH deploy utility for MIPS64, > > > > > we are following the manual procedure to create and activate an OSD. > > > > > We have disabled authentication between the clients and the > > > > > OSD?s for now. > > > > > > > > > > Has any body tried CEPH on a MIPS64? > > > > > /dev/sda is a 2TB local hard drive. > > > > > > > > > > This is how my partition looks after ceph-disk-prepare > > > > > /home/prashan/ceph-0.86/src# parted GNU Parted 2.3 Using > > > > > /dev/sda Welcome to GNU Parted! Type 'help' to view a list of commands. > > > > > (parted) p > > > > > Model: ATA TOSHIBA MQ01ABB2 (scsi) Disk /dev/sda: 2000GB Sector > > > > > size (logical/physical): 512B/4096B Partition Table: gpt > > > > > > > > > > Number Start End Size File system Name Flags > > > > > 2 1049kB 5369MB 5368MB ceph journal > > > > > 1 5370MB 2000GB 1995GB xfs ceph data > > > > > > > > > > > > > > > > > > > > The following are the steps to create an OSD > > > > > 1) ceph-disk zap /dev/sda > > > > > 2) ceph-disk-prepare --cluster f615496c-b40a-4905-bbcd- > > > > > 2d3e181ff21a --fs-type xfs /dev/sda > > > > > 3) mount /dev/sda1 /var/lib/ceph/osd/ceph-0/ > > > > > 4) ceph-osd -i 0 ?mkfs is giving an error , > > > > > filestore(/var/lib/ceph/osd/ceph-0) could not find > > > > > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file. > > > > > After this it segfaults. We have analyzed this further with the > > > > > help of strace and root caused this as objectmap file reading issue. > > > > > open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", > > > > > O_RDONLY) > > > = > > > > > 11, the first time it reads 32k, the read succeeds with 63 bytes > > > > > and it tries to read again with 27k and the read returns 0 bytes > > > > > and the CEPH osd segfaults. > > > > > > > > Can you generate a full log with --debug-osd 20 --debug-filestore > > > > 20 -- > > > debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere? > > > It should tell us where things are going wrong. In particular, we > > > want to see if that file/object is being written properly. It will > > > also have a backtrace showing exactly where it crashed. > > > > > > > > > Please note that ceph-disk prepare creates a journal in a path > > > > > which is not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492- > > > 080ad2a505cb). > > > > > So after step3 above I remove this journal below and manually > > > > > create a journal file before doing step4 above. > > > > > > > > > > > > > > > ls -l /var/lib/ceph/osd/ceph-0/ > > > > > total 16 > > > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid > > > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid lrwxrwxrwx 1 root > > > > > root > > > > > 58 Oct 22 21:40 journal -> /dev/disk/by- > > > > > partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb > > > > > > > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely? > > > > Maybe you have an old udev. What distro is this? > > > > > > > > sage > > > > > > > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid > > > > > -rw-r--r-- 1 root root 21 Oct 22 21:40 magic > > > > > > > > > > Any pointers to move ahead will be greatly appreciated?? > > > > > > > > > > thanks > > > > > Prashanth > > > > > > > > > > > > > > > > > > > > -- > > > > > To unsubscribe from this list: send the line "unsubscribe ceph- devel" > > > > > in the body of a message to majordomo <at> vger.kernel.org More > > > majordomo > > > > > info at http://vger.kernel.org/majordomo-info.html > > > > > > > > > > > > > > > > > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo <at> vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Hi All, I am experience the following error when compile ceph on MIPS64 /bin/sh ../libtool --tag=CXX --mode=link /usr/local/buildroot-2015.02- rc1/output/host/usr/bin/mips64-linux-g++ -I/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/include/nss -I/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/include/nspr -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith -Werror=format-security -fno-strict-aliasing -fsigned-char -rdynamic -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno- invalid-offsetof -Wstrict-null-sentinel -mxgot -mlong-calls -I/usr/local/buildroot-2015.02-rc1/output/host/usr/include -I/usr/local/mips64/include -latomic_ops -L/usr/local/mips64/lib -L/usr/local/buildroot-2015.02-rc1/output/host/usr/lib -L/usr/local/buildroot-2015.02-rc1/output/host/usr/lib64 -o ceph_objectstore_tool tools/ceph_objectstore_tool.o libosd.la libosdc.la libos.la -laio -lleveldb -lsnappy libperfglue.la libos.la -laio -lleveldb -lsnappy libglobal.la libcommon.la -lpthread -lm -lnss3 -lnssutil3 -lsmime3 -lssl3 -L/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib -lplds4 -lplc4 -lnspr4 -luuid -lm -lboost_program_options librados.la -lboost_system libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/../lib64/libsnappy.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/libsnappy.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/libatomic_ops.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/../lib64/libatomic_ops.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/lib/gcc/mips64-buildroot-linux- gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.la' was moved. libtool: warning: library '/usr/local/mips64/lib/libuuid.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/../lib64/libsnappy.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/libsnappy.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/libatomic_ops.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/../lib64/libatomic_ops.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/lib/gcc/mips64-buildroot-linux- gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.la' was moved. libtool: warning: library '/usr/local/mips64/lib/libuuid.la' was moved. libtool: link: /usr/local/buildroot-2015.02-rc1/output/host/usr/bin/mips64- linux-g++ -I/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64- buildroot-linux-gnu/sysroot/usr/include/nss -I/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/include/nspr -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith -Werror=format-security -fno-strict-aliasing -fsigned-char -rdynamic -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-invalid-offsetof -Wstrict- null-sentinel -mxgot -mlong-calls -I/usr/local/buildroot-2015.02- rc1/output/host/usr/include -I/usr/local/mips64/include -o ceph_objectstore_tool tools/ceph_objectstore_tool.o -L/usr/local/mips64/lib -L/usr/local/buildroot-2015.02- rc1/output/host/usr/lib -L/usr/local/buildroot-2015.02- rc1/output/host/usr/lib64 ./.libs/libosd.a /usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/../lib64/libsnappy.so ./.libs/libosdc.a ./.libs/libperfg lue.a ./.libs/libos.a -laio -lleveldb /usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/libsnappy.so ./.libs/libglobal.a ./.libs/libcommon.a -L/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib -lboost_program_options ./.libs/librados.a /usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/libatomic_ops.so -lboost_thread /usr/local/buildroot- 2015.02-rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/../lib64/libatomic_ops.so /usr/local/buildroot-2015.02- rc1/output/host/usr/lib/gcc/mips64-buildroot-linux- gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.so -lm -lpthread -lnss3 -lnssutil3 -lsmime3 -lssl3 -lplds4 -lplc4 -lnspr4 /usr/local/mips64/lib/libuuid.a -lboost_system -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/lib/gcc/mips64- buildroot-linux-gnu/4.9.2/../../../../mips64-buildroot-linux- gnu/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64- buildroot-linux-gnu/sysroot/usr/lib -Wl,-rpath -Wl,/usr/local/buildroot- 2015.02-rc1/output/host/usr/lib/gcc/mips64-buildroot-linux- gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64 /usr/local/buildroot-2015.02-rc1/output/host/usr/lib/gcc/mips64-buildroot- linux-gnu/4.9.2/../../../../mips64-buildroot-linux- gnu/bin/ld: ./.libs/libosd.a(libosd_la-ClassHandler.o): undefined reference to symbol 'dlopen@@GLIBC_2.2' /usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/lib64/libdl.so.2: error adding symbols: DSO missing from command line collect2: error: ld returned 1 exit status I use the following configure: ./configure --prefix=/usr/local/ceph_mips64 --exec- prefix=/usr/local/ceph_mips64 --build=mips64 --host=x86_64 --target=mips64 --without-tcmalloc --disable-silent-rules --with- sysroot=/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot- linux-gnu/sysroot make When I run the link command manually, with additional "-ldl". I hit different errors: /bin/sh ../libtool --tag=CXX --mode=link /usr/local/buildroot-2015.02- rc1/output/host/usr/bin/mips64-linux-g++ -I/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/include/nss -I/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/include/nspr -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith -Werror=format-security -fno-strict-aliasing -fsigned-char -rdynamic -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno- invalid-offsetof -Wstrict-null-sentinel -mxgot -mlong-calls -I/usr/local/buildroot-2015.02-rc1/output/host/usr/include -I/usr/local/mips64/include -latomic_ops -L/usr/local/mips64/lib -L/usr/local/buildroot-2015.02-rc1/output/host/usr/lib -L/usr/local/buildroot-2015.02-rc1/output/host/usr/lib64 -o ceph_objectstore_tool tools/ceph_objectstore_tool.o libosd.la libosdc.la libos.la -laio -lleveldb -lsnappy libperfglue.la libos.la -laio -lleveldb -lsnappy libglobal.la libcommon.la -lpthread -lm -lnss3 -lnssutil3 -lsmime3 -lssl3 -L/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib -lplds4 -lplc4 -lnspr4 -luuid -lm -lboost_program_options librados.la -lboost_system -ldl libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/../lib64/libsnappy.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/libsnappy.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/libatomic_ops.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/../lib64/libatomic_ops.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/lib/gcc/mips64-buildroot-linux- gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.la' was moved. libtool: warning: library '/usr/local/mips64/lib/libuuid.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/../lib64/libsnappy.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/libsnappy.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/libatomic_ops.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/../lib64/libatomic_ops.la' was moved. libtool: warning: library '/usr/local/buildroot-2015.02- rc1/output/host/usr/lib/gcc/mips64-buildroot-linux- gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.la' was moved. libtool: warning: library '/usr/local/mips64/lib/libuuid.la' was moved. libtool: link: /usr/local/buildroot-2015.02-rc1/output/host/usr/bin/mips64- linux-g++ -I/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64- buildroot-linux-gnu/sysroot/usr/include/nss -I/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/include/nspr -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith -Werror=format-security -fno-strict-aliasing -fsigned-char -rdynamic -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-invalid-offsetof -Wstrict- null-sentinel -mxgot -mlong-calls -I/usr/local/buildroot-2015.02- rc1/output/host/usr/include -I/usr/local/mips64/include -o ceph_objectstore_tool tools/ceph_objectstore_tool.o -L/usr/local/mips64/lib -L/usr/local/buildroot-2015.02- rc1/output/host/usr/lib -L/usr/local/buildroot-2015.02- rc1/output/host/usr/lib64 ./.libs/libosd.a /usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/../lib64/libsnappy.so ./.libs/libosdc.a ./.libs/libperfg lue.a ./.libs/libos.a -laio -lleveldb /usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/libsnappy.so ./.libs/libglobal.a ./.libs/libcommon.a -L/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib -lboost_program_options ./.libs/librados.a /usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/libatomic_ops.so -lboost_thread /usr/local/buildroot- 2015.02-rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/../lib64/libatomic_ops.so /usr/local/buildroot-2015.02- rc1/output/host/usr/lib/gcc/mips64-buildroot-linux- gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.so -lm -lpthread -lnss3 -lnssutil3 -lsmime3 -lssl3 -lplds4 -lplc4 -lnspr4 /usr/local/mips64/lib/libuuid.a -lboost_system -ldl -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux- gnu/sysroot/usr/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/lib/gcc/mips64- buildroot-linux-gnu/4.9.2/../../../../mips64-buildroot-linux- gnu/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02- rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64- buildroot-linux-gnu/sysroot/usr/lib -Wl,-rpath -Wl,/usr/local/buildroot- 2015.02-rc1/output/host/usr/lib/gcc/mips64-buildroot-linux- gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64 ./.libs/libosd.a(libosd_la-ReplicatedPG.o): In function `std::ostream& operator<< <boost::optional<unsigned long>, unsigned long>(std::ostream&, std::pair<boost::optional<unsigned long>, unsigned long> const&)': ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E [_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x34): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< <char, std::char_traits<char>, unsigned long> (std::basic_ostream<char, std::char_traits<char> >&, boost::optional<unsigned long> const&)' ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E [_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x3c): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< <char, std::char_traits<char>, unsigned long> (std::basic_ostream<char, std::char_traits<char> >&, boost::optional<unsigned long> const&)' ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E [_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x44): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< <char, std::char_traits<char>, unsigned long> (std::basic_ostream<char, std::char_traits<char> >&, boost::optional<unsigned long> const&)' ./.libs/libosd.a(libosd_la-ECBackend.o): In function `operator<< (std::ostream&, ECBackend::read_result_t const&)': ECBackend.cc:(.text+0x10f0): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< <char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> >&, boost::optional<std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > > > const&)' ECBackend.cc:(.text+0x10f8): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< <char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> >&, boost::optional<std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > > > const&)' ECBackend.cc:(.text+0x1100): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< <char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> >&, boost::optional<std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > > > const&)' ./.libs/libosd.a(libos_la-FileStore.o): In function `FileStoreBackend::create(long, FileStore*)': FileStore.cc:(.text+0x8f1c): undefined reference to `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)' FileStore.cc:(.text+0x8f24): undefined reference to `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)' FileStore.cc:(.text+0x8f2c): undefined reference to `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)' collect2: error: ld returned 1 exit status Could someone please help? Thank you, Bao ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Having issues trying to get the OSD up on a MIPS64!!! 2015-02-17 7:58 ` Bao Vu @ 2015-02-17 14:06 ` Sage Weil 2015-02-17 16:57 ` Bao Vu 0 siblings, 1 reply; 12+ messages in thread From: Sage Weil @ 2015-02-17 14:06 UTC (permalink / raw) To: Bao Vu; +Cc: ceph-devel > When I run the link command manually, with additional "-ldl". I hit > different errors: > > /bin/sh ../libtool --tag=CXX --mode=link /usr/local/buildroot-2015.02- > rc1/output/host/usr/bin/mips64-linux-g++ -I/usr/local/buildroot-2015.02- > rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/include/nss > -I/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux- > gnu/sysroot/usr/include/nspr -Wall -Wtype-limits -Wignored-qualifiers > -Winit-self -Wpointer-arith -Werror=format-security -fno-strict-aliasing > -fsigned-char -rdynamic -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno- > invalid-offsetof -Wstrict-null-sentinel -mxgot -mlong-calls > -I/usr/local/buildroot-2015.02-rc1/output/host/usr/include > -I/usr/local/mips64/include -latomic_ops -L/usr/local/mips64/lib > -L/usr/local/buildroot-2015.02-rc1/output/host/usr/lib > -L/usr/local/buildroot-2015.02-rc1/output/host/usr/lib64 -o > ceph_objectstore_tool tools/ceph_objectstore_tool.o libosd.la libosdc.la > libos.la -laio -lleveldb -lsnappy libperfglue.la libos.la -laio > -lleveldb -lsnappy libglobal.la libcommon.la -lpthread -lm -lnss3 > -lnssutil3 -lsmime3 -lssl3 -L/usr/local/buildroot-2015.02- > rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib -lplds4 > -lplc4 -lnspr4 -luuid -lm -lboost_program_options librados.la > -lboost_system -ldl > libtool: warning: library '/usr/local/buildroot-2015.02- > rc1/output/host/usr/mips64-buildroot-linux- > gnu/sysroot/usr/lib/../lib64/libsnappy.la' was moved. > libtool: warning: library '/usr/local/buildroot-2015.02- > rc1/output/host/usr/mips64-buildroot-linux- > gnu/sysroot/usr/lib/libsnappy.la' was moved. > libtool: warning: library '/usr/local/buildroot-2015.02- > rc1/output/host/usr/mips64-buildroot-linux- > gnu/sysroot/usr/lib/libatomic_ops.la' was moved. > libtool: warning: library '/usr/local/buildroot-2015.02- > rc1/output/host/usr/mips64-buildroot-linux- > gnu/sysroot/usr/lib/../lib64/libatomic_ops.la' was moved. > libtool: warning: library '/usr/local/buildroot-2015.02- > rc1/output/host/usr/lib/gcc/mips64-buildroot-linux- > gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.la' > was moved. > libtool: warning: library '/usr/local/mips64/lib/libuuid.la' was moved. > libtool: warning: library '/usr/local/buildroot-2015.02- > rc1/output/host/usr/mips64-buildroot-linux- > gnu/sysroot/usr/lib/../lib64/libsnappy.la' was moved. > libtool: warning: library '/usr/local/buildroot-2015.02- > rc1/output/host/usr/mips64-buildroot-linux- > gnu/sysroot/usr/lib/libsnappy.la' was moved. > libtool: warning: library '/usr/local/buildroot-2015.02- > rc1/output/host/usr/mips64-buildroot-linux- > gnu/sysroot/usr/lib/libatomic_ops.la' was moved. > libtool: warning: library '/usr/local/buildroot-2015.02- > rc1/output/host/usr/mips64-buildroot-linux- > gnu/sysroot/usr/lib/../lib64/libatomic_ops.la' was moved. > libtool: warning: library '/usr/local/buildroot-2015.02- > rc1/output/host/usr/lib/gcc/mips64-buildroot-linux- > gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.la' > was moved. > libtool: warning: library '/usr/local/mips64/lib/libuuid.la' was moved. > libtool: link: /usr/local/buildroot-2015.02-rc1/output/host/usr/bin/mips64- > linux-g++ -I/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64- > buildroot-linux-gnu/sysroot/usr/include/nss -I/usr/local/buildroot-2015.02- > rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/include/nspr > -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith > -Werror=format-security -fno-strict-aliasing -fsigned-char -rdynamic > -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-invalid-offsetof -Wstrict- > null-sentinel -mxgot -mlong-calls -I/usr/local/buildroot-2015.02- > rc1/output/host/usr/include -I/usr/local/mips64/include -o > ceph_objectstore_tool tools/ceph_objectstore_tool.o > -L/usr/local/mips64/lib -L/usr/local/buildroot-2015.02- > rc1/output/host/usr/lib -L/usr/local/buildroot-2015.02- > rc1/output/host/usr/lib64 ./.libs/libosd.a /usr/local/buildroot-2015.02- > rc1/output/host/usr/mips64-buildroot-linux- > gnu/sysroot/usr/lib/../lib64/libsnappy.so ./.libs/libosdc.a ./.libs/libperfg > lue.a ./.libs/libos.a -laio -lleveldb /usr/local/buildroot-2015.02- > rc1/output/host/usr/mips64-buildroot-linux- > gnu/sysroot/usr/lib/libsnappy.so ./.libs/libglobal.a ./.libs/libcommon.a > -L/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux- > gnu/sysroot/usr/lib > -lboost_program_options ./.libs/librados.a /usr/local/buildroot-2015.02- > rc1/output/host/usr/mips64-buildroot-linux- > gnu/sysroot/usr/lib/libatomic_ops.so -lboost_thread /usr/local/buildroot- > 2015.02-rc1/output/host/usr/mips64-buildroot-linux- > gnu/sysroot/usr/lib/../lib64/libatomic_ops.so /usr/local/buildroot-2015.02- > rc1/output/host/usr/lib/gcc/mips64-buildroot-linux- > gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.so > -lm -lpthread -lnss3 -lnssutil3 -lsmime3 -lssl3 -lplds4 -lplc4 > -lnspr4 /usr/local/mips64/lib/libuuid.a -lboost_system -ldl -Wl,-rpath > -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux- > gnu/sysroot/usr/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02- > rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib -Wl,-rpath > -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/lib/gcc/mips64- > buildroot-linux-gnu/4.9.2/../../../../mips64-buildroot-linux- > gnu/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02- > rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib/../lib64 > -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64- > buildroot-linux-gnu/sysroot/usr/lib -Wl,-rpath -Wl,/usr/local/buildroot- > 2015.02-rc1/output/host/usr/lib/gcc/mips64-buildroot-linux- > gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64 > ./.libs/libosd.a(libosd_la-ReplicatedPG.o): In function `std::ostream& > operator<< <boost::optional<unsigned long>, unsigned long>(std::ostream&, > std::pair<boost::optional<unsigned long>, unsigned long> const&)': > ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E > [_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x34): undefined > reference to `std::basic_ostream<char, std::char_traits<char> >& > boost::operator<< <char, std::char_traits<char>, unsigned long> > (std::basic_ostream<char, std::char_traits<char> >&, > boost::optional<unsigned long> const&)' > ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E > [_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x3c): undefined > reference to `std::basic_ostream<char, std::char_traits<char> >& > boost::operator<< <char, std::char_traits<char>, unsigned long> > (std::basic_ostream<char, std::char_traits<char> >&, > boost::optional<unsigned long> const&)' > ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E > [_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x44): undefined > reference to `std::basic_ostream<char, std::char_traits<char> >& > boost::operator<< <char, std::char_traits<char>, unsigned long> > (std::basic_ostream<char, std::char_traits<char> >&, > boost::optional<unsigned long> const&)' > ./.libs/libosd.a(libosd_la-ECBackend.o): In function `operator<< > (std::ostream&, ECBackend::read_result_t const&)': > ECBackend.cc:(.text+0x10f0): undefined reference to > `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< > <char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, > std::less<std::string>, std::allocator<std::pair<std::string const, > ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> > >&, boost::optional<std::map<std::string, ceph::buffer::list, > std::less<std::string>, std::allocator<std::pair<std::string const, > ceph::buffer::list> > > > const&)' > ECBackend.cc:(.text+0x10f8): undefined reference to > `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< > <char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, > std::less<std::string>, std::allocator<std::pair<std::string const, > ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> > >&, boost::optional<std::map<std::string, ceph::buffer::list, > std::less<std::string>, std::allocator<std::pair<std::string const, > ceph::buffer::list> > > > const&)' > ECBackend.cc:(.text+0x1100): undefined reference to > `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< > <char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, > std::less<std::string>, std::allocator<std::pair<std::string const, > ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> > >&, boost::optional<std::map<std::string, ceph::buffer::list, > std::less<std::string>, std::allocator<std::pair<std::string const, > ceph::buffer::list> > > > const&)' > ./.libs/libosd.a(libos_la-FileStore.o): In function > `FileStoreBackend::create(long, FileStore*)': > FileStore.cc:(.text+0x8f1c): undefined reference to > `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)' > FileStore.cc:(.text+0x8f24): undefined reference to > `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)' > FileStore.cc:(.text+0x8f2c): undefined reference to > `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)' > collect2: error: ld returned 1 exit status It looks like maybe the BtrfsFileStoreBackend is not being linked in? Perhaps there is some Makefile magic going on? Which version/commit are you on, and have you made any Makefile modifications? Unfortunately we don't have a MPIS64 box in the community lab to play with. sage ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Having issues trying to get the OSD up on a MIPS64!!! 2015-02-17 14:06 ` Sage Weil @ 2015-02-17 16:57 ` Bao Vu 2015-02-17 17:06 ` Sage Weil 0 siblings, 1 reply; 12+ messages in thread From: Bao Vu @ 2015-02-17 16:57 UTC (permalink / raw) To: Sage Weil; +Cc: ceph-devel On Feb 17, 2015, at 6:06 AM, Sage Weil <sage@newdream.net> wrote: >> When I run the link command manually, with additional "-ldl". I hit >> different errors: >> >> /bin/sh ../libtool --tag=CXX --mode=link /usr/local/buildroot-2015.02- >> rc1/output/host/usr/bin/mips64-linux-g++ -I/usr/local/buildroot-2015.02- >> rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/include/nss >> -I/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux- >> gnu/sysroot/usr/include/nspr -Wall -Wtype-limits -Wignored-qualifiers >> -Winit-self -Wpointer-arith -Werror=format-security -fno-strict-aliasing >> -fsigned-char -rdynamic -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno- >> invalid-offsetof -Wstrict-null-sentinel -mxgot -mlong-calls >> -I/usr/local/buildroot-2015.02-rc1/output/host/usr/include >> -I/usr/local/mips64/include -latomic_ops -L/usr/local/mips64/lib >> -L/usr/local/buildroot-2015.02-rc1/output/host/usr/lib >> -L/usr/local/buildroot-2015.02-rc1/output/host/usr/lib64 -o >> ceph_objectstore_tool tools/ceph_objectstore_tool.o libosd.la libosdc.la >> libos.la -laio -lleveldb -lsnappy libperfglue.la libos.la -laio >> -lleveldb -lsnappy libglobal.la libcommon.la -lpthread -lm -lnss3 >> -lnssutil3 -lsmime3 -lssl3 -L/usr/local/buildroot-2015.02- >> rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib -lplds4 >> -lplc4 -lnspr4 -luuid -lm -lboost_program_options librados.la >> -lboost_system -ldl >> libtool: warning: library '/usr/local/buildroot-2015.02- >> rc1/output/host/usr/mips64-buildroot-linux- >> gnu/sysroot/usr/lib/../lib64/libsnappy.la' was moved. >> libtool: warning: library '/usr/local/buildroot-2015.02- >> rc1/output/host/usr/mips64-buildroot-linux- >> gnu/sysroot/usr/lib/libsnappy.la' was moved. >> libtool: warning: library '/usr/local/buildroot-2015.02- >> rc1/output/host/usr/mips64-buildroot-linux- >> gnu/sysroot/usr/lib/libatomic_ops.la' was moved. >> libtool: warning: library '/usr/local/buildroot-2015.02- >> rc1/output/host/usr/mips64-buildroot-linux- >> gnu/sysroot/usr/lib/../lib64/libatomic_ops.la' was moved. >> libtool: warning: library '/usr/local/buildroot-2015.02- >> rc1/output/host/usr/lib/gcc/mips64-buildroot-linux- >> gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.la' >> was moved. >> libtool: warning: library '/usr/local/mips64/lib/libuuid.la' was moved. >> libtool: warning: library '/usr/local/buildroot-2015.02- >> rc1/output/host/usr/mips64-buildroot-linux- >> gnu/sysroot/usr/lib/../lib64/libsnappy.la' was moved. >> libtool: warning: library '/usr/local/buildroot-2015.02- >> rc1/output/host/usr/mips64-buildroot-linux- >> gnu/sysroot/usr/lib/libsnappy.la' was moved. >> libtool: warning: library '/usr/local/buildroot-2015.02- >> rc1/output/host/usr/mips64-buildroot-linux- >> gnu/sysroot/usr/lib/libatomic_ops.la' was moved. >> libtool: warning: library '/usr/local/buildroot-2015.02- >> rc1/output/host/usr/mips64-buildroot-linux- >> gnu/sysroot/usr/lib/../lib64/libatomic_ops.la' was moved. >> libtool: warning: library '/usr/local/buildroot-2015.02- >> rc1/output/host/usr/lib/gcc/mips64-buildroot-linux- >> gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.la' >> was moved. >> libtool: warning: library '/usr/local/mips64/lib/libuuid.la' was moved. >> libtool: link: /usr/local/buildroot-2015.02-rc1/output/host/usr/bin/mips64- >> linux-g++ -I/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64- >> buildroot-linux-gnu/sysroot/usr/include/nss -I/usr/local/buildroot-2015.02- >> rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/include/nspr >> -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith >> -Werror=format-security -fno-strict-aliasing -fsigned-char -rdynamic >> -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-invalid-offsetof -Wstrict- >> null-sentinel -mxgot -mlong-calls -I/usr/local/buildroot-2015.02- >> rc1/output/host/usr/include -I/usr/local/mips64/include -o >> ceph_objectstore_tool tools/ceph_objectstore_tool.o >> -L/usr/local/mips64/lib -L/usr/local/buildroot-2015.02- >> rc1/output/host/usr/lib -L/usr/local/buildroot-2015.02- >> rc1/output/host/usr/lib64 ./.libs/libosd.a /usr/local/buildroot-2015.02- >> rc1/output/host/usr/mips64-buildroot-linux- >> gnu/sysroot/usr/lib/../lib64/libsnappy.so ./.libs/libosdc.a ./.libs/libperfg >> lue.a ./.libs/libos.a -laio -lleveldb /usr/local/buildroot-2015.02- >> rc1/output/host/usr/mips64-buildroot-linux- >> gnu/sysroot/usr/lib/libsnappy.so ./.libs/libglobal.a ./.libs/libcommon.a >> -L/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux- >> gnu/sysroot/usr/lib >> -lboost_program_options ./.libs/librados.a /usr/local/buildroot-2015.02- >> rc1/output/host/usr/mips64-buildroot-linux- >> gnu/sysroot/usr/lib/libatomic_ops.so -lboost_thread /usr/local/buildroot- >> 2015.02-rc1/output/host/usr/mips64-buildroot-linux- >> gnu/sysroot/usr/lib/../lib64/libatomic_ops.so /usr/local/buildroot-2015.02- >> rc1/output/host/usr/lib/gcc/mips64-buildroot-linux- >> gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.so >> -lm -lpthread -lnss3 -lnssutil3 -lsmime3 -lssl3 -lplds4 -lplc4 >> -lnspr4 /usr/local/mips64/lib/libuuid.a -lboost_system -ldl -Wl,-rpath >> -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux- >> gnu/sysroot/usr/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02- >> rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib -Wl,-rpath >> -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/lib/gcc/mips64- >> buildroot-linux-gnu/4.9.2/../../../../mips64-buildroot-linux- >> gnu/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02- >> rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib/../lib64 >> -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64- >> buildroot-linux-gnu/sysroot/usr/lib -Wl,-rpath -Wl,/usr/local/buildroot- >> 2015.02-rc1/output/host/usr/lib/gcc/mips64-buildroot-linux- >> gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64 >> ./.libs/libosd.a(libosd_la-ReplicatedPG.o): In function `std::ostream& >> operator<< <boost::optional<unsigned long>, unsigned long>(std::ostream&, >> std::pair<boost::optional<unsigned long>, unsigned long> const&)': >> ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E >> [_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x34): undefined >> reference to `std::basic_ostream<char, std::char_traits<char> >& >> boost::operator<< <char, std::char_traits<char>, unsigned long> >> (std::basic_ostream<char, std::char_traits<char> >&, >> boost::optional<unsigned long> const&)' >> ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E >> [_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x3c): undefined >> reference to `std::basic_ostream<char, std::char_traits<char> >& >> boost::operator<< <char, std::char_traits<char>, unsigned long> >> (std::basic_ostream<char, std::char_traits<char> >&, >> boost::optional<unsigned long> const&)' >> ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E >> [_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x44): undefined >> reference to `std::basic_ostream<char, std::char_traits<char> >& >> boost::operator<< <char, std::char_traits<char>, unsigned long> >> (std::basic_ostream<char, std::char_traits<char> >&, >> boost::optional<unsigned long> const&)' >> ./.libs/libosd.a(libosd_la-ECBackend.o): In function `operator<< >> (std::ostream&, ECBackend::read_result_t const&)': >> ECBackend.cc:(.text+0x10f0): undefined reference to >> `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< >> <char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, >> std::less<std::string>, std::allocator<std::pair<std::string const, >> ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> >>> &, boost::optional<std::map<std::string, ceph::buffer::list, >> std::less<std::string>, std::allocator<std::pair<std::string const, >> ceph::buffer::list> > > > const&)' >> ECBackend.cc:(.text+0x10f8): undefined reference to >> `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< >> <char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, >> std::less<std::string>, std::allocator<std::pair<std::string const, >> ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> >>> &, boost::optional<std::map<std::string, ceph::buffer::list, >> std::less<std::string>, std::allocator<std::pair<std::string const, >> ceph::buffer::list> > > > const&)' >> ECBackend.cc:(.text+0x1100): undefined reference to >> `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< >> <char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, >> std::less<std::string>, std::allocator<std::pair<std::string const, >> ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> >>> &, boost::optional<std::map<std::string, ceph::buffer::list, >> std::less<std::string>, std::allocator<std::pair<std::string const, >> ceph::buffer::list> > > > const&)' >> ./.libs/libosd.a(libos_la-FileStore.o): In function >> `FileStoreBackend::create(long, FileStore*)': >> FileStore.cc:(.text+0x8f1c): undefined reference to >> `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)' >> FileStore.cc:(.text+0x8f24): undefined reference to >> `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)' >> FileStore.cc:(.text+0x8f2c): undefined reference to >> `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)' >> collect2: error: ld returned 1 exit status > > It looks like maybe the BtrfsFileStoreBackend is not being linked in? > Perhaps there is some Makefile magic going on? Which version/commit are > you on, and have you made any Makefile modifications? > > Unfortunately we don't have a MPIS64 box in the community lab to play > with. > > sage I am using 0.87. I do not change makefile . It is created by the configure. I try to build ceph for MIPS64. So far, I have no luck. Bao ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Having issues trying to get the OSD up on a MIPS64!!! 2015-02-17 16:57 ` Bao Vu @ 2015-02-17 17:06 ` Sage Weil 0 siblings, 0 replies; 12+ messages in thread From: Sage Weil @ 2015-02-17 17:06 UTC (permalink / raw) To: Bao Vu; +Cc: ceph-devel On Tue, 17 Feb 2015, Bao Vu wrote: > >> ./.libs/libosd.a(libos_la-FileStore.o): In function > >> `FileStoreBackend::create(long, FileStore*)': > >> FileStore.cc:(.text+0x8f1c): undefined reference to > >> `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)' > >> FileStore.cc:(.text+0x8f24): undefined reference to > >> `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)' > >> FileStore.cc:(.text+0x8f2c): undefined reference to > >> `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)' > >> collect2: error: ld returned 1 exit status > > > > It looks like maybe the BtrfsFileStoreBackend is not being linked in? > > Perhaps there is some Makefile magic going on? Which version/commit are > > you on, and have you made any Makefile modifications? > > > > Unfortunately we don't have a MPIS64 box in the community lab to play > > with. > > > > sage > > I am using 0.87. I do not change makefile . It is created by the configure. I try to build ceph for MIPS64. So far, I have no luck. It is probably best to work from the latest git master to make sure you've captured any recent changes. MIPS64 support isn't something that would have been backported. sage ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2015-02-17 17:06 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-10-23 1:28 Having issues trying to get the OSD up on a MIPS64!!! Prashanth Nednoor 2014-10-24 0:53 ` Sage Weil 2014-10-24 21:38 ` Prashanth Nednoor 2014-10-25 0:47 ` Sage Weil 2014-10-25 0:55 ` Philip Kufeldt 2014-10-25 1:11 ` Sage Weil 2014-10-27 2:31 ` Prashanth Nednoor 2014-10-27 4:14 ` Prashanth Nednoor 2015-02-17 7:58 ` Bao Vu 2015-02-17 14:06 ` Sage Weil 2015-02-17 16:57 ` Bao Vu 2015-02-17 17:06 ` Sage Weil
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.