Having issues trying to get the OSD up on a MIPS64!!!

Having issues trying to get the OSD up on a MIPS64!!!

[Prashanth Nednoor]

Hello Everyone,

We are using ceph-0.86, good news is we were able to compile and load 
all the libraries and binaries needed to configure a CEPH-OSD on MIPS 64 
platform. The CEPH monitor is also able to detect the OSD, but not up 
yet, as the osd activate failed.
Since we don’t have the required CEPH deploy utility for MIPS64, we are 
following the manual procedure to create and activate an OSD.
We have disabled authentication between the clients and the OSD’s for 
now.

Has any body tried CEPH on a MIPS64?
/dev/sda is a 2TB local hard drive.

This is how my partition looks after ceph-disk-prepare
/home/prashan/ceph-0.86/src# parted
GNU Parted 2.3
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Model: ATA TOSHIBA MQ01ABB2 (scsi)
Disk /dev/sda: 2000GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name          Flags
 2      1049kB  5369MB  5368MB               ceph journal
 1      5370MB  2000GB  1995GB  xfs          ceph data



The following are the steps to create an OSD
1)	ceph-disk zap /dev/sda
2)	ceph-disk-prepare --cluster  f615496c-b40a-4905-bbcd-
2d3e181ff21a --fs-type xfs /dev/sda
3)	mount /dev/sda1 /var/lib/ceph/osd/ceph-0/
4)	ceph-osd -i 0 –mkfs is giving an error , 
filestore(/var/lib/ceph/osd/ceph-0) could not find 
23c2fcde/osd_superblock/0//-1 in index: (2) No such file.
After this it segfaults. We have analyzed this further with the help of 
strace and root caused this as objectmap file reading issue.
open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", O_RDONLY) = 11, 
the first time it reads 32k, the read succeeds with 63 bytes and it
tries to read again with 27k and the read returns 0 bytes and the CEPH 
osd segfaults.

Please note that ceph-disk prepare creates a journal in a path which is 
not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb).
So after step3 above I remove this journal below and manually create a 
journal file before doing step4 above.


ls -l /var/lib/ceph/osd/ceph-0/
total 16
-rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid
-rw-r--r-- 1 root root 37 Oct 22 21:40 fsid
lrwxrwxrwx 1 root root 58 Oct 22 21:40 journal -> /dev/disk/by-
partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb
-rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid
-rw-r--r-- 1 root root 21 Oct 22 21:40 magic

Any pointers to move ahead will be greatly appreciated??

thanks
Prashanth



--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Having issues trying to get the OSD up on a MIPS64!!!

[Sage Weil]

Hi Prashanth,

On Thu, 23 Oct 2014, Prashanth Nednoor wrote:
> Hello Everyone,
> 
> We are using ceph-0.86, good news is we were able to compile and load 
> all the libraries and binaries needed to configure a CEPH-OSD on MIPS 64 
> platform. The CEPH monitor is also able to detect the OSD, but not up 
> yet, as the osd activate failed.
> Since we don?t have the required CEPH deploy utility for MIPS64, we are 
> following the manual procedure to create and activate an OSD.
> We have disabled authentication between the clients and the OSD?s for 
> now.
> 
> Has any body tried CEPH on a MIPS64?
> /dev/sda is a 2TB local hard drive.
> 
> This is how my partition looks after ceph-disk-prepare
> /home/prashan/ceph-0.86/src# parted
> GNU Parted 2.3
> Using /dev/sda
> Welcome to GNU Parted! Type 'help' to view a list of commands.
> (parted) p
> Model: ATA TOSHIBA MQ01ABB2 (scsi)
> Disk /dev/sda: 2000GB
> Sector size (logical/physical): 512B/4096B
> Partition Table: gpt
> 
> Number  Start   End     Size    File system  Name          Flags
>  2      1049kB  5369MB  5368MB               ceph journal
>  1      5370MB  2000GB  1995GB  xfs          ceph data
> 
> 
> 
> The following are the steps to create an OSD
> 1)	ceph-disk zap /dev/sda
> 2)	ceph-disk-prepare --cluster  f615496c-b40a-4905-bbcd-
> 2d3e181ff21a --fs-type xfs /dev/sda
> 3)	mount /dev/sda1 /var/lib/ceph/osd/ceph-0/
> 4)	ceph-osd -i 0 ?mkfs is giving an error , 
> filestore(/var/lib/ceph/osd/ceph-0) could not find 
> 23c2fcde/osd_superblock/0//-1 in index: (2) No such file.
> After this it segfaults. We have analyzed this further with the help of 
> strace and root caused this as objectmap file reading issue.
> open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", O_RDONLY) = 11, 
> the first time it reads 32k, the read succeeds with 63 bytes and it
> tries to read again with 27k and the read returns 0 bytes and the CEPH 
> osd segfaults.

Can you generate a full log with --debug-osd 20 --debug-filestore 20 
--debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere?  It 
should tell us where things are going wrong.  In particular, we want to 
see if that file/object is being written properly.  It will also have a 
backtrace showing exactly where it crashed.

> Please note that ceph-disk prepare creates a journal in a path which is 
> not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb).
> So after step3 above I remove this journal below and manually create a 
> journal file before doing step4 above.
> 
> 
> ls -l /var/lib/ceph/osd/ceph-0/
> total 16
> -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid
> -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid
> lrwxrwxrwx 1 root root 58 Oct 22 21:40 journal -> /dev/disk/by-
> partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb

Is there anything in /dev/disk/by-partuuid/ or is it missing entirely?  
Maybe you have an old udev.  What distro is this?

sage

> -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid
> -rw-r--r-- 1 root root 21 Oct 22 21:40 magic
> 
> Any pointers to move ahead will be greatly appreciated??
> 
> thanks
> Prashanth
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

RE: Having issues trying to get the OSD up on a MIPS64!!!

[Prashanth Nednoor]

[-- Attachment #1: Type: text/plain, Size: 6900 bytes --]

Hi Sage,

Thank you for the prompt response.
Is there anything in /dev/disk/by-partuuid/ or is it missing entirely?  
  Nothing , it was Missing Entirely. 
  GOOD NEWS:  I worked around  this issue, if I set my journal path in the /etc/ceph.conf.

My udev version is
udevd --version
164 

I still see the segfaults, I have attached details.
I put the osd debug logs(osd-output.txt) and the leveldb_bt(leveldb_bt.txt).
Looks like we have an issue in leveldb....

HERE IS THE BACK TRACE: I have attached the gdb before running it.
#0  0x77f68ee0 in leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const* const&, leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::Node**) const () from /usr/local/lib/libleveldb.so.1
#1  0x77f69054 in leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::Insert(char const* const&) () from /usr/local/lib/libleveldb.so.1
#2  0x77f68618 in leveldb::MemTable::Add(unsigned long long, leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) ()
   from /usr/local/lib/libleveldb.so.1
#3  0x77f7e434 in leveldb::(anonymous namespace)::MemTableInserter::Put(leveldb::Slice const&, leveldb::Slice const&) ()
   from /usr/local/lib/libleveldb.so.1
#4  0x77f7e93c in leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const () from /usr/local/lib/libleveldb.so.1
#5  0x77f7eb8c in leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch const*, leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1
#6  0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions const&, leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1
#7  0x00a5dda0 in LevelDBStore::submit_transaction_sync (this=0x1f77d10, t=<value optimized out>) at os/LevelDBStore.cc:146
#8  0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0, spos=0x72cfe3b8) at os/DBObjectMap.cc:1126
#9  0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, fd=17, spos=..., hoid=0x0, in_progress=false) at os/FileStore.cc:2070
#10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450, cid=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at os/FileStore.cc:2047
#11 0x009b2138 in FileStore::_create_collection (this=0x1f72450, c=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at os/FileStore.cc:4753
#12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, t=..., op_seq=<value optimized out>, trans_num=0, handle=0x72cfec3c) at os/FileStore.cc:2413
#13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450, tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952
#14 0x009eb858 in FileStore::_do_op (this=0x1f72450, osr=0x1f801b8, handle=...) at os/FileStore.cc:1761
#15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0, wt=0x1f7ea90) at common/WorkQueue.cc:128
#16 0x00c91b94 in ThreadPool::WorkThread::entry() ()
#17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0
#18 0x777c1738 in ?? () from /lib/libc.so.6

Do  I need to set any variable to set the cache size etcetc in ceph.conf.
I only have osd_leveldb_cache_size=5242880 for now.


Thanks
Prashanth







-----Original Message-----
From: Sage Weil [mailto:sage@newdream.net] 
Sent: Thursday, October 23, 2014 5:54 PM
To: Prashanth Nednoor
Cc: ceph-devel@vger.kernel.org
Subject: Re: Having issues trying to get the OSD up on a MIPS64!!!

Hi Prashanth,

On Thu, 23 Oct 2014, Prashanth Nednoor wrote:
> Hello Everyone,
> 
> We are using ceph-0.86, good news is we were able to compile and load 
> all the libraries and binaries needed to configure a CEPH-OSD on MIPS 
> 64 platform. The CEPH monitor is also able to detect the OSD, but not 
> up yet, as the osd activate failed.
> Since we don?t have the required CEPH deploy utility for MIPS64, we 
> are following the manual procedure to create and activate an OSD.
> We have disabled authentication between the clients and the OSD?s for 
> now.
> 
> Has any body tried CEPH on a MIPS64?
> /dev/sda is a 2TB local hard drive.
> 
> This is how my partition looks after ceph-disk-prepare 
> /home/prashan/ceph-0.86/src# parted GNU Parted 2.3 Using /dev/sda 
> Welcome to GNU Parted! Type 'help' to view a list of commands.
> (parted) p
> Model: ATA TOSHIBA MQ01ABB2 (scsi)
> Disk /dev/sda: 2000GB
> Sector size (logical/physical): 512B/4096B Partition Table: gpt
> 
> Number  Start   End     Size    File system  Name          Flags
>  2      1049kB  5369MB  5368MB               ceph journal
>  1      5370MB  2000GB  1995GB  xfs          ceph data
> 
> 
> 
> The following are the steps to create an OSD
> 1)	ceph-disk zap /dev/sda
> 2)	ceph-disk-prepare --cluster  f615496c-b40a-4905-bbcd-
> 2d3e181ff21a --fs-type xfs /dev/sda
> 3)	mount /dev/sda1 /var/lib/ceph/osd/ceph-0/
> 4)	ceph-osd -i 0 ?mkfs is giving an error , 
> filestore(/var/lib/ceph/osd/ceph-0) could not find
> 23c2fcde/osd_superblock/0//-1 in index: (2) No such file.
> After this it segfaults. We have analyzed this further with the help 
> of strace and root caused this as objectmap file reading issue.
> open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", O_RDONLY) = 
> 11, the first time it reads 32k, the read succeeds with 63 bytes and 
> it tries to read again with 27k and the read returns 0 bytes and the 
> CEPH osd segfaults.

Can you generate a full log with --debug-osd 20 --debug-filestore 20 --debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere?  It should tell us where things are going wrong.  In particular, we want to see if that file/object is being written properly.  It will also have a backtrace showing exactly where it crashed.

> Please note that ceph-disk prepare creates a journal in a path which 
> is not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb).
> So after step3 above I remove this journal below and manually create a 
> journal file before doing step4 above.
> 
> 
> ls -l /var/lib/ceph/osd/ceph-0/
> total 16
> -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid
> -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid lrwxrwxrwx 1 root root 58 
> Oct 22 21:40 journal -> /dev/disk/by- 
> partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb

Is there anything in /dev/disk/by-partuuid/ or is it missing entirely?  
Maybe you have an old udev.  What distro is this?

sage

> -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid
> -rw-r--r-- 1 root root 21 Oct 22 21:40 magic
> 
> Any pointers to move ahead will be greatly appreciated??
> 
> thanks
> Prashanth
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
> 
> 

[-- Attachment #2: leveldb_bt.txt --]
[-- Type: text/plain, Size: 2548 bytes --]


#0  0x77f68ee0 in leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const* const&, leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::Node**) const () from /usr/local/lib/libleveldb.so.1
#1  0x77f69054 in leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::Insert(char const* const&) () from /usr/local/lib/libleveldb.so.1
#2  0x77f68618 in leveldb::MemTable::Add(unsigned long long, leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) ()
   from /usr/local/lib/libleveldb.so.1
#3  0x77f7e434 in leveldb::(anonymous namespace)::MemTableInserter::Put(leveldb::Slice const&, leveldb::Slice const&) ()
   from /usr/local/lib/libleveldb.so.1
#4  0x77f7e93c in leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const () from /usr/local/lib/libleveldb.so.1
#5  0x77f7eb8c in leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch const*, leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1
#6  0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions const&, leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1
#7  0x00a5dda0 in LevelDBStore::submit_transaction_sync (this=0x1f77d10, t=<value optimized out>) at os/LevelDBStore.cc:146
#8  0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0, spos=0x72cfe3b8) at os/DBObjectMap.cc:1126
#9  0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, fd=17, spos=..., hoid=0x0, in_progress=false) at os/FileStore.cc:2070
#10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450, cid=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at os/FileStore.cc:2047
#11 0x009b2138 in FileStore::_create_collection (this=0x1f72450, c=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at os/FileStore.cc:4753
#12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, t=..., op_seq=<value optimized out>, trans_num=0, handle=0x72cfec3c) at os/FileStore.cc:2413
#13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450, tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952
#14 0x009eb858 in FileStore::_do_op (this=0x1f72450, osr=0x1f801b8, handle=...) at os/FileStore.cc:1761
#15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0, wt=0x1f7ea90) at common/WorkQueue.cc:128
#16 0x00c91b94 in ThreadPool::WorkThread::entry() ()
#17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0
#18 0x777c1738 in ?? () from /lib/libc.so.6

[-- Attachment #3: osd-output.txt --]
[-- Type: text/plain, Size: 12344 bytes --]

This is were my journal points to
/var/lib/ceph/osd/ceph-0/journal -> /dev/disk/by-partuuid/9a868ac8-5ec8-4047-aa4c-8e9e6ce0

In the above output, for my system The path "/dev/disk/by-partuuid/" is missing.

udevd --version
164


gdb --args ceph-osd -i=0 --mkfs --debug-osd 20 --debug-filestore 20 --debug-journal 20 --mkfs -d
root@octeon:/home/prashan/ceph-0.86/src# ./ceph-osd -i=0 --mkfs --debug-osd 20 --debug-filestore 20 --debug-journal 20 --mkfs -d
 ./ceph-osd -i=0 --mkfs --debug-osd 20 --debug-filestore 20 --debug-journal 20 --mkfs -d
2014-10-24 19:26:59.981239 7760b000  0 ceph version 0.86 (97dcc0539dfa7dac3de74852305d51580b7b1f82), process ceph-osd, pid 3563
2014-10-24 19:26:59.982021 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) dump_stop
2014-10-24 19:26:59.983418 7760b000  1 filestore(/var/lib/ceph/osd/ceph-0) mkfs in /var/lib/ceph/osd/ceph-0
2014-10-24 19:27:00.001999 7760b000  1 filestore(/var/lib/ceph/osd/ceph-0) mkfs fsid is already set to 536991b9-cc6d-4023-860d-53ad361dce7e
2014-10-24 19:27:00.126542 7760b000  0 filestore(/var/lib/ceph/osd/ceph-0) backend generic (magic 0x58465342)
2014-10-24 19:27:00.126590 7760b000  1 filestore(/var/lib/ceph/osd/ceph-0)  disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs
2014-10-24 19:27:00.338490 7760b000  1 filestore(/var/lib/ceph/osd/ceph-0) leveldb db exists/created
2014-10-24 19:27:00.338577 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) open_journal at /dev/sda2
2014-10-24 19:27:00.338752 7760b000 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056)
2014-10-24 19:27:00.345391 7760b000 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/dev/sda2)
2014-10-24 19:27:00.345793 7760b000  1 journal _open /dev/sda2 fd 8: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-10-24 19:27:00.345857 7760b000 10 journal read_header
2014-10-24 19:27:00.360413 7760b000 10 journal header: block_size 4096 alignment 4096 max_size 5367660544
2014-10-24 19:27:00.360442 7760b000 10 journal header: start 4096
2014-10-24 19:27:00.360448 7760b000 10 journal  write_pos 0
2014-10-24 19:27:00.360485 7760b000 -1 journal check: ondisk fsid 8b1f947b-0a47-4f4e-bcd7-57ed33d5d300 doesn't match expected 536991b9-cc6d-4023-860d-53ad361dce7e, invalid (someone else's?) journal
2014-10-24 19:27:00.360636 7760b000  2 journal create /dev/sda2 fsid 536991b9-cc6d-4023-860d-53ad361dce7e
2014-10-24 19:27:00.360726 7760b000 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056)
2014-10-24 19:27:00.372395 7760b000 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/dev/sda2)
2014-10-24 19:27:00.372554 7760b000  1 journal _open /dev/sda2 fd 8: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-10-24 19:27:00.372588 7760b000 10 journal header: block_size 4096 alignment 4096 max_size 5367660544
2014-10-24 19:27:00.372596 7760b000 10 journal header: start 4096
2014-10-24 19:27:00.372601 7760b000 10 journal  write_pos 0
2014-10-24 19:27:00.393775 7760b000  2 journal create done
2014-10-24 19:27:00.394579 7760b000  0 filestore(/var/lib/ceph/osd/ceph-0) mkjournal created journal on /dev/sda2
2014-10-24 19:27:00.394712 7760b000  1 filestore(/var/lib/ceph/osd/ceph-0) mkfs done in /var/lib/ceph/osd/ceph-0
2014-10-24 19:27:00.394774 7760b000  5 filestore(/var/lib/ceph/osd/ceph-0) basedir /var/lib/ceph/osd/ceph-0 journal /dev/sda2
2014-10-24 19:27:00.394897 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) mount fsid is 536991b9-cc6d-4023-860d-53ad361dce7e
2014-10-24 19:27:00.395107 7760b000  0 filestore(/var/lib/ceph/osd/ceph-0) backend generic (magic 0x58465342)
2014-10-24 19:27:00.526956 7760b000  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is supported and appears to work
2014-10-24 19:27:00.526999 7760b000  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2014-10-24 19:27:00.527553 7760b000  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall not supported
2014-10-24 19:27:00.527595 7760b000  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: no syncfs(2), must use sync(2).
2014-10-24 19:27:00.527600 7760b000  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: WARNING: multiple ceph-osd daemons on the same host will be slow
2014-10-24 19:27:00.528461 7760b000  5 filestore(/var/lib/ceph/osd/ceph-0) mount op_seq is 1
2014-10-24 19:27:00.660719 7760b000 20 filestore (init)dbobjectmap: seq is 1
2014-10-24 19:27:00.660791 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) open_journal at /dev/sda2
2014-10-24 19:27:00.660883 7760b000  0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2014-10-24 19:27:00.660901 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) list_collections
2014-10-24 19:27:00.661284 7760b000 10 journal journal_replay fs op_seq 1
2014-10-24 19:27:00.661293 752ff460 20 filestore(/var/lib/ceph/osd/ceph-0) sync_entry waiting for max_interval 5.000000
2014-10-24 19:27:00.661318 7760b000  2 journal open /dev/sda2 fsid 536991b9-cc6d-4023-860d-53ad361dce7e fs_op_seq 1
2014-10-24 19:27:00.661377 7760b000 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056)
2014-10-24 19:27:00.668052 7760b000 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/dev/sda2)
2014-10-24 19:27:00.668387 7760b000  1 journal _open /dev/sda2 fd 14: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-10-24 19:27:00.668426 7760b000 10 journal read_header
2014-10-24 19:27:00.682659 7760b000 10 journal header: block_size 4096 alignment 4096 max_size 5367660544
2014-10-24 19:27:00.682686 7760b000 10 journal header: start 4096
2014-10-24 19:27:00.682692 7760b000 10 journal  write_pos 4096
2014-10-24 19:27:00.682705 7760b000 10 journal open header.fsid = 536991b9-cc6d-4023-860d-53ad361dce7e
2014-10-24 19:27:00.683010 7760b000  2 journal No further valid entries found, journal is most likely valid
2014-10-24 19:27:00.683024 7760b000 10 journal open reached end of journal.
2014-10-24 19:27:00.683070 7760b000  2 journal No further valid entries found, journal is most likely valid
2014-10-24 19:27:00.683078 7760b000  3 journal journal_replay: end of journal, done.
2014-10-24 19:27:00.683084 7760b000 10 journal make_writeable
2014-10-24 19:27:00.683372 7760b000 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056)
2014-10-24 19:27:00.694396 7760b000 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/dev/sda2)
2014-10-24 19:27:00.694542 7760b000  1 journal _open /dev/sda2 fd 14: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-10-24 19:27:00.694677 74aff460 10 journal write_thread_entry start
2014-10-24 19:27:00.694719 74aff460 20 journal prepare_multi_write queue_pos now 4096
2014-10-24 19:27:00.694753 73aff460 10 journal write_finish_thread_entry enter
2014-10-24 19:27:00.694759 7760b000 10 journal journal_start
2014-10-24 19:27:00.694773 73aff460 20 journal write_finish_thread_entry sleeping
2014-10-24 19:27:00.694867 74aff460 15 journal do_aio_write writing 4096~0 + header
2014-10-24 19:27:00.694890 74aff460 20 journal write_aio_bl 0~4096 seq 0
2014-10-24 19:27:00.694958 74aff460 20 journal write_aio_bl .. 0~4096 in 1
2014-10-24 19:27:00.695129 74aff460 20 journal write_aio_bl 4096~0 seq 0
2014-10-24 19:27:00.695306 73aff460 20 journal write_finish_thread_entry waiting for aio(s)
2014-10-24 19:27:00.695306 74aff460  5 journal put_throttle finished 0 ops and 0 bytes, now 0 ops and 0 bytes
2014-10-24 19:27:00.695333 74aff460 20 journal write_thread_entry going to sleep
2014-10-24 19:27:00.695357 73aff460 10 journal write_finish_thread_entry aio 0~4096 done
2014-10-24 19:27:00.695373 73aff460 20 journal check_aio_completion
2014-10-24 19:27:00.695377 73aff460 20 journal check_aio_completion completed seq 0 0~4096
2014-10-24 19:27:00.695392 73aff460 20 journal write_finish_thread_entry sleeping
2014-10-24 19:27:00.695822 7760b000 15 filestore(/var/lib/ceph/osd/ceph-0) read meta/23c2fcde/osd_superblock/0//-1 0~0
2014-10-24 19:27:00.696088 7760b000 -1 filestore(/var/lib/ceph/osd/ceph-0) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2014-10-24 19:27:00.696207 7760b000 10 filestore(/var/lib/ceph/osd/ceph-0) FileStore::read(meta/23c2fcde/osd_superblock/0//-1) open error: (2) No such file or directory
2014-10-24 19:27:00.696479 7760b000  5 filestore(/var/lib/ceph/osd/ceph-0) queue_transactions new osr(default 0x1f72ac8)/0x1f72ac8
2014-10-24 19:27:00.696612 7760b000 10 journal op_submit_start 2
2014-10-24 19:27:00.696692 7760b000  5 filestore(/var/lib/ceph/osd/ceph-0) queue_transactions (writeahead) 2 0x7f73ea18
2014-10-24 19:27:00.696766 7760b000 10 journal op_journal_transactions 2 0x7f73ea18
2014-10-24 19:27:00.696908 7760b000  5 journal submit_entry seq 2 len 534 (0x7630c3f8)
2014-10-24 19:27:00.697037 7760b000 10 journal op_submit_finish 2
2014-10-24 19:27:00.697046 74aff460 20 journal write_thread_entry woke up
2014-10-24 19:27:00.697088 74aff460 10 journal room 5367656447 max_size 5367660544 pos 4096 header.start 4096 top 4096
2014-10-24 19:27:00.697104 74aff460 10 journal check_for_full at 4096 : 4096 < 5367656447
2014-10-24 19:27:00.697115 74aff460 15 journal prepare_single_write 1 will write 4096 : seq 2 len 534 -> 4096 (head 40 pre_pad 0 ebl 534 post_pad 3482 tail 40) (ebl alignment -1)
2014-10-24 19:27:00.697166 74aff460 20 journal prepare_multi_write queue_pos now 8192
2014-10-24 19:27:00.697176 74aff460 15 journal do_aio_write writing 4096~4096
2014-10-24 19:27:00.697226 74aff460 20 journal write_aio_bl 4096~4096 seq 2
2014-10-24 19:27:00.697244 74aff460 20 journal write_aio_bl .. 4096~4096 in 1
2014-10-24 19:27:00.704964 73aff460 20 journal write_finish_thread_entry waiting for aio(s)
2014-10-24 19:27:00.704958 74aff460  5 journal put_throttle finished 1 ops and 534 bytes, now 0 ops and 0 bytes
2014-10-24 19:27:00.704995 74aff460 20 journal write_thread_entry going to sleep
2014-10-24 19:27:00.705062 73aff460 10 journal write_finish_thread_entry aio 4096~4096 done
2014-10-24 19:27:00.705078 73aff460 20 journal check_aio_completion
2014-10-24 19:27:00.705082 73aff460 20 journal check_aio_completion completed seq 2 4096~4096
2014-10-24 19:27:00.705094 73aff460 20 journal check_aio_completion queueing finishers through seq 2
2014-10-24 19:27:00.705104 73aff460 10 journal queue_completions_thru seq 2 queueing seq 2 0x7630c3f8 lat 0.008102
2014-10-24 19:27:00.705137 73aff460 20 journal write_finish_thread_entry sleeping
2014-10-24 19:27:00.705154 732ff460  5 filestore(/var/lib/ceph/osd/ceph-0) _journaled_ahead 0x7630c0e0 seq 2 osr(default 0x1f72ac8) 0x7f73ea18
2014-10-24 19:27:00.705174 732ff460  5 filestore(/var/lib/ceph/osd/ceph-0) queue_op 0x7630c0e0 seq 2 osr(default 0x1f72ac8) 528 bytes   (queue has 1 ops and 528 bytes)
2014-10-24 19:27:00.705220 72aff460 10 journal op_apply_start 2 open_ops 0 -> 1
2014-10-24 19:27:00.705229 72aff460  5 filestore(/var/lib/ceph/osd/ceph-0) _do_op 0x7630c0e0 seq 2 osr(default 0x1f72ac8)/0x1f72ac8 start
2014-10-24 19:27:00.705242 72aff460 10 filestore(/var/lib/ceph/osd/ceph-0) _do_transaction on 0x7f73ea18
2014-10-24 19:27:00.705283 72aff460 15 filestore(/var/lib/ceph/osd/ceph-0) create_collection /var/lib/ceph/osd/ceph-0/current/meta
2014-10-24 19:27:00.733384 72aff460 10 filestore(/var/lib/ceph/osd/ceph-0) create_collection /var/lib/ceph/osd/ceph-0/current/meta = 0
2014-10-24 19:27:00.733695 72aff460 10 filestore(/var/lib/ceph/osd/ceph-0) _set_replay_guard 2.0.0
2014-10-24 19:27:00.771441 72aff460 20 filestore dbobjectmap: seq is 1
*** Caught signal (Segmentation fault) **
 in thread 72aff460
Segmentation fault

RE: Having issues trying to get the OSD up on a MIPS64!!!

[Sage Weil]

Hi Prashanth,

On Fri, 24 Oct 2014, Prashanth Nednoor wrote:
> Hi Sage,
> 
> Thank you for the prompt response.
> Is there anything in /dev/disk/by-partuuid/ or is it missing entirely?  
>   Nothing , it was Missing Entirely. 
>   GOOD NEWS:  I worked around  this issue, if I set my journal path in the /etc/ceph.conf.
> 
> My udev version is udevd --version 164

Hmm, that should be new enough, but it seems like it isn't setting up the 
links.  What distro is it?  On most systems it's 
/lib/udev/rules.d/60-persistent-storage.rules that does it.  Maybe see if 
running partprobe /dev/sda or run 'udevadm monitor' and do 'udevadm 
trigger /dev/sda' in another terminal to see what happens.

Or, work around it like you did. :)

> I still see the segfaults, I have attached details.
> I put the osd debug logs(osd-output.txt) and the leveldb_bt(leveldb_bt.txt).
> Looks like we have an issue in leveldb....

Yeah, that looks like a problem with leveldb.  What distro is this?  What 
version leveldb?

I don't actually know anything about MIPS.. what's teh wordsize and 
endianess?

sage


> 
> HERE IS THE BACK TRACE: I have attached the gdb before running it.
> #0  0x77f68ee0 in leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const* const&, leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::Node**) const () from /usr/local/lib/libleveldb.so.1
> #1  0x77f69054 in leveldb::SkipList<char const*, leveldb::MemTable::KeyComparator>::Insert(char const* const&) () from /usr/local/lib/libleveldb.so.1
> #2  0x77f68618 in leveldb::MemTable::Add(unsigned long long, leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) ()
>    from /usr/local/lib/libleveldb.so.1
> #3  0x77f7e434 in leveldb::(anonymous namespace)::MemTableInserter::Put(leveldb::Slice const&, leveldb::Slice const&) ()
>    from /usr/local/lib/libleveldb.so.1
> #4  0x77f7e93c in leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const () from /usr/local/lib/libleveldb.so.1
> #5  0x77f7eb8c in leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch const*, leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1
> #6  0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions const&, leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1
> #7  0x00a5dda0 in LevelDBStore::submit_transaction_sync (this=0x1f77d10, t=<value optimized out>) at os/LevelDBStore.cc:146
> #8  0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0, spos=0x72cfe3b8) at os/DBObjectMap.cc:1126
> #9  0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, fd=17, spos=..., hoid=0x0, in_progress=false) at os/FileStore.cc:2070
> #10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450, cid=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
> ) at os/FileStore.cc:2047
> #11 0x009b2138 in FileStore::_create_collection (this=0x1f72450, c=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
> ) at os/FileStore.cc:4753
> #12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, t=..., op_seq=<value optimized out>, trans_num=0, handle=0x72cfec3c) at os/FileStore.cc:2413
> #13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450, tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952
> #14 0x009eb858 in FileStore::_do_op (this=0x1f72450, osr=0x1f801b8, handle=...) at os/FileStore.cc:1761
> #15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0, wt=0x1f7ea90) at common/WorkQueue.cc:128
> #16 0x00c91b94 in ThreadPool::WorkThread::entry() ()
> #17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0
> #18 0x777c1738 in ?? () from /lib/libc.so.6
> 
> Do  I need to set any variable to set the cache size etcetc in ceph.conf.
> I only have osd_leveldb_cache_size=5242880 for now.
> 
> 
> Thanks
> Prashanth
> 
> 
> 
> 
> 
> 
> 
> -----Original Message-----
> From: Sage Weil [mailto:sage@newdream.net] 
> Sent: Thursday, October 23, 2014 5:54 PM
> To: Prashanth Nednoor
> Cc: ceph-devel@vger.kernel.org
> Subject: Re: Having issues trying to get the OSD up on a MIPS64!!!
> 
> Hi Prashanth,
> 
> On Thu, 23 Oct 2014, Prashanth Nednoor wrote:
> > Hello Everyone,
> > 
> > We are using ceph-0.86, good news is we were able to compile and load 
> > all the libraries and binaries needed to configure a CEPH-OSD on MIPS 
> > 64 platform. The CEPH monitor is also able to detect the OSD, but not 
> > up yet, as the osd activate failed.
> > Since we don?t have the required CEPH deploy utility for MIPS64, we 
> > are following the manual procedure to create and activate an OSD.
> > We have disabled authentication between the clients and the OSD?s for 
> > now.
> > 
> > Has any body tried CEPH on a MIPS64?
> > /dev/sda is a 2TB local hard drive.
> > 
> > This is how my partition looks after ceph-disk-prepare 
> > /home/prashan/ceph-0.86/src# parted GNU Parted 2.3 Using /dev/sda 
> > Welcome to GNU Parted! Type 'help' to view a list of commands.
> > (parted) p
> > Model: ATA TOSHIBA MQ01ABB2 (scsi)
> > Disk /dev/sda: 2000GB
> > Sector size (logical/physical): 512B/4096B Partition Table: gpt
> > 
> > Number  Start   End     Size    File system  Name          Flags
> >  2      1049kB  5369MB  5368MB               ceph journal
> >  1      5370MB  2000GB  1995GB  xfs          ceph data
> > 
> > 
> > 
> > The following are the steps to create an OSD
> > 1)	ceph-disk zap /dev/sda
> > 2)	ceph-disk-prepare --cluster  f615496c-b40a-4905-bbcd-
> > 2d3e181ff21a --fs-type xfs /dev/sda
> > 3)	mount /dev/sda1 /var/lib/ceph/osd/ceph-0/
> > 4)	ceph-osd -i 0 ?mkfs is giving an error , 
> > filestore(/var/lib/ceph/osd/ceph-0) could not find
> > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file.
> > After this it segfaults. We have analyzed this further with the help 
> > of strace and root caused this as objectmap file reading issue.
> > open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", O_RDONLY) = 
> > 11, the first time it reads 32k, the read succeeds with 63 bytes and 
> > it tries to read again with 27k and the read returns 0 bytes and the 
> > CEPH osd segfaults.
> 
> Can you generate a full log with --debug-osd 20 --debug-filestore 20 --debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere?  It should tell us where things are going wrong.  In particular, we want to see if that file/object is being written properly.  It will also have a backtrace showing exactly where it crashed.
> 
> > Please note that ceph-disk prepare creates a journal in a path which 
> > is not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb).
> > So after step3 above I remove this journal below and manually create a 
> > journal file before doing step4 above.
> > 
> > 
> > ls -l /var/lib/ceph/osd/ceph-0/
> > total 16
> > -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid
> > -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid lrwxrwxrwx 1 root root 58 
> > Oct 22 21:40 journal -> /dev/disk/by- 
> > partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb
> 
> Is there anything in /dev/disk/by-partuuid/ or is it missing entirely?  
> Maybe you have an old udev.  What distro is this?
> 
> sage
> 
> > -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid
> > -rw-r--r-- 1 root root 21 Oct 22 21:40 magic
> > 
> > Any pointers to move ahead will be greatly appreciated??
> > 
> > thanks
> > Prashanth
> > 
> > 
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> > in the body of a message to majordomo@vger.kernel.org More majordomo 
> > info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> 

RE: Having issues trying to get the OSD up on a MIPS64!!!

[Philip Kufeldt]

64 bit big endian

> -----Original Message-----
> From: Sage Weil [mailto:sage@newdream.net]
> Sent: Friday, October 24, 2014 5:47 PM
> To: Prashanth Nednoor
> Cc: ceph-devel@vger.kernel.org; Philip Kufeldt
> Subject: RE: Having issues trying to get the OSD up on a MIPS64!!!
> 
> Hi Prashanth,
> 
> On Fri, 24 Oct 2014, Prashanth Nednoor wrote:
> > Hi Sage,
> >
> > Thank you for the prompt response.
> > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely?
> >   Nothing , it was Missing Entirely.
> >   GOOD NEWS:  I worked around  this issue, if I set my journal path in the
> /etc/ceph.conf.
> >
> > My udev version is udevd --version 164
> 
> Hmm, that should be new enough, but it seems like it isn't setting up the
> links.  What distro is it?  On most systems it's /lib/udev/rules.d/60-persistent-
> storage.rules that does it.  Maybe see if running partprobe /dev/sda or run
> 'udevadm monitor' and do 'udevadm trigger /dev/sda' in another terminal to
> see what happens.
> 
> Or, work around it like you did. :)
> 
> > I still see the segfaults, I have attached details.
> > I put the osd debug logs(osd-output.txt) and the
> leveldb_bt(leveldb_bt.txt).
> > Looks like we have an issue in leveldb....
> 
> Yeah, that looks like a problem with leveldb.  What distro is this?  What
> version leveldb?
> 
> I don't actually know anything about MIPS.. what's teh wordsize and
> endianess?
> 
> sage
> 
> 
> >
> > HERE IS THE BACK TRACE: I have attached the gdb before running it.
> > #0  0x77f68ee0 in leveldb::SkipList<char const*,
> > leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const*
> > const&, leveldb::SkipList<char const*,
> > leveldb::MemTable::KeyComparator>::Node**) const () from
> > /usr/local/lib/libleveldb.so.1
> > #1  0x77f69054 in leveldb::SkipList<char const*,
> > leveldb::MemTable::KeyComparator>::Insert(char const* const&) () from
> > /usr/local/lib/libleveldb.so.1
> > #2  0x77f68618 in leveldb::MemTable::Add(unsigned long long,
> leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) ()
> >    from /usr/local/lib/libleveldb.so.1
> > #3  0x77f7e434 in leveldb::(anonymous
> namespace)::MemTableInserter::Put(leveldb::Slice const&, leveldb::Slice
> const&) ()
> >    from /usr/local/lib/libleveldb.so.1
> > #4  0x77f7e93c in
> > leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const ()
> > from /usr/local/lib/libleveldb.so.1
> > #5  0x77f7eb8c in
> > leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch const*,
> > leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1
> > #6  0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions const&,
> > leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1
> > #7  0x00a5dda0 in LevelDBStore::submit_transaction_sync
> > (this=0x1f77d10, t=<value optimized out>) at os/LevelDBStore.cc:146
> > #8  0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0,
> > spos=0x72cfe3b8) at os/DBObjectMap.cc:1126
> > #9  0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, fd=17,
> > spos=..., hoid=0x0, in_progress=false) at os/FileStore.cc:2070
> > #10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450,
> cid=DWARF-2 expression error: DW_OP_reg operations must be used either
> alone or in conjuction with DW_OP_piece.
> > ) at os/FileStore.cc:2047
> > #11 0x009b2138 in FileStore::_create_collection (this=0x1f72450, c=DWARF-
> 2 expression error: DW_OP_reg operations must be used either alone or in
> conjuction with DW_OP_piece.
> > ) at os/FileStore.cc:4753
> > #12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, t=...,
> > op_seq=<value optimized out>, trans_num=0, handle=0x72cfec3c) at
> > os/FileStore.cc:2413
> > #13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450,
> > tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952
> > #14 0x009eb858 in FileStore::_do_op (this=0x1f72450, osr=0x1f801b8,
> > handle=...) at os/FileStore.cc:1761
> > #15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0, wt=0x1f7ea90) at
> > common/WorkQueue.cc:128
> > #16 0x00c91b94 in ThreadPool::WorkThread::entry() ()
> > #17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0
> > #18 0x777c1738 in ?? () from /lib/libc.so.6
> >
> > Do  I need to set any variable to set the cache size etcetc in ceph.conf.
> > I only have osd_leveldb_cache_size=5242880 for now.
> >
> >
> > Thanks
> > Prashanth
> >
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Sage Weil [mailto:sage@newdream.net]
> > Sent: Thursday, October 23, 2014 5:54 PM
> > To: Prashanth Nednoor
> > Cc: ceph-devel@vger.kernel.org
> > Subject: Re: Having issues trying to get the OSD up on a MIPS64!!!
> >
> > Hi Prashanth,
> >
> > On Thu, 23 Oct 2014, Prashanth Nednoor wrote:
> > > Hello Everyone,
> > >
> > > We are using ceph-0.86, good news is we were able to compile and
> > > load all the libraries and binaries needed to configure a CEPH-OSD
> > > on MIPS
> > > 64 platform. The CEPH monitor is also able to detect the OSD, but
> > > not up yet, as the osd activate failed.
> > > Since we don?t have the required CEPH deploy utility for MIPS64, we
> > > are following the manual procedure to create and activate an OSD.
> > > We have disabled authentication between the clients and the OSD?s
> > > for now.
> > >
> > > Has any body tried CEPH on a MIPS64?
> > > /dev/sda is a 2TB local hard drive.
> > >
> > > This is how my partition looks after ceph-disk-prepare
> > > /home/prashan/ceph-0.86/src# parted GNU Parted 2.3 Using /dev/sda
> > > Welcome to GNU Parted! Type 'help' to view a list of commands.
> > > (parted) p
> > > Model: ATA TOSHIBA MQ01ABB2 (scsi)
> > > Disk /dev/sda: 2000GB
> > > Sector size (logical/physical): 512B/4096B Partition Table: gpt
> > >
> > > Number  Start   End     Size    File system  Name          Flags
> > >  2      1049kB  5369MB  5368MB               ceph journal
> > >  1      5370MB  2000GB  1995GB  xfs          ceph data
> > >
> > >
> > >
> > > The following are the steps to create an OSD
> > > 1)	ceph-disk zap /dev/sda
> > > 2)	ceph-disk-prepare --cluster  f615496c-b40a-4905-bbcd-
> > > 2d3e181ff21a --fs-type xfs /dev/sda
> > > 3)	mount /dev/sda1 /var/lib/ceph/osd/ceph-0/
> > > 4)	ceph-osd -i 0 ?mkfs is giving an error ,
> > > filestore(/var/lib/ceph/osd/ceph-0) could not find
> > > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file.
> > > After this it segfaults. We have analyzed this further with the help
> > > of strace and root caused this as objectmap file reading issue.
> > > open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", O_RDONLY)
> =
> > > 11, the first time it reads 32k, the read succeeds with 63 bytes and
> > > it tries to read again with 27k and the read returns 0 bytes and the
> > > CEPH osd segfaults.
> >
> > Can you generate a full log with --debug-osd 20 --debug-filestore 20 --
> debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere?  It
> should tell us where things are going wrong.  In particular, we want to see if
> that file/object is being written properly.  It will also have a backtrace
> showing exactly where it crashed.
> >
> > > Please note that ceph-disk prepare creates a journal in a path which
> > > is not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492-
> 080ad2a505cb).
> > > So after step3 above I remove this journal below and manually create
> > > a journal file before doing step4 above.
> > >
> > >
> > > ls -l /var/lib/ceph/osd/ceph-0/
> > > total 16
> > > -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid
> > > -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid lrwxrwxrwx 1 root root
> > > 58 Oct 22 21:40 journal -> /dev/disk/by-
> > > partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb
> >
> > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely?
> > Maybe you have an old udev.  What distro is this?
> >
> > sage
> >
> > > -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid
> > > -rw-r--r-- 1 root root 21 Oct 22 21:40 magic
> > >
> > > Any pointers to move ahead will be greatly appreciated??
> > >
> > > thanks
> > > Prashanth
> > >
> > >
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > > in the body of a message to majordomo@vger.kernel.org More
> majordomo
> > > info at  http://vger.kernel.org/majordomo-info.html
> > >
> > >
> >

RE: Having issues trying to get the OSD up on a MIPS64!!!

[Sage Weil]

On Sat, 25 Oct 2014, Philip Kufeldt wrote:
> 64 bit big endian

My guess is that there is an endianness bug in leveldb then.  I wonder who 
else has tried it on MIPS?

sage


> 
> > -----Original Message-----
> > From: Sage Weil [mailto:sage@newdream.net]
> > Sent: Friday, October 24, 2014 5:47 PM
> > To: Prashanth Nednoor
> > Cc: ceph-devel@vger.kernel.org; Philip Kufeldt
> > Subject: RE: Having issues trying to get the OSD up on a MIPS64!!!
> > 
> > Hi Prashanth,
> > 
> > On Fri, 24 Oct 2014, Prashanth Nednoor wrote:
> > > Hi Sage,
> > >
> > > Thank you for the prompt response.
> > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely?
> > >   Nothing , it was Missing Entirely.
> > >   GOOD NEWS:  I worked around  this issue, if I set my journal path in the
> > /etc/ceph.conf.
> > >
> > > My udev version is udevd --version 164
> > 
> > Hmm, that should be new enough, but it seems like it isn't setting up the
> > links.  What distro is it?  On most systems it's /lib/udev/rules.d/60-persistent-
> > storage.rules that does it.  Maybe see if running partprobe /dev/sda or run
> > 'udevadm monitor' and do 'udevadm trigger /dev/sda' in another terminal to
> > see what happens.
> > 
> > Or, work around it like you did. :)
> > 
> > > I still see the segfaults, I have attached details.
> > > I put the osd debug logs(osd-output.txt) and the
> > leveldb_bt(leveldb_bt.txt).
> > > Looks like we have an issue in leveldb....
> > 
> > Yeah, that looks like a problem with leveldb.  What distro is this?  What
> > version leveldb?
> > 
> > I don't actually know anything about MIPS.. what's teh wordsize and
> > endianess?
> > 
> > sage
> > 
> > 
> > >
> > > HERE IS THE BACK TRACE: I have attached the gdb before running it.
> > > #0  0x77f68ee0 in leveldb::SkipList<char const*,
> > > leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const*
> > > const&, leveldb::SkipList<char const*,
> > > leveldb::MemTable::KeyComparator>::Node**) const () from
> > > /usr/local/lib/libleveldb.so.1
> > > #1  0x77f69054 in leveldb::SkipList<char const*,
> > > leveldb::MemTable::KeyComparator>::Insert(char const* const&) () from
> > > /usr/local/lib/libleveldb.so.1
> > > #2  0x77f68618 in leveldb::MemTable::Add(unsigned long long,
> > leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) ()
> > >    from /usr/local/lib/libleveldb.so.1
> > > #3  0x77f7e434 in leveldb::(anonymous
> > namespace)::MemTableInserter::Put(leveldb::Slice const&, leveldb::Slice
> > const&) ()
> > >    from /usr/local/lib/libleveldb.so.1
> > > #4  0x77f7e93c in
> > > leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const ()
> > > from /usr/local/lib/libleveldb.so.1
> > > #5  0x77f7eb8c in
> > > leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch const*,
> > > leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1
> > > #6  0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions const&,
> > > leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1
> > > #7  0x00a5dda0 in LevelDBStore::submit_transaction_sync
> > > (this=0x1f77d10, t=<value optimized out>) at os/LevelDBStore.cc:146
> > > #8  0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0,
> > > spos=0x72cfe3b8) at os/DBObjectMap.cc:1126
> > > #9  0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, fd=17,
> > > spos=..., hoid=0x0, in_progress=false) at os/FileStore.cc:2070
> > > #10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450,
> > cid=DWARF-2 expression error: DW_OP_reg operations must be used either
> > alone or in conjuction with DW_OP_piece.
> > > ) at os/FileStore.cc:2047
> > > #11 0x009b2138 in FileStore::_create_collection (this=0x1f72450, c=DWARF-
> > 2 expression error: DW_OP_reg operations must be used either alone or in
> > conjuction with DW_OP_piece.
> > > ) at os/FileStore.cc:4753
> > > #12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, t=...,
> > > op_seq=<value optimized out>, trans_num=0, handle=0x72cfec3c) at
> > > os/FileStore.cc:2413
> > > #13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450,
> > > tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952
> > > #14 0x009eb858 in FileStore::_do_op (this=0x1f72450, osr=0x1f801b8,
> > > handle=...) at os/FileStore.cc:1761
> > > #15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0, wt=0x1f7ea90) at
> > > common/WorkQueue.cc:128
> > > #16 0x00c91b94 in ThreadPool::WorkThread::entry() ()
> > > #17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0
> > > #18 0x777c1738 in ?? () from /lib/libc.so.6
> > >
> > > Do  I need to set any variable to set the cache size etcetc in ceph.conf.
> > > I only have osd_leveldb_cache_size=5242880 for now.
> > >
> > >
> > > Thanks
> > > Prashanth
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Sage Weil [mailto:sage@newdream.net]
> > > Sent: Thursday, October 23, 2014 5:54 PM
> > > To: Prashanth Nednoor
> > > Cc: ceph-devel@vger.kernel.org
> > > Subject: Re: Having issues trying to get the OSD up on a MIPS64!!!
> > >
> > > Hi Prashanth,
> > >
> > > On Thu, 23 Oct 2014, Prashanth Nednoor wrote:
> > > > Hello Everyone,
> > > >
> > > > We are using ceph-0.86, good news is we were able to compile and
> > > > load all the libraries and binaries needed to configure a CEPH-OSD
> > > > on MIPS
> > > > 64 platform. The CEPH monitor is also able to detect the OSD, but
> > > > not up yet, as the osd activate failed.
> > > > Since we don?t have the required CEPH deploy utility for MIPS64, we
> > > > are following the manual procedure to create and activate an OSD.
> > > > We have disabled authentication between the clients and the OSD?s
> > > > for now.
> > > >
> > > > Has any body tried CEPH on a MIPS64?
> > > > /dev/sda is a 2TB local hard drive.
> > > >
> > > > This is how my partition looks after ceph-disk-prepare
> > > > /home/prashan/ceph-0.86/src# parted GNU Parted 2.3 Using /dev/sda
> > > > Welcome to GNU Parted! Type 'help' to view a list of commands.
> > > > (parted) p
> > > > Model: ATA TOSHIBA MQ01ABB2 (scsi)
> > > > Disk /dev/sda: 2000GB
> > > > Sector size (logical/physical): 512B/4096B Partition Table: gpt
> > > >
> > > > Number  Start   End     Size    File system  Name          Flags
> > > >  2      1049kB  5369MB  5368MB               ceph journal
> > > >  1      5370MB  2000GB  1995GB  xfs          ceph data
> > > >
> > > >
> > > >
> > > > The following are the steps to create an OSD
> > > > 1)	ceph-disk zap /dev/sda
> > > > 2)	ceph-disk-prepare --cluster  f615496c-b40a-4905-bbcd-
> > > > 2d3e181ff21a --fs-type xfs /dev/sda
> > > > 3)	mount /dev/sda1 /var/lib/ceph/osd/ceph-0/
> > > > 4)	ceph-osd -i 0 ?mkfs is giving an error ,
> > > > filestore(/var/lib/ceph/osd/ceph-0) could not find
> > > > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file.
> > > > After this it segfaults. We have analyzed this further with the help
> > > > of strace and root caused this as objectmap file reading issue.
> > > > open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", O_RDONLY)
> > =
> > > > 11, the first time it reads 32k, the read succeeds with 63 bytes and
> > > > it tries to read again with 27k and the read returns 0 bytes and the
> > > > CEPH osd segfaults.
> > >
> > > Can you generate a full log with --debug-osd 20 --debug-filestore 20 --
> > debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere?  It
> > should tell us where things are going wrong.  In particular, we want to see if
> > that file/object is being written properly.  It will also have a backtrace
> > showing exactly where it crashed.
> > >
> > > > Please note that ceph-disk prepare creates a journal in a path which
> > > > is not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492-
> > 080ad2a505cb).
> > > > So after step3 above I remove this journal below and manually create
> > > > a journal file before doing step4 above.
> > > >
> > > >
> > > > ls -l /var/lib/ceph/osd/ceph-0/
> > > > total 16
> > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid
> > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid lrwxrwxrwx 1 root root
> > > > 58 Oct 22 21:40 journal -> /dev/disk/by-
> > > > partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb
> > >
> > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely?
> > > Maybe you have an old udev.  What distro is this?
> > >
> > > sage
> > >
> > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid
> > > > -rw-r--r-- 1 root root 21 Oct 22 21:40 magic
> > > >
> > > > Any pointers to move ahead will be greatly appreciated??
> > > >
> > > > thanks
> > > > Prashanth
> > > >
> > > >
> > > >
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > > > in the body of a message to majordomo@vger.kernel.org More
> > majordomo
> > > > info at  http://vger.kernel.org/majordomo-info.html
> > > >
> > > >
> > >
> 
> 

RE: Having issues trying to get the OSD up on a MIPS64!!!

[Prashanth Nednoor]

Hi Sage,

Leveldb version is 1.17.

Thanks
Prashanth

-----Original Message-----
From: Sage Weil [mailto:sage@newdream.net] 
Sent: Friday, October 24, 2014 6:11 PM
To: Philip Kufeldt
Cc: Prashanth Nednoor; ceph-devel@vger.kernel.org
Subject: RE: Having issues trying to get the OSD up on a MIPS64!!!

On Sat, 25 Oct 2014, Philip Kufeldt wrote:
> 64 bit big endian

My guess is that there is an endianness bug in leveldb then.  I wonder who else has tried it on MIPS?

sage


> 
> > -----Original Message-----
> > From: Sage Weil [mailto:sage@newdream.net]
> > Sent: Friday, October 24, 2014 5:47 PM
> > To: Prashanth Nednoor
> > Cc: ceph-devel@vger.kernel.org; Philip Kufeldt
> > Subject: RE: Having issues trying to get the OSD up on a MIPS64!!!
> > 
> > Hi Prashanth,
> > 
> > On Fri, 24 Oct 2014, Prashanth Nednoor wrote:
> > > Hi Sage,
> > >
> > > Thank you for the prompt response.
> > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely?
> > >   Nothing , it was Missing Entirely.
> > >   GOOD NEWS:  I worked around  this issue, if I set my journal 
> > > path in the
> > /etc/ceph.conf.
> > >
> > > My udev version is udevd --version 164
> > 
> > Hmm, that should be new enough, but it seems like it isn't setting 
> > up the links.  What distro is it?  On most systems it's 
> > /lib/udev/rules.d/60-persistent- storage.rules that does it.  Maybe 
> > see if running partprobe /dev/sda or run 'udevadm monitor' and do 
> > 'udevadm trigger /dev/sda' in another terminal to see what happens.
> > 
> > Or, work around it like you did. :)
> > 
> > > I still see the segfaults, I have attached details.
> > > I put the osd debug logs(osd-output.txt) and the
> > leveldb_bt(leveldb_bt.txt).
> > > Looks like we have an issue in leveldb....
> > 
> > Yeah, that looks like a problem with leveldb.  What distro is this?  
> > What version leveldb?
> > 
> > I don't actually know anything about MIPS.. what's teh wordsize and 
> > endianess?
> > 
> > sage
> > 
> > 
> > >
> > > HERE IS THE BACK TRACE: I have attached the gdb before running it.
> > > #0  0x77f68ee0 in leveldb::SkipList<char const*, 
> > > leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const* 
> > > const&, leveldb::SkipList<char const*,
> > > leveldb::MemTable::KeyComparator>::Node**) const () from
> > > /usr/local/lib/libleveldb.so.1
> > > #1  0x77f69054 in leveldb::SkipList<char const*, 
> > > leveldb::MemTable::KeyComparator>::Insert(char const* const&) () 
> > > from
> > > /usr/local/lib/libleveldb.so.1
> > > #2  0x77f68618 in leveldb::MemTable::Add(unsigned long long,
> > leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) ()
> > >    from /usr/local/lib/libleveldb.so.1
> > > #3  0x77f7e434 in leveldb::(anonymous
> > namespace)::MemTableInserter::Put(leveldb::Slice const&, 
> > leveldb::Slice
> > const&) ()
> > >    from /usr/local/lib/libleveldb.so.1
> > > #4  0x77f7e93c in
> > > leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const 
> > > () from /usr/local/lib/libleveldb.so.1
> > > #5  0x77f7eb8c in
> > > leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch 
> > > const*,
> > > leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1
> > > #6  0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions 
> > > const&,
> > > leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1
> > > #7  0x00a5dda0 in LevelDBStore::submit_transaction_sync
> > > (this=0x1f77d10, t=<value optimized out>) at 
> > > os/LevelDBStore.cc:146
> > > #8  0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0,
> > > spos=0x72cfe3b8) at os/DBObjectMap.cc:1126
> > > #9  0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, 
> > > fd=17, spos=..., hoid=0x0, in_progress=false) at 
> > > os/FileStore.cc:2070
> > > #10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450,
> > cid=DWARF-2 expression error: DW_OP_reg operations must be used 
> > either alone or in conjuction with DW_OP_piece.
> > > ) at os/FileStore.cc:2047
> > > #11 0x009b2138 in FileStore::_create_collection (this=0x1f72450, 
> > > c=DWARF-
> > 2 expression error: DW_OP_reg operations must be used either alone 
> > or in conjuction with DW_OP_piece.
> > > ) at os/FileStore.cc:4753
> > > #12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, 
> > > t=..., op_seq=<value optimized out>, trans_num=0, 
> > > handle=0x72cfec3c) at
> > > os/FileStore.cc:2413
> > > #13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450, 
> > > tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952
> > > #14 0x009eb858 in FileStore::_do_op (this=0x1f72450, 
> > > osr=0x1f801b8,
> > > handle=...) at os/FileStore.cc:1761
> > > #15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0, 
> > > wt=0x1f7ea90) at
> > > common/WorkQueue.cc:128
> > > #16 0x00c91b94 in ThreadPool::WorkThread::entry() ()
> > > #17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0
> > > #18 0x777c1738 in ?? () from /lib/libc.so.6
> > >
> > > Do  I need to set any variable to set the cache size etcetc in ceph.conf.
> > > I only have osd_leveldb_cache_size=5242880 for now.
> > >
> > >
> > > Thanks
> > > Prashanth
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Sage Weil [mailto:sage@newdream.net]
> > > Sent: Thursday, October 23, 2014 5:54 PM
> > > To: Prashanth Nednoor
> > > Cc: ceph-devel@vger.kernel.org
> > > Subject: Re: Having issues trying to get the OSD up on a MIPS64!!!
> > >
> > > Hi Prashanth,
> > >
> > > On Thu, 23 Oct 2014, Prashanth Nednoor wrote:
> > > > Hello Everyone,
> > > >
> > > > We are using ceph-0.86, good news is we were able to compile and 
> > > > load all the libraries and binaries needed to configure a 
> > > > CEPH-OSD on MIPS
> > > > 64 platform. The CEPH monitor is also able to detect the OSD, 
> > > > but not up yet, as the osd activate failed.
> > > > Since we don?t have the required CEPH deploy utility for MIPS64, 
> > > > we are following the manual procedure to create and activate an OSD.
> > > > We have disabled authentication between the clients and the 
> > > > OSD?s for now.
> > > >
> > > > Has any body tried CEPH on a MIPS64?
> > > > /dev/sda is a 2TB local hard drive.
> > > >
> > > > This is how my partition looks after ceph-disk-prepare 
> > > > /home/prashan/ceph-0.86/src# parted GNU Parted 2.3 Using 
> > > > /dev/sda Welcome to GNU Parted! Type 'help' to view a list of commands.
> > > > (parted) p
> > > > Model: ATA TOSHIBA MQ01ABB2 (scsi) Disk /dev/sda: 2000GB Sector 
> > > > size (logical/physical): 512B/4096B Partition Table: gpt
> > > >
> > > > Number  Start   End     Size    File system  Name          Flags
> > > >  2      1049kB  5369MB  5368MB               ceph journal
> > > >  1      5370MB  2000GB  1995GB  xfs          ceph data
> > > >
> > > >
> > > >
> > > > The following are the steps to create an OSD
> > > > 1)	ceph-disk zap /dev/sda
> > > > 2)	ceph-disk-prepare --cluster  f615496c-b40a-4905-bbcd-
> > > > 2d3e181ff21a --fs-type xfs /dev/sda
> > > > 3)	mount /dev/sda1 /var/lib/ceph/osd/ceph-0/
> > > > 4)	ceph-osd -i 0 ?mkfs is giving an error ,
> > > > filestore(/var/lib/ceph/osd/ceph-0) could not find
> > > > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file.
> > > > After this it segfaults. We have analyzed this further with the 
> > > > help of strace and root caused this as objectmap file reading issue.
> > > > open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log", 
> > > > O_RDONLY)
> > =
> > > > 11, the first time it reads 32k, the read succeeds with 63 bytes 
> > > > and it tries to read again with 27k and the read returns 0 bytes 
> > > > and the CEPH osd segfaults.
> > >
> > > Can you generate a full log with --debug-osd 20 --debug-filestore 
> > > 20 --
> > debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere?  
> > It should tell us where things are going wrong.  In particular, we 
> > want to see if that file/object is being written properly.  It will 
> > also have a backtrace showing exactly where it crashed.
> > >
> > > > Please note that ceph-disk prepare creates a journal in a path 
> > > > which is not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492-
> > 080ad2a505cb).
> > > > So after step3 above I remove this journal below and manually 
> > > > create a journal file before doing step4 above.
> > > >
> > > >
> > > > ls -l /var/lib/ceph/osd/ceph-0/
> > > > total 16
> > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid
> > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid lrwxrwxrwx 1 root 
> > > > root
> > > > 58 Oct 22 21:40 journal -> /dev/disk/by- 
> > > > partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb
> > >
> > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely?
> > > Maybe you have an old udev.  What distro is this?
> > >
> > > sage
> > >
> > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid
> > > > -rw-r--r-- 1 root root 21 Oct 22 21:40 magic
> > > >
> > > > Any pointers to move ahead will be greatly appreciated??
> > > >
> > > > thanks
> > > > Prashanth
> > > >
> > > >
> > > >
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > > > in the body of a message to majordomo@vger.kernel.org More
> > majordomo
> > > > info at  http://vger.kernel.org/majordomo-info.html
> > > >
> > > >
> > >
> 
> 

RE: Having issues trying to get the OSD up on a MIPS64!!!

[Prashanth Nednoor]

Sage,

Good news, I am able to create the OSD successfully, let's see what's in store next.

It was an issue with  leveldb1.17 not having  either memory barrier or atomic operation support for DEBIAN MIPS???
Not even the latest version leveldb1.18  I pulled from https://github.com/google/leveldb.

But this link  talks about that https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=681945

So, I ported over the memory barrier/atomic fix for MIPS onto leveldb1.17... I had to look into the mips/barrier.h files on our eval board, to make sure
We had the correct macros.

Now, my osd creation is successful on the MIPS, created object store /var/lib/ceph/osd/ceph-0 journal /dev/sda2 for osd.0 fsid f615496c-b40a-4905-bbcd-2d3e181ff21a
I have to start looking into the CLIENT/MONITOR side to make sure everything is good.

Really thankful for your suggestions for this quick resolution, for now we are good, untill the next and then the next......

Thanks
Prashanth

-----Original Message-----
From: Prashanth Nednoor 
Sent: Sunday, October 26, 2014 7:32 PM
To: 'Sage Weil'; Philip Kufeldt
Cc: ceph-devel@vger.kernel.org
Subject: RE: Having issues trying to get the OSD up on a MIPS64!!!

Hi Sage,

Leveldb version is 1.17.

Thanks
Prashanth

-----Original Message-----
From: Sage Weil [mailto:sage@newdream.net]
Sent: Friday, October 24, 2014 6:11 PM
To: Philip Kufeldt
Cc: Prashanth Nednoor; ceph-devel@vger.kernel.org
Subject: RE: Having issues trying to get the OSD up on a MIPS64!!!

On Sat, 25 Oct 2014, Philip Kufeldt wrote:
> 64 bit big endian

My guess is that there is an endianness bug in leveldb then.  I wonder who else has tried it on MIPS?

sage


> 
> > -----Original Message-----
> > From: Sage Weil [mailto:sage@newdream.net]
> > Sent: Friday, October 24, 2014 5:47 PM
> > To: Prashanth Nednoor
> > Cc: ceph-devel@vger.kernel.org; Philip Kufeldt
> > Subject: RE: Having issues trying to get the OSD up on a MIPS64!!!
> > 
> > Hi Prashanth,
> > 
> > On Fri, 24 Oct 2014, Prashanth Nednoor wrote:
> > > Hi Sage,
> > >
> > > Thank you for the prompt response.
> > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely?
> > >   Nothing , it was Missing Entirely.
> > >   GOOD NEWS:  I worked around  this issue, if I set my journal 
> > > path in the
> > /etc/ceph.conf.
> > >
> > > My udev version is udevd --version 164
> > 
> > Hmm, that should be new enough, but it seems like it isn't setting 
> > up the links.  What distro is it?  On most systems it's
> > /lib/udev/rules.d/60-persistent- storage.rules that does it.  Maybe 
> > see if running partprobe /dev/sda or run 'udevadm monitor' and do 
> > 'udevadm trigger /dev/sda' in another terminal to see what happens.
> > 
> > Or, work around it like you did. :)
> > 
> > > I still see the segfaults, I have attached details.
> > > I put the osd debug logs(osd-output.txt) and the
> > leveldb_bt(leveldb_bt.txt).
> > > Looks like we have an issue in leveldb....
> > 
> > Yeah, that looks like a problem with leveldb.  What distro is this?  
> > What version leveldb?
> > 
> > I don't actually know anything about MIPS.. what's teh wordsize and 
> > endianess?
> > 
> > sage
> > 
> > 
> > >
> > > HERE IS THE BACK TRACE: I have attached the gdb before running it.
> > > #0  0x77f68ee0 in leveldb::SkipList<char const*, 
> > > leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const* 
> > > const&, leveldb::SkipList<char const*,
> > > leveldb::MemTable::KeyComparator>::Node**) const () from
> > > /usr/local/lib/libleveldb.so.1
> > > #1  0x77f69054 in leveldb::SkipList<char const*, 
> > > leveldb::MemTable::KeyComparator>::Insert(char const* const&) () 
> > > from
> > > /usr/local/lib/libleveldb.so.1
> > > #2  0x77f68618 in leveldb::MemTable::Add(unsigned long long,
> > leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) ()
> > >    from /usr/local/lib/libleveldb.so.1
> > > #3  0x77f7e434 in leveldb::(anonymous
> > namespace)::MemTableInserter::Put(leveldb::Slice const&, 
> > leveldb::Slice
> > const&) ()
> > >    from /usr/local/lib/libleveldb.so.1
> > > #4  0x77f7e93c in
> > > leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const
> > > () from /usr/local/lib/libleveldb.so.1
> > > #5  0x77f7eb8c in
> > > leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch
> > > const*,
> > > leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1
> > > #6  0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions
> > > const&,
> > > leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1
> > > #7  0x00a5dda0 in LevelDBStore::submit_transaction_sync
> > > (this=0x1f77d10, t=<value optimized out>) at
> > > os/LevelDBStore.cc:146
> > > #8  0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0,
> > > spos=0x72cfe3b8) at os/DBObjectMap.cc:1126
> > > #9  0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, 
> > > fd=17, spos=..., hoid=0x0, in_progress=false) at
> > > os/FileStore.cc:2070
> > > #10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450,
> > cid=DWARF-2 expression error: DW_OP_reg operations must be used 
> > either alone or in conjuction with DW_OP_piece.
> > > ) at os/FileStore.cc:2047
> > > #11 0x009b2138 in FileStore::_create_collection (this=0x1f72450,
> > > c=DWARF-
> > 2 expression error: DW_OP_reg operations must be used either alone 
> > or in conjuction with DW_OP_piece.
> > > ) at os/FileStore.cc:4753
> > > #12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, 
> > > t=..., op_seq=<value optimized out>, trans_num=0,
> > > handle=0x72cfec3c) at
> > > os/FileStore.cc:2413
> > > #13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450, 
> > > tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952
> > > #14 0x009eb858 in FileStore::_do_op (this=0x1f72450, 
> > > osr=0x1f801b8,
> > > handle=...) at os/FileStore.cc:1761
> > > #15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0,
> > > wt=0x1f7ea90) at
> > > common/WorkQueue.cc:128
> > > #16 0x00c91b94 in ThreadPool::WorkThread::entry() ()
> > > #17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0
> > > #18 0x777c1738 in ?? () from /lib/libc.so.6
> > >
> > > Do  I need to set any variable to set the cache size etcetc in ceph.conf.
> > > I only have osd_leveldb_cache_size=5242880 for now.
> > >
> > >
> > > Thanks
> > > Prashanth
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Sage Weil [mailto:sage@newdream.net]
> > > Sent: Thursday, October 23, 2014 5:54 PM
> > > To: Prashanth Nednoor
> > > Cc: ceph-devel@vger.kernel.org
> > > Subject: Re: Having issues trying to get the OSD up on a MIPS64!!!
> > >
> > > Hi Prashanth,
> > >
> > > On Thu, 23 Oct 2014, Prashanth Nednoor wrote:
> > > > Hello Everyone,
> > > >
> > > > We are using ceph-0.86, good news is we were able to compile and 
> > > > load all the libraries and binaries needed to configure a 
> > > > CEPH-OSD on MIPS
> > > > 64 platform. The CEPH monitor is also able to detect the OSD, 
> > > > but not up yet, as the osd activate failed.
> > > > Since we don?t have the required CEPH deploy utility for MIPS64, 
> > > > we are following the manual procedure to create and activate an OSD.
> > > > We have disabled authentication between the clients and the 
> > > > OSD?s for now.
> > > >
> > > > Has any body tried CEPH on a MIPS64?
> > > > /dev/sda is a 2TB local hard drive.
> > > >
> > > > This is how my partition looks after ceph-disk-prepare 
> > > > /home/prashan/ceph-0.86/src# parted GNU Parted 2.3 Using 
> > > > /dev/sda Welcome to GNU Parted! Type 'help' to view a list of commands.
> > > > (parted) p
> > > > Model: ATA TOSHIBA MQ01ABB2 (scsi) Disk /dev/sda: 2000GB Sector 
> > > > size (logical/physical): 512B/4096B Partition Table: gpt
> > > >
> > > > Number  Start   End     Size    File system  Name          Flags
> > > >  2      1049kB  5369MB  5368MB               ceph journal
> > > >  1      5370MB  2000GB  1995GB  xfs          ceph data
> > > >
> > > >
> > > >
> > > > The following are the steps to create an OSD
> > > > 1)	ceph-disk zap /dev/sda
> > > > 2)	ceph-disk-prepare --cluster  f615496c-b40a-4905-bbcd-
> > > > 2d3e181ff21a --fs-type xfs /dev/sda
> > > > 3)	mount /dev/sda1 /var/lib/ceph/osd/ceph-0/
> > > > 4)	ceph-osd -i 0 ?mkfs is giving an error ,
> > > > filestore(/var/lib/ceph/osd/ceph-0) could not find
> > > > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file.
> > > > After this it segfaults. We have analyzed this further with the 
> > > > help of strace and root caused this as objectmap file reading issue.
> > > > open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log",
> > > > O_RDONLY)
> > =
> > > > 11, the first time it reads 32k, the read succeeds with 63 bytes 
> > > > and it tries to read again with 27k and the read returns 0 bytes 
> > > > and the CEPH osd segfaults.
> > >
> > > Can you generate a full log with --debug-osd 20 --debug-filestore
> > > 20 --
> > debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere?  
> > It should tell us where things are going wrong.  In particular, we 
> > want to see if that file/object is being written properly.  It will 
> > also have a backtrace showing exactly where it crashed.
> > >
> > > > Please note that ceph-disk prepare creates a journal in a path 
> > > > which is not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492-
> > 080ad2a505cb).
> > > > So after step3 above I remove this journal below and manually 
> > > > create a journal file before doing step4 above.
> > > >
> > > >
> > > > ls -l /var/lib/ceph/osd/ceph-0/
> > > > total 16
> > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid
> > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid lrwxrwxrwx 1 root 
> > > > root
> > > > 58 Oct 22 21:40 journal -> /dev/disk/by- 
> > > > partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb
> > >
> > > Is there anything in /dev/disk/by-partuuid/ or is it missing entirely?
> > > Maybe you have an old udev.  What distro is this?
> > >
> > > sage
> > >
> > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid
> > > > -rw-r--r-- 1 root root 21 Oct 22 21:40 magic
> > > >
> > > > Any pointers to move ahead will be greatly appreciated??
> > > >
> > > > thanks
> > > > Prashanth
> > > >
> > > >
> > > >
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > > > in the body of a message to majordomo@vger.kernel.org More
> > majordomo
> > > > info at  http://vger.kernel.org/majordomo-info.html
> > > >
> > > >
> > >
> 
> 

Re: Having issues trying to get the OSD up on a MIPS64!!!

[Bao Vu]

Prashanth Nednoor <Prashanth.Nednoor <at> taec.toshiba.com> writes:

> 
> Sage,
> 
> Good news, I am able to create the OSD successfully, let's see what's in 
store next.
> 
> It was an issue with  leveldb1.17 not having  either memory barrier or 
atomic operation support for DEBIAN MIPS???
> Not even the latest version leveldb1.18  I pulled from 
https://github.com/google/leveldb.
> 
> But this link  talks about that https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=681945
> 
> So, I ported over the memory barrier/atomic fix for MIPS onto 
leveldb1.17... I had to look into the
> mips/barrier.h files on our eval board, to make sure
> We had the correct macros.
> 
> Now, my osd creation is successful on the MIPS, created object 
store /var/lib/ceph/osd/ceph-0 journal
> /dev/sda2 for osd.0 fsid f615496c-b40a-4905-bbcd-2d3e181ff21a
> I have to start looking into the CLIENT/MONITOR side to make sure 
everything is good.
> 
> Really thankful for your suggestions for this quick resolution, for now 
we are good, untill the next and
> then the next......
> 
> Thanks
> Prashanth
> 
> -----Original Message-----
> From: Prashanth Nednoor 
> Sent: Sunday, October 26, 2014 7:32 PM
> To: 'Sage Weil'; Philip Kufeldt
> Cc: ceph-devel <at> vger.kernel.org
> Subject: RE: Having issues trying to get the OSD up on a MIPS64!!!
> 
> Hi Sage,
> 
> Leveldb version is 1.17.
> 
> Thanks
> Prashanth
> 
> -----Original Message-----
> From: Sage Weil [mailto:sage <at> newdream.net]
> Sent: Friday, October 24, 2014 6:11 PM
> To: Philip Kufeldt
> Cc: Prashanth Nednoor; ceph-devel <at> vger.kernel.org
> Subject: RE: Having issues trying to get the OSD up on a MIPS64!!!
> 
> On Sat, 25 Oct 2014, Philip Kufeldt wrote:
> > 64 bit big endian
> 
> My guess is that there is an endianness bug in leveldb then.  I wonder 
who else has tried it on MIPS?
> 
> sage
> 
> > 
> > > -----Original Message-----
> > > From: Sage Weil [mailto:sage <at> newdream.net]
> > > Sent: Friday, October 24, 2014 5:47 PM
> > > To: Prashanth Nednoor
> > > Cc: ceph-devel <at> vger.kernel.org; Philip Kufeldt
> > > Subject: RE: Having issues trying to get the OSD up on a MIPS64!!!
> > > 
> > > Hi Prashanth,
> > > 
> > > On Fri, 24 Oct 2014, Prashanth Nednoor wrote:
> > > > Hi Sage,
> > > >
> > > > Thank you for the prompt response.
> > > > Is there anything in /dev/disk/by-partuuid/ or is it missing 
entirely?
> > > >   Nothing , it was Missing Entirely.
> > > >   GOOD NEWS:  I worked around  this issue, if I set my journal 
> > > > path in the
> > > /etc/ceph.conf.
> > > >
> > > > My udev version is udevd --version 164
> > > 
> > > Hmm, that should be new enough, but it seems like it isn't setting 
> > > up the links.  What distro is it?  On most systems it's
> > > /lib/udev/rules.d/60-persistent- storage.rules that does it.  Maybe 
> > > see if running partprobe /dev/sda or run 'udevadm monitor' and do 
> > > 'udevadm trigger /dev/sda' in another terminal to see what happens.
> > > 
> > > Or, work around it like you did. :)
> > > 
> > > > I still see the segfaults, I have attached details.
> > > > I put the osd debug logs(osd-output.txt) and the
> > > leveldb_bt(leveldb_bt.txt).
> > > > Looks like we have an issue in leveldb....
> > > 
> > > Yeah, that looks like a problem with leveldb.  What distro is this?  
> > > What version leveldb?
> > > 
> > > I don't actually know anything about MIPS.. what's teh wordsize and 
> > > endianess?
> > > 
> > > sage
> > > 
> > > 
> > > >
> > > > HERE IS THE BACK TRACE: I have attached the gdb before running it.
> > > > #0  0x77f68ee0 in leveldb::SkipList<char const*, 
> > > > leveldb::MemTable::KeyComparator>::FindGreaterOrEqual(char const* 
> > > > const&, leveldb::SkipList<char const*,
> > > > leveldb::MemTable::KeyComparator>::Node**) const () from
> > > > /usr/local/lib/libleveldb.so.1
> > > > #1  0x77f69054 in leveldb::SkipList<char const*, 
> > > > leveldb::MemTable::KeyComparator>::Insert(char const* const&) () 
> > > > from
> > > > /usr/local/lib/libleveldb.so.1
> > > > #2  0x77f68618 in leveldb::MemTable::Add(unsigned long long,
> > > leveldb::ValueType, leveldb::Slice const&, leveldb::Slice const&) ()
> > > >    from /usr/local/lib/libleveldb.so.1
> > > > #3  0x77f7e434 in leveldb::(anonymous
> > > namespace)::MemTableInserter::Put(leveldb::Slice const&, 
> > > leveldb::Slice
> > > const&) ()
> > > >    from /usr/local/lib/libleveldb.so.1
> > > > #4  0x77f7e93c in
> > > > leveldb::WriteBatch::Iterate(leveldb::WriteBatch::Handler*) const
> > > > () from /usr/local/lib/libleveldb.so.1
> > > > #5  0x77f7eb8c in
> > > > leveldb::WriteBatchInternal::InsertInto(leveldb::WriteBatch
> > > > const*,
> > > > leveldb::MemTable*) () from /usr/local/lib/libleveldb.so.1
> > > > #6  0x77f59360 in leveldb::DBImpl::Write(leveldb::WriteOptions
> > > > const&,
> > > > leveldb::WriteBatch*) () from /usr/local/lib/libleveldb.so.1
> > > > #7  0x00a5dda0 in LevelDBStore::submit_transaction_sync
> > > > (this=0x1f77d10, t=<value optimized out>) at
> > > > os/LevelDBStore.cc:146
> > > > #8  0x00b0d344 in DBObjectMap::sync (this=0x1f7af28, oid=0x0,
> > > > spos=0x72cfe3b8) at os/DBObjectMap.cc:1126
> > > > #9  0x009b10b8 in FileStore::_set_replay_guard (this=0x1f72450, 
> > > > fd=17, spos=..., hoid=0x0, in_progress=false) at
> > > > os/FileStore.cc:2070
> > > > #10 0x009b1c0c in FileStore::_set_replay_guard (this=0x1f72450,
> > > cid=DWARF-2 expression error: DW_OP_reg operations must be used 
> > > either alone or in conjuction with DW_OP_piece.
> > > > ) at os/FileStore.cc:2047
> > > > #11 0x009b2138 in FileStore::_create_collection (this=0x1f72450,
> > > > c=DWARF-
> > > 2 expression error: DW_OP_reg operations must be used either alone 
> > > or in conjuction with DW_OP_piece.
> > > > ) at os/FileStore.cc:4753
> > > > #12 0x009e42a8 in FileStore::_do_transaction (this=0x1f72450, 
> > > > t=..., op_seq=<value optimized out>, trans_num=0,
> > > > handle=0x72cfec3c) at
> > > > os/FileStore.cc:2413
> > > > #13 0x009eb47c in FileStore::_do_transactions (this=0x1f72450, 
> > > > tls=..., op_seq=2, handle=0x72cfec3c) at os/FileStore.cc:1952
> > > > #14 0x009eb858 in FileStore::_do_op (this=0x1f72450, 
> > > > osr=0x1f801b8,
> > > > handle=...) at os/FileStore.cc:1761
> > > > #15 0x00c8f0bc in ThreadPool::worker (this=0x1f72cf0,
> > > > wt=0x1f7ea90) at
> > > > common/WorkQueue.cc:128
> > > > #16 0x00c91b94 in ThreadPool::WorkThread::entry() ()
> > > > #17 0x77f1c0a8 in start_thread () from /lib/libpthread.so.0
> > > > #18 0x777c1738 in ?? () from /lib/libc.so.6
> > > >
> > > > Do  I need to set any variable to set the cache size etcetc in 
ceph.conf.
> > > > I only have osd_leveldb_cache_size=5242880 for now.
> > > >
> > > >
> > > > Thanks
> > > > Prashanth
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Sage Weil [mailto:sage <at> newdream.net]
> > > > Sent: Thursday, October 23, 2014 5:54 PM
> > > > To: Prashanth Nednoor
> > > > Cc: ceph-devel <at> vger.kernel.org
> > > > Subject: Re: Having issues trying to get the OSD up on a MIPS64!!!
> > > >
> > > > Hi Prashanth,
> > > >
> > > > On Thu, 23 Oct 2014, Prashanth Nednoor wrote:
> > > > > Hello Everyone,
> > > > >
> > > > > We are using ceph-0.86, good news is we were able to compile and 
> > > > > load all the libraries and binaries needed to configure a 
> > > > > CEPH-OSD on MIPS
> > > > > 64 platform. The CEPH monitor is also able to detect the OSD, 
> > > > > but not up yet, as the osd activate failed.
> > > > > Since we don?t have the required CEPH deploy utility for MIPS64, 
> > > > > we are following the manual procedure to create and activate an 
OSD.
> > > > > We have disabled authentication between the clients and the 
> > > > > OSD?s for now.
> > > > >
> > > > > Has any body tried CEPH on a MIPS64?
> > > > > /dev/sda is a 2TB local hard drive.
> > > > >
> > > > > This is how my partition looks after ceph-disk-prepare 
> > > > > /home/prashan/ceph-0.86/src# parted GNU Parted 2.3 Using 
> > > > > /dev/sda Welcome to GNU Parted! Type 'help' to view a list of 
commands.
> > > > > (parted) p
> > > > > Model: ATA TOSHIBA MQ01ABB2 (scsi) Disk /dev/sda: 2000GB Sector 
> > > > > size (logical/physical): 512B/4096B Partition Table: gpt
> > > > >
> > > > > Number  Start   End     Size    File system  Name          Flags
> > > > >  2      1049kB  5369MB  5368MB               ceph journal
> > > > >  1      5370MB  2000GB  1995GB  xfs          ceph data
> > > > >
> > > > >
> > > > >
> > > > > The following are the steps to create an OSD
> > > > > 1)	ceph-disk zap /dev/sda
> > > > > 2)	ceph-disk-prepare --cluster  f615496c-b40a-4905-bbcd-
> > > > > 2d3e181ff21a --fs-type xfs /dev/sda
> > > > > 3)	mount /dev/sda1 /var/lib/ceph/osd/ceph-0/
> > > > > 4)	ceph-osd -i 0 ?mkfs is giving an error ,
> > > > > filestore(/var/lib/ceph/osd/ceph-0) could not find
> > > > > 23c2fcde/osd_superblock/0//-1 in index: (2) No such file.
> > > > > After this it segfaults. We have analyzed this further with the 
> > > > > help of strace and root caused this as objectmap file reading 
issue.
> > > > > open("/var/lib/ceph/osd/ceph-0/current/omap/000005.log",
> > > > > O_RDONLY)
> > > =
> > > > > 11, the first time it reads 32k, the read succeeds with 63 bytes 
> > > > > and it tries to read again with 27k and the read returns 0 bytes 
> > > > > and the CEPH osd segfaults.
> > > >
> > > > Can you generate a full log with --debug-osd 20 --debug-filestore
> > > > 20 --
> > > debug-jouranl 20 passed to ceph-osd --mkfs and post that somewhere?  
> > > It should tell us where things are going wrong.  In particular, we 
> > > want to see if that file/object is being written properly.  It will 
> > > also have a backtrace showing exactly where it crashed.
> > > >
> > > > > Please note that ceph-disk prepare creates a journal in a path 
> > > > > which is not valid(dev/disk/by-partuuid/cbd4a5d1-012f-4863-b492-
> > > 080ad2a505cb).
> > > > > So after step3 above I remove this journal below and manually 
> > > > > create a journal file before doing step4 above.
> > > > >
> > > > >
> > > > > ls -l /var/lib/ceph/osd/ceph-0/
> > > > > total 16
> > > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 ceph_fsid
> > > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 fsid lrwxrwxrwx 1 root 
> > > > > root
> > > > > 58 Oct 22 21:40 journal -> /dev/disk/by- 
> > > > > partuuid/cbd4a5d1-012f-4863-b492-080ad2a505cb
> > > >
> > > > Is there anything in /dev/disk/by-partuuid/ or is it missing 
entirely?
> > > > Maybe you have an old udev.  What distro is this?
> > > >
> > > > sage
> > > >
> > > > > -rw-r--r-- 1 root root 37 Oct 22 21:40 journal_uuid
> > > > > -rw-r--r-- 1 root root 21 Oct 22 21:40 magic
> > > > >
> > > > > Any pointers to move ahead will be greatly appreciated??
> > > > >
> > > > > thanks
> > > > > Prashanth
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > To unsubscribe from this list: send the line "unsubscribe ceph-
devel"
> > > > > in the body of a message to majordomo <at> vger.kernel.org More
> > > majordomo
> > > > > info at  http://vger.kernel.org/majordomo-info.html
> > > > >
> > > > >
> > > >
> > 
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo <at> vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Hi All,

I am experience the following error when compile ceph on MIPS64
/bin/sh ../libtool  --tag=CXX   --mode=link /usr/local/buildroot-2015.02-
rc1/output/host/usr/bin/mips64-linux-g++ -I/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/include/nss 
-I/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/include/nspr    -Wall -Wtype-limits -Wignored-qualifiers 
-Winit-self -Wpointer-arith -Werror=format-security -fno-strict-aliasing 
-fsigned-char -rdynamic -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-
invalid-offsetof    -Wstrict-null-sentinel -mxgot -mlong-calls 
-I/usr/local/buildroot-2015.02-rc1/output/host/usr/include 
-I/usr/local/mips64/include  -latomic_ops -L/usr/local/mips64/lib 
-L/usr/local/buildroot-2015.02-rc1/output/host/usr/lib 
-L/usr/local/buildroot-2015.02-rc1/output/host/usr/lib64 -o 
ceph_objectstore_tool tools/ceph_objectstore_tool.o libosd.la libosdc.la 
libos.la -laio   -lleveldb -lsnappy libperfglue.la  libos.la -laio   
-lleveldb -lsnappy libglobal.la libcommon.la -lpthread -lm -lnss3 
-lnssutil3 -lsmime3 -lssl3 -L/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib -lplds4 
-lplc4 -lnspr4   -luuid -lm     -lboost_program_options  librados.la  
-lboost_system 
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/../lib64/libsnappy.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/libsnappy.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/libatomic_ops.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/../lib64/libatomic_ops.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/lib/gcc/mips64-buildroot-linux-
gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.la' 
was moved.
libtool: warning: library '/usr/local/mips64/lib/libuuid.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/../lib64/libsnappy.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/libsnappy.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/libatomic_ops.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/../lib64/libatomic_ops.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/lib/gcc/mips64-buildroot-linux-
gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.la' 
was moved.
libtool: warning: library '/usr/local/mips64/lib/libuuid.la' was moved.
libtool: link: /usr/local/buildroot-2015.02-rc1/output/host/usr/bin/mips64-
linux-g++ -I/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-
buildroot-linux-gnu/sysroot/usr/include/nss -I/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/include/nspr 
-Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith 
-Werror=format-security -fno-strict-aliasing -fsigned-char -rdynamic 
-ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-invalid-offsetof -Wstrict-
null-sentinel -mxgot -mlong-calls -I/usr/local/buildroot-2015.02-
rc1/output/host/usr/include -I/usr/local/mips64/include -o 
ceph_objectstore_tool tools/ceph_objectstore_tool.o  
-L/usr/local/mips64/lib -L/usr/local/buildroot-2015.02-
rc1/output/host/usr/lib -L/usr/local/buildroot-2015.02-
rc1/output/host/usr/lib64 ./.libs/libosd.a /usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/../lib64/libsnappy.so ./.libs/libosdc.a ./.libs/libperfg
lue.a ./.libs/libos.a -laio -lleveldb /usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/libsnappy.so ./.libs/libglobal.a ./.libs/libcommon.a 
-L/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib 
-lboost_program_options ./.libs/librados.a /usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/libatomic_ops.so -lboost_thread /usr/local/buildroot-
2015.02-rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/../lib64/libatomic_ops.so /usr/local/buildroot-2015.02-
rc1/output/host/usr/lib/gcc/mips64-buildroot-linux-
gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.so 
-lm -lpthread -lnss3 -lnssutil3 -lsmime3 -lssl3 -lplds4 -lplc4 
-lnspr4 /usr/local/mips64/lib/libuuid.a -lboost_system -Wl,-rpath 
-Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib -Wl,-rpath 
-Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/lib/gcc/mips64-
buildroot-linux-gnu/4.9.2/../../../../mips64-buildroot-linux-
gnu/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib/../lib64 
-Wl,-rpath -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-
buildroot-linux-gnu/sysroot/usr/lib -Wl,-rpath -Wl,/usr/local/buildroot-
2015.02-rc1/output/host/usr/lib/gcc/mips64-buildroot-linux-
gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64
/usr/local/buildroot-2015.02-rc1/output/host/usr/lib/gcc/mips64-buildroot-
linux-gnu/4.9.2/../../../../mips64-buildroot-linux-
gnu/bin/ld: ./.libs/libosd.a(libosd_la-ClassHandler.o): undefined reference 
to symbol 'dlopen@@GLIBC_2.2'
/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/lib64/libdl.so.2: error adding symbols: DSO missing from 
command line
collect2: error: ld returned 1 exit status


I use the following configure:
./configure --prefix=/usr/local/ceph_mips64 --exec-
prefix=/usr/local/ceph_mips64 --build=mips64 --host=x86_64 --target=mips64 
--without-tcmalloc --disable-silent-rules --with-
sysroot=/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-
linux-gnu/sysroot
make

When I run the link command manually, with additional "-ldl". I hit 
different errors:

/bin/sh ../libtool  --tag=CXX   --mode=link /usr/local/buildroot-2015.02-
rc1/output/host/usr/bin/mips64-linux-g++ -I/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/include/nss 
-I/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/include/nspr    -Wall -Wtype-limits -Wignored-qualifiers 
-Winit-self -Wpointer-arith -Werror=format-security -fno-strict-aliasing 
-fsigned-char -rdynamic -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-
invalid-offsetof    -Wstrict-null-sentinel -mxgot -mlong-calls 
-I/usr/local/buildroot-2015.02-rc1/output/host/usr/include 
-I/usr/local/mips64/include  -latomic_ops -L/usr/local/mips64/lib 
-L/usr/local/buildroot-2015.02-rc1/output/host/usr/lib 
-L/usr/local/buildroot-2015.02-rc1/output/host/usr/lib64 -o 
ceph_objectstore_tool tools/ceph_objectstore_tool.o libosd.la libosdc.la 
libos.la -laio   -lleveldb -lsnappy libperfglue.la  libos.la -laio   
-lleveldb -lsnappy libglobal.la libcommon.la -lpthread -lm -lnss3 
-lnssutil3 -lsmime3 -lssl3 -L/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib -lplds4 
-lplc4 -lnspr4   -luuid -lm     -lboost_program_options  librados.la  
-lboost_system -ldl 
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/../lib64/libsnappy.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/libsnappy.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/libatomic_ops.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/../lib64/libatomic_ops.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/lib/gcc/mips64-buildroot-linux-
gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.la' 
was moved.
libtool: warning: library '/usr/local/mips64/lib/libuuid.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/../lib64/libsnappy.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/libsnappy.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/libatomic_ops.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/../lib64/libatomic_ops.la' was moved.
libtool: warning: library '/usr/local/buildroot-2015.02-
rc1/output/host/usr/lib/gcc/mips64-buildroot-linux-
gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.la' 
was moved.
libtool: warning: library '/usr/local/mips64/lib/libuuid.la' was moved.
libtool: link: /usr/local/buildroot-2015.02-rc1/output/host/usr/bin/mips64-
linux-g++ -I/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-
buildroot-linux-gnu/sysroot/usr/include/nss -I/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/include/nspr 
-Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith 
-Werror=format-security -fno-strict-aliasing -fsigned-char -rdynamic 
-ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-invalid-offsetof -Wstrict-
null-sentinel -mxgot -mlong-calls -I/usr/local/buildroot-2015.02-
rc1/output/host/usr/include -I/usr/local/mips64/include -o 
ceph_objectstore_tool tools/ceph_objectstore_tool.o  
-L/usr/local/mips64/lib -L/usr/local/buildroot-2015.02-
rc1/output/host/usr/lib -L/usr/local/buildroot-2015.02-
rc1/output/host/usr/lib64 ./.libs/libosd.a /usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/../lib64/libsnappy.so ./.libs/libosdc.a ./.libs/libperfg
lue.a ./.libs/libos.a -laio -lleveldb /usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/libsnappy.so ./.libs/libglobal.a ./.libs/libcommon.a 
-L/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib 
-lboost_program_options ./.libs/librados.a /usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/libatomic_ops.so -lboost_thread /usr/local/buildroot-
2015.02-rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/../lib64/libatomic_ops.so /usr/local/buildroot-2015.02-
rc1/output/host/usr/lib/gcc/mips64-buildroot-linux-
gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.so 
-lm -lpthread -lnss3 -lnssutil3 -lsmime3 -lssl3 -lplds4 -lplc4 
-lnspr4 /usr/local/mips64/lib/libuuid.a -lboost_system -ldl -Wl,-rpath 
-Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux-
gnu/sysroot/usr/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib -Wl,-rpath 
-Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/lib/gcc/mips64-
buildroot-linux-gnu/4.9.2/../../../../mips64-buildroot-linux-
gnu/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-
rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib/../lib64 
-Wl,-rpath -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-
buildroot-linux-gnu/sysroot/usr/lib -Wl,-rpath -Wl,/usr/local/buildroot-
2015.02-rc1/output/host/usr/lib/gcc/mips64-buildroot-linux-
gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64
./.libs/libosd.a(libosd_la-ReplicatedPG.o): In function `std::ostream& 
operator<< <boost::optional<unsigned long>, unsigned long>(std::ostream&, 
std::pair<boost::optional<unsigned long>, unsigned long> const&)':
ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E
[_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x34): undefined 
reference to `std::basic_ostream<char, std::char_traits<char> >& 
boost::operator<< <char, std::char_traits<char>, unsigned long>
(std::basic_ostream<char, std::char_traits<char> >&, 
boost::optional<unsigned long> const&)'
ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E
[_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x3c): undefined 
reference to `std::basic_ostream<char, std::char_traits<char> >& 
boost::operator<< <char, std::char_traits<char>, unsigned long>
(std::basic_ostream<char, std::char_traits<char> >&, 
boost::optional<unsigned long> const&)'
ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E
[_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x44): undefined 
reference to `std::basic_ostream<char, std::char_traits<char> >& 
boost::operator<< <char, std::char_traits<char>, unsigned long>
(std::basic_ostream<char, std::char_traits<char> >&, 
boost::optional<unsigned long> const&)'
./.libs/libosd.a(libosd_la-ECBackend.o): In function `operator<<
(std::ostream&, ECBackend::read_result_t const&)':
ECBackend.cc:(.text+0x10f0): undefined reference to 
`std::basic_ostream<char, std::char_traits<char> >& boost::operator<< 
<char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, 
std::less<std::string>, std::allocator<std::pair<std::string const, 
ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> 
>&, boost::optional<std::map<std::string, ceph::buffer::list, 
std::less<std::string>, std::allocator<std::pair<std::string const, 
ceph::buffer::list> > > > const&)'
ECBackend.cc:(.text+0x10f8): undefined reference to 
`std::basic_ostream<char, std::char_traits<char> >& boost::operator<< 
<char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, 
std::less<std::string>, std::allocator<std::pair<std::string const, 
ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> 
>&, boost::optional<std::map<std::string, ceph::buffer::list, 
std::less<std::string>, std::allocator<std::pair<std::string const, 
ceph::buffer::list> > > > const&)'
ECBackend.cc:(.text+0x1100): undefined reference to 
`std::basic_ostream<char, std::char_traits<char> >& boost::operator<< 
<char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, 
std::less<std::string>, std::allocator<std::pair<std::string const, 
ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> 
>&, boost::optional<std::map<std::string, ceph::buffer::list, 
std::less<std::string>, std::allocator<std::pair<std::string const, 
ceph::buffer::list> > > > const&)'
./.libs/libosd.a(libos_la-FileStore.o): In function 
`FileStoreBackend::create(long, FileStore*)':
FileStore.cc:(.text+0x8f1c): undefined reference to 
`BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)'
FileStore.cc:(.text+0x8f24): undefined reference to 
`BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)'
FileStore.cc:(.text+0x8f2c): undefined reference to 
`BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)'
collect2: error: ld returned 1 exit status

Could someone please help?

Thank you,
Bao

Re: Having issues trying to get the OSD up on a MIPS64!!!

[Sage Weil]

> When I run the link command manually, with additional "-ldl". I hit 
> different errors:
> 
> /bin/sh ../libtool  --tag=CXX   --mode=link /usr/local/buildroot-2015.02-
> rc1/output/host/usr/bin/mips64-linux-g++ -I/usr/local/buildroot-2015.02-
> rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/include/nss 
> -I/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux-
> gnu/sysroot/usr/include/nspr    -Wall -Wtype-limits -Wignored-qualifiers 
> -Winit-self -Wpointer-arith -Werror=format-security -fno-strict-aliasing 
> -fsigned-char -rdynamic -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-
> invalid-offsetof    -Wstrict-null-sentinel -mxgot -mlong-calls 
> -I/usr/local/buildroot-2015.02-rc1/output/host/usr/include 
> -I/usr/local/mips64/include  -latomic_ops -L/usr/local/mips64/lib 
> -L/usr/local/buildroot-2015.02-rc1/output/host/usr/lib 
> -L/usr/local/buildroot-2015.02-rc1/output/host/usr/lib64 -o 
> ceph_objectstore_tool tools/ceph_objectstore_tool.o libosd.la libosdc.la 
> libos.la -laio   -lleveldb -lsnappy libperfglue.la  libos.la -laio   
> -lleveldb -lsnappy libglobal.la libcommon.la -lpthread -lm -lnss3 
> -lnssutil3 -lsmime3 -lssl3 -L/usr/local/buildroot-2015.02-
> rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib -lplds4 
> -lplc4 -lnspr4   -luuid -lm     -lboost_program_options  librados.la  
> -lboost_system -ldl 
> libtool: warning: library '/usr/local/buildroot-2015.02-
> rc1/output/host/usr/mips64-buildroot-linux-
> gnu/sysroot/usr/lib/../lib64/libsnappy.la' was moved.
> libtool: warning: library '/usr/local/buildroot-2015.02-
> rc1/output/host/usr/mips64-buildroot-linux-
> gnu/sysroot/usr/lib/libsnappy.la' was moved.
> libtool: warning: library '/usr/local/buildroot-2015.02-
> rc1/output/host/usr/mips64-buildroot-linux-
> gnu/sysroot/usr/lib/libatomic_ops.la' was moved.
> libtool: warning: library '/usr/local/buildroot-2015.02-
> rc1/output/host/usr/mips64-buildroot-linux-
> gnu/sysroot/usr/lib/../lib64/libatomic_ops.la' was moved.
> libtool: warning: library '/usr/local/buildroot-2015.02-
> rc1/output/host/usr/lib/gcc/mips64-buildroot-linux-
> gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.la' 
> was moved.
> libtool: warning: library '/usr/local/mips64/lib/libuuid.la' was moved.
> libtool: warning: library '/usr/local/buildroot-2015.02-
> rc1/output/host/usr/mips64-buildroot-linux-
> gnu/sysroot/usr/lib/../lib64/libsnappy.la' was moved.
> libtool: warning: library '/usr/local/buildroot-2015.02-
> rc1/output/host/usr/mips64-buildroot-linux-
> gnu/sysroot/usr/lib/libsnappy.la' was moved.
> libtool: warning: library '/usr/local/buildroot-2015.02-
> rc1/output/host/usr/mips64-buildroot-linux-
> gnu/sysroot/usr/lib/libatomic_ops.la' was moved.
> libtool: warning: library '/usr/local/buildroot-2015.02-
> rc1/output/host/usr/mips64-buildroot-linux-
> gnu/sysroot/usr/lib/../lib64/libatomic_ops.la' was moved.
> libtool: warning: library '/usr/local/buildroot-2015.02-
> rc1/output/host/usr/lib/gcc/mips64-buildroot-linux-
> gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.la' 
> was moved.
> libtool: warning: library '/usr/local/mips64/lib/libuuid.la' was moved.
> libtool: link: /usr/local/buildroot-2015.02-rc1/output/host/usr/bin/mips64-
> linux-g++ -I/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-
> buildroot-linux-gnu/sysroot/usr/include/nss -I/usr/local/buildroot-2015.02-
> rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/include/nspr 
> -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith 
> -Werror=format-security -fno-strict-aliasing -fsigned-char -rdynamic 
> -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-invalid-offsetof -Wstrict-
> null-sentinel -mxgot -mlong-calls -I/usr/local/buildroot-2015.02-
> rc1/output/host/usr/include -I/usr/local/mips64/include -o 
> ceph_objectstore_tool tools/ceph_objectstore_tool.o  
> -L/usr/local/mips64/lib -L/usr/local/buildroot-2015.02-
> rc1/output/host/usr/lib -L/usr/local/buildroot-2015.02-
> rc1/output/host/usr/lib64 ./.libs/libosd.a /usr/local/buildroot-2015.02-
> rc1/output/host/usr/mips64-buildroot-linux-
> gnu/sysroot/usr/lib/../lib64/libsnappy.so ./.libs/libosdc.a ./.libs/libperfg
> lue.a ./.libs/libos.a -laio -lleveldb /usr/local/buildroot-2015.02-
> rc1/output/host/usr/mips64-buildroot-linux-
> gnu/sysroot/usr/lib/libsnappy.so ./.libs/libglobal.a ./.libs/libcommon.a 
> -L/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux-
> gnu/sysroot/usr/lib 
> -lboost_program_options ./.libs/librados.a /usr/local/buildroot-2015.02-
> rc1/output/host/usr/mips64-buildroot-linux-
> gnu/sysroot/usr/lib/libatomic_ops.so -lboost_thread /usr/local/buildroot-
> 2015.02-rc1/output/host/usr/mips64-buildroot-linux-
> gnu/sysroot/usr/lib/../lib64/libatomic_ops.so /usr/local/buildroot-2015.02-
> rc1/output/host/usr/lib/gcc/mips64-buildroot-linux-
> gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.so 
> -lm -lpthread -lnss3 -lnssutil3 -lsmime3 -lssl3 -lplds4 -lplc4 
> -lnspr4 /usr/local/mips64/lib/libuuid.a -lboost_system -ldl -Wl,-rpath 
> -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux-
> gnu/sysroot/usr/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-
> rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib -Wl,-rpath 
> -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/lib/gcc/mips64-
> buildroot-linux-gnu/4.9.2/../../../../mips64-buildroot-linux-
> gnu/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-
> rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib/../lib64 
> -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-
> buildroot-linux-gnu/sysroot/usr/lib -Wl,-rpath -Wl,/usr/local/buildroot-
> 2015.02-rc1/output/host/usr/lib/gcc/mips64-buildroot-linux-
> gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64
> ./.libs/libosd.a(libosd_la-ReplicatedPG.o): In function `std::ostream& 
> operator<< <boost::optional<unsigned long>, unsigned long>(std::ostream&, 
> std::pair<boost::optional<unsigned long>, unsigned long> const&)':
> ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E
> [_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x34): undefined 
> reference to `std::basic_ostream<char, std::char_traits<char> >& 
> boost::operator<< <char, std::char_traits<char>, unsigned long>
> (std::basic_ostream<char, std::char_traits<char> >&, 
> boost::optional<unsigned long> const&)'
> ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E
> [_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x3c): undefined 
> reference to `std::basic_ostream<char, std::char_traits<char> >& 
> boost::operator<< <char, std::char_traits<char>, unsigned long>
> (std::basic_ostream<char, std::char_traits<char> >&, 
> boost::optional<unsigned long> const&)'
> ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E
> [_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x44): undefined 
> reference to `std::basic_ostream<char, std::char_traits<char> >& 
> boost::operator<< <char, std::char_traits<char>, unsigned long>
> (std::basic_ostream<char, std::char_traits<char> >&, 
> boost::optional<unsigned long> const&)'
> ./.libs/libosd.a(libosd_la-ECBackend.o): In function `operator<<
> (std::ostream&, ECBackend::read_result_t const&)':
> ECBackend.cc:(.text+0x10f0): undefined reference to 
> `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< 
> <char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, 
> std::less<std::string>, std::allocator<std::pair<std::string const, 
> ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> 
> >&, boost::optional<std::map<std::string, ceph::buffer::list, 
> std::less<std::string>, std::allocator<std::pair<std::string const, 
> ceph::buffer::list> > > > const&)'
> ECBackend.cc:(.text+0x10f8): undefined reference to 
> `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< 
> <char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, 
> std::less<std::string>, std::allocator<std::pair<std::string const, 
> ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> 
> >&, boost::optional<std::map<std::string, ceph::buffer::list, 
> std::less<std::string>, std::allocator<std::pair<std::string const, 
> ceph::buffer::list> > > > const&)'
> ECBackend.cc:(.text+0x1100): undefined reference to 
> `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< 
> <char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, 
> std::less<std::string>, std::allocator<std::pair<std::string const, 
> ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> 
> >&, boost::optional<std::map<std::string, ceph::buffer::list, 
> std::less<std::string>, std::allocator<std::pair<std::string const, 
> ceph::buffer::list> > > > const&)'
> ./.libs/libosd.a(libos_la-FileStore.o): In function 
> `FileStoreBackend::create(long, FileStore*)':
> FileStore.cc:(.text+0x8f1c): undefined reference to 
> `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)'
> FileStore.cc:(.text+0x8f24): undefined reference to 
> `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)'
> FileStore.cc:(.text+0x8f2c): undefined reference to 
> `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)'
> collect2: error: ld returned 1 exit status

It looks like maybe the BtrfsFileStoreBackend is not being linked in?  
Perhaps there is some Makefile magic going on?  Which version/commit are 
you on, and have you made any Makefile modifications?

Unfortunately we don't have a MPIS64 box in the community lab to play 
with.

sage

Re: Having issues trying to get the OSD up on a MIPS64!!!

[Bao Vu]

On Feb 17, 2015, at 6:06 AM, Sage Weil <sage@newdream.net> wrote:

>> When I run the link command manually, with additional "-ldl". I hit 
>> different errors:
>> 
>> /bin/sh ../libtool  --tag=CXX   --mode=link /usr/local/buildroot-2015.02-
>> rc1/output/host/usr/bin/mips64-linux-g++ -I/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/include/nss 
>> -I/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux-
>> gnu/sysroot/usr/include/nspr    -Wall -Wtype-limits -Wignored-qualifiers 
>> -Winit-self -Wpointer-arith -Werror=format-security -fno-strict-aliasing 
>> -fsigned-char -rdynamic -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-
>> invalid-offsetof    -Wstrict-null-sentinel -mxgot -mlong-calls 
>> -I/usr/local/buildroot-2015.02-rc1/output/host/usr/include 
>> -I/usr/local/mips64/include  -latomic_ops -L/usr/local/mips64/lib 
>> -L/usr/local/buildroot-2015.02-rc1/output/host/usr/lib 
>> -L/usr/local/buildroot-2015.02-rc1/output/host/usr/lib64 -o 
>> ceph_objectstore_tool tools/ceph_objectstore_tool.o libosd.la libosdc.la 
>> libos.la -laio   -lleveldb -lsnappy libperfglue.la  libos.la -laio   
>> -lleveldb -lsnappy libglobal.la libcommon.la -lpthread -lm -lnss3 
>> -lnssutil3 -lsmime3 -lssl3 -L/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib -lplds4 
>> -lplc4 -lnspr4   -luuid -lm     -lboost_program_options  librados.la  
>> -lboost_system -ldl 
>> libtool: warning: library '/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/mips64-buildroot-linux-
>> gnu/sysroot/usr/lib/../lib64/libsnappy.la' was moved.
>> libtool: warning: library '/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/mips64-buildroot-linux-
>> gnu/sysroot/usr/lib/libsnappy.la' was moved.
>> libtool: warning: library '/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/mips64-buildroot-linux-
>> gnu/sysroot/usr/lib/libatomic_ops.la' was moved.
>> libtool: warning: library '/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/mips64-buildroot-linux-
>> gnu/sysroot/usr/lib/../lib64/libatomic_ops.la' was moved.
>> libtool: warning: library '/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/lib/gcc/mips64-buildroot-linux-
>> gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.la' 
>> was moved.
>> libtool: warning: library '/usr/local/mips64/lib/libuuid.la' was moved.
>> libtool: warning: library '/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/mips64-buildroot-linux-
>> gnu/sysroot/usr/lib/../lib64/libsnappy.la' was moved.
>> libtool: warning: library '/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/mips64-buildroot-linux-
>> gnu/sysroot/usr/lib/libsnappy.la' was moved.
>> libtool: warning: library '/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/mips64-buildroot-linux-
>> gnu/sysroot/usr/lib/libatomic_ops.la' was moved.
>> libtool: warning: library '/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/mips64-buildroot-linux-
>> gnu/sysroot/usr/lib/../lib64/libatomic_ops.la' was moved.
>> libtool: warning: library '/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/lib/gcc/mips64-buildroot-linux-
>> gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.la' 
>> was moved.
>> libtool: warning: library '/usr/local/mips64/lib/libuuid.la' was moved.
>> libtool: link: /usr/local/buildroot-2015.02-rc1/output/host/usr/bin/mips64-
>> linux-g++ -I/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-
>> buildroot-linux-gnu/sysroot/usr/include/nss -I/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/include/nspr 
>> -Wall -Wtype-limits -Wignored-qualifiers -Winit-self -Wpointer-arith 
>> -Werror=format-security -fno-strict-aliasing -fsigned-char -rdynamic 
>> -ftemplate-depth-1024 -Wnon-virtual-dtor -Wno-invalid-offsetof -Wstrict-
>> null-sentinel -mxgot -mlong-calls -I/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/include -I/usr/local/mips64/include -o 
>> ceph_objectstore_tool tools/ceph_objectstore_tool.o  
>> -L/usr/local/mips64/lib -L/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/lib -L/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/lib64 ./.libs/libosd.a /usr/local/buildroot-2015.02-
>> rc1/output/host/usr/mips64-buildroot-linux-
>> gnu/sysroot/usr/lib/../lib64/libsnappy.so ./.libs/libosdc.a ./.libs/libperfg
>> lue.a ./.libs/libos.a -laio -lleveldb /usr/local/buildroot-2015.02-
>> rc1/output/host/usr/mips64-buildroot-linux-
>> gnu/sysroot/usr/lib/libsnappy.so ./.libs/libglobal.a ./.libs/libcommon.a 
>> -L/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux-
>> gnu/sysroot/usr/lib 
>> -lboost_program_options ./.libs/librados.a /usr/local/buildroot-2015.02-
>> rc1/output/host/usr/mips64-buildroot-linux-
>> gnu/sysroot/usr/lib/libatomic_ops.so -lboost_thread /usr/local/buildroot-
>> 2015.02-rc1/output/host/usr/mips64-buildroot-linux-
>> gnu/sysroot/usr/lib/../lib64/libatomic_ops.so /usr/local/buildroot-2015.02-
>> rc1/output/host/usr/lib/gcc/mips64-buildroot-linux-
>> gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64/libstdc++.so 
>> -lm -lpthread -lnss3 -lnssutil3 -lsmime3 -lssl3 -lplds4 -lplc4 
>> -lnspr4 /usr/local/mips64/lib/libuuid.a -lboost_system -ldl -Wl,-rpath 
>> -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-buildroot-linux-
>> gnu/sysroot/usr/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib -Wl,-rpath 
>> -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/lib/gcc/mips64-
>> buildroot-linux-gnu/4.9.2/../../../../mips64-buildroot-linux-
>> gnu/lib/../lib64 -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-
>> rc1/output/host/usr/mips64-buildroot-linux-gnu/sysroot/usr/lib/../lib64 
>> -Wl,-rpath -Wl,/usr/local/buildroot-2015.02-rc1/output/host/usr/mips64-
>> buildroot-linux-gnu/sysroot/usr/lib -Wl,-rpath -Wl,/usr/local/buildroot-
>> 2015.02-rc1/output/host/usr/lib/gcc/mips64-buildroot-linux-
>> gnu/4.9.2/../../../../mips64-buildroot-linux-gnu/lib/../lib64
>> ./.libs/libosd.a(libosd_la-ReplicatedPG.o): In function `std::ostream& 
>> operator<< <boost::optional<unsigned long>, unsigned long>(std::ostream&, 
>> std::pair<boost::optional<unsigned long>, unsigned long> const&)':
>> ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E
>> [_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x34): undefined 
>> reference to `std::basic_ostream<char, std::char_traits<char> >& 
>> boost::operator<< <char, std::char_traits<char>, unsigned long>
>> (std::basic_ostream<char, std::char_traits<char> >&, 
>> boost::optional<unsigned long> const&)'
>> ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E
>> [_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x3c): undefined 
>> reference to `std::basic_ostream<char, std::char_traits<char> >& 
>> boost::operator<< <char, std::char_traits<char>, unsigned long>
>> (std::basic_ostream<char, std::char_traits<char> >&, 
>> boost::optional<unsigned long> const&)'
>> ReplicatedPG.cc:(.text._ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E
>> [_ZlsIN5boost8optionalImEEmERSoS3_RKSt4pairIT_T0_E]+0x44): undefined 
>> reference to `std::basic_ostream<char, std::char_traits<char> >& 
>> boost::operator<< <char, std::char_traits<char>, unsigned long>
>> (std::basic_ostream<char, std::char_traits<char> >&, 
>> boost::optional<unsigned long> const&)'
>> ./.libs/libosd.a(libosd_la-ECBackend.o): In function `operator<<
>> (std::ostream&, ECBackend::read_result_t const&)':
>> ECBackend.cc:(.text+0x10f0): undefined reference to 
>> `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< 
>> <char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, 
>> std::less<std::string>, std::allocator<std::pair<std::string const, 
>> ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> 
>>> &, boost::optional<std::map<std::string, ceph::buffer::list,
>> std::less<std::string>, std::allocator<std::pair<std::string const, 
>> ceph::buffer::list> > > > const&)'
>> ECBackend.cc:(.text+0x10f8): undefined reference to 
>> `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< 
>> <char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, 
>> std::less<std::string>, std::allocator<std::pair<std::string const, 
>> ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> 
>>> &, boost::optional<std::map<std::string, ceph::buffer::list,
>> std::less<std::string>, std::allocator<std::pair<std::string const, 
>> ceph::buffer::list> > > > const&)'
>> ECBackend.cc:(.text+0x1100): undefined reference to 
>> `std::basic_ostream<char, std::char_traits<char> >& boost::operator<< 
>> <char, std::char_traits<char>, std::map<std::string, ceph::buffer::list, 
>> std::less<std::string>, std::allocator<std::pair<std::string const, 
>> ceph::buffer::list> > > >(std::basic_ostream<char, std::char_traits<char> 
>>> &, boost::optional<std::map<std::string, ceph::buffer::list,
>> std::less<std::string>, std::allocator<std::pair<std::string const, 
>> ceph::buffer::list> > > > const&)'
>> ./.libs/libosd.a(libos_la-FileStore.o): In function 
>> `FileStoreBackend::create(long, FileStore*)':
>> FileStore.cc:(.text+0x8f1c): undefined reference to 
>> `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)'
>> FileStore.cc:(.text+0x8f24): undefined reference to 
>> `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)'
>> FileStore.cc:(.text+0x8f2c): undefined reference to 
>> `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)'
>> collect2: error: ld returned 1 exit status
> 
> It looks like maybe the BtrfsFileStoreBackend is not being linked in?  
> Perhaps there is some Makefile magic going on?  Which version/commit are 
> you on, and have you made any Makefile modifications?
> 
> Unfortunately we don't have a MPIS64 box in the community lab to play 
> with.
> 
> sage

I am using 0.87. I do not change makefile . It is created by the configure. I try to build ceph for MIPS64. So far, I have no luck. 

Bao

Re: Having issues trying to get the OSD up on a MIPS64!!!

[Sage Weil]

On Tue, 17 Feb 2015, Bao Vu wrote:
> >> ./.libs/libosd.a(libos_la-FileStore.o): In function 
> >> `FileStoreBackend::create(long, FileStore*)':
> >> FileStore.cc:(.text+0x8f1c): undefined reference to 
> >> `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)'
> >> FileStore.cc:(.text+0x8f24): undefined reference to 
> >> `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)'
> >> FileStore.cc:(.text+0x8f2c): undefined reference to 
> >> `BtrfsFileStoreBackend::BtrfsFileStoreBackend(FileStore*)'
> >> collect2: error: ld returned 1 exit status
> > 
> > It looks like maybe the BtrfsFileStoreBackend is not being linked in?  
> > Perhaps there is some Makefile magic going on?  Which version/commit are 
> > you on, and have you made any Makefile modifications?
> > 
> > Unfortunately we don't have a MPIS64 box in the community lab to play 
> > with.
> > 
> > sage
> 
> I am using 0.87. I do not change makefile . It is created by the configure. I try to build ceph for MIPS64. So far, I have no luck. 

It is probably best to work from the latest git master to make sure you've 
captured any recent changes.  MIPS64 support isn't something that would 
have been backported.

sage