All of lore.kernel.org
 help / color / mirror / Atom feed
* setup ceph cluster failed
@ 2010-06-15 20:09 Min Zhou
  2010-06-15 20:12 ` Yehuda Sadeh Weinraub
  0 siblings, 1 reply; 9+ messages in thread
From: Min Zhou @ 2010-06-15 20:09 UTC (permalink / raw)
  To: ceph-devel

Hi all,

I am a newbie of ceph, and didn't  find any ceph mailing list for
users, so I send a mail here ask for helping.
below is the error information

$ mkcephfs --allhosts -c ceph_conf/ceph.conf -v
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t mon  "mon addr" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "mon addr" ""
/home/ceph/ceph/bin/monmaptool --create --clobber --add
192.168.0.2:6789 --add 192.168.0.3:6789 --print /tmp/monmap.8107
/home/ceph/ceph/bin/monmaptool: monmap file /tmp/monmap.8107
/home/ceph/ceph/bin/monmaptool: generated fsid
2b00c868-cfd6-2473-c27d-302b5cddc194
epoch 1
fsid 2b00c868-cfd6-2473-c27d-302b5cddc194
last_changed 10.06.16 04:02:28.247612
created 10.06.16 04:02:28.247612
        mon0 192.168.0.2:6789/0
        mon1 192.168.0.3:6789/0
/home/ceph/ceph/bin/monmaptool: writing epoch 1 to /tmp/monmap.8107 (2 monitors)
max osd in ceph_conf/ceph.conf is 2, num osd is 3
/home/ceph/ceph/bin/osdmaptool: osdmap file '/tmp/osdmap.8107'
/home/ceph/ceph/bin/osdmaptool: writing epoch 1 to /tmp/osdmap.8107
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "crush map src" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "crush map" ""
Building admin keyring at /tmp/admin.keyring.8107
creating /tmp/admin.keyring.8107
Building monitor keyring with all service keys
creating /tmp/monkeyring.8107
importing contents of /tmp/admin.keyring.8107 into /tmp/monkeyring.8107
creating /tmp/keyring.mds.0
importing contents of /tmp/keyring.mds.0 into /tmp/monkeyring.8107
creating /tmp/keyring.mds.1
importing contents of /tmp/keyring.mds.1 into /tmp/monkeyring.8107
creating /tmp/keyring.mds.a
importing contents of /tmp/keyring.mds.a into /tmp/monkeyring.8107
creating /tmp/keyring.osd.0
importing contents of /tmp/keyring.osd.0 into /tmp/monkeyring.8107
creating /tmp/keyring.osd.1
importing contents of /tmp/keyring.osd.1 into /tmp/monkeyring.8107
creating /tmp/keyring.osd.2
importing contents of /tmp/keyring.osd.2 into /tmp/monkeyring.8107
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t mon  "user" ""
=== mon0 ===
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t mon  "mon data" ""
/home/ceph/ceph/lib/ceph/ceph_common.sh: eval: line 167: unexpected
EOF while looking for matching `"'
/home/ceph/ceph/lib/ceph/ceph_common.sh: eval: line 168: syntax error:
unexpected end of file
--- hdp2# /home/ceph/ceph/bin/mkmonfs --clobber --mon-data  -i 0
--monmap /tmp/monmap.8107 --osdmap /tmp/osdmap.8107 -k
/tmp/keyring.8107 ; rm -f /tmp/keyring.8107
usage: ./mkmonfs [--clobber] --mon-data <monfsdir> -i <monid> --monmap
<file> --osdmap <file>
`/tmp/admin.keyring.8107' -> `/admin_keyring.bin'
cp: cannot create regular file `/admin_keyring.bin': Permission denied
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "user" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "ssh
path" "/home/ceph"
=== mon1 ===
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "mon data" ""
/home/ceph/ceph/lib/ceph/ceph_common.sh: eval: line 167: unexpected
EOF while looking for matching `"'
/home/ceph/ceph/lib/ceph/ceph_common.sh: eval: line 168: syntax error:
unexpected end of file
--- ssh hdp3  "cd /home/ceph ; ulimit -c unlimited ;
/home/ceph/ceph/bin/mkmonfs --clobber --mon-data  -i 1 --monmap
/tmp/monmap.8107 --osdmap /tmp/osdmap.8107 -k /tmp/keyring.8107 ; rm
-f /tmp/keyring.8107"
usage: ./mkmonfs [--clobber] --mon-data <monfsdir> -i <monid> --monmap
<file> --osdmap <file>
scp: /admin_keyring.bin: Permission denied
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t mds  "user" ""
=== mds0 ===
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t mds  "keyring" ""
WARNING: no keyring specified for mds0
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mds  "user" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mds  "ssh
path" "/home/ceph"
=== mds1 ===
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mds  "keyring" ""
WARNING: no keyring specified for mds1
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i a -t mds  "user" ""
=== mds.a ===
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i a -t mds  "keyring" ""
WARNING: no keyring specified for mds.a
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t osd  "user" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t osd  "ssh
path" "/home/ceph"
=== osd0 ===
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t osd  "osd data" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t osd  "keyring" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t osd  "btrfs
path" "(0µá=_OSD_DIR/dev/osd0"
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t osd  "btrfs devs" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t osd  "btrfs
options" "noatime"
--- ssh hdp4  "cd /home/ceph ; ulimit -c unlimited ; test -d
(0µá=_OSD_DIR/dev/osd0 || mkdir -p (0µá=_OSD_DIR/dev/osd0"
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `cd /home/ceph ; ulimit -c unlimited ; test -d
(0µá=_OSD_DIR/dev/osd0 || mkdir -p (0µá=_OSD_DIR/dev/osd0'
failed: 'ssh hdp4 test -d (0µá=_OSD_DIR/dev/osd0 || mkdir -p
(0µá=_OSD_DIR/dev/osd0'

Can someone help me on this?


Thanks,
Min

-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: setup ceph cluster failed
  2010-06-15 20:09 setup ceph cluster failed Min Zhou
@ 2010-06-15 20:12 ` Yehuda Sadeh Weinraub
  2010-06-15 20:17   ` Min Zhou
  0 siblings, 1 reply; 9+ messages in thread
From: Yehuda Sadeh Weinraub @ 2010-06-15 20:12 UTC (permalink / raw)
  To: Min Zhou; +Cc: ceph-devel

On Tue, Jun 15, 2010 at 1:09 PM, Min Zhou <coderplay@gmail.com> wrote:
> Hi all,
>
> I am a newbie of ceph, and didn't  find any ceph mailing list for
> users, so I send a mail here ask for helping.
> below is the error information
>
> $ mkcephfs --allhosts -c ceph_conf/ceph.conf -v
> /home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t mon  "mon addr" ""
> /home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "mon addr" ""
> /home/ceph/ceph/bin/monmaptool --create --clobber --add
> 192.168.0.2:6789 --add 192.168.0.3:6789 --print /tmp/monmap.8107
> /home/ceph/ceph/bin/monmaptool: monmap file /tmp/monmap.8107
> /home/ceph/ceph/bin/monmaptool: generated fsid
> 2b00c868-cfd6-2473-c27d-302b5cddc194
> epoch 1
> fsid 2b00c868-cfd6-2473-c27d-302b5cddc194
> last_changed 10.06.16 04:02:28.247612
> created 10.06.16 04:02:28.247612
>        mon0 192.168.0.2:6789/0
>        mon1 192.168.0.3:6789/0
> /home/ceph/ceph/bin/monmaptool: writing epoch 1 to /tmp/monmap.8107 (2 monitors)
> max osd in ceph_conf/ceph.conf is 2, num osd is 3
> /home/ceph/ceph/bin/osdmaptool: osdmap file '/tmp/osdmap.8107'
> /home/ceph/ceph/bin/osdmaptool: writing epoch 1 to /tmp/osdmap.8107
> /home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "crush map src" ""
> /home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "crush map" ""
> Building admin keyring at /tmp/admin.keyring.8107
> creating /tmp/admin.keyring.8107
> Building monitor keyring with all service keys
> creating /tmp/monkeyring.8107
> importing contents of /tmp/admin.keyring.8107 into /tmp/monkeyring.8107
> creating /tmp/keyring.mds.0
> importing contents of /tmp/keyring.mds.0 into /tmp/monkeyring.8107
> creating /tmp/keyring.mds.1
> importing contents of /tmp/keyring.mds.1 into /tmp/monkeyring.8107
> creating /tmp/keyring.mds.a
> importing contents of /tmp/keyring.mds.a into /tmp/monkeyring.8107
> creating /tmp/keyring.osd.0
> importing contents of /tmp/keyring.osd.0 into /tmp/monkeyring.8107
> creating /tmp/keyring.osd.1
> importing contents of /tmp/keyring.osd.1 into /tmp/monkeyring.8107
> creating /tmp/keyring.osd.2
> importing contents of /tmp/keyring.osd.2 into /tmp/monkeyring.8107
> /home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t mon  "user" ""
> === mon0 ===
> /home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t mon  "mon data" ""
> /home/ceph/ceph/lib/ceph/ceph_common.sh: eval: line 167: unexpected

Can you send your ceph.conf?

Thanks,
Yehuda
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: setup ceph cluster failed
  2010-06-15 20:12 ` Yehuda Sadeh Weinraub
@ 2010-06-15 20:17   ` Min Zhou
  2010-06-15 20:31     ` Yehuda Sadeh Weinraub
  0 siblings, 1 reply; 9+ messages in thread
From: Min Zhou @ 2010-06-15 20:17 UTC (permalink / raw)
  To: Yehuda Sadeh Weinraub; +Cc: ceph-devel

Here is my ceph.conf at /home/ceph/ceph.conf

$ cat ceph_conf/ceph.conf
[global]
       log dir = $CEPH_CONF/out
       logger dir = $CEPH_CONF/log
       chdir = ""
       pid file = $CEPH_CONF/out/$type$id.pid
[mds]
       pid file = $CEPH_CONF/out/$name.pid
       lockdep = 1
       debug ms = 1
       debug mds = 20
       mds log max segments = 2
[mds0]
       host=hdp2
[mds1]
       host=hdp3
[mon]
       lockdep = 1
       debug mon = 20
       debug paxos = 20
       debug ms = 1
[mon0]
       host = hdp2
       mon data = $CEPH_OSD_DIR/dev/mon0"
       mon addr = 192.168.0.2:6789
[mon1]
       host = hdp3
       mon data = $CEPH_OSD_DIR/dev/mon1"
       mon addr = 192.168.0.3:6789
[osd]
       lockdep = 1
       debug ms = 1
       debug osd = 25
       debug journal = 20
       debug filestore = 10
[osd0]
       host = hdp4
       user $USER
       osd data = $CEPH_OSD_DIR/dev/osd0
       osd journal = $CEPH_OSD_DIR/dev/osd0/journal
       osd journal size = 100
[osd1]
       host = hdp5
       user $USER
       osd data = $CEPH_OSD_DIR/dev/osd1
       osd journal = $CEPH_OSD_DIR/dev/osd1/journal
       osd journal size = 100
[osd2]
       host = hdp6
       user $USER
       osd data = $CEPH_OSD_DIR/dev/osd2
       osd journal = $CEPH_OSD_DIR/dev/osd2/journal
       osd journal size = 100
[mds.a]

Some envionment variables were defined at file ~/.bashrc on each node.
# ceph
export CEPH_HOME=$HOME/ceph
export CEPH_CONF_DIR=$HOME/ceph_conf
export CEPH_OSD_DIR=$HOME/ceph_osd
export PATH=${CEPH_HOME}/bin:${CEPH_HOME}/sbin:${PATH}
# ceph end

I built ceph from the source into a sub-directory of the user home.

Thanks,
Min

On Wed, Jun 16, 2010 at 4:12 AM, Yehuda Sadeh Weinraub
<yehudasa@gmail.com> wrote:
> On Tue, Jun 15, 2010 at 1:09 PM, Min Zhou <coderplay@gmail.com> wrote:
>> Hi all,
>>
>> I am a newbie of ceph, and didn't  find any ceph mailing list for
>> users, so I send a mail here ask for helping.
>> below is the error information
>>
>> $ mkcephfs --allhosts -c ceph_conf/ceph.conf -v
>> /home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t mon  "mon addr" ""
>> /home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "mon addr" ""
>> /home/ceph/ceph/bin/monmaptool --create --clobber --add
>> 192.168.0.2:6789 --add 192.168.0.3:6789 --print /tmp/monmap.8107
>> /home/ceph/ceph/bin/monmaptool: monmap file /tmp/monmap.8107
>> /home/ceph/ceph/bin/monmaptool: generated fsid
>> 2b00c868-cfd6-2473-c27d-302b5cddc194
>> epoch 1
>> fsid 2b00c868-cfd6-2473-c27d-302b5cddc194
>> last_changed 10.06.16 04:02:28.247612
>> created 10.06.16 04:02:28.247612
>>        mon0 192.168.0.2:6789/0
>>        mon1 192.168.0.3:6789/0
>> /home/ceph/ceph/bin/monmaptool: writing epoch 1 to /tmp/monmap.8107 (2 monitors)
>> max osd in ceph_conf/ceph.conf is 2, num osd is 3
>> /home/ceph/ceph/bin/osdmaptool: osdmap file '/tmp/osdmap.8107'
>> /home/ceph/ceph/bin/osdmaptool: writing epoch 1 to /tmp/osdmap.8107
>> /home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "crush map src" ""
>> /home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "crush map" ""
>> Building admin keyring at /tmp/admin.keyring.8107
>> creating /tmp/admin.keyring.8107
>> Building monitor keyring with all service keys
>> creating /tmp/monkeyring.8107
>> importing contents of /tmp/admin.keyring.8107 into /tmp/monkeyring.8107
>> creating /tmp/keyring.mds.0
>> importing contents of /tmp/keyring.mds.0 into /tmp/monkeyring.8107
>> creating /tmp/keyring.mds.1
>> importing contents of /tmp/keyring.mds.1 into /tmp/monkeyring.8107
>> creating /tmp/keyring.mds.a
>> importing contents of /tmp/keyring.mds.a into /tmp/monkeyring.8107
>> creating /tmp/keyring.osd.0
>> importing contents of /tmp/keyring.osd.0 into /tmp/monkeyring.8107
>> creating /tmp/keyring.osd.1
>> importing contents of /tmp/keyring.osd.1 into /tmp/monkeyring.8107
>> creating /tmp/keyring.osd.2
>> importing contents of /tmp/keyring.osd.2 into /tmp/monkeyring.8107
>> /home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t mon  "user" ""
>> === mon0 ===
>> /home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t mon  "mon data" ""
>> /home/ceph/ceph/lib/ceph/ceph_common.sh: eval: line 167: unexpected
>
> Can you send your ceph.conf?
>
> Thanks,
> Yehuda
>



-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: setup ceph cluster failed
  2010-06-15 20:17   ` Min Zhou
@ 2010-06-15 20:31     ` Yehuda Sadeh Weinraub
  2010-06-15 20:54       ` Min Zhou
  0 siblings, 1 reply; 9+ messages in thread
From: Yehuda Sadeh Weinraub @ 2010-06-15 20:31 UTC (permalink / raw)
  To: Min Zhou; +Cc: ceph-devel

On Tue, Jun 15, 2010 at 1:17 PM, Min Zhou <coderplay@gmail.com> wrote:
> Here is my ceph.conf at /home/ceph/ceph.conf
.. snip
>       pid file = $CEPH_CONF/out/$name.pid
Currently there are only a limited number of variables that you can
use for replacement: $type, $id, $num, $name and $host. You can't use
environment variables yet, although adding it might be a good idea.

>       mon data = $CEPH_OSD_DIR/dev/mon0"
...
>       mon data = $CEPH_OSD_DIR/dev/mon1"
The same goes here. Note also that you're missing quotes anyway.


Thanks,
Yehuda
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: setup ceph cluster failed
  2010-06-15 20:31     ` Yehuda Sadeh Weinraub
@ 2010-06-15 20:54       ` Min Zhou
  2010-06-15 21:19         ` Yehuda Sadeh Weinraub
  0 siblings, 1 reply; 9+ messages in thread
From: Min Zhou @ 2010-06-15 20:54 UTC (permalink / raw)
  To: Yehuda Sadeh Weinraub; +Cc: ceph-devel

Thanks Yehuda,

After correcting the my ceph.conf, it seems work now besides from some warnings.
Here is the new version of my ceph.conf

$ cat ~/ceph_conf/ceph.conf
[global]
       log dir = ceph_conf/out
       logger dir = ceph_conf/log
       chdir = ""
       pid file = ceph_conf/out/$type$id.pid
[mds]
       pid file = ceph_conf/out/$name.pid
       lockdep = 1
       debug ms = 1
       debug mds = 20
       mds log max segments = 2
[mds0]
       host=hdp2
[mds1]
       host=hdp3
[mon]
       lockdep = 1
       debug mon = 20
       debug paxos = 20
       debug ms = 1
       mon data = ceph_osd/dev/mon$id
[mon0]
       host = hdp2
       mon addr = 192.168.0.2:6789
[mon1]
       host = hdp3
       mon addr = 192.168.0.3:6789
[osd]
       lockdep = 1
       debug ms = 1
       debug osd = 25
       debug journal = 20
       debug filestore = 10
[osd0]
       host = hdp4
       user $USER
       osd data = ceph_osd/dev/osd0
       osd journal = ceph_osd/dev/osd0/journal
       osd journal size = 100
[osd1]
       host = hdp5
       user $USER
       osd data = ceph_osd/dev/osd1
       osd journal = ceph_osd/dev/osd1/journal
       osd journal size = 100
[osd2]
       host = hdp6
       user $USER
       osd data = ceph_osd/dev/osd2
       osd journal = ceph_osd/dev/osd2/journal
       osd journal size = 100
[mds.a]


and the results

$ mkcephfs --allhosts -c ceph_conf/ceph.conf -v
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t mon  "mon addr" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "mon addr" ""
/home/ceph/ceph/bin/monmaptool --create --clobber --add
172.19.201.2:6789 --add 172.19.201.3:6789 --print /tmp/monmap.10072
/home/ceph/ceph/bin/monmaptool: monmap file /tmp/monmap.10072
/home/ceph/ceph/bin/monmaptool: generated fsid
75902357-7c87-74f5-4166-88b92901c675
epoch 1
fsid 75902357-7c87-74f5-4166-88b92901c675
last_changed 10.06.16 04:44:32.874850
created 10.06.16 04:44:32.874850
        mon0 172.19.201.2:6789/0
        mon1 172.19.201.3:6789/0
/home/ceph/ceph/bin/monmaptool: writing epoch 1 to /tmp/monmap.10072
(2 monitors)
max osd in ceph_conf/ceph.conf is 2, num osd is 3
/home/ceph/ceph/bin/osdmaptool: osdmap file '/tmp/osdmap.10072'
/home/ceph/ceph/bin/osdmaptool: writing epoch 1 to /tmp/osdmap.10072
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "crush map src" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "crush map" ""
Building admin keyring at /tmp/admin.keyring.10072
creating /tmp/admin.keyring.10072
Building monitor keyring with all service keys
creating /tmp/monkeyring.10072
importing contents of /tmp/admin.keyring.10072 into /tmp/monkeyring.10072
creating /tmp/keyring.mds.0
importing contents of /tmp/keyring.mds.0 into /tmp/monkeyring.10072
creating /tmp/keyring.mds.1
importing contents of /tmp/keyring.mds.1 into /tmp/monkeyring.10072
creating /tmp/keyring.mds.a
importing contents of /tmp/keyring.mds.a into /tmp/monkeyring.10072
creating /tmp/keyring.osd.0
importing contents of /tmp/keyring.osd.0 into /tmp/monkeyring.10072
creating /tmp/keyring.osd.1
importing contents of /tmp/keyring.osd.1 into /tmp/monkeyring.10072
creating /tmp/keyring.osd.2
importing contents of /tmp/keyring.osd.2 into /tmp/monkeyring.10072
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t mon  "user" ""
=== mon0 ===
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t mon  "mon data" ""
--- hdp2# /home/ceph/ceph/bin/mkmonfs --clobber --mon-data
ceph_osd/dev/mon0 -i 0 --monmap /tmp/monmap.10072 --osdmap
/tmp/osdmap.10072 -k /tmp/keyring.10072 ; rm -f /tmp/keyring.10072
10.06.16 04:44:33.043908 2ab098621d30 store(ceph_osd/dev/mon0) mkfs
10.06.16 04:44:33.044025 2ab098621d30 store(ceph_osd/dev/mon0) test -d
ceph_osd/dev/mon0 && /bin/rm -rf ceph_osd/dev/mon0 ; mkdir -p
ceph_osd/dev/mon0
10.06.16 04:44:33.053747 2ab098621d30 mon0(starting).class v0
create_initial -- creating initial map
10.06.16 04:44:33.055299 2ab098621d30 mon0(starting).auth v0
create_initial -- creating initial map
10.06.16 04:44:33.055318 2ab098621d30 mon0(starting).auth v0 reading
initial keyring
/home/ceph/ceph/bin/mkmonfs: created monfs at ceph_osd/dev/mon0 for mon0
`/tmp/admin.keyring.10072' -> `ceph_osd/dev/mon0/admin_keyring.bin'
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "user" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "ssh
path" "/home/ceph"
=== mon1 ===
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mon  "mon data" ""
--- ssh hdp3  "cd /home/ceph ; ulimit -c unlimited ;
/home/ceph/ceph/bin/mkmonfs --clobber --mon-data ceph_osd/dev/mon1 -i
1 --monmap /tmp/monmap.10072 --osdmap /tmp/osdmap.10072 -k
/tmp/keyring.10072 ; rm -f /tmp/keyring.10072"
10.06.16 04:44:33.552910 2b19d7082d30 store(ceph_osd/dev/mon1) mkfs
10.06.16 04:44:33.553057 2b19d7082d30 store(ceph_osd/dev/mon1) test -d
ceph_osd/dev/mon1 && /bin/rm -rf ceph_osd/dev/mon1 ; mkdir -p
ceph_osd/dev/mon1
10.06.16 04:44:33.562586 2b19d7082d30 mon1(starting).class v0
create_initial -- creating initial map
10.06.16 04:44:33.564058 2b19d7082d30 mon1(starting).auth v0
create_initial -- creating initial map
10.06.16 04:44:33.564077 2b19d7082d30 mon1(starting).auth v0 reading
initial keyring
/home/ceph/ceph/bin/mkmonfs: created monfs at ceph_osd/dev/mon1 for mon1
admin.keyring.10072
                           100%  123     0.1KB/s   00:00
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t mds  "user" ""
=== mds0 ===
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t mds  "keyring" ""
WARNING: no keyring specified for mds0
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mds  "user" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mds  "ssh
path" "/home/ceph"
=== mds1 ===
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t mds  "keyring" ""
WARNING: no keyring specified for mds1
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i a -t mds  "user" ""
=== mds.a ===
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i a -t mds  "keyring" ""
WARNING: no keyring specified for mds.a
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t osd  "user" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t osd  "ssh
path" "/home/ceph"
=== osd0 ===
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t osd  "osd data" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t osd  "keyring" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t osd  "btrfs
path" "ceph_osd/dev/osd0"
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t osd  "btrfs devs" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 0 -t osd  "btrfs
options" "noatime"
--- ssh hdp4  "cd /home/ceph ; ulimit -c unlimited ; test -d
ceph_osd/dev/osd0 || mkdir -p ceph_osd/dev/osd0"
monmap.10072
                           100%  332     0.3KB/s   00:00
--- ssh hdp4  "cd /home/ceph ; ulimit -c unlimited ;
/home/ceph/ceph/bin/cosd -c ceph_conf/ceph.conf --monmap
/tmp/monmap.10072 -i 0 --mkfs --osd-data ceph_osd/dev/osd0"
root@hdp4's password:
 ** WARNING: Ceph is still under heavy development, and is only suitable for **
 **          testing and review.  Do not trust it with important data.       **
created object store for osd0 fsid
75902357-7c87-74f5-4166-88b92901c675 on ceph_osd/dev/osd0
WARNING: no keyring specified for osd0
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t osd  "user" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t osd  "ssh
path" "/home/ceph"
=== osd1 ===
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t osd  "osd data" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t osd  "keyring" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t osd  "btrfs
path" "ceph_osd/dev/osd1"
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t osd  "btrfs devs" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 1 -t osd  "btrfs
options" "noatime"
--- ssh hdp5  "cd /home/ceph ; ulimit -c unlimited ; test -d
ceph_osd/dev/osd1 || mkdir -p ceph_osd/dev/osd1"
monmap.10072
                           100%  332     0.3KB/s   00:00
--- ssh hdp5  "cd /home/ceph ; ulimit -c unlimited ;
/home/ceph/ceph/bin/cosd -c ceph_conf/ceph.conf --monmap
/tmp/monmap.10072 -i 1 --mkfs --osd-data ceph_osd/dev/osd1"
root@hdp5's password:
 ** WARNING: Ceph is still under heavy development, and is only suitable for **
 **          testing and review.  Do not trust it with important data.       **
created object store for osd1 fsid
75902357-7c87-74f5-4166-88b92901c675 on ceph_osd/dev/osd1
WARNING: no keyring specified for osd1
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 2 -t osd  "user" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 2 -t osd  "ssh
path" "/home/ceph"
=== osd2 ===
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 2 -t osd  "osd data" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 2 -t osd  "keyring" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 2 -t osd  "btrfs
path" "ceph_osd/dev/osd2"
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 2 -t osd  "btrfs devs" ""
/home/ceph/ceph/bin/cconf -c ceph_conf/ceph.conf -i 2 -t osd  "btrfs
options" "noatime"
--- ssh hdp6  "cd /home/ceph ; ulimit -c unlimited ; test -d
ceph_osd/dev/osd2 || mkdir -p ceph_osd/dev/osd2"
monmap.10072
                           100%  332     0.3KB/s   00:00
--- ssh hdp6  "cd /home/ceph ; ulimit -c unlimited ;
/home/ceph/ceph/bin/cosd -c ceph_conf/ceph.conf --monmap
/tmp/monmap.10072 -i 2 --mkfs --osd-data ceph_osd/dev/osd2"
root@hdp6's password:
 ** WARNING: Ceph is still under heavy development, and is only suitable for **
 **          testing and review.  Do not trust it with important data.       **
created object store for osd2 fsid
75902357-7c87-74f5-4166-88b92901c675 on ceph_osd/dev/osd2
WARNING: no keyring specified for osd2



Still two questions:
1. Why it prompted the need of all osd's root passwords?
2. Why it still has warning that tell me no keyring specified for each osd.

Thanks a lot,
Min

On Wed, Jun 16, 2010 at 4:31 AM, Yehuda Sadeh Weinraub
<yehudasa@gmail.com> wrote:
> On Tue, Jun 15, 2010 at 1:17 PM, Min Zhou <coderplay@gmail.com> wrote:
>> Here is my ceph.conf at /home/ceph/ceph.conf
> .. snip
>>       pid file = $CEPH_CONF/out/$name.pid
> Currently there are only a limited number of variables that you can
> use for replacement: $type, $id, $num, $name and $host. You can't use
> environment variables yet, although adding it might be a good idea.
>
>>       mon data = $CEPH_OSD_DIR/dev/mon0"
> ...
>>       mon data = $CEPH_OSD_DIR/dev/mon1"
> The same goes here. Note also that you're missing quotes anyway.
>
>
> Thanks,
> Yehuda
>



-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: setup ceph cluster failed
  2010-06-15 20:54       ` Min Zhou
@ 2010-06-15 21:19         ` Yehuda Sadeh Weinraub
  2010-06-16  2:32           ` Min Zhou
  0 siblings, 1 reply; 9+ messages in thread
From: Yehuda Sadeh Weinraub @ 2010-06-15 21:19 UTC (permalink / raw)
  To: Min Zhou; +Cc: ceph-devel

On Tue, Jun 15, 2010 at 1:54 PM, Min Zhou <coderplay@gmail.com> wrote:
>
>
> Still two questions:
> 1. Why it prompted the need of all osd's root passwords?
You specified --allhosts in the mkcephfs script command line. The
script uses ssh to run commands on the remote hosts, so you'd probably
need to setup your environment so that it doesn't prompt for password
when you login there (e.g., one way to do it is to create a
private/public keys pair, put the copy of your public key in the
remote host under ~/.ssh/authorized_keys).

> 2. Why it still has warning that tell me no keyring specified for each osd.

Probably because there wasn't any keyring specified in your ceph.conf.
As you don't setup your ceph cluster with authentication it doesn't
really matter.

Yehuda

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: setup ceph cluster failed
  2010-06-15 21:19         ` Yehuda Sadeh Weinraub
@ 2010-06-16  2:32           ` Min Zhou
  2010-06-16 19:54             ` Yehuda Sadeh Weinraub
  0 siblings, 1 reply; 9+ messages in thread
From: Min Zhou @ 2010-06-16  2:32 UTC (permalink / raw)
  To: Yehuda Sadeh Weinraub; +Cc: ceph-devel

Hi, Yehuda,




On Wed, Jun 16, 2010 at 5:19 AM, Yehuda Sadeh Weinraub
<yehudasa@gmail.com> wrote:
> On Tue, Jun 15, 2010 at 1:54 PM, Min Zhou <coderplay@gmail.com> wrote:
>>
>>
>> Still two questions:
>> 1. Why it prompted the need of all osd's root passwords?
> You specified --allhosts in the mkcephfs script command line. The
> script uses ssh to run commands on the remote hosts, so you'd probably
> need to setup your environment so that it doesn't prompt for password
> when you login there (e.g., one way to do it is to create a
> private/public keys pair, put the copy of your public key in the
> remote host under ~/.ssh/authorized_keys).


Actually, my machines were configured to SSH to without being prompted
for a password from any node to another one under the same user
account ceph. Why the system ask me for root passwords?

Btw, it seems there is still somewhat incorrectnesses of my
deployment, see below

$ ceph  -c ceph_conf/ceph.conf  start
10.06.16 05:00:29.182841 439bc940 -- :/10572 >> 192.168.0.2:6789/0
pipe(0x108f2df0 sd=-1 pgs=0 cs=0 l=0).fault first fault
10.06.16 05:00:39.185779 42fbb940 monclient(hunting): continuing hunt
10.06.16 05:00:39.186014 41ab6940 -- :/10572 >> 192.168.0.2:6789/0
pipe(0x108f58b0 sd=-1 pgs=0 cs=0 l=0).fault first fault
10.06.16 05:00:49.188963 42fbb940 monclient(hunting): continuing hunt
10.06.16 05:00:49.189299 443bd940 -- :/10572 >> 192.168.0.3:6789/0
pipe(0x108f5ee0 sd=-1 pgs=0 cs=0 l=0).fault first fault
10.06.16 05:00:59.192191 42fbb940 monclient(hunting): continuing hunt
10.06.16 05:00:59.192630 44dbe940 -- :/10572 >> 192.168.0.3:6789/0
pipe(0x108f7020 sd=-1 pgs=0 cs=0 l=0).fault first fault
10.06.16 05:01:09.195363 42fbb940 monclient(hunting): continuing hunt
10.06.16 05:01:09.195714 457bf940 -- :/10572 >> 192.168.0.3:6789/0
pipe(0x108f65e0 sd=-1 pgs=0 cs=0 l=0).fault first fault
10.06.16 05:01:19.198552 42fbb940 monclient(hunting): continuing hunt


Thanks,
Min


>> 2. Why it still has warning that tell me no keyring specified for each osd.
>
> Probably because there wasn't any keyring specified in your ceph.conf.
> As you don't setup your ceph cluster with authentication it doesn't
> really matter.
>
> Yehuda
>



-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: setup ceph cluster failed
  2010-06-16  2:32           ` Min Zhou
@ 2010-06-16 19:54             ` Yehuda Sadeh Weinraub
  2010-06-20  4:34               ` Min Zhou
  0 siblings, 1 reply; 9+ messages in thread
From: Yehuda Sadeh Weinraub @ 2010-06-16 19:54 UTC (permalink / raw)
  To: Min Zhou; +Cc: ceph-devel

On Tue, Jun 15, 2010 at 7:32 PM, Min Zhou <coderplay@gmail.com> wrote:
> Hi, Yehuda,
>
>
>
>
> On Wed, Jun 16, 2010 at 5:19 AM, Yehuda Sadeh Weinraub
> <yehudasa@gmail.com> wrote:
>> On Tue, Jun 15, 2010 at 1:54 PM, Min Zhou <coderplay@gmail.com> wrote:
>>>
>>>
>>> Still two questions:
>>> 1. Why it prompted the need of all osd's root passwords?
>> You specified --allhosts in the mkcephfs script command line. The
>> script uses ssh to run commands on the remote hosts, so you'd probably
>> need to setup your environment so that it doesn't prompt for password
>> when you login there (e.g., one way to do it is to create a
>> private/public keys pair, put the copy of your public key in the
>> remote host under ~/.ssh/authorized_keys).
>
>
> Actually, my machines were configured to SSH to without being prompted
> for a password from any node to another one under the same user
> account ceph. Why the system ask me for root passwords?

In order to use btrfs volumes the system needs to access the machines
as root (this is the default behavior). Also note that your 'user'
entry in your ceph.conf is messed up.

>
> Btw, it seems there is still somewhat incorrectnesses of my
> deployment, see below
>
> $ ceph  -c ceph_conf/ceph.conf  start
> 10.06.16 05:00:29.182841 439bc940 -- :/10572 >> 192.168.0.2:6789/0
> pipe(0x108f2df0 sd=-1 pgs=0 cs=0 l=0).fault first fault
> 10.06.16 05:00:39.185779 42fbb940 monclient(hunting): continuing hunt
> 10.06.16 05:00:39.186014 41ab6940 -- :/10572 >> 192.168.0.2:6789/0
> pipe(0x108f58b0 sd=-1 pgs=0 cs=0 l=0).fault first fault

This looks like your monitors are down.

Yehuda
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: setup ceph cluster failed
  2010-06-16 19:54             ` Yehuda Sadeh Weinraub
@ 2010-06-20  4:34               ` Min Zhou
  0 siblings, 0 replies; 9+ messages in thread
From: Min Zhou @ 2010-06-20  4:34 UTC (permalink / raw)
  To: Yehuda Sadeh Weinraub; +Cc: ceph-devel

Hi,

On Thu, Jun 17, 2010 at 3:54 AM, Yehuda Sadeh Weinraub
<yehudasa@gmail.com> wrote:
> On Tue, Jun 15, 2010 at 7:32 PM, Min Zhou <coderplay@gmail.com> wrote:
>> Hi, Yehuda,
>>
>>
>>
>>
>> On Wed, Jun 16, 2010 at 5:19 AM, Yehuda Sadeh Weinraub
>> <yehudasa@gmail.com> wrote:
>>> On Tue, Jun 15, 2010 at 1:54 PM, Min Zhou <coderplay@gmail.com> wrote:
>>>>
>>>>
>>>> Still two questions:
>>>> 1. Why it prompted the need of all osd's root passwords?
>>> You specified --allhosts in the mkcephfs script command line. The
>>> script uses ssh to run commands on the remote hosts, so you'd probably
>>> need to setup your environment so that it doesn't prompt for password
>>> when you login there (e.g., one way to do it is to create a
>>> private/public keys pair, put the copy of your public key in the
>>> remote host under ~/.ssh/authorized_keys).
>>
>>
>> Actually, my machines were configured to SSH to without being prompted
>> for a password from any node to another one under the same user
>> account ceph. Why the system ask me for root passwords?
>
> In order to use btrfs volumes the system needs to access the machines
> as root (this is the default behavior). Also note that your 'user'
> entry in your ceph.conf is messed up.
>

Currently, I use ext3 as ceph cluster 's  underlying ods, is that okay?


>>
>> Btw, it seems there is still somewhat incorrectnesses of my
>> deployment, see below
>>
>> $ ceph  -c ceph_conf/ceph.conf  start
>> 10.06.16 05:00:29.182841 439bc940 -- :/10572 >> 192.168.0.2:6789/0
>> pipe(0x108f2df0 sd=-1 pgs=0 cs=0 l=0).fault first fault
>> 10.06.16 05:00:39.185779 42fbb940 monclient(hunting): continuing hunt
>> 10.06.16 05:00:39.186014 41ab6940 -- :/10572 >> 192.168.0.2:6789/0
>> pipe(0x108f58b0 sd=-1 pgs=0 cs=0 l=0).fault first fault
>
> This looks like your monitors are down.
>
> Yehuda
>

Thanks,
Min
-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 9+ messages in thread
end of thread, other threads:[~2010-06-20  4:34 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-06-15 20:09 setup ceph cluster failed Min Zhou
2010-06-15 20:12 ` Yehuda Sadeh Weinraub
2010-06-15 20:17   ` Min Zhou
2010-06-15 20:31     ` Yehuda Sadeh Weinraub
2010-06-15 20:54       ` Min Zhou
2010-06-15 21:19         ` Yehuda Sadeh Weinraub
2010-06-16  2:32           ` Min Zhou
2010-06-16 19:54             ` Yehuda Sadeh Weinraub
2010-06-20  4:34               ` Min Zhou

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.