* The cluster do not aware some osd are disappear
@ 2012-07-31 7:48 Eric_YH_Chen
2012-07-31 21:42 ` Josh Durgin
0 siblings, 1 reply; 7+ messages in thread
From: Eric_YH_Chen @ 2012-07-31 7:48 UTC (permalink / raw)
To: ceph-devel; +Cc: Chris_YT_Huang, Victor_CY_Chang
Dear All:
My Environment: two servers, and 12 hard-disk on each server.
Version: Ceph 0.48, Kernel: 3.2.0-27
We create a ceph cluster with 24 osd, 3 monitors
Osd.0 ~ osd.11 is on server1
Osd.12 ~ osd.23 is on server2
Mon.0 is on server1
Mon.1 is on server2
Mon.2 is on server3 which has no osd
When I turn off the network of server1, we expect that server2 would aware 12 osd (on server 1) disappear.
However, when I type ceph -s, it still show 24 osd there.
And from the log of osd.0 and osd.11, we can find heartbeat check on server1, but not on server2.
What happened to server2? Can we restart the heartbeat server? Thanks!
root@wistor-002:~# ceph -s
health HEALTH_WARN 1 mons down, quorum 1,2 008,009
monmap e1: 3 mons at {006=192.168.200.84:6789/0,008=192.168.200.86:6789/0,009=192.168.200.87:6789/0}, election epoch 522, quorum 1,2 008,009
osdmap e1388: 24 osds: 24 up, 24 in
pgmap v288663: 4608 pgs: 4608 active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
mdsmap e1: 0/0/1 up
log of ceph -w (we turn of server1 arround 15:20, that cause the new monitor election)
2012-07-31 15:21:25.966572 mon.0 [INF] pgmap v288658: 4608 pgs: 4608 active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
2012-07-31 15:20:10.400566 mon.1 [INF] mon.008 calling new monitor election
2012-07-31 15:21:36.030473 mon.1 [INF] mon.008 calling new monitor election
2012-07-31 15:21:36.079772 mon.2 [INF] mon.009 calling new monitor election
2012-07-31 15:21:46.102587 mon.1 [INF] mon.008@1 won leader election with quorum 1,2
2012-07-31 15:21:46.273253 mon.1 [INF] pgmap v288659: 4608 pgs: 4608 active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
2012-07-31 15:21:46.273379 mon.1 [INF] mdsmap e1: 0/0/1 up
2012-07-31 15:21:46.273495 mon.1 [INF] osdmap e1388: 24 osds: 24 up, 24 in
2012-07-31 15:21:46.273814 mon.1 [INF] monmap e1: 3 mons at {006=192.168.200.84:6789/0,008=192.168.200.86:6789/0,009=192.168.200.87:6789/0}
2012-07-31 15:21:46.587679 mon.1 [INF] pgmap v288660: 4608 pgs: 4608 active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
2012-07-31 15:22:01.245813 mon.1 [INF] pgmap v288661: 4608 pgs: 4608 active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
2012-07-31 15:22:33.970838 mon.1 [INF] pgmap v288662: 4608 pgs: 4608 active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
Log of osd.0 (on server 1)
2012-07-31 15:20:25.309264 7fdc06470700 0 -- 192.168.200.81:6825/12162 >> 192.168.200.82:6840/8772 pipe(0x4dbea00 sd=52 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state 1
2012-07-31 15:20:25.310887 7fdc1c551700 0 -- 192.168.200.81:6825/12162 >> 192.168.200.82:6833/15570 pipe(0x4dbec80 sd=51 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state 1
2012-07-31 15:21:46.861458 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.12 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
2012-07-31 15:21:46.861496 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.13 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
2012-07-31 15:21:46.861506 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.14 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
2012-07-31 15:21:46.861514 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.15 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
2012-07-31 15:21:46.861522 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.16 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
2012-07-31 15:21:46.861530 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.17 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
2012-07-31 15:21:46.861538 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.18 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
2012-07-31 15:21:46.861546 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.19 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
2012-07-31 15:21:46.861556 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.20 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
2012-07-31 15:21:46.861576 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.21 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
2012-07-31 15:21:46.861609 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.22 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
2012-07-31 15:21:46.861618 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.23 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
Log of osd.12 (on server 2)
2012-07-31 15:20:31.475815 7f9eac5ba700 0 osd.12 1387 pg[2.16f( v 1356'10485 (465'9480,1356'10485] n=42 ec=1 les/c 1387/1387 1383/1383/1383) [12,0] r=0 lpr=1383 mlcod 0'0 active+clean] watch: oi.user_version=45
2012-07-31 15:20:31.475817 7f9eabdb9700 0 osd.12 1387 pg[2.205( v 1282'26975 (1254'25973,1282'26975] n=86 ec=1 les/c 1387/1387 1383/1383/1383) [12,9] r=0 lpr=1383 lcod 0'0 mlcod 0'0 active+clean] watch: ctx->obc=0x5838dc0 cookie=9 oi.version=26975 ctx->at_version=1387'26976
2012-07-31 15:20:31.475837 7f9eabdb9700 0 osd.12 1387 pg[2.205( v 1282'26975 (1254'25973,1282'26975] n=86 ec=1 les/c 1387/1387 1383/1383/1383) [12,9] r=0 lpr=1383 lcod 0'0 mlcod 0'0 active+clean] watch: oi.user_version=1043
2012-07-31 15:35:31.512306 7f9ea6f8e700 0 -- 192.168.200.82:6840/8772 >> 192.168.200.81:6847/18544 pipe(0x4633780 sd=41 pgs=82 cs=1 l=0).fault with nothing to send, going to standby
2012-07-31 15:35:31.512342 7f9ea7897700 0 -- 192.168.200.82:6840/8772 >> 192.168.200.81:6853/19122 pipe(0x4a68280 sd=43 pgs=83 cs=1 l=0).fault with nothing to send, going to standby
2012-07-31 15:35:31.579095 7f9ea6c8b700 0 -- 192.168.200.82:6840/8772 >> 192.168.200.81:6809/17957 pipe(0x6309c80 sd=55 pgs=80 cs=1 l=0).fault with nothing to send, going to standby
2012-07-31 15:35:31.592368 7f9ea7a99700 0 -- 192.168.200.82:6840/8772 >> 192.168.200.81:6840/12656 pipe(0x4b44780 sd=44 pgs=104 cs=1 l=0).fault with nothing to send, going to standby
2012-07-31 15:35:31.596484 7f9ea94b3700 0 -- 192.168.200.82:6840/8772 >> 192.168.200.81:6836/18275 pipe(0x4cfb780 sd=48 pgs=76 cs=1 l=0).fault with nothing to send, going to standby
2012-07-31 15:35:31.720803 7f9ea5a79700 0 -- 192.168.200.82:6840/8772 >> 192.168.200.81:6838/12409 pipe(0xeb4000 sd=38 pgs=105 cs=1 l=0).fault with nothing to send, going to standby
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: The cluster do not aware some osd are disappear
2012-07-31 7:48 The cluster do not aware some osd are disappear Eric_YH_Chen
@ 2012-07-31 21:42 ` Josh Durgin
2012-08-01 1:07 ` Eric_YH_Chen
0 siblings, 1 reply; 7+ messages in thread
From: Josh Durgin @ 2012-07-31 21:42 UTC (permalink / raw)
To: Eric_YH_Chen; +Cc: ceph-devel, Chris_YT_Huang, Victor_CY_Chang
On 07/31/2012 12:48 AM, Eric_YH_Chen@wiwynn.com wrote:
> Dear All:
>
> My Environment: two servers, and 12 hard-disk on each server.
> Version: Ceph 0.48, Kernel: 3.2.0-27
>
> We create a ceph cluster with 24 osd, 3 monitors
> Osd.0 ~ osd.11 is on server1
> Osd.12 ~ osd.23 is on server2
> Mon.0 is on server1
> Mon.1 is on server2
> Mon.2 is on server3 which has no osd
>
> When I turn off the network of server1, we expect that server2 would aware 12 osd (on server 1) disappear.
> However, when I type ceph -s, it still show 24 osd there.
There's a grace period before osds are marked down by their heartbeat
peers (configured by osd_heartbeat_grace), which is 20 seconds by
default. This avoids unnecessary data movement from short-lived issues.
> And from the log of osd.0 and osd.11, we can find heartbeat check on server1, but not on server2.
> What happened to server2? Can we restart the heartbeat server? Thanks!
There's no separate heartbeat server, it's a thread in the osd. If the
heartbeat thread stopped, the process would exit. If waiting longer
doesn't trigger the heartbeat failures, your crushmap might not
be separating peers onto different hosts correctly. OSDs heartbeat
with their peers.
Josh
> root@wistor-002:~# ceph -s
> health HEALTH_WARN 1 mons down, quorum 1,2 008,009
> monmap e1: 3 mons at {006=192.168.200.84:6789/0,008=192.168.200.86:6789/0,009=192.168.200.87:6789/0}, election epoch 522, quorum 1,2 008,009
> osdmap e1388: 24 osds: 24 up, 24 in
> pgmap v288663: 4608 pgs: 4608 active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
> mdsmap e1: 0/0/1 up
>
> log of ceph -w (we turn of server1 arround 15:20, that cause the new monitor election)
> 2012-07-31 15:21:25.966572 mon.0 [INF] pgmap v288658: 4608 pgs: 4608 active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
> 2012-07-31 15:20:10.400566 mon.1 [INF] mon.008 calling new monitor election
> 2012-07-31 15:21:36.030473 mon.1 [INF] mon.008 calling new monitor election
> 2012-07-31 15:21:36.079772 mon.2 [INF] mon.009 calling new monitor election
> 2012-07-31 15:21:46.102587 mon.1 [INF] mon.008@1 won leader election with quorum 1,2
> 2012-07-31 15:21:46.273253 mon.1 [INF] pgmap v288659: 4608 pgs: 4608 active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
> 2012-07-31 15:21:46.273379 mon.1 [INF] mdsmap e1: 0/0/1 up
> 2012-07-31 15:21:46.273495 mon.1 [INF] osdmap e1388: 24 osds: 24 up, 24 in
> 2012-07-31 15:21:46.273814 mon.1 [INF] monmap e1: 3 mons at {006=192.168.200.84:6789/0,008=192.168.200.86:6789/0,009=192.168.200.87:6789/0}
> 2012-07-31 15:21:46.587679 mon.1 [INF] pgmap v288660: 4608 pgs: 4608 active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
> 2012-07-31 15:22:01.245813 mon.1 [INF] pgmap v288661: 4608 pgs: 4608 active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
> 2012-07-31 15:22:33.970838 mon.1 [INF] pgmap v288662: 4608 pgs: 4608 active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
>
> Log of osd.0 (on server 1)
> 2012-07-31 15:20:25.309264 7fdc06470700 0 -- 192.168.200.81:6825/12162 >> 192.168.200.82:6840/8772 pipe(0x4dbea00 sd=52 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state 1
> 2012-07-31 15:20:25.310887 7fdc1c551700 0 -- 192.168.200.81:6825/12162 >> 192.168.200.82:6833/15570 pipe(0x4dbec80 sd=51 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state 1
> 2012-07-31 15:21:46.861458 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.12 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861496 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.13 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861506 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.14 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861514 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.15 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861522 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.16 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861530 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.17 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861538 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.18 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861546 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.19 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861556 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.20 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861576 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.21 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861609 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.22 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861618 7fdc14e9d700 -1 osd.0 1388 heartbeat_check: no reply from osd.23 since 2012-07-31 15:21:26.770108 (cutoff 2012-07-31 15:21:26.861458)
>
> Log of osd.12 (on server 2)
> 2012-07-31 15:20:31.475815 7f9eac5ba700 0 osd.12 1387 pg[2.16f( v 1356'10485 (465'9480,1356'10485] n=42 ec=1 les/c 1387/1387 1383/1383/1383) [12,0] r=0 lpr=1383 mlcod 0'0 active+clean] watch: oi.user_version=45
> 2012-07-31 15:20:31.475817 7f9eabdb9700 0 osd.12 1387 pg[2.205( v 1282'26975 (1254'25973,1282'26975] n=86 ec=1 les/c 1387/1387 1383/1383/1383) [12,9] r=0 lpr=1383 lcod 0'0 mlcod 0'0 active+clean] watch: ctx->obc=0x5838dc0 cookie=9 oi.version=26975 ctx->at_version=1387'26976
> 2012-07-31 15:20:31.475837 7f9eabdb9700 0 osd.12 1387 pg[2.205( v 1282'26975 (1254'25973,1282'26975] n=86 ec=1 les/c 1387/1387 1383/1383/1383) [12,9] r=0 lpr=1383 lcod 0'0 mlcod 0'0 active+clean] watch: oi.user_version=1043
> 2012-07-31 15:35:31.512306 7f9ea6f8e700 0 -- 192.168.200.82:6840/8772 >> 192.168.200.81:6847/18544 pipe(0x4633780 sd=41 pgs=82 cs=1 l=0).fault with nothing to send, going to standby
> 2012-07-31 15:35:31.512342 7f9ea7897700 0 -- 192.168.200.82:6840/8772 >> 192.168.200.81:6853/19122 pipe(0x4a68280 sd=43 pgs=83 cs=1 l=0).fault with nothing to send, going to standby
> 2012-07-31 15:35:31.579095 7f9ea6c8b700 0 -- 192.168.200.82:6840/8772 >> 192.168.200.81:6809/17957 pipe(0x6309c80 sd=55 pgs=80 cs=1 l=0).fault with nothing to send, going to standby
> 2012-07-31 15:35:31.592368 7f9ea7a99700 0 -- 192.168.200.82:6840/8772 >> 192.168.200.81:6840/12656 pipe(0x4b44780 sd=44 pgs=104 cs=1 l=0).fault with nothing to send, going to standby
> 2012-07-31 15:35:31.596484 7f9ea94b3700 0 -- 192.168.200.82:6840/8772 >> 192.168.200.81:6836/18275 pipe(0x4cfb780 sd=48 pgs=76 cs=1 l=0).fault with nothing to send, going to standby
> 2012-07-31 15:35:31.720803 7f9ea5a79700 0 -- 192.168.200.82:6840/8772 >> 192.168.200.81:6838/12409 pipe(0xeb4000 sd=38 pgs=105 cs=1 l=0).fault with nothing to send, going to standby
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: The cluster do not aware some osd are disappear
2012-07-31 21:42 ` Josh Durgin
@ 2012-08-01 1:07 ` Eric_YH_Chen
2012-08-01 2:05 ` rbd create issues - like the bug Ryan Nicholson
2012-08-01 15:57 ` The cluster do not aware some osd are disappear Tommi Virtanen
0 siblings, 2 replies; 7+ messages in thread
From: Eric_YH_Chen @ 2012-08-01 1:07 UTC (permalink / raw)
To: josh.durgin; +Cc: ceph-devel, Chris_YT_Huang, Victor_CY_Chang
Hi, Josh:
I do not assign the crushmap by myself, I use the default setting.
And after I reboot the server, I cannot reproduce this situation.
The heartbeat check works fine when one of the server not available.
-----Original Message-----
From: Josh Durgin [mailto:josh.durgin@inktank.com]
Sent: Wednesday, August 01, 2012 5:43 AM
To: Eric YH Chen/WYHQ/Wiwynn
Cc: ceph-devel@vger.kernel.org; Chris YT Huang/WYHQ/Wiwynn; Victor CY Chang/WYHQ/Wiwynn
Subject: Re: The cluster do not aware some osd are disappear
On 07/31/2012 12:48 AM, Eric_YH_Chen@wiwynn.com wrote:
> Dear All:
>
> My Environment: two servers, and 12 hard-disk on each server.
> Version: Ceph 0.48, Kernel: 3.2.0-27
>
> We create a ceph cluster with 24 osd, 3 monitors
> Osd.0 ~ osd.11 is on server1
> Osd.12 ~ osd.23 is on server2
> Mon.0 is on server1
> Mon.1 is on server2
> Mon.2 is on server3 which has no osd
>
> When I turn off the network of server1, we expect that server2 would aware 12 osd (on server 1) disappear.
> However, when I type ceph -s, it still show 24 osd there.
There's a grace period before osds are marked down by their heartbeat peers (configured by osd_heartbeat_grace), which is 20 seconds by default. This avoids unnecessary data movement from short-lived issues.
> And from the log of osd.0 and osd.11, we can find heartbeat check on server1, but not on server2.
> What happened to server2? Can we restart the heartbeat server? Thanks!
There's no separate heartbeat server, it's a thread in the osd. If the heartbeat thread stopped, the process would exit. If waiting longer doesn't trigger the heartbeat failures, your crushmap might not be separating peers onto different hosts correctly. OSDs heartbeat with their peers.
Josh
> root@wistor-002:~# ceph -s
> health HEALTH_WARN 1 mons down, quorum 1,2 008,009
> monmap e1: 3 mons at {006=192.168.200.84:6789/0,008=192.168.200.86:6789/0,009=192.168.200.87:6789/0}, election epoch 522, quorum 1,2 008,009
> osdmap e1388: 24 osds: 24 up, 24 in
> pgmap v288663: 4608 pgs: 4608 active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
> mdsmap e1: 0/0/1 up
>
> log of ceph -w (we turn of server1 arround 15:20, that cause the new
> monitor election)
> 2012-07-31 15:21:25.966572 mon.0 [INF] pgmap v288658: 4608 pgs: 4608
> active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
> 2012-07-31 15:20:10.400566 mon.1 [INF] mon.008 calling new monitor
> election
> 2012-07-31 15:21:36.030473 mon.1 [INF] mon.008 calling new monitor
> election
> 2012-07-31 15:21:36.079772 mon.2 [INF] mon.009 calling new monitor
> election
> 2012-07-31 15:21:46.102587 mon.1 [INF] mon.008@1 won leader election
> with quorum 1,2
> 2012-07-31 15:21:46.273253 mon.1 [INF] pgmap v288659: 4608 pgs: 4608
> active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
> 2012-07-31 15:21:46.273379 mon.1 [INF] mdsmap e1: 0/0/1 up
> 2012-07-31 15:21:46.273495 mon.1 [INF] osdmap e1388: 24 osds: 24 up,
> 24 in
> 2012-07-31 15:21:46.273814 mon.1 [INF] monmap e1: 3 mons at
> {006=192.168.200.84:6789/0,008=192.168.200.86:6789/0,009=192.168.200.8
> 7:6789/0}
> 2012-07-31 15:21:46.587679 mon.1 [INF] pgmap v288660: 4608 pgs: 4608
> active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
> 2012-07-31 15:22:01.245813 mon.1 [INF] pgmap v288661: 4608 pgs: 4608
> active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
> 2012-07-31 15:22:33.970838 mon.1 [INF] pgmap v288662: 4608 pgs: 4608
> active+clean; 257 GB data, 988 GB used, 20214 GB / 22320 GB avail
>
> Log of osd.0 (on server 1)
> 2012-07-31 15:20:25.309264 7fdc06470700 0 --
> 192.168.200.81:6825/12162 >> 192.168.200.82:6840/8772 pipe(0x4dbea00
> sd=52 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state 1
> 2012-07-31 15:20:25.310887 7fdc1c551700 0 --
> 192.168.200.81:6825/12162 >> 192.168.200.82:6833/15570 pipe(0x4dbec80
> sd=51 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state 1
> 2012-07-31 15:21:46.861458 7fdc14e9d700 -1 osd.0 1388 heartbeat_check:
> no reply from osd.12 since 2012-07-31 15:21:26.770108 (cutoff
> 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861496 7fdc14e9d700 -1 osd.0 1388 heartbeat_check:
> no reply from osd.13 since 2012-07-31 15:21:26.770108 (cutoff
> 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861506 7fdc14e9d700 -1 osd.0 1388 heartbeat_check:
> no reply from osd.14 since 2012-07-31 15:21:26.770108 (cutoff
> 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861514 7fdc14e9d700 -1 osd.0 1388 heartbeat_check:
> no reply from osd.15 since 2012-07-31 15:21:26.770108 (cutoff
> 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861522 7fdc14e9d700 -1 osd.0 1388 heartbeat_check:
> no reply from osd.16 since 2012-07-31 15:21:26.770108 (cutoff
> 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861530 7fdc14e9d700 -1 osd.0 1388 heartbeat_check:
> no reply from osd.17 since 2012-07-31 15:21:26.770108 (cutoff
> 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861538 7fdc14e9d700 -1 osd.0 1388 heartbeat_check:
> no reply from osd.18 since 2012-07-31 15:21:26.770108 (cutoff
> 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861546 7fdc14e9d700 -1 osd.0 1388 heartbeat_check:
> no reply from osd.19 since 2012-07-31 15:21:26.770108 (cutoff
> 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861556 7fdc14e9d700 -1 osd.0 1388 heartbeat_check:
> no reply from osd.20 since 2012-07-31 15:21:26.770108 (cutoff
> 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861576 7fdc14e9d700 -1 osd.0 1388 heartbeat_check:
> no reply from osd.21 since 2012-07-31 15:21:26.770108 (cutoff
> 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861609 7fdc14e9d700 -1 osd.0 1388 heartbeat_check:
> no reply from osd.22 since 2012-07-31 15:21:26.770108 (cutoff
> 2012-07-31 15:21:26.861458)
> 2012-07-31 15:21:46.861618 7fdc14e9d700 -1 osd.0 1388 heartbeat_check:
> no reply from osd.23 since 2012-07-31 15:21:26.770108 (cutoff
> 2012-07-31 15:21:26.861458)
>
> Log of osd.12 (on server 2)
> 2012-07-31 15:20:31.475815 7f9eac5ba700 0 osd.12 1387 pg[2.16f( v
> 1356'10485 (465'9480,1356'10485] n=42 ec=1 les/c 1387/1387
> 1383/1383/1383) [12,0] r=0 lpr=1383 mlcod 0'0 active+clean] watch:
> oi.user_version=45
> 2012-07-31 15:20:31.475817 7f9eabdb9700 0 osd.12 1387 pg[2.205( v
> 1282'26975 (1254'25973,1282'26975] n=86 ec=1 les/c 1387/1387
> 1383/1383/1383) [12,9] r=0 lpr=1383 lcod 0'0 mlcod 0'0 active+clean]
> watch: ctx->obc=0x5838dc0 cookie=9 oi.version=26975
> ctx->at_version=1387'26976
> 2012-07-31 15:20:31.475837 7f9eabdb9700 0 osd.12 1387 pg[2.205( v
> 1282'26975 (1254'25973,1282'26975] n=86 ec=1 les/c 1387/1387
> 1383/1383/1383) [12,9] r=0 lpr=1383 lcod 0'0 mlcod 0'0 active+clean]
> watch: oi.user_version=1043
> 2012-07-31 15:35:31.512306 7f9ea6f8e700 0 -- 192.168.200.82:6840/8772
> >> 192.168.200.81:6847/18544 pipe(0x4633780 sd=41 pgs=82 cs=1
> l=0).fault with nothing to send, going to standby
> 2012-07-31 15:35:31.512342 7f9ea7897700 0 -- 192.168.200.82:6840/8772
> >> 192.168.200.81:6853/19122 pipe(0x4a68280 sd=43 pgs=83 cs=1
> l=0).fault with nothing to send, going to standby
> 2012-07-31 15:35:31.579095 7f9ea6c8b700 0 -- 192.168.200.82:6840/8772
> >> 192.168.200.81:6809/17957 pipe(0x6309c80 sd=55 pgs=80 cs=1
> l=0).fault with nothing to send, going to standby
> 2012-07-31 15:35:31.592368 7f9ea7a99700 0 -- 192.168.200.82:6840/8772
> >> 192.168.200.81:6840/12656 pipe(0x4b44780 sd=44 pgs=104 cs=1
> l=0).fault with nothing to send, going to standby
> 2012-07-31 15:35:31.596484 7f9ea94b3700 0 -- 192.168.200.82:6840/8772
> >> 192.168.200.81:6836/18275 pipe(0x4cfb780 sd=48 pgs=76 cs=1
> l=0).fault with nothing to send, going to standby
> 2012-07-31 15:35:31.720803 7f9ea5a79700 0 -- 192.168.200.82:6840/8772
> >> 192.168.200.81:6838/12409 pipe(0xeb4000 sd=38 pgs=105 cs=1
> l=0).fault with nothing to send, going to standby
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@vger.kernel.org More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: rbd create issues - like the bug...
2012-08-01 1:07 ` Eric_YH_Chen
@ 2012-08-01 2:05 ` Ryan Nicholson
2012-08-01 5:47 ` Ryan Nicholson
2012-08-01 15:57 ` The cluster do not aware some osd are disappear Tommi Virtanen
1 sibling, 1 reply; 7+ messages in thread
From: Ryan Nicholson @ 2012-08-01 2:05 UTC (permalink / raw)
To: ceph-devel
All: I cannot get this to work:
Rbd create testImage --size 2000
My ceph.conf. file contains
#
[osd]
osd_class_dir = /usr/local/lib/rados-classes
#
I've added /usr/local/lib/rados-classes to my ld.so.conf
I've done a ldconfig.
libcls_rbd.so & so.1 & so.1.0.0 are in the cache. (ldconfig -p | grep ...)
However, I'm still getting the error noted here http://ceph.com/docs/master/dev/osd-class-path/
[librbd: failed to assign a block name for image]
Not sure where to go next...
Thanks, for any help.
Ryan Nicholson
^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: rbd create issues - like the bug...
2012-08-01 2:05 ` rbd create issues - like the bug Ryan Nicholson
@ 2012-08-01 5:47 ` Ryan Nicholson
0 siblings, 0 replies; 7+ messages in thread
From: Ryan Nicholson @ 2012-08-01 5:47 UTC (permalink / raw)
To: Ryan Nicholson, ceph-devel
Nevermind. One OSD node was missing the lib.
Cheers!
-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Ryan Nicholson
Sent: Tuesday, July 31, 2012 9:05 PM
To: ceph-devel@vger.kernel.org
Subject: RE: rbd create issues - like the bug...
All: I cannot get this to work:
Rbd create testImage --size 2000
My ceph.conf. file contains
#
[osd]
osd_class_dir = /usr/local/lib/rados-classes #
I've added /usr/local/lib/rados-classes to my ld.so.conf I've done a ldconfig.
libcls_rbd.so & so.1 & so.1.0.0 are in the cache. (ldconfig -p | grep ...)
However, I'm still getting the error noted here http://ceph.com/docs/master/dev/osd-class-path/
[librbd: failed to assign a block name for image]
Not sure where to go next...
Thanks, for any help.
Ryan Nicholson
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: The cluster do not aware some osd are disappear
2012-08-01 1:07 ` Eric_YH_Chen
2012-08-01 2:05 ` rbd create issues - like the bug Ryan Nicholson
@ 2012-08-01 15:57 ` Tommi Virtanen
2012-08-03 1:43 ` Eric_YH_Chen
1 sibling, 1 reply; 7+ messages in thread
From: Tommi Virtanen @ 2012-08-01 15:57 UTC (permalink / raw)
To: Eric_YH_Chen; +Cc: josh.durgin, ceph-devel, Chris_YT_Huang, Victor_CY_Chang
On Tue, Jul 31, 2012 at 6:07 PM, <Eric_YH_Chen@wiwynn.com> wrote:
> Hi, Josh:
>
> I do not assign the crushmap by myself, I use the default setting.
> And after I reboot the server, I cannot reproduce this situation.
> The heartbeat check works fine when one of the server not available.
If you don't do anything to your crushmap, all yours osds are in a
flat tree, with no understanding of your failure domains. You really
should configure it. (We really should document it better!)
The newer upstart scripts (/etc/init/ceph-osd.conf instead of
/etc/init.d/ceph) at least set the hostname by default, but that still
ignores racks, rooms etc.
^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: The cluster do not aware some osd are disappear
2012-08-01 15:57 ` The cluster do not aware some osd are disappear Tommi Virtanen
@ 2012-08-03 1:43 ` Eric_YH_Chen
0 siblings, 0 replies; 7+ messages in thread
From: Eric_YH_Chen @ 2012-08-03 1:43 UTC (permalink / raw)
To: tv; +Cc: josh.durgin, ceph-devel, Chris_YT_Huang, Victor_CY_Chang
Hi, Tommi:
I use this two command to get the crush map, how should I modify it?
ceph osd getcrushmap -o curmap
crushtool -d curmap -o curmap.txt
# begin crush map
# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2
device 3 osd.3
device 4 osd.4
device 5 osd.5
device 6 osd.6
device 7 osd.7
device 8 osd.8
device 9 osd.9
device 10 osd.10
device 11 osd.11
device 12 osd.12
device 13 osd.13
device 14 osd.14
device 15 osd.15
device 16 osd.16
device 17 osd.17
device 18 osd.18
device 19 osd.19
device 20 osd.20
device 21 osd.21
device 22 osd.22
device 23 osd.23
# types
type 0 osd
type 1 host
type 2 rack
type 3 row
type 4 room
type 5 datacenter
type 6 pool
# buckets
host wistor-001 {
id -2 # do not change unnecessarily
# weight 12.000
alg straw
hash 0 # rjenkins1
item osd.0 weight 1.000
item osd.1 weight 1.000
item osd.10 weight 1.000
item osd.11 weight 1.000
item osd.2 weight 1.000
item osd.3 weight 1.000
item osd.4 weight 1.000
item osd.5 weight 1.000
item osd.6 weight 1.000
item osd.7 weight 1.000
item osd.8 weight 1.000
item osd.9 weight 1.000
}
host wistor-002 {
id -4 # do not change unnecessarily
# weight 12.000
alg straw
hash 0 # rjenkins1
item osd.12 weight 1.000
item osd.13 weight 1.000
item osd.14 weight 1.000
item osd.15 weight 1.000
item osd.16 weight 1.000
item osd.17 weight 1.000
item osd.18 weight 1.000
item osd.19 weight 1.000
item osd.20 weight 1.000
item osd.21 weight 1.000
item osd.22 weight 1.000
item osd.23 weight 1.000
}
rack unknownrack {
id -3 # do not change unnecessarily
# weight 24.000
alg straw
hash 0 # rjenkins1
item wistor-001 weight 12.000
item wistor-002 weight 12.000
}
pool default {
id -1 # do not change unnecessarily
# weight 24.000
alg straw
hash 0 # rjenkins1
item unknownrack weight 24.000
}
# rules
rule data {
ruleset 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
rule metadata {
ruleset 1
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
rule rbd {
ruleset 2
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
# end crush map
-----Original Message-----
From: Tommi Virtanen [mailto:tv@inktank.com]
Sent: Wednesday, August 01, 2012 11:58 PM
To: Eric YH Chen/WYHQ/Wiwynn
Cc: josh.durgin@inktank.com; ceph-devel@vger.kernel.org; Chris YT Huang/WYHQ/Wiwynn; Victor CY Chang/WYHQ/Wiwynn
Subject: Re: The cluster do not aware some osd are disappear
On Tue, Jul 31, 2012 at 6:07 PM, <Eric_YH_Chen@wiwynn.com> wrote:
> Hi, Josh:
>
> I do not assign the crushmap by myself, I use the default setting.
> And after I reboot the server, I cannot reproduce this situation.
> The heartbeat check works fine when one of the server not available.
If you don't do anything to your crushmap, all yours osds are in a flat tree, with no understanding of your failure domains. You really should configure it. (We really should document it better!)
The newer upstart scripts (/etc/init/ceph-osd.conf instead of
/etc/init.d/ceph) at least set the hostname by default, but that still ignores racks, rooms etc.
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2012-08-03 1:43 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-07-31 7:48 The cluster do not aware some osd are disappear Eric_YH_Chen
2012-07-31 21:42 ` Josh Durgin
2012-08-01 1:07 ` Eric_YH_Chen
2012-08-01 2:05 ` rbd create issues - like the bug Ryan Nicholson
2012-08-01 5:47 ` Ryan Nicholson
2012-08-01 15:57 ` The cluster do not aware some osd are disappear Tommi Virtanen
2012-08-03 1:43 ` Eric_YH_Chen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.