* Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
@ 2013-01-24 17:24 Isaac Otsiabah
2013-01-24 21:28 ` Gregory Farnum
0 siblings, 1 reply; 13+ messages in thread
From: Isaac Otsiabah @ 2013-01-24 17:24 UTC (permalink / raw)
To: ceph-devel
[-- Attachment #1: Type: text/plain, Size: 5466 bytes --]
Gregory, i tried send the the attached debug output several times and
the mail server rejected them all probably becauseof the file size so i cut the log file size down and it is attached. You will see the
reconnection failures by the error message line below. The ceph version
is 0.56
it appears to be a timing issue because with the flag (debug ms=1) turned on, the system ran slower and became harder to fail.
I
ran it several times and finally got it to fail on (osd.0) using
default crush map. The attached tar file contains log files for all
components on g8ct plus the ceph.conf. By the way, the log file contain only the last 1384 lines where the error occurs.
I started with a 1-node cluster on host g8ct (osd.0, osd.1, osd.2) and then added host g13ct (osd.3, osd.4, osd.5)
id weight type name up/down reweight
-1 6 root default
-3 6 rack unknownrack
-2 3 host g8ct
0 1 osd.0 down 1
1 1 osd.1 up 1
2 1 osd.2 up 1
-4 3 host g13ct
3 1 osd.3 up 1
4 1 osd.4 up 1
5 1 osd.5 up 1
The error messages are in ceph.log and ceph-osd.0.log:
ceph.log:2013-01-08
05:41:38.080470 osd.0 192.168.0.124:6801/25571 3 : [ERR] map e15 had
wrong cluster addr (192.168.0.124:6802/25571 != my
192.168.1.124:6802/25571)
ceph-osd.0.log:2013-01-08 05:41:38.080458 7f06757fa710 0 log [ERR] : map e15 had wrong cluster addr
(192.168.0.124:6802/25571 != my 192.168.1.124:6802/25571)
[root@g8ct ceph]# ceph -v
ceph version 0.56 (1a32f0a0b42f169a7b55ed48ec3208f6d4edc1e8)
Isaac
----- Original Message -----
From: Gregory Farnum <greg@inktank.com>
To: Isaac Otsiabah <zmoo76b@yahoo.com>
Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Sent: Monday, January 7, 2013 1:27 PM
Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
On Monday, January 7, 2013 at 1:00 PM, Isaac Otsiabah wrote:
>
>
>
When i add a new host (with osd's) to my existing cluster, 1 or 2
previous osd(s) goes down for about 2 minutes and then they come back
up.
>
>
> [root@h1ct ~]# ceph osd tree
>
> # id weight type name up/down reweight
> -1
> 3 root default
> -3 3 rack unknownrack
> -2 3 host h1
> 0 1 osd.0 up 1
> 1 1 osd.1 up 1
> 2
> 1 osd.2 up 1
>
>
>
For example, after adding host h2 (with 3 new osd) to the above cluster
and running the "ceph osd tree" command, i see this:
>
>
> [root@h1 ~]# ceph osd tree
>
> # id weight type name up/down reweight
> -1 6 root default
> -3
> 6 rack unknownrack
> -2 3 host h1
> 0 1 osd.0 up 1
> 1 1 osd.1 down 1
> 2
> 1 osd.2 up 1
> -4 3 host h2
> 3 1 osd.3 up 1
> 4 1 osd.4 up
> 1
> 5 1 osd.5 up 1
>
>
>
The down osd always come back up after 2 minutes or less andi see the
following error message in the respective osd log file:
> 2013-01-07 04:40:17.613028 7fec7f092760 1 journal _open
> /ceph_journal/journals/journal_2 fd 26: 1073741824 bytes, block size
> 4096 bytes, directio = 1, aio = 0
> 2013-01-07 04:40:17.613122
> 7fec7f092760 1 journal _open /ceph_journal/journals/journal_2 fd 26:
> 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> 2013-01-07
> 04:42:10.006533 7fec746f7710 0 -- 192.168.0.124:6808/19449 >>
> 192.168.1.123:6800/18287 pipe(0x7fec20000e10 sd=31 :6808 pgs=0 cs=0
> l=0).accept connect_seq 0 vs existing 0 state connecting
> 2013-01-07
> 04:45:29.834341 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
> 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45438 pgs=7 cs=1
> l=0).fault, initiating reconnect
> 2013-01-07 04:45:29.835748
> 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
> 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45439 pgs=15 cs=3
> l=0).fault, initiating reconnect
> 2013-01-07 04:45:30.835219 7fec743f4710 0 --
> 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
> pipe(0x7fec5402f320 sd=28 :45894 pgs=482 cs=903 l=0).fault, initiating
> reconnect
> 2013-01-07 04:45:30.837318 7fec743f4710 0 --
> 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
> pipe(0x7fec5402f320 sd=28 :45895 pgs=483 cs=905 l=0).fault, initiating
> reconnect
> 2013-01-07 04:45:30.851984 7fec637fe710 0 log [ERR] : map
> e27 had wrong cluster addr (192.168.0.124:6808/19449 != my
> 192.168.1.124:6808/19449)
>
> Also, this only happens only when the cluster ip address and the public ip address are different for example
> ....
> ....
> ....
> [osd.0]
> host = g8ct
> public address = 192.168.0.124
> cluster address = 192.168.1.124
> btrfs devs = /dev/sdb
>
> ....
> ....
>
> but does not happen when they are the same. Any idea what may be the issue?
>
This isn't familiar to me at first glance. What version of Ceph are you using?
If
this is easy to reproduce, can you pastebin your ceph.conf and then add
"debug ms = 1" to your global config and gather up the logs from each
daemon?
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo
[-- Attachment #2: ceph-osd.0.log.tar.gz --]
[-- Type: application/x-gzip, Size: 30099 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
2013-01-24 17:24 osd down (for 2 about 2 minutes) error after adding a new host to my cluster Isaac Otsiabah
@ 2013-01-24 21:28 ` Gregory Farnum
2013-01-25 17:51 ` Isaac Otsiabah
0 siblings, 1 reply; 13+ messages in thread
From: Gregory Farnum @ 2013-01-24 21:28 UTC (permalink / raw)
To: Isaac Otsiabah; +Cc: ceph-devel
What's the physical layout of your networking? This additional log may prove helpful as well, but I really need a bit more context in evaluating the messages I see from the first one. :)
-Greg
On Thursday, January 24, 2013 at 9:24 AM, Isaac Otsiabah wrote:
>
>
> Gregory, i tried send the the attached debug output several times and
> the mail server rejected them all probably becauseof the file size so i cut the log file size down and it is attached. You will see the
> reconnection failures by the error message line below. The ceph version
> is 0.56
>
>
> it appears to be a timing issue because with the flag (debug ms=1) turned on, the system ran slower and became harder to fail.
> I
> ran it several times and finally got it to fail on (osd.0) using
> default crush map. The attached tar file contains log files for all
> components on g8ct plus the ceph.conf. By the way, the log file contain only the last 1384 lines where the error occurs.
>
>
> I started with a 1-node cluster on host g8ct (osd.0, osd.1, osd.2) and then added host g13ct (osd.3, osd.4, osd.5)
>
>
> id weight type name up/down reweight
> -1 6 root default
> -3 6 rack unknownrack
> -2 3 host g8ct
> 0 1 osd.0 down 1
> 1 1 osd.1 up 1
> 2 1 osd.2 up 1
> -4 3 host g13ct
> 3 1 osd.3 up 1
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
>
>
>
> The error messages are in ceph.log and ceph-osd.0.log:
>
> ceph.log:2013-01-08
> 05:41:38.080470 osd.0 192.168.0.124:6801/25571 3 : [ERR] map e15 had
> wrong cluster addr (192.168.0.124:6802/25571 != my
> 192.168.1.124:6802/25571)
> ceph-osd.0.log:2013-01-08 05:41:38.080458 7f06757fa710 0 log [ERR] : map e15 had wrong cluster addr
> (192.168.0.124:6802/25571 != my 192.168.1.124:6802/25571)
>
>
>
> [root@g8ct ceph]# ceph -v
> ceph version 0.56 (1a32f0a0b42f169a7b55ed48ec3208f6d4edc1e8)
>
>
> Isaac
>
>
> ----- Original Message -----
> From: Gregory Farnum <greg@inktank.com (mailto:greg@inktank.com)>
> To: Isaac Otsiabah <zmoo76b@yahoo.com (mailto:zmoo76b@yahoo.com)>
> Cc: "ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)" <ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)>
> Sent: Monday, January 7, 2013 1:27 PM
> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>
> On Monday, January 7, 2013 at 1:00 PM, Isaac Otsiabah wrote:
>
>
> When i add a new host (with osd's) to my existing cluster, 1 or 2
> previous osd(s) goes down for about 2 minutes and then they come back
> up.
> >
> >
> > [root@h1ct ~]# ceph osd tree
> >
> > # id weight type name up/down reweight
> > -1
> > 3 root default
> > -3 3 rack unknownrack
> > -2 3 host h1
> > 0 1 osd.0 up 1
> > 1 1 osd.1 up 1
> > 2
> > 1 osd.2 up 1
>
>
> For example, after adding host h2 (with 3 new osd) to the above cluster
> and running the "ceph osd tree" command, i see this:
> >
> >
> > [root@h1 ~]# ceph osd tree
> >
> > # id weight type name up/down reweight
> > -1 6 root default
> > -3
> > 6 rack unknownrack
> > -2 3 host h1
> > 0 1 osd.0 up 1
> > 1 1 osd.1 down 1
> > 2
> > 1 osd.2 up 1
> > -4 3 host h2
> > 3 1 osd.3 up 1
> > 4 1 osd.4 up
> > 1
> > 5 1 osd.5 up 1
>
>
> The down osd always come back up after 2 minutes or less andi see the
> following error message in the respective osd log file:
> > 2013-01-07 04:40:17.613028 7fec7f092760 1 journal _open
> > /ceph_journal/journals/journal_2 fd 26: 1073741824 bytes, block size
> > 4096 bytes, directio = 1, aio = 0
> > 2013-01-07 04:40:17.613122
> > 7fec7f092760 1 journal _open /ceph_journal/journals/journal_2 fd 26:
> > 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> > 2013-01-07
> > 04:42:10.006533 7fec746f7710 0 -- 192.168.0.124:6808/19449 >>
> > 192.168.1.123:6800/18287 pipe(0x7fec20000e10 sd=31 :6808 pgs=0 cs=0
> > l=0).accept connect_seq 0 vs existing 0 state connecting
> > 2013-01-07
> > 04:45:29.834341 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45438 pgs=7 cs=1
> > l=0).fault, initiating reconnect
> > 2013-01-07 04:45:29.835748
> > 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45439 pgs=15 cs=3
> > l=0).fault, initiating reconnect
> > 2013-01-07 04:45:30.835219 7fec743f4710 0 --
> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
> > pipe(0x7fec5402f320 sd=28 :45894 pgs=482 cs=903 l=0).fault, initiating
> > reconnect
> > 2013-01-07 04:45:30.837318 7fec743f4710 0 --
> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
> > pipe(0x7fec5402f320 sd=28 :45895 pgs=483 cs=905 l=0).fault, initiating
> > reconnect
> > 2013-01-07 04:45:30.851984 7fec637fe710 0 log [ERR] : map
> > e27 had wrong cluster addr (192.168.0.124:6808/19449 != my
> > 192.168.1.124:6808/19449)
> >
> > Also, this only happens only when the cluster ip address and the public ip address are different for example
> > ....
> > ....
> > ....
> > [osd.0]
> > host = g8ct
> > public address = 192.168.0.124
> > cluster address = 192.168.1.124
> > btrfs devs = /dev/sdb
> >
> > ....
> > ....
> >
> > but does not happen when they are the same. Any idea what may be the issue?
> This isn't familiar to me at first glance. What version of Ceph are you using?
>
> If
> this is easy to reproduce, can you pastebin your ceph.conf and then add
> "debug ms = 1" to your global config and gather up the logs from each
> daemon?
> -Greg
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
> More majordomo info at http://vger.kernel.org/majordomo
>
>
> Attachments:
> - ceph-osd.0.log.tar.gz
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
2013-01-24 21:28 ` Gregory Farnum
@ 2013-01-25 17:51 ` Isaac Otsiabah
2013-01-25 23:46 ` Sam Lang
2013-01-28 20:17 ` Isaac Otsiabah
0 siblings, 2 replies; 13+ messages in thread
From: Isaac Otsiabah @ 2013-01-25 17:51 UTC (permalink / raw)
To: Gregory Farnum; +Cc: ceph-devel
Gregory, the network physical layout is simple, the two networks are
separate. the 192.168.0 and the 192.168.1 are not subnets within a
network.
Isaac
----- Original Message -----
From: Gregory Farnum <greg@inktank.com>
To: Isaac Otsiabah <zmoo76b@yahoo.com>
Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Sent: Thursday, January 24, 2013 1:28 PM
Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
What's the physical layout of your networking? This additional log may prove helpful as well, but I really need a bit more context in evaluating the messages I see from the first one. :)
-Greg
On Thursday, January 24, 2013 at 9:24 AM, Isaac Otsiabah wrote:
>
>
> Gregory, i tried send the the attached debug output several times and
> the mail server rejected them all probably becauseof the file size so i cut the log file size down and it is attached. You will see the
> reconnection failures by the error message line below. The ceph version
> is 0.56
>
>
> it appears to be a timing issue because with the flag (debug ms=1) turned on, the system ran slower and became harder to fail.
> I
> ran it several times and finally got it to fail on (osd.0) using
> default crush map. The attached tar file contains log files for all
> components on g8ct plus the ceph.conf. By the way, the log file contain only the last 1384 lines where the error occurs.
>
>
> I started with a 1-node cluster on host g8ct (osd.0, osd.1, osd.2) and then added host g13ct (osd.3, osd.4, osd.5)
>
>
> id weight type name up/down reweight
> -1 6 root default
> -3 6 rack unknownrack
> -2 3 host g8ct
> 0 1 osd.0 down 1
> 1 1 osd.1 up 1
> 2 1 osd.2 up 1
> -4 3 host g13ct
> 3 1 osd.3 up 1
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
>
>
>
> The error messages are in ceph.log and ceph-osd.0.log:
>
> ceph.log:2013-01-08
> 05:41:38.080470 osd.0 192.168.0.124:6801/25571 3 : [ERR] map e15 had
> wrong cluster addr (192.168.0.124:6802/25571 != my
> 192.168.1.124:6802/25571)
> ceph-osd.0.log:2013-01-08 05:41:38.080458 7f06757fa710 0 log [ERR] : map e15 had wrong cluster addr
> (192.168.0.124:6802/25571 != my 192.168.1.124:6802/25571)
>
>
>
> [root@g8ct ceph]# ceph -v
> ceph version 0.56 (1a32f0a0b42f169a7b55ed48ec3208f6d4edc1e8)
>
>
> Isaac
>
>
> ----- Original Message -----
> From: Gregory Farnum <greg@inktank.com (mailto:greg@inktank.com)>
> To: Isaac Otsiabah <zmoo76b@yahoo.com (mailto:zmoo76b@yahoo.com)>
> Cc: "ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)" <ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)>
> Sent: Monday, January 7, 2013 1:27 PM
> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>
> On Monday, January 7, 2013 at 1:00 PM, Isaac Otsiabah wrote:
>
>
> When i add a new host (with osd's) to my existing cluster, 1 or 2
> previous osd(s) goes down for about 2 minutes and then they come back
> up.
> >
> >
> > [root@h1ct ~]# ceph osd tree
> >
> > # id weight type name up/down reweight
> > -1
> > 3 root default
> > -3 3 rack unknownrack
> > -2 3 host h1
> > 0 1 osd.0 up 1
> > 1 1 osd.1 up 1
> > 2
> > 1 osd.2 up 1
>
>
> For example, after adding host h2 (with 3 new osd) to the above cluster
> and running the "ceph osd tree" command, i see this:
> >
> >
> > [root@h1 ~]# ceph osd tree
> >
> > # id weight type name up/down reweight
> > -1 6 root default
> > -3
> > 6 rack unknownrack
> > -2 3 host h1
> > 0 1 osd.0 up 1
> > 1 1 osd.1 down 1
> > 2
> > 1 osd.2 up 1
> > -4 3 host h2
> > 3 1 osd.3 up 1
> > 4 1 osd.4 up
> > 1
> > 5 1 osd.5 up 1
>
>
> The down osd always come back up after 2 minutes or less andi see the
> following error message in the respective osd log file:
> > 2013-01-07 04:40:17.613028 7fec7f092760 1 journal _open
> > /ceph_journal/journals/journal_2 fd 26: 1073741824 bytes, block size
> > 4096 bytes, directio = 1, aio = 0
> > 2013-01-07 04:40:17.613122
> > 7fec7f092760 1 journal _open /ceph_journal/journals/journal_2 fd 26:
> > 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> > 2013-01-07
> > 04:42:10.006533 7fec746f7710 0 -- 192.168.0.124:6808/19449 >>
> > 192.168.1.123:6800/18287 pipe(0x7fec20000e10 sd=31 :6808 pgs=0 cs=0
> > l=0).accept connect_seq 0 vs existing 0 state connecting
> > 2013-01-07
> > 04:45:29.834341 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45438 pgs=7 cs=1
> > l=0).fault, initiating reconnect
> > 2013-01-07 04:45:29.835748
> > 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45439 pgs=15 cs=3
> > l=0).fault, initiating reconnect
> > 2013-01-07 04:45:30.835219 7fec743f4710 0 --
> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
> > pipe(0x7fec5402f320 sd=28 :45894 pgs=482 cs=903 l=0).fault, initiating
> > reconnect
> > 2013-01-07 04:45:30.837318 7fec743f4710 0 --
> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
> > pipe(0x7fec5402f320 sd=28 :45895 pgs=483 cs=905 l=0).fault, initiating
> > reconnect
> > 2013-01-07 04:45:30.851984 7fec637fe710 0 log [ERR] : map
> > e27 had wrong cluster addr (192.168.0.124:6808/19449 != my
> > 192.168.1.124:6808/19449)
> >
> > Also, this only happens only when the cluster ip address and the public ip address are different for example
> > ....
> > ....
> > ....
> > [osd.0]
> > host = g8ct
> > public address = 192.168.0.124
> > cluster address = 192.168.1.124
> > btrfs devs = /dev/sdb
> >
> > ....
> > ....
> >
> > but does not happen when they are the same. Any idea what may be the issue?
> This isn't familiar to me at first glance. What version of Ceph are you using?
>
> If
> this is easy to reproduce, can you pastebin your ceph.conf and then add
> "debug ms = 1" to your global config and gather up the logs from each
> daemon?
> -Greg
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
> More majordomo info at http://vger.kernel.org/majordomo
>
>
> Attachments:
> - ceph-osd.0.log.tar.gz
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
2013-01-25 17:51 ` Isaac Otsiabah
@ 2013-01-25 23:46 ` Sam Lang
2013-01-28 20:17 ` Isaac Otsiabah
1 sibling, 0 replies; 13+ messages in thread
From: Sam Lang @ 2013-01-25 23:46 UTC (permalink / raw)
To: Isaac Otsiabah; +Cc: Gregory Farnum, ceph-devel
On Fri, Jan 25, 2013 at 11:51 AM, Isaac Otsiabah <zmoo76b@yahoo.com> wrote:
>
>
> Gregory, the network physical layout is simple, the two networks are
> separate. the 192.168.0 and the 192.168.1 are not subnets within a
> network.
Hi Isaac,
Could you send us your routing tables on the osds (route -n). That's
one more bit of information that might be useful for tracking this
down.
Thanks,
-sam
>
> Isaac
>
>
>
>
> ----- Original Message -----
> From: Gregory Farnum <greg@inktank.com>
> To: Isaac Otsiabah <zmoo76b@yahoo.com>
> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
> Sent: Thursday, January 24, 2013 1:28 PM
> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>
> What's the physical layout of your networking? This additional log may prove helpful as well, but I really need a bit more context in evaluating the messages I see from the first one. :)
> -Greg
>
>
> On Thursday, January 24, 2013 at 9:24 AM, Isaac Otsiabah wrote:
>
>>
>>
>> Gregory, i tried send the the attached debug output several times and
>> the mail server rejected them all probably becauseof the file size so i cut the log file size down and it is attached. You will see the
>> reconnection failures by the error message line below. The ceph version
>> is 0.56
>>
>>
>> it appears to be a timing issue because with the flag (debug ms=1) turned on, the system ran slower and became harder to fail.
>> I
>> ran it several times and finally got it to fail on (osd.0) using
>> default crush map. The attached tar file contains log files for all
>> components on g8ct plus the ceph.conf. By the way, the log file contain only the last 1384 lines where the error occurs.
>>
>>
>> I started with a 1-node cluster on host g8ct (osd.0, osd.1, osd.2) and then added host g13ct (osd.3, osd.4, osd.5)
>>
>>
>> id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g8ct
>> 0 1 osd.0 down 1
>> 1 1 osd.1 up 1
>> 2 1 osd.2 up 1
>> -4 3 host g13ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>>
>>
>> The error messages are in ceph.log and ceph-osd.0.log:
>>
>> ceph.log:2013-01-08
>> 05:41:38.080470 osd.0 192.168.0.124:6801/25571 3 : [ERR] map e15 had
>> wrong cluster addr (192.168.0.124:6802/25571 != my
>> 192.168.1.124:6802/25571)
>> ceph-osd.0.log:2013-01-08 05:41:38.080458 7f06757fa710 0 log [ERR] : map e15 had wrong cluster addr
>> (192.168.0.124:6802/25571 != my 192.168.1.124:6802/25571)
>>
>>
>>
>> [root@g8ct ceph]# ceph -v
>> ceph version 0.56 (1a32f0a0b42f169a7b55ed48ec3208f6d4edc1e8)
>>
>>
>> Isaac
>>
>>
>> ----- Original Message -----
>> From: Gregory Farnum <greg@inktank.com (mailto:greg@inktank.com)>
>> To: Isaac Otsiabah <zmoo76b@yahoo.com (mailto:zmoo76b@yahoo.com)>
>> Cc: "ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)" <ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)>
>> Sent: Monday, January 7, 2013 1:27 PM
>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>
>> On Monday, January 7, 2013 at 1:00 PM, Isaac Otsiabah wrote:
>>
>>
>> When i add a new host (with osd's) to my existing cluster, 1 or 2
>> previous osd(s) goes down for about 2 minutes and then they come back
>> up.
>> >
>> >
>> > [root@h1ct ~]# ceph osd tree
>> >
>> > # id weight type name up/down reweight
>> > -1
>> > 3 root default
>> > -3 3 rack unknownrack
>> > -2 3 host h1
>> > 0 1 osd.0 up 1
>> > 1 1 osd.1 up 1
>> > 2
>> > 1 osd.2 up 1
>>
>>
>> For example, after adding host h2 (with 3 new osd) to the above cluster
>> and running the "ceph osd tree" command, i see this:
>> >
>> >
>> > [root@h1 ~]# ceph osd tree
>> >
>> > # id weight type name up/down reweight
>> > -1 6 root default
>> > -3
>> > 6 rack unknownrack
>> > -2 3 host h1
>> > 0 1 osd.0 up 1
>> > 1 1 osd.1 down 1
>> > 2
>> > 1 osd.2 up 1
>> > -4 3 host h2
>> > 3 1 osd.3 up 1
>> > 4 1 osd.4 up
>> > 1
>> > 5 1 osd.5 up 1
>>
>>
>> The down osd always come back up after 2 minutes or less andi see the
>> following error message in the respective osd log file:
>> > 2013-01-07 04:40:17.613028 7fec7f092760 1 journal _open
>> > /ceph_journal/journals/journal_2 fd 26: 1073741824 bytes, block size
>> > 4096 bytes, directio = 1, aio = 0
>> > 2013-01-07 04:40:17.613122
>> > 7fec7f092760 1 journal _open /ceph_journal/journals/journal_2 fd 26:
>> > 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
>> > 2013-01-07
>> > 04:42:10.006533 7fec746f7710 0 -- 192.168.0.124:6808/19449 >>
>> > 192.168.1.123:6800/18287 pipe(0x7fec20000e10 sd=31 :6808 pgs=0 cs=0
>> > l=0).accept connect_seq 0 vs existing 0 state connecting
>> > 2013-01-07
>> > 04:45:29.834341 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45438 pgs=7 cs=1
>> > l=0).fault, initiating reconnect
>> > 2013-01-07 04:45:29.835748
>> > 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45439 pgs=15 cs=3
>> > l=0).fault, initiating reconnect
>> > 2013-01-07 04:45:30.835219 7fec743f4710 0 --
>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>> > pipe(0x7fec5402f320 sd=28 :45894 pgs=482 cs=903 l=0).fault, initiating
>> > reconnect
>> > 2013-01-07 04:45:30.837318 7fec743f4710 0 --
>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>> > pipe(0x7fec5402f320 sd=28 :45895 pgs=483 cs=905 l=0).fault, initiating
>> > reconnect
>> > 2013-01-07 04:45:30.851984 7fec637fe710 0 log [ERR] : map
>> > e27 had wrong cluster addr (192.168.0.124:6808/19449 != my
>> > 192.168.1.124:6808/19449)
>> >
>> > Also, this only happens only when the cluster ip address and the public ip address are different for example
>> > ....
>> > ....
>> > ....
>> > [osd.0]
>> > host = g8ct
>> > public address = 192.168.0.124
>> > cluster address = 192.168.1.124
>> > btrfs devs = /dev/sdb
>> >
>> > ....
>> > ....
>> >
>> > but does not happen when they are the same. Any idea what may be the issue?
>> This isn't familiar to me at first glance. What version of Ceph are you using?
>>
>> If
>> this is easy to reproduce, can you pastebin your ceph.conf and then add
>> "debug ms = 1" to your global config and gather up the logs from each
>> daemon?
>> -Greg
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
>> More majordomo info at http://vger.kernel.org/majordomo
>>
>>
>> Attachments:
>> - ceph-osd.0.log.tar.gz
>>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
2013-01-25 17:51 ` Isaac Otsiabah
2013-01-25 23:46 ` Sam Lang
@ 2013-01-28 20:17 ` Isaac Otsiabah
2013-02-11 20:29 ` Gregory Farnum
1 sibling, 1 reply; 13+ messages in thread
From: Isaac Otsiabah @ 2013-01-28 20:17 UTC (permalink / raw)
To: Gregory Farnum; +Cc: ceph-devel
[-- Attachment #1: Type: text/plain, Size: 10982 bytes --]
Gregory, i recreated the osd down problem again this morning on two nodes (g13ct, g14ct). First, i created a 1-node cluster on g13ct (with osd.0, 1 ,2) and then added host g14ct (osd3. 4, 5). osd.1 went down for about 1 minute and half after adding osd 3, 4, 5 were adde4d. i have included the routing table of each node at the time osd.1 went down. ceph.conf and ceph-osd.1.log files are attached. The crush map was default. Also, it could be a timing issue because it does not always fail when using default crush map, it takes several trials before you see it. Thank you.
[root@g13ct ~]# netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
default 133.164.98.250 0.0.0.0 UG 0 0 0 eth2
133.164.98.0 * 255.255.255.0 U 0 0 0 eth2
link-local * 255.255.0.0 U 0 0 0 eth3
link-local * 255.255.0.0 U 0 0 0 eth0
link-local * 255.255.0.0 U 0 0 0 eth2
192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
192.0.0.0 * 255.0.0.0 U 0 0 0 eth0
192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
[root@g13ct ~]# ceph osd tree
# id weight type name up/down reweight
-1 6 root default
-3 6 rack unknownrack
-2 3 host g13ct
0 1 osd.0 up 1
1 1 osd.1 down 1
2 1 osd.2 up 1
-4 3 host g14ct
3 1 osd.3 up 1
4 1 osd.4 up 1
5 1 osd.5 up 1
[root@g14ct ~]# ceph osd tree
# id weight type name up/down reweight
-1 6 root default
-3 6 rack unknownrack
-2 3 host g13ct
0 1 osd.0 up 1
1 1 osd.1 down 1
2 1 osd.2 up 1
-4 3 host g14ct
3 1 osd.3 up 1
4 1 osd.4 up 1
5 1 osd.5 up 1
[root@g14ct ~]# netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
default 133.164.98.250 0.0.0.0 UG 0 0 0 eth0
133.164.98.0 * 255.255.255.0 U 0 0 0 eth0
link-local * 255.255.0.0 U 0 0 0 eth3
link-local * 255.255.0.0 U 0 0 0 eth5
link-local * 255.255.0.0 U 0 0 0 eth0
192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
192.0.0.0 * 255.0.0.0 U 0 0 0 eth5
192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
192.168.1.0 * 255.255.255.0 U 0 0 0 eth5
[root@g14ct ~]# ceph osd tree
# id weight type name up/down reweight
-1 6 root default
-3 6 rack unknownrack
-2 3 host g13ct
0 1 osd.0 up 1
1 1 osd.1 down 1
2 1 osd.2 up 1
-4 3 host g14ct
3 1 osd.3 up 1
4 1 osd.4 up 1
5 1 osd.5 up 1
Isaac
----- Original Message -----
From: Isaac Otsiabah <zmoo76b@yahoo.com>
To: Gregory Farnum <greg@inktank.com>
Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Sent: Friday, January 25, 2013 9:51 AM
Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
Gregory, the network physical layout is simple, the two networks are
separate. the 192.168.0 and the 192.168.1 are not subnets within a
network.
Isaac
----- Original Message -----
From: Gregory Farnum <greg@inktank.com>
To: Isaac Otsiabah <zmoo76b@yahoo.com>
Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Sent: Thursday, January 24, 2013 1:28 PM
Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
What's the physical layout of your networking? This additional log may prove helpful as well, but I really need a bit more context in evaluating the messages I see from the first one. :)
-Greg
On Thursday, January 24, 2013 at 9:24 AM, Isaac Otsiabah wrote:
>
>
> Gregory, i tried send the the attached debug output several times and
> the mail server rejected them all probably becauseof the file size so i cut the log file size down and it is attached. You will see the
> reconnection failures by the error message line below. The ceph version
> is 0.56
>
>
> it appears to be a timing issue because with the flag (debug ms=1) turned on, the system ran slower and became harder to fail.
> I
> ran it several times and finally got it to fail on (osd.0) using
> default crush map. The attached tar file contains log files for all
> components on g8ct plus the ceph.conf. By the way, the log file contain only the last 1384 lines where the error occurs.
>
>
> I started with a 1-node cluster on host g8ct (osd.0, osd.1, osd.2) and then added host g13ct (osd.3, osd.4, osd.5)
>
>
> id weight type name up/down reweight
> -1 6 root default
> -3 6 rack unknownrack
> -2 3 host g8ct
> 0 1 osd.0 down 1
> 1 1 osd.1 up 1
> 2 1 osd.2 up 1
> -4 3 host g13ct
> 3 1 osd.3 up 1
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
>
>
>
> The error messages are in ceph.log and ceph-osd.0.log:
>
> ceph.log:2013-01-08
> 05:41:38.080470 osd.0 192.168.0.124:6801/25571 3 : [ERR] map e15 had
> wrong cluster addr (192.168.0.124:6802/25571 != my
> 192.168.1.124:6802/25571)
> ceph-osd.0.log:2013-01-08 05:41:38.080458 7f06757fa710 0 log [ERR] : map e15 had wrong cluster addr
> (192.168.0.124:6802/25571 != my 192.168.1.124:6802/25571)
>
>
>
> [root@g8ct ceph]# ceph -v
> ceph version 0.56 (1a32f0a0b42f169a7b55ed48ec3208f6d4edc1e8)
>
>
> Isaac
>
>
> ----- Original Message -----
> From: Gregory Farnum <greg@inktank.com (mailto:greg@inktank.com)>
> To: Isaac Otsiabah <zmoo76b@yahoo.com (mailto:zmoo76b@yahoo.com)>
> Cc: "ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)" <ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)>
> Sent: Monday, January 7, 2013 1:27 PM
> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>
> On Monday, January 7, 2013 at 1:00 PM, Isaac Otsiabah wrote:
>
>
> When i add a new host (with osd's) to my existing cluster, 1 or 2
> previous osd(s) goes down for about 2 minutes and then they come back
> up.
> >
> >
> > [root@h1ct ~]# ceph osd tree
> >
> > # id weight type name up/down reweight
> > -1
> > 3 root default
> > -3 3 rack unknownrack
> > -2 3 host h1
> > 0 1 osd.0 up 1
> > 1 1 osd.1 up 1
> > 2
> > 1 osd.2 up 1
>
>
> For example, after adding host h2 (with 3 new osd) to the above cluster
> and running the "ceph osd tree" command, i see this:
> >
> >
> > [root@h1 ~]# ceph osd tree
> >
> > # id weight type name up/down reweight
> > -1 6 root default
> > -3
> > 6 rack unknownrack
> > -2 3 host h1
> > 0 1 osd.0 up 1
> > 1 1 osd.1 down 1
> > 2
> > 1 osd.2 up 1
> > -4 3 host h2
> > 3 1 osd.3 up 1
> > 4 1 osd.4 up
> > 1
> > 5 1 osd.5 up 1
>
>
> The down osd always come back up after 2 minutes or less andi see the
> following error message in the respective osd log file:
> > 2013-01-07 04:40:17.613028 7fec7f092760 1 journal _open
> > /ceph_journal/journals/journal_2 fd 26: 1073741824 bytes, block size
> > 4096 bytes, directio = 1, aio = 0
> > 2013-01-07 04:40:17.613122
> > 7fec7f092760 1 journal _open /ceph_journal/journals/journal_2 fd 26:
> > 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> > 2013-01-07
> > 04:42:10.006533 7fec746f7710 0 -- 192.168.0.124:6808/19449 >>
> > 192.168.1.123:6800/18287 pipe(0x7fec20000e10 sd=31 :6808 pgs=0 cs=0
> > l=0).accept connect_seq 0 vs existing 0 state connecting
> > 2013-01-07
> > 04:45:29.834341 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45438 pgs=7 cs=1
> > l=0).fault, initiating reconnect
> > 2013-01-07 04:45:29.835748
> > 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45439 pgs=15 cs=3
> > l=0).fault, initiating reconnect
> > 2013-01-07 04:45:30.835219 7fec743f4710 0 --
> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
> > pipe(0x7fec5402f320 sd=28 :45894 pgs=482 cs=903 l=0).fault, initiating
> > reconnect
> > 2013-01-07 04:45:30.837318 7fec743f4710 0 --
> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
> > pipe(0x7fec5402f320 sd=28 :45895 pgs=483 cs=905 l=0).fault, initiating
> > reconnect
> > 2013-01-07 04:45:30.851984 7fec637fe710 0 log [ERR] : map
> > e27 had wrong cluster addr (192.168.0.124:6808/19449 != my
> > 192.168.1.124:6808/19449)
> >
> > Also, this only happens only when the cluster ip address and the public ip address are different for example
> > ....
> > ....
> > ....
> > [osd.0]
> > host = g8ct
> > public address = 192.168.0.124
> > cluster address = 192.168.1.124
> > btrfs devs = /dev/sdb
> >
> > ....
> > ....
> >
> > but does not happen when they are the same. Any idea what may be the issue?
> This isn't familiar to me at first glance. What version of Ceph are you using?
>
> If
> this is easy to reproduce, can you pastebin your ceph.conf and then add
> "debug ms = 1" to your global config and gather up the logs from each
> daemon?
> -Greg
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
> More majordomo info at http://vger.kernel.org/majordomo
>
>
> Attachments:
> - ceph-osd.0.log.tar.gz
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: osd.1.tar.gz --]
[-- Type: application/x-gzip, Size: 11360 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
2013-01-28 20:17 ` Isaac Otsiabah
@ 2013-02-11 20:29 ` Gregory Farnum
2013-02-12 1:39 ` Isaac Otsiabah
2013-05-21 20:21 ` ceph-deploy errors on CentOS Isaac Otsiabah
0 siblings, 2 replies; 13+ messages in thread
From: Gregory Farnum @ 2013-02-11 20:29 UTC (permalink / raw)
To: Isaac Otsiabah; +Cc: ceph-devel
jIsaac,
I'm sorry I haven't been able to wrangle any time to look into this
more yet, but Sage pointed out in a related thread that there might be
some buggy handling of things like this if the OSD and the monitor are
located on the same host. Am I correct in assuming that with your
small cluster, all your OSDs are co-located with a monitor daemon?
-Greg
On Mon, Jan 28, 2013 at 12:17 PM, Isaac Otsiabah <zmoo76b@yahoo.com> wrote:
>
>
> Gregory, i recreated the osd down problem again this morning on two nodes (g13ct, g14ct). First, i created a 1-node cluster on g13ct (with osd.0, 1 ,2) and then added host g14ct (osd3. 4, 5). osd.1 went down for about 1 minute and half after adding osd 3, 4, 5 were adde4d. i have included the routing table of each node at the time osd.1 went down. ceph.conf and ceph-osd.1.log files are attached. The crush map was default. Also, it could be a timing issue because it does not always fail when using default crush map, it takes several trials before you see it. Thank you.
>
>
> [root@g13ct ~]# netstat -r
> Kernel IP routing table
> Destination Gateway Genmask Flags MSS Window irtt Iface
> default 133.164.98.250 0.0.0.0 UG 0 0 0 eth2
> 133.164.98.0 * 255.255.255.0 U 0 0 0 eth2
> link-local * 255.255.0.0 U 0 0 0 eth3
> link-local * 255.255.0.0 U 0 0 0 eth0
> link-local * 255.255.0.0 U 0 0 0 eth2
> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth0
> 192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
> 192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
> [root@g13ct ~]# ceph osd tree
>
> # id weight type name up/down reweight
> -1 6 root default
> -3 6 rack unknownrack
> -2 3 host g13ct
> 0 1 osd.0 up 1
> 1 1 osd.1 down 1
> 2 1 osd.2 up 1
> -4 3 host g14ct
> 3 1 osd.3 up 1
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
>
>
>
> [root@g14ct ~]# ceph osd tree
>
> # id weight type name up/down reweight
> -1 6 root default
> -3 6 rack unknownrack
> -2 3 host g13ct
> 0 1 osd.0 up 1
> 1 1 osd.1 down 1
> 2 1 osd.2 up 1
> -4 3 host g14ct
> 3 1 osd.3 up 1
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
>
> [root@g14ct ~]# netstat -r
> Kernel IP routing table
> Destination Gateway Genmask Flags MSS Window irtt Iface
> default 133.164.98.250 0.0.0.0 UG 0 0 0 eth0
> 133.164.98.0 * 255.255.255.0 U 0 0 0 eth0
> link-local * 255.255.0.0 U 0 0 0 eth3
> link-local * 255.255.0.0 U 0 0 0 eth5
> link-local * 255.255.0.0 U 0 0 0 eth0
> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth5
> 192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
> 192.168.1.0 * 255.255.255.0 U 0 0 0 eth5
> [root@g14ct ~]# ceph osd tree
>
> # id weight type name up/down reweight
> -1 6 root default
> -3 6 rack unknownrack
> -2 3 host g13ct
> 0 1 osd.0 up 1
> 1 1 osd.1 down 1
> 2 1 osd.2 up 1
> -4 3 host g14ct
> 3 1 osd.3 up 1
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
>
>
>
>
>
> Isaac
>
>
>
>
>
>
>
>
>
>
> ----- Original Message -----
> From: Isaac Otsiabah <zmoo76b@yahoo.com>
> To: Gregory Farnum <greg@inktank.com>
> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
> Sent: Friday, January 25, 2013 9:51 AM
> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>
>
>
> Gregory, the network physical layout is simple, the two networks are
> separate. the 192.168.0 and the 192.168.1 are not subnets within a
> network.
>
> Isaac
>
>
>
>
> ----- Original Message -----
> From: Gregory Farnum <greg@inktank.com>
> To: Isaac Otsiabah <zmoo76b@yahoo.com>
> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
> Sent: Thursday, January 24, 2013 1:28 PM
> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>
> What's the physical layout of your networking? This additional log may prove helpful as well, but I really need a bit more context in evaluating the messages I see from the first one. :)
> -Greg
>
>
> On Thursday, January 24, 2013 at 9:24 AM, Isaac Otsiabah wrote:
>
>>
>>
>> Gregory, i tried send the the attached debug output several times and
>> the mail server rejected them all probably becauseof the file size so i cut the log file size down and it is attached. You will see the
>> reconnection failures by the error message line below. The ceph version
>> is 0.56
>>
>>
>> it appears to be a timing issue because with the flag (debug ms=1) turned on, the system ran slower and became harder to fail.
>> I
>> ran it several times and finally got it to fail on (osd.0) using
>> default crush map. The attached tar file contains log files for all
>> components on g8ct plus the ceph.conf. By the way, the log file contain only the last 1384 lines where the error occurs.
>>
>>
>> I started with a 1-node cluster on host g8ct (osd.0, osd.1, osd.2) and then added host g13ct (osd.3, osd.4, osd.5)
>>
>>
>> id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g8ct
>> 0 1 osd.0 down 1
>> 1 1 osd.1 up 1
>> 2 1 osd.2 up 1
>> -4 3 host g13ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>>
>>
>> The error messages are in ceph.log and ceph-osd.0.log:
>>
>> ceph.log:2013-01-08
>> 05:41:38.080470 osd.0 192.168.0.124:6801/25571 3 : [ERR] map e15 had
>> wrong cluster addr (192.168.0.124:6802/25571 != my
>> 192.168.1.124:6802/25571)
>> ceph-osd.0.log:2013-01-08 05:41:38.080458 7f06757fa710 0 log [ERR] : map e15 had wrong cluster addr
>> (192.168.0.124:6802/25571 != my 192.168.1.124:6802/25571)
>>
>>
>>
>> [root@g8ct ceph]# ceph -v
>> ceph version 0.56 (1a32f0a0b42f169a7b55ed48ec3208f6d4edc1e8)
>>
>>
>> Isaac
>>
>>
>> ----- Original Message -----
>> From: Gregory Farnum <greg@inktank.com (mailto:greg@inktank.com)>
>> To: Isaac Otsiabah <zmoo76b@yahoo.com (mailto:zmoo76b@yahoo.com)>
>> Cc: "ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)" <ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)>
>> Sent: Monday, January 7, 2013 1:27 PM
>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>
>> On Monday, January 7, 2013 at 1:00 PM, Isaac Otsiabah wrote:
>>
>>
>> When i add a new host (with osd's) to my existing cluster, 1 or 2
>> previous osd(s) goes down for about 2 minutes and then they come back
>> up.
>> >
>> >
>> > [root@h1ct ~]# ceph osd tree
>> >
>> > # id weight type name up/down reweight
>> > -1
>> > 3 root default
>> > -3 3 rack unknownrack
>> > -2 3 host h1
>> > 0 1 osd.0 up 1
>> > 1 1 osd.1 up 1
>> > 2
>> > 1 osd.2 up 1
>>
>>
>> For example, after adding host h2 (with 3 new osd) to the above cluster
>> and running the "ceph osd tree" command, i see this:
>> >
>> >
>> > [root@h1 ~]# ceph osd tree
>> >
>> > # id weight type name up/down reweight
>> > -1 6 root default
>> > -3
>> > 6 rack unknownrack
>> > -2 3 host h1
>> > 0 1 osd.0 up 1
>> > 1 1 osd.1 down 1
>> > 2
>> > 1 osd.2 up 1
>> > -4 3 host h2
>> > 3 1 osd.3 up 1
>> > 4 1 osd.4 up
>> > 1
>> > 5 1 osd.5 up 1
>>
>>
>> The down osd always come back up after 2 minutes or less andi see the
>> following error message in the respective osd log file:
>> > 2013-01-07 04:40:17.613028 7fec7f092760 1 journal _open
>> > /ceph_journal/journals/journal_2 fd 26: 1073741824 bytes, block size
>> > 4096 bytes, directio = 1, aio = 0
>> > 2013-01-07 04:40:17.613122
>> > 7fec7f092760 1 journal _open /ceph_journal/journals/journal_2 fd 26:
>> > 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
>> > 2013-01-07
>> > 04:42:10.006533 7fec746f7710 0 -- 192.168.0.124:6808/19449 >>
>> > 192.168.1.123:6800/18287 pipe(0x7fec20000e10 sd=31 :6808 pgs=0 cs=0
>> > l=0).accept connect_seq 0 vs existing 0 state connecting
>> > 2013-01-07
>> > 04:45:29.834341 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45438 pgs=7 cs=1
>> > l=0).fault, initiating reconnect
>> > 2013-01-07 04:45:29.835748
>> > 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45439 pgs=15 cs=3
>> > l=0).fault, initiating reconnect
>> > 2013-01-07 04:45:30.835219 7fec743f4710 0 --
>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>> > pipe(0x7fec5402f320 sd=28 :45894 pgs=482 cs=903 l=0).fault, initiating
>> > reconnect
>> > 2013-01-07 04:45:30.837318 7fec743f4710 0 --
>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>> > pipe(0x7fec5402f320 sd=28 :45895 pgs=483 cs=905 l=0).fault, initiating
>> > reconnect
>> > 2013-01-07 04:45:30.851984 7fec637fe710 0 log [ERR] : map
>> > e27 had wrong cluster addr (192.168.0.124:6808/19449 != my
>> > 192.168.1.124:6808/19449)
>> >
>> > Also, this only happens only when the cluster ip address and the public ip address are different for example
>> > ....
>> > ....
>> > ....
>> > [osd.0]
>> > host = g8ct
>> > public address = 192.168.0.124
>> > cluster address = 192.168.1.124
>> > btrfs devs = /dev/sdb
>> >
>> > ....
>> > ....
>> >
>> > but does not happen when they are the same. Any idea what may be the issue?
>> This isn't familiar to me at first glance. What version of Ceph are you using?
>>
>> If
>> this is easy to reproduce, can you pastebin your ceph.conf and then add
>> "debug ms = 1" to your global config and gather up the logs from each
>> daemon?
>> -Greg
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
>> More majordomo info at http://vger.kernel.org/majordomo
>>
>>
>> Attachments:
>> - ceph-osd.0.log.tar.gz
>>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Jan 28, 2013 at 12:17 PM, Isaac Otsiabah <zmoo76b@yahoo.com> wrote:
>
>
> Gregory, i recreated the osd down problem again this morning on two nodes (g13ct, g14ct). First, i created a 1-node cluster on g13ct (with osd.0, 1 ,2) and then added host g14ct (osd3. 4, 5). osd.1 went down for about 1 minute and half after adding osd 3, 4, 5 were adde4d. i have included the routing table of each node at the time osd.1 went down. ceph.conf and ceph-osd.1.log files are attached. The crush map was default. Also, it could be a timing issue because it does not always fail when using default crush map, it takes several trials before you see it. Thank you.
>
>
> [root@g13ct ~]# netstat -r
> Kernel IP routing table
> Destination Gateway Genmask Flags MSS Window irtt Iface
> default 133.164.98.250 0.0.0.0 UG 0 0 0 eth2
> 133.164.98.0 * 255.255.255.0 U 0 0 0 eth2
> link-local * 255.255.0.0 U 0 0 0 eth3
> link-local * 255.255.0.0 U 0 0 0 eth0
> link-local * 255.255.0.0 U 0 0 0 eth2
> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth0
> 192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
> 192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
> [root@g13ct ~]# ceph osd tree
>
> # id weight type name up/down reweight
> -1 6 root default
> -3 6 rack unknownrack
> -2 3 host g13ct
> 0 1 osd.0 up 1
> 1 1 osd.1 down 1
> 2 1 osd.2 up 1
> -4 3 host g14ct
> 3 1 osd.3 up 1
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
>
>
>
> [root@g14ct ~]# ceph osd tree
>
> # id weight type name up/down reweight
> -1 6 root default
> -3 6 rack unknownrack
> -2 3 host g13ct
> 0 1 osd.0 up 1
> 1 1 osd.1 down 1
> 2 1 osd.2 up 1
> -4 3 host g14ct
> 3 1 osd.3 up 1
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
>
> [root@g14ct ~]# netstat -r
> Kernel IP routing table
> Destination Gateway Genmask Flags MSS Window irtt Iface
> default 133.164.98.250 0.0.0.0 UG 0 0 0 eth0
> 133.164.98.0 * 255.255.255.0 U 0 0 0 eth0
> link-local * 255.255.0.0 U 0 0 0 eth3
> link-local * 255.255.0.0 U 0 0 0 eth5
> link-local * 255.255.0.0 U 0 0 0 eth0
> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth5
> 192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
> 192.168.1.0 * 255.255.255.0 U 0 0 0 eth5
> [root@g14ct ~]# ceph osd tree
>
> # id weight type name up/down reweight
> -1 6 root default
> -3 6 rack unknownrack
> -2 3 host g13ct
> 0 1 osd.0 up 1
> 1 1 osd.1 down 1
> 2 1 osd.2 up 1
> -4 3 host g14ct
> 3 1 osd.3 up 1
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
>
>
>
>
>
> Isaac
>
>
>
>
>
>
>
>
>
>
> ----- Original Message -----
> From: Isaac Otsiabah <zmoo76b@yahoo.com>
> To: Gregory Farnum <greg@inktank.com>
> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
> Sent: Friday, January 25, 2013 9:51 AM
> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>
>
>
> Gregory, the network physical layout is simple, the two networks are
> separate. the 192.168.0 and the 192.168.1 are not subnets within a
> network.
>
> Isaac
>
>
>
>
> ----- Original Message -----
> From: Gregory Farnum <greg@inktank.com>
> To: Isaac Otsiabah <zmoo76b@yahoo.com>
> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
> Sent: Thursday, January 24, 2013 1:28 PM
> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>
> What's the physical layout of your networking? This additional log may prove helpful as well, but I really need a bit more context in evaluating the messages I see from the first one. :)
> -Greg
>
>
> On Thursday, January 24, 2013 at 9:24 AM, Isaac Otsiabah wrote:
>
>>
>>
>> Gregory, i tried send the the attached debug output several times and
>> the mail server rejected them all probably becauseof the file size so i cut the log file size down and it is attached. You will see the
>> reconnection failures by the error message line below. The ceph version
>> is 0.56
>>
>>
>> it appears to be a timing issue because with the flag (debug ms=1) turned on, the system ran slower and became harder to fail.
>> I
>> ran it several times and finally got it to fail on (osd.0) using
>> default crush map. The attached tar file contains log files for all
>> components on g8ct plus the ceph.conf. By the way, the log file contain only the last 1384 lines where the error occurs.
>>
>>
>> I started with a 1-node cluster on host g8ct (osd.0, osd.1, osd.2) and then added host g13ct (osd.3, osd.4, osd.5)
>>
>>
>> id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g8ct
>> 0 1 osd.0 down 1
>> 1 1 osd.1 up 1
>> 2 1 osd.2 up 1
>> -4 3 host g13ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>>
>>
>> The error messages are in ceph.log and ceph-osd.0.log:
>>
>> ceph.log:2013-01-08
>> 05:41:38.080470 osd.0 192.168.0.124:6801/25571 3 : [ERR] map e15 had
>> wrong cluster addr (192.168.0.124:6802/25571 != my
>> 192.168.1.124:6802/25571)
>> ceph-osd.0.log:2013-01-08 05:41:38.080458 7f06757fa710 0 log [ERR] : map e15 had wrong cluster addr
>> (192.168.0.124:6802/25571 != my 192.168.1.124:6802/25571)
>>
>>
>>
>> [root@g8ct ceph]# ceph -v
>> ceph version 0.56 (1a32f0a0b42f169a7b55ed48ec3208f6d4edc1e8)
>>
>>
>> Isaac
>>
>>
>> ----- Original Message -----
>> From: Gregory Farnum <greg@inktank.com (mailto:greg@inktank.com)>
>> To: Isaac Otsiabah <zmoo76b@yahoo.com (mailto:zmoo76b@yahoo.com)>
>> Cc: "ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)" <ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)>
>> Sent: Monday, January 7, 2013 1:27 PM
>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>
>> On Monday, January 7, 2013 at 1:00 PM, Isaac Otsiabah wrote:
>>
>>
>> When i add a new host (with osd's) to my existing cluster, 1 or 2
>> previous osd(s) goes down for about 2 minutes and then they come back
>> up.
>> >
>> >
>> > [root@h1ct ~]# ceph osd tree
>> >
>> > # id weight type name up/down reweight
>> > -1
>> > 3 root default
>> > -3 3 rack unknownrack
>> > -2 3 host h1
>> > 0 1 osd.0 up 1
>> > 1 1 osd.1 up 1
>> > 2
>> > 1 osd.2 up 1
>>
>>
>> For example, after adding host h2 (with 3 new osd) to the above cluster
>> and running the "ceph osd tree" command, i see this:
>> >
>> >
>> > [root@h1 ~]# ceph osd tree
>> >
>> > # id weight type name up/down reweight
>> > -1 6 root default
>> > -3
>> > 6 rack unknownrack
>> > -2 3 host h1
>> > 0 1 osd.0 up 1
>> > 1 1 osd.1 down 1
>> > 2
>> > 1 osd.2 up 1
>> > -4 3 host h2
>> > 3 1 osd.3 up 1
>> > 4 1 osd.4 up
>> > 1
>> > 5 1 osd.5 up 1
>>
>>
>> The down osd always come back up after 2 minutes or less andi see the
>> following error message in the respective osd log file:
>> > 2013-01-07 04:40:17.613028 7fec7f092760 1 journal _open
>> > /ceph_journal/journals/journal_2 fd 26: 1073741824 bytes, block size
>> > 4096 bytes, directio = 1, aio = 0
>> > 2013-01-07 04:40:17.613122
>> > 7fec7f092760 1 journal _open /ceph_journal/journals/journal_2 fd 26:
>> > 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
>> > 2013-01-07
>> > 04:42:10.006533 7fec746f7710 0 -- 192.168.0.124:6808/19449 >>
>> > 192.168.1.123:6800/18287 pipe(0x7fec20000e10 sd=31 :6808 pgs=0 cs=0
>> > l=0).accept connect_seq 0 vs existing 0 state connecting
>> > 2013-01-07
>> > 04:45:29.834341 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45438 pgs=7 cs=1
>> > l=0).fault, initiating reconnect
>> > 2013-01-07 04:45:29.835748
>> > 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45439 pgs=15 cs=3
>> > l=0).fault, initiating reconnect
>> > 2013-01-07 04:45:30.835219 7fec743f4710 0 --
>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>> > pipe(0x7fec5402f320 sd=28 :45894 pgs=482 cs=903 l=0).fault, initiating
>> > reconnect
>> > 2013-01-07 04:45:30.837318 7fec743f4710 0 --
>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>> > pipe(0x7fec5402f320 sd=28 :45895 pgs=483 cs=905 l=0).fault, initiating
>> > reconnect
>> > 2013-01-07 04:45:30.851984 7fec637fe710 0 log [ERR] : map
>> > e27 had wrong cluster addr (192.168.0.124:6808/19449 != my
>> > 192.168.1.124:6808/19449)
>> >
>> > Also, this only happens only when the cluster ip address and the public ip address are different for example
>> > ....
>> > ....
>> > ....
>> > [osd.0]
>> > host = g8ct
>> > public address = 192.168.0.124
>> > cluster address = 192.168.1.124
>> > btrfs devs = /dev/sdb
>> >
>> > ....
>> > ....
>> >
>> > but does not happen when they are the same. Any idea what may be the issue?
>> This isn't familiar to me at first glance. What version of Ceph are you using?
>>
>> If
>> this is easy to reproduce, can you pastebin your ceph.conf and then add
>> "debug ms = 1" to your global config and gather up the logs from each
>> daemon?
>> -Greg
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
>> More majordomo info at http://vger.kernel.org/majordomo
>>
>>
>> Attachments:
>> - ceph-osd.0.log.tar.gz
>>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
2013-02-11 20:29 ` Gregory Farnum
@ 2013-02-12 1:39 ` Isaac Otsiabah
2013-02-15 17:20 ` Sam Lang
2013-05-21 20:21 ` ceph-deploy errors on CentOS Isaac Otsiabah
1 sibling, 1 reply; 13+ messages in thread
From: Isaac Otsiabah @ 2013-02-12 1:39 UTC (permalink / raw)
To: Gregory Farnum; +Cc: ceph-devel
Yes, there were osd daemons running on the same node that the monitor was
running on. If that is the case then i will run a test case with the
monitor running on a different node where no osd is running and see what happens. Thank you.
Isaac
________________________________
From: Gregory Farnum <greg@inktank.com>
To: Isaac Otsiabah <zmoo76b@yahoo.com>
Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Sent: Monday, February 11, 2013 12:29 PM
Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
jIsaac,
I'm sorry I haven't been able to wrangle any time to look into this
more yet, but Sage pointed out in a related thread that there might be
some buggy handling of things like this if the OSD and the monitor are
located on the same host. Am I correct in assuming that with your
small cluster, all your OSDs are co-located with a monitor daemon?
-Greg
On Mon, Jan 28, 2013 at 12:17 PM, Isaac Otsiabah <zmoo76b@yahoo.com> wrote:
>
>
> Gregory, i recreated the osd down problem again this morning on two nodes (g13ct, g14ct). First, i created a 1-node cluster on g13ct (with osd.0, 1 ,2) and then added host g14ct (osd3. 4, 5). osd.1 went down for about 1 minute and half after adding osd 3, 4, 5 were adde4d. i have included the routing table of each node at the time osd.1 went down. ceph.conf and ceph-osd.1.log files are attached. The crush map was default. Also, it could be a timing issue because it does not always fail when using default crush map, it takes several trials before you see it. Thank you.
>
>
> [root@g13ct ~]# netstat -r
> Kernel IP routing table
> Destination Gateway Genmask Flags MSS Window irtt Iface
> default 133.164.98.250 0.0.0.0 UG 0 0 0 eth2
> 133.164.98.0 * 255.255.255.0 U 0 0 0 eth2
> link-local * 255.255.0.0 U 0 0 0 eth3
> link-local * 255.255.0.0 U 0 0 0 eth0
> link-local * 255.255.0.0 U 0 0 0 eth2
> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth0
> 192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
> 192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
> [root@g13ct ~]# ceph osd tree
>
> # id weight type name up/down reweight
> -1 6 root default
> -3 6 rack unknownrack
> -2 3 host g13ct
> 0 1 osd.0 up 1
> 1 1 osd.1 down 1
> 2 1 osd.2 up 1
> -4 3 host g14ct
> 3 1 osd.3 up 1
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
>
>
>
> [root@g14ct ~]# ceph osd tree
>
> # id weight type name up/down reweight
> -1 6 root default
> -3 6 rack unknownrack
> -2 3 host g13ct
> 0 1 osd.0 up 1
> 1 1 osd.1 down 1
> 2 1 osd.2 up 1
> -4 3 host g14ct
> 3 1 osd.3 up 1
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
>
> [root@g14ct ~]# netstat -r
> Kernel IP routing table
> Destination Gateway Genmask Flags MSS Window irtt Iface
> default 133.164.98.250 0.0.0.0 UG 0 0 0 eth0
> 133.164.98.0 * 255.255.255.0 U 0 0 0 eth0
> link-local * 255.255.0.0 U 0 0 0 eth3
> link-local * 255.255.0.0 U 0 0 0 eth5
> link-local * 255.255.0.0 U 0 0 0 eth0
> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth5
> 192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
> 192.168.1.0 * 255.255.255.0 U 0 0 0 eth5
> [root@g14ct ~]# ceph osd tree
>
> # id weight type name up/down reweight
> -1 6 root default
> -3 6 rack unknownrack
> -2 3 host g13ct
> 0 1 osd.0 up 1
> 1 1 osd.1 down 1
> 2 1 osd.2 up 1
> -4 3 host g14ct
> 3 1 osd.3 up 1
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
>
>
>
>
>
> Isaac
>
>
>
>
>
>
>
>
>
>
> ----- Original Message -----
> From: Isaac Otsiabah <zmoo76b@yahoo.com>
> To: Gregory Farnum <greg@inktank.com>
> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
> Sent: Friday, January 25, 2013 9:51 AM
> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>
>
>
> Gregory, the network physical layout is simple, the two networks are
> separate. the 192.168.0 and the 192.168.1 are not subnets within a
> network.
>
> Isaac
>
>
>
>
> ----- Original Message -----
> From: Gregory Farnum <greg@inktank.com>
> To: Isaac Otsiabah <zmoo76b@yahoo.com>
> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
> Sent: Thursday, January 24, 2013 1:28 PM
> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>
> What's the physical layout of your networking? This additional log may prove helpful as well, but I really need a bit more context in evaluating the messages I see from the first one. :)
> -Greg
>
>
> On Thursday, January 24, 2013 at 9:24 AM, Isaac Otsiabah wrote:
>
>>
>>
>> Gregory, i tried send the the attached debug output several times and
>> the mail server rejected them all probably becauseof the file size so i cut the log file size down and it is attached. You will see the
>> reconnection failures by the error message line below. The ceph version
>> is 0.56
>>
>>
>> it appears to be a timing issue because with the flag (debug ms=1) turned on, the system ran slower and became harder to fail.
>> I
>> ran it several times and finally got it to fail on (osd.0) using
>> default crush map. The attached tar file contains log files for all
>> components on g8ct plus the ceph.conf. By the way, the log file contain only the last 1384 lines where the error occurs.
>>
>>
>> I started with a 1-node cluster on host g8ct (osd.0, osd.1, osd.2) and then added host g13ct (osd.3, osd.4, osd.5)
>>
>>
>> id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g8ct
>> 0 1 osd.0 down 1
>> 1 1 osd.1 up 1
>> 2 1 osd.2 up 1
>> -4 3 host g13ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>>
>>
>> The error messages are in ceph.log and ceph-osd.0.log:
>>
>> ceph.log:2013-01-08
>> 05:41:38.080470 osd.0 192.168.0.124:6801/25571 3 : [ERR] map e15 had
>> wrong cluster addr (192.168.0.124:6802/25571 != my
>> 192.168.1.124:6802/25571)
>> ceph-osd.0.log:2013-01-08 05:41:38.080458 7f06757fa710 0 log [ERR] : map e15 had wrong cluster addr
>> (192.168.0.124:6802/25571 != my 192.168.1.124:6802/25571)
>>
>>
>>
>> [root@g8ct ceph]# ceph -v
>> ceph version 0.56 (1a32f0a0b42f169a7b55ed48ec3208f6d4edc1e8)
>>
>>
>> Isaac
>>
>>
>> ----- Original Message -----
>> From: Gregory Farnum <greg@inktank.com (mailto:greg@inktank.com)>
>> To: Isaac Otsiabah <zmoo76b@yahoo.com (mailto:zmoo76b@yahoo.com)>
>> Cc: "ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)" <ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)>
>> Sent: Monday, January 7, 2013 1:27 PM
>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>
>> On Monday, January 7, 2013 at 1:00 PM, Isaac Otsiabah wrote:
>>
>>
>> When i add a new host (with osd's) to my existing cluster, 1 or 2
>> previous osd(s) goes down for about 2 minutes and then they come back
>> up.
>> >
>> >
>> > [root@h1ct ~]# ceph osd tree
>> >
>> > # id weight type name up/down reweight
>> > -1
>> > 3 root default
>> > -3 3 rack unknownrack
>> > -2 3 host h1
>> > 0 1 osd.0 up 1
>> > 1 1 osd.1 up 1
>> > 2
>> > 1 osd.2 up 1
>>
>>
>> For example, after adding host h2 (with 3 new osd) to the above cluster
>> and running the "ceph osd tree" command, i see this:
>> >
>> >
>> > [root@h1 ~]# ceph osd tree
>> >
>> > # id weight type name up/down reweight
>> > -1 6 root default
>> > -3
>> > 6 rack unknownrack
>> > -2 3 host h1
>> > 0 1 osd.0 up 1
>> > 1 1 osd.1 down 1
>> > 2
>> > 1 osd.2 up 1
>> > -4 3 host h2
>> > 3 1 osd.3 up 1
>> > 4 1 osd.4 up
>> > 1
>> > 5 1 osd.5 up 1
>>
>>
>> The down osd always come back up after 2 minutes or less andi see the
>> following error message in the respective osd log file:
>> > 2013-01-07 04:40:17.613028 7fec7f092760 1 journal _open
>> > /ceph_journal/journals/journal_2 fd 26: 1073741824 bytes, block size
>> > 4096 bytes, directio = 1, aio = 0
>> > 2013-01-07 04:40:17.613122
>> > 7fec7f092760 1 journal _open /ceph_journal/journals/journal_2 fd 26:
>> > 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
>> > 2013-01-07
>> > 04:42:10.006533 7fec746f7710 0 -- 192.168.0.124:6808/19449 >>
>> > 192.168.1.123:6800/18287 pipe(0x7fec20000e10 sd=31 :6808 pgs=0 cs=0
>> > l=0).accept connect_seq 0 vs existing 0 state connecting
>> > 2013-01-07
>> > 04:45:29.834341 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45438 pgs=7 cs=1
>> > l=0).fault, initiating reconnect
>> > 2013-01-07 04:45:29.835748
>> > 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45439 pgs=15 cs=3
>> > l=0).fault, initiating reconnect
>> > 2013-01-07 04:45:30.835219 7fec743f4710 0 --
>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>> > pipe(0x7fec5402f320 sd=28 :45894 pgs=482 cs=903 l=0).fault, initiating
>> > reconnect
>> > 2013-01-07 04:45:30.837318 7fec743f4710 0 --
>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>> > pipe(0x7fec5402f320 sd=28 :45895 pgs=483 cs=905 l=0).fault, initiating
>> > reconnect
>> > 2013-01-07 04:45:30.851984 7fec637fe710 0 log [ERR] : map
>> > e27 had wrong cluster addr (192.168.0.124:6808/19449 != my
>> > 192.168.1.124:6808/19449)
>> >
>> > Also, this only happens only when the cluster ip address and the public ip address are different for example
>> > ....
>> > ....
>> > ....
>> > [osd.0]
>> > host = g8ct
>> > public address = 192.168.0.124
>> > cluster address = 192.168.1.124
>> > btrfs devs = /dev/sdb
>> >
>> > ....
>> > ....
>> >
>> > but does not happen when they are the same. Any idea what may be the issue?
>> This isn't familiar to me at first glance. What version of Ceph are you using?
>>
>> If
>> this is easy to reproduce, can you pastebin your ceph.conf and then add
>> "debug ms = 1" to your global config and gather up the logs from each
>> daemon?
>> -Greg
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
>> More majordomo info at http://vger.kernel.org/majordomo
>>
>>
>> Attachments:
>> - ceph-osd.0.log.tar.gz
>>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Jan 28, 2013 at 12:17 PM, Isaac Otsiabah <zmoo76b@yahoo.com> wrote:
>
>
> Gregory, i recreated the osd down problem again this morning on two nodes (g13ct, g14ct). First, i created a 1-node cluster on g13ct (with osd.0, 1 ,2) and then added host g14ct (osd3. 4, 5). osd.1 went down for about 1 minute and half after adding osd 3, 4, 5 were adde4d. i have included the routing table of each node at the time osd.1 went down. ceph.conf and ceph-osd.1.log files are attached. The crush map was default. Also, it could be a timing issue because it does not always fail when using default crush map, it takes several trials before you see it. Thank you.
>
>
> [root@g13ct ~]# netstat -r
> Kernel IP routing table
> Destination Gateway Genmask Flags MSS Window irtt Iface
> default 133.164.98.250 0.0.0.0 UG 0 0 0 eth2
> 133.164.98.0 * 255.255.255.0 U 0 0 0 eth2
> link-local * 255.255.0.0 U 0 0 0 eth3
> link-local * 255.255.0.0 U 0 0 0 eth0
> link-local * 255.255.0.0 U 0 0 0 eth2
> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth0
> 192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
> 192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
> [root@g13ct ~]# ceph osd tree
>
> # id weight type name up/down reweight
> -1 6 root default
> -3 6 rack unknownrack
> -2 3 host g13ct
> 0 1 osd.0 up 1
> 1 1 osd.1 down 1
> 2 1 osd.2 up 1
> -4 3 host g14ct
> 3 1 osd.3 up 1
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
>
>
>
> [root@g14ct ~]# ceph osd tree
>
> # id weight type name up/down reweight
> -1 6 root default
> -3 6 rack unknownrack
> -2 3 host g13ct
> 0 1 osd.0 up 1
> 1 1 osd.1 down 1
> 2 1 osd.2 up 1
> -4 3 host g14ct
> 3 1 osd.3 up 1
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
>
> [root@g14ct ~]# netstat -r
> Kernel IP routing table
> Destination Gateway Genmask Flags MSS Window irtt Iface
> default 133.164.98.250 0.0.0.0 UG 0 0 0 eth0
> 133.164.98.0 * 255.255.255.0 U 0 0 0 eth0
> link-local * 255.255.0.0 U 0 0 0 eth3
> link-local * 255.255.0.0 U 0 0 0 eth5
> link-local * 255.255.0.0 U 0 0 0 eth0
> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth5
> 192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
> 192.168.1.0 * 255.255.255.0 U 0 0 0 eth5
> [root@g14ct ~]# ceph osd tree
>
> # id weight type name up/down reweight
> -1 6 root default
> -3 6 rack unknownrack
> -2 3 host g13ct
> 0 1 osd.0 up 1
> 1 1 osd.1 down 1
> 2 1 osd.2 up 1
> -4 3 host g14ct
> 3 1 osd.3 up 1
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
>
>
>
>
>
> Isaac
>
>
>
>
>
>
>
>
>
>
> ----- Original Message -----
> From: Isaac Otsiabah <zmoo76b@yahoo.com>
> To: Gregory Farnum <greg@inktank.com>
> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
> Sent: Friday, January 25, 2013 9:51 AM
> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>
>
>
> Gregory, the network physical layout is simple, the two networks are
> separate. the 192.168.0 and the 192.168.1 are not subnets within a
> network.
>
> Isaac
>
>
>
>
> ----- Original Message -----
> From: Gregory Farnum <greg@inktank.com>
> To: Isaac Otsiabah <zmoo76b@yahoo.com>
> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
> Sent: Thursday, January 24, 2013 1:28 PM
> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>
> What's the physical layout of your networking? This additional log may prove helpful as well, but I really need a bit more context in evaluating the messages I see from the first one. :)
> -Greg
>
>
> On Thursday, January 24, 2013 at 9:24 AM, Isaac Otsiabah wrote:
>
>>
>>
>> Gregory, i tried send the the attached debug output several times and
>> the mail server rejected them all probably becauseof the file size so i cut the log file size down and it is attached. You will see the
>> reconnection failures by the error message line below. The ceph version
>> is 0.56
>>
>>
>> it appears to be a timing issue because with the flag (debug ms=1) turned on, the system ran slower and became harder to fail.
>> I
>> ran it several times and finally got it to fail on (osd.0) using
>> default crush map. The attached tar file contains log files for all
>> components on g8ct plus the ceph.conf. By the way, the log file contain only the last 1384 lines where the error occurs.
>>
>>
>> I started with a 1-node cluster on host g8ct (osd.0, osd.1, osd.2) and then added host g13ct (osd.3, osd.4, osd.5)
>>
>>
>> id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g8ct
>> 0 1 osd.0 down 1
>> 1 1 osd.1 up 1
>> 2 1 osd.2 up 1
>> -4 3 host g13ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>>
>>
>> The error messages are in ceph.log and ceph-osd.0.log:
>>
>> ceph.log:2013-01-08
>> 05:41:38.080470 osd.0 192.168.0.124:6801/25571 3 : [ERR] map e15 had
>> wrong cluster addr (192.168.0.124:6802/25571 != my
>> 192.168.1.124:6802/25571)
>> ceph-osd.0.log:2013-01-08 05:41:38.080458 7f06757fa710 0 log [ERR] : map e15 had wrong cluster addr
>> (192.168.0.124:6802/25571 != my 192.168.1.124:6802/25571)
>>
>>
>>
>> [root@g8ct ceph]# ceph -v
>> ceph version 0.56 (1a32f0a0b42f169a7b55ed48ec3208f6d4edc1e8)
>>
>>
>> Isaac
>>
>>
>> ----- Original Message -----
>> From: Gregory Farnum <greg@inktank.com (mailto:greg@inktank.com)>
>> To: Isaac Otsiabah <zmoo76b@yahoo.com (mailto:zmoo76b@yahoo.com)>
>> Cc: "ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)" <ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)>
>> Sent: Monday, January 7, 2013 1:27 PM
>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>
>> On Monday, January 7, 2013 at 1:00 PM, Isaac Otsiabah wrote:
>>
>>
>> When i add a new host (with osd's) to my existing cluster, 1 or 2
>> previous osd(s) goes down for about 2 minutes and then they come back
>> up.
>> >
>> >
>> > [root@h1ct ~]# ceph osd tree
>> >
>> > # id weight type name up/down reweight
>> > -1
>> > 3 root default
>> > -3 3 rack unknownrack
>> > -2 3 host h1
>> > 0 1 osd.0 up 1
>> > 1 1 osd.1 up 1
>> > 2
>> > 1 osd.2 up 1
>>
>>
>> For example, after adding host h2 (with 3 new osd) to the above cluster
>> and running the "ceph osd tree" command, i see this:
>> >
>> >
>> > [root@h1 ~]# ceph osd tree
>> >
>> > # id weight type name up/down reweight
>> > -1 6 root default
>> > -3
>> > 6 rack unknownrack
>> > -2 3 host h1
>> > 0 1 osd.0 up 1
>> > 1 1 osd.1 down 1
>> > 2
>> > 1 osd.2 up 1
>> > -4 3 host h2
>> > 3 1 osd.3 up 1
>> > 4 1 osd.4 up
>> > 1
>> > 5 1 osd.5 up 1
>>
>>
>> The down osd always come back up after 2 minutes or less andi see the
>> following error message in the respective osd log file:
>> > 2013-01-07 04:40:17.613028 7fec7f092760 1 journal _open
>> > /ceph_journal/journals/journal_2 fd 26: 1073741824 bytes, block size
>> > 4096 bytes, directio = 1, aio = 0
>> > 2013-01-07 04:40:17.613122
>> > 7fec7f092760 1 journal _open /ceph_journal/journals/journal_2 fd 26:
>> > 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
>> > 2013-01-07
>> > 04:42:10.006533 7fec746f7710 0 -- 192.168.0.124:6808/19449 >>
>> > 192.168.1.123:6800/18287 pipe(0x7fec20000e10 sd=31 :6808 pgs=0 cs=0
>> > l=0).accept connect_seq 0 vs existing 0 state connecting
>> > 2013-01-07
>> > 04:45:29.834341 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45438 pgs=7 cs=1
>> > l=0).fault, initiating reconnect
>> > 2013-01-07 04:45:29.835748
>> > 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45439 pgs=15 cs=3
>> > l=0).fault, initiating reconnect
>> > 2013-01-07 04:45:30.835219 7fec743f4710 0 --
>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>> > pipe(0x7fec5402f320 sd=28 :45894 pgs=482 cs=903 l=0).fault, initiating
>> > reconnect
>> > 2013-01-07 04:45:30.837318 7fec743f4710 0 --
>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>> > pipe(0x7fec5402f320 sd=28 :45895 pgs=483 cs=905 l=0).fault, initiating
>> > reconnect
>> > 2013-01-07 04:45:30.851984 7fec637fe710 0 log [ERR] : map
>> > e27 had wrong cluster addr (192.168.0.124:6808/19449 != my
>> > 192.168.1.124:6808/19449)
>> >
>> > Also, this only happens only when the cluster ip address and the public ip address are different for example
>> > ....
>> > ....
>> > ....
>> > [osd.0]
>> > host = g8ct
>> > public address = 192.168.0.124
>> > cluster address = 192.168.1.124
>> > btrfs devs = /dev/sdb
>> >
>> > ....
>> > ....
>> >
>> > but does not happen when they are the same. Any idea what may be the issue?
>> This isn't familiar to me at first glance. What version of Ceph are you using?
>>
>> If
>> this is easy to reproduce, can you pastebin your ceph.conf and then add
>> "debug ms = 1" to your global config and gather up the logs from each
>> daemon?
>> -Greg
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
>> More majordomo info at http://vger.kernel.org/majordomo
>>
>>
>> Attachments:
>> - ceph-osd.0.log.tar.gz
>>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
2013-02-12 1:39 ` Isaac Otsiabah
@ 2013-02-15 17:20 ` Sam Lang
2013-02-16 2:00 ` Isaac Otsiabah
0 siblings, 1 reply; 13+ messages in thread
From: Sam Lang @ 2013-02-15 17:20 UTC (permalink / raw)
To: Isaac Otsiabah; +Cc: Gregory Farnum, ceph-devel
On Mon, Feb 11, 2013 at 7:39 PM, Isaac Otsiabah <zmoo76b@yahoo.com> wrote:
>
>
> Yes, there were osd daemons running on the same node that the monitor was
> running on. If that is the case then i will run a test case with the
> monitor running on a different node where no osd is running and see what happens. Thank you.
Hi Isaac,
Any luck? Does the problem reproduce with the mon running on a separate host?
-sam
>
> Isaac
>
> ________________________________
> From: Gregory Farnum <greg@inktank.com>
> To: Isaac Otsiabah <zmoo76b@yahoo.com>
> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
> Sent: Monday, February 11, 2013 12:29 PM
> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>
> jIsaac,
> I'm sorry I haven't been able to wrangle any time to look into this
> more yet, but Sage pointed out in a related thread that there might be
> some buggy handling of things like this if the OSD and the monitor are
> located on the same host. Am I correct in assuming that with your
> small cluster, all your OSDs are co-located with a monitor daemon?
> -Greg
>
> On Mon, Jan 28, 2013 at 12:17 PM, Isaac Otsiabah <zmoo76b@yahoo.com> wrote:
>>
>>
>> Gregory, i recreated the osd down problem again this morning on two nodes (g13ct, g14ct). First, i created a 1-node cluster on g13ct (with osd.0, 1 ,2) and then added host g14ct (osd3. 4, 5). osd.1 went down for about 1 minute and half after adding osd 3, 4, 5 were adde4d. i have included the routing table of each node at the time osd.1 went down. ceph.conf and ceph-osd.1.log files are attached. The crush map was default. Also, it could be a timing issue because it does not always fail when using default crush map, it takes several trials before you see it. Thank you.
>>
>>
>> [root@g13ct ~]# netstat -r
>> Kernel IP routing table
>> Destination Gateway Genmask Flags MSS Window irtt Iface
>> default 133.164.98.250 0.0.0.0 UG 0 0 0 eth2
>> 133.164.98.0 * 255.255.255.0 U 0 0 0 eth2
>> link-local * 255.255.0.0 U 0 0 0 eth3
>> link-local * 255.255.0.0 U 0 0 0 eth0
>> link-local * 255.255.0.0 U 0 0 0 eth2
>> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
>> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth0
>> 192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
>> 192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
>> [root@g13ct ~]# ceph osd tree
>>
>> # id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g13ct
>> 0 1 osd.0 up 1
>> 1 1 osd.1 down 1
>> 2 1 osd.2 up 1
>> -4 3 host g14ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>>
>>
>> [root@g14ct ~]# ceph osd tree
>>
>> # id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g13ct
>> 0 1 osd.0 up 1
>> 1 1 osd.1 down 1
>> 2 1 osd.2 up 1
>> -4 3 host g14ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>> [root@g14ct ~]# netstat -r
>> Kernel IP routing table
>> Destination Gateway Genmask Flags MSS Window irtt Iface
>> default 133.164.98.250 0.0.0.0 UG 0 0 0 eth0
>> 133.164.98.0 * 255.255.255.0 U 0 0 0 eth0
>> link-local * 255.255.0.0 U 0 0 0 eth3
>> link-local * 255.255.0.0 U 0 0 0 eth5
>> link-local * 255.255.0.0 U 0 0 0 eth0
>> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
>> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth5
>> 192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
>> 192.168.1.0 * 255.255.255.0 U 0 0 0 eth5
>> [root@g14ct ~]# ceph osd tree
>>
>> # id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g13ct
>> 0 1 osd.0 up 1
>> 1 1 osd.1 down 1
>> 2 1 osd.2 up 1
>> -4 3 host g14ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>>
>>
>>
>>
>> Isaac
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ----- Original Message -----
>> From: Isaac Otsiabah <zmoo76b@yahoo.com>
>> To: Gregory Farnum <greg@inktank.com>
>> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
>> Sent: Friday, January 25, 2013 9:51 AM
>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>
>>
>>
>> Gregory, the network physical layout is simple, the two networks are
>> separate. the 192.168.0 and the 192.168.1 are not subnets within a
>> network.
>>
>> Isaac
>>
>>
>>
>>
>> ----- Original Message -----
>> From: Gregory Farnum <greg@inktank.com>
>> To: Isaac Otsiabah <zmoo76b@yahoo.com>
>> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
>> Sent: Thursday, January 24, 2013 1:28 PM
>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>
>> What's the physical layout of your networking? This additional log may prove helpful as well, but I really need a bit more context in evaluating the messages I see from the first one. :)
>> -Greg
>>
>>
>> On Thursday, January 24, 2013 at 9:24 AM, Isaac Otsiabah wrote:
>>
>>>
>>>
>>> Gregory, i tried send the the attached debug output several times and
>>> the mail server rejected them all probably becauseof the file size so i cut the log file size down and it is attached. You will see the
>>> reconnection failures by the error message line below. The ceph version
>>> is 0.56
>>>
>>>
>>> it appears to be a timing issue because with the flag (debug ms=1) turned on, the system ran slower and became harder to fail.
>>> I
>>> ran it several times and finally got it to fail on (osd.0) using
>>> default crush map. The attached tar file contains log files for all
>>> components on g8ct plus the ceph.conf. By the way, the log file contain only the last 1384 lines where the error occurs.
>>>
>>>
>>> I started with a 1-node cluster on host g8ct (osd.0, osd.1, osd.2) and then added host g13ct (osd.3, osd.4, osd.5)
>>>
>>>
>>> id weight type name up/down reweight
>>> -1 6 root default
>>> -3 6 rack unknownrack
>>> -2 3 host g8ct
>>> 0 1 osd.0 down 1
>>> 1 1 osd.1 up 1
>>> 2 1 osd.2 up 1
>>> -4 3 host g13ct
>>> 3 1 osd.3 up 1
>>> 4 1 osd.4 up 1
>>> 5 1 osd.5 up 1
>>>
>>>
>>>
>>> The error messages are in ceph.log and ceph-osd.0.log:
>>>
>>> ceph.log:2013-01-08
>>> 05:41:38.080470 osd.0 192.168.0.124:6801/25571 3 : [ERR] map e15 had
>>> wrong cluster addr (192.168.0.124:6802/25571 != my
>>> 192.168.1.124:6802/25571)
>>> ceph-osd.0.log:2013-01-08 05:41:38.080458 7f06757fa710 0 log [ERR] : map e15 had wrong cluster addr
>>> (192.168.0.124:6802/25571 != my 192.168.1.124:6802/25571)
>>>
>>>
>>>
>>> [root@g8ct ceph]# ceph -v
>>> ceph version 0.56 (1a32f0a0b42f169a7b55ed48ec3208f6d4edc1e8)
>>>
>>>
>>> Isaac
>>>
>>>
>>> ----- Original Message -----
>>> From: Gregory Farnum <greg@inktank.com (mailto:greg@inktank.com)>
>>> To: Isaac Otsiabah <zmoo76b@yahoo.com (mailto:zmoo76b@yahoo.com)>
>>> Cc: "ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)" <ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)>
>>> Sent: Monday, January 7, 2013 1:27 PM
>>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>>
>>> On Monday, January 7, 2013 at 1:00 PM, Isaac Otsiabah wrote:
>>>
>>>
>>> When i add a new host (with osd's) to my existing cluster, 1 or 2
>>> previous osd(s) goes down for about 2 minutes and then they come back
>>> up.
>>> >
>>> >
>>> > [root@h1ct ~]# ceph osd tree
>>> >
>>> > # id weight type name up/down reweight
>>> > -1
>>> > 3 root default
>>> > -3 3 rack unknownrack
>>> > -2 3 host h1
>>> > 0 1 osd.0 up 1
>>> > 1 1 osd.1 up 1
>>> > 2
>>> > 1 osd.2 up 1
>>>
>>>
>>> For example, after adding host h2 (with 3 new osd) to the above cluster
>>> and running the "ceph osd tree" command, i see this:
>>> >
>>> >
>>> > [root@h1 ~]# ceph osd tree
>>> >
>>> > # id weight type name up/down reweight
>>> > -1 6 root default
>>> > -3
>>> > 6 rack unknownrack
>>> > -2 3 host h1
>>> > 0 1 osd.0 up 1
>>> > 1 1 osd.1 down 1
>>> > 2
>>> > 1 osd.2 up 1
>>> > -4 3 host h2
>>> > 3 1 osd.3 up 1
>>> > 4 1 osd.4 up
>>> > 1
>>> > 5 1 osd.5 up 1
>>>
>>>
>>> The down osd always come back up after 2 minutes or less andi see the
>>> following error message in the respective osd log file:
>>> > 2013-01-07 04:40:17.613028 7fec7f092760 1 journal _open
>>> > /ceph_journal/journals/journal_2 fd 26: 1073741824 bytes, block size
>>> > 4096 bytes, directio = 1, aio = 0
>>> > 2013-01-07 04:40:17.613122
>>> > 7fec7f092760 1 journal _open /ceph_journal/journals/journal_2 fd 26:
>>> > 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
>>> > 2013-01-07
>>> > 04:42:10.006533 7fec746f7710 0 -- 192.168.0.124:6808/19449 >>
>>> > 192.168.1.123:6800/18287 pipe(0x7fec20000e10 sd=31 :6808 pgs=0 cs=0
>>> > l=0).accept connect_seq 0 vs existing 0 state connecting
>>> > 2013-01-07
>>> > 04:45:29.834341 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45438 pgs=7 cs=1
>>> > l=0).fault, initiating reconnect
>>> > 2013-01-07 04:45:29.835748
>>> > 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45439 pgs=15 cs=3
>>> > l=0).fault, initiating reconnect
>>> > 2013-01-07 04:45:30.835219 7fec743f4710 0 --
>>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>>> > pipe(0x7fec5402f320 sd=28 :45894 pgs=482 cs=903 l=0).fault, initiating
>>> > reconnect
>>> > 2013-01-07 04:45:30.837318 7fec743f4710 0 --
>>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>>> > pipe(0x7fec5402f320 sd=28 :45895 pgs=483 cs=905 l=0).fault, initiating
>>> > reconnect
>>> > 2013-01-07 04:45:30.851984 7fec637fe710 0 log [ERR] : map
>>> > e27 had wrong cluster addr (192.168.0.124:6808/19449 != my
>>> > 192.168.1.124:6808/19449)
>>> >
>>> > Also, this only happens only when the cluster ip address and the public ip address are different for example
>>> > ....
>>> > ....
>>> > ....
>>> > [osd.0]
>>> > host = g8ct
>>> > public address = 192.168.0.124
>>> > cluster address = 192.168.1.124
>>> > btrfs devs = /dev/sdb
>>> >
>>> > ....
>>> > ....
>>> >
>>> > but does not happen when they are the same. Any idea what may be the issue?
>>> This isn't familiar to me at first glance. What version of Ceph are you using?
>>>
>>> If
>>> this is easy to reproduce, can you pastebin your ceph.conf and then add
>>> "debug ms = 1" to your global config and gather up the logs from each
>>> daemon?
>>> -Greg
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
>>> More majordomo info at http://vger.kernel.org/majordomo
>>>
>>>
>>> Attachments:
>>> - ceph-osd.0.log.tar.gz
>>>
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
> On Mon, Jan 28, 2013 at 12:17 PM, Isaac Otsiabah <zmoo76b@yahoo.com> wrote:
>>
>>
>> Gregory, i recreated the osd down problem again this morning on two nodes (g13ct, g14ct). First, i created a 1-node cluster on g13ct (with osd.0, 1 ,2) and then added host g14ct (osd3. 4, 5). osd.1 went down for about 1 minute and half after adding osd 3, 4, 5 were adde4d. i have included the routing table of each node at the time osd.1 went down. ceph.conf and ceph-osd.1.log files are attached. The crush map was default. Also, it could be a timing issue because it does not always fail when using default crush map, it takes several trials before you see it. Thank you.
>>
>>
>> [root@g13ct ~]# netstat -r
>> Kernel IP routing table
>> Destination Gateway Genmask Flags MSS Window irtt Iface
>> default 133.164.98.250 0.0.0.0 UG 0 0 0 eth2
>> 133.164.98.0 * 255.255.255.0 U 0 0 0 eth2
>> link-local * 255.255.0.0 U 0 0 0 eth3
>> link-local * 255.255.0.0 U 0 0 0 eth0
>> link-local * 255.255.0.0 U 0 0 0 eth2
>> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
>> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth0
>> 192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
>> 192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
>> [root@g13ct ~]# ceph osd tree
>>
>> # id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g13ct
>> 0 1 osd.0 up 1
>> 1 1 osd.1 down 1
>> 2 1 osd.2 up 1
>> -4 3 host g14ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>>
>>
>> [root@g14ct ~]# ceph osd tree
>>
>> # id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g13ct
>> 0 1 osd.0 up 1
>> 1 1 osd.1 down 1
>> 2 1 osd.2 up 1
>> -4 3 host g14ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>> [root@g14ct ~]# netstat -r
>> Kernel IP routing table
>> Destination Gateway Genmask Flags MSS Window irtt Iface
>> default 133.164.98.250 0.0.0.0 UG 0 0 0 eth0
>> 133.164.98.0 * 255.255.255.0 U 0 0 0 eth0
>> link-local * 255.255.0.0 U 0 0 0 eth3
>> link-local * 255.255.0.0 U 0 0 0 eth5
>> link-local * 255.255.0.0 U 0 0 0 eth0
>> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
>> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth5
>> 192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
>> 192.168.1.0 * 255.255.255.0 U 0 0 0 eth5
>> [root@g14ct ~]# ceph osd tree
>>
>> # id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g13ct
>> 0 1 osd.0 up 1
>> 1 1 osd.1 down 1
>> 2 1 osd.2 up 1
>> -4 3 host g14ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>>
>>
>>
>>
>> Isaac
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ----- Original Message -----
>> From: Isaac Otsiabah <zmoo76b@yahoo.com>
>> To: Gregory Farnum <greg@inktank.com>
>> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
>> Sent: Friday, January 25, 2013 9:51 AM
>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>
>>
>>
>> Gregory, the network physical layout is simple, the two networks are
>> separate. the 192.168.0 and the 192.168.1 are not subnets within a
>> network.
>>
>> Isaac
>>
>>
>>
>>
>> ----- Original Message -----
>> From: Gregory Farnum <greg@inktank.com>
>> To: Isaac Otsiabah <zmoo76b@yahoo.com>
>> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
>> Sent: Thursday, January 24, 2013 1:28 PM
>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>
>> What's the physical layout of your networking? This additional log may prove helpful as well, but I really need a bit more context in evaluating the messages I see from the first one. :)
>> -Greg
>>
>>
>> On Thursday, January 24, 2013 at 9:24 AM, Isaac Otsiabah wrote:
>>
>>>
>>>
>>> Gregory, i tried send the the attached debug output several times and
>>> the mail server rejected them all probably becauseof the file size so i cut the log file size down and it is attached. You will see the
>>> reconnection failures by the error message line below. The ceph version
>>> is 0.56
>>>
>>>
>>> it appears to be a timing issue because with the flag (debug ms=1) turned on, the system ran slower and became harder to fail.
>>> I
>>> ran it several times and finally got it to fail on (osd.0) using
>>> default crush map. The attached tar file contains log files for all
>>> components on g8ct plus the ceph.conf. By the way, the log file contain only the last 1384 lines where the error occurs.
>>>
>>>
>>> I started with a 1-node cluster on host g8ct (osd.0, osd.1, osd.2) and then added host g13ct (osd.3, osd.4, osd.5)
>>>
>>>
>>> id weight type name up/down reweight
>>> -1 6 root default
>>> -3 6 rack unknownrack
>>> -2 3 host g8ct
>>> 0 1 osd.0 down 1
>>> 1 1 osd.1 up 1
>>> 2 1 osd.2 up 1
>>> -4 3 host g13ct
>>> 3 1 osd.3 up 1
>>> 4 1 osd.4 up 1
>>> 5 1 osd.5 up 1
>>>
>>>
>>>
>>> The error messages are in ceph.log and ceph-osd.0.log:
>>>
>>> ceph.log:2013-01-08
>>> 05:41:38.080470 osd.0 192.168.0.124:6801/25571 3 : [ERR] map e15 had
>>> wrong cluster addr (192.168.0.124:6802/25571 != my
>>> 192.168.1.124:6802/25571)
>>> ceph-osd.0.log:2013-01-08 05:41:38.080458 7f06757fa710 0 log [ERR] : map e15 had wrong cluster addr
>>> (192.168.0.124:6802/25571 != my 192.168.1.124:6802/25571)
>>>
>>>
>>>
>>> [root@g8ct ceph]# ceph -v
>>> ceph version 0.56 (1a32f0a0b42f169a7b55ed48ec3208f6d4edc1e8)
>>>
>>>
>>> Isaac
>>>
>>>
>>> ----- Original Message -----
>>> From: Gregory Farnum <greg@inktank.com (mailto:greg@inktank.com)>
>>> To: Isaac Otsiabah <zmoo76b@yahoo.com (mailto:zmoo76b@yahoo.com)>
>>> Cc: "ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)" <ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)>
>>> Sent: Monday, January 7, 2013 1:27 PM
>>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>>
>>> On Monday, January 7, 2013 at 1:00 PM, Isaac Otsiabah wrote:
>>>
>>>
>>> When i add a new host (with osd's) to my existing cluster, 1 or 2
>>> previous osd(s) goes down for about 2 minutes and then they come back
>>> up.
>>> >
>>> >
>>> > [root@h1ct ~]# ceph osd tree
>>> >
>>> > # id weight type name up/down reweight
>>> > -1
>>> > 3 root default
>>> > -3 3 rack unknownrack
>>> > -2 3 host h1
>>> > 0 1 osd.0 up 1
>>> > 1 1 osd.1 up 1
>>> > 2
>>> > 1 osd.2 up 1
>>>
>>>
>>> For example, after adding host h2 (with 3 new osd) to the above cluster
>>> and running the "ceph osd tree" command, i see this:
>>> >
>>> >
>>> > [root@h1 ~]# ceph osd tree
>>> >
>>> > # id weight type name up/down reweight
>>> > -1 6 root default
>>> > -3
>>> > 6 rack unknownrack
>>> > -2 3 host h1
>>> > 0 1 osd.0 up 1
>>> > 1 1 osd.1 down 1
>>> > 2
>>> > 1 osd.2 up 1
>>> > -4 3 host h2
>>> > 3 1 osd.3 up 1
>>> > 4 1 osd.4 up
>>> > 1
>>> > 5 1 osd.5 up 1
>>>
>>>
>>> The down osd always come back up after 2 minutes or less andi see the
>>> following error message in the respective osd log file:
>>> > 2013-01-07 04:40:17.613028 7fec7f092760 1 journal _open
>>> > /ceph_journal/journals/journal_2 fd 26: 1073741824 bytes, block size
>>> > 4096 bytes, directio = 1, aio = 0
>>> > 2013-01-07 04:40:17.613122
>>> > 7fec7f092760 1 journal _open /ceph_journal/journals/journal_2 fd 26:
>>> > 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
>>> > 2013-01-07
>>> > 04:42:10.006533 7fec746f7710 0 -- 192.168.0.124:6808/19449 >>
>>> > 192.168.1.123:6800/18287 pipe(0x7fec20000e10 sd=31 :6808 pgs=0 cs=0
>>> > l=0).accept connect_seq 0 vs existing 0 state connecting
>>> > 2013-01-07
>>> > 04:45:29.834341 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45438 pgs=7 cs=1
>>> > l=0).fault, initiating reconnect
>>> > 2013-01-07 04:45:29.835748
>>> > 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45439 pgs=15 cs=3
>>> > l=0).fault, initiating reconnect
>>> > 2013-01-07 04:45:30.835219 7fec743f4710 0 --
>>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>>> > pipe(0x7fec5402f320 sd=28 :45894 pgs=482 cs=903 l=0).fault, initiating
>>> > reconnect
>>> > 2013-01-07 04:45:30.837318 7fec743f4710 0 --
>>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>>> > pipe(0x7fec5402f320 sd=28 :45895 pgs=483 cs=905 l=0).fault, initiating
>>> > reconnect
>>> > 2013-01-07 04:45:30.851984 7fec637fe710 0 log [ERR] : map
>>> > e27 had wrong cluster addr (192.168.0.124:6808/19449 != my
>>> > 192.168.1.124:6808/19449)
>>> >
>>> > Also, this only happens only when the cluster ip address and the public ip address are different for example
>>> > ....
>>> > ....
>>> > ....
>>> > [osd.0]
>>> > host = g8ct
>>> > public address = 192.168.0.124
>>> > cluster address = 192.168.1.124
>>> > btrfs devs = /dev/sdb
>>> >
>>> > ....
>>> > ....
>>> >
>>> > but does not happen when they are the same. Any idea what may be the issue?
>>> This isn't familiar to me at first glance. What version of Ceph are you using?
>>>
>>> If
>>> this is easy to reproduce, can you pastebin your ceph.conf and then add
>>> "debug ms = 1" to your global config and gather up the logs from each
>>> daemon?
>>> -Greg
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
>>> More majordomo info at http://vger.kernel.org/majordomo
>>>
>>>
>>> Attachments:
>>> - ceph-osd.0.log.tar.gz
>>>
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
2013-02-15 17:20 ` Sam Lang
@ 2013-02-16 2:00 ` Isaac Otsiabah
0 siblings, 0 replies; 13+ messages in thread
From: Isaac Otsiabah @ 2013-02-16 2:00 UTC (permalink / raw)
To: Sam Lang; +Cc: Gregory Farnum, ceph-devel
Hello Sam and Gregory, i got machines today and tested it with the monitor process running on a separate system with no osd daemons and i did not see the problem. On Monday i will do a few test to confirm.
Isaac
----- Original Message -----
From: Sam Lang <sam.lang@inktank.com>
To: Isaac Otsiabah <zmoo76b@yahoo.com>
Cc: Gregory Farnum <greg@inktank.com>; "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Sent: Friday, February 15, 2013 9:20 AM
Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
On Mon, Feb 11, 2013 at 7:39 PM, Isaac Otsiabah <zmoo76b@yahoo.com> wrote:
>
>
> Yes, there were osd daemons running on the same node that the monitor was
> running on. If that is the case then i will run a test case with the
> monitor running on a different node where no osd is running and see what happens. Thank you.
Hi Isaac,
Any luck? Does the problem reproduce with the mon running on a separate host?
-sam
>
> Isaac
>
> ________________________________
> From: Gregory Farnum <greg@inktank.com>
> To: Isaac Otsiabah <zmoo76b@yahoo.com>
> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
> Sent: Monday, February 11, 2013 12:29 PM
> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>
> jIsaac,
> I'm sorry I haven't been able to wrangle any time to look into this
> more yet, but Sage pointed out in a related thread that there might be
> some buggy handling of things like this if the OSD and the monitor are
> located on the same host. Am I correct in assuming that with your
> small cluster, all your OSDs are co-located with a monitor daemon?
> -Greg
>
> On Mon, Jan 28, 2013 at 12:17 PM, Isaac Otsiabah <zmoo76b@yahoo.com> wrote:
>>
>>
>> Gregory, i recreated the osd down problem again this morning on two nodes (g13ct, g14ct). First, i created a 1-node cluster on g13ct (with osd.0, 1 ,2) and then added host g14ct (osd3. 4, 5). osd.1 went down for about 1 minute and half after adding osd 3, 4, 5 were adde4d. i have included the routing table of each node at the time osd.1 went down. ceph.conf and ceph-osd.1.log files are attached. The crush map was default. Also, it could be a timing issue because it does not always fail when using default crush map, it takes several trials before you see it. Thank you.
>>
>>
>> [root@g13ct ~]# netstat -r
>> Kernel IP routing table
>> Destination Gateway Genmask Flags MSS Window irtt Iface
>> default 133.164.98.250 0.0.0.0 UG 0 0 0 eth2
>> 133.164.98.0 * 255.255.255.0 U 0 0 0 eth2
>> link-local * 255.255.0.0 U 0 0 0 eth3
>> link-local * 255.255.0.0 U 0 0 0 eth0
>> link-local * 255.255.0.0 U 0 0 0 eth2
>> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
>> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth0
>> 192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
>> 192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
>> [root@g13ct ~]# ceph osd tree
>>
>> # id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g13ct
>> 0 1 osd.0 up 1
>> 1 1 osd.1 down 1
>> 2 1 osd.2 up 1
>> -4 3 host g14ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>>
>>
>> [root@g14ct ~]# ceph osd tree
>>
>> # id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g13ct
>> 0 1 osd.0 up 1
>> 1 1 osd.1 down 1
>> 2 1 osd.2 up 1
>> -4 3 host g14ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>> [root@g14ct ~]# netstat -r
>> Kernel IP routing table
>> Destination Gateway Genmask Flags MSS Window irtt Iface
>> default 133.164.98.250 0.0.0.0 UG 0 0 0 eth0
>> 133.164.98.0 * 255.255.255.0 U 0 0 0 eth0
>> link-local * 255.255.0.0 U 0 0 0 eth3
>> link-local * 255.255.0.0 U 0 0 0 eth5
>> link-local * 255.255.0.0 U 0 0 0 eth0
>> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
>> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth5
>> 192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
>> 192.168.1.0 * 255.255.255.0 U 0 0 0 eth5
>> [root@g14ct ~]# ceph osd tree
>>
>> # id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g13ct
>> 0 1 osd.0 up 1
>> 1 1 osd.1 down 1
>> 2 1 osd.2 up 1
>> -4 3 host g14ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>>
>>
>>
>>
>> Isaac
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ----- Original Message -----
>> From: Isaac Otsiabah <zmoo76b@yahoo.com>
>> To: Gregory Farnum <greg@inktank.com>
>> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
>> Sent: Friday, January 25, 2013 9:51 AM
>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>
>>
>>
>> Gregory, the network physical layout is simple, the two networks are
>> separate. the 192.168.0 and the 192.168.1 are not subnets within a
>> network.
>>
>> Isaac
>>
>>
>>
>>
>> ----- Original Message -----
>> From: Gregory Farnum <greg@inktank.com>
>> To: Isaac Otsiabah <zmoo76b@yahoo.com>
>> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
>> Sent: Thursday, January 24, 2013 1:28 PM
>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>
>> What's the physical layout of your networking? This additional log may prove helpful as well, but I really need a bit more context in evaluating the messages I see from the first one. :)
>> -Greg
>>
>>
>> On Thursday, January 24, 2013 at 9:24 AM, Isaac Otsiabah wrote:
>>
>>>
>>>
>>> Gregory, i tried send the the attached debug output several times and
>>> the mail server rejected them all probably becauseof the file size so i cut the log file size down and it is attached. You will see the
>>> reconnection failures by the error message line below. The ceph version
>>> is 0.56
>>>
>>>
>>> it appears to be a timing issue because with the flag (debug ms=1) turned on, the system ran slower and became harder to fail.
>>> I
>>> ran it several times and finally got it to fail on (osd.0) using
>>> default crush map. The attached tar file contains log files for all
>>> components on g8ct plus the ceph.conf. By the way, the log file contain only the last 1384 lines where the error occurs.
>>>
>>>
>>> I started with a 1-node cluster on host g8ct (osd.0, osd.1, osd.2) and then added host g13ct (osd.3, osd.4, osd.5)
>>>
>>>
>>> id weight type name up/down reweight
>>> -1 6 root default
>>> -3 6 rack unknownrack
>>> -2 3 host g8ct
>>> 0 1 osd.0 down 1
>>> 1 1 osd.1 up 1
>>> 2 1 osd.2 up 1
>>> -4 3 host g13ct
>>> 3 1 osd.3 up 1
>>> 4 1 osd.4 up 1
>>> 5 1 osd.5 up 1
>>>
>>>
>>>
>>> The error messages are in ceph.log and ceph-osd.0.log:
>>>
>>> ceph.log:2013-01-08
>>> 05:41:38.080470 osd.0 192.168.0.124:6801/25571 3 : [ERR] map e15 had
>>> wrong cluster addr (192.168.0.124:6802/25571 != my
>>> 192.168.1.124:6802/25571)
>>> ceph-osd.0.log:2013-01-08 05:41:38.080458 7f06757fa710 0 log [ERR] : map e15 had wrong cluster addr
>>> (192.168.0.124:6802/25571 != my 192.168.1.124:6802/25571)
>>>
>>>
>>>
>>> [root@g8ct ceph]# ceph -v
>>> ceph version 0.56 (1a32f0a0b42f169a7b55ed48ec3208f6d4edc1e8)
>>>
>>>
>>> Isaac
>>>
>>>
>>> ----- Original Message -----
>>> From: Gregory Farnum <greg@inktank.com (mailto:greg@inktank.com)>
>>> To: Isaac Otsiabah <zmoo76b@yahoo.com (mailto:zmoo76b@yahoo.com)>
>>> Cc: "ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)" <ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)>
>>> Sent: Monday, January 7, 2013 1:27 PM
>>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>>
>>> On Monday, January 7, 2013 at 1:00 PM, Isaac Otsiabah wrote:
>>>
>>>
>>> When i add a new host (with osd's) to my existing cluster, 1 or 2
>>> previous osd(s) goes down for about 2 minutes and then they come back
>>> up.
>>> >
>>> >
>>> > [root@h1ct ~]# ceph osd tree
>>> >
>>> > # id weight type name up/down reweight
>>> > -1
>>> > 3 root default
>>> > -3 3 rack unknownrack
>>> > -2 3 host h1
>>> > 0 1 osd.0 up 1
>>> > 1 1 osd.1 up 1
>>> > 2
>>> > 1 osd.2 up 1
>>>
>>>
>>> For example, after adding host h2 (with 3 new osd) to the above cluster
>>> and running the "ceph osd tree" command, i see this:
>>> >
>>> >
>>> > [root@h1 ~]# ceph osd tree
>>> >
>>> > # id weight type name up/down reweight
>>> > -1 6 root default
>>> > -3
>>> > 6 rack unknownrack
>>> > -2 3 host h1
>>> > 0 1 osd.0 up 1
>>> > 1 1 osd.1 down 1
>>> > 2
>>> > 1 osd.2 up 1
>>> > -4 3 host h2
>>> > 3 1 osd.3 up 1
>>> > 4 1 osd.4 up
>>> > 1
>>> > 5 1 osd.5 up 1
>>>
>>>
>>> The down osd always come back up after 2 minutes or less andi see the
>>> following error message in the respective osd log file:
>>> > 2013-01-07 04:40:17.613028 7fec7f092760 1 journal _open
>>> > /ceph_journal/journals/journal_2 fd 26: 1073741824 bytes, block size
>>> > 4096 bytes, directio = 1, aio = 0
>>> > 2013-01-07 04:40:17.613122
>>> > 7fec7f092760 1 journal _open /ceph_journal/journals/journal_2 fd 26:
>>> > 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
>>> > 2013-01-07
>>> > 04:42:10.006533 7fec746f7710 0 -- 192.168.0.124:6808/19449 >>
>>> > 192.168.1.123:6800/18287 pipe(0x7fec20000e10 sd=31 :6808 pgs=0 cs=0
>>> > l=0).accept connect_seq 0 vs existing 0 state connecting
>>> > 2013-01-07
>>> > 04:45:29.834341 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45438 pgs=7 cs=1
>>> > l=0).fault, initiating reconnect
>>> > 2013-01-07 04:45:29.835748
>>> > 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45439 pgs=15 cs=3
>>> > l=0).fault, initiating reconnect
>>> > 2013-01-07 04:45:30.835219 7fec743f4710 0 --
>>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>>> > pipe(0x7fec5402f320 sd=28 :45894 pgs=482 cs=903 l=0).fault, initiating
>>> > reconnect
>>> > 2013-01-07 04:45:30.837318 7fec743f4710 0 --
>>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>>> > pipe(0x7fec5402f320 sd=28 :45895 pgs=483 cs=905 l=0).fault, initiating
>>> > reconnect
>>> > 2013-01-07 04:45:30.851984 7fec637fe710 0 log [ERR] : map
>>> > e27 had wrong cluster addr (192.168.0.124:6808/19449 != my
>>> > 192.168.1.124:6808/19449)
>>> >
>>> > Also, this only happens only when the cluster ip address and the public ip address are different for example
>>> > ....
>>> > ....
>>> > ....
>>> > [osd.0]
>>> > host = g8ct
>>> > public address = 192.168.0.124
>>> > cluster address = 192.168.1.124
>>> > btrfs devs = /dev/sdb
>>> >
>>> > ....
>>> > ....
>>> >
>>> > but does not happen when they are the same. Any idea what may be the issue?
>>> This isn't familiar to me at first glance. What version of Ceph are you using?
>>>
>>> If
>>> this is easy to reproduce, can you pastebin your ceph.conf and then add
>>> "debug ms = 1" to your global config and gather up the logs from each
>>> daemon?
>>> -Greg
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
>>> More majordomo info at http://vger.kernel.org/majordomo
>>>
>>>
>>> Attachments:
>>> - ceph-osd.0.log.tar.gz
>>>
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
> On Mon, Jan 28, 2013 at 12:17 PM, Isaac Otsiabah <zmoo76b@yahoo.com> wrote:
>>
>>
>> Gregory, i recreated the osd down problem again this morning on two nodes (g13ct, g14ct). First, i created a 1-node cluster on g13ct (with osd.0, 1 ,2) and then added host g14ct (osd3. 4, 5). osd.1 went down for about 1 minute and half after adding osd 3, 4, 5 were adde4d. i have included the routing table of each node at the time osd.1 went down. ceph.conf and ceph-osd.1.log files are attached. The crush map was default. Also, it could be a timing issue because it does not always fail when using default crush map, it takes several trials before you see it. Thank you.
>>
>>
>> [root@g13ct ~]# netstat -r
>> Kernel IP routing table
>> Destination Gateway Genmask Flags MSS Window irtt Iface
>> default 133.164.98.250 0.0.0.0 UG 0 0 0 eth2
>> 133.164.98.0 * 255.255.255.0 U 0 0 0 eth2
>> link-local * 255.255.0.0 U 0 0 0 eth3
>> link-local * 255.255.0.0 U 0 0 0 eth0
>> link-local * 255.255.0.0 U 0 0 0 eth2
>> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
>> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth0
>> 192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
>> 192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
>> [root@g13ct ~]# ceph osd tree
>>
>> # id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g13ct
>> 0 1 osd.0 up 1
>> 1 1 osd.1 down 1
>> 2 1 osd.2 up 1
>> -4 3 host g14ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>>
>>
>> [root@g14ct ~]# ceph osd tree
>>
>> # id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g13ct
>> 0 1 osd.0 up 1
>> 1 1 osd.1 down 1
>> 2 1 osd.2 up 1
>> -4 3 host g14ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>> [root@g14ct ~]# netstat -r
>> Kernel IP routing table
>> Destination Gateway Genmask Flags MSS Window irtt Iface
>> default 133.164.98.250 0.0.0.0 UG 0 0 0 eth0
>> 133.164.98.0 * 255.255.255.0 U 0 0 0 eth0
>> link-local * 255.255.0.0 U 0 0 0 eth3
>> link-local * 255.255.0.0 U 0 0 0 eth5
>> link-local * 255.255.0.0 U 0 0 0 eth0
>> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth3
>> 192.0.0.0 * 255.0.0.0 U 0 0 0 eth5
>> 192.168.0.0 * 255.255.255.0 U 0 0 0 eth3
>> 192.168.1.0 * 255.255.255.0 U 0 0 0 eth5
>> [root@g14ct ~]# ceph osd tree
>>
>> # id weight type name up/down reweight
>> -1 6 root default
>> -3 6 rack unknownrack
>> -2 3 host g13ct
>> 0 1 osd.0 up 1
>> 1 1 osd.1 down 1
>> 2 1 osd.2 up 1
>> -4 3 host g14ct
>> 3 1 osd.3 up 1
>> 4 1 osd.4 up 1
>> 5 1 osd.5 up 1
>>
>>
>>
>>
>>
>> Isaac
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ----- Original Message -----
>> From: Isaac Otsiabah <zmoo76b@yahoo.com>
>> To: Gregory Farnum <greg@inktank.com>
>> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
>> Sent: Friday, January 25, 2013 9:51 AM
>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>
>>
>>
>> Gregory, the network physical layout is simple, the two networks are
>> separate. the 192.168.0 and the 192.168.1 are not subnets within a
>> network.
>>
>> Isaac
>>
>>
>>
>>
>> ----- Original Message -----
>> From: Gregory Farnum <greg@inktank.com>
>> To: Isaac Otsiabah <zmoo76b@yahoo.com>
>> Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
>> Sent: Thursday, January 24, 2013 1:28 PM
>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>
>> What's the physical layout of your networking? This additional log may prove helpful as well, but I really need a bit more context in evaluating the messages I see from the first one. :)
>> -Greg
>>
>>
>> On Thursday, January 24, 2013 at 9:24 AM, Isaac Otsiabah wrote:
>>
>>>
>>>
>>> Gregory, i tried send the the attached debug output several times and
>>> the mail server rejected them all probably becauseof the file size so i cut the log file size down and it is attached. You will see the
>>> reconnection failures by the error message line below. The ceph version
>>> is 0.56
>>>
>>>
>>> it appears to be a timing issue because with the flag (debug ms=1) turned on, the system ran slower and became harder to fail.
>>> I
>>> ran it several times and finally got it to fail on (osd.0) using
>>> default crush map. The attached tar file contains log files for all
>>> components on g8ct plus the ceph.conf. By the way, the log file contain only the last 1384 lines where the error occurs.
>>>
>>>
>>> I started with a 1-node cluster on host g8ct (osd.0, osd.1, osd.2) and then added host g13ct (osd.3, osd.4, osd.5)
>>>
>>>
>>> id weight type name up/down reweight
>>> -1 6 root default
>>> -3 6 rack unknownrack
>>> -2 3 host g8ct
>>> 0 1 osd.0 down 1
>>> 1 1 osd.1 up 1
>>> 2 1 osd.2 up 1
>>> -4 3 host g13ct
>>> 3 1 osd.3 up 1
>>> 4 1 osd.4 up 1
>>> 5 1 osd.5 up 1
>>>
>>>
>>>
>>> The error messages are in ceph.log and ceph-osd.0.log:
>>>
>>> ceph.log:2013-01-08
>>> 05:41:38.080470 osd.0 192.168.0.124:6801/25571 3 : [ERR] map e15 had
>>> wrong cluster addr (192.168.0.124:6802/25571 != my
>>> 192.168.1.124:6802/25571)
>>> ceph-osd.0.log:2013-01-08 05:41:38.080458 7f06757fa710 0 log [ERR] : map e15 had wrong cluster addr
>>> (192.168.0.124:6802/25571 != my 192.168.1.124:6802/25571)
>>>
>>>
>>>
>>> [root@g8ct ceph]# ceph -v
>>> ceph version 0.56 (1a32f0a0b42f169a7b55ed48ec3208f6d4edc1e8)
>>>
>>>
>>> Isaac
>>>
>>>
>>> ----- Original Message -----
>>> From: Gregory Farnum <greg@inktank.com (mailto:greg@inktank.com)>
>>> To: Isaac Otsiabah <zmoo76b@yahoo.com (mailto:zmoo76b@yahoo.com)>
>>> Cc: "ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)" <ceph-devel@vger.kernel.org (mailto:ceph-devel@vger.kernel.org)>
>>> Sent: Monday, January 7, 2013 1:27 PM
>>> Subject: Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
>>>
>>> On Monday, January 7, 2013 at 1:00 PM, Isaac Otsiabah wrote:
>>>
>>>
>>> When i add a new host (with osd's) to my existing cluster, 1 or 2
>>> previous osd(s) goes down for about 2 minutes and then they come back
>>> up.
>>> >
>>> >
>>> > [root@h1ct ~]# ceph osd tree
>>> >
>>> > # id weight type name up/down reweight
>>> > -1
>>> > 3 root default
>>> > -3 3 rack unknownrack
>>> > -2 3 host h1
>>> > 0 1 osd.0 up 1
>>> > 1 1 osd.1 up 1
>>> > 2
>>> > 1 osd.2 up 1
>>>
>>>
>>> For example, after adding host h2 (with 3 new osd) to the above cluster
>>> and running the "ceph osd tree" command, i see this:
>>> >
>>> >
>>> > [root@h1 ~]# ceph osd tree
>>> >
>>> > # id weight type name up/down reweight
>>> > -1 6 root default
>>> > -3
>>> > 6 rack unknownrack
>>> > -2 3 host h1
>>> > 0 1 osd.0 up 1
>>> > 1 1 osd.1 down 1
>>> > 2
>>> > 1 osd.2 up 1
>>> > -4 3 host h2
>>> > 3 1 osd.3 up 1
>>> > 4 1 osd.4 up
>>> > 1
>>> > 5 1 osd.5 up 1
>>>
>>>
>>> The down osd always come back up after 2 minutes or less andi see the
>>> following error message in the respective osd log file:
>>> > 2013-01-07 04:40:17.613028 7fec7f092760 1 journal _open
>>> > /ceph_journal/journals/journal_2 fd 26: 1073741824 bytes, block size
>>> > 4096 bytes, directio = 1, aio = 0
>>> > 2013-01-07 04:40:17.613122
>>> > 7fec7f092760 1 journal _open /ceph_journal/journals/journal_2 fd 26:
>>> > 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
>>> > 2013-01-07
>>> > 04:42:10.006533 7fec746f7710 0 -- 192.168.0.124:6808/19449 >>
>>> > 192.168.1.123:6800/18287 pipe(0x7fec20000e10 sd=31 :6808 pgs=0 cs=0
>>> > l=0).accept connect_seq 0 vs existing 0 state connecting
>>> > 2013-01-07
>>> > 04:45:29.834341 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45438 pgs=7 cs=1
>>> > l=0).fault, initiating reconnect
>>> > 2013-01-07 04:45:29.835748
>>> > 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
>>> > 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45439 pgs=15 cs=3
>>> > l=0).fault, initiating reconnect
>>> > 2013-01-07 04:45:30.835219 7fec743f4710 0 --
>>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>>> > pipe(0x7fec5402f320 sd=28 :45894 pgs=482 cs=903 l=0).fault, initiating
>>> > reconnect
>>> > 2013-01-07 04:45:30.837318 7fec743f4710 0 --
>>> > 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
>>> > pipe(0x7fec5402f320 sd=28 :45895 pgs=483 cs=905 l=0).fault, initiating
>>> > reconnect
>>> > 2013-01-07 04:45:30.851984 7fec637fe710 0 log [ERR] : map
>>> > e27 had wrong cluster addr (192.168.0.124:6808/19449 != my
>>> > 192.168.1.124:6808/19449)
>>> >
>>> > Also, this only happens only when the cluster ip address and the public ip address are different for example
>>> > ....
>>> > ....
>>> > ....
>>> > [osd.0]
>>> > host = g8ct
>>> > public address = 192.168.0.124
>>> > cluster address = 192.168.1.124
>>> > btrfs devs = /dev/sdb
>>> >
>>> > ....
>>> > ....
>>> >
>>> > but does not happen when they are the same. Any idea what may be the issue?
>>> This isn't familiar to me at first glance. What version of Ceph are you using?
>>>
>>> If
>>> this is easy to reproduce, can you pastebin your ceph.conf and then add
>>> "debug ms = 1" to your global config and gather up the logs from each
>>> daemon?
>>> -Greg
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org (mailto:majordomo@vger.kernel.org)
>>> More majordomo info at http://vger.kernel.org/majordomo
>>>
>>>
>>> Attachments:
>>> - ceph-osd.0.log.tar.gz
>>>
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 13+ messages in thread
* ceph-deploy errors on CentOS
2013-02-11 20:29 ` Gregory Farnum
2013-02-12 1:39 ` Isaac Otsiabah
@ 2013-05-21 20:21 ` Isaac Otsiabah
1 sibling, 0 replies; 13+ messages in thread
From: Isaac Otsiabah @ 2013-05-21 20:21 UTC (permalink / raw)
To: Gregory Farnum; +Cc: ceph-devel
Hello everyone, i am testing ceph-deploy on Centos 6.3 i am getting errors. i have a simple one node setup as follows:
OS: CentOS 6.3
kernel 3.5 and also kernel 2.6.32-279.el6.x86_64
Journal partition size=2GB
/dev/sdb label=gpt
selinux=OFF
iptables=OFF
NUMBER OF OSD=2
Test 1:
ceph-deploy new gclient158
ceph-deploy mon create gclient158
ceph-deploy disk zap gclient158:/dev/sdc
ceph-deploy disk zap gclient158:/dev/sdd
ceph-deploy gatherkeys gclient158
ceph-deploy mds create gclient158
ceph-deploy osd prepare gclient158:sdc:/dev/sdb1
ceph-deploy osd prepare gclient158:sdd:/dev/sdb2
ceph-deploy osd activate gclient158:/dev/sdc:/dev/sdb1
ceph-deploy osd activate gclient158:/dev/sdd:/dev/sdb2
The result of the above ceph-deploy commands are shown below. The 2 osd are running but "ceph health" command nnnever show HEALTK_OK. It stays in HEALTH_WARN forever and is degraded. By the way /var/log/ceph/ceph-osd.0.log and /var/log/ceph/ceph-osd.1.log contain no real errors.This behavior is the same for kernel 3.5 and 2.6.32-279.el6.x86_64. What am i missing?
[root@gclient158 ~]# ps -elf|grep ceph
5 S root 3124 1 0 80 0 - 40727 futex_ 10:49 ? 00:00:00 /usr/bin/ceph-mon -i gclient158 --pid-file /var/run/ceph/mon.gclient158.pid -c /etc/ceph/ceph.conf
5 S root 3472 1 0 80 0 - 41194 futex_ 10:49 ? 00:00:00 /usr/bin/ceph-mds -i gclient158 --pid-file /var/run/ceph/mds.gclient158.pid -c /etc/ceph/ceph.conf
5 S root 4035 1 1 78 -2 - 115119 futex_ 10:50 ? 00:00:00 /usr/bin/ceph-osd -i 0 --pid-file /var/run/ceph/osd.0.pid -c /etc/ceph/ceph.conf
5 S root 4769 1 0 78 -2 - 112304 futex_ 10:50 ? 00:00:00 /usr/bin/ceph-osd -i 1 --pid-file /var/run/ceph/osd.1.pid -c /etc/ceph/ceph.conf
0 S root 5025 2710 0 80 0 - 25811 pipe_w 10:50 pts/0 00:00:00 grep ceph
[root@gclient158 ~]# ceph osd tree
# id weight type name up/down reweight
-1 0.14 root default
-2 0.14 host gclient158
0 0.06999 osd.0 up 1
1 0.06999 osd.1 up 1
[root@gclient158 ~]# ceph health
HEALTH_WARN 91 pgs degraded; 192 pgs stuck unclean; recovery 9/42 degraded (21.429%); recovering 2 o/s, 1492B/s
[root@gclient158 ~]# ceph health
HEALTH_WARN 91 pgs degraded; 192 pgs stuck unclean; recovery 9/42 degraded (21.429%); recovering 2 o/s, 1492B/s
[root@gclient158 ~]# ceph health
HEALTH_WARN 91 pgs degraded; 192 pgs stuck unclean; recovery 9/42 degraded (21.429%); recovering 2 o/s, 1492B/s
By the way /var/log/ceph/ceph-osd.0.log and /var/log/ceph/ceph-osd.1.log contain no real errors but one thing that happens after osd prepare and activate commands is the error below:
Traceback (most recent call last):
File "/usr/sbin/ceph-deploy", line 8, in <module>
load_entry_point('ceph-deploy==0.1', 'console_scripts', 'ceph-deploy')()
File "/root/ceph-deploy/ceph_deploy/cli.py", line 112, in main
return args.func(args)
File "/root/ceph-deploy/ceph_deploy/osd.py", line 426, in osd
prepare(args, cfg, activate_prepared_disk=False)
File "/root/ceph-deploy/ceph_deploy/osd.py", line 273, in prepare
s = '{} returned {}\n{}\n{}'.format(cmd, ret, out, err)
ValueError: zero length field name in format
The above error probably has to with the journal device. I had the same error when the journal device label=gpt and also with journal device label=msdos. Please, what am i missing here and why does the cluster never reaches HEALTH_OK?
TEST 2: the setup for this test is same as above except i used same disk for both ceph data and journal as follows:
ceph-deploy osd prepare gclient158:/dev/sdc
ceph-deploy osd prepare gclient158:/dev/sdd
ceph-deploy osd activate gclient158:/dev/sdc
ceph-deploy osd activate gclient158:/dev/sdd
For test 2, i do not get the error in test 1 but osd's fail to start and both osd log files contain this error:
2013-05-21 11:54:24.806747 7f26cfa26780 -1 journal check: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected 942af534-ccc0-4843-8598-79420592317a, invalid (someone else's?) journal
2013-05-21 11:54:24.806784 7f26cfa26780 -1 filestore(/var/lib/ceph/tmp/mnt.3YsEmH) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.3YsEmH/journal: (22) Invalid argument
2013-05-21 11:54:24.806802 7f26cfa26780 -1 OSD::mkfs: FileStore::mkfs failed with error -22
2013-05-21 11:54:24.806838 7f26cfa26780 -1 ^[[0;31m ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.3YsEmH: (22) Invalid argument^[[0m
What am i missing? any suggestion on both test cases would be appreciated. Thank you.
Isaac
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
[not found] ` <1357680673.72602.YahooMailNeo@web121904.mail.ne1.yahoo.com>
@ 2013-01-10 18:32 ` Gregory Farnum
0 siblings, 0 replies; 13+ messages in thread
From: Gregory Farnum @ 2013-01-10 18:32 UTC (permalink / raw)
To: Isaac Otsiabah; +Cc: ceph-devel
On Tue, Jan 8, 2013 at 1:31 PM, Isaac Otsiabah <zmoo76b@yahoo.com> wrote:
>
>
> Hi Greg, it appears to be a timing issue because with the flag (debug ms=1) turned on, the system ran slower and became harder to fail. I ran it several times and finally got it to fail on (osd.0) using default crush map. The attached tar file contains log files for all components on g8ct plus the ceph.conf.
>
> I started with a 1-node cluster on host g8ct (osd.0, osd.1, osd.2) and then added host g13ct (osd.3, osd.4, osd.5)
>
>
>
> id weight type name up/down reweight
> -1 6 root default
> -3 6 rack unknownrack
> -2 3 host g8ct
> 0 1 osd.0 down 1
> 1 1 osd.1 up 1
> 2 1 osd.2 up 1
> -4 3 host g13ct
> 3 1 osd.3 up 1
> 4 1 osd.4 up 1
> 5 1 osd.5 up 1
>
>
>
> The error messages are in ceph.log and ceph-osd.0.log:
>
> ceph.log:2013-01-08 05:41:38.080470 osd.0 192.168.0.124:6801/25571 3 : [ERR] map e15 had wrong cluster addr (192.168.0.124:6802/25571 != my 192.168.1.124:6802/25571)
> ceph-osd.0.log:2013-01-08 05:41:38.080458 7f06757fa710 0 log [ERR] : map e15 had wrong cluster addr (192.168.0.124:6802/25571 != my 192.168.1.124:6802/25571)
Thanks. I had a brief look through these logs on Tuesday and want to
spend more time with them because they have some odd stuff in them. It
*looks* like the OSD is starting out using a single IP for both the
public and cluster networks and then switching over at some point,
which is...odd.
Knowing more details about how your network is actually set up would
be very helpful.
-Greg
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: osd down (for 2 about 2 minutes) error after adding a new host to my cluster
2013-01-07 21:00 ` osd down (for 2 about 2 minutes) error after adding a new host to my cluster Isaac Otsiabah
@ 2013-01-07 21:27 ` Gregory Farnum
[not found] ` <1357680673.72602.YahooMailNeo@web121904.mail.ne1.yahoo.com>
0 siblings, 1 reply; 13+ messages in thread
From: Gregory Farnum @ 2013-01-07 21:27 UTC (permalink / raw)
To: Isaac Otsiabah; +Cc: ceph-devel
On Monday, January 7, 2013 at 1:00 PM, Isaac Otsiabah wrote:
>
>
> When i add a new host (with osd's) to my existing cluster, 1 or 2 previous osd(s) goes down for about 2 minutes and then they come back up.
>
>
> [root@h1ct ~]# ceph osd tree
>
> # id weight type name up/down reweight
> -1
> 3 root default
> -3 3 rack unknownrack
> -2 3 host h1
> 0 1 osd.0 up 1
> 1 1 osd.1 up 1
> 2
> 1 osd.2 up 1
>
>
> For example, after adding host h2 (with 3 new osd) to the above cluster and running the "ceph osd tree" command, i see this:
>
>
> [root@h1 ~]# ceph osd tree
>
> # id weight type name up/down reweight
> -1 6 root default
> -3
> 6 rack unknownrack
> -2 3 host h1
> 0 1 osd.0 up 1
> 1 1 osd.1 down 1
> 2
> 1 osd.2 up 1
> -4 3 host h2
> 3 1 osd.3 up 1
> 4 1 osd.4 up
> 1
> 5 1 osd.5 up 1
>
>
> The down osd always come back up after 2 minutes or less andi see the following error message in the respective osd log file:
> 2013-01-07 04:40:17.613028 7fec7f092760 1 journal _open
> /ceph_journal/journals/journal_2 fd 26: 1073741824 bytes, block size
> 4096 bytes, directio = 1, aio = 0
> 2013-01-07 04:40:17.613122
> 7fec7f092760 1 journal _open /ceph_journal/journals/journal_2 fd 26:
> 1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
> 2013-01-07
> 04:42:10.006533 7fec746f7710 0 -- 192.168.0.124:6808/19449 >>
> 192.168.1.123:6800/18287 pipe(0x7fec20000e10 sd=31 :6808 pgs=0 cs=0
> l=0).accept connect_seq 0 vs existing 0 state connecting
> 2013-01-07
> 04:45:29.834341 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
> 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45438 pgs=7 cs=1
> l=0).fault, initiating reconnect
> 2013-01-07 04:45:29.835748
> 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
> 192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45439 pgs=15 cs=3
> l=0).fault, initiating reconnect
> 2013-01-07 04:45:30.835219 7fec743f4710 0 --
> 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
> pipe(0x7fec5402f320 sd=28 :45894 pgs=482 cs=903 l=0).fault, initiating
> reconnect
> 2013-01-07 04:45:30.837318 7fec743f4710 0 --
> 192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
> pipe(0x7fec5402f320 sd=28 :45895 pgs=483 cs=905 l=0).fault, initiating
> reconnect
> 2013-01-07 04:45:30.851984 7fec637fe710 0 log [ERR] : map
> e27 had wrong cluster addr (192.168.0.124:6808/19449 != my
> 192.168.1.124:6808/19449)
>
> Also, this only happens only when the cluster ip address and the public ip address are different for example
> ....
> ....
> ....
> [osd.0]
> host = g8ct
> public address = 192.168.0.124
> cluster address = 192.168.1.124
> btrfs devs = /dev/sdb
>
> ....
> ....
>
> but does not happen when they are the same. Any idea what may be the issue?
>
This isn't familiar to me at first glance. What version of Ceph are you using?
If this is easy to reproduce, can you pastebin your ceph.conf and then add "debug ms = 1" to your global config and gather up the logs from each daemon?
-Greg
^ permalink raw reply [flat|nested] 13+ messages in thread
* osd down (for 2 about 2 minutes) error after adding a new host to my cluster
[not found] ` <1357591756.80653.YahooMailNeo@web121903.mail.ne1.yahoo.com>
@ 2013-01-07 21:00 ` Isaac Otsiabah
2013-01-07 21:27 ` Gregory Farnum
0 siblings, 1 reply; 13+ messages in thread
From: Isaac Otsiabah @ 2013-01-07 21:00 UTC (permalink / raw)
To: ceph-devel
When i add a new host (with osd's) to my existing cluster, 1 or 2 previous osd(s) goes down for about 2 minutes and then they come back up.
[root@h1ct ~]# ceph osd tree
# id weight type name up/down reweight
-1
3 root default
-3 3 rack unknownrack
-2 3 host h1
0 1 osd.0 up 1
1 1 osd.1 up 1
2
1 osd.2 up 1
For example, after adding host h2 (with 3 new osd) to the above cluster and running the "ceph osd tree" command, i see this:
[root@h1 ~]# ceph osd tree
# id weight type name up/down reweight
-1 6 root default
-3
6 rack unknownrack
-2 3 host h1
0 1 osd.0 up 1
1 1 osd.1 down 1
2
1 osd.2 up 1
-4 3 host h2
3 1 osd.3 up 1
4 1 osd.4 up
1
5 1 osd.5 up 1
The down osd always come back up after 2 minutes or less andi see the following error message in the respective osd log file:
2013-01-07 04:40:17.613028 7fec7f092760 1 journal _open
/ceph_journal/journals/journal_2 fd 26: 1073741824 bytes, block size
4096 bytes, directio = 1, aio = 0
2013-01-07 04:40:17.613122
7fec7f092760 1 journal _open /ceph_journal/journals/journal_2 fd 26:
1073741824 bytes, block size 4096 bytes, directio = 1, aio = 0
2013-01-07
04:42:10.006533 7fec746f7710 0 -- 192.168.0.124:6808/19449 >>
192.168.1.123:6800/18287 pipe(0x7fec20000e10 sd=31 :6808 pgs=0 cs=0
l=0).accept connect_seq 0 vs existing 0 state connecting
2013-01-07
04:45:29.834341 7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45438 pgs=7 cs=1
l=0).fault, initiating reconnect
2013-01-07 04:45:29.835748
7fec743f4710 0 -- 192.168.1.124:6808/19449 >>
192.168.1.122:6800/20072 pipe(0x7fec5402f320 sd=28 :45439 pgs=15 cs=3
l=0).fault, initiating reconnect
2013-01-07 04:45:30.835219 7fec743f4710 0 --
192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
pipe(0x7fec5402f320 sd=28 :45894 pgs=482 cs=903 l=0).fault, initiating
reconnect
2013-01-07 04:45:30.837318 7fec743f4710 0 --
192.168.1.124:6808/19449 >> 192.168.1.122:6800/20072
pipe(0x7fec5402f320 sd=28 :45895 pgs=483 cs=905 l=0).fault, initiating
reconnect
2013-01-07 04:45:30.851984 7fec637fe710 0 log [ERR] : map
e27 had wrong cluster addr (192.168.0.124:6808/19449 != my
192.168.1.124:6808/19449)
Also, this only happens only when the cluster ip address and the public ip address are different for example
....
....
....
[osd.0]
host = g8ct
public address = 192.168.0.124
cluster address = 192.168.1.124
btrfs devs = /dev/sdb
....
....
but does not happen when they are the same. Any idea what may be the issue?
Isaac
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2013-05-21 20:26 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-01-24 17:24 osd down (for 2 about 2 minutes) error after adding a new host to my cluster Isaac Otsiabah
2013-01-24 21:28 ` Gregory Farnum
2013-01-25 17:51 ` Isaac Otsiabah
2013-01-25 23:46 ` Sam Lang
2013-01-28 20:17 ` Isaac Otsiabah
2013-02-11 20:29 ` Gregory Farnum
2013-02-12 1:39 ` Isaac Otsiabah
2013-02-15 17:20 ` Sam Lang
2013-02-16 2:00 ` Isaac Otsiabah
2013-05-21 20:21 ` ceph-deploy errors on CentOS Isaac Otsiabah
-- strict thread matches above, loose matches on Subject: below --
2012-11-29 7:45 parsing in the ceph osd subsystem Andrey Korolyov
2012-11-29 16:34 ` Sage Weil
2012-11-29 19:01 ` Andrey Korolyov
2012-11-30 1:04 ` Joao Eduardo Luis
[not found] ` <1354237947.86472.YahooMailNeo@web121901.mail.ne1.yahoo.com>
2012-11-30 1:22 ` What is the new command to add osd to the crushmap to enable it to receive data Isaac Otsiabah
[not found] ` <1357591756.80653.YahooMailNeo@web121903.mail.ne1.yahoo.com>
2013-01-07 21:00 ` osd down (for 2 about 2 minutes) error after adding a new host to my cluster Isaac Otsiabah
2013-01-07 21:27 ` Gregory Farnum
[not found] ` <1357680673.72602.YahooMailNeo@web121904.mail.ne1.yahoo.com>
2013-01-10 18:32 ` Gregory Farnum
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.