* Inconclusive answer when running tests for cephtool-test-mon.sh
@ 2016-09-05 11:42 Willem Jan Withagen
2016-09-05 15:18 ` Willem Jan Withagen
0 siblings, 1 reply; 4+ messages in thread
From: Willem Jan Withagen @ 2016-09-05 11:42 UTC (permalink / raw)
To: Ceph Development
Hi
I would innterested to know why I get 2 different answers:
(this is asking the OSDs directly?)
ceph osd dump
but osd.0 is in, up, up_from 179 up_thru 185 down_at 176
osd.1/2 are in, up, up_from 8/13 up_thru 224 down_at 0
ceph -s reports 1 OSD down
So all osds in the dump are in and up, but ...
I guess that osd.0 is telling me when it came back and that it has not
all the data that osd.1/2 have because they go up to 224
Which is why ceph -s tells me that one osd is down?
Or did the leading MON nog get fully informed.
What daemon is missing what part of the communication?
And what for type of warning/error should I look for in the log files?
Thanx,
--WjW
4: /home/wjw/wip/qa/workunits/cephtool/test.sh:12: check_no_osd_down:
ceph osd dump
4: epoch 230
4: fsid d00499b7-6b4c-4e71-a862-420b0d921097
4: created 2016-09-05 13:10:08.247894
4: modified 2016-09-05 13:13:17.952969
4: flags sortbitwise,require_jewel_osds,require_kraken_osds
4: pool 0 'rbd' replicated size 3 min_size 1 crush_ruleset 0 object_hash
rjenkins pg_num 8 pgp_num 8 last_change 1 flags hashpspool stripe_width 0
4: max_osd 3
4: osd.0 up in weight 1 up_from 179 up_thru 185 down_at 176
last_clean_interval [4,178) 127.0.0.1:6800/27490 127.0.0.1:6812/1027490
127.0.0.1:6813/1027490 127.0.0.1:6814/1027490 exists,up
cb894225-670c-4e80-8e15-123c161cd00e
4: osd.1 up in weight 1 up_from 8 up_thru 224 down_at 0
last_clean_interval [0,0) 127.0.0.1:6804/27505 127.0.0.1:6805/27505
127.0.0.1:6806/27505 127.0.0.1:6807/27505 exists,up
8cf8bf1f-27ce-49bb-88cf-c1d54511c434
4: osd.2 up in weight 1 up_from 13 up_thru 224 down_at 0
last_clean_interval [0,0) 127.0.0.1:6808/27520 127.0.0.1:6809/27520
127.0.0.1:6810/27520 127.0.0.1:6811/27520 exists,up
568223b6-06d6-41d0-92c3-286227819bb5
4: pg_temp 0.2 [1,2]
4: pg_temp 0.6 [1,2]
4: pg_temp 0.7 [1,2]
4: /home/wjw/wip/qa/workunits/cephtool/test.sh:13: check_no_osd_down:
ceph -s
4: cluster d00499b7-6b4c-4e71-a862-420b0d921097
4: health HEALTH_WARN
4: 5 pgs peering
4: 3 pgs stale
4: 1/3 in osds are down
4: monmap e1: 3 mons at
{a=127.0.0.1:7202/0,b=127.0.0.1:7203/0,c=127.0.0.1:7204/0}
4: election epoch 6, quorum 0,1,2 a,b,c
4: osdmap e232: 3 osds: 2 up, 3 in; 5 remapped pgs
4: flags sortbitwise,require_jewel_osds,require_kraken_osds
4: pgmap v326: 8 pgs, 1 pools, 0 bytes data, 0 objects
4: 200 GB used, 248 GB / 448 GB avail
4: 3 stale+active+clean
4: 3 remapped+peering
4: 2 peering
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Inconclusive answer when running tests for cephtool-test-mon.sh
2016-09-05 11:42 Inconclusive answer when running tests for cephtool-test-mon.sh Willem Jan Withagen
@ 2016-09-05 15:18 ` Willem Jan Withagen
2016-09-06 17:36 ` Gregory Farnum
0 siblings, 1 reply; 4+ messages in thread
From: Willem Jan Withagen @ 2016-09-05 15:18 UTC (permalink / raw)
To: Ceph Development
On 5-9-2016 13:42, Willem Jan Withagen wrote:
> Hi
>
> I would innterested to know why I get 2 different answers:
>
> (this is asking the OSDs directly?)
> ceph osd dump
> but osd.0 is in, up, up_from 179 up_thru 185 down_at 176
> osd.1/2 are in, up, up_from 8/13 up_thru 224 down_at 0
>
> ceph -s reports 1 OSD down
>
> So all osds in the dump are in and up, but ...
> I guess that osd.0 is telling me when it came back and that it has not
> all the data that osd.1/2 have because they go up to 224
>
> Which is why ceph -s tells me that one osd is down?
> Or did the leading MON nog get fully informed.
>
> What daemon is missing what part of the communication?
> And what for type of warning/error should I look for in the log files?
A bit of extra info:
The trouble starts with:
ceph osd down 0
4: marked down osd.0.
ceph osd dump
4: epoch 174
4: fsid eee1774b-3560-46ec-83b4-bac0e8763e93
4: created 2016-09-05 16:47:43.301220
4: modified 2016-09-05 16:51:16.529826
4: flags noup,sortbitwise,require_jewel_osds,require_kraken_osds
4: pool 0 'rbd' replicated size 3 min_size 1 crush_ruleset 0 object_hash
rjenkins pg_num 8 pgp_num 8 last_change 1 flags hashpspool stripe_width 0
4: max_osd 3
4: osd.0 down in weight 1 up_from 4 up_thru 167 down_at 173
last_clean_interval [0,0) 127.0.0.1:6800/45257 127.0.0.1:6801/45257
127.0.0.1:6802/45257 127.0.0.1:6803/45257 exists
7688bac7-75ec-4a3c-8edc-ce7245071a90
4: osd.1 up in weight 1 up_from 8 up_thru 173 down_at 0
last_clean_interval [0,0) 127.0.0.1:6804/45271 127.0.0.1:6805/45271
127.0.0.1:6806/45271 127.0.0.1:6807/45271 exists,up
7639c41c-be59-42e4-9eb4-fcb6372e7042
4: osd.2 up in weight 1 up_from 14 up_thru 173 down_at 0
last_clean_interval [0,0) 127.0.0.1:6808/45285 127.0.0.1:6809/45285
127.0.0.1:6810/45285 127.0.0.1:6811/45285 exists,up
39e812a4-2f50-4088-bebd-6392dc05c76c
4: pg_temp 0.0 [0,2,1]
4: pg_temp 0.1 [2,0,1]
4: pg_temp 0.2 [0,1,2]
4: pg_temp 0.3 [2,0,1]
4: pg_temp 0.4 [0,2,1]
4: pg_temp 0.5 [0,2,1]
4: pg_temp 0.6 [0,1,2]
4: pg_temp 0.7 [1,0,2]
ceph daemon osd.0 status
4: {
4: "cluster_fsid": "eee1774b-3560-46ec-83b4-bac0e8763e93",
4: "osd_fsid": "7688bac7-75ec-4a3c-8edc-ce7245071a90",
4: "whoami": 0,
4: "state": "active",
4: "oldest_map": 1,
4: "newest_map": 172,
4: "num_pgs": 8
4: }
4: /home/wjw/wip/qa/workunits/cephtool/test.sh:15: osd_state: ceph -s
4: cluster eee1774b-3560-46ec-83b4-bac0e8763e93
4: health HEALTH_WARN
4: 3 pgs degraded
4: 3 pgs stale
4: 3 pgs undersized
4: 1/3 in osds are down
4: noup,sortbitwise,require_jewel_osds,require_kraken_osds
flag(s) set
4: monmap e1: 3 mons at
{a=127.0.0.1:7202/0,b=127.0.0.1:7203/0,c=127.0.0.1:7204/0}
4: election epoch 6, quorum 0,1,2 a,b,c
4: osdmap e174: 3 osds: 2 up, 3 in; 8 remapped pgs
4: flags noup,sortbitwise,require_jewel_osds,require_kraken_osds
4: pgmap v294: 8 pgs, 1 pools, 0 bytes data, 0 objects
4: 300 GB used, 349 GB / 650 GB avail
4: 3 stale+active+clean
4: 3 active+undersized+degraded
4: 2 active+clean
And then:
ceph osd unset noup
4: noup is unset
And we wait/loop until çeph osd dump' reports osd.0 up.
ceph osd dump
4: epoch 177
4: fsid eee1774b-3560-46ec-83b4-bac0e8763e93
4: created 2016-09-05 16:47:43.301220
4: modified 2016-09-05 16:51:20.098412
4: flags sortbitwise,require_jewel_osds,require_kraken_osds
4: pool 0 'rbd' replicated size 3 min_size 1 crush_ruleset 0 object_hash
rjenkins pg_num 8 pgp_num 8 last_change 1 flags hashpspool stripe_width 0
4: max_osd 3
4: osd.0 up in weight 1 up_from 176 up_thru 176 down_at 173
last_clean_interval [4,175) 127.0.0.1:6800/45257 127.0.0.1:6812/1045257
127.0.0.1:6813/1045257 127.0.0.1:6814/1045257 exists,up
7688bac7-75ec-4a3c-8edc-ce7245071a90
4: osd.1 up in weight 1 up_from 8 up_thru 176 down_at 0
last_clean_interval [0,0) 127.0.0.1:6804/45271 127.0.0.1:6805/45271
127.0.0.1:6806/45271 127.0.0.1:6807/45271 exists,up
7639c41c-be59-42e4-9eb4-fcb6372e7042
4: osd.2 up in weight 1 up_from 14 up_thru 176 down_at 0
last_clean_interval [0,0) 127.0.0.1:6808/45285 127.0.0.1:6809/45285
127.0.0.1:6810/45285 127.0.0.1:6811/45285 exists,up
39e812a4-2f50-4088-bebd-6392dc05c76c
4: pg_temp 0.2 [1,2]
4: pg_temp 0.6 [1,2]
4: pg_temp 0.7 [1,2]
ceph daemon osd.0 status
4: {
4: "cluster_fsid": "eee1774b-3560-46ec-83b4-bac0e8763e93",
4: "osd_fsid": "7688bac7-75ec-4a3c-8edc-ce7245071a90",
4: "whoami": 0,
4: "state": "booting",
4: "oldest_map": 1,
4: "newest_map": 177,
4: "num_pgs": 8
4: }
ceph -s
4: cluster eee1774b-3560-46ec-83b4-bac0e8763e93
4: health HEALTH_WARN
4: 3 pgs degraded
4: 3 pgs peering
4: 3 pgs undersized
4: monmap e1: 3 mons at
{a=127.0.0.1:7202/0,b=127.0.0.1:7203/0,c=127.0.0.1:7204/0}
4: election epoch 6, quorum 0,1,2 a,b,c
4: osdmap e177: 3 osds: 3 up, 3 in; 3 remapped pgs
4: flags sortbitwise,require_jewel_osds,require_kraken_osds
4: pgmap v298: 8 pgs, 1 pools, 0 bytes data, 0 objects
4: 300 GB used, 349 GB / 650 GB avail
4: 3 peering
4: 3 active+undersized+degraded
4: 2 active+clean
And even though the cluster is reported UP, the osd itself is booting,
and never gets out of that state.
I have reworked the code that actually rebinds the accepters, which
seems to be working. But obviously it doesn't have enough work done to
really go to the active state.
Wido suggested to look for: wrongly marked me down, because that should
start triggering the reconnect state.
Which led me to SimpleMessenger
Does anybody have suggestions as to where to start looking this time??
Thanx,
--WjW
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Inconclusive answer when running tests for cephtool-test-mon.sh
2016-09-05 15:18 ` Willem Jan Withagen
@ 2016-09-06 17:36 ` Gregory Farnum
2016-09-06 17:54 ` Willem Jan Withagen
0 siblings, 1 reply; 4+ messages in thread
From: Gregory Farnum @ 2016-09-06 17:36 UTC (permalink / raw)
To: Willem Jan Withagen; +Cc: Ceph Development
On Mon, Sep 5, 2016 at 8:18 AM, Willem Jan Withagen <wjw@digiware.nl> wrote:
> On 5-9-2016 13:42, Willem Jan Withagen wrote:
>> Hi
>>
>> I would innterested to know why I get 2 different answers:
>>
>> (this is asking the OSDs directly?)
>> ceph osd dump
>> but osd.0 is in, up, up_from 179 up_thru 185 down_at 176
>> osd.1/2 are in, up, up_from 8/13 up_thru 224 down_at 0
>>
>> ceph -s reports 1 OSD down
>>
>> So all osds in the dump are in and up, but ...
>> I guess that osd.0 is telling me when it came back and that it has not
>> all the data that osd.1/2 have because they go up to 224
>>
>> Which is why ceph -s tells me that one osd is down?
>> Or did the leading MON nog get fully informed.
>>
>> What daemon is missing what part of the communication?
>> And what for type of warning/error should I look for in the log files?
>
> A bit of extra info:
>
> The trouble starts with:
> ceph osd down 0
> 4: marked down osd.0.
>
> ceph osd dump
> 4: epoch 174
> 4: fsid eee1774b-3560-46ec-83b4-bac0e8763e93
> 4: created 2016-09-05 16:47:43.301220
> 4: modified 2016-09-05 16:51:16.529826
> 4: flags noup,sortbitwise,require_jewel_osds,require_kraken_osds
> 4: pool 0 'rbd' replicated size 3 min_size 1 crush_ruleset 0 object_hash
> rjenkins pg_num 8 pgp_num 8 last_change 1 flags hashpspool stripe_width 0
> 4: max_osd 3
> 4: osd.0 down in weight 1 up_from 4 up_thru 167 down_at 173
> last_clean_interval [0,0) 127.0.0.1:6800/45257 127.0.0.1:6801/45257
> 127.0.0.1:6802/45257 127.0.0.1:6803/45257 exists
> 7688bac7-75ec-4a3c-8edc-ce7245071a90
> 4: osd.1 up in weight 1 up_from 8 up_thru 173 down_at 0
> last_clean_interval [0,0) 127.0.0.1:6804/45271 127.0.0.1:6805/45271
> 127.0.0.1:6806/45271 127.0.0.1:6807/45271 exists,up
> 7639c41c-be59-42e4-9eb4-fcb6372e7042
> 4: osd.2 up in weight 1 up_from 14 up_thru 173 down_at 0
> last_clean_interval [0,0) 127.0.0.1:6808/45285 127.0.0.1:6809/45285
> 127.0.0.1:6810/45285 127.0.0.1:6811/45285 exists,up
> 39e812a4-2f50-4088-bebd-6392dc05c76c
> 4: pg_temp 0.0 [0,2,1]
> 4: pg_temp 0.1 [2,0,1]
> 4: pg_temp 0.2 [0,1,2]
> 4: pg_temp 0.3 [2,0,1]
> 4: pg_temp 0.4 [0,2,1]
> 4: pg_temp 0.5 [0,2,1]
> 4: pg_temp 0.6 [0,1,2]
> 4: pg_temp 0.7 [1,0,2]
>
> ceph daemon osd.0 status
> 4: {
> 4: "cluster_fsid": "eee1774b-3560-46ec-83b4-bac0e8763e93",
> 4: "osd_fsid": "7688bac7-75ec-4a3c-8edc-ce7245071a90",
> 4: "whoami": 0,
> 4: "state": "active",
> 4: "oldest_map": 1,
> 4: "newest_map": 172,
> 4: "num_pgs": 8
> 4: }
> 4: /home/wjw/wip/qa/workunits/cephtool/test.sh:15: osd_state: ceph -s
> 4: cluster eee1774b-3560-46ec-83b4-bac0e8763e93
> 4: health HEALTH_WARN
> 4: 3 pgs degraded
> 4: 3 pgs stale
> 4: 3 pgs undersized
> 4: 1/3 in osds are down
> 4: noup,sortbitwise,require_jewel_osds,require_kraken_osds
> flag(s) set
> 4: monmap e1: 3 mons at
> {a=127.0.0.1:7202/0,b=127.0.0.1:7203/0,c=127.0.0.1:7204/0}
> 4: election epoch 6, quorum 0,1,2 a,b,c
> 4: osdmap e174: 3 osds: 2 up, 3 in; 8 remapped pgs
> 4: flags noup,sortbitwise,require_jewel_osds,require_kraken_osds
> 4: pgmap v294: 8 pgs, 1 pools, 0 bytes data, 0 objects
> 4: 300 GB used, 349 GB / 650 GB avail
> 4: 3 stale+active+clean
> 4: 3 active+undersized+degraded
> 4: 2 active+clean
>
> And then:
> ceph osd unset noup
> 4: noup is unset
>
> And we wait/loop until çeph osd dump' reports osd.0 up.
>
> ceph osd dump
> 4: epoch 177
> 4: fsid eee1774b-3560-46ec-83b4-bac0e8763e93
> 4: created 2016-09-05 16:47:43.301220
> 4: modified 2016-09-05 16:51:20.098412
> 4: flags sortbitwise,require_jewel_osds,require_kraken_osds
> 4: pool 0 'rbd' replicated size 3 min_size 1 crush_ruleset 0 object_hash
> rjenkins pg_num 8 pgp_num 8 last_change 1 flags hashpspool stripe_width 0
> 4: max_osd 3
> 4: osd.0 up in weight 1 up_from 176 up_thru 176 down_at 173
> last_clean_interval [4,175) 127.0.0.1:6800/45257 127.0.0.1:6812/1045257
> 127.0.0.1:6813/1045257 127.0.0.1:6814/1045257 exists,up
> 7688bac7-75ec-4a3c-8edc-ce7245071a90
> 4: osd.1 up in weight 1 up_from 8 up_thru 176 down_at 0
> last_clean_interval [0,0) 127.0.0.1:6804/45271 127.0.0.1:6805/45271
> 127.0.0.1:6806/45271 127.0.0.1:6807/45271 exists,up
> 7639c41c-be59-42e4-9eb4-fcb6372e7042
> 4: osd.2 up in weight 1 up_from 14 up_thru 176 down_at 0
> last_clean_interval [0,0) 127.0.0.1:6808/45285 127.0.0.1:6809/45285
> 127.0.0.1:6810/45285 127.0.0.1:6811/45285 exists,up
> 39e812a4-2f50-4088-bebd-6392dc05c76c
> 4: pg_temp 0.2 [1,2]
> 4: pg_temp 0.6 [1,2]
> 4: pg_temp 0.7 [1,2]
>
> ceph daemon osd.0 status
> 4: {
> 4: "cluster_fsid": "eee1774b-3560-46ec-83b4-bac0e8763e93",
> 4: "osd_fsid": "7688bac7-75ec-4a3c-8edc-ce7245071a90",
> 4: "whoami": 0,
> 4: "state": "booting",
> 4: "oldest_map": 1,
> 4: "newest_map": 177,
> 4: "num_pgs": 8
> 4: }
>
> ceph -s
> 4: cluster eee1774b-3560-46ec-83b4-bac0e8763e93
> 4: health HEALTH_WARN
> 4: 3 pgs degraded
> 4: 3 pgs peering
> 4: 3 pgs undersized
> 4: monmap e1: 3 mons at
> {a=127.0.0.1:7202/0,b=127.0.0.1:7203/0,c=127.0.0.1:7204/0}
> 4: election epoch 6, quorum 0,1,2 a,b,c
> 4: osdmap e177: 3 osds: 3 up, 3 in; 3 remapped pgs
> 4: flags sortbitwise,require_jewel_osds,require_kraken_osds
> 4: pgmap v298: 8 pgs, 1 pools, 0 bytes data, 0 objects
> 4: 300 GB used, 349 GB / 650 GB avail
> 4: 3 peering
> 4: 3 active+undersized+degraded
> 4: 2 active+clean
>
> And even though the cluster is reported UP, the osd itself is booting,
> and never gets out of that state.
>
> I have reworked the code that actually rebinds the accepters, which
> seems to be working. But obviously it doesn't have enough work done to
> really go to the active state.
> Wido suggested to look for: wrongly marked me down, because that should
> start triggering the reconnect state.
> Which led me to SimpleMessenger
>
> Does anybody have suggestions as to where to start looking this time??
Just at a first guess, I think your rebind changes might have busted
something. I haven't looked at it in any depth but
https://github.com/ceph/ceph/pull/10976 made me nervous when I saw it:
in general we expect a slightly different address to guarantee that
the OSD knows it was previously marked down.
-Greg
>
> Thanx,
> --WjW
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Inconclusive answer when running tests for cephtool-test-mon.sh
2016-09-06 17:36 ` Gregory Farnum
@ 2016-09-06 17:54 ` Willem Jan Withagen
0 siblings, 0 replies; 4+ messages in thread
From: Willem Jan Withagen @ 2016-09-06 17:54 UTC (permalink / raw)
To: Gregory Farnum; +Cc: Ceph Development
On 6-9-2016 19:36, Gregory Farnum wrote:
> On Mon, Sep 5, 2016 at 8:18 AM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>> On 5-9-2016 13:42, Willem Jan Withagen wrote:
>>> Hi
>>>
>>> I would innterested to know why I get 2 different answers:
>>>
>>> (this is asking the OSDs directly?)
>>> ceph osd dump
>>> but osd.0 is in, up, up_from 179 up_thru 185 down_at 176
>>> osd.1/2 are in, up, up_from 8/13 up_thru 224 down_at 0
>>>
>>> ceph -s reports 1 OSD down
>>>
>>> So all osds in the dump are in and up, but ...
>>> I guess that osd.0 is telling me when it came back and that it has not
>>> all the data that osd.1/2 have because they go up to 224
>>>
>>> Which is why ceph -s tells me that one osd is down?
>>> Or did the leading MON nog get fully informed.
>>>
>>> What daemon is missing what part of the communication?
>>> And what for type of warning/error should I look for in the log files?
>>
>> A bit of extra info:
>>
>> The trouble starts with:
>> ceph osd down 0
>> 4: marked down osd.0.
>>
>> ceph osd dump
>> 4: epoch 174
>> 4: fsid eee1774b-3560-46ec-83b4-bac0e8763e93
>> 4: created 2016-09-05 16:47:43.301220
>> 4: modified 2016-09-05 16:51:16.529826
>> 4: flags noup,sortbitwise,require_jewel_osds,require_kraken_osds
>> 4: pool 0 'rbd' replicated size 3 min_size 1 crush_ruleset 0 object_hash
>> rjenkins pg_num 8 pgp_num 8 last_change 1 flags hashpspool stripe_width 0
>> 4: max_osd 3
>> 4: osd.0 down in weight 1 up_from 4 up_thru 167 down_at 173
>> last_clean_interval [0,0) 127.0.0.1:6800/45257 127.0.0.1:6801/45257
>> 127.0.0.1:6802/45257 127.0.0.1:6803/45257 exists
>> 7688bac7-75ec-4a3c-8edc-ce7245071a90
>> 4: osd.1 up in weight 1 up_from 8 up_thru 173 down_at 0
>> last_clean_interval [0,0) 127.0.0.1:6804/45271 127.0.0.1:6805/45271
>> 127.0.0.1:6806/45271 127.0.0.1:6807/45271 exists,up
>> 7639c41c-be59-42e4-9eb4-fcb6372e7042
>> 4: osd.2 up in weight 1 up_from 14 up_thru 173 down_at 0
>> last_clean_interval [0,0) 127.0.0.1:6808/45285 127.0.0.1:6809/45285
>> 127.0.0.1:6810/45285 127.0.0.1:6811/45285 exists,up
>> 39e812a4-2f50-4088-bebd-6392dc05c76c
>> 4: pg_temp 0.0 [0,2,1]
>> 4: pg_temp 0.1 [2,0,1]
>> 4: pg_temp 0.2 [0,1,2]
>> 4: pg_temp 0.3 [2,0,1]
>> 4: pg_temp 0.4 [0,2,1]
>> 4: pg_temp 0.5 [0,2,1]
>> 4: pg_temp 0.6 [0,1,2]
>> 4: pg_temp 0.7 [1,0,2]
>>
>> ceph daemon osd.0 status
>> 4: {
>> 4: "cluster_fsid": "eee1774b-3560-46ec-83b4-bac0e8763e93",
>> 4: "osd_fsid": "7688bac7-75ec-4a3c-8edc-ce7245071a90",
>> 4: "whoami": 0,
>> 4: "state": "active",
>> 4: "oldest_map": 1,
>> 4: "newest_map": 172,
>> 4: "num_pgs": 8
>> 4: }
>> 4: /home/wjw/wip/qa/workunits/cephtool/test.sh:15: osd_state: ceph -s
>> 4: cluster eee1774b-3560-46ec-83b4-bac0e8763e93
>> 4: health HEALTH_WARN
>> 4: 3 pgs degraded
>> 4: 3 pgs stale
>> 4: 3 pgs undersized
>> 4: 1/3 in osds are down
>> 4: noup,sortbitwise,require_jewel_osds,require_kraken_osds
>> flag(s) set
>> 4: monmap e1: 3 mons at
>> {a=127.0.0.1:7202/0,b=127.0.0.1:7203/0,c=127.0.0.1:7204/0}
>> 4: election epoch 6, quorum 0,1,2 a,b,c
>> 4: osdmap e174: 3 osds: 2 up, 3 in; 8 remapped pgs
>> 4: flags noup,sortbitwise,require_jewel_osds,require_kraken_osds
>> 4: pgmap v294: 8 pgs, 1 pools, 0 bytes data, 0 objects
>> 4: 300 GB used, 349 GB / 650 GB avail
>> 4: 3 stale+active+clean
>> 4: 3 active+undersized+degraded
>> 4: 2 active+clean
>>
>> And then:
>> ceph osd unset noup
>> 4: noup is unset
>>
>> And we wait/loop until çeph osd dump' reports osd.0 up.
>>
>> ceph osd dump
>> 4: epoch 177
>> 4: fsid eee1774b-3560-46ec-83b4-bac0e8763e93
>> 4: created 2016-09-05 16:47:43.301220
>> 4: modified 2016-09-05 16:51:20.098412
>> 4: flags sortbitwise,require_jewel_osds,require_kraken_osds
>> 4: pool 0 'rbd' replicated size 3 min_size 1 crush_ruleset 0 object_hash
>> rjenkins pg_num 8 pgp_num 8 last_change 1 flags hashpspool stripe_width 0
>> 4: max_osd 3
>> 4: osd.0 up in weight 1 up_from 176 up_thru 176 down_at 173
>> last_clean_interval [4,175) 127.0.0.1:6800/45257 127.0.0.1:6812/1045257
>> 127.0.0.1:6813/1045257 127.0.0.1:6814/1045257 exists,up
>> 7688bac7-75ec-4a3c-8edc-ce7245071a90
>> 4: osd.1 up in weight 1 up_from 8 up_thru 176 down_at 0
>> last_clean_interval [0,0) 127.0.0.1:6804/45271 127.0.0.1:6805/45271
>> 127.0.0.1:6806/45271 127.0.0.1:6807/45271 exists,up
>> 7639c41c-be59-42e4-9eb4-fcb6372e7042
>> 4: osd.2 up in weight 1 up_from 14 up_thru 176 down_at 0
>> last_clean_interval [0,0) 127.0.0.1:6808/45285 127.0.0.1:6809/45285
>> 127.0.0.1:6810/45285 127.0.0.1:6811/45285 exists,up
>> 39e812a4-2f50-4088-bebd-6392dc05c76c
>> 4: pg_temp 0.2 [1,2]
>> 4: pg_temp 0.6 [1,2]
>> 4: pg_temp 0.7 [1,2]
>>
>> ceph daemon osd.0 status
>> 4: {
>> 4: "cluster_fsid": "eee1774b-3560-46ec-83b4-bac0e8763e93",
>> 4: "osd_fsid": "7688bac7-75ec-4a3c-8edc-ce7245071a90",
>> 4: "whoami": 0,
>> 4: "state": "booting",
>> 4: "oldest_map": 1,
>> 4: "newest_map": 177,
>> 4: "num_pgs": 8
>> 4: }
>>
>> ceph -s
>> 4: cluster eee1774b-3560-46ec-83b4-bac0e8763e93
>> 4: health HEALTH_WARN
>> 4: 3 pgs degraded
>> 4: 3 pgs peering
>> 4: 3 pgs undersized
>> 4: monmap e1: 3 mons at
>> {a=127.0.0.1:7202/0,b=127.0.0.1:7203/0,c=127.0.0.1:7204/0}
>> 4: election epoch 6, quorum 0,1,2 a,b,c
>> 4: osdmap e177: 3 osds: 3 up, 3 in; 3 remapped pgs
>> 4: flags sortbitwise,require_jewel_osds,require_kraken_osds
>> 4: pgmap v298: 8 pgs, 1 pools, 0 bytes data, 0 objects
>> 4: 300 GB used, 349 GB / 650 GB avail
>> 4: 3 peering
>> 4: 3 active+undersized+degraded
>> 4: 2 active+clean
>>
>> And even though the cluster is reported UP, the osd itself is booting,
>> and never gets out of that state.
>>
>> I have reworked the code that actually rebinds the accepters, which
>> seems to be working. But obviously it doesn't have enough work done to
>> really go to the active state.
>> Wido suggested to look for: wrongly marked me down, because that should
>> start triggering the reconnect state.
>> Which led me to SimpleMessenger
>>
>> Does anybody have suggestions as to where to start looking this time??
>
> Just at a first guess, I think your rebind changes might have busted
> something. I haven't looked at it in any depth but
> https://github.com/ceph/ceph/pull/10976 made me nervous when I saw
> it: in general we expect a slightly different address to guarantee
> that the OSD knows it was previously marked down.
Right,
Wat you say is the other side of what I noticed: During the rebind there
are a few attempts to actually get connected, causing a rather long
delay before the rebind completes. And the test did not complete also.
The first part of the change is:
https://github.com/ceph/ceph/pull/10720
Which is really needed because shutdown() on a socket-listner does not
work on FreeBSD.
I'll take the change (#10976) out, and start watching more careful for
the rebinding. And start working from the state machine why it does not
leave booting...
Thanx for the suggestion.
--WjW
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2016-09-06 17:54 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-09-05 11:42 Inconclusive answer when running tests for cephtool-test-mon.sh Willem Jan Withagen
2016-09-05 15:18 ` Willem Jan Withagen
2016-09-06 17:36 ` Gregory Farnum
2016-09-06 17:54 ` Willem Jan Withagen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.