* luminous filesystem is degraded
@ 2017-09-03 13:14 Two Spirit
2017-09-04 9:05 ` John Spray
0 siblings, 1 reply; 7+ messages in thread
From: Two Spirit @ 2017-09-03 13:14 UTC (permalink / raw)
To: ceph-devel
Setup: luminous on
Ubuntu 14.04/16.04 mix. 5 OSD. all up. 3 or 4 mds, 3mon,cephx
rebooting all 6 ceph systems did not clear the problem. Failure
occurred within 6 hours of start of test.
similar stress test with 4OSD,1MDS,1MON,cephx worked fine.
stress test
# cp * /mnt/cephfs
# ceph -s
health: HEALTH_WARN
1 filesystem is degraded
crush map has straw_calc_version=0
1/731529 unfound (0.000%)
Degraded data redundancy: 22519/1463058 objects degraded
(1.539%), 2 pgs unclean, 2 pgs degraded, 1 pg undersized
services:
mon: 3 daemons, quorum xxx233,xxx266,xxx272
mgr: xxx266(active)
mds: cephfs-1/1/1 up {0=xxx233=up:replay}, 3 up:standby
osd: 5 osds: 5 up, 5 in
rgw: 1 daemon active
# ceph mds dump
dumped fsmap epoch 590
fs_name cephfs
epoch 589
flags c
created 2017-08-24 14:35:33.735399
modified 2017-08-24 14:35:33.735400
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
last_failure 0
last_failure_osd_epoch 1573
compat compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in
separate object,5=mds uses versioned encoding,6=dirfrag is stored in
omap,8=file layout v2}
max_mds 1
in 0
up {0=579217}
failed
damaged
stopped
data_pools [5]
metadata_pool 6
inline_data disabled
balancer
standby_count_wanted 1
579217: x.x.x.233:6804/1176521332 'xxx233' mds.0.589 up:replay seq 2
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous filesystem is degraded
2017-09-03 13:14 luminous filesystem is degraded Two Spirit
@ 2017-09-04 9:05 ` John Spray
2017-09-04 15:45 ` Two Spirit
0 siblings, 1 reply; 7+ messages in thread
From: John Spray @ 2017-09-04 9:05 UTC (permalink / raw)
To: Two Spirit; +Cc: ceph-devel
On Sun, Sep 3, 2017 at 2:14 PM, Two Spirit <twospirit6905@gmail.com> wrote:
> Setup: luminous on
> Ubuntu 14.04/16.04 mix. 5 OSD. all up. 3 or 4 mds, 3mon,cephx
> rebooting all 6 ceph systems did not clear the problem. Failure
> occurred within 6 hours of start of test.
> similar stress test with 4OSD,1MDS,1MON,cephx worked fine.
>
>
> stress test
> # cp * /mnt/cephfs
>
> # ceph -s
> health: HEALTH_WARN
> 1 filesystem is degraded
> crush map has straw_calc_version=0
> 1/731529 unfound (0.000%)
> Degraded data redundancy: 22519/1463058 objects degraded
> (1.539%), 2 pgs unclean, 2 pgs degraded, 1 pg undersized
>
> services:
> mon: 3 daemons, quorum xxx233,xxx266,xxx272
> mgr: xxx266(active)
> mds: cephfs-1/1/1 up {0=xxx233=up:replay}, 3 up:standby
> osd: 5 osds: 5 up, 5 in
> rgw: 1 daemon active
Your MDS is probably stuck in the replay state because it can't read
from one of your degraded PGs. Given that you have all your OSDs in,
but one of your PGs is undersized (i.e. is short on OSDs), I would
guess that something is wrong with your choice of CRUSH rules or EC
config.
John
>
> # ceph mds dump
> dumped fsmap epoch 590
> fs_name cephfs
> epoch 589
> flags c
> created 2017-08-24 14:35:33.735399
> modified 2017-08-24 14:35:33.735400
> tableserver 0
> root 0
> session_timeout 60
> session_autoclose 300
> max_file_size 1099511627776
> last_failure 0
> last_failure_osd_epoch 1573
> compat compat={},rocompat={},incompat={1=base v0.20,2=client
> writeable ranges,3=default file layouts on dirs,4=dir inode in
> separate object,5=mds uses versioned encoding,6=dirfrag is stored in
> omap,8=file layout v2}
> max_mds 1
> in 0
> up {0=579217}
> failed
> damaged
> stopped
> data_pools [5]
> metadata_pool 6
> inline_data disabled
> balancer
> standby_count_wanted 1
> 579217: x.x.x.233:6804/1176521332 'xxx233' mds.0.589 up:replay seq 2
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous filesystem is degraded
2017-09-04 9:05 ` John Spray
@ 2017-09-04 15:45 ` Two Spirit
2017-09-05 13:33 ` Sage Weil
0 siblings, 1 reply; 7+ messages in thread
From: Two Spirit @ 2017-09-04 15:45 UTC (permalink / raw)
To: John Spray; +Cc: ceph-devel
Thanks for the info. I'm stumped what to do right now to get back to
an operation cluster -- still trying to find documentation on how to
recover.
1) I have not yet modified any CRUSH rules from the defaults. I have
one ubuntu 14.04 OSD in the mix, and I had to set "ceph osd crush
tunables legacy" just to get it to work.
2) I have not yet implemented any Erasure Code pool. That is probably
one of the next tests I was going to do. I'm still testing with basic
replication.
The degraded data redundancy seems to be stuck and not reducing
anymore. If I manually clear [if this is even possible] the 1 pg
undersized, should my degraded filesystem go back online?
On Mon, Sep 4, 2017 at 2:05 AM, John Spray <jspray@redhat.com> wrote:
> On Sun, Sep 3, 2017 at 2:14 PM, Two Spirit <twospirit6905@gmail.com> wrote:
>> Setup: luminous on
>> Ubuntu 14.04/16.04 mix. 5 OSD. all up. 3 or 4 mds, 3mon,cephx
>> rebooting all 6 ceph systems did not clear the problem. Failure
>> occurred within 6 hours of start of test.
>> similar stress test with 4OSD,1MDS,1MON,cephx worked fine.
>>
>>
>> stress test
>> # cp * /mnt/cephfs
>>
>> # ceph -s
>> health: HEALTH_WARN
>> 1 filesystem is degraded
>> crush map has straw_calc_version=0
>> 1/731529 unfound (0.000%)
>> Degraded data redundancy: 22519/1463058 objects degraded
>> (1.539%), 2 pgs unclean, 2 pgs degraded, 1 pg undersized
>>
>> services:
>> mon: 3 daemons, quorum xxx233,xxx266,xxx272
>> mgr: xxx266(active)
>> mds: cephfs-1/1/1 up {0=xxx233=up:replay}, 3 up:standby
>> osd: 5 osds: 5 up, 5 in
>> rgw: 1 daemon active
>
> Your MDS is probably stuck in the replay state because it can't read
> from one of your degraded PGs. Given that you have all your OSDs in,
> but one of your PGs is undersized (i.e. is short on OSDs), I would
> guess that something is wrong with your choice of CRUSH rules or EC
> config.
>
> John
>
>>
>> # ceph mds dump
>> dumped fsmap epoch 590
>> fs_name cephfs
>> epoch 589
>> flags c
>> created 2017-08-24 14:35:33.735399
>> modified 2017-08-24 14:35:33.735400
>> tableserver 0
>> root 0
>> session_timeout 60
>> session_autoclose 300
>> max_file_size 1099511627776
>> last_failure 0
>> last_failure_osd_epoch 1573
>> compat compat={},rocompat={},incompat={1=base v0.20,2=client
>> writeable ranges,3=default file layouts on dirs,4=dir inode in
>> separate object,5=mds uses versioned encoding,6=dirfrag is stored in
>> omap,8=file layout v2}
>> max_mds 1
>> in 0
>> up {0=579217}
>> failed
>> damaged
>> stopped
>> data_pools [5]
>> metadata_pool 6
>> inline_data disabled
>> balancer
>> standby_count_wanted 1
>> 579217: x.x.x.233:6804/1176521332 'xxx233' mds.0.589 up:replay seq 2
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous filesystem is degraded
2017-09-04 15:45 ` Two Spirit
@ 2017-09-05 13:33 ` Sage Weil
2017-09-12 18:31 ` Two Spirit
0 siblings, 1 reply; 7+ messages in thread
From: Sage Weil @ 2017-09-05 13:33 UTC (permalink / raw)
To: Two Spirit; +Cc: John Spray, ceph-devel
On Mon, 4 Sep 2017, Two Spirit wrote:
> Thanks for the info. I'm stumped what to do right now to get back to
> an operation cluster -- still trying to find documentation on how to
> recover.
>
>
> 1) I have not yet modified any CRUSH rules from the defaults. I have
> one ubuntu 14.04 OSD in the mix, and I had to set "ceph osd crush
> tunables legacy" just to get it to work.
>
> 2) I have not yet implemented any Erasure Code pool. That is probably
> one of the next tests I was going to do. I'm still testing with basic
> replication.
Can you attach 'ceph health detail', 'ceph osd crush dump', and 'ceph osd
dump'?
> The degraded data redundancy seems to be stuck and not reducing
> anymore. If I manually clear [if this is even possible] the 1 pg
> undersized, should my degraded filesystem go back online?
The problem is likely the 1 unfound object. Are there any OSDs that are
down that failed recently? (Try 'ceph osd tree down' to see a simple
summary.)
sage
>
> On Mon, Sep 4, 2017 at 2:05 AM, John Spray <jspray@redhat.com> wrote:
> > On Sun, Sep 3, 2017 at 2:14 PM, Two Spirit <twospirit6905@gmail.com> wrote:
> >> Setup: luminous on
> >> Ubuntu 14.04/16.04 mix. 5 OSD. all up. 3 or 4 mds, 3mon,cephx
> >> rebooting all 6 ceph systems did not clear the problem. Failure
> >> occurred within 6 hours of start of test.
> >> similar stress test with 4OSD,1MDS,1MON,cephx worked fine.
> >>
> >>
> >> stress test
> >> # cp * /mnt/cephfs
> >>
> >> # ceph -s
> >> health: HEALTH_WARN
> >> 1 filesystem is degraded
> >> crush map has straw_calc_version=0
> >> 1/731529 unfound (0.000%)
> >> Degraded data redundancy: 22519/1463058 objects degraded
> >> (1.539%), 2 pgs unclean, 2 pgs degraded, 1 pg undersized
> >>
> >> services:
> >> mon: 3 daemons, quorum xxx233,xxx266,xxx272
> >> mgr: xxx266(active)
> >> mds: cephfs-1/1/1 up {0=xxx233=up:replay}, 3 up:standby
> >> osd: 5 osds: 5 up, 5 in
> >> rgw: 1 daemon active
> >
> > Your MDS is probably stuck in the replay state because it can't read
> > from one of your degraded PGs. Given that you have all your OSDs in,
> > but one of your PGs is undersized (i.e. is short on OSDs), I would
> > guess that something is wrong with your choice of CRUSH rules or EC
> > config.
> >
> > John
> >
> >>
> >> # ceph mds dump
> >> dumped fsmap epoch 590
> >> fs_name cephfs
> >> epoch 589
> >> flags c
> >> created 2017-08-24 14:35:33.735399
> >> modified 2017-08-24 14:35:33.735400
> >> tableserver 0
> >> root 0
> >> session_timeout 60
> >> session_autoclose 300
> >> max_file_size 1099511627776
> >> last_failure 0
> >> last_failure_osd_epoch 1573
> >> compat compat={},rocompat={},incompat={1=base v0.20,2=client
> >> writeable ranges,3=default file layouts on dirs,4=dir inode in
> >> separate object,5=mds uses versioned encoding,6=dirfrag is stored in
> >> omap,8=file layout v2}
> >> max_mds 1
> >> in 0
> >> up {0=579217}
> >> failed
> >> damaged
> >> stopped
> >> data_pools [5]
> >> metadata_pool 6
> >> inline_data disabled
> >> balancer
> >> standby_count_wanted 1
> >> 579217: x.x.x.233:6804/1176521332 'xxx233' mds.0.589 up:replay seq 2
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous filesystem is degraded
2017-09-05 13:33 ` Sage Weil
@ 2017-09-12 18:31 ` Two Spirit
2017-09-12 19:51 ` Sage Weil
0 siblings, 1 reply; 7+ messages in thread
From: Two Spirit @ 2017-09-12 18:31 UTC (permalink / raw)
To: Sage Weil; +Cc: John Spray, ceph-devel
[-- Attachment #1: Type: text/plain, Size: 4299 bytes --]
I don't have any OSDs that are down, so the 1 unfound object I think
needs to be manually cleared. I ran across a webpage a while ago that
talked about how to clear it, but if you have a reference, would save
me a little time.
I've included the outputs of the commands you asked. The ceph test
network contains 6 osds, 3 mons, 3 mds, 1rgw 1mgr. ubuntu 64 bit
14.04/16.04 mix.
file system is degraded. Are there procedures how to get this back in operation?
On Tue, Sep 5, 2017 at 6:33 AM, Sage Weil <sweil@redhat.com> wrote:
> On Mon, 4 Sep 2017, Two Spirit wrote:
>> Thanks for the info. I'm stumped what to do right now to get back to
>> an operation cluster -- still trying to find documentation on how to
>> recover.
>>
>>
>> 1) I have not yet modified any CRUSH rules from the defaults. I have
>> one ubuntu 14.04 OSD in the mix, and I had to set "ceph osd crush
>> tunables legacy" just to get it to work.
>>
>> 2) I have not yet implemented any Erasure Code pool. That is probably
>> one of the next tests I was going to do. I'm still testing with basic
>> replication.
>
> Can you attach 'ceph health detail', 'ceph osd crush dump', and 'ceph osd
> dump'?
>
>> The degraded data redundancy seems to be stuck and not reducing
>> anymore. If I manually clear [if this is even possible] the 1 pg
>> undersized, should my degraded filesystem go back online?
>
> The problem is likely the 1 unfound object. Are there any OSDs that are
> down that failed recently? (Try 'ceph osd tree down' to see a simple
> summary.)
>
> sage
>
>
>>
>> On Mon, Sep 4, 2017 at 2:05 AM, John Spray <jspray@redhat.com> wrote:
>> > On Sun, Sep 3, 2017 at 2:14 PM, Two Spirit <twospirit6905@gmail.com> wrote:
>> >> Setup: luminous on
>> >> Ubuntu 14.04/16.04 mix. 5 OSD. all up. 3 or 4 mds, 3mon,cephx
>> >> rebooting all 6 ceph systems did not clear the problem. Failure
>> >> occurred within 6 hours of start of test.
>> >> similar stress test with 4OSD,1MDS,1MON,cephx worked fine.
>> >>
>> >>
>> >> stress test
>> >> # cp * /mnt/cephfs
>> >>
>> >> # ceph -s
>> >> health: HEALTH_WARN
>> >> 1 filesystem is degraded
>> >> crush map has straw_calc_version=0
>> >> 1/731529 unfound (0.000%)
>> >> Degraded data redundancy: 22519/1463058 objects degraded
>> >> (1.539%), 2 pgs unclean, 2 pgs degraded, 1 pg undersized
>> >>
>> >> services:
>> >> mon: 3 daemons, quorum xxx233,xxx266,xxx272
>> >> mgr: xxx266(active)
>> >> mds: cephfs-1/1/1 up {0=xxx233=up:replay}, 3 up:standby
>> >> osd: 5 osds: 5 up, 5 in
>> >> rgw: 1 daemon active
>> >
>> > Your MDS is probably stuck in the replay state because it can't read
>> > from one of your degraded PGs. Given that you have all your OSDs in,
>> > but one of your PGs is undersized (i.e. is short on OSDs), I would
>> > guess that something is wrong with your choice of CRUSH rules or EC
>> > config.
>> >
>> > John
>> >
>> >>
>> >> # ceph mds dump
>> >> dumped fsmap epoch 590
>> >> fs_name cephfs
>> >> epoch 589
>> >> flags c
>> >> created 2017-08-24 14:35:33.735399
>> >> modified 2017-08-24 14:35:33.735400
>> >> tableserver 0
>> >> root 0
>> >> session_timeout 60
>> >> session_autoclose 300
>> >> max_file_size 1099511627776
>> >> last_failure 0
>> >> last_failure_osd_epoch 1573
>> >> compat compat={},rocompat={},incompat={1=base v0.20,2=client
>> >> writeable ranges,3=default file layouts on dirs,4=dir inode in
>> >> separate object,5=mds uses versioned encoding,6=dirfrag is stored in
>> >> omap,8=file layout v2}
>> >> max_mds 1
>> >> in 0
>> >> up {0=579217}
>> >> failed
>> >> damaged
>> >> stopped
>> >> data_pools [5]
>> >> metadata_pool 6
>> >> inline_data disabled
>> >> balancer
>> >> standby_count_wanted 1
>> >> 579217: x.x.x.233:6804/1176521332 'xxx233' mds.0.589 up:replay seq 2
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
[-- Attachment #2: ceph_health_detail.out --]
[-- Type: application/octet-stream, Size: 2742 bytes --]
HEALTH_WARN 1 filesystem is degraded; crush map has straw_calc_version=0; 177902/1463058 objects misplaced (12.160%); 1/731529 unfound (0.000%); Degraded data redundancy: 22519/1463058 objects degraded (1.539%), 16 pgs unclean, 2 pgs degraded, 1 pg undersized; clock skew detected on mon.osdmonmgr66, mon.osdmon72
FS_DEGRADED 1 filesystem is degraded
fs cephfs is degraded
OLD_CRUSH_STRAW_CALC_VERSION crush map has straw_calc_version=0
see http://docs.ceph.com/docs/master/rados/operations/crush-map/#tunables
OBJECT_MISPLACED 177902/1463058 objects misplaced (12.160%)
OBJECT_UNFOUND 1/731529 unfound (0.000%)
pg 6.2 has 1 unfound objects
PG_DEGRADED Degraded data redundancy: 22519/1463058 objects degraded (1.539%), 16 pgs unclean, 2 pgs degraded, 1 pg undersized
pg 4.5 is stuck unclean for 1342.896172, current state active+remapped+backfill_wait, last acting [1,3]
pg 5.3 is stuck unclean for 1342.976130, current state active+remapped+backfilling, last acting [2,1]
pg 5.6 is stuck unclean for 1342.975955, current state active+remapped+backfill_wait, last acting [2,5]
pg 5.c is stuck unclean for 1342.971200, current state active+remapped+backfill_wait, last acting [3,1]
pg 5.11 is stuck undersized for 7774.128674, current state active+undersized+degraded, last acting [1]
pg 5.14 is stuck unclean for 1342.223608, current state active+remapped+backfill_wait, last acting [3,2]
pg 5.1a is stuck unclean for 1342.959107, current state active+remapped+backfill_wait, last acting [0,3]
pg 5.1b is stuck unclean for 1343.006102, current state active+remapped+backfill_wait, last acting [3,0]
pg 5.1c is stuck unclean for 1342.976034, current state active+remapped+backfill_wait, last acting [2,0]
pg 5.1e is stuck unclean for 1342.959184, current state active+remapped+backfill_wait, last acting [0,1]
pg 6.0 is stuck unclean for 1343.006407, current state active+remapped+backfill_wait, last acting [3,0]
pg 6.2 is active+recovery_wait+degraded+remapped, acting [3,1], 1 unfound
pg 6.3 is stuck unclean for 1342.896018, current state active+remapped+backfill_wait, last acting [1,2]
pg 6.9 is stuck unclean for 1086.179526, current state active+remapped+backfill_wait, last acting [3,4]
pg 6.12 is stuck unclean for 1342.976018, current state active+remapped+backfill_wait, last acting [2,3]
pg 6.1c is stuck unclean for 1342.975969, current state active+remapped+backfill_wait, last acting [2,1]
MON_CLOCK_SKEW clock skew detected on mon.osdmonmgr66, mon.osdmon72
mon.osdmonmgr66 addr 10.1.2.66:6789/0 clock skew 0.0806478s > max 0.05s (latency 0.00741028s)
mon.osdmon72 addr 10.1.2.72:6789/0 clock skew 0.0820921s > max 0.05s (latency 0.0109876s)
[-- Attachment #3: ceph_osd_crush_dump.out --]
[-- Type: application/octet-stream, Size: 9686 bytes --]
{
"devices": [
{
"id": 0,
"name": "osd.0",
"class": "hdd"
},
{
"id": 1,
"name": "osd.1",
"class": "hdd"
},
{
"id": 2,
"name": "osd.2",
"class": "hdd"
},
{
"id": 3,
"name": "osd.3",
"class": "hdd"
},
{
"id": 4,
"name": "osd.4",
"class": "hdd"
},
{
"id": 5,
"name": "osd.5",
"class": "hdd"
}
],
"types": [
{
"type_id": 0,
"name": "osd"
},
{
"type_id": 1,
"name": "host"
},
{
"type_id": 2,
"name": "chassis"
},
{
"type_id": 3,
"name": "rack"
},
{
"type_id": 4,
"name": "row"
},
{
"type_id": 5,
"name": "pdu"
},
{
"type_id": 6,
"name": "pod"
},
{
"type_id": 7,
"name": "room"
},
{
"type_id": 8,
"name": "datacenter"
},
{
"type_id": 9,
"name": "region"
},
{
"type_id": 10,
"name": "root"
}
],
"buckets": [
{
"id": -1,
"name": "default",
"type_id": 10,
"type_name": "root",
"weight": 417284,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": -3,
"weight": 89417,
"pos": 0
},
{
"id": -5,
"weight": 119229,
"pos": 1
},
{
"id": -7,
"weight": 59611,
"pos": 2
},
{
"id": -9,
"weight": 59611,
"pos": 3
},
{
"id": -11,
"weight": 44708,
"pos": 4
},
{
"id": -13,
"weight": 44708,
"pos": 5
}
]
},
{
"id": -2,
"name": "default~hdd",
"type_id": 10,
"type_name": "root",
"weight": 417284,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": -4,
"weight": 89417,
"pos": 0
},
{
"id": -6,
"weight": 119229,
"pos": 1
},
{
"id": -8,
"weight": 59611,
"pos": 2
},
{
"id": -10,
"weight": 59611,
"pos": 3
},
{
"id": -12,
"weight": 44708,
"pos": 4
},
{
"id": -14,
"weight": 44708,
"pos": 5
}
]
},
{
"id": -3,
"name": "osdmonmgr66",
"type_id": 1,
"type_name": "host",
"weight": 89417,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": 0,
"weight": 89417,
"pos": 0
}
]
},
{
"id": -4,
"name": "osdmonmgr66~hdd",
"type_id": 1,
"type_name": "host",
"weight": 89417,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": 0,
"weight": 89417,
"pos": 0
}
]
},
{
"id": -5,
"name": "osdmon33",
"type_id": 1,
"type_name": "host",
"weight": 119229,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": 1,
"weight": 119229,
"pos": 0
}
]
},
{
"id": -6,
"name": "osdmon33~hdd",
"type_id": 1,
"type_name": "host",
"weight": 119229,
"alg": "straw2",
"hash": "rjenkins1",
"items": [
{
"id": 1,
"weight": 119229,
"pos": 0
}
]
},
{
"id": -7,
"name": "osdmon72",
"type_id": 1,
"type_name": "host",
"weight": 59611,
"alg": "straw",
"hash": "rjenkins1",
"items": [
{
"id": 2,
"weight": 59611,
"pos": 0
}
]
},
{
"id": -8,
"name": "osdmon72~hdd",
"type_id": 1,
"type_name": "host",
"weight": 59611,
"alg": "straw",
"hash": "rjenkins1",
"items": [
{
"id": 2,
"weight": 59611,
"pos": 0
}
]
},
{
"id": -9,
"name": "osd43",
"type_id": 1,
"type_name": "host",
"weight": 59611,
"alg": "straw",
"hash": "rjenkins1",
"items": [
{
"id": 3,
"weight": 59611,
"pos": 0
}
]
},
{
"id": -10,
"name": "osd43~hdd",
"type_id": 1,
"type_name": "host",
"weight": 59611,
"alg": "straw",
"hash": "rjenkins1",
"items": [
{
"id": 3,
"weight": 59611,
"pos": 0
}
]
},
{
"id": -11,
"name": "osd40",
"type_id": 1,
"type_name": "host",
"weight": 44708,
"alg": "straw",
"hash": "rjenkins1",
"items": [
{
"id": 5,
"weight": 44708,
"pos": 0
}
]
},
{
"id": -12,
"name": "osd40~hdd",
"type_id": 1,
"type_name": "host",
"weight": 44708,
"alg": "straw",
"hash": "rjenkins1",
"items": [
{
"id": 5,
"weight": 44708,
"pos": 0
}
]
},
{
"id": -13,
"name": "osd59",
"type_id": 1,
"type_name": "host",
"weight": 44708,
"alg": "straw",
"hash": "rjenkins1",
"items": [
{
"id": 4,
"weight": 44708,
"pos": 0
}
]
},
{
"id": -14,
"name": "osd59~hdd",
"type_id": 1,
"type_name": "host",
"weight": 44708,
"alg": "straw",
"hash": "rjenkins1",
"items": [
{
"id": 4,
"weight": 44708,
"pos": 0
}
]
}
],
"rules": [
{
"rule_id": 0,
"rule_name": "replicated_rule",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "choose_firstn",
"num": 0,
"type": "osd"
},
{
"op": "emit"
}
]
}
],
"tunables": {
"choose_local_tries": 2,
"choose_local_fallback_tries": 5,
"choose_total_tries": 19,
"chooseleaf_descend_once": 0,
"chooseleaf_vary_r": 0,
"chooseleaf_stable": 0,
"straw_calc_version": 0,
"allowed_bucket_algs": 22,
"profile": "argonaut",
"optimal_tunables": 0,
"legacy_tunables": 1,
"minimum_required_version": "hammer",
"require_feature_tunables": 0,
"require_feature_tunables2": 0,
"has_v2_rules": 0,
"require_feature_tunables3": 0,
"has_v3_rules": 0,
"has_v4_buckets": 1,
"require_feature_tunables5": 0,
"has_v5_rules": 0
},
"choose_args": {}
}
[-- Attachment #4: ceph_osd_dump.out --]
[-- Type: application/octet-stream, Size: 3093 bytes --]
epoch 1667
fsid d432ca51-9141-4853-b6b9-7c22f621e9c5
created 2017-08-24 13:47:57.530896
modified 2017-09-12 10:45:31.097347
flags sortbitwise,recovery_deletes
crush_version 20
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client jewel
min_compat_client hammer
require_osd_release luminous
pool 1 '.rgw.root' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 11 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 2 'default.rgw.control' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 13 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 3 'default.rgw.meta' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 15 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 4 'default.rgw.log' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 17 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 5 'cephfs_data' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 1200 lfor 0/1194 stripe_width 0 application cephfs
pool 6 'cephfs_metadata' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 1207 lfor 0/1191 flags hashpspool stripe_width 0 application cephfs
max_osd 6
osd.0 up in weight 1 up_from 1621 up_thru 1664 down_at 1613 last_clean_interval [1590,1612) 10.1.2.66:6801/1154 10.1.2.66:6802/1154 10.1.2.66:6803/1154 10.1.2.66:6804/1154 exists,up e345f131-3603-4446-9de8-e6f64a7356fd
osd.1 up in weight 1 up_from 1588 up_thru 1664 down_at 1587 last_clean_interval [1567,1585) 10.1.2.33:6800/1771 10.1.2.33:6801/1771 10.1.2.33:6802/1771 10.1.2.33:6803/1771 exists,up 15f9a4b3-c088-4882-a9dd-443fa0466367
osd.2 up in weight 1 up_from 1600 up_thru 1664 down_at 1577 last_clean_interval [1465,1576) 10.1.2.72:6801/3879 10.1.2.72:6802/3879 10.1.2.72:6803/3879 10.1.2.72:6804/3879 exists,up 059c4895-03db-418b-a596-d6bb7071b39b
osd.3 up in weight 1 up_from 1638 up_thru 1666 down_at 1636 last_clean_interval [1633,1637) 10.1.2.43:6800/4353 10.1.2.43:6804/1004353 10.1.2.43:6805/1004353 10.1.2.43:6806/1004353 exists,up 3abaf555-a684-4182-b592-4476e75367f6
osd.4 up in weight 1 up_from 1653 up_thru 1662 down_at 1649 last_clean_interval [1645,1648) 10.1.2.59:6800/1671 10.1.2.59:6801/1671 10.1.2.59:6802/1671 10.1.2.59:6803/1671 exists,up fd2cc782-49f3-4f32-bdf8-331d4fee880f
osd.5 up in weight 1 up_from 1626 up_thru 1666 down_at 1589 last_clean_interval [1556,1585) 10.1.2.40:6800/5921 10.1.2.40:6801/5921 10.1.2.40:6802/5921 10.1.2.40:6803/5921 exists,up ddbc4815-541c-451d-a478-712863c529a6
pg_temp 4.5 [1,3]
pg_temp 5.3 [2,1]
pg_temp 5.6 [2,5]
pg_temp 5.c [3,1]
pg_temp 5.14 [3,2]
pg_temp 5.1a [0,3]
pg_temp 5.1b [3,0]
pg_temp 5.1c [2,0]
pg_temp 5.1e [0,1]
pg_temp 6.0 [3,0]
pg_temp 6.2 [3,1]
pg_temp 6.3 [1,2]
pg_temp 6.9 [3,4]
pg_temp 6.12 [2,3]
pg_temp 6.1c [2,1]
[-- Attachment #5: ceph_osd_tree_down.out --]
[-- Type: application/octet-stream, Size: 51 bytes --]
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous filesystem is degraded
2017-09-12 18:31 ` Two Spirit
@ 2017-09-12 19:51 ` Sage Weil
2017-09-13 22:16 ` Two Spirit
0 siblings, 1 reply; 7+ messages in thread
From: Sage Weil @ 2017-09-12 19:51 UTC (permalink / raw)
To: Two Spirit; +Cc: John Spray, ceph-devel
On Tue, 12 Sep 2017, Two Spirit wrote:
> I don't have any OSDs that are down, so the 1 unfound object I think
> needs to be manually cleared. I ran across a webpage a while ago that
> talked about how to clear it, but if you have a reference, would save
> me a little time.
http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/#failures-osd-unfound
sage
> I've included the outputs of the commands you asked. The ceph test
> network contains 6 osds, 3 mons, 3 mds, 1rgw 1mgr. ubuntu 64 bit
> 14.04/16.04 mix.
>
> file system is degraded. Are there procedures how to get this back in operation?
>
> On Tue, Sep 5, 2017 at 6:33 AM, Sage Weil <sweil@redhat.com> wrote:
> > On Mon, 4 Sep 2017, Two Spirit wrote:
> >> Thanks for the info. I'm stumped what to do right now to get back to
> >> an operation cluster -- still trying to find documentation on how to
> >> recover.
> >>
> >>
> >> 1) I have not yet modified any CRUSH rules from the defaults. I have
> >> one ubuntu 14.04 OSD in the mix, and I had to set "ceph osd crush
> >> tunables legacy" just to get it to work.
> >>
> >> 2) I have not yet implemented any Erasure Code pool. That is probably
> >> one of the next tests I was going to do. I'm still testing with basic
> >> replication.
> >
> > Can you attach 'ceph health detail', 'ceph osd crush dump', and 'ceph osd
> > dump'?
> >
> >> The degraded data redundancy seems to be stuck and not reducing
> >> anymore. If I manually clear [if this is even possible] the 1 pg
> >> undersized, should my degraded filesystem go back online?
> >
> > The problem is likely the 1 unfound object. Are there any OSDs that are
> > down that failed recently? (Try 'ceph osd tree down' to see a simple
> > summary.)
> >
> > sage
> >
> >
> >>
> >> On Mon, Sep 4, 2017 at 2:05 AM, John Spray <jspray@redhat.com> wrote:
> >> > On Sun, Sep 3, 2017 at 2:14 PM, Two Spirit <twospirit6905@gmail.com> wrote:
> >> >> Setup: luminous on
> >> >> Ubuntu 14.04/16.04 mix. 5 OSD. all up. 3 or 4 mds, 3mon,cephx
> >> >> rebooting all 6 ceph systems did not clear the problem. Failure
> >> >> occurred within 6 hours of start of test.
> >> >> similar stress test with 4OSD,1MDS,1MON,cephx worked fine.
> >> >>
> >> >>
> >> >> stress test
> >> >> # cp * /mnt/cephfs
> >> >>
> >> >> # ceph -s
> >> >> health: HEALTH_WARN
> >> >> 1 filesystem is degraded
> >> >> crush map has straw_calc_version=0
> >> >> 1/731529 unfound (0.000%)
> >> >> Degraded data redundancy: 22519/1463058 objects degraded
> >> >> (1.539%), 2 pgs unclean, 2 pgs degraded, 1 pg undersized
> >> >>
> >> >> services:
> >> >> mon: 3 daemons, quorum xxx233,xxx266,xxx272
> >> >> mgr: xxx266(active)
> >> >> mds: cephfs-1/1/1 up {0=xxx233=up:replay}, 3 up:standby
> >> >> osd: 5 osds: 5 up, 5 in
> >> >> rgw: 1 daemon active
> >> >
> >> > Your MDS is probably stuck in the replay state because it can't read
> >> > from one of your degraded PGs. Given that you have all your OSDs in,
> >> > but one of your PGs is undersized (i.e. is short on OSDs), I would
> >> > guess that something is wrong with your choice of CRUSH rules or EC
> >> > config.
> >> >
> >> > John
> >> >
> >> >>
> >> >> # ceph mds dump
> >> >> dumped fsmap epoch 590
> >> >> fs_name cephfs
> >> >> epoch 589
> >> >> flags c
> >> >> created 2017-08-24 14:35:33.735399
> >> >> modified 2017-08-24 14:35:33.735400
> >> >> tableserver 0
> >> >> root 0
> >> >> session_timeout 60
> >> >> session_autoclose 300
> >> >> max_file_size 1099511627776
> >> >> last_failure 0
> >> >> last_failure_osd_epoch 1573
> >> >> compat compat={},rocompat={},incompat={1=base v0.20,2=client
> >> >> writeable ranges,3=default file layouts on dirs,4=dir inode in
> >> >> separate object,5=mds uses versioned encoding,6=dirfrag is stored in
> >> >> omap,8=file layout v2}
> >> >> max_mds 1
> >> >> in 0
> >> >> up {0=579217}
> >> >> failed
> >> >> damaged
> >> >> stopped
> >> >> data_pools [5]
> >> >> metadata_pool 6
> >> >> inline_data disabled
> >> >> balancer
> >> >> standby_count_wanted 1
> >> >> 579217: x.x.x.233:6804/1176521332 'xxx233' mds.0.589 up:replay seq 2
> >> >> --
> >> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> >> the body of a message to majordomo@vger.kernel.org
> >> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>
> >>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous filesystem is degraded
2017-09-12 19:51 ` Sage Weil
@ 2017-09-13 22:16 ` Two Spirit
0 siblings, 0 replies; 7+ messages in thread
From: Two Spirit @ 2017-09-13 22:16 UTC (permalink / raw)
To: Sage Weil; +Cc: John Spray, ceph-devel
I reverted the 1 unfound object(in the MDS). I think that eventually
cleared despite getting a message initially saying it wasn't found.
My filesystem is still degraded. The revert action seemed to have
damaged my mds. also the Clearing the unfound didn't seem to unstuck
the Degraded data redundancy. There is no recovery going on.
I'm looking into clearing the unclean, degraded, & undersized pg, but
I don't think that will restore the damaged mds, and degraded
filesystem.
help.
health: HEALTH_ERR
1 filesystem is degraded
1 mds daemon damaged
Degraded data redundancy: 22517/1463016 objects degraded
(1.539%), 1 pg unclean, 1 pg degraded, 1 pg undersized
services:
mon: 3 daemons, quorum osdmon33,osdmonmgr66,osdmon72
mds: cephfs-0/1/1 up , 3 up:standby, 1 damaged
osd: 6 osds: 6 up, 6 in
On Tue, Sep 12, 2017 at 12:51 PM, Sage Weil <sweil@redhat.com> wrote:
> On Tue, 12 Sep 2017, Two Spirit wrote:
>> I don't have any OSDs that are down, so the 1 unfound object I think
>> needs to be manually cleared. I ran across a webpage a while ago that
>> talked about how to clear it, but if you have a reference, would save
>> me a little time.
>
> http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/#failures-osd-unfound
>
> sage
>
>> I've included the outputs of the commands you asked. The ceph test
>> network contains 6 osds, 3 mons, 3 mds, 1rgw 1mgr. ubuntu 64 bit
>> 14.04/16.04 mix.
>>
>> file system is degraded. Are there procedures how to get this back in operation?
>>
>> On Tue, Sep 5, 2017 at 6:33 AM, Sage Weil <sweil@redhat.com> wrote:
>> > On Mon, 4 Sep 2017, Two Spirit wrote:
>> >> Thanks for the info. I'm stumped what to do right now to get back to
>> >> an operation cluster -- still trying to find documentation on how to
>> >> recover.
>> >>
>> >>
>> >> 1) I have not yet modified any CRUSH rules from the defaults. I have
>> >> one ubuntu 14.04 OSD in the mix, and I had to set "ceph osd crush
>> >> tunables legacy" just to get it to work.
>> >>
>> >> 2) I have not yet implemented any Erasure Code pool. That is probably
>> >> one of the next tests I was going to do. I'm still testing with basic
>> >> replication.
>> >
>> > Can you attach 'ceph health detail', 'ceph osd crush dump', and 'ceph osd
>> > dump'?
>> >
>> >> The degraded data redundancy seems to be stuck and not reducing
>> >> anymore. If I manually clear [if this is even possible] the 1 pg
>> >> undersized, should my degraded filesystem go back online?
>> >
>> > The problem is likely the 1 unfound object. Are there any OSDs that are
>> > down that failed recently? (Try 'ceph osd tree down' to see a simple
>> > summary.)
>> >
>> > sage
>> >
>> >
>> >>
>> >> On Mon, Sep 4, 2017 at 2:05 AM, John Spray <jspray@redhat.com> wrote:
>> >> > On Sun, Sep 3, 2017 at 2:14 PM, Two Spirit <twospirit6905@gmail.com> wrote:
>> >> >> Setup: luminous on
>> >> >> Ubuntu 14.04/16.04 mix. 5 OSD. all up. 3 or 4 mds, 3mon,cephx
>> >> >> rebooting all 6 ceph systems did not clear the problem. Failure
>> >> >> occurred within 6 hours of start of test.
>> >> >> similar stress test with 4OSD,1MDS,1MON,cephx worked fine.
>> >> >>
>> >> >>
>> >> >> stress test
>> >> >> # cp * /mnt/cephfs
>> >> >>
>> >> >> # ceph -s
>> >> >> health: HEALTH_WARN
>> >> >> 1 filesystem is degraded
>> >> >> crush map has straw_calc_version=0
>> >> >> 1/731529 unfound (0.000%)
>> >> >> Degraded data redundancy: 22519/1463058 objects degraded
>> >> >> (1.539%), 2 pgs unclean, 2 pgs degraded, 1 pg undersized
>> >> >>
>> >> >> services:
>> >> >> mon: 3 daemons, quorum xxx233,xxx266,xxx272
>> >> >> mgr: xxx266(active)
>> >> >> mds: cephfs-1/1/1 up {0=xxx233=up:replay}, 3 up:standby
>> >> >> osd: 5 osds: 5 up, 5 in
>> >> >> rgw: 1 daemon active
>> >> >
>> >> > Your MDS is probably stuck in the replay state because it can't read
>> >> > from one of your degraded PGs. Given that you have all your OSDs in,
>> >> > but one of your PGs is undersized (i.e. is short on OSDs), I would
>> >> > guess that something is wrong with your choice of CRUSH rules or EC
>> >> > config.
>> >> >
>> >> > John
>> >> >
>> >> >>
>> >> >> # ceph mds dump
>> >> >> dumped fsmap epoch 590
>> >> >> fs_name cephfs
>> >> >> epoch 589
>> >> >> flags c
>> >> >> created 2017-08-24 14:35:33.735399
>> >> >> modified 2017-08-24 14:35:33.735400
>> >> >> tableserver 0
>> >> >> root 0
>> >> >> session_timeout 60
>> >> >> session_autoclose 300
>> >> >> max_file_size 1099511627776
>> >> >> last_failure 0
>> >> >> last_failure_osd_epoch 1573
>> >> >> compat compat={},rocompat={},incompat={1=base v0.20,2=client
>> >> >> writeable ranges,3=default file layouts on dirs,4=dir inode in
>> >> >> separate object,5=mds uses versioned encoding,6=dirfrag is stored in
>> >> >> omap,8=file layout v2}
>> >> >> max_mds 1
>> >> >> in 0
>> >> >> up {0=579217}
>> >> >> failed
>> >> >> damaged
>> >> >> stopped
>> >> >> data_pools [5]
>> >> >> metadata_pool 6
>> >> >> inline_data disabled
>> >> >> balancer
>> >> >> standby_count_wanted 1
>> >> >> 579217: x.x.x.233:6804/1176521332 'xxx233' mds.0.589 up:replay seq 2
>> >> >> --
>> >> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> >> >> the body of a message to majordomo@vger.kernel.org
>> >> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> >>
>> >>
>>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-09-13 22:16 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-09-03 13:14 luminous filesystem is degraded Two Spirit
2017-09-04 9:05 ` John Spray
2017-09-04 15:45 ` Two Spirit
2017-09-05 13:33 ` Sage Weil
2017-09-12 18:31 ` Two Spirit
2017-09-12 19:51 ` Sage Weil
2017-09-13 22:16 ` Two Spirit
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.