* luminous filesystem is degraded
@ 2017-09-03 13:14 Two Spirit
2017-09-04 9:05 ` John Spray
0 siblings, 1 reply; 7+ messages in thread
From: Two Spirit @ 2017-09-03 13:14 UTC (permalink / raw)
To: ceph-devel
Setup: luminous on
Ubuntu 14.04/16.04 mix. 5 OSD. all up. 3 or 4 mds, 3mon,cephx
rebooting all 6 ceph systems did not clear the problem. Failure
occurred within 6 hours of start of test.
similar stress test with 4OSD,1MDS,1MON,cephx worked fine.
stress test
# cp * /mnt/cephfs
# ceph -s
health: HEALTH_WARN
1 filesystem is degraded
crush map has straw_calc_version=0
1/731529 unfound (0.000%)
Degraded data redundancy: 22519/1463058 objects degraded
(1.539%), 2 pgs unclean, 2 pgs degraded, 1 pg undersized
services:
mon: 3 daemons, quorum xxx233,xxx266,xxx272
mgr: xxx266(active)
mds: cephfs-1/1/1 up {0=xxx233=up:replay}, 3 up:standby
osd: 5 osds: 5 up, 5 in
rgw: 1 daemon active
# ceph mds dump
dumped fsmap epoch 590
fs_name cephfs
epoch 589
flags c
created 2017-08-24 14:35:33.735399
modified 2017-08-24 14:35:33.735400
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
last_failure 0
last_failure_osd_epoch 1573
compat compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in
separate object,5=mds uses versioned encoding,6=dirfrag is stored in
omap,8=file layout v2}
max_mds 1
in 0
up {0=579217}
failed
damaged
stopped
data_pools [5]
metadata_pool 6
inline_data disabled
balancer
standby_count_wanted 1
579217: x.x.x.233:6804/1176521332 'xxx233' mds.0.589 up:replay seq 2
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: luminous filesystem is degraded 2017-09-03 13:14 luminous filesystem is degraded Two Spirit @ 2017-09-04 9:05 ` John Spray 2017-09-04 15:45 ` Two Spirit 0 siblings, 1 reply; 7+ messages in thread From: John Spray @ 2017-09-04 9:05 UTC (permalink / raw) To: Two Spirit; +Cc: ceph-devel On Sun, Sep 3, 2017 at 2:14 PM, Two Spirit <twospirit6905@gmail.com> wrote: > Setup: luminous on > Ubuntu 14.04/16.04 mix. 5 OSD. all up. 3 or 4 mds, 3mon,cephx > rebooting all 6 ceph systems did not clear the problem. Failure > occurred within 6 hours of start of test. > similar stress test with 4OSD,1MDS,1MON,cephx worked fine. > > > stress test > # cp * /mnt/cephfs > > # ceph -s > health: HEALTH_WARN > 1 filesystem is degraded > crush map has straw_calc_version=0 > 1/731529 unfound (0.000%) > Degraded data redundancy: 22519/1463058 objects degraded > (1.539%), 2 pgs unclean, 2 pgs degraded, 1 pg undersized > > services: > mon: 3 daemons, quorum xxx233,xxx266,xxx272 > mgr: xxx266(active) > mds: cephfs-1/1/1 up {0=xxx233=up:replay}, 3 up:standby > osd: 5 osds: 5 up, 5 in > rgw: 1 daemon active Your MDS is probably stuck in the replay state because it can't read from one of your degraded PGs. Given that you have all your OSDs in, but one of your PGs is undersized (i.e. is short on OSDs), I would guess that something is wrong with your choice of CRUSH rules or EC config. John > > # ceph mds dump > dumped fsmap epoch 590 > fs_name cephfs > epoch 589 > flags c > created 2017-08-24 14:35:33.735399 > modified 2017-08-24 14:35:33.735400 > tableserver 0 > root 0 > session_timeout 60 > session_autoclose 300 > max_file_size 1099511627776 > last_failure 0 > last_failure_osd_epoch 1573 > compat compat={},rocompat={},incompat={1=base v0.20,2=client > writeable ranges,3=default file layouts on dirs,4=dir inode in > separate object,5=mds uses versioned encoding,6=dirfrag is stored in > omap,8=file layout v2} > max_mds 1 > in 0 > up {0=579217} > failed > damaged > stopped > data_pools [5] > metadata_pool 6 > inline_data disabled > balancer > standby_count_wanted 1 > 579217: x.x.x.233:6804/1176521332 'xxx233' mds.0.589 up:replay seq 2 > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous filesystem is degraded 2017-09-04 9:05 ` John Spray @ 2017-09-04 15:45 ` Two Spirit 2017-09-05 13:33 ` Sage Weil 0 siblings, 1 reply; 7+ messages in thread From: Two Spirit @ 2017-09-04 15:45 UTC (permalink / raw) To: John Spray; +Cc: ceph-devel Thanks for the info. I'm stumped what to do right now to get back to an operation cluster -- still trying to find documentation on how to recover. 1) I have not yet modified any CRUSH rules from the defaults. I have one ubuntu 14.04 OSD in the mix, and I had to set "ceph osd crush tunables legacy" just to get it to work. 2) I have not yet implemented any Erasure Code pool. That is probably one of the next tests I was going to do. I'm still testing with basic replication. The degraded data redundancy seems to be stuck and not reducing anymore. If I manually clear [if this is even possible] the 1 pg undersized, should my degraded filesystem go back online? On Mon, Sep 4, 2017 at 2:05 AM, John Spray <jspray@redhat.com> wrote: > On Sun, Sep 3, 2017 at 2:14 PM, Two Spirit <twospirit6905@gmail.com> wrote: >> Setup: luminous on >> Ubuntu 14.04/16.04 mix. 5 OSD. all up. 3 or 4 mds, 3mon,cephx >> rebooting all 6 ceph systems did not clear the problem. Failure >> occurred within 6 hours of start of test. >> similar stress test with 4OSD,1MDS,1MON,cephx worked fine. >> >> >> stress test >> # cp * /mnt/cephfs >> >> # ceph -s >> health: HEALTH_WARN >> 1 filesystem is degraded >> crush map has straw_calc_version=0 >> 1/731529 unfound (0.000%) >> Degraded data redundancy: 22519/1463058 objects degraded >> (1.539%), 2 pgs unclean, 2 pgs degraded, 1 pg undersized >> >> services: >> mon: 3 daemons, quorum xxx233,xxx266,xxx272 >> mgr: xxx266(active) >> mds: cephfs-1/1/1 up {0=xxx233=up:replay}, 3 up:standby >> osd: 5 osds: 5 up, 5 in >> rgw: 1 daemon active > > Your MDS is probably stuck in the replay state because it can't read > from one of your degraded PGs. Given that you have all your OSDs in, > but one of your PGs is undersized (i.e. is short on OSDs), I would > guess that something is wrong with your choice of CRUSH rules or EC > config. > > John > >> >> # ceph mds dump >> dumped fsmap epoch 590 >> fs_name cephfs >> epoch 589 >> flags c >> created 2017-08-24 14:35:33.735399 >> modified 2017-08-24 14:35:33.735400 >> tableserver 0 >> root 0 >> session_timeout 60 >> session_autoclose 300 >> max_file_size 1099511627776 >> last_failure 0 >> last_failure_osd_epoch 1573 >> compat compat={},rocompat={},incompat={1=base v0.20,2=client >> writeable ranges,3=default file layouts on dirs,4=dir inode in >> separate object,5=mds uses versioned encoding,6=dirfrag is stored in >> omap,8=file layout v2} >> max_mds 1 >> in 0 >> up {0=579217} >> failed >> damaged >> stopped >> data_pools [5] >> metadata_pool 6 >> inline_data disabled >> balancer >> standby_count_wanted 1 >> 579217: x.x.x.233:6804/1176521332 'xxx233' mds.0.589 up:replay seq 2 >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous filesystem is degraded 2017-09-04 15:45 ` Two Spirit @ 2017-09-05 13:33 ` Sage Weil 2017-09-12 18:31 ` Two Spirit 0 siblings, 1 reply; 7+ messages in thread From: Sage Weil @ 2017-09-05 13:33 UTC (permalink / raw) To: Two Spirit; +Cc: John Spray, ceph-devel On Mon, 4 Sep 2017, Two Spirit wrote: > Thanks for the info. I'm stumped what to do right now to get back to > an operation cluster -- still trying to find documentation on how to > recover. > > > 1) I have not yet modified any CRUSH rules from the defaults. I have > one ubuntu 14.04 OSD in the mix, and I had to set "ceph osd crush > tunables legacy" just to get it to work. > > 2) I have not yet implemented any Erasure Code pool. That is probably > one of the next tests I was going to do. I'm still testing with basic > replication. Can you attach 'ceph health detail', 'ceph osd crush dump', and 'ceph osd dump'? > The degraded data redundancy seems to be stuck and not reducing > anymore. If I manually clear [if this is even possible] the 1 pg > undersized, should my degraded filesystem go back online? The problem is likely the 1 unfound object. Are there any OSDs that are down that failed recently? (Try 'ceph osd tree down' to see a simple summary.) sage > > On Mon, Sep 4, 2017 at 2:05 AM, John Spray <jspray@redhat.com> wrote: > > On Sun, Sep 3, 2017 at 2:14 PM, Two Spirit <twospirit6905@gmail.com> wrote: > >> Setup: luminous on > >> Ubuntu 14.04/16.04 mix. 5 OSD. all up. 3 or 4 mds, 3mon,cephx > >> rebooting all 6 ceph systems did not clear the problem. Failure > >> occurred within 6 hours of start of test. > >> similar stress test with 4OSD,1MDS,1MON,cephx worked fine. > >> > >> > >> stress test > >> # cp * /mnt/cephfs > >> > >> # ceph -s > >> health: HEALTH_WARN > >> 1 filesystem is degraded > >> crush map has straw_calc_version=0 > >> 1/731529 unfound (0.000%) > >> Degraded data redundancy: 22519/1463058 objects degraded > >> (1.539%), 2 pgs unclean, 2 pgs degraded, 1 pg undersized > >> > >> services: > >> mon: 3 daemons, quorum xxx233,xxx266,xxx272 > >> mgr: xxx266(active) > >> mds: cephfs-1/1/1 up {0=xxx233=up:replay}, 3 up:standby > >> osd: 5 osds: 5 up, 5 in > >> rgw: 1 daemon active > > > > Your MDS is probably stuck in the replay state because it can't read > > from one of your degraded PGs. Given that you have all your OSDs in, > > but one of your PGs is undersized (i.e. is short on OSDs), I would > > guess that something is wrong with your choice of CRUSH rules or EC > > config. > > > > John > > > >> > >> # ceph mds dump > >> dumped fsmap epoch 590 > >> fs_name cephfs > >> epoch 589 > >> flags c > >> created 2017-08-24 14:35:33.735399 > >> modified 2017-08-24 14:35:33.735400 > >> tableserver 0 > >> root 0 > >> session_timeout 60 > >> session_autoclose 300 > >> max_file_size 1099511627776 > >> last_failure 0 > >> last_failure_osd_epoch 1573 > >> compat compat={},rocompat={},incompat={1=base v0.20,2=client > >> writeable ranges,3=default file layouts on dirs,4=dir inode in > >> separate object,5=mds uses versioned encoding,6=dirfrag is stored in > >> omap,8=file layout v2} > >> max_mds 1 > >> in 0 > >> up {0=579217} > >> failed > >> damaged > >> stopped > >> data_pools [5] > >> metadata_pool 6 > >> inline_data disabled > >> balancer > >> standby_count_wanted 1 > >> 579217: x.x.x.233:6804/1176521332 'xxx233' mds.0.589 up:replay seq 2 > >> -- > >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > >> the body of a message to majordomo@vger.kernel.org > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous filesystem is degraded 2017-09-05 13:33 ` Sage Weil @ 2017-09-12 18:31 ` Two Spirit 2017-09-12 19:51 ` Sage Weil 0 siblings, 1 reply; 7+ messages in thread From: Two Spirit @ 2017-09-12 18:31 UTC (permalink / raw) To: Sage Weil; +Cc: John Spray, ceph-devel [-- Attachment #1: Type: text/plain, Size: 4299 bytes --] I don't have any OSDs that are down, so the 1 unfound object I think needs to be manually cleared. I ran across a webpage a while ago that talked about how to clear it, but if you have a reference, would save me a little time. I've included the outputs of the commands you asked. The ceph test network contains 6 osds, 3 mons, 3 mds, 1rgw 1mgr. ubuntu 64 bit 14.04/16.04 mix. file system is degraded. Are there procedures how to get this back in operation? On Tue, Sep 5, 2017 at 6:33 AM, Sage Weil <sweil@redhat.com> wrote: > On Mon, 4 Sep 2017, Two Spirit wrote: >> Thanks for the info. I'm stumped what to do right now to get back to >> an operation cluster -- still trying to find documentation on how to >> recover. >> >> >> 1) I have not yet modified any CRUSH rules from the defaults. I have >> one ubuntu 14.04 OSD in the mix, and I had to set "ceph osd crush >> tunables legacy" just to get it to work. >> >> 2) I have not yet implemented any Erasure Code pool. That is probably >> one of the next tests I was going to do. I'm still testing with basic >> replication. > > Can you attach 'ceph health detail', 'ceph osd crush dump', and 'ceph osd > dump'? > >> The degraded data redundancy seems to be stuck and not reducing >> anymore. If I manually clear [if this is even possible] the 1 pg >> undersized, should my degraded filesystem go back online? > > The problem is likely the 1 unfound object. Are there any OSDs that are > down that failed recently? (Try 'ceph osd tree down' to see a simple > summary.) > > sage > > >> >> On Mon, Sep 4, 2017 at 2:05 AM, John Spray <jspray@redhat.com> wrote: >> > On Sun, Sep 3, 2017 at 2:14 PM, Two Spirit <twospirit6905@gmail.com> wrote: >> >> Setup: luminous on >> >> Ubuntu 14.04/16.04 mix. 5 OSD. all up. 3 or 4 mds, 3mon,cephx >> >> rebooting all 6 ceph systems did not clear the problem. Failure >> >> occurred within 6 hours of start of test. >> >> similar stress test with 4OSD,1MDS,1MON,cephx worked fine. >> >> >> >> >> >> stress test >> >> # cp * /mnt/cephfs >> >> >> >> # ceph -s >> >> health: HEALTH_WARN >> >> 1 filesystem is degraded >> >> crush map has straw_calc_version=0 >> >> 1/731529 unfound (0.000%) >> >> Degraded data redundancy: 22519/1463058 objects degraded >> >> (1.539%), 2 pgs unclean, 2 pgs degraded, 1 pg undersized >> >> >> >> services: >> >> mon: 3 daemons, quorum xxx233,xxx266,xxx272 >> >> mgr: xxx266(active) >> >> mds: cephfs-1/1/1 up {0=xxx233=up:replay}, 3 up:standby >> >> osd: 5 osds: 5 up, 5 in >> >> rgw: 1 daemon active >> > >> > Your MDS is probably stuck in the replay state because it can't read >> > from one of your degraded PGs. Given that you have all your OSDs in, >> > but one of your PGs is undersized (i.e. is short on OSDs), I would >> > guess that something is wrong with your choice of CRUSH rules or EC >> > config. >> > >> > John >> > >> >> >> >> # ceph mds dump >> >> dumped fsmap epoch 590 >> >> fs_name cephfs >> >> epoch 589 >> >> flags c >> >> created 2017-08-24 14:35:33.735399 >> >> modified 2017-08-24 14:35:33.735400 >> >> tableserver 0 >> >> root 0 >> >> session_timeout 60 >> >> session_autoclose 300 >> >> max_file_size 1099511627776 >> >> last_failure 0 >> >> last_failure_osd_epoch 1573 >> >> compat compat={},rocompat={},incompat={1=base v0.20,2=client >> >> writeable ranges,3=default file layouts on dirs,4=dir inode in >> >> separate object,5=mds uses versioned encoding,6=dirfrag is stored in >> >> omap,8=file layout v2} >> >> max_mds 1 >> >> in 0 >> >> up {0=579217} >> >> failed >> >> damaged >> >> stopped >> >> data_pools [5] >> >> metadata_pool 6 >> >> inline_data disabled >> >> balancer >> >> standby_count_wanted 1 >> >> 579217: x.x.x.233:6804/1176521332 'xxx233' mds.0.589 up:replay seq 2 >> >> -- >> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> >> the body of a message to majordomo@vger.kernel.org >> >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> [-- Attachment #2: ceph_health_detail.out --] [-- Type: application/octet-stream, Size: 2742 bytes --] HEALTH_WARN 1 filesystem is degraded; crush map has straw_calc_version=0; 177902/1463058 objects misplaced (12.160%); 1/731529 unfound (0.000%); Degraded data redundancy: 22519/1463058 objects degraded (1.539%), 16 pgs unclean, 2 pgs degraded, 1 pg undersized; clock skew detected on mon.osdmonmgr66, mon.osdmon72 FS_DEGRADED 1 filesystem is degraded fs cephfs is degraded OLD_CRUSH_STRAW_CALC_VERSION crush map has straw_calc_version=0 see http://docs.ceph.com/docs/master/rados/operations/crush-map/#tunables OBJECT_MISPLACED 177902/1463058 objects misplaced (12.160%) OBJECT_UNFOUND 1/731529 unfound (0.000%) pg 6.2 has 1 unfound objects PG_DEGRADED Degraded data redundancy: 22519/1463058 objects degraded (1.539%), 16 pgs unclean, 2 pgs degraded, 1 pg undersized pg 4.5 is stuck unclean for 1342.896172, current state active+remapped+backfill_wait, last acting [1,3] pg 5.3 is stuck unclean for 1342.976130, current state active+remapped+backfilling, last acting [2,1] pg 5.6 is stuck unclean for 1342.975955, current state active+remapped+backfill_wait, last acting [2,5] pg 5.c is stuck unclean for 1342.971200, current state active+remapped+backfill_wait, last acting [3,1] pg 5.11 is stuck undersized for 7774.128674, current state active+undersized+degraded, last acting [1] pg 5.14 is stuck unclean for 1342.223608, current state active+remapped+backfill_wait, last acting [3,2] pg 5.1a is stuck unclean for 1342.959107, current state active+remapped+backfill_wait, last acting [0,3] pg 5.1b is stuck unclean for 1343.006102, current state active+remapped+backfill_wait, last acting [3,0] pg 5.1c is stuck unclean for 1342.976034, current state active+remapped+backfill_wait, last acting [2,0] pg 5.1e is stuck unclean for 1342.959184, current state active+remapped+backfill_wait, last acting [0,1] pg 6.0 is stuck unclean for 1343.006407, current state active+remapped+backfill_wait, last acting [3,0] pg 6.2 is active+recovery_wait+degraded+remapped, acting [3,1], 1 unfound pg 6.3 is stuck unclean for 1342.896018, current state active+remapped+backfill_wait, last acting [1,2] pg 6.9 is stuck unclean for 1086.179526, current state active+remapped+backfill_wait, last acting [3,4] pg 6.12 is stuck unclean for 1342.976018, current state active+remapped+backfill_wait, last acting [2,3] pg 6.1c is stuck unclean for 1342.975969, current state active+remapped+backfill_wait, last acting [2,1] MON_CLOCK_SKEW clock skew detected on mon.osdmonmgr66, mon.osdmon72 mon.osdmonmgr66 addr 10.1.2.66:6789/0 clock skew 0.0806478s > max 0.05s (latency 0.00741028s) mon.osdmon72 addr 10.1.2.72:6789/0 clock skew 0.0820921s > max 0.05s (latency 0.0109876s) [-- Attachment #3: ceph_osd_crush_dump.out --] [-- Type: application/octet-stream, Size: 9686 bytes --] { "devices": [ { "id": 0, "name": "osd.0", "class": "hdd" }, { "id": 1, "name": "osd.1", "class": "hdd" }, { "id": 2, "name": "osd.2", "class": "hdd" }, { "id": 3, "name": "osd.3", "class": "hdd" }, { "id": 4, "name": "osd.4", "class": "hdd" }, { "id": 5, "name": "osd.5", "class": "hdd" } ], "types": [ { "type_id": 0, "name": "osd" }, { "type_id": 1, "name": "host" }, { "type_id": 2, "name": "chassis" }, { "type_id": 3, "name": "rack" }, { "type_id": 4, "name": "row" }, { "type_id": 5, "name": "pdu" }, { "type_id": 6, "name": "pod" }, { "type_id": 7, "name": "room" }, { "type_id": 8, "name": "datacenter" }, { "type_id": 9, "name": "region" }, { "type_id": 10, "name": "root" } ], "buckets": [ { "id": -1, "name": "default", "type_id": 10, "type_name": "root", "weight": 417284, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -3, "weight": 89417, "pos": 0 }, { "id": -5, "weight": 119229, "pos": 1 }, { "id": -7, "weight": 59611, "pos": 2 }, { "id": -9, "weight": 59611, "pos": 3 }, { "id": -11, "weight": 44708, "pos": 4 }, { "id": -13, "weight": 44708, "pos": 5 } ] }, { "id": -2, "name": "default~hdd", "type_id": 10, "type_name": "root", "weight": 417284, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -4, "weight": 89417, "pos": 0 }, { "id": -6, "weight": 119229, "pos": 1 }, { "id": -8, "weight": 59611, "pos": 2 }, { "id": -10, "weight": 59611, "pos": 3 }, { "id": -12, "weight": 44708, "pos": 4 }, { "id": -14, "weight": 44708, "pos": 5 } ] }, { "id": -3, "name": "osdmonmgr66", "type_id": 1, "type_name": "host", "weight": 89417, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 0, "weight": 89417, "pos": 0 } ] }, { "id": -4, "name": "osdmonmgr66~hdd", "type_id": 1, "type_name": "host", "weight": 89417, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 0, "weight": 89417, "pos": 0 } ] }, { "id": -5, "name": "osdmon33", "type_id": 1, "type_name": "host", "weight": 119229, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 1, "weight": 119229, "pos": 0 } ] }, { "id": -6, "name": "osdmon33~hdd", "type_id": 1, "type_name": "host", "weight": 119229, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 1, "weight": 119229, "pos": 0 } ] }, { "id": -7, "name": "osdmon72", "type_id": 1, "type_name": "host", "weight": 59611, "alg": "straw", "hash": "rjenkins1", "items": [ { "id": 2, "weight": 59611, "pos": 0 } ] }, { "id": -8, "name": "osdmon72~hdd", "type_id": 1, "type_name": "host", "weight": 59611, "alg": "straw", "hash": "rjenkins1", "items": [ { "id": 2, "weight": 59611, "pos": 0 } ] }, { "id": -9, "name": "osd43", "type_id": 1, "type_name": "host", "weight": 59611, "alg": "straw", "hash": "rjenkins1", "items": [ { "id": 3, "weight": 59611, "pos": 0 } ] }, { "id": -10, "name": "osd43~hdd", "type_id": 1, "type_name": "host", "weight": 59611, "alg": "straw", "hash": "rjenkins1", "items": [ { "id": 3, "weight": 59611, "pos": 0 } ] }, { "id": -11, "name": "osd40", "type_id": 1, "type_name": "host", "weight": 44708, "alg": "straw", "hash": "rjenkins1", "items": [ { "id": 5, "weight": 44708, "pos": 0 } ] }, { "id": -12, "name": "osd40~hdd", "type_id": 1, "type_name": "host", "weight": 44708, "alg": "straw", "hash": "rjenkins1", "items": [ { "id": 5, "weight": 44708, "pos": 0 } ] }, { "id": -13, "name": "osd59", "type_id": 1, "type_name": "host", "weight": 44708, "alg": "straw", "hash": "rjenkins1", "items": [ { "id": 4, "weight": 44708, "pos": 0 } ] }, { "id": -14, "name": "osd59~hdd", "type_id": 1, "type_name": "host", "weight": 44708, "alg": "straw", "hash": "rjenkins1", "items": [ { "id": 4, "weight": 44708, "pos": 0 } ] } ], "rules": [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "choose_firstn", "num": 0, "type": "osd" }, { "op": "emit" } ] } ], "tunables": { "choose_local_tries": 2, "choose_local_fallback_tries": 5, "choose_total_tries": 19, "chooseleaf_descend_once": 0, "chooseleaf_vary_r": 0, "chooseleaf_stable": 0, "straw_calc_version": 0, "allowed_bucket_algs": 22, "profile": "argonaut", "optimal_tunables": 0, "legacy_tunables": 1, "minimum_required_version": "hammer", "require_feature_tunables": 0, "require_feature_tunables2": 0, "has_v2_rules": 0, "require_feature_tunables3": 0, "has_v3_rules": 0, "has_v4_buckets": 1, "require_feature_tunables5": 0, "has_v5_rules": 0 }, "choose_args": {} } [-- Attachment #4: ceph_osd_dump.out --] [-- Type: application/octet-stream, Size: 3093 bytes --] epoch 1667 fsid d432ca51-9141-4853-b6b9-7c22f621e9c5 created 2017-08-24 13:47:57.530896 modified 2017-09-12 10:45:31.097347 flags sortbitwise,recovery_deletes crush_version 20 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client jewel min_compat_client hammer require_osd_release luminous pool 1 '.rgw.root' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 11 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 2 'default.rgw.control' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 13 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 3 'default.rgw.meta' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 15 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 4 'default.rgw.log' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 17 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 5 'cephfs_data' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 1200 lfor 0/1194 stripe_width 0 application cephfs pool 6 'cephfs_metadata' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 1207 lfor 0/1191 flags hashpspool stripe_width 0 application cephfs max_osd 6 osd.0 up in weight 1 up_from 1621 up_thru 1664 down_at 1613 last_clean_interval [1590,1612) 10.1.2.66:6801/1154 10.1.2.66:6802/1154 10.1.2.66:6803/1154 10.1.2.66:6804/1154 exists,up e345f131-3603-4446-9de8-e6f64a7356fd osd.1 up in weight 1 up_from 1588 up_thru 1664 down_at 1587 last_clean_interval [1567,1585) 10.1.2.33:6800/1771 10.1.2.33:6801/1771 10.1.2.33:6802/1771 10.1.2.33:6803/1771 exists,up 15f9a4b3-c088-4882-a9dd-443fa0466367 osd.2 up in weight 1 up_from 1600 up_thru 1664 down_at 1577 last_clean_interval [1465,1576) 10.1.2.72:6801/3879 10.1.2.72:6802/3879 10.1.2.72:6803/3879 10.1.2.72:6804/3879 exists,up 059c4895-03db-418b-a596-d6bb7071b39b osd.3 up in weight 1 up_from 1638 up_thru 1666 down_at 1636 last_clean_interval [1633,1637) 10.1.2.43:6800/4353 10.1.2.43:6804/1004353 10.1.2.43:6805/1004353 10.1.2.43:6806/1004353 exists,up 3abaf555-a684-4182-b592-4476e75367f6 osd.4 up in weight 1 up_from 1653 up_thru 1662 down_at 1649 last_clean_interval [1645,1648) 10.1.2.59:6800/1671 10.1.2.59:6801/1671 10.1.2.59:6802/1671 10.1.2.59:6803/1671 exists,up fd2cc782-49f3-4f32-bdf8-331d4fee880f osd.5 up in weight 1 up_from 1626 up_thru 1666 down_at 1589 last_clean_interval [1556,1585) 10.1.2.40:6800/5921 10.1.2.40:6801/5921 10.1.2.40:6802/5921 10.1.2.40:6803/5921 exists,up ddbc4815-541c-451d-a478-712863c529a6 pg_temp 4.5 [1,3] pg_temp 5.3 [2,1] pg_temp 5.6 [2,5] pg_temp 5.c [3,1] pg_temp 5.14 [3,2] pg_temp 5.1a [0,3] pg_temp 5.1b [3,0] pg_temp 5.1c [2,0] pg_temp 5.1e [0,1] pg_temp 6.0 [3,0] pg_temp 6.2 [3,1] pg_temp 6.3 [1,2] pg_temp 6.9 [3,4] pg_temp 6.12 [2,3] pg_temp 6.1c [2,1] [-- Attachment #5: ceph_osd_tree_down.out --] [-- Type: application/octet-stream, Size: 51 bytes --] ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous filesystem is degraded 2017-09-12 18:31 ` Two Spirit @ 2017-09-12 19:51 ` Sage Weil 2017-09-13 22:16 ` Two Spirit 0 siblings, 1 reply; 7+ messages in thread From: Sage Weil @ 2017-09-12 19:51 UTC (permalink / raw) To: Two Spirit; +Cc: John Spray, ceph-devel On Tue, 12 Sep 2017, Two Spirit wrote: > I don't have any OSDs that are down, so the 1 unfound object I think > needs to be manually cleared. I ran across a webpage a while ago that > talked about how to clear it, but if you have a reference, would save > me a little time. http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/#failures-osd-unfound sage > I've included the outputs of the commands you asked. The ceph test > network contains 6 osds, 3 mons, 3 mds, 1rgw 1mgr. ubuntu 64 bit > 14.04/16.04 mix. > > file system is degraded. Are there procedures how to get this back in operation? > > On Tue, Sep 5, 2017 at 6:33 AM, Sage Weil <sweil@redhat.com> wrote: > > On Mon, 4 Sep 2017, Two Spirit wrote: > >> Thanks for the info. I'm stumped what to do right now to get back to > >> an operation cluster -- still trying to find documentation on how to > >> recover. > >> > >> > >> 1) I have not yet modified any CRUSH rules from the defaults. I have > >> one ubuntu 14.04 OSD in the mix, and I had to set "ceph osd crush > >> tunables legacy" just to get it to work. > >> > >> 2) I have not yet implemented any Erasure Code pool. That is probably > >> one of the next tests I was going to do. I'm still testing with basic > >> replication. > > > > Can you attach 'ceph health detail', 'ceph osd crush dump', and 'ceph osd > > dump'? > > > >> The degraded data redundancy seems to be stuck and not reducing > >> anymore. If I manually clear [if this is even possible] the 1 pg > >> undersized, should my degraded filesystem go back online? > > > > The problem is likely the 1 unfound object. Are there any OSDs that are > > down that failed recently? (Try 'ceph osd tree down' to see a simple > > summary.) > > > > sage > > > > > >> > >> On Mon, Sep 4, 2017 at 2:05 AM, John Spray <jspray@redhat.com> wrote: > >> > On Sun, Sep 3, 2017 at 2:14 PM, Two Spirit <twospirit6905@gmail.com> wrote: > >> >> Setup: luminous on > >> >> Ubuntu 14.04/16.04 mix. 5 OSD. all up. 3 or 4 mds, 3mon,cephx > >> >> rebooting all 6 ceph systems did not clear the problem. Failure > >> >> occurred within 6 hours of start of test. > >> >> similar stress test with 4OSD,1MDS,1MON,cephx worked fine. > >> >> > >> >> > >> >> stress test > >> >> # cp * /mnt/cephfs > >> >> > >> >> # ceph -s > >> >> health: HEALTH_WARN > >> >> 1 filesystem is degraded > >> >> crush map has straw_calc_version=0 > >> >> 1/731529 unfound (0.000%) > >> >> Degraded data redundancy: 22519/1463058 objects degraded > >> >> (1.539%), 2 pgs unclean, 2 pgs degraded, 1 pg undersized > >> >> > >> >> services: > >> >> mon: 3 daemons, quorum xxx233,xxx266,xxx272 > >> >> mgr: xxx266(active) > >> >> mds: cephfs-1/1/1 up {0=xxx233=up:replay}, 3 up:standby > >> >> osd: 5 osds: 5 up, 5 in > >> >> rgw: 1 daemon active > >> > > >> > Your MDS is probably stuck in the replay state because it can't read > >> > from one of your degraded PGs. Given that you have all your OSDs in, > >> > but one of your PGs is undersized (i.e. is short on OSDs), I would > >> > guess that something is wrong with your choice of CRUSH rules or EC > >> > config. > >> > > >> > John > >> > > >> >> > >> >> # ceph mds dump > >> >> dumped fsmap epoch 590 > >> >> fs_name cephfs > >> >> epoch 589 > >> >> flags c > >> >> created 2017-08-24 14:35:33.735399 > >> >> modified 2017-08-24 14:35:33.735400 > >> >> tableserver 0 > >> >> root 0 > >> >> session_timeout 60 > >> >> session_autoclose 300 > >> >> max_file_size 1099511627776 > >> >> last_failure 0 > >> >> last_failure_osd_epoch 1573 > >> >> compat compat={},rocompat={},incompat={1=base v0.20,2=client > >> >> writeable ranges,3=default file layouts on dirs,4=dir inode in > >> >> separate object,5=mds uses versioned encoding,6=dirfrag is stored in > >> >> omap,8=file layout v2} > >> >> max_mds 1 > >> >> in 0 > >> >> up {0=579217} > >> >> failed > >> >> damaged > >> >> stopped > >> >> data_pools [5] > >> >> metadata_pool 6 > >> >> inline_data disabled > >> >> balancer > >> >> standby_count_wanted 1 > >> >> 579217: x.x.x.233:6804/1176521332 'xxx233' mds.0.589 up:replay seq 2 > >> >> -- > >> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > >> >> the body of a message to majordomo@vger.kernel.org > >> >> More majordomo info at http://vger.kernel.org/majordomo-info.html > >> -- > >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > >> the body of a message to majordomo@vger.kernel.org > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > >> > >> > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: luminous filesystem is degraded 2017-09-12 19:51 ` Sage Weil @ 2017-09-13 22:16 ` Two Spirit 0 siblings, 0 replies; 7+ messages in thread From: Two Spirit @ 2017-09-13 22:16 UTC (permalink / raw) To: Sage Weil; +Cc: John Spray, ceph-devel I reverted the 1 unfound object(in the MDS). I think that eventually cleared despite getting a message initially saying it wasn't found. My filesystem is still degraded. The revert action seemed to have damaged my mds. also the Clearing the unfound didn't seem to unstuck the Degraded data redundancy. There is no recovery going on. I'm looking into clearing the unclean, degraded, & undersized pg, but I don't think that will restore the damaged mds, and degraded filesystem. help. health: HEALTH_ERR 1 filesystem is degraded 1 mds daemon damaged Degraded data redundancy: 22517/1463016 objects degraded (1.539%), 1 pg unclean, 1 pg degraded, 1 pg undersized services: mon: 3 daemons, quorum osdmon33,osdmonmgr66,osdmon72 mds: cephfs-0/1/1 up , 3 up:standby, 1 damaged osd: 6 osds: 6 up, 6 in On Tue, Sep 12, 2017 at 12:51 PM, Sage Weil <sweil@redhat.com> wrote: > On Tue, 12 Sep 2017, Two Spirit wrote: >> I don't have any OSDs that are down, so the 1 unfound object I think >> needs to be manually cleared. I ran across a webpage a while ago that >> talked about how to clear it, but if you have a reference, would save >> me a little time. > > http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/#failures-osd-unfound > > sage > >> I've included the outputs of the commands you asked. The ceph test >> network contains 6 osds, 3 mons, 3 mds, 1rgw 1mgr. ubuntu 64 bit >> 14.04/16.04 mix. >> >> file system is degraded. Are there procedures how to get this back in operation? >> >> On Tue, Sep 5, 2017 at 6:33 AM, Sage Weil <sweil@redhat.com> wrote: >> > On Mon, 4 Sep 2017, Two Spirit wrote: >> >> Thanks for the info. I'm stumped what to do right now to get back to >> >> an operation cluster -- still trying to find documentation on how to >> >> recover. >> >> >> >> >> >> 1) I have not yet modified any CRUSH rules from the defaults. I have >> >> one ubuntu 14.04 OSD in the mix, and I had to set "ceph osd crush >> >> tunables legacy" just to get it to work. >> >> >> >> 2) I have not yet implemented any Erasure Code pool. That is probably >> >> one of the next tests I was going to do. I'm still testing with basic >> >> replication. >> > >> > Can you attach 'ceph health detail', 'ceph osd crush dump', and 'ceph osd >> > dump'? >> > >> >> The degraded data redundancy seems to be stuck and not reducing >> >> anymore. If I manually clear [if this is even possible] the 1 pg >> >> undersized, should my degraded filesystem go back online? >> > >> > The problem is likely the 1 unfound object. Are there any OSDs that are >> > down that failed recently? (Try 'ceph osd tree down' to see a simple >> > summary.) >> > >> > sage >> > >> > >> >> >> >> On Mon, Sep 4, 2017 at 2:05 AM, John Spray <jspray@redhat.com> wrote: >> >> > On Sun, Sep 3, 2017 at 2:14 PM, Two Spirit <twospirit6905@gmail.com> wrote: >> >> >> Setup: luminous on >> >> >> Ubuntu 14.04/16.04 mix. 5 OSD. all up. 3 or 4 mds, 3mon,cephx >> >> >> rebooting all 6 ceph systems did not clear the problem. Failure >> >> >> occurred within 6 hours of start of test. >> >> >> similar stress test with 4OSD,1MDS,1MON,cephx worked fine. >> >> >> >> >> >> >> >> >> stress test >> >> >> # cp * /mnt/cephfs >> >> >> >> >> >> # ceph -s >> >> >> health: HEALTH_WARN >> >> >> 1 filesystem is degraded >> >> >> crush map has straw_calc_version=0 >> >> >> 1/731529 unfound (0.000%) >> >> >> Degraded data redundancy: 22519/1463058 objects degraded >> >> >> (1.539%), 2 pgs unclean, 2 pgs degraded, 1 pg undersized >> >> >> >> >> >> services: >> >> >> mon: 3 daemons, quorum xxx233,xxx266,xxx272 >> >> >> mgr: xxx266(active) >> >> >> mds: cephfs-1/1/1 up {0=xxx233=up:replay}, 3 up:standby >> >> >> osd: 5 osds: 5 up, 5 in >> >> >> rgw: 1 daemon active >> >> > >> >> > Your MDS is probably stuck in the replay state because it can't read >> >> > from one of your degraded PGs. Given that you have all your OSDs in, >> >> > but one of your PGs is undersized (i.e. is short on OSDs), I would >> >> > guess that something is wrong with your choice of CRUSH rules or EC >> >> > config. >> >> > >> >> > John >> >> > >> >> >> >> >> >> # ceph mds dump >> >> >> dumped fsmap epoch 590 >> >> >> fs_name cephfs >> >> >> epoch 589 >> >> >> flags c >> >> >> created 2017-08-24 14:35:33.735399 >> >> >> modified 2017-08-24 14:35:33.735400 >> >> >> tableserver 0 >> >> >> root 0 >> >> >> session_timeout 60 >> >> >> session_autoclose 300 >> >> >> max_file_size 1099511627776 >> >> >> last_failure 0 >> >> >> last_failure_osd_epoch 1573 >> >> >> compat compat={},rocompat={},incompat={1=base v0.20,2=client >> >> >> writeable ranges,3=default file layouts on dirs,4=dir inode in >> >> >> separate object,5=mds uses versioned encoding,6=dirfrag is stored in >> >> >> omap,8=file layout v2} >> >> >> max_mds 1 >> >> >> in 0 >> >> >> up {0=579217} >> >> >> failed >> >> >> damaged >> >> >> stopped >> >> >> data_pools [5] >> >> >> metadata_pool 6 >> >> >> inline_data disabled >> >> >> balancer >> >> >> standby_count_wanted 1 >> >> >> 579217: x.x.x.233:6804/1176521332 'xxx233' mds.0.589 up:replay seq 2 >> >> >> -- >> >> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> >> >> the body of a message to majordomo@vger.kernel.org >> >> >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> -- >> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> >> the body of a message to majordomo@vger.kernel.org >> >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> >> >> >> ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-09-13 22:16 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-09-03 13:14 luminous filesystem is degraded Two Spirit 2017-09-04 9:05 ` John Spray 2017-09-04 15:45 ` Two Spirit 2017-09-05 13:33 ` Sage Weil 2017-09-12 18:31 ` Two Spirit 2017-09-12 19:51 ` Sage Weil 2017-09-13 22:16 ` Two Spirit
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.