Permanent MDS restarting under load

* Permanent MDS restarting under load
@ 2015-11-10 14:32 Oleksandr Natalenko
  2015-11-10 20:38 ` [ceph-users] " Gregory Farnum
  0 siblings, 1 reply; 3+ messages in thread
From: Oleksandr Natalenko @ 2015-11-10 14:32 UTC (permalink / raw)
  To: ceph-users-idqoXFIVOFJgJs9I8MT0rw; +Cc: ceph-devel-u79uwXL29TY76Z2rM5mHXA

Hello.

We have CephFS deployed over Ceph cluster (0.94.5).

We experience constant MDS restarting under high IOPS workload (e.g. 
rsyncing lots of small mailboxes from another storage to CephFS using 
ceph-fuse client). First, cluster health goes to HEALTH_WARN state with 
the following disclaimer:

===
mds0: Behind on trimming (321/30)
===

Also, slow requests start to appear:

===
2 requests are blocked > 32 sec
===

Then, after a while, one of MDSes fails with the following log:

===
лис 10 16:07:41 baikal bash[10122]: 2015-11-10 16:07:41.915540 
7f2484f13700 -1 MDSIOContextBase: blacklisted!  Restarting...
лис 10 16:07:41 baikal bash[10122]: starting mds.baikal at :/0
лис 10 16:07:42 baikal bash[10122]: 2015-11-10 16:07:42.003189 
7f82b477e7c0 -1 mds.-1.0 log_to_monitors {default=true}
===

I guess writing lots of small files bloats MDS log, and MDS doesn't 
catch trimming in time. That's why it is marked as failed and is 
replaced by standby MDS. We tried to limit mds_log_max_events to 30 
events, but that caused MDS to fail very quickly with the following 
stacktrace:

===
Stacktrace: https://gist.github.com/4c8a89682e81b0049f3e
===

Is that normal situation, or one could rate-limit client requests? May 
be there should be additional knobs to tune CephFS for handling such a 
workload?

Cluster info goes below.

CentOS 7.1, Ceph 0.94.5.

Cluster maps:

===
      osdmap e5894: 20 osds: 20 up, 20 in
       pgmap v8959901: 1024 pgs, 12 pools, 5156 GB data, 23074 kobjects
             20101 GB used, 30468 GB / 50570 GB avail
                 1024 active+clean
===

CephFS list:

===
name: myfs, metadata pool: mds_meta_storage, data pools: 
[mds_xattrs_storage fs_samba fs_pbx fs_misc fs_web fs_mail fs_ott ]
===

Both MDS data and metadata pools are located on PCI-E SSDs:

===
  -9  0.44800 root pcie-ssd
  -7  0.22400     host data-pcie-ssd
   7  0.22400         osd.7                                 up  1.00000   
        1.00000
  -8  0.22400     host baikal-pcie-ssd
   6  0.22400         osd.6                                 up  1.00000   
        1.00000

pool 20 'mds_meta_storage' replicated size 2 min_size 1 crush_ruleset 2 
object_hash rjenkins pg_num 64 pgp_num 64 last_change 4333 flags 
hashpspool stripe_width 0
pool 21 'mds_xattrs_storage' replicated size 2 min_size 1 crush_ruleset 
2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 4337 flags 
hashpspool crash_replay_interval 45 stripe_width 0

     mds_meta_storage       20     37422k         0          169G       
234714
     mds_xattrs_storage     21          0         0          169G     
11271588

rule pcie-ssd {
         ruleset 2
         type replicated
         min_size 1
         max_size 2
         step take pcie-ssd
         step chooseleaf firstn 0 type host
         step emit
}
===

There is 1 active MDS as well as 1 stand-by MDS:

===
mdsmap e9035: 1/1/1 up {0=data=up:active}, 1 up:standby
===

Also we have 10 OSDs on HDDs for additional data pools:

===
  -6 37.00000 root sata-hdd-misc
  -4 18.50000     host data-sata-hdd-misc
   1  3.70000         osd.1                                 up  1.00000   
        1.00000
   3  3.70000         osd.3                                 up  1.00000   
        1.00000
   4  3.70000         osd.4                                 up  1.00000   
        1.00000
   5  3.70000         osd.5                                 up  1.00000   
        1.00000
  10  3.70000         osd.10                                up  1.00000   
        1.00000
  -5 18.50000     host baikal-sata-hdd-misc
   0  3.70000         osd.0                                 up  1.00000   
        1.00000
  11  3.70000         osd.11                                up  1.00000   
        1.00000
  12  3.70000         osd.12                                up  1.00000   
        1.00000
  13  3.70000         osd.13                                up  1.00000   
        1.00000
  14  3.70000         osd.14                                up  1.00000   
        1.00000

     fs_samba               22      2162G      4.28         3814G      
1168619
     fs_pbx                 23      1551G      3.07         3814G      
3908813
     fs_misc                24       436G      0.86         3814G       
112114
     fs_web                 25     58642M      0.11         3814G       
378946
     fs_mail                26       442G      0.88         3814G      
6414073
     fs_ott                 27          0         0         3814G         
    0

rule sata-hdd-misc {
         ruleset 4
         type replicated
         min_size 1
         max_size 4
         step take sata-hdd-misc
         step choose firstn 2 type host
         step chooseleaf firstn 2 type osd
         step emit
}
===

CephFS folders pool affinity is done via setfattr. For example:

===
# file: mail
ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 
pool=fs_mail"
===
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

^ permalink raw reply	[flat|nested] 3+ messages in thread