* ceph-mds crash v12.0.3
@ 2017-06-12 9:13 Georgi Chorbadzhiyski
2017-06-12 10:22 ` John Spray
0 siblings, 1 reply; 17+ messages in thread
From: Georgi Chorbadzhiyski @ 2017-06-12 9:13 UTC (permalink / raw)
To: ceph-devel
We started getting these on all of our 3 MDS-es. Any idea how to fix it or at least debug
it and remove the dir entries that are causing the problem?
[root@amssn3 ~]# yum info ceph-mds
Name : ceph-mds
Arch : x86_64
Epoch : 1
Version : 12.0.3
Release : 0.el7
Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: *** Caught signal (Segmentation fault) **
Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: in thread 7f9e0ae70700 thread_name:mds_rank_progr
Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: ceph version 12.0.3 (f2337d1b42fa49dbb0a93e4048a42762e3dffbbf)
Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 1: (()+0x563caf) [0x7f9e16d46caf]
Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 2: (()+0xf370) [0x7f9e148cc370]
Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xbb9) [0x7f9e16ac3559]
Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9b1) [0x7f9e16af2231]
Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 5: (MDSInternalContextBase::complete(int)+0x1eb) [0x7f9e16cd1bcb]
Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 6: (MDSRank::_advance_queues()+0x4a5) [0x7f9e16a7e375]
Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 7: (MDSRank::ProgressThread::entry()+0x4a) [0x7f9e16a7e7ea]
Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 8: (()+0x7dc5) [0x7f9e148c4dc5]
Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 9: (clone()+0x6d) [0x7f9e137a476d]
Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 2017-06-12 04:11:39.585944 7f9e0ae70700 -1 *** Caught signal (Segmentation fault) **
Jun 12 03:36:19 amssn3.sgvps.net ceph-mds[3503]: ceph version 12.0.3 (f2337d1b42fa49dbb0a93e4048a42762e3dffbbf)
Jun 12 03:36:19 amssn3 ceph-mds[3503]: 1: (()+0x563caf) [0x7f24fe425caf]
Jun 12 03:36:19 amssn3 ceph-mds[3503]: 2: (()+0xf370) [0x7f24fbfab370]
Jun 12 03:36:19 amssn3 ceph-mds[3503]: 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xbb9) [0x7f24fe1a2559]
Jun 12 03:36:19 amssn3 ceph-mds[3503]: 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9b1) [0x7f24fe1d1231]
Jun 12 03:36:19 amssn3 ceph-mds[3503]: 5: (Server::handle_client_request(MClientRequest*)+0x48d) [0x7f24fe1d1a6d]
Jun 12 03:36:19 amssn3 ceph-mds[3503]: 6: (Server::dispatch(Message*)+0x38b) [0x7f24fe1d619b]
Jun 12 03:36:19 amssn3 ceph-mds[3503]: 7: (MDSRank::handle_deferrable_message(Message*)+0x7fc) [0x7f24fe152bbc]
Jun 12 03:36:19 amssn3 ceph-mds[3503]: 8: (MDSRank::_dispatch(Message*, bool)+0x1eb) [0x7f24fe15db4b]
Jun 12 03:36:19 amssn3 ceph-mds[3503]: 9: (MDSRankDispatcher::ms_dispatch(Message*)+0x15) [0x7f24fe15ea95]
Jun 12 03:36:19 amssn3 ceph-mds[3503]: 10: (MDSDaemon::ms_dispatch(Message*)+0xf3) [0x7f24fe14a7c3]
Jun 12 03:36:19 amssn3 ceph-mds[3503]: 11: (DispatchQueue::entry()+0x7a2) [0x7f24fe6a9a02]
Jun 12 03:36:19 amssn3 ceph-mds[3503]: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f24fe4dd23d]
Jun 12 03:36:19 amssn3 ceph-mds[3503]: 13: (()+0x7dc5) [0x7f24fbfa3dc5]
Jun 12 03:36:19 amssn3 ceph-mds[3503]: 14: (clone()+0x6d) [0x7f24fae8376d]
Jun 12 04:01:33 amssn5 ceph-mds[2544]: starting mds.amssn5 at -
Jun 12 04:01:43 amssn5 ceph-mds[2544]: *** Caught signal (Segmentation fault) **
Jun 12 04:01:43 amssn5 ceph-mds[2544]: in thread 7f45d2595700 thread_name:mds_rank_progr
Jun 12 04:01:43 amssn5 ceph-mds[2544]: ceph version 12.0.3 (f2337d1b42fa49dbb0a93e4048a42762e3dffbbf)
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 1: (()+0x563caf) [0x7f45de46bcaf]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 2: (()+0xf370) [0x7f45dbff1370]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xbb9) [0x7f45de1e8559]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9b1) [0x7f45de217231]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 5: (MDSInternalContextBase::complete(int)+0x1eb) [0x7f45de3f6bcb]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 6: (MDSRank::_advance_queues()+0x4a5) [0x7f45de1a3375]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 7: (MDSRank::ProgressThread::entry()+0x4a) [0x7f45de1a37ea]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 8: (()+0x7dc5) [0x7f45dbfe9dc5]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 9: (clone()+0x6d) [0x7f45daec976d]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 2017-06-12 04:01:43.579491 7f45d2595700 -1 *** Caught signal (Segmentation fault) **
Jun 12 04:01:43 amssn5 ceph-mds[2544]: in thread 7f45d2595700 thread_name:mds_rank_progr
Jun 12 04:01:43 amssn5 ceph-mds[2544]: ceph version 12.0.3 (f2337d1b42fa49dbb0a93e4048a42762e3dffbbf)
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 1: (()+0x563caf) [0x7f45de46bcaf]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 2: (()+0xf370) [0x7f45dbff1370]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xbb9) [0x7f45de1e8559]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9b1) [0x7f45de217231]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 5: (MDSInternalContextBase::complete(int)+0x1eb) [0x7f45de3f6bcb]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 6: (MDSRank::_advance_queues()+0x4a5) [0x7f45de1a3375]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 7: (MDSRank::ProgressThread::entry()+0x4a) [0x7f45de1a37ea]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 8: (()+0x7dc5) [0x7f45dbfe9dc5]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 9: (clone()+0x6d) [0x7f45daec976d]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 0> 2017-06-12 04:01:43.579491 7f45d2595700 -1 *** Caught signal (Segmentation fault) **
Jun 12 04:01:43 amssn5 ceph-mds[2544]: in thread 7f45d2595700 thread_name:mds_rank_progr
Jun 12 04:01:43 amssn5 ceph-mds[2544]: ceph version 12.0.3 (f2337d1b42fa49dbb0a93e4048a42762e3dffbbf)
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 1: (()+0x563caf) [0x7f45de46bcaf]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 2: (()+0xf370) [0x7f45dbff1370]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xbb9) [0x7f45de1e8559]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9b1) [0x7f45de217231]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 5: (MDSInternalContextBase::complete(int)+0x1eb) [0x7f45de3f6bcb]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 6: (MDSRank::_advance_queues()+0x4a5) [0x7f45de1a3375]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 7: (MDSRank::ProgressThread::entry()+0x4a) [0x7f45de1a37ea]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 8: (()+0x7dc5) [0x7f45dbfe9dc5]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: 9: (clone()+0x6d) [0x7f45daec976d]
Jun 12 04:01:43 amssn5 ceph-mds[2544]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: ceph-mds crash v12.0.3
2017-06-12 9:13 ceph-mds crash v12.0.3 Georgi Chorbadzhiyski
@ 2017-06-12 10:22 ` John Spray
2017-06-12 10:38 ` Georgi Chorbadzhiyski
2017-06-12 10:56 ` Georgi Chorbadzhiyski
0 siblings, 2 replies; 17+ messages in thread
From: John Spray @ 2017-06-12 10:22 UTC (permalink / raw)
To: Georgi Chorbadzhiyski; +Cc: Ceph Development
On Mon, Jun 12, 2017 at 5:13 AM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
> We started getting these on all of our 3 MDS-es. Any idea how to fix it or at least debug
> it and remove the dir entries that are causing the problem?
Assuming it's easy to reproduce, set "debug mds = 20", "debug ms = 1"
and gather the logs in the run up to the crash.
What is the workload? Is there anything unusual about this directory?
Has the cluster ever experienced severe damage like a lost PG?
John
>
> [root@amssn3 ~]# yum info ceph-mds
> Name : ceph-mds
> Arch : x86_64
> Epoch : 1
> Version : 12.0.3
> Release : 0.el7
>
> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: *** Caught signal (Segmentation fault) **
> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: in thread 7f9e0ae70700 thread_name:mds_rank_progr
> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: ceph version 12.0.3 (f2337d1b42fa49dbb0a93e4048a42762e3dffbbf)
> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 1: (()+0x563caf) [0x7f9e16d46caf]
> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 2: (()+0xf370) [0x7f9e148cc370]
> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xbb9) [0x7f9e16ac3559]
> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9b1) [0x7f9e16af2231]
> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 5: (MDSInternalContextBase::complete(int)+0x1eb) [0x7f9e16cd1bcb]
> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 6: (MDSRank::_advance_queues()+0x4a5) [0x7f9e16a7e375]
> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 7: (MDSRank::ProgressThread::entry()+0x4a) [0x7f9e16a7e7ea]
> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 8: (()+0x7dc5) [0x7f9e148c4dc5]
> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 9: (clone()+0x6d) [0x7f9e137a476d]
> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 2017-06-12 04:11:39.585944 7f9e0ae70700 -1 *** Caught signal (Segmentation fault) **
>
>
> Jun 12 03:36:19 amssn3.sgvps.net ceph-mds[3503]: ceph version 12.0.3 (f2337d1b42fa49dbb0a93e4048a42762e3dffbbf)
> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 1: (()+0x563caf) [0x7f24fe425caf]
> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 2: (()+0xf370) [0x7f24fbfab370]
> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xbb9) [0x7f24fe1a2559]
> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9b1) [0x7f24fe1d1231]
> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 5: (Server::handle_client_request(MClientRequest*)+0x48d) [0x7f24fe1d1a6d]
> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 6: (Server::dispatch(Message*)+0x38b) [0x7f24fe1d619b]
> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 7: (MDSRank::handle_deferrable_message(Message*)+0x7fc) [0x7f24fe152bbc]
> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 8: (MDSRank::_dispatch(Message*, bool)+0x1eb) [0x7f24fe15db4b]
> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 9: (MDSRankDispatcher::ms_dispatch(Message*)+0x15) [0x7f24fe15ea95]
> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 10: (MDSDaemon::ms_dispatch(Message*)+0xf3) [0x7f24fe14a7c3]
> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 11: (DispatchQueue::entry()+0x7a2) [0x7f24fe6a9a02]
> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f24fe4dd23d]
> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 13: (()+0x7dc5) [0x7f24fbfa3dc5]
> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 14: (clone()+0x6d) [0x7f24fae8376d]
>
>
> Jun 12 04:01:33 amssn5 ceph-mds[2544]: starting mds.amssn5 at -
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: *** Caught signal (Segmentation fault) **
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: in thread 7f45d2595700 thread_name:mds_rank_progr
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: ceph version 12.0.3 (f2337d1b42fa49dbb0a93e4048a42762e3dffbbf)
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 1: (()+0x563caf) [0x7f45de46bcaf]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 2: (()+0xf370) [0x7f45dbff1370]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xbb9) [0x7f45de1e8559]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9b1) [0x7f45de217231]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 5: (MDSInternalContextBase::complete(int)+0x1eb) [0x7f45de3f6bcb]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 6: (MDSRank::_advance_queues()+0x4a5) [0x7f45de1a3375]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 7: (MDSRank::ProgressThread::entry()+0x4a) [0x7f45de1a37ea]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 8: (()+0x7dc5) [0x7f45dbfe9dc5]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 9: (clone()+0x6d) [0x7f45daec976d]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 2017-06-12 04:01:43.579491 7f45d2595700 -1 *** Caught signal (Segmentation fault) **
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: in thread 7f45d2595700 thread_name:mds_rank_progr
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: ceph version 12.0.3 (f2337d1b42fa49dbb0a93e4048a42762e3dffbbf)
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 1: (()+0x563caf) [0x7f45de46bcaf]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 2: (()+0xf370) [0x7f45dbff1370]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xbb9) [0x7f45de1e8559]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9b1) [0x7f45de217231]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 5: (MDSInternalContextBase::complete(int)+0x1eb) [0x7f45de3f6bcb]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 6: (MDSRank::_advance_queues()+0x4a5) [0x7f45de1a3375]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 7: (MDSRank::ProgressThread::entry()+0x4a) [0x7f45de1a37ea]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 8: (()+0x7dc5) [0x7f45dbfe9dc5]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 9: (clone()+0x6d) [0x7f45daec976d]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 0> 2017-06-12 04:01:43.579491 7f45d2595700 -1 *** Caught signal (Segmentation fault) **
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: in thread 7f45d2595700 thread_name:mds_rank_progr
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: ceph version 12.0.3 (f2337d1b42fa49dbb0a93e4048a42762e3dffbbf)
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 1: (()+0x563caf) [0x7f45de46bcaf]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 2: (()+0xf370) [0x7f45dbff1370]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xbb9) [0x7f45de1e8559]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9b1) [0x7f45de217231]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 5: (MDSInternalContextBase::complete(int)+0x1eb) [0x7f45de3f6bcb]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 6: (MDSRank::_advance_queues()+0x4a5) [0x7f45de1a3375]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 7: (MDSRank::ProgressThread::entry()+0x4a) [0x7f45de1a37ea]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 8: (()+0x7dc5) [0x7f45dbfe9dc5]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 9: (clone()+0x6d) [0x7f45daec976d]
> Jun 12 04:01:43 amssn5 ceph-mds[2544]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: ceph-mds crash v12.0.3
2017-06-12 10:22 ` John Spray
@ 2017-06-12 10:38 ` Georgi Chorbadzhiyski
2017-06-12 10:56 ` Georgi Chorbadzhiyski
1 sibling, 0 replies; 17+ messages in thread
From: Georgi Chorbadzhiyski @ 2017-06-12 10:38 UTC (permalink / raw)
To: John Spray; +Cc: Ceph Development
On 6/12/17 1:22 PM, John Spray wrote:
> On Mon, Jun 12, 2017 at 5:13 AM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
>> We started getting these on all of our 3 MDS-es. Any idea how to fix it or at least debug
>> it and remove the dir entries that are causing the problem?
>
> Assuming it's easy to reproduce, set "debug mds = 20", "debug ms = 1"
> and gather the logs in the run up to the crash.
I'll try that.
> What is the workload? Is there anything unusual about this directory?
The problem is that I don't know which directory is causing the problems.
I'll try to remove all clients and the try to pinpoint the directory,
because currently the thing is just unusable.
> Has the cluster ever experienced severe damage like a lost PG?
Nope, the cluster was working just fine until two hours ago. We've installed
it last week and tere were no problems until today.
>> [root@amssn3 ~]# yum info ceph-mds
>> Name : ceph-mds
>> Arch : x86_64
>> Epoch : 1
>> Version : 12.0.3
>> Release : 0.el7
>>
>> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: *** Caught signal (Segmentation fault) **
>> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: in thread 7f9e0ae70700 thread_name:mds_rank_progr
>> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: ceph version 12.0.3 (f2337d1b42fa49dbb0a93e4048a42762e3dffbbf)
>> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 1: (()+0x563caf) [0x7f9e16d46caf]
>> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 2: (()+0xf370) [0x7f9e148cc370]
>> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xbb9) [0x7f9e16ac3559]
>> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9b1) [0x7f9e16af2231]
>> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 5: (MDSInternalContextBase::complete(int)+0x1eb) [0x7f9e16cd1bcb]
>> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 6: (MDSRank::_advance_queues()+0x4a5) [0x7f9e16a7e375]
>> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 7: (MDSRank::ProgressThread::entry()+0x4a) [0x7f9e16a7e7ea]
>> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 8: (()+0x7dc5) [0x7f9e148c4dc5]
>> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 9: (clone()+0x6d) [0x7f9e137a476d]
>> Jun 12 04:11:39 amssn1.sgvps.net ceph-mds[3532]: 2017-06-12 04:11:39.585944 7f9e0ae70700 -1 *** Caught signal (Segmentation fault) **
>>
>>
>> Jun 12 03:36:19 amssn3.sgvps.net ceph-mds[3503]: ceph version 12.0.3 (f2337d1b42fa49dbb0a93e4048a42762e3dffbbf)
>> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 1: (()+0x563caf) [0x7f24fe425caf]
>> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 2: (()+0xf370) [0x7f24fbfab370]
>> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xbb9) [0x7f24fe1a2559]
>> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9b1) [0x7f24fe1d1231]
>> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 5: (Server::handle_client_request(MClientRequest*)+0x48d) [0x7f24fe1d1a6d]
>> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 6: (Server::dispatch(Message*)+0x38b) [0x7f24fe1d619b]
>> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 7: (MDSRank::handle_deferrable_message(Message*)+0x7fc) [0x7f24fe152bbc]
>> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 8: (MDSRank::_dispatch(Message*, bool)+0x1eb) [0x7f24fe15db4b]
>> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 9: (MDSRankDispatcher::ms_dispatch(Message*)+0x15) [0x7f24fe15ea95]
>> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 10: (MDSDaemon::ms_dispatch(Message*)+0xf3) [0x7f24fe14a7c3]
>> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 11: (DispatchQueue::entry()+0x7a2) [0x7f24fe6a9a02]
>> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f24fe4dd23d]
>> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 13: (()+0x7dc5) [0x7f24fbfa3dc5]
>> Jun 12 03:36:19 amssn3 ceph-mds[3503]: 14: (clone()+0x6d) [0x7f24fae8376d]
>>
>>
>> Jun 12 04:01:33 amssn5 ceph-mds[2544]: starting mds.amssn5 at -
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: *** Caught signal (Segmentation fault) **
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: in thread 7f45d2595700 thread_name:mds_rank_progr
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: ceph version 12.0.3 (f2337d1b42fa49dbb0a93e4048a42762e3dffbbf)
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 1: (()+0x563caf) [0x7f45de46bcaf]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 2: (()+0xf370) [0x7f45dbff1370]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xbb9) [0x7f45de1e8559]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9b1) [0x7f45de217231]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 5: (MDSInternalContextBase::complete(int)+0x1eb) [0x7f45de3f6bcb]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 6: (MDSRank::_advance_queues()+0x4a5) [0x7f45de1a3375]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 7: (MDSRank::ProgressThread::entry()+0x4a) [0x7f45de1a37ea]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 8: (()+0x7dc5) [0x7f45dbfe9dc5]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 9: (clone()+0x6d) [0x7f45daec976d]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 2017-06-12 04:01:43.579491 7f45d2595700 -1 *** Caught signal (Segmentation fault) **
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: in thread 7f45d2595700 thread_name:mds_rank_progr
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: ceph version 12.0.3 (f2337d1b42fa49dbb0a93e4048a42762e3dffbbf)
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 1: (()+0x563caf) [0x7f45de46bcaf]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 2: (()+0xf370) [0x7f45dbff1370]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xbb9) [0x7f45de1e8559]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9b1) [0x7f45de217231]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 5: (MDSInternalContextBase::complete(int)+0x1eb) [0x7f45de3f6bcb]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 6: (MDSRank::_advance_queues()+0x4a5) [0x7f45de1a3375]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 7: (MDSRank::ProgressThread::entry()+0x4a) [0x7f45de1a37ea]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 8: (()+0x7dc5) [0x7f45dbfe9dc5]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 9: (clone()+0x6d) [0x7f45daec976d]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 0> 2017-06-12 04:01:43.579491 7f45d2595700 -1 *** Caught signal (Segmentation fault) **
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: in thread 7f45d2595700 thread_name:mds_rank_progr
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: ceph version 12.0.3 (f2337d1b42fa49dbb0a93e4048a42762e3dffbbf)
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 1: (()+0x563caf) [0x7f45de46bcaf]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 2: (()+0xf370) [0x7f45dbff1370]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xbb9) [0x7f45de1e8559]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9b1) [0x7f45de217231]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 5: (MDSInternalContextBase::complete(int)+0x1eb) [0x7f45de3f6bcb]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 6: (MDSRank::_advance_queues()+0x4a5) [0x7f45de1a3375]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 7: (MDSRank::ProgressThread::entry()+0x4a) [0x7f45de1a37ea]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 8: (()+0x7dc5) [0x7f45dbfe9dc5]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: 9: (clone()+0x6d) [0x7f45daec976d]
>> Jun 12 04:01:43 amssn5 ceph-mds[2544]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Georgi Chorbadzhiyski | http://georgi.unixsol.org/ | http://github.com/gfto/
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: ceph-mds crash v12.0.3
2017-06-12 10:22 ` John Spray
2017-06-12 10:38 ` Georgi Chorbadzhiyski
@ 2017-06-12 10:56 ` Georgi Chorbadzhiyski
2017-06-12 12:16 ` Georgi Chorbadzhiyski
2017-06-12 12:58 ` Yan, Zheng
1 sibling, 2 replies; 17+ messages in thread
From: Georgi Chorbadzhiyski @ 2017-06-12 10:56 UTC (permalink / raw)
To: John Spray; +Cc: Ceph Development
On 6/12/17 1:22 PM, John Spray wrote:
> On Mon, Jun 12, 2017 at 5:13 AM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
>> We started getting these on all of our 3 MDS-es. Any idea how to fix it or at least debug
>> it and remove the dir entries that are causing the problem?
>
> Assuming it's easy to reproduce, set "debug mds = 20", "debug ms = 1"
> and gather the logs in the run up to the crash.
After turning up debug as you suggested here are the logs before the crash.
-10> 2017-06-12 05:46:05.808399 7fc8038d2700 10 MDSInternalContextBase::complete: 18C_MDS_RetryRequest
-9> 2017-06-12 05:46:05.808401 7fc8038d2700 7 mds.0.server dispatch_client_request client_request(client.467465:121280822 readdir #1000007d3e8 sess_umos5jqii50rttcag852lros73 2017-06-12 03:36:19.073284 RETRY=115 caller_uid=1838, caller_gid=1838{}) v2
-8> 2017-06-12 05:46:05.808407 7fc8038d2700 10 mds.0.server rdlock_path_pin_ref request(client.467465:121280822 cr=0x7fc819988f00) #1000007d3e8
-7> 2017-06-12 05:46:05.808410 7fc8038d2700 10 mds.0.locker acquire_locks request(client.467465:121280822 cr=0x7fc819988f00) - done locking
-6> 2017-06-12 05:46:05.808416 7fc8038d2700 20 Session check_access path /client/shared/site.com/oc-content/uploads/session
-5> 2017-06-12 05:46:05.808418 7fc8038d2700 10 MDSAuthCap is_capable inode(path /client/shared/site.com/oc-content/uploads/session owner 1838:1838 mode 040755) by caller 1838:1838 mask 1 new 1024:524288 cap: MDSAuthCaps[allow *]
-4> 2017-06-12 05:46:05.808423 7fc8038d2700 10 mds.0.server frag 1* offset 'sess_umos5jqii50rttcag852lros73' offset_hash 13141348 flags 0
-3> 2017-06-12 05:46:05.808426 7fc8038d2700 10 mds.0.server adjust frag 1* -> 1000* fragtree_t(*^1 0*^1 1*^3)
-2> 2017-06-12 05:46:05.808430 7fc8038d2700 10 mds.0.server handle_client_readdir on [dir 1000007d3e8.1000* /client/shared/site.com/oc-content/uploads/session/ [2,head] auth pv=1077482 v=1077481 cv=0/0 ap=1+2+2 state=1610612738|complete f(v86 218=218+0) n(v10 b67671 218=218+0) hs=218+6,ss=0+0 dirty=10 | child=1 dirty=1 waiter=0 authpin=1 0x7fc828eae340]
-1> 2017-06-12 05:46:05.808447 7fc8038d2700 10 mds.0.server snapid head
0> 2017-06-12 05:46:05.843735 7fc8038d2700 -1 *** Caught signal (Segmentation fault) **
in thread 7fc8038d2700 thread_name:ms_dispatch
I've looked at mds code but nothing caught my eye and after "snapid head" debug
message unfortunately there is not a lot debug printing going on to get a better
idea what's going on.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: ceph-mds crash v12.0.3
2017-06-12 10:56 ` Georgi Chorbadzhiyski
@ 2017-06-12 12:16 ` Georgi Chorbadzhiyski
2017-06-12 12:58 ` Yan, Zheng
1 sibling, 0 replies; 17+ messages in thread
From: Georgi Chorbadzhiyski @ 2017-06-12 12:16 UTC (permalink / raw)
To: John Spray; +Cc: Ceph Development
On 6/12/17 1:56 PM, Georgi Chorbadzhiyski wrote:
> On 6/12/17 1:22 PM, John Spray wrote:
>> On Mon, Jun 12, 2017 at 5:13 AM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
>>> We started getting these on all of our 3 MDS-es. Any idea how to fix it or at least debug
>>> it and remove the dir entries that are causing the problem?
>>
>> Assuming it's easy to reproduce, set "debug mds = 20", "debug ms = 1"
>> and gather the logs in the run up to the crash.
>
> After turning up debug as you suggested here are the logs before the crash.
>
> -10> 2017-06-12 05:46:05.808399 7fc8038d2700 10 MDSInternalContextBase::complete: 18C_MDS_RetryRequest
> -9> 2017-06-12 05:46:05.808401 7fc8038d2700 7 mds.0.server dispatch_client_request client_request(client.467465:121280822 readdir #1000007d3e8 sess_umos5jqii50rttcag852lros73 2017-06-12 03:36:19.073284 RETRY=115 caller_uid=1838, caller_gid=1838{}) v2
> -8> 2017-06-12 05:46:05.808407 7fc8038d2700 10 mds.0.server rdlock_path_pin_ref request(client.467465:121280822 cr=0x7fc819988f00) #1000007d3e8
> -7> 2017-06-12 05:46:05.808410 7fc8038d2700 10 mds.0.locker acquire_locks request(client.467465:121280822 cr=0x7fc819988f00) - done locking
> -6> 2017-06-12 05:46:05.808416 7fc8038d2700 20 Session check_access path /client/shared/site.com/oc-content/uploads/session
> -5> 2017-06-12 05:46:05.808418 7fc8038d2700 10 MDSAuthCap is_capable inode(path /client/shared/site.com/oc-content/uploads/session owner 1838:1838 mode 040755) by caller 1838:1838 mask 1 new 1024:524288 cap: MDSAuthCaps[allow *]
>
> -4> 2017-06-12 05:46:05.808423 7fc8038d2700 10 mds.0.server frag 1* offset 'sess_umos5jqii50rttcag852lros73' offset_hash 13141348 flags 0
> -3> 2017-06-12 05:46:05.808426 7fc8038d2700 10 mds.0.server adjust frag 1* -> 1000* fragtree_t(*^1 0*^1 1*^3)
> -2> 2017-06-12 05:46:05.808430 7fc8038d2700 10 mds.0.server handle_client_readdir on [dir 1000007d3e8.1000* /client/shared/site.com/oc-content/uploads/session/ [2,head] auth pv=1077482 v=1077481 cv=0/0 ap=1+2+2 state=1610612738|complete f(v86 218=218+0) n(v10 b67671 218=218+0) hs=218+6,ss=0+0 dirty=10 | child=1 dirty=1 waiter=0 authpin=1 0x7fc828eae340]
> -1> 2017-06-12 05:46:05.808447 7fc8038d2700 10 mds.0.server snapid head
> 0> 2017-06-12 05:46:05.843735 7fc8038d2700 -1 *** Caught signal (Segmentation fault) **
> in thread 7fc8038d2700 thread_name:ms_dispatch
>
> I've looked at mds code but nothing caught my eye and after "snapid head" debug
> message unfortunately there is not a lot debug printing going on to get a better
> idea what's going on.
A bit more info about the directory structure. The directory from the log above
had >3500 session files names. The debug from the crashes always mentioned this
file: sess_umos5jqii50rttcag852lros73 (unfortunately it was deleted before
I tried to preserve it).
The parent directory is wordpress 'uploads' directory and contains > 78000 files
(pictures and thumbnails).
I've workaround the problem by stopping all cephfs clients and renaming 'session'
directory. I'm currently unable to recreate the problem but I have the files
and the directory structure.
root@cephfs-client:/mnt/test/client/shared/site.com/oc-content/uploads# du -sh .
3.1G .
root@cephfs-client:/mnt/test/client/shared/site.com/oc-content/uploads# find -type f | wc -l
78321
root@cephfs-client:/mnt/test/client/shared/site.com/oc-content/uploads# ls -l | wc -l
54068
root@cephfs-client:/mnt/test/client/shared/site.com/oc-content/uploads# ls -l session/ | wc -l
3513
root@cephfs-client:/mnt/test/client/shared/site.com/oc-content/uploads# find -type d | wc -l
2435
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: ceph-mds crash v12.0.3
2017-06-12 10:56 ` Georgi Chorbadzhiyski
2017-06-12 12:16 ` Georgi Chorbadzhiyski
@ 2017-06-12 12:58 ` Yan, Zheng
2017-06-12 14:45 ` Georgi Chorbadzhiyski
1 sibling, 1 reply; 17+ messages in thread
From: Yan, Zheng @ 2017-06-12 12:58 UTC (permalink / raw)
To: Georgi Chorbadzhiyski; +Cc: John Spray, Ceph Development
On Mon, Jun 12, 2017 at 6:56 PM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
> On 6/12/17 1:22 PM, John Spray wrote:
>> On Mon, Jun 12, 2017 at 5:13 AM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
>>> We started getting these on all of our 3 MDS-es. Any idea how to fix it or at least debug
>>> it and remove the dir entries that are causing the problem?
>>
>> Assuming it's easy to reproduce, set "debug mds = 20", "debug ms = 1"
>> and gather the logs in the run up to the crash.
>
> After turning up debug as you suggested here are the logs before the crash.
>
> -10> 2017-06-12 05:46:05.808399 7fc8038d2700 10 MDSInternalContextBase::complete: 18C_MDS_RetryRequest
> -9> 2017-06-12 05:46:05.808401 7fc8038d2700 7 mds.0.server dispatch_client_request client_request(client.467465:121280822 readdir #1000007d3e8 sess_umos5jqii50rttcag852lros73 2017-06-12 03:36:19.073284 RETRY=115 caller_uid=1838, caller_gid=1838{}) v2
> -8> 2017-06-12 05:46:05.808407 7fc8038d2700 10 mds.0.server rdlock_path_pin_ref request(client.467465:121280822 cr=0x7fc819988f00) #1000007d3e8
> -7> 2017-06-12 05:46:05.808410 7fc8038d2700 10 mds.0.locker acquire_locks request(client.467465:121280822 cr=0x7fc819988f00) - done locking
> -6> 2017-06-12 05:46:05.808416 7fc8038d2700 20 Session check_access path /client/shared/site.com/oc-content/uploads/session
> -5> 2017-06-12 05:46:05.808418 7fc8038d2700 10 MDSAuthCap is_capable inode(path /client/shared/site.com/oc-content/uploads/session owner 1838:1838 mode 040755) by caller 1838:1838 mask 1 new 1024:524288 cap: MDSAuthCaps[allow *]
>
> -4> 2017-06-12 05:46:05.808423 7fc8038d2700 10 mds.0.server frag 1* offset 'sess_umos5jqii50rttcag852lros73' offset_hash 13141348 flags 0
> -3> 2017-06-12 05:46:05.808426 7fc8038d2700 10 mds.0.server adjust frag 1* -> 1000* fragtree_t(*^1 0*^1 1*^3)
> -2> 2017-06-12 05:46:05.808430 7fc8038d2700 10 mds.0.server handle_client_readdir on [dir 1000007d3e8.1000* /client/shared/site.com/oc-content/uploads/session/ [2,head] auth pv=1077482 v=1077481 cv=0/0 ap=1+2+2 state=1610612738|complete f(v86 218=218+0) n(v10 b67671 218=218+0) hs=218+6,ss=0+0 dirty=10 | child=1 dirty=1 waiter=0 authpin=1 0x7fc828eae340]
> -1> 2017-06-12 05:46:05.808447 7fc8038d2700 10 mds.0.server snapid head
> 0> 2017-06-12 05:46:05.843735 7fc8038d2700 -1 *** Caught signal (Segmentation fault) **
> in thread 7fc8038d2700 thread_name:ms_dispatch
>
The log doesn't contain useful information for this issue. Could you
please enable coredump try again. Besides, please install debug info
for ceph-mds or re-compile ceph with debug info. After got coredump,
using gdb to debug this issue is convenient.
Regards
Yan, Zheng
> I've looked at mds code but nothing caught my eye and after "snapid head" debug
> message unfortunately there is not a lot debug printing going on to get a better
> idea what's going on.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: ceph-mds crash v12.0.3
2017-06-12 12:58 ` Yan, Zheng
@ 2017-06-12 14:45 ` Georgi Chorbadzhiyski
2017-06-13 11:46 ` Yan, Zheng
0 siblings, 1 reply; 17+ messages in thread
From: Georgi Chorbadzhiyski @ 2017-06-12 14:45 UTC (permalink / raw)
To: Yan, Zheng; +Cc: John Spray, Ceph Development
On 6/12/17 3:58 PM, Yan, Zheng wrote:
> On Mon, Jun 12, 2017 at 6:56 PM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
>> On 6/12/17 1:22 PM, John Spray wrote:
>>> On Mon, Jun 12, 2017 at 5:13 AM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
>>>> We started getting these on all of our 3 MDS-es. Any idea how to fix it or at least debug
>>>> it and remove the dir entries that are causing the problem?
>>>
>>> Assuming it's easy to reproduce, set "debug mds = 20", "debug ms = 1"
>>> and gather the logs in the run up to the crash.
>>
>> After turning up debug as you suggested here are the logs before the crash.
>>
>> -10> 2017-06-12 05:46:05.808399 7fc8038d2700 10 MDSInternalContextBase::complete: 18C_MDS_RetryRequest
>> -9> 2017-06-12 05:46:05.808401 7fc8038d2700 7 mds.0.server dispatch_client_request client_request(client.467465:121280822 readdir #1000007d3e8 sess_umos5jqii50rttcag852lros73 2017-06-12 03:36:19.073284 RETRY=115 caller_uid=1838, caller_gid=1838{}) v2
>> -8> 2017-06-12 05:46:05.808407 7fc8038d2700 10 mds.0.server rdlock_path_pin_ref request(client.467465:121280822 cr=0x7fc819988f00) #1000007d3e8
>> -7> 2017-06-12 05:46:05.808410 7fc8038d2700 10 mds.0.locker acquire_locks request(client.467465:121280822 cr=0x7fc819988f00) - done locking
>> -6> 2017-06-12 05:46:05.808416 7fc8038d2700 20 Session check_access path /client/shared/site.com/oc-content/uploads/session
>> -5> 2017-06-12 05:46:05.808418 7fc8038d2700 10 MDSAuthCap is_capable inode(path /client/shared/site.com/oc-content/uploads/session owner 1838:1838 mode 040755) by caller 1838:1838 mask 1 new 1024:524288 cap: MDSAuthCaps[allow *]
>>
>> -4> 2017-06-12 05:46:05.808423 7fc8038d2700 10 mds.0.server frag 1* offset 'sess_umos5jqii50rttcag852lros73' offset_hash 13141348 flags 0
>> -3> 2017-06-12 05:46:05.808426 7fc8038d2700 10 mds.0.server adjust frag 1* -> 1000* fragtree_t(*^1 0*^1 1*^3)
>> -2> 2017-06-12 05:46:05.808430 7fc8038d2700 10 mds.0.server handle_client_readdir on [dir 1000007d3e8.1000* /client/shared/site.com/oc-content/uploads/session/ [2,head] auth pv=1077482 v=1077481 cv=0/0 ap=1+2+2 state=1610612738|complete f(v86 218=218+0) n(v10 b67671 218=218+0) hs=218+6,ss=0+0 dirty=10 | child=1 dirty=1 waiter=0 authpin=1 0x7fc828eae340]
>> -1> 2017-06-12 05:46:05.808447 7fc8038d2700 10 mds.0.server snapid head
>> 0> 2017-06-12 05:46:05.843735 7fc8038d2700 -1 *** Caught signal (Segmentation fault) **
>> in thread 7fc8038d2700 thread_name:ms_dispatch
>
> The log doesn't contain useful information for this issue. Could you
> please enable coredump try again. Besides, please install debug info
> for ceph-mds or re-compile ceph with debug info. After got coredump,
> using gdb to debug this issue is convenient.
I have core dump, where can I get the debug info for official centos 7
packages?
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: ceph-mds crash v12.0.3
2017-06-12 14:45 ` Georgi Chorbadzhiyski
@ 2017-06-13 11:46 ` Yan, Zheng
2017-06-14 14:40 ` Georgi Chorbadzhiyski
0 siblings, 1 reply; 17+ messages in thread
From: Yan, Zheng @ 2017-06-13 11:46 UTC (permalink / raw)
To: Georgi Chorbadzhiyski; +Cc: John Spray, Ceph Development
On Mon, Jun 12, 2017 at 10:45 PM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
> On 6/12/17 3:58 PM, Yan, Zheng wrote:
>> On Mon, Jun 12, 2017 at 6:56 PM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
>>> On 6/12/17 1:22 PM, John Spray wrote:
>>>> On Mon, Jun 12, 2017 at 5:13 AM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
>>>>> We started getting these on all of our 3 MDS-es. Any idea how to fix it or at least debug
>>>>> it and remove the dir entries that are causing the problem?
>>>>
>>>> Assuming it's easy to reproduce, set "debug mds = 20", "debug ms = 1"
>>>> and gather the logs in the run up to the crash.
>>>
>>> After turning up debug as you suggested here are the logs before the crash.
>>>
>>> -10> 2017-06-12 05:46:05.808399 7fc8038d2700 10 MDSInternalContextBase::complete: 18C_MDS_RetryRequest
>>> -9> 2017-06-12 05:46:05.808401 7fc8038d2700 7 mds.0.server dispatch_client_request client_request(client.467465:121280822 readdir #1000007d3e8 sess_umos5jqii50rttcag852lros73 2017-06-12 03:36:19.073284 RETRY=115 caller_uid=1838, caller_gid=1838{}) v2
>>> -8> 2017-06-12 05:46:05.808407 7fc8038d2700 10 mds.0.server rdlock_path_pin_ref request(client.467465:121280822 cr=0x7fc819988f00) #1000007d3e8
>>> -7> 2017-06-12 05:46:05.808410 7fc8038d2700 10 mds.0.locker acquire_locks request(client.467465:121280822 cr=0x7fc819988f00) - done locking
>>> -6> 2017-06-12 05:46:05.808416 7fc8038d2700 20 Session check_access path /client/shared/site.com/oc-content/uploads/session
>>> -5> 2017-06-12 05:46:05.808418 7fc8038d2700 10 MDSAuthCap is_capable inode(path /client/shared/site.com/oc-content/uploads/session owner 1838:1838 mode 040755) by caller 1838:1838 mask 1 new 1024:524288 cap: MDSAuthCaps[allow *]
>>>
>>> -4> 2017-06-12 05:46:05.808423 7fc8038d2700 10 mds.0.server frag 1* offset 'sess_umos5jqii50rttcag852lros73' offset_hash 13141348 flags 0
>>> -3> 2017-06-12 05:46:05.808426 7fc8038d2700 10 mds.0.server adjust frag 1* -> 1000* fragtree_t(*^1 0*^1 1*^3)
>>> -2> 2017-06-12 05:46:05.808430 7fc8038d2700 10 mds.0.server handle_client_readdir on [dir 1000007d3e8.1000* /client/shared/site.com/oc-content/uploads/session/ [2,head] auth pv=1077482 v=1077481 cv=0/0 ap=1+2+2 state=1610612738|complete f(v86 218=218+0) n(v10 b67671 218=218+0) hs=218+6,ss=0+0 dirty=10 | child=1 dirty=1 waiter=0 authpin=1 0x7fc828eae340]
>>> -1> 2017-06-12 05:46:05.808447 7fc8038d2700 10 mds.0.server snapid head
>>> 0> 2017-06-12 05:46:05.843735 7fc8038d2700 -1 *** Caught signal (Segmentation fault) **
>>> in thread 7fc8038d2700 thread_name:ms_dispatch
>>
>> The log doesn't contain useful information for this issue. Could you
>> please enable coredump try again. Besides, please install debug info
>> for ceph-mds or re-compile ceph with debug info. After got coredump,
>> using gdb to debug this issue is convenient.
>
> I have core dump, where can I get the debug info for official centos 7
> packages?
I don't use centos. I found link
https://wiki.centos.org/AdditionalResources/Repositories/DebugInfo
Regards
Yan, Zheng
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: ceph-mds crash v12.0.3
2017-06-13 11:46 ` Yan, Zheng
@ 2017-06-14 14:40 ` Georgi Chorbadzhiyski
2017-06-15 8:27 ` Yan, Zheng
0 siblings, 1 reply; 17+ messages in thread
From: Georgi Chorbadzhiyski @ 2017-06-14 14:40 UTC (permalink / raw)
To: Yan, Zheng; +Cc: John Spray, Ceph Development
On 6/13/17 2:46 PM, Yan, Zheng wrote:
> On Mon, Jun 12, 2017 at 10:45 PM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
>> On 6/12/17 3:58 PM, Yan, Zheng wrote:
>>> On Mon, Jun 12, 2017 at 6:56 PM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
>>>> On 6/12/17 1:22 PM, John Spray wrote:
>>>>> On Mon, Jun 12, 2017 at 5:13 AM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
>>>>>> We started getting these on all of our 3 MDS-es. Any idea how to fix it or at least debug
>>>>>> it and remove the dir entries that are causing the problem?
>>>>>
>>>>> Assuming it's easy to reproduce, set "debug mds = 20", "debug ms = 1"
>>>>> and gather the logs in the run up to the crash.
>>>>
>>>> After turning up debug as you suggested here are the logs before the crash.
>>>>
>>>> -10> 2017-06-12 05:46:05.808399 7fc8038d2700 10 MDSInternalContextBase::complete: 18C_MDS_RetryRequest
>>>> -9> 2017-06-12 05:46:05.808401 7fc8038d2700 7 mds.0.server dispatch_client_request client_request(client.467465:121280822 readdir #1000007d3e8 sess_umos5jqii50rttcag852lros73 2017-06-12 03:36:19.073284 RETRY=115 caller_uid=1838, caller_gid=1838{}) v2
>>>> -8> 2017-06-12 05:46:05.808407 7fc8038d2700 10 mds.0.server rdlock_path_pin_ref request(client.467465:121280822 cr=0x7fc819988f00) #1000007d3e8
>>>> -7> 2017-06-12 05:46:05.808410 7fc8038d2700 10 mds.0.locker acquire_locks request(client.467465:121280822 cr=0x7fc819988f00) - done locking
>>>> -6> 2017-06-12 05:46:05.808416 7fc8038d2700 20 Session check_access path /client/shared/site.com/oc-content/uploads/session
>>>> -5> 2017-06-12 05:46:05.808418 7fc8038d2700 10 MDSAuthCap is_capable inode(path /client/shared/site.com/oc-content/uploads/session owner 1838:1838 mode 040755) by caller 1838:1838 mask 1 new 1024:524288 cap: MDSAuthCaps[allow *]
>>>>
>>>> -4> 2017-06-12 05:46:05.808423 7fc8038d2700 10 mds.0.server frag 1* offset 'sess_umos5jqii50rttcag852lros73' offset_hash 13141348 flags 0
>>>> -3> 2017-06-12 05:46:05.808426 7fc8038d2700 10 mds.0.server adjust frag 1* -> 1000* fragtree_t(*^1 0*^1 1*^3)
>>>> -2> 2017-06-12 05:46:05.808430 7fc8038d2700 10 mds.0.server handle_client_readdir on [dir 1000007d3e8.1000* /client/shared/site.com/oc-content/uploads/session/ [2,head] auth pv=1077482 v=1077481 cv=0/0 ap=1+2+2 state=1610612738|complete f(v86 218=218+0) n(v10 b67671 218=218+0) hs=218+6,ss=0+0 dirty=10 | child=1 dirty=1 waiter=0 authpin=1 0x7fc828eae340]
>>>> -1> 2017-06-12 05:46:05.808447 7fc8038d2700 10 mds.0.server snapid head
>>>> 0> 2017-06-12 05:46:05.843735 7fc8038d2700 -1 *** Caught signal (Segmentation fault) **
>>>> in thread 7fc8038d2700 thread_name:ms_dispatch
>>>
>>> The log doesn't contain useful information for this issue. Could you
>>> please enable coredump try again. Besides, please install debug info
>>> for ceph-mds or re-compile ceph with debug info. After got coredump,
>>> using gdb to debug this issue is convenient.
>>
>> I have core dump, where can I get the debug info for official centos 7
>> packages?
>
> I don't use centos. I found link
> https://wiki.centos.org/AdditionalResources/Repositories/DebugInfo
I already have this :) I needed ceph binaries with debug info, not the generic
ones.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: ceph-mds crash v12.0.3
2017-06-14 14:40 ` Georgi Chorbadzhiyski
@ 2017-06-15 8:27 ` Yan, Zheng
0 siblings, 0 replies; 17+ messages in thread
From: Yan, Zheng @ 2017-06-15 8:27 UTC (permalink / raw)
To: Georgi Chorbadzhiyski; +Cc: John Spray, Ceph Development
On Wed, Jun 14, 2017 at 10:40 PM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
> On 6/13/17 2:46 PM, Yan, Zheng wrote:
>> On Mon, Jun 12, 2017 at 10:45 PM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
>>> On 6/12/17 3:58 PM, Yan, Zheng wrote:
>>>> On Mon, Jun 12, 2017 at 6:56 PM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
>>>>> On 6/12/17 1:22 PM, John Spray wrote:
>>>>>> On Mon, Jun 12, 2017 at 5:13 AM, Georgi Chorbadzhiyski <gf@unixsol.org> wrote:
>>>>>>> We started getting these on all of our 3 MDS-es. Any idea how to fix it or at least debug
>>>>>>> it and remove the dir entries that are causing the problem?
>>>>>>
>>>>>> Assuming it's easy to reproduce, set "debug mds = 20", "debug ms = 1"
>>>>>> and gather the logs in the run up to the crash.
>>>>>
>>>>> After turning up debug as you suggested here are the logs before the crash.
>>>>>
>>>>> -10> 2017-06-12 05:46:05.808399 7fc8038d2700 10 MDSInternalContextBase::complete: 18C_MDS_RetryRequest
>>>>> -9> 2017-06-12 05:46:05.808401 7fc8038d2700 7 mds.0.server dispatch_client_request client_request(client.467465:121280822 readdir #1000007d3e8 sess_umos5jqii50rttcag852lros73 2017-06-12 03:36:19.073284 RETRY=115 caller_uid=1838, caller_gid=1838{}) v2
>>>>> -8> 2017-06-12 05:46:05.808407 7fc8038d2700 10 mds.0.server rdlock_path_pin_ref request(client.467465:121280822 cr=0x7fc819988f00) #1000007d3e8
>>>>> -7> 2017-06-12 05:46:05.808410 7fc8038d2700 10 mds.0.locker acquire_locks request(client.467465:121280822 cr=0x7fc819988f00) - done locking
>>>>> -6> 2017-06-12 05:46:05.808416 7fc8038d2700 20 Session check_access path /client/shared/site.com/oc-content/uploads/session
>>>>> -5> 2017-06-12 05:46:05.808418 7fc8038d2700 10 MDSAuthCap is_capable inode(path /client/shared/site.com/oc-content/uploads/session owner 1838:1838 mode 040755) by caller 1838:1838 mask 1 new 1024:524288 cap: MDSAuthCaps[allow *]
>>>>>
>>>>> -4> 2017-06-12 05:46:05.808423 7fc8038d2700 10 mds.0.server frag 1* offset 'sess_umos5jqii50rttcag852lros73' offset_hash 13141348 flags 0
>>>>> -3> 2017-06-12 05:46:05.808426 7fc8038d2700 10 mds.0.server adjust frag 1* -> 1000* fragtree_t(*^1 0*^1 1*^3)
>>>>> -2> 2017-06-12 05:46:05.808430 7fc8038d2700 10 mds.0.server handle_client_readdir on [dir 1000007d3e8.1000* /client/shared/site.com/oc-content/uploads/session/ [2,head] auth pv=1077482 v=1077481 cv=0/0 ap=1+2+2 state=1610612738|complete f(v86 218=218+0) n(v10 b67671 218=218+0) hs=218+6,ss=0+0 dirty=10 | child=1 dirty=1 waiter=0 authpin=1 0x7fc828eae340]
>>>>> -1> 2017-06-12 05:46:05.808447 7fc8038d2700 10 mds.0.server snapid head
>>>>> 0> 2017-06-12 05:46:05.843735 7fc8038d2700 -1 *** Caught signal (Segmentation fault) **
>>>>> in thread 7fc8038d2700 thread_name:ms_dispatch
>>>>
>>>> The log doesn't contain useful information for this issue. Could you
>>>> please enable coredump try again. Besides, please install debug info
>>>> for ceph-mds or re-compile ceph with debug info. After got coredump,
>>>> using gdb to debug this issue is convenient.
>>>
>>> I have core dump, where can I get the debug info for official centos 7
>>> packages?
>>
>> I don't use centos. I found link
>> https://wiki.centos.org/AdditionalResources/Repositories/DebugInfo
>
> I already have this :) I needed ceph binaries with debug info, not the generic
> ones.
where did you get the ceph 12.0.3 packages. debuginfo packages are
probably at the same place.
Regards
Yan, Zheng
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: ceph-mds crash v12.0.3
2017-06-16 8:19 ` Jake Grimmett
@ 2017-06-16 13:13 ` Jake Grimmett
0 siblings, 0 replies; 17+ messages in thread
From: Jake Grimmett @ 2017-06-16 13:13 UTC (permalink / raw)
To: Yan, Zheng; +Cc: ceph-devel
Hi Yan,
I've just checked my build process again...
Your patch did get applied to journal.cc in the tree cloned from git.
However, when I ran make-dist the resulting
ceph-12.0.3-1744-g84d57eb.tar.bz2 now contains an un-patched journal.cc
- presumably make-dist is downloading a new copy of journal.cc from github?
If I run ./do_cmake.sh ; cd build ; make
the new ceph-mds binary works perfectly,
So I've copied the good copy over /usr/bin/ceph-mds and, good news, my
MDS servers now work, so the the file system is accessible.
By the way, I recall Greg Farnum warning against snapshots in June 2016:
<http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-June/010812.html>
Are snapshots still considered to be highly dangerous? And if so, is
there a likelyhood of this changing in the next year?
thanks again,
Jake
On 16/06/17 09:19, Jake Grimmett wrote:
> Hi Yan,
>
> Many thanks for getting back to me - sorry to cause you bother.
>
> I think I'm patching OK, but can you please check my methodology?
>
> git clone git://github.com/ceph/ceph ; cd ceph
>
> git apply ceph-mds.patch ; ./make-srpm.sh
>
> rpmbuild --rebuild /root/ceph/ceph/ceph-12.0.3-1661-g3ddbfcd.el7.src.rpm
>
>
> here is the section of the patched src/mds/journal.cc
>
> 2194 // note which segments inodes belong to, so we don't have to
> start rejournaling them
> 2195 for (const auto &ino : inos) {
> 2196 CInode *in = mds->mdcache->get_inode(ino);
> 2197 if (!in) {
> 2198 dout(0) << "EOpen.replay ino " << ino << " not in
> metablob" << dendl;
> 2199 assert(in);
> 2200 }
> 2201 _segment->open_files.push_back(&in->item_open_file);
> 2202 }
> 2203 for (const auto &vino : snap_inos) {
> 2204 CInode *in = mds->mdcache->get_inode(vino);
> 2205 if (!in) {
> 2206 dout(0) << "EOpen.replay ino " << vino << " not in
> metablob" << dendl;
> 2207 continue;
> 2208 }
>
> many thanks for your time,
>
> Jake
>
>
> On 16/06/17 08:04, Yan, Zheng wrote:
>> On Thu, Jun 15, 2017 at 7:32 PM, Jake Grimmett <jog@mrc-lmb.cam.ac.uk> wrote:
>>> Hi Yan,
>>>
>>> Many thanks for looking into this and providing a patch.
>>>
>>> I've downloaded ceph 12.0.3-1661-g3ddbfcd, applied your patch, rebuilt
>>> the rpms, and installed across my cluster.
>>>
>>> Unfortunately, the MDS are still crashing, any ideas welcome :)
>>>
>>> With "debug_mds = 10" the full Log is 140MB, a truncated version of the
>>> log immediately preceding the crash follows:
>>>
>>> best,
>>>
>>> Jake
>>>
>>> -5> 2017-06-15 12:21:14.084373 7f77fe590700 10 mds.0.journal
>>> EMetaBlob.replay added (full) [dentry
>>> #1/isilon/sc/users/spc/JessComb_AB_230115/JessB_TO_190115_F6_1/n0/JessB_TO_190115_F6_1.peaks_int
>>> [9f,head] auth NULL (dversion lock) v=3104 inode=0
>>> state=1073741888|bottomlru 0x7f781a3f1860]
>>> -4> 2017-06-15 12:21:14.084375 7f77fe590700 10 mds.0.journal
>>> EMetaBlob.replay added [inode 1000147f773 [9f,head]
>>> /isilon/sc/users/spc/JessComb_AB_230115/JessB_TO_190115_F6_1/n0/JessB_TO_190115_F6_1.peaks_int
>>> auth v3104 s=4 n(v0 b4 1=1+0) (iversion lock) cr={3554272=0-4194304@9e}
>>> 0x7f781a3f5800]
>>> -3> 2017-06-15 12:21:14.084379 7f77fe590700 10 mds.0.journal
>>> EMetaBlob.replay added (full) [dentry
>>> #1/isilon/sc/users/spc/JessComb_AB_230115/JessB_TO_190115_F6_1/n0/JessB_TO_190115_F6_1.peaks_maxt
>>> [9f,head] auth NULL (dversion lock) v=3132 inode=0
>>> state=1073741888|bottomlru 0x7f781a3f1d40]
>>> -2> 2017-06-15 12:21:14.084381 7f77fe590700 10 mds.0.journal
>>> EMetaBlob.replay added [inode 1000147f775 [9f,head]
>>> /isilon/sc/users/spc/JessComb_AB_230115/JessB_TO_190115_F6_1/n0/JessB_TO_190115_F6_1.peaks_maxt
>>> auth v3132 s=4 n(v0 b4 1=1+0) (iversion lock) cr={3554272=0-4194304@9e}
>>> 0x7f781a3f5e00]
>>> -1> 2017-06-15 12:21:14.084406 7f77fe590700 0 mds.0.journal
>>> EOpen.replay ino 1000147761b.9a not in metablob
>>> 0> 2017-06-15 12:21:14.085348 7f77fe590700 -1
>>> /root/rpmbuild/BUILD/ceph-12.0.3-1661-g3ddbfcd/src/mds/journal.cc: In
>>> function 'virtual void EOpen::replay(MDSRank*)' thread 7f77fe590700 time
>>> 2017-06-15 12:21:14.084409
>>> /root/rpmbuild/BUILD/ceph-12.0.3-1661-g3ddbfcd/src/mds/journal.cc: 2207:
>>> FAILED assert(in)
>>>
>> The assertion should be removed by my patch. Maybe you didn't cleanly
>> apply the patch.
>>
>>
>> Regards
>> Yan, Zheng
>>
>>> ceph version 12.0.3-1661-g3ddbfcd
>>> (3ddbfcd4357ab3a3c2f17f86f88dc83172d4ce0d) luminous (dev)
>>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>>> const*)+0x110) [0x7f780d290500]
>>> 2: (EOpen::replay(MDSRank*)+0x3e5) [0x7f780d2397b5]
>>> 3: (MDLog::_replay_thread()+0x5f2) [0x7f780d1efd12]
>>> 4: (MDLog::ReplayThread::entry()+0xd) [0x7f780cf9b6ad]
>>> 5: (()+0x7dc5) [0x7f780adb4dc5]
>>> 6: (clone()+0x6d) [0x7f7809e9476d]
>>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>>> needed to interpret this.
>>>
>>> --- logging levels ---
>>> 0/ 5 none
>>> 0/ 1 lockdep
>>> 0/ 1 context
>>> 1/ 1 crush
>>> 10/10 mds
>>> 1/ 5 mds_balancer
>>> 1/ 5 mds_locker
>>> 1/ 5 mds_log
>>> 1/ 5 mds_log_expire
>>> 1/ 5 mds_migrator
>>> 0/ 1 buffer
>>> 0/ 1 timer
>>> 0/ 1 filer
>>> 0/ 1 striper
>>> 0/ 1 objecter
>>> 0/ 5 rados
>>> 0/ 5 rbd
>>> 0/ 5 rbd_mirror
>>> 0/ 5 rbd_replay
>>> 0/ 5 journaler
>>> 0/ 5 objectcacher
>>> 0/ 5 client
>>> 1/ 5 osd
>>> 0/ 5 optracker
>>> 0/ 5 objclass
>>> 1/ 3 filestore
>>> 1/ 3 journal
>>> 0/ 5 ms
>>> 1/ 5 mon
>>> 0/10 monc
>>> 1/ 5 paxos
>>> 0/ 5 tp
>>> 1/ 5 auth
>>> 1/ 5 crypto
>>> 1/ 1 finisher
>>> 1/ 5 heartbeatmap
>>> 1/ 5 perfcounter
>>> 1/ 5 rgw
>>> 1/10 civetweb
>>> 1/ 5 javaclient
>>> 1/ 5 asok
>>> 1/ 1 throttle
>>> 0/ 0 refs
>>> 1/ 5 xio
>>> 1/ 5 compressor
>>> 1/ 5 bluestore
>>> 1/ 5 bluefs
>>> 1/ 3 bdev
>>> 1/ 5 kstore
>>> 4/ 5 rocksdb
>>> 4/ 5 leveldb
>>> 4/ 5 memdb
>>> 1/ 5 kinetic
>>> 1/ 5 fuse
>>> 1/ 5 mgr
>>> 1/ 5 mgrc
>>> 1/ 5 dpdk
>>> 1/ 5 eventtrace
>>> -2/-2 (syslog threshold)
>>> -1/-1 (stderr threshold)
>>> max_recent 10000
>>> max_new 1000
>>> log_file /var/log/ceph/ceph-mds.cephfs1.log
>>> --- end dump of recent events ---
>>> 2017-06-15 12:21:14.101761 7f77fe590700 -1 *** Caught signal (Aborted) **
>>> in thread 7f77fe590700 thread_name:md_log_replay
>>>
>>> ceph version 12.0.3-1661-g3ddbfcd
>>> (3ddbfcd4357ab3a3c2f17f86f88dc83172d4ce0d) luminous (dev)
>>> 1: (()+0x57d7ff) [0x7f780d2507ff]
>>> 2: (()+0xf370) [0x7f780adbc370]
>>> 3: (gsignal()+0x37) [0x7f7809dd21d7]
>>> 4: (abort()+0x148) [0x7f7809dd38c8]
>>> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>>> const*)+0x284) [0x7f780d290674]
>>> 6: (EOpen::replay(MDSRank*)+0x3e5) [0x7f780d2397b5]
>>> 7: (MDLog::_replay_thread()+0x5f2) [0x7f780d1efd12]
>>> 8: (MDLog::ReplayThread::entry()+0xd) [0x7f780cf9b6ad]
>>> 9: (()+0x7dc5) [0x7f780adb4dc5]
>>> 10: (clone()+0x6d) [0x7f7809e9476d]
>>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>>> needed to interpret this.
>>>
>>> --- begin dump of recent events ---
>>> 0> 2017-06-15 12:21:14.101761 7f77fe590700 -1 *** Caught signal
>>> (Aborted) **
>>> in thread 7f77fe590700 thread_name:md_log_replay
>>>
>>> ceph version 12.0.3-1661-g3ddbfcd
>>> (3ddbfcd4357ab3a3c2f17f86f88dc83172d4ce0d) luminous (dev)
>>> 1: (()+0x57d7ff) [0x7f780d2507ff]
>>> 2: (()+0xf370) [0x7f780adbc370]
>>> 3: (gsignal()+0x37) [0x7f7809dd21d7]
>>> 4: (abort()+0x148) [0x7f7809dd38c8]
>>> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>>> const*)+0x284) [0x7f780d290674]
>>> 6: (EOpen::replay(MDSRank*)+0x3e5) [0x7f780d2397b5]
>>> 7: (MDLog::_replay_thread()+0x5f2) [0x7f780d1efd12]
>>> 8: (MDLog::ReplayThread::entry()+0xd) [0x7f780cf9b6ad]
>>> 9: (()+0x7dc5) [0x7f780adb4dc5]
>>> 10: (clone()+0x6d) [0x7f7809e9476d]
>>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>>> needed to interpret this.
>>>
>>> --- logging levels ---
>>> 0/ 5 none
>>> 0/ 1 lockdep
>>> 0/ 1 context
>>> 1/ 1 crush
>>> 10/10 mds
>>> 1/ 5 mds_balancer
>>> 1/ 5 mds_locker
>>> 1/ 5 mds_log
>>> 1/ 5 mds_log_expire
>>> 1/ 5 mds_migrator
>>> 0/ 1 buffer
>>> 0/ 1 timer
>>> 0/ 1 filer
>>> 0/ 1 striper
>>> 0/ 1 objecter
>>> 0/ 5 rados
>>> 0/ 5 rbd
>>> 0/ 5 rbd_mirror
>>> 0/ 5 rbd_replay
>>> 0/ 5 journaler
>>> 0/ 5 objectcacher
>>> 0/ 5 client
>>> 1/ 5 osd
>>> 0/ 5 optracker
>>> 0/ 5 objclass
>>> 1/ 3 filestore
>>> 1/ 3 journal
>>> 0/ 5 ms
>>> 1/ 5 mon
>>> 0/10 monc
>>> 1/ 5 paxos
>>> 0/ 5 tp
>>> 1/ 5 auth
>>> 1/ 5 crypto
>>> 1/ 1 finisher
>>> 1/ 5 heartbeatmap
>>> 1/ 5 perfcounter
>>> 1/ 5 rgw
>>> 1/10 civetweb
>>> 1/ 5 javaclient
>>> 1/ 5 asok
>>> 1/ 1 throttle
>>> 0/ 0 refs
>>> 1/ 5 xio
>>> 1/ 5 compressor
>>> 1/ 5 bluestore
>>> 1/ 5 bluefs
>>> 1/ 3 bdev
>>> 1/ 5 kstore
>>> 4/ 5 rocksdb
>>> 4/ 5 leveldb
>>> 4/ 5 memdb
>>> 1/ 5 kinetic
>>> 1/ 5 fuse
>>> 1/ 5 mgr
>>> 1/ 5 mgrc
>>> 1/ 5 dpdk
>>> 1/ 5 eventtrace
>>> -2/-2 (syslog threshold)
>>> -1/-1 (stderr threshold)
>>> max_recent 10000
>>> max_new 1000
>>> log_file /var/log/ceph/ceph-mds.cephfs1.log
>>> --- end dump of recent events ---
>>>
>>>
>>> On 15/06/17 08:10, Yan, Zheng wrote:
>>>> On Wed, Jun 14, 2017 at 11:49 PM, Jake Grimmett <jog@mrc-lmb.cam.ac.uk> wrote:
>>>>> Dear All,
>>>>>
>>>>> Sorry, but I need to add +1 to the mds crash reports with ceph
>>>>> 12.0.3-1507-g52f0deb
>>>>>
>>>>> This happened to me after updating from 12.0.2
>>>>> All was fairly OK for a few hours, I/O around 500MB/s, then both MDS
>>>>> servers crashed, and have not worked since.
>>>>>
>>>>> The two MDS servers, are active:standby, both now crash immediately
>>>>> after being started.
>>>>>
>>>>> This cluster has been upgraded from Kraken, through several Luminous
>>>>> versions, so I did a clean install of SL7.3 on one MDS server, and still
>>>>> have crashes on this machine.
>>>>>
>>>>> Cluster has 40 x 8TB drives (EC 4+1), with dual replicated NVME
>>>>> providing a hotpool to drive the Cephfs layer. df -h /cephfs is/was
>>>>> 200TB. All OSD's are bluestore, and were generated on Luminous.
>>>>>
>>>>> I enabled snapshots a few days ago, and keep 144 snapshots (one taken
>>>>> every 10 minutes, each is kept for 24 hours only) about 30TB is copied
>>>>> into the fs each day. If snapshots caused the crash, I can regenerate
>>>>> the data, but they are very useful.
>>>>>
>>>>> One MDS gave this log...
>>>>>
>>>>> <http://www.mrc-lmb.cam.ac.uk/jog/ceph-mds.cephfs1.log>
>>>> It is a snapshot related bug. The Attached patch should prevent mds
>>>> from crashing.
>>>> Next time you restart mds, please set debug_mds=10 and upload the log.
>>>>
>>>> Regards
>>>> Yan, Zheng
>>>>
>>>>> many thanks for any suggestions, and it's great to see the experimental
>>>>> flag removed from bluestore!
>>>>>
>>>>> Jake
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: ceph-mds crash v12.0.3
2017-06-16 7:04 ` Yan, Zheng
@ 2017-06-16 8:19 ` Jake Grimmett
2017-06-16 13:13 ` Jake Grimmett
0 siblings, 1 reply; 17+ messages in thread
From: Jake Grimmett @ 2017-06-16 8:19 UTC (permalink / raw)
To: Yan, Zheng, Jake Grimmett; +Cc: ceph-devel
Hi Yan,
Many thanks for getting back to me - sorry to cause you bother.
I think I'm patching OK, but can you please check my methodology?
git clone git://github.com/ceph/ceph ; cd ceph
git apply ceph-mds.patch ; ./make-srpm.sh
rpmbuild --rebuild /root/ceph/ceph/ceph-12.0.3-1661-g3ddbfcd.el7.src.rpm
here is the section of the patched src/mds/journal.cc
2194 // note which segments inodes belong to, so we don't have to
start rejournaling them
2195 for (const auto &ino : inos) {
2196 CInode *in = mds->mdcache->get_inode(ino);
2197 if (!in) {
2198 dout(0) << "EOpen.replay ino " << ino << " not in
metablob" << dendl;
2199 assert(in);
2200 }
2201 _segment->open_files.push_back(&in->item_open_file);
2202 }
2203 for (const auto &vino : snap_inos) {
2204 CInode *in = mds->mdcache->get_inode(vino);
2205 if (!in) {
2206 dout(0) << "EOpen.replay ino " << vino << " not in
metablob" << dendl;
2207 continue;
2208 }
many thanks for your time,
Jake
On 16/06/17 08:04, Yan, Zheng wrote:
> On Thu, Jun 15, 2017 at 7:32 PM, Jake Grimmett <jog@mrc-lmb.cam.ac.uk> wrote:
>> Hi Yan,
>>
>> Many thanks for looking into this and providing a patch.
>>
>> I've downloaded ceph 12.0.3-1661-g3ddbfcd, applied your patch, rebuilt
>> the rpms, and installed across my cluster.
>>
>> Unfortunately, the MDS are still crashing, any ideas welcome :)
>>
>> With "debug_mds = 10" the full Log is 140MB, a truncated version of the
>> log immediately preceding the crash follows:
>>
>> best,
>>
>> Jake
>>
>> -5> 2017-06-15 12:21:14.084373 7f77fe590700 10 mds.0.journal
>> EMetaBlob.replay added (full) [dentry
>> #1/isilon/sc/users/spc/JessComb_AB_230115/JessB_TO_190115_F6_1/n0/JessB_TO_190115_F6_1.peaks_int
>> [9f,head] auth NULL (dversion lock) v=3104 inode=0
>> state=1073741888|bottomlru 0x7f781a3f1860]
>> -4> 2017-06-15 12:21:14.084375 7f77fe590700 10 mds.0.journal
>> EMetaBlob.replay added [inode 1000147f773 [9f,head]
>> /isilon/sc/users/spc/JessComb_AB_230115/JessB_TO_190115_F6_1/n0/JessB_TO_190115_F6_1.peaks_int
>> auth v3104 s=4 n(v0 b4 1=1+0) (iversion lock) cr={3554272=0-4194304@9e}
>> 0x7f781a3f5800]
>> -3> 2017-06-15 12:21:14.084379 7f77fe590700 10 mds.0.journal
>> EMetaBlob.replay added (full) [dentry
>> #1/isilon/sc/users/spc/JessComb_AB_230115/JessB_TO_190115_F6_1/n0/JessB_TO_190115_F6_1.peaks_maxt
>> [9f,head] auth NULL (dversion lock) v=3132 inode=0
>> state=1073741888|bottomlru 0x7f781a3f1d40]
>> -2> 2017-06-15 12:21:14.084381 7f77fe590700 10 mds.0.journal
>> EMetaBlob.replay added [inode 1000147f775 [9f,head]
>> /isilon/sc/users/spc/JessComb_AB_230115/JessB_TO_190115_F6_1/n0/JessB_TO_190115_F6_1.peaks_maxt
>> auth v3132 s=4 n(v0 b4 1=1+0) (iversion lock) cr={3554272=0-4194304@9e}
>> 0x7f781a3f5e00]
>> -1> 2017-06-15 12:21:14.084406 7f77fe590700 0 mds.0.journal
>> EOpen.replay ino 1000147761b.9a not in metablob
>> 0> 2017-06-15 12:21:14.085348 7f77fe590700 -1
>> /root/rpmbuild/BUILD/ceph-12.0.3-1661-g3ddbfcd/src/mds/journal.cc: In
>> function 'virtual void EOpen::replay(MDSRank*)' thread 7f77fe590700 time
>> 2017-06-15 12:21:14.084409
>> /root/rpmbuild/BUILD/ceph-12.0.3-1661-g3ddbfcd/src/mds/journal.cc: 2207:
>> FAILED assert(in)
>>
> The assertion should be removed by my patch. Maybe you didn't cleanly
> apply the patch.
>
>
> Regards
> Yan, Zheng
>
>> ceph version 12.0.3-1661-g3ddbfcd
>> (3ddbfcd4357ab3a3c2f17f86f88dc83172d4ce0d) luminous (dev)
>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x110) [0x7f780d290500]
>> 2: (EOpen::replay(MDSRank*)+0x3e5) [0x7f780d2397b5]
>> 3: (MDLog::_replay_thread()+0x5f2) [0x7f780d1efd12]
>> 4: (MDLog::ReplayThread::entry()+0xd) [0x7f780cf9b6ad]
>> 5: (()+0x7dc5) [0x7f780adb4dc5]
>> 6: (clone()+0x6d) [0x7f7809e9476d]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>> needed to interpret this.
>>
>> --- logging levels ---
>> 0/ 5 none
>> 0/ 1 lockdep
>> 0/ 1 context
>> 1/ 1 crush
>> 10/10 mds
>> 1/ 5 mds_balancer
>> 1/ 5 mds_locker
>> 1/ 5 mds_log
>> 1/ 5 mds_log_expire
>> 1/ 5 mds_migrator
>> 0/ 1 buffer
>> 0/ 1 timer
>> 0/ 1 filer
>> 0/ 1 striper
>> 0/ 1 objecter
>> 0/ 5 rados
>> 0/ 5 rbd
>> 0/ 5 rbd_mirror
>> 0/ 5 rbd_replay
>> 0/ 5 journaler
>> 0/ 5 objectcacher
>> 0/ 5 client
>> 1/ 5 osd
>> 0/ 5 optracker
>> 0/ 5 objclass
>> 1/ 3 filestore
>> 1/ 3 journal
>> 0/ 5 ms
>> 1/ 5 mon
>> 0/10 monc
>> 1/ 5 paxos
>> 0/ 5 tp
>> 1/ 5 auth
>> 1/ 5 crypto
>> 1/ 1 finisher
>> 1/ 5 heartbeatmap
>> 1/ 5 perfcounter
>> 1/ 5 rgw
>> 1/10 civetweb
>> 1/ 5 javaclient
>> 1/ 5 asok
>> 1/ 1 throttle
>> 0/ 0 refs
>> 1/ 5 xio
>> 1/ 5 compressor
>> 1/ 5 bluestore
>> 1/ 5 bluefs
>> 1/ 3 bdev
>> 1/ 5 kstore
>> 4/ 5 rocksdb
>> 4/ 5 leveldb
>> 4/ 5 memdb
>> 1/ 5 kinetic
>> 1/ 5 fuse
>> 1/ 5 mgr
>> 1/ 5 mgrc
>> 1/ 5 dpdk
>> 1/ 5 eventtrace
>> -2/-2 (syslog threshold)
>> -1/-1 (stderr threshold)
>> max_recent 10000
>> max_new 1000
>> log_file /var/log/ceph/ceph-mds.cephfs1.log
>> --- end dump of recent events ---
>> 2017-06-15 12:21:14.101761 7f77fe590700 -1 *** Caught signal (Aborted) **
>> in thread 7f77fe590700 thread_name:md_log_replay
>>
>> ceph version 12.0.3-1661-g3ddbfcd
>> (3ddbfcd4357ab3a3c2f17f86f88dc83172d4ce0d) luminous (dev)
>> 1: (()+0x57d7ff) [0x7f780d2507ff]
>> 2: (()+0xf370) [0x7f780adbc370]
>> 3: (gsignal()+0x37) [0x7f7809dd21d7]
>> 4: (abort()+0x148) [0x7f7809dd38c8]
>> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x284) [0x7f780d290674]
>> 6: (EOpen::replay(MDSRank*)+0x3e5) [0x7f780d2397b5]
>> 7: (MDLog::_replay_thread()+0x5f2) [0x7f780d1efd12]
>> 8: (MDLog::ReplayThread::entry()+0xd) [0x7f780cf9b6ad]
>> 9: (()+0x7dc5) [0x7f780adb4dc5]
>> 10: (clone()+0x6d) [0x7f7809e9476d]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>> needed to interpret this.
>>
>> --- begin dump of recent events ---
>> 0> 2017-06-15 12:21:14.101761 7f77fe590700 -1 *** Caught signal
>> (Aborted) **
>> in thread 7f77fe590700 thread_name:md_log_replay
>>
>> ceph version 12.0.3-1661-g3ddbfcd
>> (3ddbfcd4357ab3a3c2f17f86f88dc83172d4ce0d) luminous (dev)
>> 1: (()+0x57d7ff) [0x7f780d2507ff]
>> 2: (()+0xf370) [0x7f780adbc370]
>> 3: (gsignal()+0x37) [0x7f7809dd21d7]
>> 4: (abort()+0x148) [0x7f7809dd38c8]
>> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x284) [0x7f780d290674]
>> 6: (EOpen::replay(MDSRank*)+0x3e5) [0x7f780d2397b5]
>> 7: (MDLog::_replay_thread()+0x5f2) [0x7f780d1efd12]
>> 8: (MDLog::ReplayThread::entry()+0xd) [0x7f780cf9b6ad]
>> 9: (()+0x7dc5) [0x7f780adb4dc5]
>> 10: (clone()+0x6d) [0x7f7809e9476d]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>> needed to interpret this.
>>
>> --- logging levels ---
>> 0/ 5 none
>> 0/ 1 lockdep
>> 0/ 1 context
>> 1/ 1 crush
>> 10/10 mds
>> 1/ 5 mds_balancer
>> 1/ 5 mds_locker
>> 1/ 5 mds_log
>> 1/ 5 mds_log_expire
>> 1/ 5 mds_migrator
>> 0/ 1 buffer
>> 0/ 1 timer
>> 0/ 1 filer
>> 0/ 1 striper
>> 0/ 1 objecter
>> 0/ 5 rados
>> 0/ 5 rbd
>> 0/ 5 rbd_mirror
>> 0/ 5 rbd_replay
>> 0/ 5 journaler
>> 0/ 5 objectcacher
>> 0/ 5 client
>> 1/ 5 osd
>> 0/ 5 optracker
>> 0/ 5 objclass
>> 1/ 3 filestore
>> 1/ 3 journal
>> 0/ 5 ms
>> 1/ 5 mon
>> 0/10 monc
>> 1/ 5 paxos
>> 0/ 5 tp
>> 1/ 5 auth
>> 1/ 5 crypto
>> 1/ 1 finisher
>> 1/ 5 heartbeatmap
>> 1/ 5 perfcounter
>> 1/ 5 rgw
>> 1/10 civetweb
>> 1/ 5 javaclient
>> 1/ 5 asok
>> 1/ 1 throttle
>> 0/ 0 refs
>> 1/ 5 xio
>> 1/ 5 compressor
>> 1/ 5 bluestore
>> 1/ 5 bluefs
>> 1/ 3 bdev
>> 1/ 5 kstore
>> 4/ 5 rocksdb
>> 4/ 5 leveldb
>> 4/ 5 memdb
>> 1/ 5 kinetic
>> 1/ 5 fuse
>> 1/ 5 mgr
>> 1/ 5 mgrc
>> 1/ 5 dpdk
>> 1/ 5 eventtrace
>> -2/-2 (syslog threshold)
>> -1/-1 (stderr threshold)
>> max_recent 10000
>> max_new 1000
>> log_file /var/log/ceph/ceph-mds.cephfs1.log
>> --- end dump of recent events ---
>>
>>
>> On 15/06/17 08:10, Yan, Zheng wrote:
>>> On Wed, Jun 14, 2017 at 11:49 PM, Jake Grimmett <jog@mrc-lmb.cam.ac.uk> wrote:
>>>> Dear All,
>>>>
>>>> Sorry, but I need to add +1 to the mds crash reports with ceph
>>>> 12.0.3-1507-g52f0deb
>>>>
>>>> This happened to me after updating from 12.0.2
>>>> All was fairly OK for a few hours, I/O around 500MB/s, then both MDS
>>>> servers crashed, and have not worked since.
>>>>
>>>> The two MDS servers, are active:standby, both now crash immediately
>>>> after being started.
>>>>
>>>> This cluster has been upgraded from Kraken, through several Luminous
>>>> versions, so I did a clean install of SL7.3 on one MDS server, and still
>>>> have crashes on this machine.
>>>>
>>>> Cluster has 40 x 8TB drives (EC 4+1), with dual replicated NVME
>>>> providing a hotpool to drive the Cephfs layer. df -h /cephfs is/was
>>>> 200TB. All OSD's are bluestore, and were generated on Luminous.
>>>>
>>>> I enabled snapshots a few days ago, and keep 144 snapshots (one taken
>>>> every 10 minutes, each is kept for 24 hours only) about 30TB is copied
>>>> into the fs each day. If snapshots caused the crash, I can regenerate
>>>> the data, but they are very useful.
>>>>
>>>> One MDS gave this log...
>>>>
>>>> <http://www.mrc-lmb.cam.ac.uk/jog/ceph-mds.cephfs1.log>
>>> It is a snapshot related bug. The Attached patch should prevent mds
>>> from crashing.
>>> Next time you restart mds, please set debug_mds=10 and upload the log.
>>>
>>> Regards
>>> Yan, Zheng
>>>
>>>> many thanks for any suggestions, and it's great to see the experimental
>>>> flag removed from bluestore!
>>>>
>>>> Jake
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: ceph-mds crash v12.0.3
2017-06-15 11:32 ` Jake Grimmett
@ 2017-06-16 7:04 ` Yan, Zheng
2017-06-16 8:19 ` Jake Grimmett
0 siblings, 1 reply; 17+ messages in thread
From: Yan, Zheng @ 2017-06-16 7:04 UTC (permalink / raw)
To: Jake Grimmett; +Cc: ceph-devel, Jake Grimmett
On Thu, Jun 15, 2017 at 7:32 PM, Jake Grimmett <jog@mrc-lmb.cam.ac.uk> wrote:
> Hi Yan,
>
> Many thanks for looking into this and providing a patch.
>
> I've downloaded ceph 12.0.3-1661-g3ddbfcd, applied your patch, rebuilt
> the rpms, and installed across my cluster.
>
> Unfortunately, the MDS are still crashing, any ideas welcome :)
>
> With "debug_mds = 10" the full Log is 140MB, a truncated version of the
> log immediately preceding the crash follows:
>
> best,
>
> Jake
>
> -5> 2017-06-15 12:21:14.084373 7f77fe590700 10 mds.0.journal
> EMetaBlob.replay added (full) [dentry
> #1/isilon/sc/users/spc/JessComb_AB_230115/JessB_TO_190115_F6_1/n0/JessB_TO_190115_F6_1.peaks_int
> [9f,head] auth NULL (dversion lock) v=3104 inode=0
> state=1073741888|bottomlru 0x7f781a3f1860]
> -4> 2017-06-15 12:21:14.084375 7f77fe590700 10 mds.0.journal
> EMetaBlob.replay added [inode 1000147f773 [9f,head]
> /isilon/sc/users/spc/JessComb_AB_230115/JessB_TO_190115_F6_1/n0/JessB_TO_190115_F6_1.peaks_int
> auth v3104 s=4 n(v0 b4 1=1+0) (iversion lock) cr={3554272=0-4194304@9e}
> 0x7f781a3f5800]
> -3> 2017-06-15 12:21:14.084379 7f77fe590700 10 mds.0.journal
> EMetaBlob.replay added (full) [dentry
> #1/isilon/sc/users/spc/JessComb_AB_230115/JessB_TO_190115_F6_1/n0/JessB_TO_190115_F6_1.peaks_maxt
> [9f,head] auth NULL (dversion lock) v=3132 inode=0
> state=1073741888|bottomlru 0x7f781a3f1d40]
> -2> 2017-06-15 12:21:14.084381 7f77fe590700 10 mds.0.journal
> EMetaBlob.replay added [inode 1000147f775 [9f,head]
> /isilon/sc/users/spc/JessComb_AB_230115/JessB_TO_190115_F6_1/n0/JessB_TO_190115_F6_1.peaks_maxt
> auth v3132 s=4 n(v0 b4 1=1+0) (iversion lock) cr={3554272=0-4194304@9e}
> 0x7f781a3f5e00]
> -1> 2017-06-15 12:21:14.084406 7f77fe590700 0 mds.0.journal
> EOpen.replay ino 1000147761b.9a not in metablob
> 0> 2017-06-15 12:21:14.085348 7f77fe590700 -1
> /root/rpmbuild/BUILD/ceph-12.0.3-1661-g3ddbfcd/src/mds/journal.cc: In
> function 'virtual void EOpen::replay(MDSRank*)' thread 7f77fe590700 time
> 2017-06-15 12:21:14.084409
> /root/rpmbuild/BUILD/ceph-12.0.3-1661-g3ddbfcd/src/mds/journal.cc: 2207:
> FAILED assert(in)
>
The assertion should be removed by my patch. Maybe you didn't cleanly
apply the patch.
Regards
Yan, Zheng
> ceph version 12.0.3-1661-g3ddbfcd
> (3ddbfcd4357ab3a3c2f17f86f88dc83172d4ce0d) luminous (dev)
> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x110) [0x7f780d290500]
> 2: (EOpen::replay(MDSRank*)+0x3e5) [0x7f780d2397b5]
> 3: (MDLog::_replay_thread()+0x5f2) [0x7f780d1efd12]
> 4: (MDLog::ReplayThread::entry()+0xd) [0x7f780cf9b6ad]
> 5: (()+0x7dc5) [0x7f780adb4dc5]
> 6: (clone()+0x6d) [0x7f7809e9476d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> --- logging levels ---
> 0/ 5 none
> 0/ 1 lockdep
> 0/ 1 context
> 1/ 1 crush
> 10/10 mds
> 1/ 5 mds_balancer
> 1/ 5 mds_locker
> 1/ 5 mds_log
> 1/ 5 mds_log_expire
> 1/ 5 mds_migrator
> 0/ 1 buffer
> 0/ 1 timer
> 0/ 1 filer
> 0/ 1 striper
> 0/ 1 objecter
> 0/ 5 rados
> 0/ 5 rbd
> 0/ 5 rbd_mirror
> 0/ 5 rbd_replay
> 0/ 5 journaler
> 0/ 5 objectcacher
> 0/ 5 client
> 1/ 5 osd
> 0/ 5 optracker
> 0/ 5 objclass
> 1/ 3 filestore
> 1/ 3 journal
> 0/ 5 ms
> 1/ 5 mon
> 0/10 monc
> 1/ 5 paxos
> 0/ 5 tp
> 1/ 5 auth
> 1/ 5 crypto
> 1/ 1 finisher
> 1/ 5 heartbeatmap
> 1/ 5 perfcounter
> 1/ 5 rgw
> 1/10 civetweb
> 1/ 5 javaclient
> 1/ 5 asok
> 1/ 1 throttle
> 0/ 0 refs
> 1/ 5 xio
> 1/ 5 compressor
> 1/ 5 bluestore
> 1/ 5 bluefs
> 1/ 3 bdev
> 1/ 5 kstore
> 4/ 5 rocksdb
> 4/ 5 leveldb
> 4/ 5 memdb
> 1/ 5 kinetic
> 1/ 5 fuse
> 1/ 5 mgr
> 1/ 5 mgrc
> 1/ 5 dpdk
> 1/ 5 eventtrace
> -2/-2 (syslog threshold)
> -1/-1 (stderr threshold)
> max_recent 10000
> max_new 1000
> log_file /var/log/ceph/ceph-mds.cephfs1.log
> --- end dump of recent events ---
> 2017-06-15 12:21:14.101761 7f77fe590700 -1 *** Caught signal (Aborted) **
> in thread 7f77fe590700 thread_name:md_log_replay
>
> ceph version 12.0.3-1661-g3ddbfcd
> (3ddbfcd4357ab3a3c2f17f86f88dc83172d4ce0d) luminous (dev)
> 1: (()+0x57d7ff) [0x7f780d2507ff]
> 2: (()+0xf370) [0x7f780adbc370]
> 3: (gsignal()+0x37) [0x7f7809dd21d7]
> 4: (abort()+0x148) [0x7f7809dd38c8]
> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x284) [0x7f780d290674]
> 6: (EOpen::replay(MDSRank*)+0x3e5) [0x7f780d2397b5]
> 7: (MDLog::_replay_thread()+0x5f2) [0x7f780d1efd12]
> 8: (MDLog::ReplayThread::entry()+0xd) [0x7f780cf9b6ad]
> 9: (()+0x7dc5) [0x7f780adb4dc5]
> 10: (clone()+0x6d) [0x7f7809e9476d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> --- begin dump of recent events ---
> 0> 2017-06-15 12:21:14.101761 7f77fe590700 -1 *** Caught signal
> (Aborted) **
> in thread 7f77fe590700 thread_name:md_log_replay
>
> ceph version 12.0.3-1661-g3ddbfcd
> (3ddbfcd4357ab3a3c2f17f86f88dc83172d4ce0d) luminous (dev)
> 1: (()+0x57d7ff) [0x7f780d2507ff]
> 2: (()+0xf370) [0x7f780adbc370]
> 3: (gsignal()+0x37) [0x7f7809dd21d7]
> 4: (abort()+0x148) [0x7f7809dd38c8]
> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x284) [0x7f780d290674]
> 6: (EOpen::replay(MDSRank*)+0x3e5) [0x7f780d2397b5]
> 7: (MDLog::_replay_thread()+0x5f2) [0x7f780d1efd12]
> 8: (MDLog::ReplayThread::entry()+0xd) [0x7f780cf9b6ad]
> 9: (()+0x7dc5) [0x7f780adb4dc5]
> 10: (clone()+0x6d) [0x7f7809e9476d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> --- logging levels ---
> 0/ 5 none
> 0/ 1 lockdep
> 0/ 1 context
> 1/ 1 crush
> 10/10 mds
> 1/ 5 mds_balancer
> 1/ 5 mds_locker
> 1/ 5 mds_log
> 1/ 5 mds_log_expire
> 1/ 5 mds_migrator
> 0/ 1 buffer
> 0/ 1 timer
> 0/ 1 filer
> 0/ 1 striper
> 0/ 1 objecter
> 0/ 5 rados
> 0/ 5 rbd
> 0/ 5 rbd_mirror
> 0/ 5 rbd_replay
> 0/ 5 journaler
> 0/ 5 objectcacher
> 0/ 5 client
> 1/ 5 osd
> 0/ 5 optracker
> 0/ 5 objclass
> 1/ 3 filestore
> 1/ 3 journal
> 0/ 5 ms
> 1/ 5 mon
> 0/10 monc
> 1/ 5 paxos
> 0/ 5 tp
> 1/ 5 auth
> 1/ 5 crypto
> 1/ 1 finisher
> 1/ 5 heartbeatmap
> 1/ 5 perfcounter
> 1/ 5 rgw
> 1/10 civetweb
> 1/ 5 javaclient
> 1/ 5 asok
> 1/ 1 throttle
> 0/ 0 refs
> 1/ 5 xio
> 1/ 5 compressor
> 1/ 5 bluestore
> 1/ 5 bluefs
> 1/ 3 bdev
> 1/ 5 kstore
> 4/ 5 rocksdb
> 4/ 5 leveldb
> 4/ 5 memdb
> 1/ 5 kinetic
> 1/ 5 fuse
> 1/ 5 mgr
> 1/ 5 mgrc
> 1/ 5 dpdk
> 1/ 5 eventtrace
> -2/-2 (syslog threshold)
> -1/-1 (stderr threshold)
> max_recent 10000
> max_new 1000
> log_file /var/log/ceph/ceph-mds.cephfs1.log
> --- end dump of recent events ---
>
>
> On 15/06/17 08:10, Yan, Zheng wrote:
>> On Wed, Jun 14, 2017 at 11:49 PM, Jake Grimmett <jog@mrc-lmb.cam.ac.uk> wrote:
>>> Dear All,
>>>
>>> Sorry, but I need to add +1 to the mds crash reports with ceph
>>> 12.0.3-1507-g52f0deb
>>>
>>> This happened to me after updating from 12.0.2
>>> All was fairly OK for a few hours, I/O around 500MB/s, then both MDS
>>> servers crashed, and have not worked since.
>>>
>>> The two MDS servers, are active:standby, both now crash immediately
>>> after being started.
>>>
>>> This cluster has been upgraded from Kraken, through several Luminous
>>> versions, so I did a clean install of SL7.3 on one MDS server, and still
>>> have crashes on this machine.
>>>
>>> Cluster has 40 x 8TB drives (EC 4+1), with dual replicated NVME
>>> providing a hotpool to drive the Cephfs layer. df -h /cephfs is/was
>>> 200TB. All OSD's are bluestore, and were generated on Luminous.
>>>
>>> I enabled snapshots a few days ago, and keep 144 snapshots (one taken
>>> every 10 minutes, each is kept for 24 hours only) about 30TB is copied
>>> into the fs each day. If snapshots caused the crash, I can regenerate
>>> the data, but they are very useful.
>>>
>>> One MDS gave this log...
>>>
>>> <http://www.mrc-lmb.cam.ac.uk/jog/ceph-mds.cephfs1.log>
>>
>> It is a snapshot related bug. The Attached patch should prevent mds
>> from crashing.
>> Next time you restart mds, please set debug_mds=10 and upload the log.
>>
>> Regards
>> Yan, Zheng
>>
>>>
>>> many thanks for any suggestions, and it's great to see the experimental
>>> flag removed from bluestore!
>>>
>>> Jake
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
> --
> Dr Jake Grimmett
> Head Of Scientific Computing
> MRC Laboratory of Molecular Biology
> Francis Crick Avenue,
> Cambridge CB2 0QH, UK.
> Phone 01223 267019
> Mobile 0776 9886539
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: ceph-mds crash v12.0.3
2017-06-15 7:10 ` Yan, Zheng
@ 2017-06-15 11:32 ` Jake Grimmett
2017-06-16 7:04 ` Yan, Zheng
0 siblings, 1 reply; 17+ messages in thread
From: Jake Grimmett @ 2017-06-15 11:32 UTC (permalink / raw)
To: Yan, Zheng; +Cc: ceph-devel, Jake Grimmett
Hi Yan,
Many thanks for looking into this and providing a patch.
I've downloaded ceph 12.0.3-1661-g3ddbfcd, applied your patch, rebuilt
the rpms, and installed across my cluster.
Unfortunately, the MDS are still crashing, any ideas welcome :)
With "debug_mds = 10" the full Log is 140MB, a truncated version of the
log immediately preceding the crash follows:
best,
Jake
-5> 2017-06-15 12:21:14.084373 7f77fe590700 10 mds.0.journal
EMetaBlob.replay added (full) [dentry
#1/isilon/sc/users/spc/JessComb_AB_230115/JessB_TO_190115_F6_1/n0/JessB_TO_190115_F6_1.peaks_int
[9f,head] auth NULL (dversion lock) v=3104 inode=0
state=1073741888|bottomlru 0x7f781a3f1860]
-4> 2017-06-15 12:21:14.084375 7f77fe590700 10 mds.0.journal
EMetaBlob.replay added [inode 1000147f773 [9f,head]
/isilon/sc/users/spc/JessComb_AB_230115/JessB_TO_190115_F6_1/n0/JessB_TO_190115_F6_1.peaks_int
auth v3104 s=4 n(v0 b4 1=1+0) (iversion lock) cr={3554272=0-4194304@9e}
0x7f781a3f5800]
-3> 2017-06-15 12:21:14.084379 7f77fe590700 10 mds.0.journal
EMetaBlob.replay added (full) [dentry
#1/isilon/sc/users/spc/JessComb_AB_230115/JessB_TO_190115_F6_1/n0/JessB_TO_190115_F6_1.peaks_maxt
[9f,head] auth NULL (dversion lock) v=3132 inode=0
state=1073741888|bottomlru 0x7f781a3f1d40]
-2> 2017-06-15 12:21:14.084381 7f77fe590700 10 mds.0.journal
EMetaBlob.replay added [inode 1000147f775 [9f,head]
/isilon/sc/users/spc/JessComb_AB_230115/JessB_TO_190115_F6_1/n0/JessB_TO_190115_F6_1.peaks_maxt
auth v3132 s=4 n(v0 b4 1=1+0) (iversion lock) cr={3554272=0-4194304@9e}
0x7f781a3f5e00]
-1> 2017-06-15 12:21:14.084406 7f77fe590700 0 mds.0.journal
EOpen.replay ino 1000147761b.9a not in metablob
0> 2017-06-15 12:21:14.085348 7f77fe590700 -1
/root/rpmbuild/BUILD/ceph-12.0.3-1661-g3ddbfcd/src/mds/journal.cc: In
function 'virtual void EOpen::replay(MDSRank*)' thread 7f77fe590700 time
2017-06-15 12:21:14.084409
/root/rpmbuild/BUILD/ceph-12.0.3-1661-g3ddbfcd/src/mds/journal.cc: 2207:
FAILED assert(in)
ceph version 12.0.3-1661-g3ddbfcd
(3ddbfcd4357ab3a3c2f17f86f88dc83172d4ce0d) luminous (dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x110) [0x7f780d290500]
2: (EOpen::replay(MDSRank*)+0x3e5) [0x7f780d2397b5]
3: (MDLog::_replay_thread()+0x5f2) [0x7f780d1efd12]
4: (MDLog::ReplayThread::entry()+0xd) [0x7f780cf9b6ad]
5: (()+0x7dc5) [0x7f780adb4dc5]
6: (clone()+0x6d) [0x7f7809e9476d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
10/10 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mds.cephfs1.log
--- end dump of recent events ---
2017-06-15 12:21:14.101761 7f77fe590700 -1 *** Caught signal (Aborted) **
in thread 7f77fe590700 thread_name:md_log_replay
ceph version 12.0.3-1661-g3ddbfcd
(3ddbfcd4357ab3a3c2f17f86f88dc83172d4ce0d) luminous (dev)
1: (()+0x57d7ff) [0x7f780d2507ff]
2: (()+0xf370) [0x7f780adbc370]
3: (gsignal()+0x37) [0x7f7809dd21d7]
4: (abort()+0x148) [0x7f7809dd38c8]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x284) [0x7f780d290674]
6: (EOpen::replay(MDSRank*)+0x3e5) [0x7f780d2397b5]
7: (MDLog::_replay_thread()+0x5f2) [0x7f780d1efd12]
8: (MDLog::ReplayThread::entry()+0xd) [0x7f780cf9b6ad]
9: (()+0x7dc5) [0x7f780adb4dc5]
10: (clone()+0x6d) [0x7f7809e9476d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- begin dump of recent events ---
0> 2017-06-15 12:21:14.101761 7f77fe590700 -1 *** Caught signal
(Aborted) **
in thread 7f77fe590700 thread_name:md_log_replay
ceph version 12.0.3-1661-g3ddbfcd
(3ddbfcd4357ab3a3c2f17f86f88dc83172d4ce0d) luminous (dev)
1: (()+0x57d7ff) [0x7f780d2507ff]
2: (()+0xf370) [0x7f780adbc370]
3: (gsignal()+0x37) [0x7f7809dd21d7]
4: (abort()+0x148) [0x7f7809dd38c8]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x284) [0x7f780d290674]
6: (EOpen::replay(MDSRank*)+0x3e5) [0x7f780d2397b5]
7: (MDLog::_replay_thread()+0x5f2) [0x7f780d1efd12]
8: (MDLog::ReplayThread::entry()+0xd) [0x7f780cf9b6ad]
9: (()+0x7dc5) [0x7f780adb4dc5]
10: (clone()+0x6d) [0x7f7809e9476d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
10/10 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mds.cephfs1.log
--- end dump of recent events ---
On 15/06/17 08:10, Yan, Zheng wrote:
> On Wed, Jun 14, 2017 at 11:49 PM, Jake Grimmett <jog@mrc-lmb.cam.ac.uk> wrote:
>> Dear All,
>>
>> Sorry, but I need to add +1 to the mds crash reports with ceph
>> 12.0.3-1507-g52f0deb
>>
>> This happened to me after updating from 12.0.2
>> All was fairly OK for a few hours, I/O around 500MB/s, then both MDS
>> servers crashed, and have not worked since.
>>
>> The two MDS servers, are active:standby, both now crash immediately
>> after being started.
>>
>> This cluster has been upgraded from Kraken, through several Luminous
>> versions, so I did a clean install of SL7.3 on one MDS server, and still
>> have crashes on this machine.
>>
>> Cluster has 40 x 8TB drives (EC 4+1), with dual replicated NVME
>> providing a hotpool to drive the Cephfs layer. df -h /cephfs is/was
>> 200TB. All OSD's are bluestore, and were generated on Luminous.
>>
>> I enabled snapshots a few days ago, and keep 144 snapshots (one taken
>> every 10 minutes, each is kept for 24 hours only) about 30TB is copied
>> into the fs each day. If snapshots caused the crash, I can regenerate
>> the data, but they are very useful.
>>
>> One MDS gave this log...
>>
>> <http://www.mrc-lmb.cam.ac.uk/jog/ceph-mds.cephfs1.log>
>
> It is a snapshot related bug. The Attached patch should prevent mds
> from crashing.
> Next time you restart mds, please set debug_mds=10 and upload the log.
>
> Regards
> Yan, Zheng
>
>>
>> many thanks for any suggestions, and it's great to see the experimental
>> flag removed from bluestore!
>>
>> Jake
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Dr Jake Grimmett
Head Of Scientific Computing
MRC Laboratory of Molecular Biology
Francis Crick Avenue,
Cambridge CB2 0QH, UK.
Phone 01223 267019
Mobile 0776 9886539
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: ceph-mds crash v12.0.3
2017-06-14 15:49 Jake Grimmett
2017-06-14 16:38 ` John Spray
@ 2017-06-15 7:10 ` Yan, Zheng
2017-06-15 11:32 ` Jake Grimmett
1 sibling, 1 reply; 17+ messages in thread
From: Yan, Zheng @ 2017-06-15 7:10 UTC (permalink / raw)
To: Jake Grimmett; +Cc: ceph-devel, Jake Grimmett
[-- Attachment #1: Type: text/plain, Size: 1669 bytes --]
On Wed, Jun 14, 2017 at 11:49 PM, Jake Grimmett <jog@mrc-lmb.cam.ac.uk> wrote:
> Dear All,
>
> Sorry, but I need to add +1 to the mds crash reports with ceph
> 12.0.3-1507-g52f0deb
>
> This happened to me after updating from 12.0.2
> All was fairly OK for a few hours, I/O around 500MB/s, then both MDS
> servers crashed, and have not worked since.
>
> The two MDS servers, are active:standby, both now crash immediately
> after being started.
>
> This cluster has been upgraded from Kraken, through several Luminous
> versions, so I did a clean install of SL7.3 on one MDS server, and still
> have crashes on this machine.
>
> Cluster has 40 x 8TB drives (EC 4+1), with dual replicated NVME
> providing a hotpool to drive the Cephfs layer. df -h /cephfs is/was
> 200TB. All OSD's are bluestore, and were generated on Luminous.
>
> I enabled snapshots a few days ago, and keep 144 snapshots (one taken
> every 10 minutes, each is kept for 24 hours only) about 30TB is copied
> into the fs each day. If snapshots caused the crash, I can regenerate
> the data, but they are very useful.
>
> One MDS gave this log...
>
> <http://www.mrc-lmb.cam.ac.uk/jog/ceph-mds.cephfs1.log>
It is a snapshot related bug. The Attached patch should prevent mds
from crashing.
Next time you restart mds, please set debug_mds=10 and upload the log.
Regards
Yan, Zheng
>
> many thanks for any suggestions, and it's great to see the experimental
> flag removed from bluestore!
>
> Jake
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: ceph-mds.patch --]
[-- Type: application/octet-stream, Size: 437 bytes --]
diff --git a/src/mds/journal.cc b/src/mds/journal.cc
index f7570c6..1754608 100644
--- a/src/mds/journal.cc
+++ b/src/mds/journal.cc
@@ -2204,7 +2204,7 @@ void EOpen::replay(MDSRank *mds)
CInode *in = mds->mdcache->get_inode(vino);
if (!in) {
dout(0) << "EOpen.replay ino " << vino << " not in metablob" << dendl;
- assert(in);
+ continue;
}
_segment->open_files.push_back(&in->item_open_file);
}
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: ceph-mds crash v12.0.3
2017-06-14 15:49 Jake Grimmett
@ 2017-06-14 16:38 ` John Spray
2017-06-15 7:10 ` Yan, Zheng
1 sibling, 0 replies; 17+ messages in thread
From: John Spray @ 2017-06-14 16:38 UTC (permalink / raw)
To: Jake Grimmett; +Cc: Ceph Development, Jake Grimmett
On Wed, Jun 14, 2017 at 11:49 AM, Jake Grimmett <jog@mrc-lmb.cam.ac.uk> wrote:
> Dear All,
>
> Sorry, but I need to add +1 to the mds crash reports with ceph
> 12.0.3-1507-g52f0deb
>
> This happened to me after updating from 12.0.2
> All was fairly OK for a few hours, I/O around 500MB/s, then both MDS
> servers crashed, and have not worked since.
>
> The two MDS servers, are active:standby, both now crash immediately
> after being started.
>
> This cluster has been upgraded from Kraken, through several Luminous
> versions, so I did a clean install of SL7.3 on one MDS server, and still
> have crashes on this machine.
>
> Cluster has 40 x 8TB drives (EC 4+1), with dual replicated NVME
> providing a hotpool to drive the Cephfs layer. df -h /cephfs is/was
> 200TB. All OSD's are bluestore, and were generated on Luminous.
>
> I enabled snapshots a few days ago, and keep 144 snapshots (one taken
> every 10 minutes, each is kept for 24 hours only) about 30TB is copied
> into the fs each day. If snapshots caused the crash, I can regenerate
> the data, but they are very useful.
>
> One MDS gave this log...
>
> <http://www.mrc-lmb.cam.ac.uk/jog/ceph-mds.cephfs1.log>
I'm getting a Forbidden trying to load that.
John
> many thanks for any suggestions, and it's great to see the experimental
> flag removed from bluestore!
>
> Jake
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: ceph-mds crash v12.0.3
@ 2017-06-14 15:49 Jake Grimmett
2017-06-14 16:38 ` John Spray
2017-06-15 7:10 ` Yan, Zheng
0 siblings, 2 replies; 17+ messages in thread
From: Jake Grimmett @ 2017-06-14 15:49 UTC (permalink / raw)
To: ceph-devel, Jake Grimmett
Dear All,
Sorry, but I need to add +1 to the mds crash reports with ceph
12.0.3-1507-g52f0deb
This happened to me after updating from 12.0.2
All was fairly OK for a few hours, I/O around 500MB/s, then both MDS
servers crashed, and have not worked since.
The two MDS servers, are active:standby, both now crash immediately
after being started.
This cluster has been upgraded from Kraken, through several Luminous
versions, so I did a clean install of SL7.3 on one MDS server, and still
have crashes on this machine.
Cluster has 40 x 8TB drives (EC 4+1), with dual replicated NVME
providing a hotpool to drive the Cephfs layer. df -h /cephfs is/was
200TB. All OSD's are bluestore, and were generated on Luminous.
I enabled snapshots a few days ago, and keep 144 snapshots (one taken
every 10 minutes, each is kept for 24 hours only) about 30TB is copied
into the fs each day. If snapshots caused the crash, I can regenerate
the data, but they are very useful.
One MDS gave this log...
<http://www.mrc-lmb.cam.ac.uk/jog/ceph-mds.cephfs1.log>
many thanks for any suggestions, and it's great to see the experimental
flag removed from bluestore!
Jake
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2017-06-16 13:14 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-06-12 9:13 ceph-mds crash v12.0.3 Georgi Chorbadzhiyski
2017-06-12 10:22 ` John Spray
2017-06-12 10:38 ` Georgi Chorbadzhiyski
2017-06-12 10:56 ` Georgi Chorbadzhiyski
2017-06-12 12:16 ` Georgi Chorbadzhiyski
2017-06-12 12:58 ` Yan, Zheng
2017-06-12 14:45 ` Georgi Chorbadzhiyski
2017-06-13 11:46 ` Yan, Zheng
2017-06-14 14:40 ` Georgi Chorbadzhiyski
2017-06-15 8:27 ` Yan, Zheng
2017-06-14 15:49 Jake Grimmett
2017-06-14 16:38 ` John Spray
2017-06-15 7:10 ` Yan, Zheng
2017-06-15 11:32 ` Jake Grimmett
2017-06-16 7:04 ` Yan, Zheng
2017-06-16 8:19 ` Jake Grimmett
2017-06-16 13:13 ` Jake Grimmett
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.