[ceph-mgr] Signal Raised in tcmalloc

[ceph-mgr] Signal Raised in tcmalloc

[zengran zhang]

Hi, Kefu:

   we found a mgr instance hang, and attach it with gdb. we can see a
signal raised(#8) in or after
`DaemonPerfCounters::update(MMgrReport*)` , so its because the
tcmalloc heap corrupt. can you help to disgnose the problem? ceph
version is 12.0.2. thanks so much.

#0  0x00007f2245fc8026 in SpinLock::SlowLock() () from /usr/lib/libtcmalloc.so.4
#1  0x00007f2245fb907a in
tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from
/usr/lib/libtcmalloc.so.4
#2  0x00007f2245fbbfaa in
tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned
long) () from /usr/lib/libtcmalloc.so.4
#3  0x00007f2245fce368 in tc_newarray () from /usr/lib/libtcmalloc.so.4
#4  0x000055840e8f670d in ?? ()
#5  0x000055840e8f6a9c in pretty_version_to_str[abi:cxx11]() ()
#6  0x000055840e7122b6 in ceph::BackTrace::print(std::ostream&) const ()
#7  0x000055840e6f8a27 in ?? ()
#8  <signal handler called>
#9  0x00007f2245fb8c33 in
tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) ()
from /usr/lib/libtcmalloc.so.4
#10 0x00007f2245fb8f0a in
tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)
() from /usr/lib/libtcmalloc.so.4
#11 0x00007f2245fb8fbf in
tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from
/usr/lib/libtcmalloc.so.4
#12 0x00007f2245fbbfaa in
tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned
long) () from /usr/lib/libtcmalloc.so.4
#13 0x00007f2245fce368 in tc_newarray () from /usr/lib/libtcmalloc.so.4
#14 0x000055840e67036d in ?? ()
#15 0x000055840e671e46 in DaemonPerfCounters::update(MMgrReport*) ()
#16 0x000055840e6770e4 in DaemonServer::handle_report(MMgrReport*) ()
#17 0x000055840e67cb70 in DaemonServer::ms_dispatch(Message*) ()
#18 0x000055840e970f6a in DispatchQueue::entry() ()
#19 0x000055840e809d7d in DispatchQueue::DispatchThread::entry() ()
#20 0x00007f22453d96ba in start_thread (arg=0x7f223b6dc700) at
pthread_create.c:333
#21 0x00007f224444582d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Re: [ceph-mgr] Signal Raised in tcmalloc

[Sage Weil]

On Wed, 21 Jun 2017, zengran zhang wrote:
> Hi, Kefu:
> 
>    we found a mgr instance hang, and attach it with gdb. we can see a
> signal raised(#8) in or after
> `DaemonPerfCounters::update(MMgrReport*)` , so its because the
> tcmalloc heap corrupt. can you help to disgnose the problem? ceph
> version is 12.0.2. thanks so much.

This is likely to be fixed on the latest master.  We were seeing lots of 
crashes when the mgr went from active to inactive; now it is quite 
stable.

sage


> 
> #0  0x00007f2245fc8026 in SpinLock::SlowLock() () from /usr/lib/libtcmalloc.so.4
> #1  0x00007f2245fb907a in
> tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from
> /usr/lib/libtcmalloc.so.4
> #2  0x00007f2245fbbfaa in
> tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned
> long) () from /usr/lib/libtcmalloc.so.4
> #3  0x00007f2245fce368 in tc_newarray () from /usr/lib/libtcmalloc.so.4
> #4  0x000055840e8f670d in ?? ()
> #5  0x000055840e8f6a9c in pretty_version_to_str[abi:cxx11]() ()
> #6  0x000055840e7122b6 in ceph::BackTrace::print(std::ostream&) const ()
> #7  0x000055840e6f8a27 in ?? ()
> #8  <signal handler called>
> #9  0x00007f2245fb8c33 in
> tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) ()
> from /usr/lib/libtcmalloc.so.4
> #10 0x00007f2245fb8f0a in
> tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)
> () from /usr/lib/libtcmalloc.so.4
> #11 0x00007f2245fb8fbf in
> tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from
> /usr/lib/libtcmalloc.so.4
> #12 0x00007f2245fbbfaa in
> tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned
> long) () from /usr/lib/libtcmalloc.so.4
> #13 0x00007f2245fce368 in tc_newarray () from /usr/lib/libtcmalloc.so.4
> #14 0x000055840e67036d in ?? ()
> #15 0x000055840e671e46 in DaemonPerfCounters::update(MMgrReport*) ()
> #16 0x000055840e6770e4 in DaemonServer::handle_report(MMgrReport*) ()
> #17 0x000055840e67cb70 in DaemonServer::ms_dispatch(Message*) ()
> #18 0x000055840e970f6a in DispatchQueue::entry() ()
> #19 0x000055840e809d7d in DispatchQueue::DispatchThread::entry() ()
> #20 0x00007f22453d96ba in start_thread (arg=0x7f223b6dc700) at
> pthread_create.c:333
> #21 0x00007f224444582d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
>