From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ning Yao Subject: Re: fio multithreads meets Segmentation fault Date: Tue, 28 Mar 2017 21:03:07 +0800 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from mail-it0-f49.google.com ([209.85.214.49]:36909 "EHLO mail-it0-f49.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755380AbdC1NDX (ORCPT ); Tue, 28 Mar 2017 09:03:23 -0400 Received: by mail-it0-f49.google.com with SMTP id 190so17399951itm.0 for ; Tue, 28 Mar 2017 06:03:08 -0700 (PDT) In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: =?UTF-8?B?6Zer5Yib?= , Haomai Wang Cc: ceph-devel Hi, Haomai AsyncMessenger uses threadpools to handle connection and some thread unsafe function such as PK11_CreateContextBySymKey in libnss causes the corruptions here? Regards Ning Yao 2017-03-28 17:36 GMT+08:00 =E9=97=AB=E5=88=9B : > Hi all, > > When I make a fio-rbd-test, it says segmentation fault. I reckon that > there may be something wrong with the usage of libnss. > Ceph version is 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7) > using AsyncMessenger. > Is libnss3.so thread safe? or other thoughts to resolve this problem? > > Here is the output of #gdb /usr/bin/fio corefile > > ...... > warning: the debug information found in > "/usr/lib/debug/usr/lib64//librados.so.2.0.0.debug" does not match > "/lib64/librados.so.2" (CRC mismatch). > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > Core was generated by `fio rbd.fio --output 20G_rbd.log'. > Program terminated with signal 11, Segmentation fault. > #0 0x00007f799b7ebdbc in _int_malloc () from /lib64/libc.so.6 > Missing separate debuginfos, use: debuginfo-install fio-2.2.8-2.el7.x86_6= 4 > (gdb) bt > #0 0x00007f799b7ebdbc in _int_malloc () from /lib64/libc.so.6 > #1 0x00007f799b7ee26c in malloc () from /lib64/libc.so.6 > #2 0x00007f799a75a440 in PORT_Alloc_Util () from /lib64/libnssutil3.so > #3 0x00007f799a758d32 in SECITEM_ArenaDupItem_Util () from > /lib64/libnssutil3.so > #4 0x00007f799a9a4238 in pk11_CreateNewContextInSlot () from /lib64/libn= ss3.so > #5 0x00007f799a9a42df in PK11_CreateContextBySymKey () from /lib64/libns= s3.so > #6 0x00007f799cd78b7f in nss_aes_operation(unsigned long, unsigned > long, PK11SymKeyStr*, SECItemStr*, ceph::buffer::list const&, > ceph::buffer::list&, std::string*) () from /lib64/librados.so.2 > #7 0x00007f799cd7990a in > CryptoAESKeyHandler::encrypt(ceph::buffer::list const&, > ceph::buffer::list&, std::string*) const () > from /lib64/librados.so.2 > #8 0x00007f799ce2446f in > CephxSessionHandler::_calc_signature(Message*, unsigned long*) () from > /lib64/librados.so.2 > #9 0x00007f799ce24c0d in > CephxSessionHandler::check_message_signature(Message*) () from > /lib64/librados.so.2 > #10 0x00007f799ce1d3ef in AsyncConnection::process() () from > /lib64/librados.so.2 > #11 0x00007f799cc827e9 in EventCenter::process_events(int) () from > /lib64/librados.so.2 > #12 0x00007f799cc851ea in NetworkStack::add_thread(unsigned > int)::{lambda()#1}::operator()() const () from /lib64/librados.so.2 > #13 0x00007f7999eaa220 in ?? () from /lib64/libstdc++.so.6 > #14 0x00007f799bd3bdc5 in start_thread () from /lib64/libpthread.so.0 > #15 0x00007f799b864ced in clone () from /lib64/libc.so.6 > (gdb) > > Regards, > Chuang Yan. > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html