All of lore.kernel.org
 help / color / mirror / Atom feed
* fio multithreads meets Segmentation fault
@ 2017-03-28  9:36 闫创
  2017-03-28 13:03 ` Ning Yao
  0 siblings, 1 reply; 3+ messages in thread
From: 闫创 @ 2017-03-28  9:36 UTC (permalink / raw)
  To: ceph-devel

Hi all,

When I make a fio-rbd-test, it says segmentation fault. I reckon that
there may be something wrong with the usage of libnss.
Ceph version is 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
using AsyncMessenger.
Is libnss3.so thread safe? or other thoughts to resolve this problem?

Here is the output of  #gdb /usr/bin/fio corefile

......
warning: the debug information found in
"/usr/lib/debug/usr/lib64//librados.so.2.0.0.debug" does not match
"/lib64/librados.so.2" (CRC mismatch).
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `fio rbd.fio --output 20G_rbd.log'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f799b7ebdbc in _int_malloc () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install fio-2.2.8-2.el7.x86_64
(gdb) bt
#0  0x00007f799b7ebdbc in _int_malloc () from /lib64/libc.so.6
#1  0x00007f799b7ee26c in malloc () from /lib64/libc.so.6
#2  0x00007f799a75a440 in PORT_Alloc_Util () from /lib64/libnssutil3.so
#3  0x00007f799a758d32 in SECITEM_ArenaDupItem_Util () from
/lib64/libnssutil3.so
#4  0x00007f799a9a4238 in pk11_CreateNewContextInSlot () from /lib64/libnss3.so
#5  0x00007f799a9a42df in PK11_CreateContextBySymKey () from /lib64/libnss3.so
#6  0x00007f799cd78b7f in nss_aes_operation(unsigned long, unsigned
long, PK11SymKeyStr*, SECItemStr*, ceph::buffer::list const&,
ceph::buffer::list&, std::string*) () from /lib64/librados.so.2
#7  0x00007f799cd7990a in
CryptoAESKeyHandler::encrypt(ceph::buffer::list const&,
ceph::buffer::list&, std::string*) const ()
   from /lib64/librados.so.2
#8  0x00007f799ce2446f in
CephxSessionHandler::_calc_signature(Message*, unsigned long*) () from
/lib64/librados.so.2
#9  0x00007f799ce24c0d in
CephxSessionHandler::check_message_signature(Message*) () from
/lib64/librados.so.2
#10 0x00007f799ce1d3ef in AsyncConnection::process() () from
/lib64/librados.so.2
#11 0x00007f799cc827e9 in EventCenter::process_events(int) () from
/lib64/librados.so.2
#12 0x00007f799cc851ea in NetworkStack::add_thread(unsigned
int)::{lambda()#1}::operator()() const () from /lib64/librados.so.2
#13 0x00007f7999eaa220 in ?? () from /lib64/libstdc++.so.6
#14 0x00007f799bd3bdc5 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f799b864ced in clone () from /lib64/libc.so.6
(gdb)

Regards,
Chuang Yan.

^ permalink raw reply	[flat|nested] 3+ messages in thread
* Re: fio multithreads meets Segmentation fault
  2017-03-28  9:36 fio multithreads meets Segmentation fault 闫创
@ 2017-03-28 13:03 ` Ning Yao
  2017-03-28 14:19   ` Haomai Wang
  0 siblings, 1 reply; 3+ messages in thread
From: Ning Yao @ 2017-03-28 13:03 UTC (permalink / raw)
  To: 闫创, Haomai Wang; +Cc: ceph-devel

Hi, Haomai

AsyncMessenger uses threadpools to handle connection and some thread
unsafe function such as PK11_CreateContextBySymKey in libnss causes
the corruptions here?

Regards
Ning Yao


2017-03-28 17:36 GMT+08:00 闫创 <yanchuang1994@gmail.com>:
> Hi all,
>
> When I make a fio-rbd-test, it says segmentation fault. I reckon that
> there may be something wrong with the usage of libnss.
> Ceph version is 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
> using AsyncMessenger.
> Is libnss3.so thread safe? or other thoughts to resolve this problem?
>
> Here is the output of  #gdb /usr/bin/fio corefile
>
> ......
> warning: the debug information found in
> "/usr/lib/debug/usr/lib64//librados.so.2.0.0.debug" does not match
> "/lib64/librados.so.2" (CRC mismatch).
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `fio rbd.fio --output 20G_rbd.log'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x00007f799b7ebdbc in _int_malloc () from /lib64/libc.so.6
> Missing separate debuginfos, use: debuginfo-install fio-2.2.8-2.el7.x86_64
> (gdb) bt
> #0  0x00007f799b7ebdbc in _int_malloc () from /lib64/libc.so.6
> #1  0x00007f799b7ee26c in malloc () from /lib64/libc.so.6
> #2  0x00007f799a75a440 in PORT_Alloc_Util () from /lib64/libnssutil3.so
> #3  0x00007f799a758d32 in SECITEM_ArenaDupItem_Util () from
> /lib64/libnssutil3.so
> #4  0x00007f799a9a4238 in pk11_CreateNewContextInSlot () from /lib64/libnss3.so
> #5  0x00007f799a9a42df in PK11_CreateContextBySymKey () from /lib64/libnss3.so
> #6  0x00007f799cd78b7f in nss_aes_operation(unsigned long, unsigned
> long, PK11SymKeyStr*, SECItemStr*, ceph::buffer::list const&,
> ceph::buffer::list&, std::string*) () from /lib64/librados.so.2
> #7  0x00007f799cd7990a in
> CryptoAESKeyHandler::encrypt(ceph::buffer::list const&,
> ceph::buffer::list&, std::string*) const ()
>    from /lib64/librados.so.2
> #8  0x00007f799ce2446f in
> CephxSessionHandler::_calc_signature(Message*, unsigned long*) () from
> /lib64/librados.so.2
> #9  0x00007f799ce24c0d in
> CephxSessionHandler::check_message_signature(Message*) () from
> /lib64/librados.so.2
> #10 0x00007f799ce1d3ef in AsyncConnection::process() () from
> /lib64/librados.so.2
> #11 0x00007f799cc827e9 in EventCenter::process_events(int) () from
> /lib64/librados.so.2
> #12 0x00007f799cc851ea in NetworkStack::add_thread(unsigned
> int)::{lambda()#1}::operator()() const () from /lib64/librados.so.2
> #13 0x00007f7999eaa220 in ?? () from /lib64/libstdc++.so.6
> #14 0x00007f799bd3bdc5 in start_thread () from /lib64/libpthread.so.0
> #15 0x00007f799b864ced in clone () from /lib64/libc.so.6
> (gdb)
>
> Regards,
> Chuang Yan.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 3+ messages in thread
* Re: fio multithreads meets Segmentation fault
  2017-03-28 13:03 ` Ning Yao
@ 2017-03-28 14:19   ` Haomai Wang
  0 siblings, 0 replies; 3+ messages in thread
From: Haomai Wang @ 2017-03-28 14:19 UTC (permalink / raw)
  To: Ning Yao; +Cc: 闫创, ceph-devel

I haven't see this problem before. Each connection is thread locally
so it's safe to use unsafe function without sync.

On Tue, Mar 28, 2017 at 9:03 AM, Ning Yao <zay11022@gmail.com> wrote:
> Hi, Haomai
>
> AsyncMessenger uses threadpools to handle connection and some thread
> unsafe function such as PK11_CreateContextBySymKey in libnss causes
> the corruptions here?
>
> Regards
> Ning Yao
>
>
> 2017-03-28 17:36 GMT+08:00 闫创 <yanchuang1994@gmail.com>:
>> Hi all,
>>
>> When I make a fio-rbd-test, it says segmentation fault. I reckon that
>> there may be something wrong with the usage of libnss.
>> Ceph version is 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
>> using AsyncMessenger.
>> Is libnss3.so thread safe? or other thoughts to resolve this problem?
>>
>> Here is the output of  #gdb /usr/bin/fio corefile
>>
>> ......
>> warning: the debug information found in
>> "/usr/lib/debug/usr/lib64//librados.so.2.0.0.debug" does not match
>> "/lib64/librados.so.2" (CRC mismatch).
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib64/libthread_db.so.1".
>> Core was generated by `fio rbd.fio --output 20G_rbd.log'.
>> Program terminated with signal 11, Segmentation fault.
>> #0  0x00007f799b7ebdbc in _int_malloc () from /lib64/libc.so.6
>> Missing separate debuginfos, use: debuginfo-install fio-2.2.8-2.el7.x86_64
>> (gdb) bt
>> #0  0x00007f799b7ebdbc in _int_malloc () from /lib64/libc.so.6
>> #1  0x00007f799b7ee26c in malloc () from /lib64/libc.so.6
>> #2  0x00007f799a75a440 in PORT_Alloc_Util () from /lib64/libnssutil3.so
>> #3  0x00007f799a758d32 in SECITEM_ArenaDupItem_Util () from
>> /lib64/libnssutil3.so
>> #4  0x00007f799a9a4238 in pk11_CreateNewContextInSlot () from /lib64/libnss3.so
>> #5  0x00007f799a9a42df in PK11_CreateContextBySymKey () from /lib64/libnss3.so
>> #6  0x00007f799cd78b7f in nss_aes_operation(unsigned long, unsigned
>> long, PK11SymKeyStr*, SECItemStr*, ceph::buffer::list const&,
>> ceph::buffer::list&, std::string*) () from /lib64/librados.so.2
>> #7  0x00007f799cd7990a in
>> CryptoAESKeyHandler::encrypt(ceph::buffer::list const&,
>> ceph::buffer::list&, std::string*) const ()
>>    from /lib64/librados.so.2
>> #8  0x00007f799ce2446f in
>> CephxSessionHandler::_calc_signature(Message*, unsigned long*) () from
>> /lib64/librados.so.2
>> #9  0x00007f799ce24c0d in
>> CephxSessionHandler::check_message_signature(Message*) () from
>> /lib64/librados.so.2
>> #10 0x00007f799ce1d3ef in AsyncConnection::process() () from
>> /lib64/librados.so.2
>> #11 0x00007f799cc827e9 in EventCenter::process_events(int) () from
>> /lib64/librados.so.2
>> #12 0x00007f799cc851ea in NetworkStack::add_thread(unsigned
>> int)::{lambda()#1}::operator()() const () from /lib64/librados.so.2
>> #13 0x00007f7999eaa220 in ?? () from /lib64/libstdc++.so.6
>> #14 0x00007f799bd3bdc5 in start_thread () from /lib64/libpthread.so.0
>> #15 0x00007f799b864ced in clone () from /lib64/libc.so.6
>> (gdb)
>>
>> Regards,
>> Chuang Yan.
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 3+ messages in thread
end of thread, other threads:[~2017-03-28 14:55 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-03-28  9:36 fio multithreads meets Segmentation fault 闫创
2017-03-28 13:03 ` Ning Yao
2017-03-28 14:19   ` Haomai Wang

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.