[Qemu-devel] Qemu + RBD = ceph::buffer::end_of_buffer

[Qemu-devel] Qemu + RBD = ceph::buffer::end_of_buffer

[Dyweni - Qemu-Devel]

Hi List!

I am tripping across this error as soon as the qemu rbd disk is
probed by the windows 2000 installer:

VNC server running on `127.0.0.1:5900'
terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
  what():  buffer::end_of_buffer
Aborted (core dumped)



Has anyone else tripped across this?



I am running the following:
  Linux Kernel 2.6.37-gentoo-r4
  Ceph 0.27
  QEMU-KVM (commit 28262112181f27f302b5186f0df6428df6b513e7)
    Pulled from: git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
    And patched with Josh Durgin's "rbd improvements" patchset
     (http://lists.gnu.org/archive/html/qemu-devel/2011-04/msg01211.html)



The GDB Backtrace is:

#0  0x00007f7733c2a495 in raise (sig=<value optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007f7733c2b81f in abort () at abort.c:92
#2  0x00007f7733175a25 in __gnu_cxx::__verbose_terminate_handler () at
/usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
#3  0x00007f7733172c64 in __cxxabiv1::__terminate (handler=0x7f7733175817
<__gnu_cxx::__verbose_terminate_handler()>)
    at
/usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
#4  0x00007f7733172c8c in std::terminate () at
/usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
#5  0x00007f7733172ea4 in __cxxabiv1::__cxa_throw (obj=0x18d62a0,
tinfo=0x7f7735d57ce0, dest=0x7f7735b4548a
<ceph::buffer::end_of_buffer::~end_of_buffer()>)
    at
/usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
#6  0x00007f7735b45b31 in ceph::buffer::list::iterator::copy
(this=0x7f7731cb2930, len=4, dest=0x7f7731cb28dc "") at
include/buffer.h:381
#7  0x00007f7735b483de in decode_raw<__le32> (t=@0x7f7731cb28dc, p=...) at
include/encoding.h:35
#8  0x00007f7735b46a40 in decode (v=@0x7f7731cb290c, p=...) at
include/encoding.h:80
#9  0x00007f7735b46b94 in decode (s=..., p=...) at include/encoding.h:189
#10 0x00007f77356ea3b5 in
librados::RadosClient::C_aio_sparse_read_Ack::finish (this=0x7f77280618d0,
r=0) at librados.cc:463
#11 0x00007f773572dff4 in Objecter::handle_osd_op_reply (this=0x181ff10,
m=0x18f1440) at osdc/Objecter.cc:801
#12 0x00007f77356d347c in librados::RadosClient::_dispatch
(this=0x181cdc0, m=0x18f1440) at librados.cc:751
#13 0x00007f77356d327c in librados::RadosClient::ms_dispatch
(this=0x181cdc0, m=0x18f1440) at librados.cc:717
#14 0x00007f773571d93d in Messenger::ms_deliver_dispatch (this=0x181f300,
m=0x18f1440) at msg/Messenger.h:98
#15 0x00007f773570b135 in SimpleMessenger::dispatch_entry (this=0x181f300)
at msg/SimpleMessenger.cc:352
#16 0x00007f77356e49ba in SimpleMessenger::DispatchThread::entry
(this=0x181f790) at msg/SimpleMessenger.h:533
#17 0x00007f773571c75d in Thread::_entry_func (arg=0x181f790) at
common/Thread.h:41
#18 0x00007f7736372ac4 in start_thread (arg=<value optimized out>) at
pthread_create.c:297
#19 0x00007f7733cc938d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115



Thanks,
Dyweni

Re: [Qemu-devel] Qemu + RBD = ceph::buffer::end_of_buffer

[Dyweni - Qemu-Devel]

Hi List!

I upgraded Ceph to the latest development version
	Commit: 0edbc75a5fe8c3028faf85546f3264d28653ea3f
	Pulled from: git://ceph.newdream.net/ceph.git

I recompiled the latest GIT version of QEMU-KVM (with Josh Durgin's
patches) against the latest git version of Ceph.

However, this error is still occurring:

terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
  what():  buffer::end_of_buffer
Aborted (core dumped)



Here's another backtrace from GDB:

#0  0x00007f16ff829495 in raise (sig=<value optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007f16ff82a81f in abort () at abort.c:92
#2  0x00007f16fed74a25 in __gnu_cxx::__verbose_terminate_handler () at
/usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
#3  0x00007f16fed71c64 in __cxxabiv1::__terminate (handler=0x7f16fed74817
<__gnu_cxx::__verbose_terminate_handler()>)
    at
/usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
#4  0x00007f16fed71c8c in std::terminate () at
/usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
#5  0x00007f16fed71ea4 in __cxxabiv1::__cxa_throw (obj=0x1346470,
tinfo=0x7f1701952ce0, dest=0x7f17017403d4
<ceph::buffer::end_of_buffer::~end_of_buffer()>)
    at
/usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
#6  0x00007f1701740a7b in ceph::buffer::list::iterator::copy
(this=0x7f16fd8b1930, len=4, dest=0x7f16fd8b18dc "") at
include/buffer.h:379
#7  0x00007f1701743328 in decode_raw<__le32> (t=@0x7f16fd8b18dc, p=...) at
include/encoding.h:35
#8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
include/encoding.h:80
#9  0x00007f1701741ade in decode (s=..., p=...) at include/encoding.h:189
#10 0x00007f17012e8369 in
librados::RadosClient::C_aio_sparse_read_Ack::finish (this=0x7f16f40d6200,
r=0) at librados.cc:463
#11 0x00007f170132bb5a in Objecter::handle_osd_op_reply (this=0x13423e0,
m=0x1346520) at osdc/Objecter.cc:794
#12 0x00007f17012d1444 in librados::RadosClient::_dispatch
(this=0x133f810, m=0x1346520) at librados.cc:751
#13 0x00007f17012d1244 in librados::RadosClient::ms_dispatch
(this=0x133f810, m=0x1346520) at librados.cc:717
#14 0x00007f170131b57b in Messenger::ms_deliver_dispatch (this=0x1341910,
m=0x1346520) at msg/Messenger.h:98
#15 0x00007f17013090d3 in SimpleMessenger::dispatch_entry (this=0x1341910)
at msg/SimpleMessenger.cc:352
#16 0x00007f17012e296e in SimpleMessenger::DispatchThread::entry
(this=0x1341da0) at msg/SimpleMessenger.h:533
#17 0x00007f170131a39b in Thread::_entry_func (arg=0x1341da0) at
common/Thread.h:41
#18 0x00007f1701f6dac4 in start_thread (arg=<value optimized out>) at
pthread_create.c:297
#19 0x00007f16ff8c838d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:115


Thanks,
Dyweni




> Hi List!
>
> I am tripping across this error as soon as the qemu rbd disk is
> probed by the windows 2000 installer:
>
> VNC server running on `127.0.0.1:5900'
> terminate called after throwing an instance of
> 'ceph::buffer::end_of_buffer'
>   what():  buffer::end_of_buffer
> Aborted (core dumped)
>
>
>
> Has anyone else tripped across this?
>
>
>
> I am running the following:
>   Linux Kernel 2.6.37-gentoo-r4
>   Ceph 0.27
>   QEMU-KVM (commit 28262112181f27f302b5186f0df6428df6b513e7)
>     Pulled from: git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
>     And patched with Josh Durgin's "rbd improvements" patchset
>      (http://lists.gnu.org/archive/html/qemu-devel/2011-04/msg01211.html)
>
>
>
> The GDB Backtrace is:
>
> #0  0x00007f7733c2a495 in raise (sig=<value optimized out>) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1  0x00007f7733c2b81f in abort () at abort.c:92
> #2  0x00007f7733175a25 in __gnu_cxx::__verbose_terminate_handler () at
> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
> #3  0x00007f7733172c64 in __cxxabiv1::__terminate (handler=0x7f7733175817
> <__gnu_cxx::__verbose_terminate_handler()>)
>     at
> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
> #4  0x00007f7733172c8c in std::terminate () at
> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
> #5  0x00007f7733172ea4 in __cxxabiv1::__cxa_throw (obj=0x18d62a0,
> tinfo=0x7f7735d57ce0, dest=0x7f7735b4548a
> <ceph::buffer::end_of_buffer::~end_of_buffer()>)
>     at
> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
> #6  0x00007f7735b45b31 in ceph::buffer::list::iterator::copy
> (this=0x7f7731cb2930, len=4, dest=0x7f7731cb28dc "") at
> include/buffer.h:381
> #7  0x00007f7735b483de in decode_raw<__le32> (t=@0x7f7731cb28dc, p=...) at
> include/encoding.h:35
> #8  0x00007f7735b46a40 in decode (v=@0x7f7731cb290c, p=...) at
> include/encoding.h:80
> #9  0x00007f7735b46b94 in decode (s=..., p=...) at include/encoding.h:189
> #10 0x00007f77356ea3b5 in
> librados::RadosClient::C_aio_sparse_read_Ack::finish (this=0x7f77280618d0,
> r=0) at librados.cc:463
> #11 0x00007f773572dff4 in Objecter::handle_osd_op_reply (this=0x181ff10,
> m=0x18f1440) at osdc/Objecter.cc:801
> #12 0x00007f77356d347c in librados::RadosClient::_dispatch
> (this=0x181cdc0, m=0x18f1440) at librados.cc:751
> #13 0x00007f77356d327c in librados::RadosClient::ms_dispatch
> (this=0x181cdc0, m=0x18f1440) at librados.cc:717
> #14 0x00007f773571d93d in Messenger::ms_deliver_dispatch (this=0x181f300,
> m=0x18f1440) at msg/Messenger.h:98
> #15 0x00007f773570b135 in SimpleMessenger::dispatch_entry (this=0x181f300)
> at msg/SimpleMessenger.cc:352
> #16 0x00007f77356e49ba in SimpleMessenger::DispatchThread::entry
> (this=0x181f790) at msg/SimpleMessenger.h:533
> #17 0x00007f773571c75d in Thread::_entry_func (arg=0x181f790) at
> common/Thread.h:41
> #18 0x00007f7736372ac4 in start_thread (arg=<value optimized out>) at
> pthread_create.c:297
> #19 0x00007f7733cc938d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
>
>
>
> Thanks,
> Dyweni
>
>
>
>
>

Re: [Qemu-devel] Qemu + RBD = ceph::buffer::end_of_buffer

[Josh Durgin]

CCing the ceph list.

On 05/06/2011 12:23 PM, Dyweni - Qemu-Devel wrote:
> Hi List!
>
> I upgraded Ceph to the latest development version
> 	Commit: 0edbc75a5fe8c3028faf85546f3264d28653ea3f
> 	Pulled from: git://ceph.newdream.net/ceph.git
>
> I recompiled the latest GIT version of QEMU-KVM (with Josh Durgin's
> patches) against the latest git version of Ceph.
>
> However, this error is still occurring:
>
> terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
>    what():  buffer::end_of_buffer
> Aborted (core dumped)
>
>
>
> Here's another backtrace from GDB:
>
> #0  0x00007f16ff829495 in raise (sig=<value optimized out>) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1  0x00007f16ff82a81f in abort () at abort.c:92
> #2  0x00007f16fed74a25 in __gnu_cxx::__verbose_terminate_handler () at
> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
> #3  0x00007f16fed71c64 in __cxxabiv1::__terminate (handler=0x7f16fed74817
> <__gnu_cxx::__verbose_terminate_handler()>)
>      at
> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
> #4  0x00007f16fed71c8c in std::terminate () at
> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
> #5  0x00007f16fed71ea4 in __cxxabiv1::__cxa_throw (obj=0x1346470,
> tinfo=0x7f1701952ce0, dest=0x7f17017403d4
> <ceph::buffer::end_of_buffer::~end_of_buffer()>)
>      at
> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
> #6  0x00007f1701740a7b in ceph::buffer::list::iterator::copy
> (this=0x7f16fd8b1930, len=4, dest=0x7f16fd8b18dc "") at
> include/buffer.h:379
> #7  0x00007f1701743328 in decode_raw<__le32>  (t=@0x7f16fd8b18dc, p=...) at
> include/encoding.h:35
> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
> include/encoding.h:80
> #9  0x00007f1701741ade in decode (s=..., p=...) at include/encoding.h:189
> #10 0x00007f17012e8369 in
> librados::RadosClient::C_aio_sparse_read_Ack::finish (this=0x7f16f40d6200,
> r=0) at librados.cc:463
> #11 0x00007f170132bb5a in Objecter::handle_osd_op_reply (this=0x13423e0,
> m=0x1346520) at osdc/Objecter.cc:794
> #12 0x00007f17012d1444 in librados::RadosClient::_dispatch
> (this=0x133f810, m=0x1346520) at librados.cc:751
> #13 0x00007f17012d1244 in librados::RadosClient::ms_dispatch
> (this=0x133f810, m=0x1346520) at librados.cc:717
> #14 0x00007f170131b57b in Messenger::ms_deliver_dispatch (this=0x1341910,
> m=0x1346520) at msg/Messenger.h:98
> #15 0x00007f17013090d3 in SimpleMessenger::dispatch_entry (this=0x1341910)
> at msg/SimpleMessenger.cc:352
> #16 0x00007f17012e296e in SimpleMessenger::DispatchThread::entry
> (this=0x1341da0) at msg/SimpleMessenger.h:533
> #17 0x00007f170131a39b in Thread::_entry_func (arg=0x1341da0) at
> common/Thread.h:41
> #18 0x00007f1701f6dac4 in start_thread (arg=<value optimized out>) at
> pthread_create.c:297
> #19 0x00007f16ff8c838d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

I haven't seen that error before, but it's probably a bug in the OSD 
where it doesn't set an error code. If you've still got the core file, 
could you go to frame 10 and send us the values of r, bl._len, and iter.off?

Thanks,
Josh

Re: [Qemu-devel] Qemu + RBD = ceph::buffer::end_of_buffer

[Josh Durgin]

CCing the ceph list.

On 05/06/2011 12:23 PM, Dyweni - Qemu-Devel wrote:
> Hi List!
>
> I upgraded Ceph to the latest development version
> 	Commit: 0edbc75a5fe8c3028faf85546f3264d28653ea3f
> 	Pulled from: git://ceph.newdream.net/ceph.git
>
> I recompiled the latest GIT version of QEMU-KVM (with Josh Durgin's
> patches) against the latest git version of Ceph.
>
> However, this error is still occurring:
>
> terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
>    what():  buffer::end_of_buffer
> Aborted (core dumped)
>
>
>
> Here's another backtrace from GDB:
>
> #0  0x00007f16ff829495 in raise (sig=<value optimized out>) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1  0x00007f16ff82a81f in abort () at abort.c:92
> #2  0x00007f16fed74a25 in __gnu_cxx::__verbose_terminate_handler () at
> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
> #3  0x00007f16fed71c64 in __cxxabiv1::__terminate (handler=0x7f16fed74817
> <__gnu_cxx::__verbose_terminate_handler()>)
>      at
> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
> #4  0x00007f16fed71c8c in std::terminate () at
> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
> #5  0x00007f16fed71ea4 in __cxxabiv1::__cxa_throw (obj=0x1346470,
> tinfo=0x7f1701952ce0, dest=0x7f17017403d4
> <ceph::buffer::end_of_buffer::~end_of_buffer()>)
>      at
> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
> #6  0x00007f1701740a7b in ceph::buffer::list::iterator::copy
> (this=0x7f16fd8b1930, len=4, dest=0x7f16fd8b18dc "") at
> include/buffer.h:379
> #7  0x00007f1701743328 in decode_raw<__le32>  (t=@0x7f16fd8b18dc, p=...) at
> include/encoding.h:35
> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
> include/encoding.h:80
> #9  0x00007f1701741ade in decode (s=..., p=...) at include/encoding.h:189
> #10 0x00007f17012e8369 in
> librados::RadosClient::C_aio_sparse_read_Ack::finish (this=0x7f16f40d6200,
> r=0) at librados.cc:463
> #11 0x00007f170132bb5a in Objecter::handle_osd_op_reply (this=0x13423e0,
> m=0x1346520) at osdc/Objecter.cc:794
> #12 0x00007f17012d1444 in librados::RadosClient::_dispatch
> (this=0x133f810, m=0x1346520) at librados.cc:751
> #13 0x00007f17012d1244 in librados::RadosClient::ms_dispatch
> (this=0x133f810, m=0x1346520) at librados.cc:717
> #14 0x00007f170131b57b in Messenger::ms_deliver_dispatch (this=0x1341910,
> m=0x1346520) at msg/Messenger.h:98
> #15 0x00007f17013090d3 in SimpleMessenger::dispatch_entry (this=0x1341910)
> at msg/SimpleMessenger.cc:352
> #16 0x00007f17012e296e in SimpleMessenger::DispatchThread::entry
> (this=0x1341da0) at msg/SimpleMessenger.h:533
> #17 0x00007f170131a39b in Thread::_entry_func (arg=0x1341da0) at
> common/Thread.h:41
> #18 0x00007f1701f6dac4 in start_thread (arg=<value optimized out>) at
> pthread_create.c:297
> #19 0x00007f16ff8c838d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

I haven't seen that error before, but it's probably a bug in the OSD 
where it doesn't set an error code. If you've still got the core file, 
could you go to frame 10 and send us the values of r, bl._len, and iter.off?

Thanks,
Josh

Re: Qemu + RBD = ceph::buffer::end_of_buffer

[Dyweni - Qemu-Devel]

Hi Josh/Lists!

463             ::decode(*data_bl, iter);
(gdb) print r
$1 = 0
(gdb) print data_bl
$2 = (ceph::bufferlist *) 0x7f16f40d6060
(gdb) print data_bl->_len
$3 = 0
(gdb) print iter->off
$4 = 20


Thanks,
Dyweni



> CCing the ceph list.
>
> On 05/06/2011 12:23 PM, Dyweni - Qemu-Devel wrote:
>> Hi List!
>>
>> I upgraded Ceph to the latest development version
>> 	Commit: 0edbc75a5fe8c3028faf85546f3264d28653ea3f
>> 	Pulled from: git://ceph.newdream.net/ceph.git
>>
>> I recompiled the latest GIT version of QEMU-KVM (with Josh Durgin's
>> patches) against the latest git version of Ceph.
>>
>> However, this error is still occurring:
>>
>> terminate called after throwing an instance of
>> 'ceph::buffer::end_of_buffer'
>>    what():  buffer::end_of_buffer
>> Aborted (core dumped)
>>
>>
>>
>> Here's another backtrace from GDB:
>>
>> #0  0x00007f16ff829495 in raise (sig=<value optimized out>) at
>> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
>> #1  0x00007f16ff82a81f in abort () at abort.c:92
>> #2  0x00007f16fed74a25 in __gnu_cxx::__verbose_terminate_handler () at
>> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
>> #3  0x00007f16fed71c64 in __cxxabiv1::__terminate
>> (handler=0x7f16fed74817
>> <__gnu_cxx::__verbose_terminate_handler()>)
>>      at
>> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
>> #4  0x00007f16fed71c8c in std::terminate () at
>> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
>> #5  0x00007f16fed71ea4 in __cxxabiv1::__cxa_throw (obj=0x1346470,
>> tinfo=0x7f1701952ce0, dest=0x7f17017403d4
>> <ceph::buffer::end_of_buffer::~end_of_buffer()>)
>>      at
>> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
>> #6  0x00007f1701740a7b in ceph::buffer::list::iterator::copy
>> (this=0x7f16fd8b1930, len=4, dest=0x7f16fd8b18dc "") at
>> include/buffer.h:379
>> #7  0x00007f1701743328 in decode_raw<__le32>  (t=@0x7f16fd8b18dc, p=...)
>> at
>> include/encoding.h:35
>> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
>> include/encoding.h:80
>> #9  0x00007f1701741ade in decode (s=..., p=...) at
>> include/encoding.h:189
>> #10 0x00007f17012e8369 in
>> librados::RadosClient::C_aio_sparse_read_Ack::finish
>> (this=0x7f16f40d6200,
>> r=0) at librados.cc:463
>> #11 0x00007f170132bb5a in Objecter::handle_osd_op_reply (this=0x13423e0,
>> m=0x1346520) at osdc/Objecter.cc:794
>> #12 0x00007f17012d1444 in librados::RadosClient::_dispatch
>> (this=0x133f810, m=0x1346520) at librados.cc:751
>> #13 0x00007f17012d1244 in librados::RadosClient::ms_dispatch
>> (this=0x133f810, m=0x1346520) at librados.cc:717
>> #14 0x00007f170131b57b in Messenger::ms_deliver_dispatch
>> (this=0x1341910,
>> m=0x1346520) at msg/Messenger.h:98
>> #15 0x00007f17013090d3 in SimpleMessenger::dispatch_entry
>> (this=0x1341910)
>> at msg/SimpleMessenger.cc:352
>> #16 0x00007f17012e296e in SimpleMessenger::DispatchThread::entry
>> (this=0x1341da0) at msg/SimpleMessenger.h:533
>> #17 0x00007f170131a39b in Thread::_entry_func (arg=0x1341da0) at
>> common/Thread.h:41
>> #18 0x00007f1701f6dac4 in start_thread (arg=<value optimized out>) at
>> pthread_create.c:297
>> #19 0x00007f16ff8c838d in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
>
> I haven't seen that error before, but it's probably a bug in the OSD
> where it doesn't set an error code. If you've still got the core file,
> could you go to frame 10 and send us the values of r, bl._len, and
> iter.off?
>
> Thanks,
> Josh
>

Re: [Qemu-devel] Qemu + RBD = ceph::buffer::end_of_buffer

[Dyweni - Qemu-Devel]

Hi Josh/Lists!

463             ::decode(*data_bl, iter);
(gdb) print r
$1 = 0
(gdb) print data_bl
$2 = (ceph::bufferlist *) 0x7f16f40d6060
(gdb) print data_bl->_len
$3 = 0
(gdb) print iter->off
$4 = 20


Thanks,
Dyweni



> CCing the ceph list.
>
> On 05/06/2011 12:23 PM, Dyweni - Qemu-Devel wrote:
>> Hi List!
>>
>> I upgraded Ceph to the latest development version
>> 	Commit: 0edbc75a5fe8c3028faf85546f3264d28653ea3f
>> 	Pulled from: git://ceph.newdream.net/ceph.git
>>
>> I recompiled the latest GIT version of QEMU-KVM (with Josh Durgin's
>> patches) against the latest git version of Ceph.
>>
>> However, this error is still occurring:
>>
>> terminate called after throwing an instance of
>> 'ceph::buffer::end_of_buffer'
>>    what():  buffer::end_of_buffer
>> Aborted (core dumped)
>>
>>
>>
>> Here's another backtrace from GDB:
>>
>> #0  0x00007f16ff829495 in raise (sig=<value optimized out>) at
>> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
>> #1  0x00007f16ff82a81f in abort () at abort.c:92
>> #2  0x00007f16fed74a25 in __gnu_cxx::__verbose_terminate_handler () at
>> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
>> #3  0x00007f16fed71c64 in __cxxabiv1::__terminate
>> (handler=0x7f16fed74817
>> <__gnu_cxx::__verbose_terminate_handler()>)
>>      at
>> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
>> #4  0x00007f16fed71c8c in std::terminate () at
>> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
>> #5  0x00007f16fed71ea4 in __cxxabiv1::__cxa_throw (obj=0x1346470,
>> tinfo=0x7f1701952ce0, dest=0x7f17017403d4
>> <ceph::buffer::end_of_buffer::~end_of_buffer()>)
>>      at
>> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
>> #6  0x00007f1701740a7b in ceph::buffer::list::iterator::copy
>> (this=0x7f16fd8b1930, len=4, dest=0x7f16fd8b18dc "") at
>> include/buffer.h:379
>> #7  0x00007f1701743328 in decode_raw<__le32>  (t=@0x7f16fd8b18dc, p=...)
>> at
>> include/encoding.h:35
>> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
>> include/encoding.h:80
>> #9  0x00007f1701741ade in decode (s=..., p=...) at
>> include/encoding.h:189
>> #10 0x00007f17012e8369 in
>> librados::RadosClient::C_aio_sparse_read_Ack::finish
>> (this=0x7f16f40d6200,
>> r=0) at librados.cc:463
>> #11 0x00007f170132bb5a in Objecter::handle_osd_op_reply (this=0x13423e0,
>> m=0x1346520) at osdc/Objecter.cc:794
>> #12 0x00007f17012d1444 in librados::RadosClient::_dispatch
>> (this=0x133f810, m=0x1346520) at librados.cc:751
>> #13 0x00007f17012d1244 in librados::RadosClient::ms_dispatch
>> (this=0x133f810, m=0x1346520) at librados.cc:717
>> #14 0x00007f170131b57b in Messenger::ms_deliver_dispatch
>> (this=0x1341910,
>> m=0x1346520) at msg/Messenger.h:98
>> #15 0x00007f17013090d3 in SimpleMessenger::dispatch_entry
>> (this=0x1341910)
>> at msg/SimpleMessenger.cc:352
>> #16 0x00007f17012e296e in SimpleMessenger::DispatchThread::entry
>> (this=0x1341da0) at msg/SimpleMessenger.h:533
>> #17 0x00007f170131a39b in Thread::_entry_func (arg=0x1341da0) at
>> common/Thread.h:41
>> #18 0x00007f1701f6dac4 in start_thread (arg=<value optimized out>) at
>> pthread_create.c:297
>> #19 0x00007f16ff8c838d in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
>
> I haven't seen that error before, but it's probably a bug in the OSD
> where it doesn't set an error code. If you've still got the core file,
> could you go to frame 10 and send us the values of r, bl._len, and
> iter.off?
>
> Thanks,
> Josh
>

Re: [Qemu-devel] Qemu + RBD = ceph::buffer::end_of_buffer

[Sage Weil]

On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
> Hi Josh/Lists!
> 
> 463             ::decode(*data_bl, iter);
> (gdb) print r
> $1 = 0
> (gdb) print data_bl
> $2 = (ceph::bufferlist *) 0x7f16f40d6060
> (gdb) print data_bl->_len
> $3 = 0

What about c->bl._len?

sage


> (gdb) print iter->off
> $4 = 20
> 
> 
> Thanks,
> Dyweni
> 
> 
> 
> > CCing the ceph list.
> >
> > On 05/06/2011 12:23 PM, Dyweni - Qemu-Devel wrote:
> >> Hi List!
> >>
> >> I upgraded Ceph to the latest development version
> >> 	Commit: 0edbc75a5fe8c3028faf85546f3264d28653ea3f
> >> 	Pulled from: git://ceph.newdream.net/ceph.git
> >>
> >> I recompiled the latest GIT version of QEMU-KVM (with Josh Durgin's
> >> patches) against the latest git version of Ceph.
> >>
> >> However, this error is still occurring:
> >>
> >> terminate called after throwing an instance of
> >> 'ceph::buffer::end_of_buffer'
> >>    what():  buffer::end_of_buffer
> >> Aborted (core dumped)
> >>
> >>
> >>
> >> Here's another backtrace from GDB:
> >>
> >> #0  0x00007f16ff829495 in raise (sig=<value optimized out>) at
> >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> >> #1  0x00007f16ff82a81f in abort () at abort.c:92
> >> #2  0x00007f16fed74a25 in __gnu_cxx::__verbose_terminate_handler () at
> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
> >> #3  0x00007f16fed71c64 in __cxxabiv1::__terminate
> >> (handler=0x7f16fed74817
> >> <__gnu_cxx::__verbose_terminate_handler()>)
> >>      at
> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
> >> #4  0x00007f16fed71c8c in std::terminate () at
> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
> >> #5  0x00007f16fed71ea4 in __cxxabiv1::__cxa_throw (obj=0x1346470,
> >> tinfo=0x7f1701952ce0, dest=0x7f17017403d4
> >> <ceph::buffer::end_of_buffer::~end_of_buffer()>)
> >>      at
> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
> >> #6  0x00007f1701740a7b in ceph::buffer::list::iterator::copy
> >> (this=0x7f16fd8b1930, len=4, dest=0x7f16fd8b18dc "") at
> >> include/buffer.h:379
> >> #7  0x00007f1701743328 in decode_raw<__le32>  (t=@0x7f16fd8b18dc, p=...)
> >> at
> >> include/encoding.h:35
> >> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
> >> include/encoding.h:80
> >> #9  0x00007f1701741ade in decode (s=..., p=...) at
> >> include/encoding.h:189
> >> #10 0x00007f17012e8369 in
> >> librados::RadosClient::C_aio_sparse_read_Ack::finish
> >> (this=0x7f16f40d6200,
> >> r=0) at librados.cc:463
> >> #11 0x00007f170132bb5a in Objecter::handle_osd_op_reply (this=0x13423e0,
> >> m=0x1346520) at osdc/Objecter.cc:794
> >> #12 0x00007f17012d1444 in librados::RadosClient::_dispatch
> >> (this=0x133f810, m=0x1346520) at librados.cc:751
> >> #13 0x00007f17012d1244 in librados::RadosClient::ms_dispatch
> >> (this=0x133f810, m=0x1346520) at librados.cc:717
> >> #14 0x00007f170131b57b in Messenger::ms_deliver_dispatch
> >> (this=0x1341910,
> >> m=0x1346520) at msg/Messenger.h:98
> >> #15 0x00007f17013090d3 in SimpleMessenger::dispatch_entry
> >> (this=0x1341910)
> >> at msg/SimpleMessenger.cc:352
> >> #16 0x00007f17012e296e in SimpleMessenger::DispatchThread::entry
> >> (this=0x1341da0) at msg/SimpleMessenger.h:533
> >> #17 0x00007f170131a39b in Thread::_entry_func (arg=0x1341da0) at
> >> common/Thread.h:41
> >> #18 0x00007f1701f6dac4 in start_thread (arg=<value optimized out>) at
> >> pthread_create.c:297
> >> #19 0x00007f16ff8c838d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> >
> > I haven't seen that error before, but it's probably a bug in the OSD
> > where it doesn't set an error code. If you've still got the core file,
> > could you go to frame 10 and send us the values of r, bl._len, and
> > iter.off?
> >
> > Thanks,
> > Josh
> >
> 
> 
> 
> 

Re: [Qemu-devel] Qemu + RBD = ceph::buffer::end_of_buffer

[Sage Weil]

On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
> Hi Josh/Lists!
> 
> 463             ::decode(*data_bl, iter);
> (gdb) print r
> $1 = 0
> (gdb) print data_bl
> $2 = (ceph::bufferlist *) 0x7f16f40d6060
> (gdb) print data_bl->_len
> $3 = 0

What about c->bl._len?

sage


> (gdb) print iter->off
> $4 = 20
> 
> 
> Thanks,
> Dyweni
> 
> 
> 
> > CCing the ceph list.
> >
> > On 05/06/2011 12:23 PM, Dyweni - Qemu-Devel wrote:
> >> Hi List!
> >>
> >> I upgraded Ceph to the latest development version
> >> 	Commit: 0edbc75a5fe8c3028faf85546f3264d28653ea3f
> >> 	Pulled from: git://ceph.newdream.net/ceph.git
> >>
> >> I recompiled the latest GIT version of QEMU-KVM (with Josh Durgin's
> >> patches) against the latest git version of Ceph.
> >>
> >> However, this error is still occurring:
> >>
> >> terminate called after throwing an instance of
> >> 'ceph::buffer::end_of_buffer'
> >>    what():  buffer::end_of_buffer
> >> Aborted (core dumped)
> >>
> >>
> >>
> >> Here's another backtrace from GDB:
> >>
> >> #0  0x00007f16ff829495 in raise (sig=<value optimized out>) at
> >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> >> #1  0x00007f16ff82a81f in abort () at abort.c:92
> >> #2  0x00007f16fed74a25 in __gnu_cxx::__verbose_terminate_handler () at
> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
> >> #3  0x00007f16fed71c64 in __cxxabiv1::__terminate
> >> (handler=0x7f16fed74817
> >> <__gnu_cxx::__verbose_terminate_handler()>)
> >>      at
> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
> >> #4  0x00007f16fed71c8c in std::terminate () at
> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
> >> #5  0x00007f16fed71ea4 in __cxxabiv1::__cxa_throw (obj=0x1346470,
> >> tinfo=0x7f1701952ce0, dest=0x7f17017403d4
> >> <ceph::buffer::end_of_buffer::~end_of_buffer()>)
> >>      at
> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
> >> #6  0x00007f1701740a7b in ceph::buffer::list::iterator::copy
> >> (this=0x7f16fd8b1930, len=4, dest=0x7f16fd8b18dc "") at
> >> include/buffer.h:379
> >> #7  0x00007f1701743328 in decode_raw<__le32>  (t=@0x7f16fd8b18dc, p=...)
> >> at
> >> include/encoding.h:35
> >> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
> >> include/encoding.h:80
> >> #9  0x00007f1701741ade in decode (s=..., p=...) at
> >> include/encoding.h:189
> >> #10 0x00007f17012e8369 in
> >> librados::RadosClient::C_aio_sparse_read_Ack::finish
> >> (this=0x7f16f40d6200,
> >> r=0) at librados.cc:463
> >> #11 0x00007f170132bb5a in Objecter::handle_osd_op_reply (this=0x13423e0,
> >> m=0x1346520) at osdc/Objecter.cc:794
> >> #12 0x00007f17012d1444 in librados::RadosClient::_dispatch
> >> (this=0x133f810, m=0x1346520) at librados.cc:751
> >> #13 0x00007f17012d1244 in librados::RadosClient::ms_dispatch
> >> (this=0x133f810, m=0x1346520) at librados.cc:717
> >> #14 0x00007f170131b57b in Messenger::ms_deliver_dispatch
> >> (this=0x1341910,
> >> m=0x1346520) at msg/Messenger.h:98
> >> #15 0x00007f17013090d3 in SimpleMessenger::dispatch_entry
> >> (this=0x1341910)
> >> at msg/SimpleMessenger.cc:352
> >> #16 0x00007f17012e296e in SimpleMessenger::DispatchThread::entry
> >> (this=0x1341da0) at msg/SimpleMessenger.h:533
> >> #17 0x00007f170131a39b in Thread::_entry_func (arg=0x1341da0) at
> >> common/Thread.h:41
> >> #18 0x00007f1701f6dac4 in start_thread (arg=<value optimized out>) at
> >> pthread_create.c:297
> >> #19 0x00007f16ff8c838d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> >
> > I haven't seen that error before, but it's probably a bug in the OSD
> > where it doesn't set an error code. If you've still got the core file,
> > could you go to frame 10 and send us the values of r, bl._len, and
> > iter.off?
> >
> > Thanks,
> > Josh
> >
> 
> 
> 
> 

Re: [Qemu-devel] Qemu + RBD = ceph::buffer::end_of_buffer

[Dyweni - Qemu-Devel]

Hi Sage/Lists!


(gdb) print c->bl._len
$1 = 20


And in case this is helpful:

(gdb) print *c
$2 = {lock = {name = 0x7f1701430f8d "AioCompletionImpl lock", id = -1,
recursive = false, lockdep = true, backtrace = false, _m = {__data =
{__lock = 1, __count = 0,
        __owner = 25800, __nusers = 1, __kind = 2, __spins = 0, __list =
{__prev = 0x0, __next = 0x0}},
      __size =
"\001\000\000\000\000\000\000\000\310d\000\000\001\000\000\000\002",
'\000' <repeats 22 times>, __align = 1}, nlock = 1}, cond = {
    _vptr.Cond = 0x7f1701952bd0, _c = {__data = {__lock = 0, __futex = 0,
__total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0,
__nwaiters = 0,
        __broadcast_seq = 0}, __size = '\000' <repeats 47 times>, __align
= 0}}, ref = 1, rval = 0, released = true, ack = true, safe =
false, objver = {version = 0,
    epoch = 0, __pad = 0}, callback_complete = 0x7f170173de33
<librbd::rados_aio_sparse_read_cb(rados_completion_t, void*)>,
  callback_safe = 0x7f170173d8bd <librbd::rados_cb(rados_completion_t,
void*)>, callback_arg = 0x7f16f40d6010, bl = {
    _buffers = {<std::_List_base<ceph::buffer::ptr,
std::allocator<ceph::buffer::ptr> >> = {
        _M_impl = {<std::allocator<std::_List_node<ceph::buffer::ptr> >> =
{<__gnu_cxx::new_allocator<std::_List_node<ceph::buffer::ptr> >> =
{<No data fields>}, <No data fields>}, _M_node = {_M_next =
0x1350530, _M_prev = 0x1350530}}}, <No data fields>}, _len = 20,
append_buffer = {_raw = 0x0, _off = 0, _len = 0}, last_p = {
      bl = 0x7f16f40d6170, ls = 0x7f16f40d6170, off = 0, p = {_M_node =
0x7f16f40d6170}, p_off = 0}}, pbl = 0x0, buf = 0x0, maxlen = 0}



Thanks,
Dyweni




> On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
>> Hi Josh/Lists!
>>
>> 463             ::decode(*data_bl, iter);
>> (gdb) print r
>> $1 = 0
>> (gdb) print data_bl
>> $2 = (ceph::bufferlist *) 0x7f16f40d6060
>> (gdb) print data_bl->_len
>> $3 = 0
>
> What about c->bl._len?
>
> sage
>
>
>> (gdb) print iter->off
>> $4 = 20
>>
>>
>> Thanks,
>> Dyweni
>>
>>
>>
>> > CCing the ceph list.
>> >
>> > On 05/06/2011 12:23 PM, Dyweni - Qemu-Devel wrote:
>> >> Hi List!
>> >>
>> >> I upgraded Ceph to the latest development version
>> >> 	Commit: 0edbc75a5fe8c3028faf85546f3264d28653ea3f
>> >> 	Pulled from: git://ceph.newdream.net/ceph.git
>> >>
>> >> I recompiled the latest GIT version of QEMU-KVM (with Josh Durgin's
>> >> patches) against the latest git version of Ceph.
>> >>
>> >> However, this error is still occurring:
>> >>
>> >> terminate called after throwing an instance of
>> >> 'ceph::buffer::end_of_buffer'
>> >>    what():  buffer::end_of_buffer
>> >> Aborted (core dumped)
>> >>
>> >>
>> >>
>> >> Here's another backtrace from GDB:
>> >>
>> >> #0  0x00007f16ff829495 in raise (sig=<value optimized out>) at
>> >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
>> >> #1  0x00007f16ff82a81f in abort () at abort.c:92
>> >> #2  0x00007f16fed74a25 in __gnu_cxx::__verbose_terminate_handler ()
>> at
>> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
>> >> #3  0x00007f16fed71c64 in __cxxabiv1::__terminate
>> >> (handler=0x7f16fed74817
>> >> <__gnu_cxx::__verbose_terminate_handler()>)
>> >>      at
>> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
>> >> #4  0x00007f16fed71c8c in std::terminate () at
>> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
>> >> #5  0x00007f16fed71ea4 in __cxxabiv1::__cxa_throw (obj=0x1346470,
>> >> tinfo=0x7f1701952ce0, dest=0x7f17017403d4
>> >> <ceph::buffer::end_of_buffer::~end_of_buffer()>)
>> >>      at
>> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
>> >> #6  0x00007f1701740a7b in ceph::buffer::list::iterator::copy
>> >> (this=0x7f16fd8b1930, len=4, dest=0x7f16fd8b18dc "") at
>> >> include/buffer.h:379
>> >> #7  0x00007f1701743328 in decode_raw<__le32>  (t=@0x7f16fd8b18dc,
>> p=...)
>> >> at
>> >> include/encoding.h:35
>> >> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
>> >> include/encoding.h:80
>> >> #9  0x00007f1701741ade in decode (s=..., p=...) at
>> >> include/encoding.h:189
>> >> #10 0x00007f17012e8369 in
>> >> librados::RadosClient::C_aio_sparse_read_Ack::finish
>> >> (this=0x7f16f40d6200,
>> >> r=0) at librados.cc:463
>> >> #11 0x00007f170132bb5a in Objecter::handle_osd_op_reply
>> (this=0x13423e0,
>> >> m=0x1346520) at osdc/Objecter.cc:794
>> >> #12 0x00007f17012d1444 in librados::RadosClient::_dispatch
>> >> (this=0x133f810, m=0x1346520) at librados.cc:751
>> >> #13 0x00007f17012d1244 in librados::RadosClient::ms_dispatch
>> >> (this=0x133f810, m=0x1346520) at librados.cc:717
>> >> #14 0x00007f170131b57b in Messenger::ms_deliver_dispatch
>> >> (this=0x1341910,
>> >> m=0x1346520) at msg/Messenger.h:98
>> >> #15 0x00007f17013090d3 in SimpleMessenger::dispatch_entry
>> >> (this=0x1341910)
>> >> at msg/SimpleMessenger.cc:352
>> >> #16 0x00007f17012e296e in SimpleMessenger::DispatchThread::entry
>> >> (this=0x1341da0) at msg/SimpleMessenger.h:533
>> >> #17 0x00007f170131a39b in Thread::_entry_func (arg=0x1341da0) at
>> >> common/Thread.h:41
>> >> #18 0x00007f1701f6dac4 in start_thread (arg=<value optimized out>) at
>> >> pthread_create.c:297
>> >> #19 0x00007f16ff8c838d in clone () at
>> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
>> >
>> > I haven't seen that error before, but it's probably a bug in the OSD
>> > where it doesn't set an error code. If you've still got the core file,
>> > could you go to frame 10 and send us the values of r, bl._len, and
>> > iter.off?
>> >
>> > Thanks,
>> > Josh
>> >
>>
>>
>>
>>
>

Re: [Qemu-devel] Qemu + RBD = ceph::buffer::end_of_buffer

[Dyweni - Qemu-Devel]

Hi Sage/Lists!


(gdb) print c->bl._len
$1 = 20


And in case this is helpful:

(gdb) print *c
$2 = {lock = {name = 0x7f1701430f8d "AioCompletionImpl lock", id = -1,
recursive = false, lockdep = true, backtrace = false, _m = {__data =
{__lock = 1, __count = 0,
        __owner = 25800, __nusers = 1, __kind = 2, __spins = 0, __list =
{__prev = 0x0, __next = 0x0}},
      __size =
"\001\000\000\000\000\000\000\000\310d\000\000\001\000\000\000\002",
'\000' <repeats 22 times>, __align = 1}, nlock = 1}, cond = {
    _vptr.Cond = 0x7f1701952bd0, _c = {__data = {__lock = 0, __futex = 0,
__total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0,
__nwaiters = 0,
        __broadcast_seq = 0}, __size = '\000' <repeats 47 times>, __align
= 0}}, ref = 1, rval = 0, released = true, ack = true, safe =
false, objver = {version = 0,
    epoch = 0, __pad = 0}, callback_complete = 0x7f170173de33
<librbd::rados_aio_sparse_read_cb(rados_completion_t, void*)>,
  callback_safe = 0x7f170173d8bd <librbd::rados_cb(rados_completion_t,
void*)>, callback_arg = 0x7f16f40d6010, bl = {
    _buffers = {<std::_List_base<ceph::buffer::ptr,
std::allocator<ceph::buffer::ptr> >> = {
        _M_impl = {<std::allocator<std::_List_node<ceph::buffer::ptr> >> =
{<__gnu_cxx::new_allocator<std::_List_node<ceph::buffer::ptr> >> =
{<No data fields>}, <No data fields>}, _M_node = {_M_next =
0x1350530, _M_prev = 0x1350530}}}, <No data fields>}, _len = 20,
append_buffer = {_raw = 0x0, _off = 0, _len = 0}, last_p = {
      bl = 0x7f16f40d6170, ls = 0x7f16f40d6170, off = 0, p = {_M_node =
0x7f16f40d6170}, p_off = 0}}, pbl = 0x0, buf = 0x0, maxlen = 0}



Thanks,
Dyweni




> On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
>> Hi Josh/Lists!
>>
>> 463             ::decode(*data_bl, iter);
>> (gdb) print r
>> $1 = 0
>> (gdb) print data_bl
>> $2 = (ceph::bufferlist *) 0x7f16f40d6060
>> (gdb) print data_bl->_len
>> $3 = 0
>
> What about c->bl._len?
>
> sage
>
>
>> (gdb) print iter->off
>> $4 = 20
>>
>>
>> Thanks,
>> Dyweni
>>
>>
>>
>> > CCing the ceph list.
>> >
>> > On 05/06/2011 12:23 PM, Dyweni - Qemu-Devel wrote:
>> >> Hi List!
>> >>
>> >> I upgraded Ceph to the latest development version
>> >> 	Commit: 0edbc75a5fe8c3028faf85546f3264d28653ea3f
>> >> 	Pulled from: git://ceph.newdream.net/ceph.git
>> >>
>> >> I recompiled the latest GIT version of QEMU-KVM (with Josh Durgin's
>> >> patches) against the latest git version of Ceph.
>> >>
>> >> However, this error is still occurring:
>> >>
>> >> terminate called after throwing an instance of
>> >> 'ceph::buffer::end_of_buffer'
>> >>    what():  buffer::end_of_buffer
>> >> Aborted (core dumped)
>> >>
>> >>
>> >>
>> >> Here's another backtrace from GDB:
>> >>
>> >> #0  0x00007f16ff829495 in raise (sig=<value optimized out>) at
>> >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
>> >> #1  0x00007f16ff82a81f in abort () at abort.c:92
>> >> #2  0x00007f16fed74a25 in __gnu_cxx::__verbose_terminate_handler ()
>> at
>> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
>> >> #3  0x00007f16fed71c64 in __cxxabiv1::__terminate
>> >> (handler=0x7f16fed74817
>> >> <__gnu_cxx::__verbose_terminate_handler()>)
>> >>      at
>> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
>> >> #4  0x00007f16fed71c8c in std::terminate () at
>> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
>> >> #5  0x00007f16fed71ea4 in __cxxabiv1::__cxa_throw (obj=0x1346470,
>> >> tinfo=0x7f1701952ce0, dest=0x7f17017403d4
>> >> <ceph::buffer::end_of_buffer::~end_of_buffer()>)
>> >>      at
>> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
>> >> #6  0x00007f1701740a7b in ceph::buffer::list::iterator::copy
>> >> (this=0x7f16fd8b1930, len=4, dest=0x7f16fd8b18dc "") at
>> >> include/buffer.h:379
>> >> #7  0x00007f1701743328 in decode_raw<__le32>  (t=@0x7f16fd8b18dc,
>> p=...)
>> >> at
>> >> include/encoding.h:35
>> >> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
>> >> include/encoding.h:80
>> >> #9  0x00007f1701741ade in decode (s=..., p=...) at
>> >> include/encoding.h:189
>> >> #10 0x00007f17012e8369 in
>> >> librados::RadosClient::C_aio_sparse_read_Ack::finish
>> >> (this=0x7f16f40d6200,
>> >> r=0) at librados.cc:463
>> >> #11 0x00007f170132bb5a in Objecter::handle_osd_op_reply
>> (this=0x13423e0,
>> >> m=0x1346520) at osdc/Objecter.cc:794
>> >> #12 0x00007f17012d1444 in librados::RadosClient::_dispatch
>> >> (this=0x133f810, m=0x1346520) at librados.cc:751
>> >> #13 0x00007f17012d1244 in librados::RadosClient::ms_dispatch
>> >> (this=0x133f810, m=0x1346520) at librados.cc:717
>> >> #14 0x00007f170131b57b in Messenger::ms_deliver_dispatch
>> >> (this=0x1341910,
>> >> m=0x1346520) at msg/Messenger.h:98
>> >> #15 0x00007f17013090d3 in SimpleMessenger::dispatch_entry
>> >> (this=0x1341910)
>> >> at msg/SimpleMessenger.cc:352
>> >> #16 0x00007f17012e296e in SimpleMessenger::DispatchThread::entry
>> >> (this=0x1341da0) at msg/SimpleMessenger.h:533
>> >> #17 0x00007f170131a39b in Thread::_entry_func (arg=0x1341da0) at
>> >> common/Thread.h:41
>> >> #18 0x00007f1701f6dac4 in start_thread (arg=<value optimized out>) at
>> >> pthread_create.c:297
>> >> #19 0x00007f16ff8c838d in clone () at
>> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
>> >
>> > I haven't seen that error before, but it's probably a bug in the OSD
>> > where it doesn't set an error code. If you've still got the core file,
>> > could you go to frame 10 and send us the values of r, bl._len, and
>> > iter.off?
>> >
>> > Thanks,
>> > Josh
>> >
>>
>>
>>
>>
>

Re: [Qemu-devel] Qemu + RBD = ceph::buffer::end_of_buffer

[Sage Weil]

f 9  (or 8?)
p n
p s

(BTW this might be faster over irc, #ceph on irc.oftc.net)

Thanks!
sage


On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:

> Hi Sage/Lists!
> 
> 
> (gdb) print c->bl._len
> $1 = 20
> 
> 
> And in case this is helpful:
> 
> (gdb) print *c
> $2 = {lock = {name = 0x7f1701430f8d "AioCompletionImpl lock", id = -1,
> recursive = false, lockdep = true, backtrace = false, _m = {__data =
> {__lock = 1, __count = 0,
>         __owner = 25800, __nusers = 1, __kind = 2, __spins = 0, __list =
> {__prev = 0x0, __next = 0x0}},
>       __size =
> "\001\000\000\000\000\000\000\000\310d\000\000\001\000\000\000\002",
> '\000' <repeats 22 times>, __align = 1}, nlock = 1}, cond = {
>     _vptr.Cond = 0x7f1701952bd0, _c = {__data = {__lock = 0, __futex = 0,
> __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0,
> __nwaiters = 0,
>         __broadcast_seq = 0}, __size = '\000' <repeats 47 times>, __align
> = 0}}, ref = 1, rval = 0, released = true, ack = true, safe =
> false, objver = {version = 0,
>     epoch = 0, __pad = 0}, callback_complete = 0x7f170173de33
> <librbd::rados_aio_sparse_read_cb(rados_completion_t, void*)>,
>   callback_safe = 0x7f170173d8bd <librbd::rados_cb(rados_completion_t,
> void*)>, callback_arg = 0x7f16f40d6010, bl = {
>     _buffers = {<std::_List_base<ceph::buffer::ptr,
> std::allocator<ceph::buffer::ptr> >> = {
>         _M_impl = {<std::allocator<std::_List_node<ceph::buffer::ptr> >> =
> {<__gnu_cxx::new_allocator<std::_List_node<ceph::buffer::ptr> >> =
> {<No data fields>}, <No data fields>}, _M_node = {_M_next =
> 0x1350530, _M_prev = 0x1350530}}}, <No data fields>}, _len = 20,
> append_buffer = {_raw = 0x0, _off = 0, _len = 0}, last_p = {
>       bl = 0x7f16f40d6170, ls = 0x7f16f40d6170, off = 0, p = {_M_node =
> 0x7f16f40d6170}, p_off = 0}}, pbl = 0x0, buf = 0x0, maxlen = 0}
> 
> 
> 
> Thanks,
> Dyweni
> 
> 
> 
> 
> > On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
> >> Hi Josh/Lists!
> >>
> >> 463             ::decode(*data_bl, iter);
> >> (gdb) print r
> >> $1 = 0
> >> (gdb) print data_bl
> >> $2 = (ceph::bufferlist *) 0x7f16f40d6060
> >> (gdb) print data_bl->_len
> >> $3 = 0
> >
> > What about c->bl._len?
> >
> > sage
> >
> >
> >> (gdb) print iter->off
> >> $4 = 20
> >>
> >>
> >> Thanks,
> >> Dyweni
> >>
> >>
> >>
> >> > CCing the ceph list.
> >> >
> >> > On 05/06/2011 12:23 PM, Dyweni - Qemu-Devel wrote:
> >> >> Hi List!
> >> >>
> >> >> I upgraded Ceph to the latest development version
> >> >> 	Commit: 0edbc75a5fe8c3028faf85546f3264d28653ea3f
> >> >> 	Pulled from: git://ceph.newdream.net/ceph.git
> >> >>
> >> >> I recompiled the latest GIT version of QEMU-KVM (with Josh Durgin's
> >> >> patches) against the latest git version of Ceph.
> >> >>
> >> >> However, this error is still occurring:
> >> >>
> >> >> terminate called after throwing an instance of
> >> >> 'ceph::buffer::end_of_buffer'
> >> >>    what():  buffer::end_of_buffer
> >> >> Aborted (core dumped)
> >> >>
> >> >>
> >> >>
> >> >> Here's another backtrace from GDB:
> >> >>
> >> >> #0  0x00007f16ff829495 in raise (sig=<value optimized out>) at
> >> >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> >> >> #1  0x00007f16ff82a81f in abort () at abort.c:92
> >> >> #2  0x00007f16fed74a25 in __gnu_cxx::__verbose_terminate_handler ()
> >> at
> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
> >> >> #3  0x00007f16fed71c64 in __cxxabiv1::__terminate
> >> >> (handler=0x7f16fed74817
> >> >> <__gnu_cxx::__verbose_terminate_handler()>)
> >> >>      at
> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
> >> >> #4  0x00007f16fed71c8c in std::terminate () at
> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
> >> >> #5  0x00007f16fed71ea4 in __cxxabiv1::__cxa_throw (obj=0x1346470,
> >> >> tinfo=0x7f1701952ce0, dest=0x7f17017403d4
> >> >> <ceph::buffer::end_of_buffer::~end_of_buffer()>)
> >> >>      at
> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
> >> >> #6  0x00007f1701740a7b in ceph::buffer::list::iterator::copy
> >> >> (this=0x7f16fd8b1930, len=4, dest=0x7f16fd8b18dc "") at
> >> >> include/buffer.h:379
> >> >> #7  0x00007f1701743328 in decode_raw<__le32>  (t=@0x7f16fd8b18dc,
> >> p=...)
> >> >> at
> >> >> include/encoding.h:35
> >> >> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
> >> >> include/encoding.h:80
> >> >> #9  0x00007f1701741ade in decode (s=..., p=...) at
> >> >> include/encoding.h:189
> >> >> #10 0x00007f17012e8369 in
> >> >> librados::RadosClient::C_aio_sparse_read_Ack::finish
> >> >> (this=0x7f16f40d6200,
> >> >> r=0) at librados.cc:463
> >> >> #11 0x00007f170132bb5a in Objecter::handle_osd_op_reply
> >> (this=0x13423e0,
> >> >> m=0x1346520) at osdc/Objecter.cc:794
> >> >> #12 0x00007f17012d1444 in librados::RadosClient::_dispatch
> >> >> (this=0x133f810, m=0x1346520) at librados.cc:751
> >> >> #13 0x00007f17012d1244 in librados::RadosClient::ms_dispatch
> >> >> (this=0x133f810, m=0x1346520) at librados.cc:717
> >> >> #14 0x00007f170131b57b in Messenger::ms_deliver_dispatch
> >> >> (this=0x1341910,
> >> >> m=0x1346520) at msg/Messenger.h:98
> >> >> #15 0x00007f17013090d3 in SimpleMessenger::dispatch_entry
> >> >> (this=0x1341910)
> >> >> at msg/SimpleMessenger.cc:352
> >> >> #16 0x00007f17012e296e in SimpleMessenger::DispatchThread::entry
> >> >> (this=0x1341da0) at msg/SimpleMessenger.h:533
> >> >> #17 0x00007f170131a39b in Thread::_entry_func (arg=0x1341da0) at
> >> >> common/Thread.h:41
> >> >> #18 0x00007f1701f6dac4 in start_thread (arg=<value optimized out>) at
> >> >> pthread_create.c:297
> >> >> #19 0x00007f16ff8c838d in clone () at
> >> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> >> >
> >> > I haven't seen that error before, but it's probably a bug in the OSD
> >> > where it doesn't set an error code. If you've still got the core file,
> >> > could you go to frame 10 and send us the values of r, bl._len, and
> >> > iter.off?
> >> >
> >> > Thanks,
> >> > Josh
> >> >
> >>
> >>
> >>
> >>
> >
> 
> 
> 
> 

Re: [Qemu-devel] Qemu + RBD = ceph::buffer::end_of_buffer

[Dyweni - Qemu-Devel]

Hi Sage/Lists!


(gdb) f 8
#8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
include/encoding.h:80
80      WRITE_INTTYPE_ENCODER(uint32_t, le32)
(gdb) p n
No symbol "n" in current context.
(gdb) p s
No symbol "s" in current context.


(gdb) f 9
#9  0x00007f1701741ade in decode (s=..., p=...) at include/encoding.h:189
189       decode(len, p);
(gdb) p n
No symbol "n" in current context.
(gdb) p s
$3 = (ceph::bufferlist &) @0x7f16f40d6060: {_buffers =
{<std::_List_base<ceph::buffer::ptr, std::allocator<ceph::buffer::ptr> >>
= {
      _M_impl = {<std::allocator<std::_List_node<ceph::buffer::ptr> >> =
{<__gnu_cxx::new_allocator<std::_List_node<ceph::buffer::ptr> >> =
{<No data fields>}, <No data fields>}, _M_node = {_M_next =
0x7f16f40d6060, _M_prev = 0x7f16f40d6060}}}, <No data fields>}, _len
= 0, append_buffer = {_raw = 0x0, _off = 0, _len = 0}, last_p = {
    bl = 0x7f16f40d6060, ls = 0x7f16f40d6060, off = 0, p = {_M_node =
0x7f16f40d6060}, p_off = 0}}


Sorry, I don't have access to IRC from where I am at.

Thanks,
Dyweni




> f 9  (or 8?)
> p n
> p s
>
> (BTW this might be faster over irc, #ceph on irc.oftc.net)
>
> Thanks!
> sage
>
>
> On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
>
>> Hi Sage/Lists!
>>
>>
>> (gdb) print c->bl._len
>> $1 = 20
>>
>>
>> And in case this is helpful:
>>
>> (gdb) print *c
>> $2 = {lock = {name = 0x7f1701430f8d "AioCompletionImpl lock", id = -1,
>> recursive = false, lockdep = true, backtrace = false, _m = {__data =
>> {__lock = 1, __count = 0,
>>         __owner = 25800, __nusers = 1, __kind = 2, __spins = 0, __list =
>> {__prev = 0x0, __next = 0x0}},
>>       __size =
>> "\001\000\000\000\000\000\000\000\310d\000\000\001\000\000\000\002",
>> '\000' <repeats 22 times>, __align = 1}, nlock = 1}, cond = {
>>     _vptr.Cond = 0x7f1701952bd0, _c = {__data = {__lock = 0, __futex =
>> 0,
>> __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0,
>> __nwaiters = 0,
>>         __broadcast_seq = 0}, __size = '\000' <repeats 47 times>,
>> __align
>> = 0}}, ref = 1, rval = 0, released = true, ack = true, safe =
>> false, objver = {version = 0,
>>     epoch = 0, __pad = 0}, callback_complete = 0x7f170173de33
>> <librbd::rados_aio_sparse_read_cb(rados_completion_t, void*)>,
>>   callback_safe = 0x7f170173d8bd <librbd::rados_cb(rados_completion_t,
>> void*)>, callback_arg = 0x7f16f40d6010, bl = {
>>     _buffers = {<std::_List_base<ceph::buffer::ptr,
>> std::allocator<ceph::buffer::ptr> >> = {
>>         _M_impl = {<std::allocator<std::_List_node<ceph::buffer::ptr> >>
>> =
>> {<__gnu_cxx::new_allocator<std::_List_node<ceph::buffer::ptr> >> =
>> {<No data fields>}, <No data fields>}, _M_node = {_M_next =
>> 0x1350530, _M_prev = 0x1350530}}}, <No data fields>}, _len = 20,
>> append_buffer = {_raw = 0x0, _off = 0, _len = 0}, last_p = {
>>       bl = 0x7f16f40d6170, ls = 0x7f16f40d6170, off = 0, p = {_M_node =
>> 0x7f16f40d6170}, p_off = 0}}, pbl = 0x0, buf = 0x0, maxlen = 0}
>>
>>
>>
>> Thanks,
>> Dyweni
>>
>>
>>
>>
>> > On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
>> >> Hi Josh/Lists!
>> >>
>> >> 463             ::decode(*data_bl, iter);
>> >> (gdb) print r
>> >> $1 = 0
>> >> (gdb) print data_bl
>> >> $2 = (ceph::bufferlist *) 0x7f16f40d6060
>> >> (gdb) print data_bl->_len
>> >> $3 = 0
>> >
>> > What about c->bl._len?
>> >
>> > sage
>> >
>> >
>> >> (gdb) print iter->off
>> >> $4 = 20
>> >>
>> >>
>> >> Thanks,
>> >> Dyweni
>> >>
>> >>
>> >>
>> >> > CCing the ceph list.
>> >> >
>> >> > On 05/06/2011 12:23 PM, Dyweni - Qemu-Devel wrote:
>> >> >> Hi List!
>> >> >>
>> >> >> I upgraded Ceph to the latest development version
>> >> >> 	Commit: 0edbc75a5fe8c3028faf85546f3264d28653ea3f
>> >> >> 	Pulled from: git://ceph.newdream.net/ceph.git
>> >> >>
>> >> >> I recompiled the latest GIT version of QEMU-KVM (with Josh
>> Durgin's
>> >> >> patches) against the latest git version of Ceph.
>> >> >>
>> >> >> However, this error is still occurring:
>> >> >>
>> >> >> terminate called after throwing an instance of
>> >> >> 'ceph::buffer::end_of_buffer'
>> >> >>    what():  buffer::end_of_buffer
>> >> >> Aborted (core dumped)
>> >> >>
>> >> >>
>> >> >>
>> >> >> Here's another backtrace from GDB:
>> >> >>
>> >> >> #0  0x00007f16ff829495 in raise (sig=<value optimized out>) at
>> >> >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
>> >> >> #1  0x00007f16ff82a81f in abort () at abort.c:92
>> >> >> #2  0x00007f16fed74a25 in __gnu_cxx::__verbose_terminate_handler
>> ()
>> >> at
>> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
>> >> >> #3  0x00007f16fed71c64 in __cxxabiv1::__terminate
>> >> >> (handler=0x7f16fed74817
>> >> >> <__gnu_cxx::__verbose_terminate_handler()>)
>> >> >>      at
>> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
>> >> >> #4  0x00007f16fed71c8c in std::terminate () at
>> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
>> >> >> #5  0x00007f16fed71ea4 in __cxxabiv1::__cxa_throw (obj=0x1346470,
>> >> >> tinfo=0x7f1701952ce0, dest=0x7f17017403d4
>> >> >> <ceph::buffer::end_of_buffer::~end_of_buffer()>)
>> >> >>      at
>> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
>> >> >> #6  0x00007f1701740a7b in ceph::buffer::list::iterator::copy
>> >> >> (this=0x7f16fd8b1930, len=4, dest=0x7f16fd8b18dc "") at
>> >> >> include/buffer.h:379
>> >> >> #7  0x00007f1701743328 in decode_raw<__le32>  (t=@0x7f16fd8b18dc,
>> >> p=...)
>> >> >> at
>> >> >> include/encoding.h:35
>> >> >> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
>> >> >> include/encoding.h:80
>> >> >> #9  0x00007f1701741ade in decode (s=..., p=...) at
>> >> >> include/encoding.h:189
>> >> >> #10 0x00007f17012e8369 in
>> >> >> librados::RadosClient::C_aio_sparse_read_Ack::finish
>> >> >> (this=0x7f16f40d6200,
>> >> >> r=0) at librados.cc:463
>> >> >> #11 0x00007f170132bb5a in Objecter::handle_osd_op_reply
>> >> (this=0x13423e0,
>> >> >> m=0x1346520) at osdc/Objecter.cc:794
>> >> >> #12 0x00007f17012d1444 in librados::RadosClient::_dispatch
>> >> >> (this=0x133f810, m=0x1346520) at librados.cc:751
>> >> >> #13 0x00007f17012d1244 in librados::RadosClient::ms_dispatch
>> >> >> (this=0x133f810, m=0x1346520) at librados.cc:717
>> >> >> #14 0x00007f170131b57b in Messenger::ms_deliver_dispatch
>> >> >> (this=0x1341910,
>> >> >> m=0x1346520) at msg/Messenger.h:98
>> >> >> #15 0x00007f17013090d3 in SimpleMessenger::dispatch_entry
>> >> >> (this=0x1341910)
>> >> >> at msg/SimpleMessenger.cc:352
>> >> >> #16 0x00007f17012e296e in SimpleMessenger::DispatchThread::entry
>> >> >> (this=0x1341da0) at msg/SimpleMessenger.h:533
>> >> >> #17 0x00007f170131a39b in Thread::_entry_func (arg=0x1341da0) at
>> >> >> common/Thread.h:41
>> >> >> #18 0x00007f1701f6dac4 in start_thread (arg=<value optimized out>)
>> at
>> >> >> pthread_create.c:297
>> >> >> #19 0x00007f16ff8c838d in clone () at
>> >> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
>> >> >
>> >> > I haven't seen that error before, but it's probably a bug in the
>> OSD
>> >> > where it doesn't set an error code. If you've still got the core
>> file,
>> >> > could you go to frame 10 and send us the values of r, bl._len, and
>> >> > iter.off?
>> >> >
>> >> > Thanks,
>> >> > Josh
>> >> >
>> >>
>> >>
>> >>
>> >>
>> >
>>
>>
>>
>>
>

Re: [Qemu-devel] Qemu + RBD = ceph::buffer::end_of_buffer

[Sage Weil]

On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
> Hi Sage/Lists!
> 
> 
> (gdb) f 8
> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
> include/encoding.h:80
> 80      WRITE_INTTYPE_ENCODER(uint32_t, le32)
> (gdb) p n
> No symbol "n" in current context.
> (gdb) p s
> No symbol "s" in current context.
> 
> 
> (gdb) f 9
> #9  0x00007f1701741ade in decode (s=..., p=...) at include/encoding.h:189
> 189       decode(len, p);
> (gdb) p n
> No symbol "n" in current context.
> (gdb) p s
> $3 = (ceph::bufferlist &) @0x7f16f40d6060: {_buffers =
> {<std::_List_base<ceph::buffer::ptr, std::allocator<ceph::buffer::ptr> >>
> = {
>       _M_impl = {<std::allocator<std::_List_node<ceph::buffer::ptr> >> =
> {<__gnu_cxx::new_allocator<std::_List_node<ceph::buffer::ptr> >> =
> {<No data fields>}, <No data fields>}, _M_node = {_M_next =
> 0x7f16f40d6060, _M_prev = 0x7f16f40d6060}}}, <No data fields>}, _len
> = 0, append_buffer = {_raw = 0x0, _off = 0, _len = 0}, last_p = {
>     bl = 0x7f16f40d6060, ls = 0x7f16f40d6060, off = 0, p = {_M_node =
> 0x7f16f40d6060}, p_off = 0}}
> 
> 
> Sorry, I don't have access to IRC from where I am at.

No worries.

Are you OSDs, by chance, running on 32bit machines?  This looks like a 
word size encoding thing.

sage


> 
> Thanks,
> Dyweni
> 
> 
> 
> 
> > f 9  (or 8?)
> > p n
> > p s
> >
> > (BTW this might be faster over irc, #ceph on irc.oftc.net)
> >
> > Thanks!
> > sage
> >
> >
> > On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
> >
> >> Hi Sage/Lists!
> >>
> >>
> >> (gdb) print c->bl._len
> >> $1 = 20
> >>
> >>
> >> And in case this is helpful:
> >>
> >> (gdb) print *c
> >> $2 = {lock = {name = 0x7f1701430f8d "AioCompletionImpl lock", id = -1,
> >> recursive = false, lockdep = true, backtrace = false, _m = {__data =
> >> {__lock = 1, __count = 0,
> >>         __owner = 25800, __nusers = 1, __kind = 2, __spins = 0, __list =
> >> {__prev = 0x0, __next = 0x0}},
> >>       __size =
> >> "\001\000\000\000\000\000\000\000\310d\000\000\001\000\000\000\002",
> >> '\000' <repeats 22 times>, __align = 1}, nlock = 1}, cond = {
> >>     _vptr.Cond = 0x7f1701952bd0, _c = {__data = {__lock = 0, __futex =
> >> 0,
> >> __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0,
> >> __nwaiters = 0,
> >>         __broadcast_seq = 0}, __size = '\000' <repeats 47 times>,
> >> __align
> >> = 0}}, ref = 1, rval = 0, released = true, ack = true, safe =
> >> false, objver = {version = 0,
> >>     epoch = 0, __pad = 0}, callback_complete = 0x7f170173de33
> >> <librbd::rados_aio_sparse_read_cb(rados_completion_t, void*)>,
> >>   callback_safe = 0x7f170173d8bd <librbd::rados_cb(rados_completion_t,
> >> void*)>, callback_arg = 0x7f16f40d6010, bl = {
> >>     _buffers = {<std::_List_base<ceph::buffer::ptr,
> >> std::allocator<ceph::buffer::ptr> >> = {
> >>         _M_impl = {<std::allocator<std::_List_node<ceph::buffer::ptr> >>
> >> =
> >> {<__gnu_cxx::new_allocator<std::_List_node<ceph::buffer::ptr> >> =
> >> {<No data fields>}, <No data fields>}, _M_node = {_M_next =
> >> 0x1350530, _M_prev = 0x1350530}}}, <No data fields>}, _len = 20,
> >> append_buffer = {_raw = 0x0, _off = 0, _len = 0}, last_p = {
> >>       bl = 0x7f16f40d6170, ls = 0x7f16f40d6170, off = 0, p = {_M_node =
> >> 0x7f16f40d6170}, p_off = 0}}, pbl = 0x0, buf = 0x0, maxlen = 0}
> >>
> >>
> >>
> >> Thanks,
> >> Dyweni
> >>
> >>
> >>
> >>
> >> > On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
> >> >> Hi Josh/Lists!
> >> >>
> >> >> 463             ::decode(*data_bl, iter);
> >> >> (gdb) print r
> >> >> $1 = 0
> >> >> (gdb) print data_bl
> >> >> $2 = (ceph::bufferlist *) 0x7f16f40d6060
> >> >> (gdb) print data_bl->_len
> >> >> $3 = 0
> >> >
> >> > What about c->bl._len?
> >> >
> >> > sage
> >> >
> >> >
> >> >> (gdb) print iter->off
> >> >> $4 = 20
> >> >>
> >> >>
> >> >> Thanks,
> >> >> Dyweni
> >> >>
> >> >>
> >> >>
> >> >> > CCing the ceph list.
> >> >> >
> >> >> > On 05/06/2011 12:23 PM, Dyweni - Qemu-Devel wrote:
> >> >> >> Hi List!
> >> >> >>
> >> >> >> I upgraded Ceph to the latest development version
> >> >> >> 	Commit: 0edbc75a5fe8c3028faf85546f3264d28653ea3f
> >> >> >> 	Pulled from: git://ceph.newdream.net/ceph.git
> >> >> >>
> >> >> >> I recompiled the latest GIT version of QEMU-KVM (with Josh
> >> Durgin's
> >> >> >> patches) against the latest git version of Ceph.
> >> >> >>
> >> >> >> However, this error is still occurring:
> >> >> >>
> >> >> >> terminate called after throwing an instance of
> >> >> >> 'ceph::buffer::end_of_buffer'
> >> >> >>    what():  buffer::end_of_buffer
> >> >> >> Aborted (core dumped)
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> Here's another backtrace from GDB:
> >> >> >>
> >> >> >> #0  0x00007f16ff829495 in raise (sig=<value optimized out>) at
> >> >> >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> >> >> >> #1  0x00007f16ff82a81f in abort () at abort.c:92
> >> >> >> #2  0x00007f16fed74a25 in __gnu_cxx::__verbose_terminate_handler
> >> ()
> >> >> at
> >> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
> >> >> >> #3  0x00007f16fed71c64 in __cxxabiv1::__terminate
> >> >> >> (handler=0x7f16fed74817
> >> >> >> <__gnu_cxx::__verbose_terminate_handler()>)
> >> >> >>      at
> >> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
> >> >> >> #4  0x00007f16fed71c8c in std::terminate () at
> >> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
> >> >> >> #5  0x00007f16fed71ea4 in __cxxabiv1::__cxa_throw (obj=0x1346470,
> >> >> >> tinfo=0x7f1701952ce0, dest=0x7f17017403d4
> >> >> >> <ceph::buffer::end_of_buffer::~end_of_buffer()>)
> >> >> >>      at
> >> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
> >> >> >> #6  0x00007f1701740a7b in ceph::buffer::list::iterator::copy
> >> >> >> (this=0x7f16fd8b1930, len=4, dest=0x7f16fd8b18dc "") at
> >> >> >> include/buffer.h:379
> >> >> >> #7  0x00007f1701743328 in decode_raw<__le32>  (t=@0x7f16fd8b18dc,
> >> >> p=...)
> >> >> >> at
> >> >> >> include/encoding.h:35
> >> >> >> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
> >> >> >> include/encoding.h:80
> >> >> >> #9  0x00007f1701741ade in decode (s=..., p=...) at
> >> >> >> include/encoding.h:189
> >> >> >> #10 0x00007f17012e8369 in
> >> >> >> librados::RadosClient::C_aio_sparse_read_Ack::finish
> >> >> >> (this=0x7f16f40d6200,
> >> >> >> r=0) at librados.cc:463
> >> >> >> #11 0x00007f170132bb5a in Objecter::handle_osd_op_reply
> >> >> (this=0x13423e0,
> >> >> >> m=0x1346520) at osdc/Objecter.cc:794
> >> >> >> #12 0x00007f17012d1444 in librados::RadosClient::_dispatch
> >> >> >> (this=0x133f810, m=0x1346520) at librados.cc:751
> >> >> >> #13 0x00007f17012d1244 in librados::RadosClient::ms_dispatch
> >> >> >> (this=0x133f810, m=0x1346520) at librados.cc:717
> >> >> >> #14 0x00007f170131b57b in Messenger::ms_deliver_dispatch
> >> >> >> (this=0x1341910,
> >> >> >> m=0x1346520) at msg/Messenger.h:98
> >> >> >> #15 0x00007f17013090d3 in SimpleMessenger::dispatch_entry
> >> >> >> (this=0x1341910)
> >> >> >> at msg/SimpleMessenger.cc:352
> >> >> >> #16 0x00007f17012e296e in SimpleMessenger::DispatchThread::entry
> >> >> >> (this=0x1341da0) at msg/SimpleMessenger.h:533
> >> >> >> #17 0x00007f170131a39b in Thread::_entry_func (arg=0x1341da0) at
> >> >> >> common/Thread.h:41
> >> >> >> #18 0x00007f1701f6dac4 in start_thread (arg=<value optimized out>)
> >> at
> >> >> >> pthread_create.c:297
> >> >> >> #19 0x00007f16ff8c838d in clone () at
> >> >> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
> >> >> >
> >> >> > I haven't seen that error before, but it's probably a bug in the
> >> OSD
> >> >> > where it doesn't set an error code. If you've still got the core
> >> file,
> >> >> > could you go to frame 10 and send us the values of r, bl._len, and
> >> >> > iter.off?
> >> >> >
> >> >> > Thanks,
> >> >> > Josh
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >>
> >> >
> >>
> >>
> >>
> >>
> >
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Qemu + RBD = ceph::buffer::end_of_buffer

[Dyweni - Qemu-Devel]

Hi Sage/Lists!


Yes!  The entire Ceph cluster (1 Mon, 1 MSD, 3 OSD) are 32bit linux.

The machine running Qemu is 64bit linux.


Thanks,
Dyweni



> On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
>> Hi Sage/Lists!
>>
>>
>> (gdb) f 8
>> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
>> include/encoding.h:80
>> 80      WRITE_INTTYPE_ENCODER(uint32_t, le32)
>> (gdb) p n
>> No symbol "n" in current context.
>> (gdb) p s
>> No symbol "s" in current context.
>>
>>
>> (gdb) f 9
>> #9  0x00007f1701741ade in decode (s=..., p=...) at
>> include/encoding.h:189
>> 189       decode(len, p);
>> (gdb) p n
>> No symbol "n" in current context.
>> (gdb) p s
>> $3 = (ceph::bufferlist &) @0x7f16f40d6060: {_buffers =
>> {<std::_List_base<ceph::buffer::ptr, std::allocator<ceph::buffer::ptr>
>> >>
>> = {
>>       _M_impl = {<std::allocator<std::_List_node<ceph::buffer::ptr> >> =
>> {<__gnu_cxx::new_allocator<std::_List_node<ceph::buffer::ptr> >> =
>> {<No data fields>}, <No data fields>}, _M_node = {_M_next =
>> 0x7f16f40d6060, _M_prev = 0x7f16f40d6060}}}, <No data fields>}, _len
>> = 0, append_buffer = {_raw = 0x0, _off = 0, _len = 0}, last_p = {
>>     bl = 0x7f16f40d6060, ls = 0x7f16f40d6060, off = 0, p = {_M_node =
>> 0x7f16f40d6060}, p_off = 0}}
>>
>>
>> Sorry, I don't have access to IRC from where I am at.
>
> No worries.
>
> Are you OSDs, by chance, running on 32bit machines?  This looks like a
> word size encoding thing.
>
> sage
>
>
>>
>> Thanks,
>> Dyweni
>>
>>
>>
>>
>> > f 9  (or 8?)
>> > p n
>> > p s
>> >
>> > (BTW this might be faster over irc, #ceph on irc.oftc.net)
>> >
>> > Thanks!
>> > sage
>> >
>> >
>> > On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
>> >
>> >> Hi Sage/Lists!
>> >>
>> >>
>> >> (gdb) print c->bl._len
>> >> $1 = 20
>> >>
>> >>
>> >> And in case this is helpful:
>> >>
>> >> (gdb) print *c
>> >> $2 = {lock = {name = 0x7f1701430f8d "AioCompletionImpl lock", id =
>> -1,
>> >> recursive = false, lockdep = true, backtrace = false, _m = {__data =
>> >> {__lock = 1, __count = 0,
>> >>         __owner = 25800, __nusers = 1, __kind = 2, __spins = 0,
>> __list =
>> >> {__prev = 0x0, __next = 0x0}},
>> >>       __size =
>> >> "\001\000\000\000\000\000\000\000\310d\000\000\001\000\000\000\002",
>> >> '\000' <repeats 22 times>, __align = 1}, nlock = 1}, cond = {
>> >>     _vptr.Cond = 0x7f1701952bd0, _c = {__data = {__lock = 0, __futex
>> =
>> >> 0,
>> >> __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0,
>> >> __nwaiters = 0,
>> >>         __broadcast_seq = 0}, __size = '\000' <repeats 47 times>,
>> >> __align
>> >> = 0}}, ref = 1, rval = 0, released = true, ack = true, safe =
>> >> false, objver = {version = 0,
>> >>     epoch = 0, __pad = 0}, callback_complete = 0x7f170173de33
>> >> <librbd::rados_aio_sparse_read_cb(rados_completion_t, void*)>,
>> >>   callback_safe = 0x7f170173d8bd
>> <librbd::rados_cb(rados_completion_t,
>> >> void*)>, callback_arg = 0x7f16f40d6010, bl = {
>> >>     _buffers = {<std::_List_base<ceph::buffer::ptr,
>> >> std::allocator<ceph::buffer::ptr> >> = {
>> >>         _M_impl = {<std::allocator<std::_List_node<ceph::buffer::ptr>
>> >>
>> >> =
>> >> {<__gnu_cxx::new_allocator<std::_List_node<ceph::buffer::ptr> >> =
>> >> {<No data fields>}, <No data fields>}, _M_node = {_M_next =
>> >> 0x1350530, _M_prev = 0x1350530}}}, <No data fields>}, _len = 20,
>> >> append_buffer = {_raw = 0x0, _off = 0, _len = 0}, last_p = {
>> >>       bl = 0x7f16f40d6170, ls = 0x7f16f40d6170, off = 0, p = {_M_node
>> =
>> >> 0x7f16f40d6170}, p_off = 0}}, pbl = 0x0, buf = 0x0, maxlen = 0}
>> >>
>> >>
>> >>
>> >> Thanks,
>> >> Dyweni
>> >>
>> >>
>> >>
>> >>
>> >> > On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
>> >> >> Hi Josh/Lists!
>> >> >>
>> >> >> 463             ::decode(*data_bl, iter);
>> >> >> (gdb) print r
>> >> >> $1 = 0
>> >> >> (gdb) print data_bl
>> >> >> $2 = (ceph::bufferlist *) 0x7f16f40d6060
>> >> >> (gdb) print data_bl->_len
>> >> >> $3 = 0
>> >> >
>> >> > What about c->bl._len?
>> >> >
>> >> > sage
>> >> >
>> >> >
>> >> >> (gdb) print iter->off
>> >> >> $4 = 20
>> >> >>
>> >> >>
>> >> >> Thanks,
>> >> >> Dyweni
>> >> >>
>> >> >>
>> >> >>
>> >> >> > CCing the ceph list.
>> >> >> >
>> >> >> > On 05/06/2011 12:23 PM, Dyweni - Qemu-Devel wrote:
>> >> >> >> Hi List!
>> >> >> >>
>> >> >> >> I upgraded Ceph to the latest development version
>> >> >> >> 	Commit: 0edbc75a5fe8c3028faf85546f3264d28653ea3f
>> >> >> >> 	Pulled from: git://ceph.newdream.net/ceph.git
>> >> >> >>
>> >> >> >> I recompiled the latest GIT version of QEMU-KVM (with Josh
>> >> Durgin's
>> >> >> >> patches) against the latest git version of Ceph.
>> >> >> >>
>> >> >> >> However, this error is still occurring:
>> >> >> >>
>> >> >> >> terminate called after throwing an instance of
>> >> >> >> 'ceph::buffer::end_of_buffer'
>> >> >> >>    what():  buffer::end_of_buffer
>> >> >> >> Aborted (core dumped)
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> Here's another backtrace from GDB:
>> >> >> >>
>> >> >> >> #0  0x00007f16ff829495 in raise (sig=<value optimized out>) at
>> >> >> >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
>> >> >> >> #1  0x00007f16ff82a81f in abort () at abort.c:92
>> >> >> >> #2  0x00007f16fed74a25 in
>> __gnu_cxx::__verbose_terminate_handler
>> >> ()
>> >> >> at
>> >> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
>> >> >> >> #3  0x00007f16fed71c64 in __cxxabiv1::__terminate
>> >> >> >> (handler=0x7f16fed74817
>> >> >> >> <__gnu_cxx::__verbose_terminate_handler()>)
>> >> >> >>      at
>> >> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
>> >> >> >> #4  0x00007f16fed71c8c in std::terminate () at
>> >> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
>> >> >> >> #5  0x00007f16fed71ea4 in __cxxabiv1::__cxa_throw
>> (obj=0x1346470,
>> >> >> >> tinfo=0x7f1701952ce0, dest=0x7f17017403d4
>> >> >> >> <ceph::buffer::end_of_buffer::~end_of_buffer()>)
>> >> >> >>      at
>> >> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
>> >> >> >> #6  0x00007f1701740a7b in ceph::buffer::list::iterator::copy
>> >> >> >> (this=0x7f16fd8b1930, len=4, dest=0x7f16fd8b18dc "") at
>> >> >> >> include/buffer.h:379
>> >> >> >> #7  0x00007f1701743328 in decode_raw<__le32>
>> (t=@0x7f16fd8b18dc,
>> >> >> p=...)
>> >> >> >> at
>> >> >> >> include/encoding.h:35
>> >> >> >> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
>> >> >> >> include/encoding.h:80
>> >> >> >> #9  0x00007f1701741ade in decode (s=..., p=...) at
>> >> >> >> include/encoding.h:189
>> >> >> >> #10 0x00007f17012e8369 in
>> >> >> >> librados::RadosClient::C_aio_sparse_read_Ack::finish
>> >> >> >> (this=0x7f16f40d6200,
>> >> >> >> r=0) at librados.cc:463
>> >> >> >> #11 0x00007f170132bb5a in Objecter::handle_osd_op_reply
>> >> >> (this=0x13423e0,
>> >> >> >> m=0x1346520) at osdc/Objecter.cc:794
>> >> >> >> #12 0x00007f17012d1444 in librados::RadosClient::_dispatch
>> >> >> >> (this=0x133f810, m=0x1346520) at librados.cc:751
>> >> >> >> #13 0x00007f17012d1244 in librados::RadosClient::ms_dispatch
>> >> >> >> (this=0x133f810, m=0x1346520) at librados.cc:717
>> >> >> >> #14 0x00007f170131b57b in Messenger::ms_deliver_dispatch
>> >> >> >> (this=0x1341910,
>> >> >> >> m=0x1346520) at msg/Messenger.h:98
>> >> >> >> #15 0x00007f17013090d3 in SimpleMessenger::dispatch_entry
>> >> >> >> (this=0x1341910)
>> >> >> >> at msg/SimpleMessenger.cc:352
>> >> >> >> #16 0x00007f17012e296e in
>> SimpleMessenger::DispatchThread::entry
>> >> >> >> (this=0x1341da0) at msg/SimpleMessenger.h:533
>> >> >> >> #17 0x00007f170131a39b in Thread::_entry_func (arg=0x1341da0)
>> at
>> >> >> >> common/Thread.h:41
>> >> >> >> #18 0x00007f1701f6dac4 in start_thread (arg=<value optimized
>> out>)
>> >> at
>> >> >> >> pthread_create.c:297
>> >> >> >> #19 0x00007f16ff8c838d in clone () at
>> >> >> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
>> >> >> >
>> >> >> > I haven't seen that error before, but it's probably a bug in the
>> >> OSD
>> >> >> > where it doesn't set an error code. If you've still got the core
>> >> file,
>> >> >> > could you go to frame 10 and send us the values of r, bl._len,
>> and
>> >> >> > iter.off?
>> >> >> >
>> >> >> > Thanks,
>> >> >> > Josh
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >
>> >>
>> >>
>> >>
>> >>
>> >
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>

Re: [Qemu-devel] Qemu + RBD = ceph::buffer::end_of_buffer

[Dyweni - Qemu-Devel]

Hi Sage/Lists!


Yes!  The entire Ceph cluster (1 Mon, 1 MSD, 3 OSD) are 32bit linux.

The machine running Qemu is 64bit linux.


Thanks,
Dyweni



> On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
>> Hi Sage/Lists!
>>
>>
>> (gdb) f 8
>> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
>> include/encoding.h:80
>> 80      WRITE_INTTYPE_ENCODER(uint32_t, le32)
>> (gdb) p n
>> No symbol "n" in current context.
>> (gdb) p s
>> No symbol "s" in current context.
>>
>>
>> (gdb) f 9
>> #9  0x00007f1701741ade in decode (s=..., p=...) at
>> include/encoding.h:189
>> 189       decode(len, p);
>> (gdb) p n
>> No symbol "n" in current context.
>> (gdb) p s
>> $3 = (ceph::bufferlist &) @0x7f16f40d6060: {_buffers =
>> {<std::_List_base<ceph::buffer::ptr, std::allocator<ceph::buffer::ptr>
>> >>
>> = {
>>       _M_impl = {<std::allocator<std::_List_node<ceph::buffer::ptr> >> =
>> {<__gnu_cxx::new_allocator<std::_List_node<ceph::buffer::ptr> >> =
>> {<No data fields>}, <No data fields>}, _M_node = {_M_next =
>> 0x7f16f40d6060, _M_prev = 0x7f16f40d6060}}}, <No data fields>}, _len
>> = 0, append_buffer = {_raw = 0x0, _off = 0, _len = 0}, last_p = {
>>     bl = 0x7f16f40d6060, ls = 0x7f16f40d6060, off = 0, p = {_M_node =
>> 0x7f16f40d6060}, p_off = 0}}
>>
>>
>> Sorry, I don't have access to IRC from where I am at.
>
> No worries.
>
> Are you OSDs, by chance, running on 32bit machines?  This looks like a
> word size encoding thing.
>
> sage
>
>
>>
>> Thanks,
>> Dyweni
>>
>>
>>
>>
>> > f 9  (or 8?)
>> > p n
>> > p s
>> >
>> > (BTW this might be faster over irc, #ceph on irc.oftc.net)
>> >
>> > Thanks!
>> > sage
>> >
>> >
>> > On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
>> >
>> >> Hi Sage/Lists!
>> >>
>> >>
>> >> (gdb) print c->bl._len
>> >> $1 = 20
>> >>
>> >>
>> >> And in case this is helpful:
>> >>
>> >> (gdb) print *c
>> >> $2 = {lock = {name = 0x7f1701430f8d "AioCompletionImpl lock", id =
>> -1,
>> >> recursive = false, lockdep = true, backtrace = false, _m = {__data =
>> >> {__lock = 1, __count = 0,
>> >>         __owner = 25800, __nusers = 1, __kind = 2, __spins = 0,
>> __list =
>> >> {__prev = 0x0, __next = 0x0}},
>> >>       __size =
>> >> "\001\000\000\000\000\000\000\000\310d\000\000\001\000\000\000\002",
>> >> '\000' <repeats 22 times>, __align = 1}, nlock = 1}, cond = {
>> >>     _vptr.Cond = 0x7f1701952bd0, _c = {__data = {__lock = 0, __futex
>> =
>> >> 0,
>> >> __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0,
>> >> __nwaiters = 0,
>> >>         __broadcast_seq = 0}, __size = '\000' <repeats 47 times>,
>> >> __align
>> >> = 0}}, ref = 1, rval = 0, released = true, ack = true, safe =
>> >> false, objver = {version = 0,
>> >>     epoch = 0, __pad = 0}, callback_complete = 0x7f170173de33
>> >> <librbd::rados_aio_sparse_read_cb(rados_completion_t, void*)>,
>> >>   callback_safe = 0x7f170173d8bd
>> <librbd::rados_cb(rados_completion_t,
>> >> void*)>, callback_arg = 0x7f16f40d6010, bl = {
>> >>     _buffers = {<std::_List_base<ceph::buffer::ptr,
>> >> std::allocator<ceph::buffer::ptr> >> = {
>> >>         _M_impl = {<std::allocator<std::_List_node<ceph::buffer::ptr>
>> >>
>> >> =
>> >> {<__gnu_cxx::new_allocator<std::_List_node<ceph::buffer::ptr> >> =
>> >> {<No data fields>}, <No data fields>}, _M_node = {_M_next =
>> >> 0x1350530, _M_prev = 0x1350530}}}, <No data fields>}, _len = 20,
>> >> append_buffer = {_raw = 0x0, _off = 0, _len = 0}, last_p = {
>> >>       bl = 0x7f16f40d6170, ls = 0x7f16f40d6170, off = 0, p = {_M_node
>> =
>> >> 0x7f16f40d6170}, p_off = 0}}, pbl = 0x0, buf = 0x0, maxlen = 0}
>> >>
>> >>
>> >>
>> >> Thanks,
>> >> Dyweni
>> >>
>> >>
>> >>
>> >>
>> >> > On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
>> >> >> Hi Josh/Lists!
>> >> >>
>> >> >> 463             ::decode(*data_bl, iter);
>> >> >> (gdb) print r
>> >> >> $1 = 0
>> >> >> (gdb) print data_bl
>> >> >> $2 = (ceph::bufferlist *) 0x7f16f40d6060
>> >> >> (gdb) print data_bl->_len
>> >> >> $3 = 0
>> >> >
>> >> > What about c->bl._len?
>> >> >
>> >> > sage
>> >> >
>> >> >
>> >> >> (gdb) print iter->off
>> >> >> $4 = 20
>> >> >>
>> >> >>
>> >> >> Thanks,
>> >> >> Dyweni
>> >> >>
>> >> >>
>> >> >>
>> >> >> > CCing the ceph list.
>> >> >> >
>> >> >> > On 05/06/2011 12:23 PM, Dyweni - Qemu-Devel wrote:
>> >> >> >> Hi List!
>> >> >> >>
>> >> >> >> I upgraded Ceph to the latest development version
>> >> >> >> 	Commit: 0edbc75a5fe8c3028faf85546f3264d28653ea3f
>> >> >> >> 	Pulled from: git://ceph.newdream.net/ceph.git
>> >> >> >>
>> >> >> >> I recompiled the latest GIT version of QEMU-KVM (with Josh
>> >> Durgin's
>> >> >> >> patches) against the latest git version of Ceph.
>> >> >> >>
>> >> >> >> However, this error is still occurring:
>> >> >> >>
>> >> >> >> terminate called after throwing an instance of
>> >> >> >> 'ceph::buffer::end_of_buffer'
>> >> >> >>    what():  buffer::end_of_buffer
>> >> >> >> Aborted (core dumped)
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> Here's another backtrace from GDB:
>> >> >> >>
>> >> >> >> #0  0x00007f16ff829495 in raise (sig=<value optimized out>) at
>> >> >> >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
>> >> >> >> #1  0x00007f16ff82a81f in abort () at abort.c:92
>> >> >> >> #2  0x00007f16fed74a25 in
>> __gnu_cxx::__verbose_terminate_handler
>> >> ()
>> >> >> at
>> >> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
>> >> >> >> #3  0x00007f16fed71c64 in __cxxabiv1::__terminate
>> >> >> >> (handler=0x7f16fed74817
>> >> >> >> <__gnu_cxx::__verbose_terminate_handler()>)
>> >> >> >>      at
>> >> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
>> >> >> >> #4  0x00007f16fed71c8c in std::terminate () at
>> >> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
>> >> >> >> #5  0x00007f16fed71ea4 in __cxxabiv1::__cxa_throw
>> (obj=0x1346470,
>> >> >> >> tinfo=0x7f1701952ce0, dest=0x7f17017403d4
>> >> >> >> <ceph::buffer::end_of_buffer::~end_of_buffer()>)
>> >> >> >>      at
>> >> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
>> >> >> >> #6  0x00007f1701740a7b in ceph::buffer::list::iterator::copy
>> >> >> >> (this=0x7f16fd8b1930, len=4, dest=0x7f16fd8b18dc "") at
>> >> >> >> include/buffer.h:379
>> >> >> >> #7  0x00007f1701743328 in decode_raw<__le32>
>> (t=@0x7f16fd8b18dc,
>> >> >> p=...)
>> >> >> >> at
>> >> >> >> include/encoding.h:35
>> >> >> >> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
>> >> >> >> include/encoding.h:80
>> >> >> >> #9  0x00007f1701741ade in decode (s=..., p=...) at
>> >> >> >> include/encoding.h:189
>> >> >> >> #10 0x00007f17012e8369 in
>> >> >> >> librados::RadosClient::C_aio_sparse_read_Ack::finish
>> >> >> >> (this=0x7f16f40d6200,
>> >> >> >> r=0) at librados.cc:463
>> >> >> >> #11 0x00007f170132bb5a in Objecter::handle_osd_op_reply
>> >> >> (this=0x13423e0,
>> >> >> >> m=0x1346520) at osdc/Objecter.cc:794
>> >> >> >> #12 0x00007f17012d1444 in librados::RadosClient::_dispatch
>> >> >> >> (this=0x133f810, m=0x1346520) at librados.cc:751
>> >> >> >> #13 0x00007f17012d1244 in librados::RadosClient::ms_dispatch
>> >> >> >> (this=0x133f810, m=0x1346520) at librados.cc:717
>> >> >> >> #14 0x00007f170131b57b in Messenger::ms_deliver_dispatch
>> >> >> >> (this=0x1341910,
>> >> >> >> m=0x1346520) at msg/Messenger.h:98
>> >> >> >> #15 0x00007f17013090d3 in SimpleMessenger::dispatch_entry
>> >> >> >> (this=0x1341910)
>> >> >> >> at msg/SimpleMessenger.cc:352
>> >> >> >> #16 0x00007f17012e296e in
>> SimpleMessenger::DispatchThread::entry
>> >> >> >> (this=0x1341da0) at msg/SimpleMessenger.h:533
>> >> >> >> #17 0x00007f170131a39b in Thread::_entry_func (arg=0x1341da0)
>> at
>> >> >> >> common/Thread.h:41
>> >> >> >> #18 0x00007f1701f6dac4 in start_thread (arg=<value optimized
>> out>)
>> >> at
>> >> >> >> pthread_create.c:297
>> >> >> >> #19 0x00007f16ff8c838d in clone () at
>> >> >> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
>> >> >> >
>> >> >> > I haven't seen that error before, but it's probably a bug in the
>> >> OSD
>> >> >> > where it doesn't set an error code. If you've still got the core
>> >> file,
>> >> >> > could you go to frame 10 and send us the values of r, bl._len,
>> and
>> >> >> > iter.off?
>> >> >> >
>> >> >> > Thanks,
>> >> >> > Josh
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >
>> >>
>> >>
>> >>
>> >>
>> >
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>

Re: [Qemu-devel] Qemu + RBD = ceph::buffer::end_of_buffer

[Sage Weil]

On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
> Hi Sage/Lists!
> 
> Yes!  The entire Ceph cluster (1 Mon, 1 MSD, 3 OSD) are 32bit linux.
> 
> The machine running Qemu is 64bit linux.

Excellent.  This is now fixed by 48d94f6e34da8ace2b5cb128de1bcfb34b3c40b5 
in the stable and master branches of ceph.git.  Or you can apply the patch 
below.

Thanks!
sage



From 48d94f6e34da8ace2b5cb128de1bcfb34b3c40b5 Mon Sep 17 00:00:00 2001
From: Sage Weil <sage@newdream.net>
Date: Fri, 6 May 2011 13:42:23 -0700
Subject: [PATCH] osd: used fixed size types for fiemap/mapext/sparseread encoding

The client expects <uint64_t,uint64_t>, so this breaks on any 32-bit osd.

Signed-off-by: Sage Weil <sage@newdream.net>
---
 src/os/FileStore.cc     |    4 ++--
 src/osd/ReplicatedPG.cc |    4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/os/FileStore.cc b/src/os/FileStore.cc
index 81bc91d..ca11435 100644
--- a/src/os/FileStore.cc
+++ b/src/os/FileStore.cc
@@ -2198,7 +2198,7 @@ int FileStore::fiemap(coll_t cid, const sobject_t& oid,
 {
 
   if (!ioctl_fiemap) {
-    map<off_t, size_t> m;
+    map<uint64_t, uint64_t> m;
     m[offset] = len;
     ::encode(m, bl);
     return 0;
@@ -2207,7 +2207,7 @@ int FileStore::fiemap(coll_t cid, const sobject_t& oid,
 
   char fn[PATH_MAX];
   struct fiemap *fiemap = NULL;
-  map<off_t, size_t> extmap;
+  map<uint64_t, uint64_t> extmap;
 
   get_coname(cid, oid, fn, sizeof(fn));
 
diff --git a/src/osd/ReplicatedPG.cc b/src/osd/ReplicatedPG.cc
index ba76b05..84399ca 100644
--- a/src/osd/ReplicatedPG.cc
+++ b/src/osd/ReplicatedPG.cc
@@ -1142,10 +1142,10 @@ int ReplicatedPG::do_osd_ops(OpContext *ctx, vector<OSDOp>& ops,
 	  result = r;
           break;
 	}
-        map<off_t, size_t> m;
+        map<uint64_t, uint64_t> m;
         bufferlist::iterator iter = bl.begin();
         ::decode(m, iter);
-        map<off_t, size_t>::iterator miter;
+        map<uint64_t, uint64_t>::iterator miter;
         bufferlist data_bl;
         for (miter = m.begin(); miter != m.end(); ++miter) {
           bufferlist tmpbl;
-- 
1.7.0

Re: [Qemu-devel] Qemu + RBD = ceph::buffer::end_of_buffer

[Sage Weil]

On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
> Hi Sage/Lists!
> 
> Yes!  The entire Ceph cluster (1 Mon, 1 MSD, 3 OSD) are 32bit linux.
> 
> The machine running Qemu is 64bit linux.

Excellent.  This is now fixed by 48d94f6e34da8ace2b5cb128de1bcfb34b3c40b5 
in the stable and master branches of ceph.git.  Or you can apply the patch 
below.

Thanks!
sage



>From 48d94f6e34da8ace2b5cb128de1bcfb34b3c40b5 Mon Sep 17 00:00:00 2001
From: Sage Weil <sage@newdream.net>
Date: Fri, 6 May 2011 13:42:23 -0700
Subject: [PATCH] osd: used fixed size types for fiemap/mapext/sparseread encoding

The client expects <uint64_t,uint64_t>, so this breaks on any 32-bit osd.

Signed-off-by: Sage Weil <sage@newdream.net>
---
 src/os/FileStore.cc     |    4 ++--
 src/osd/ReplicatedPG.cc |    4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/os/FileStore.cc b/src/os/FileStore.cc
index 81bc91d..ca11435 100644
--- a/src/os/FileStore.cc
+++ b/src/os/FileStore.cc
@@ -2198,7 +2198,7 @@ int FileStore::fiemap(coll_t cid, const sobject_t& oid,
 {
 
   if (!ioctl_fiemap) {
-    map<off_t, size_t> m;
+    map<uint64_t, uint64_t> m;
     m[offset] = len;
     ::encode(m, bl);
     return 0;
@@ -2207,7 +2207,7 @@ int FileStore::fiemap(coll_t cid, const sobject_t& oid,
 
   char fn[PATH_MAX];
   struct fiemap *fiemap = NULL;
-  map<off_t, size_t> extmap;
+  map<uint64_t, uint64_t> extmap;
 
   get_coname(cid, oid, fn, sizeof(fn));
 
diff --git a/src/osd/ReplicatedPG.cc b/src/osd/ReplicatedPG.cc
index ba76b05..84399ca 100644
--- a/src/osd/ReplicatedPG.cc
+++ b/src/osd/ReplicatedPG.cc
@@ -1142,10 +1142,10 @@ int ReplicatedPG::do_osd_ops(OpContext *ctx, vector<OSDOp>& ops,
 	  result = r;
           break;
 	}
-        map<off_t, size_t> m;
+        map<uint64_t, uint64_t> m;
         bufferlist::iterator iter = bl.begin();
         ::decode(m, iter);
-        map<off_t, size_t>::iterator miter;
+        map<uint64_t, uint64_t>::iterator miter;
         bufferlist data_bl;
         for (miter = m.begin(); miter != m.end(); ++miter) {
           bufferlist tmpbl;
-- 
1.7.0