Re: Qemu + RBD = ceph::buffer::end_of_buffer

["Dyweni - Qemu-Devel" <8SScqsnYX8tr@dyweni.com>]

Hi Sage/Lists!


Yes!  The entire Ceph cluster (1 Mon, 1 MSD, 3 OSD) are 32bit linux.

The machine running Qemu is 64bit linux.


Thanks,
Dyweni



> On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
>> Hi Sage/Lists!
>>
>>
>> (gdb) f 8
>> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
>> include/encoding.h:80
>> 80      WRITE_INTTYPE_ENCODER(uint32_t, le32)
>> (gdb) p n
>> No symbol "n" in current context.
>> (gdb) p s
>> No symbol "s" in current context.
>>
>>
>> (gdb) f 9
>> #9  0x00007f1701741ade in decode (s=..., p=...) at
>> include/encoding.h:189
>> 189       decode(len, p);
>> (gdb) p n
>> No symbol "n" in current context.
>> (gdb) p s
>> $3 = (ceph::bufferlist &) @0x7f16f40d6060: {_buffers =
>> {<std::_List_base<ceph::buffer::ptr, std::allocator<ceph::buffer::ptr>
>> >>
>> = {
>>       _M_impl = {<std::allocator<std::_List_node<ceph::buffer::ptr> >> =
>> {<__gnu_cxx::new_allocator<std::_List_node<ceph::buffer::ptr> >> =
>> {<No data fields>}, <No data fields>}, _M_node = {_M_next =
>> 0x7f16f40d6060, _M_prev = 0x7f16f40d6060}}}, <No data fields>}, _len
>> = 0, append_buffer = {_raw = 0x0, _off = 0, _len = 0}, last_p = {
>>     bl = 0x7f16f40d6060, ls = 0x7f16f40d6060, off = 0, p = {_M_node =
>> 0x7f16f40d6060}, p_off = 0}}
>>
>>
>> Sorry, I don't have access to IRC from where I am at.
>
> No worries.
>
> Are you OSDs, by chance, running on 32bit machines?  This looks like a
> word size encoding thing.
>
> sage
>
>
>>
>> Thanks,
>> Dyweni
>>
>>
>>
>>
>> > f 9  (or 8?)
>> > p n
>> > p s
>> >
>> > (BTW this might be faster over irc, #ceph on irc.oftc.net)
>> >
>> > Thanks!
>> > sage
>> >
>> >
>> > On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
>> >
>> >> Hi Sage/Lists!
>> >>
>> >>
>> >> (gdb) print c->bl._len
>> >> $1 = 20
>> >>
>> >>
>> >> And in case this is helpful:
>> >>
>> >> (gdb) print *c
>> >> $2 = {lock = {name = 0x7f1701430f8d "AioCompletionImpl lock", id =
>> -1,
>> >> recursive = false, lockdep = true, backtrace = false, _m = {__data =
>> >> {__lock = 1, __count = 0,
>> >>         __owner = 25800, __nusers = 1, __kind = 2, __spins = 0,
>> __list =
>> >> {__prev = 0x0, __next = 0x0}},
>> >>       __size =
>> >> "\001\000\000\000\000\000\000\000\310d\000\000\001\000\000\000\002",
>> >> '\000' <repeats 22 times>, __align = 1}, nlock = 1}, cond = {
>> >>     _vptr.Cond = 0x7f1701952bd0, _c = {__data = {__lock = 0, __futex
>> =
>> >> 0,
>> >> __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0,
>> >> __nwaiters = 0,
>> >>         __broadcast_seq = 0}, __size = '\000' <repeats 47 times>,
>> >> __align
>> >> = 0}}, ref = 1, rval = 0, released = true, ack = true, safe =
>> >> false, objver = {version = 0,
>> >>     epoch = 0, __pad = 0}, callback_complete = 0x7f170173de33
>> >> <librbd::rados_aio_sparse_read_cb(rados_completion_t, void*)>,
>> >>   callback_safe = 0x7f170173d8bd
>> <librbd::rados_cb(rados_completion_t,
>> >> void*)>, callback_arg = 0x7f16f40d6010, bl = {
>> >>     _buffers = {<std::_List_base<ceph::buffer::ptr,
>> >> std::allocator<ceph::buffer::ptr> >> = {
>> >>         _M_impl = {<std::allocator<std::_List_node<ceph::buffer::ptr>
>> >>
>> >> =
>> >> {<__gnu_cxx::new_allocator<std::_List_node<ceph::buffer::ptr> >> =
>> >> {<No data fields>}, <No data fields>}, _M_node = {_M_next =
>> >> 0x1350530, _M_prev = 0x1350530}}}, <No data fields>}, _len = 20,
>> >> append_buffer = {_raw = 0x0, _off = 0, _len = 0}, last_p = {
>> >>       bl = 0x7f16f40d6170, ls = 0x7f16f40d6170, off = 0, p = {_M_node
>> =
>> >> 0x7f16f40d6170}, p_off = 0}}, pbl = 0x0, buf = 0x0, maxlen = 0}
>> >>
>> >>
>> >>
>> >> Thanks,
>> >> Dyweni
>> >>
>> >>
>> >>
>> >>
>> >> > On Fri, 6 May 2011, Dyweni - Qemu-Devel wrote:
>> >> >> Hi Josh/Lists!
>> >> >>
>> >> >> 463             ::decode(*data_bl, iter);
>> >> >> (gdb) print r
>> >> >> $1 = 0
>> >> >> (gdb) print data_bl
>> >> >> $2 = (ceph::bufferlist *) 0x7f16f40d6060
>> >> >> (gdb) print data_bl->_len
>> >> >> $3 = 0
>> >> >
>> >> > What about c->bl._len?
>> >> >
>> >> > sage
>> >> >
>> >> >
>> >> >> (gdb) print iter->off
>> >> >> $4 = 20
>> >> >>
>> >> >>
>> >> >> Thanks,
>> >> >> Dyweni
>> >> >>
>> >> >>
>> >> >>
>> >> >> > CCing the ceph list.
>> >> >> >
>> >> >> > On 05/06/2011 12:23 PM, Dyweni - Qemu-Devel wrote:
>> >> >> >> Hi List!
>> >> >> >>
>> >> >> >> I upgraded Ceph to the latest development version
>> >> >> >> 	Commit: 0edbc75a5fe8c3028faf85546f3264d28653ea3f
>> >> >> >> 	Pulled from: git://ceph.newdream.net/ceph.git
>> >> >> >>
>> >> >> >> I recompiled the latest GIT version of QEMU-KVM (with Josh
>> >> Durgin's
>> >> >> >> patches) against the latest git version of Ceph.
>> >> >> >>
>> >> >> >> However, this error is still occurring:
>> >> >> >>
>> >> >> >> terminate called after throwing an instance of
>> >> >> >> 'ceph::buffer::end_of_buffer'
>> >> >> >>    what():  buffer::end_of_buffer
>> >> >> >> Aborted (core dumped)
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> Here's another backtrace from GDB:
>> >> >> >>
>> >> >> >> #0  0x00007f16ff829495 in raise (sig=<value optimized out>) at
>> >> >> >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
>> >> >> >> #1  0x00007f16ff82a81f in abort () at abort.c:92
>> >> >> >> #2  0x00007f16fed74a25 in
>> __gnu_cxx::__verbose_terminate_handler
>> >> ()
>> >> >> at
>> >> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/vterminate.cc:93
>> >> >> >> #3  0x00007f16fed71c64 in __cxxabiv1::__terminate
>> >> >> >> (handler=0x7f16fed74817
>> >> >> >> <__gnu_cxx::__verbose_terminate_handler()>)
>> >> >> >>      at
>> >> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:38
>> >> >> >> #4  0x00007f16fed71c8c in std::terminate () at
>> >> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_terminate.cc:48
>> >> >> >> #5  0x00007f16fed71ea4 in __cxxabiv1::__cxa_throw
>> (obj=0x1346470,
>> >> >> >> tinfo=0x7f1701952ce0, dest=0x7f17017403d4
>> >> >> >> <ceph::buffer::end_of_buffer::~end_of_buffer()>)
>> >> >> >>      at
>> >> >> >> /usr/src/debug/sys-devel/gcc-4.4.5/gcc-4.4.5/libstdc++-v3/libsupc++/eh_throw.cc:83
>> >> >> >> #6  0x00007f1701740a7b in ceph::buffer::list::iterator::copy
>> >> >> >> (this=0x7f16fd8b1930, len=4, dest=0x7f16fd8b18dc "") at
>> >> >> >> include/buffer.h:379
>> >> >> >> #7  0x00007f1701743328 in decode_raw<__le32>
>> (t=@0x7f16fd8b18dc,
>> >> >> p=...)
>> >> >> >> at
>> >> >> >> include/encoding.h:35
>> >> >> >> #8  0x00007f170174198a in decode (v=@0x7f16fd8b190c, p=...) at
>> >> >> >> include/encoding.h:80
>> >> >> >> #9  0x00007f1701741ade in decode (s=..., p=...) at
>> >> >> >> include/encoding.h:189
>> >> >> >> #10 0x00007f17012e8369 in
>> >> >> >> librados::RadosClient::C_aio_sparse_read_Ack::finish
>> >> >> >> (this=0x7f16f40d6200,
>> >> >> >> r=0) at librados.cc:463
>> >> >> >> #11 0x00007f170132bb5a in Objecter::handle_osd_op_reply
>> >> >> (this=0x13423e0,
>> >> >> >> m=0x1346520) at osdc/Objecter.cc:794
>> >> >> >> #12 0x00007f17012d1444 in librados::RadosClient::_dispatch
>> >> >> >> (this=0x133f810, m=0x1346520) at librados.cc:751
>> >> >> >> #13 0x00007f17012d1244 in librados::RadosClient::ms_dispatch
>> >> >> >> (this=0x133f810, m=0x1346520) at librados.cc:717
>> >> >> >> #14 0x00007f170131b57b in Messenger::ms_deliver_dispatch
>> >> >> >> (this=0x1341910,
>> >> >> >> m=0x1346520) at msg/Messenger.h:98
>> >> >> >> #15 0x00007f17013090d3 in SimpleMessenger::dispatch_entry
>> >> >> >> (this=0x1341910)
>> >> >> >> at msg/SimpleMessenger.cc:352
>> >> >> >> #16 0x00007f17012e296e in
>> SimpleMessenger::DispatchThread::entry
>> >> >> >> (this=0x1341da0) at msg/SimpleMessenger.h:533
>> >> >> >> #17 0x00007f170131a39b in Thread::_entry_func (arg=0x1341da0)
>> at
>> >> >> >> common/Thread.h:41
>> >> >> >> #18 0x00007f1701f6dac4 in start_thread (arg=<value optimized
>> out>)
>> >> at
>> >> >> >> pthread_create.c:297
>> >> >> >> #19 0x00007f16ff8c838d in clone () at
>> >> >> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
>> >> >> >
>> >> >> > I haven't seen that error before, but it's probably a bug in the
>> >> OSD
>> >> >> > where it doesn't set an error code. If you've still got the core
>> >> file,
>> >> >> > could you go to frame 10 and send us the values of r, bl._len,
>> and
>> >> >> > iter.off?
>> >> >> >
>> >> >> > Thanks,
>> >> >> > Josh
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >
>> >>
>> >>
>> >>
>> >>
>> >
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>