* Bluestore StoreTest.Synthetic is broken
@ 2016-08-11 13:25 Igor Fedotov
2016-08-11 13:34 ` Igor Fedotov
0 siblings, 1 reply; 3+ messages in thread
From: Igor Fedotov @ 2016-08-11 13:25 UTC (permalink / raw)
To: ceph-devel
Hi All!
I'm observing following crash when running Bluestore StoreTest.Synthetic
test casr( ceph_test_objectstore
--gtest_filter=ObjectStore/StoreTest.Synthetic/2 ):
-20> 2016-08-11 16:09:30.155436 7f6c1437b680 15
bluestore(store_test_temp_dir) collection_list 555.0_head start GHMIN
end GHMAX max 10
-19> 2016-08-11 16:09:30.155443 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list range
'--'0x7ffffffffffffdd300000000'.' to '--'0x7ff
ffffffffffdd3ffffffff':' and '--'0x800000000000022b00000000'.' to
'--'0x800000000000022bffffffff':' start GHMIN
-18> 2016-08-11 16:09:30.155482 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list pend
'--'0x7ffffffffffffdd3ffffffff':'
-17> 2016-08-11 16:09:30.155509 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b00000000'.!=OBJ_1049aaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xff
fffffffffffffe0000000000000000 > GHMAX
-16> 2016-08-11 16:09:30.155515 7f6c1437b680 30
bluestore(store_test_temp_dir) collection_list switch to non-temp namespace
-15> 2016-08-11 16:09:30.155521 7f6c1437b680 30
bluestore(store_test_temp_dir) collection_list pend
'--'0x800000000000022bffffffff':'
-14> 2016-08-11 16:09:30.155524 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b00000000'.!=OBJ_1049aaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xff
fffffffffffffe0000000000000000
-13> 2016-08-11 16:09:30.155536 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b00000000'.!=OBJ_1105aaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x00
0000005df155b0ffffffffffffffff
-12> 2016-08-11 16:09:30.155544 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b00000000'.!=OBJ_255aaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
000007e448de90000000000000000
-11> 2016-08-11 16:09:30.155555 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b00000000'.!=OBJ_511aaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xfff
ffffffffffffe0000000000000001
-10> 2016-08-11 16:09:30.155563 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b00000000'.!=OBJ_767aaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
000005cab38c60000000000000001
-9> 2016-08-11 16:09:30.155574 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b01000000'.!=OBJ_127aaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
0000026f324ba0000000000000000
-8> 2016-08-11 16:09:30.155584 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b01000000'.!=OBJ_639aaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
000007365f1ee0000000000000001
-7> 2016-08-11 16:09:30.155592 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b02000000'.!=OBJ_575aaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
00000060a14630000000000000000
-6> 2016-08-11 16:09:30.155601 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b02000000'.!=OBJ_831aaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xfff
ffffffffffffe0000000000000000
-5> 2016-08-11 16:09:30.155612 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b03000000 (bnode, skipping)
-4> 2016-08-11 16:09:30.155617 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b03000000'.!=OBJ_1013aaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xff
fffffffffffffe0000000000000002
-3> 2016-08-11 16:09:30.155626 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list key
'--'0x800000000000022b03000000'.!=OBJ_1243aaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x00
0000000c94deb00000000000000000
-2> 2016-08-11 16:09:30.155631 7f6c1437b680 20
bluestore(store_test_temp_dir) collection_list reached max 10
-1> 2016-08-11 16:09:30.155636 7f6c1437b680 20
bluestore.2QCache(0xa282410) trim onodes 500 / 500 buffers 1978368 / 2000000
0> 2016-08-11 16:09:56.642872 7f6c1437b680 -1 *** Caught signal
(Segmentation fault) **
in thread 7f6c1437b680 thread_name:ceph_test_objec
ceph version v11.0.0-1434-gc6e561a
(c6e561a5385105dfbd70f642bcd5732fac0bca89)
1: bin/ceph_test_objectstore() [0xa1f207]
2: (()+0xfc90) [0x7f6c13f7bc90]
3: bin/ceph_test_objectstore() [0x8cadab]
4: (operator<<(std::ostream&, hobject_t const&)+0x229) [0x8d1709]
5: (operator<<(std::ostream&, ghobject_t const&)+0x107) [0x8d1c37]
6:
(BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&,
ghobject_t, ghobject_t, bool, int, std::vector<ghobject_t, std::al
locator<ghobject_t> >*, ghobject_t*)+0xb6d) [0x78680d]
7: (BlueStore::collection_list(coll_t const&, ghobject_t, ghobject_t,
bool, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*,
ghobject_t*)+
0x1eb) [0x79d64b]
8: (SyntheticWorkloadState::shutdown()+0x2ab) [0x68b88b]
9: (doSyntheticTest(boost::scoped_ptr<ObjectStore>&, unsigned long,
unsigned long, unsigned long)+0x471) [0x651741]
It looks like it was introduced by the recent PR #10410 merge
(https://github.com/ceph/ceph/pull/10410). I don't observe the issue
prior to this patch.
Brief analysis under GDB shows that ghobject_t instance pointed by pnext
passed to operator<< at Bluestore.cc:4167 is broken - oid.name field has
huge size..
The issue is 100% reproducible.
Thanks,
Igor
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Bluestore StoreTest.Synthetic is broken
2016-08-11 13:25 Bluestore StoreTest.Synthetic is broken Igor Fedotov
@ 2016-08-11 13:34 ` Igor Fedotov
2016-08-11 14:18 ` Igor Fedotov
0 siblings, 1 reply; 3+ messages in thread
From: Igor Fedotov @ 2016-08-11 13:34 UTC (permalink / raw)
To: ceph-devel
Better output from GDB
Program received signal SIGSEGV, Segmentation fault.
append_out_escaped (in="", out=out@entry=0x7fffffffcb70) at
/home/ifed/ceph/ceph_my4/src/common/hobject.cc:207
207 if (*i == '%' || *i == ':' || *i == '/' || *i < 32 || *i >=
127) {
(gdb) bt
#0 append_out_escaped (in="", out=out@entry=0x7fffffffcb70) at
/home/ifed/ceph/ceph_my4/src/common/hobject.cc:207
#1 0x00000000008cc2c9 in operator<< (out=..., o=...) at
/home/ifed/ceph/ceph_my4/src/common/hobject.cc:258
#2 0x00000000008cc7f7 in operator<< (out=..., o=...) at
/home/ifed/ceph/ceph_my4/src/common/hobject.cc:551
#3 0x000000000078b817 in BlueStore::collection_list
(this=this@entry=0x92742f0, c_=..., start=..., end=...,
sort_bitwise=sort_bitwise@entry=true,
max=max@entry=10, ls=0x7fffffffd2a0, pnext=0x7fffffffcee0) at
/home/ifed/ceph/ceph_my4/src/os/bluestore/BlueStore.cc:4219
#4 0x000000000079465b in BlueStore::collection_list (this=0x92742f0,
cid=..., start=..., end=..., sort_bitwise=sort_bitwise@entry=true,
max=max@entry=10, ls=0x7fffffffd2a0, pnext=0x0) at
/home/ifed/ceph/ceph_my4/src/os/bluestore/BlueStore.cc:4088
#5 0x000000000069210b in SyntheticWorkloadState::shutdown
(this=0x7fffffffd7f0) at
/home/ifed/ceph/ceph_my4/src/test/objectstore/store_test.cc:3097
#6 0x0000000000657ac4 in doSyntheticTest (store=..., max_obj=<optimized
out>, max_wr=<optimized out>, align=<optimized out>)
at /home/ifed/ceph/ceph_my4/src/test/objectstore/store_test.cc:3761
#7 0x0000000000a07cb3 in
HandleSehExceptionsInMethodIfSupported<testing::Test, void>
(location=0xc09697 "the test body", method=<optimized out>,
object=<optimized out>) at
/home/ifed/ceph/ceph_my4/src/googletest/googletest/src/gtest.cc:2402
#8 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
void> (object=object@entry=0x9159ad0,
method=(void (testing::Test::*)(testing::Test * const)) 0x6584e0
<StoreTest_Synthetic_Test::TestBody()>,
location=location@entry=0xc09697 "the test body") at
/home/ifed/ceph/ceph_my4/src/googletest/googletest/src/gtest.cc:2438
#9 0x00000000009f74da in testing::Test::Run (this=0x9159ad0) at
/home/ifed/ceph/ceph_my4/src/googletest/googletest/src/gtest.cc:2475
(gdb) f 2
#2 0x00000000008cc7f7 in operator<< (out=..., o=...) at
/home/ifed/ceph/ceph_my4/src/common/hobject.cc:551
551 out << '#' << o.hobj << '#';
(gdb) p o.hobject
(gdb) p o
$8 = (const ghobject_t &) @0x7fffffffcee0: {hobj = {oid = {name = ""},
snap = {val = 18446744073709551614}, hash = 192, max = false,
nibblewise_key_cache = 201326592, hash_reverse_bits = 50331648,
static POOL_META = -1, static POOL_TEMP_START = -2, pool = 555, nspace =
"",
key = ""}, generation = 1, shard_id = {id = -1 '\377', static
NO_SHARD = {id = -1 '\377',
static NO_SHARD = <same as static member of an already seen
type>}}, max = false, static NO_GEN = 18446744073709551615}
(gdb) p o.hobj
$9 = {oid = {name = ""}, snap = {val = 18446744073709551614}, hash =
192, max = false, nibblewise_key_cache = 201326592,
hash_reverse_bits = 50331648, static POOL_META = -1, static
POOL_TEMP_START = -2, pool = 555, nspace = "", key = ""}
(gdb) p o.hobj.oid.name
$10 = ""
(gdb) p o.hobj.oid.name.size()
$11 = 140737309080040
logging immediately after the assignement *pnext = oid shows no
corruption though...
On 11.08.2016 16:25, Igor Fedotov wrote:
> Hi All!
>
> I'm observing following crash when running Bluestore
> StoreTest.Synthetic test casr( ceph_test_objectstore
> --gtest_filter=ObjectStore/StoreTest.Synthetic/2 ):
>
>
> -20> 2016-08-11 16:09:30.155436 7f6c1437b680 15
> bluestore(store_test_temp_dir) collection_list 555.0_head start GHMIN
> end GHMAX max 10
> -19> 2016-08-11 16:09:30.155443 7f6c1437b680 20
> bluestore(store_test_temp_dir) collection_list range
> '--'0x7ffffffffffffdd300000000'.' to '--'0x7ff
> ffffffffffdd3ffffffff':' and '--'0x800000000000022b00000000'.' to
> '--'0x800000000000022bffffffff':' start GHMIN
> -18> 2016-08-11 16:09:30.155482 7f6c1437b680 20
> bluestore(store_test_temp_dir) collection_list pend
> '--'0x7ffffffffffffdd3ffffffff':'
> -17> 2016-08-11 16:09:30.155509 7f6c1437b680 20
> bluestore(store_test_temp_dir) collection_list key
> '--'0x800000000000022b00000000'.!=OBJ_1049aaaaaa
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xff
>
> fffffffffffffe0000000000000000 > GHMAX
> -16> 2016-08-11 16:09:30.155515 7f6c1437b680 30
> bluestore(store_test_temp_dir) collection_list switch to non-temp
> namespace
> -15> 2016-08-11 16:09:30.155521 7f6c1437b680 30
> bluestore(store_test_temp_dir) collection_list pend
> '--'0x800000000000022bffffffff':'
> -14> 2016-08-11 16:09:30.155524 7f6c1437b680 20
> bluestore(store_test_temp_dir) collection_list key
> '--'0x800000000000022b00000000'.!=OBJ_1049aaaaaa
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xff
>
> fffffffffffffe0000000000000000
> -13> 2016-08-11 16:09:30.155536 7f6c1437b680 20
> bluestore(store_test_temp_dir) collection_list key
> '--'0x800000000000022b00000000'.!=OBJ_1105aaaaaa
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x00
>
> 0000005df155b0ffffffffffffffff
> -12> 2016-08-11 16:09:30.155544 7f6c1437b680 20
> bluestore(store_test_temp_dir) collection_list key
> '--'0x800000000000022b00000000'.!=OBJ_255aaaaaaa
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
>
> 000007e448de90000000000000000
> -11> 2016-08-11 16:09:30.155555 7f6c1437b680 20
> bluestore(store_test_temp_dir) collection_list key
> '--'0x800000000000022b00000000'.!=OBJ_511aaaaaaa
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xfff
>
> ffffffffffffe0000000000000001
> -10> 2016-08-11 16:09:30.155563 7f6c1437b680 20
> bluestore(store_test_temp_dir) collection_list key
> '--'0x800000000000022b00000000'.!=OBJ_767aaaaaaa
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
>
> 000005cab38c60000000000000001
> -9> 2016-08-11 16:09:30.155574 7f6c1437b680 20
> bluestore(store_test_temp_dir) collection_list key
> '--'0x800000000000022b01000000'.!=OBJ_127aaaaaaa
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
>
> 0000026f324ba0000000000000000
> -8> 2016-08-11 16:09:30.155584 7f6c1437b680 20
> bluestore(store_test_temp_dir) collection_list key
> '--'0x800000000000022b01000000'.!=OBJ_639aaaaaaa
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
>
> 000007365f1ee0000000000000001
> -7> 2016-08-11 16:09:30.155592 7f6c1437b680 20
> bluestore(store_test_temp_dir) collection_list key
> '--'0x800000000000022b02000000'.!=OBJ_575aaaaaaa
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
>
> 00000060a14630000000000000000
> -6> 2016-08-11 16:09:30.155601 7f6c1437b680 20
> bluestore(store_test_temp_dir) collection_list key
> '--'0x800000000000022b02000000'.!=OBJ_831aaaaaaa
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xfff
>
> ffffffffffffe0000000000000000
> -5> 2016-08-11 16:09:30.155612 7f6c1437b680 20
> bluestore(store_test_temp_dir) collection_list key
> '--'0x800000000000022b03000000 (bnode, skipping)
> -4> 2016-08-11 16:09:30.155617 7f6c1437b680 20
> bluestore(store_test_temp_dir) collection_list key
> '--'0x800000000000022b03000000'.!=OBJ_1013aaaaaa
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xff
>
> fffffffffffffe0000000000000002
> -3> 2016-08-11 16:09:30.155626 7f6c1437b680 20
> bluestore(store_test_temp_dir) collection_list key
> '--'0x800000000000022b03000000'.!=OBJ_1243aaaaaa
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x00
>
> 0000000c94deb00000000000000000
> -2> 2016-08-11 16:09:30.155631 7f6c1437b680 20
> bluestore(store_test_temp_dir) collection_list reached max 10
> -1> 2016-08-11 16:09:30.155636 7f6c1437b680 20
> bluestore.2QCache(0xa282410) trim onodes 500 / 500 buffers 1978368 /
> 2000000
> 0> 2016-08-11 16:09:56.642872 7f6c1437b680 -1 *** Caught signal
> (Segmentation fault) **
> in thread 7f6c1437b680 thread_name:ceph_test_objec
>
> ceph version v11.0.0-1434-gc6e561a
> (c6e561a5385105dfbd70f642bcd5732fac0bca89)
> 1: bin/ceph_test_objectstore() [0xa1f207]
> 2: (()+0xfc90) [0x7f6c13f7bc90]
> 3: bin/ceph_test_objectstore() [0x8cadab]
> 4: (operator<<(std::ostream&, hobject_t const&)+0x229) [0x8d1709]
> 5: (operator<<(std::ostream&, ghobject_t const&)+0x107) [0x8d1c37]
> 6:
> (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&,
> ghobject_t, ghobject_t, bool, int, std::vector<ghobject_t, std::al
> locator<ghobject_t> >*, ghobject_t*)+0xb6d) [0x78680d]
> 7: (BlueStore::collection_list(coll_t const&, ghobject_t, ghobject_t,
> bool, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*,
> ghobject_t*)+
> 0x1eb) [0x79d64b]
> 8: (SyntheticWorkloadState::shutdown()+0x2ab) [0x68b88b]
> 9: (doSyntheticTest(boost::scoped_ptr<ObjectStore>&, unsigned long,
> unsigned long, unsigned long)+0x471) [0x651741]
>
>
> It looks like it was introduced by the recent PR #10410 merge
> (https://github.com/ceph/ceph/pull/10410). I don't observe the issue
> prior to this patch.
>
> Brief analysis under GDB shows that ghobject_t instance pointed by
> pnext passed to operator<< at Bluestore.cc:4167 is broken - oid.name
> field has huge size..
>
> The issue is 100% reproducible.
>
>
> Thanks,
>
> Igor
>
>
>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Bluestore StoreTest.Synthetic is broken
2016-08-11 13:34 ` Igor Fedotov
@ 2016-08-11 14:18 ` Igor Fedotov
0 siblings, 0 replies; 3+ messages in thread
From: Igor Fedotov @ 2016-08-11 14:18 UTC (permalink / raw)
To: ceph-devel
Answering to myself.
The root cause is as follows:
store_test calls collection_list with pnext = NULL.
And pnext is assigned to an address of static_next at Bluestore.cc:4070.
The issue is that static_next is defined within a local scope while
final output for *pnext happens out of this scope.
Will publish a PR with the fix shortly.
Thanks,
Igor
On 11.08.2016 16:34, Igor Fedotov wrote:
> Better output from GDB
>
> Program received signal SIGSEGV, Segmentation fault.
> append_out_escaped (in="", out=out@entry=0x7fffffffcb70) at
> /home/ifed/ceph/ceph_my4/src/common/hobject.cc:207
> 207 if (*i == '%' || *i == ':' || *i == '/' || *i < 32 || *i >=
> 127) {
> (gdb) bt
> #0 append_out_escaped (in="", out=out@entry=0x7fffffffcb70) at
> /home/ifed/ceph/ceph_my4/src/common/hobject.cc:207
> #1 0x00000000008cc2c9 in operator<< (out=..., o=...) at
> /home/ifed/ceph/ceph_my4/src/common/hobject.cc:258
> #2 0x00000000008cc7f7 in operator<< (out=..., o=...) at
> /home/ifed/ceph/ceph_my4/src/common/hobject.cc:551
> #3 0x000000000078b817 in BlueStore::collection_list
> (this=this@entry=0x92742f0, c_=..., start=..., end=...,
> sort_bitwise=sort_bitwise@entry=true,
> max=max@entry=10, ls=0x7fffffffd2a0, pnext=0x7fffffffcee0) at
> /home/ifed/ceph/ceph_my4/src/os/bluestore/BlueStore.cc:4219
> #4 0x000000000079465b in BlueStore::collection_list (this=0x92742f0,
> cid=..., start=..., end=..., sort_bitwise=sort_bitwise@entry=true,
> max=max@entry=10, ls=0x7fffffffd2a0, pnext=0x0) at
> /home/ifed/ceph/ceph_my4/src/os/bluestore/BlueStore.cc:4088
> #5 0x000000000069210b in SyntheticWorkloadState::shutdown
> (this=0x7fffffffd7f0) at
> /home/ifed/ceph/ceph_my4/src/test/objectstore/store_test.cc:3097
> #6 0x0000000000657ac4 in doSyntheticTest (store=...,
> max_obj=<optimized out>, max_wr=<optimized out>, align=<optimized out>)
> at /home/ifed/ceph/ceph_my4/src/test/objectstore/store_test.cc:3761
> #7 0x0000000000a07cb3 in
> HandleSehExceptionsInMethodIfSupported<testing::Test, void>
> (location=0xc09697 "the test body", method=<optimized out>,
> object=<optimized out>) at
> /home/ifed/ceph/ceph_my4/src/googletest/googletest/src/gtest.cc:2402
> #8
> testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
> void> (object=object@entry=0x9159ad0,
> method=(void (testing::Test::*)(testing::Test * const)) 0x6584e0
> <StoreTest_Synthetic_Test::TestBody()>,
> location=location@entry=0xc09697 "the test body") at
> /home/ifed/ceph/ceph_my4/src/googletest/googletest/src/gtest.cc:2438
> #9 0x00000000009f74da in testing::Test::Run (this=0x9159ad0) at
> /home/ifed/ceph/ceph_my4/src/googletest/googletest/src/gtest.cc:2475
>
> (gdb) f 2
> #2 0x00000000008cc7f7 in operator<< (out=..., o=...) at
> /home/ifed/ceph/ceph_my4/src/common/hobject.cc:551
> 551 out << '#' << o.hobj << '#';
> (gdb) p o.hobject
> (gdb) p o
> $8 = (const ghobject_t &) @0x7fffffffcee0: {hobj = {oid = {name = ""},
> snap = {val = 18446744073709551614}, hash = 192, max = false,
> nibblewise_key_cache = 201326592, hash_reverse_bits = 50331648,
> static POOL_META = -1, static POOL_TEMP_START = -2, pool = 555, nspace
> = "",
> key = ""}, generation = 1, shard_id = {id = -1 '\377', static
> NO_SHARD = {id = -1 '\377',
> static NO_SHARD = <same as static member of an already seen
> type>}}, max = false, static NO_GEN = 18446744073709551615}
> (gdb) p o.hobj
> $9 = {oid = {name = ""}, snap = {val = 18446744073709551614}, hash =
> 192, max = false, nibblewise_key_cache = 201326592,
> hash_reverse_bits = 50331648, static POOL_META = -1, static
> POOL_TEMP_START = -2, pool = 555, nspace = "", key = ""}
> (gdb) p o.hobj.oid.name
> $10 = ""
> (gdb) p o.hobj.oid.name.size()
> $11 = 140737309080040
>
> logging immediately after the assignement *pnext = oid shows no
> corruption though...
>
>
> On 11.08.2016 16:25, Igor Fedotov wrote:
>> Hi All!
>>
>> I'm observing following crash when running Bluestore
>> StoreTest.Synthetic test casr( ceph_test_objectstore
>> --gtest_filter=ObjectStore/StoreTest.Synthetic/2 ):
>>
>>
>> -20> 2016-08-11 16:09:30.155436 7f6c1437b680 15
>> bluestore(store_test_temp_dir) collection_list 555.0_head start GHMIN
>> end GHMAX max 10
>> -19> 2016-08-11 16:09:30.155443 7f6c1437b680 20
>> bluestore(store_test_temp_dir) collection_list range
>> '--'0x7ffffffffffffdd300000000'.' to '--'0x7ff
>> ffffffffffdd3ffffffff':' and '--'0x800000000000022b00000000'.' to
>> '--'0x800000000000022bffffffff':' start GHMIN
>> -18> 2016-08-11 16:09:30.155482 7f6c1437b680 20
>> bluestore(store_test_temp_dir) collection_list pend
>> '--'0x7ffffffffffffdd3ffffffff':'
>> -17> 2016-08-11 16:09:30.155509 7f6c1437b680 20
>> bluestore(store_test_temp_dir) collection_list key
>> '--'0x800000000000022b00000000'.!=OBJ_1049aaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>>
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xff
>>
>> fffffffffffffe0000000000000000 > GHMAX
>> -16> 2016-08-11 16:09:30.155515 7f6c1437b680 30
>> bluestore(store_test_temp_dir) collection_list switch to non-temp
>> namespace
>> -15> 2016-08-11 16:09:30.155521 7f6c1437b680 30
>> bluestore(store_test_temp_dir) collection_list pend
>> '--'0x800000000000022bffffffff':'
>> -14> 2016-08-11 16:09:30.155524 7f6c1437b680 20
>> bluestore(store_test_temp_dir) collection_list key
>> '--'0x800000000000022b00000000'.!=OBJ_1049aaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>>
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xff
>>
>> fffffffffffffe0000000000000000
>> -13> 2016-08-11 16:09:30.155536 7f6c1437b680 20
>> bluestore(store_test_temp_dir) collection_list key
>> '--'0x800000000000022b00000000'.!=OBJ_1105aaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>>
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x00
>>
>> 0000005df155b0ffffffffffffffff
>> -12> 2016-08-11 16:09:30.155544 7f6c1437b680 20
>> bluestore(store_test_temp_dir) collection_list key
>> '--'0x800000000000022b00000000'.!=OBJ_255aaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>>
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
>>
>> 000007e448de90000000000000000
>> -11> 2016-08-11 16:09:30.155555 7f6c1437b680 20
>> bluestore(store_test_temp_dir) collection_list key
>> '--'0x800000000000022b00000000'.!=OBJ_511aaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>>
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xfff
>>
>> ffffffffffffe0000000000000001
>> -10> 2016-08-11 16:09:30.155563 7f6c1437b680 20
>> bluestore(store_test_temp_dir) collection_list key
>> '--'0x800000000000022b00000000'.!=OBJ_767aaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>>
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
>>
>> 000005cab38c60000000000000001
>> -9> 2016-08-11 16:09:30.155574 7f6c1437b680 20
>> bluestore(store_test_temp_dir) collection_list key
>> '--'0x800000000000022b01000000'.!=OBJ_127aaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>>
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
>>
>> 0000026f324ba0000000000000000
>> -8> 2016-08-11 16:09:30.155584 7f6c1437b680 20
>> bluestore(store_test_temp_dir) collection_list key
>> '--'0x800000000000022b01000000'.!=OBJ_639aaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>>
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
>>
>> 000007365f1ee0000000000000001
>> -7> 2016-08-11 16:09:30.155592 7f6c1437b680 20
>> bluestore(store_test_temp_dir) collection_list key
>> '--'0x800000000000022b02000000'.!=OBJ_575aaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>>
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x000
>>
>> 00000060a14630000000000000000
>> -6> 2016-08-11 16:09:30.155601 7f6c1437b680 20
>> bluestore(store_test_temp_dir) collection_list key
>> '--'0x800000000000022b02000000'.!=OBJ_831aaaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>>
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xfff
>>
>> ffffffffffffe0000000000000000
>> -5> 2016-08-11 16:09:30.155612 7f6c1437b680 20
>> bluestore(store_test_temp_dir) collection_list key
>> '--'0x800000000000022b03000000 (bnode, skipping)
>> -4> 2016-08-11 16:09:30.155617 7f6c1437b680 20
>> bluestore(store_test_temp_dir) collection_list key
>> '--'0x800000000000022b03000000'.!=OBJ_1013aaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>>
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0xff
>>
>> fffffffffffffe0000000000000002
>> -3> 2016-08-11 16:09:30.155626 7f6c1437b680 20
>> bluestore(store_test_temp_dir) collection_list key
>> '--'0x800000000000022b03000000'.!=OBJ_1243aaaaaa
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
>>
>> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'0x00
>>
>> 0000000c94deb00000000000000000
>> -2> 2016-08-11 16:09:30.155631 7f6c1437b680 20
>> bluestore(store_test_temp_dir) collection_list reached max 10
>> -1> 2016-08-11 16:09:30.155636 7f6c1437b680 20
>> bluestore.2QCache(0xa282410) trim onodes 500 / 500 buffers 1978368 /
>> 2000000
>> 0> 2016-08-11 16:09:56.642872 7f6c1437b680 -1 *** Caught signal
>> (Segmentation fault) **
>> in thread 7f6c1437b680 thread_name:ceph_test_objec
>>
>> ceph version v11.0.0-1434-gc6e561a
>> (c6e561a5385105dfbd70f642bcd5732fac0bca89)
>> 1: bin/ceph_test_objectstore() [0xa1f207]
>> 2: (()+0xfc90) [0x7f6c13f7bc90]
>> 3: bin/ceph_test_objectstore() [0x8cadab]
>> 4: (operator<<(std::ostream&, hobject_t const&)+0x229) [0x8d1709]
>> 5: (operator<<(std::ostream&, ghobject_t const&)+0x107) [0x8d1c37]
>> 6:
>> (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&,
>> ghobject_t, ghobject_t, bool, int, std::vector<ghobject_t, std::al
>> locator<ghobject_t> >*, ghobject_t*)+0xb6d) [0x78680d]
>> 7: (BlueStore::collection_list(coll_t const&, ghobject_t,
>> ghobject_t, bool, int, std::vector<ghobject_t,
>> std::allocator<ghobject_t> >*, ghobject_t*)+
>> 0x1eb) [0x79d64b]
>> 8: (SyntheticWorkloadState::shutdown()+0x2ab) [0x68b88b]
>> 9: (doSyntheticTest(boost::scoped_ptr<ObjectStore>&, unsigned long,
>> unsigned long, unsigned long)+0x471) [0x651741]
>>
>>
>> It looks like it was introduced by the recent PR #10410 merge
>> (https://github.com/ceph/ceph/pull/10410). I don't observe the issue
>> prior to this patch.
>>
>> Brief analysis under GDB shows that ghobject_t instance pointed by
>> pnext passed to operator<< at Bluestore.cc:4167 is broken - oid.name
>> field has huge size..
>>
>> The issue is 100% reproducible.
>>
>>
>> Thanks,
>>
>> Igor
>>
>>
>>
>
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2016-08-11 14:18 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-08-11 13:25 Bluestore StoreTest.Synthetic is broken Igor Fedotov
2016-08-11 13:34 ` Igor Fedotov
2016-08-11 14:18 ` Igor Fedotov
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.