xattr spillout appears broken :(

xattr spillout appears broken :(

[Gregory Farnum]

I got a bug report a little while ago that we were hitting leveldb on
every xattr lookup with Firefly, and after looking into it I indeed
discovered that we never set the XATTR_SPILL_OUT_NAME xattr when
creating new files.
I pushed a branch wip-xattr-spillout containing a fix for that, plus
some more general FileStore cleanups Sam asked for, but (much to my
surprise) it didn't pass testing. I tried backing off and bisecting
down to the broken commit, and although the test failures aren't
consistent, it appears that simply setting the spillout xattr is
enough to sporadically break tests (patch "FileStore: set
XATTR_NO_SPILL_OUT when creating new files.").

I've done some code inspection to see if I can find the issue and am
drawing a blank. It's possible my setup patch is just wrong, but I
think the issue must be elsewhere in the code. I wonder if you could
take a look and try to figure it out?

My test logs (these are against two slightly different branches; so
check the sha1s) are:
Test Run: gregf-2014-05-22_15:27:32-rados-wip-xattr-spillout-basic-testing-basic-plana
=================================================================
info:   http://pulpito.ceph.com/gregf-2014-05-22_15:27:32-rados-wip-xattr-spillout-basic-testing-basic-plana/
logs:   http://qa-proxy.ceph.com/teuthology/gregf-2014-05-22_15:27:32-rados-wip-xattr-spillout-basic-testing-basic-plana/

Test Run: gregf-2014-05-20_15:45:53-rados-wip-xattr-spillout-testing-basic-plana
=================================================================
info:   http://pulpito.ceph.com/gregf-2014-05-20_15:45:53-rados-wip-xattr-spillout-testing-basic-plana/
logs:   http://qa-proxy.ceph.com/teuthology/gregf-2014-05-20_15:45:53-rados-wip-xattr-spillout-testing-basic-plana/

If you don't think you'll have a chance, let me know and I'll dig into
it more. Thanks!
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com

Re: xattr spillout appears broken :(

[Haomai Wang]

Hi Gregory,

I try to reproduce the bug in my local machine but failed.

My test cmdline:
./ceph_test_rados --op read 100 --op write 100 --op delete 50
--max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000
--min-stride-size 400000 --max-stride-size 800000 --max-seconds 600
--op copy_from 50 --op snap_create 50 --op snap_remove 50 --op
rollback 50 --op setattr 25 --op rmattr 25 --pool unique_pool_0

Is there any tip to reproduce it?

On Fri, May 30, 2014 at 7:15 AM, Gregory Farnum <greg@inktank.com> wrote:
> I got a bug report a little while ago that we were hitting leveldb on
> every xattr lookup with Firefly, and after looking into it I indeed
> discovered that we never set the XATTR_SPILL_OUT_NAME xattr when
> creating new files.
> I pushed a branch wip-xattr-spillout containing a fix for that, plus
> some more general FileStore cleanups Sam asked for, but (much to my
> surprise) it didn't pass testing. I tried backing off and bisecting
> down to the broken commit, and although the test failures aren't
> consistent, it appears that simply setting the spillout xattr is
> enough to sporadically break tests (patch "FileStore: set
> XATTR_NO_SPILL_OUT when creating new files.").
>
> I've done some code inspection to see if I can find the issue and am
> drawing a blank. It's possible my setup patch is just wrong, but I
> think the issue must be elsewhere in the code. I wonder if you could
> take a look and try to figure it out?
>
> My test logs (these are against two slightly different branches; so
> check the sha1s) are:
> Test Run: gregf-2014-05-22_15:27:32-rados-wip-xattr-spillout-basic-testing-basic-plana
> =================================================================
> info:   http://pulpito.ceph.com/gregf-2014-05-22_15:27:32-rados-wip-xattr-spillout-basic-testing-basic-plana/
> logs:   http://qa-proxy.ceph.com/teuthology/gregf-2014-05-22_15:27:32-rados-wip-xattr-spillout-basic-testing-basic-plana/
>
> Test Run: gregf-2014-05-20_15:45:53-rados-wip-xattr-spillout-testing-basic-plana
> =================================================================
> info:   http://pulpito.ceph.com/gregf-2014-05-20_15:45:53-rados-wip-xattr-spillout-testing-basic-plana/
> logs:   http://qa-proxy.ceph.com/teuthology/gregf-2014-05-20_15:45:53-rados-wip-xattr-spillout-testing-basic-plana/
>
> If you don't think you'll have a chance, let me know and I'll dig into
> it more. Thanks!
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com



-- 
Best Regards,

Wheat

Re: xattr spillout appears broken :(

[Gregory Farnum]

On Fri, May 30, 2014 at 2:18 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
> Hi Gregory,
>
> I try to reproduce the bug in my local machine but failed.
>
> My test cmdline:
> ./ceph_test_rados --op read 100 --op write 100 --op delete 50
> --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000
> --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600
> --op copy_from 50 --op snap_create 50 --op snap_remove 50 --op
> rollback 50 --op setattr 25 --op rmattr 25 --pool unique_pool_0
>
> Is there any tip to reproduce it?

Hmm. I've been directly running the teuthology tests that failed; that
is the command line it's running though so I think the only difference
would be that I've got OSD thrashing (ie, recovery) happening while
the test is running.
Most of the other failures were scrub turning up inconsistencies in
the xattrs at each replica/shard of an object. I didn't see any
obvious mechanism by which storing values in xattrs versus leveldb
would impact these higher-level primitives, but maybe you have some
idea?
-Greg

Re: xattr spillout appears broken :(

[Haomai Wang]

Hi Gregory,

I checked again and again each line change about spill out codes,
still failed to find anything wrong.

I ran "ceph_test_rados" then activate scrub process several times
locally, nothing unusual. I'm familiar with teuthoghy jobs, maybe we
can find the common thing among fail jobs.


On Sat, May 31, 2014 at 2:06 AM, Gregory Farnum <greg@inktank.com> wrote:
> On Fri, May 30, 2014 at 2:18 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
>> Hi Gregory,
>>
>> I try to reproduce the bug in my local machine but failed.
>>
>> My test cmdline:
>> ./ceph_test_rados --op read 100 --op write 100 --op delete 50
>> --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000
>> --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600
>> --op copy_from 50 --op snap_create 50 --op snap_remove 50 --op
>> rollback 50 --op setattr 25 --op rmattr 25 --pool unique_pool_0
>>
>> Is there any tip to reproduce it?
>
> Hmm. I've been directly running the teuthology tests that failed; that
> is the command line it's running though so I think the only difference
> would be that I've got OSD thrashing (ie, recovery) happening while
> the test is running.
> Most of the other failures were scrub turning up inconsistencies in
> the xattrs at each replica/shard of an object. I didn't see any
> obvious mechanism by which storing values in xattrs versus leveldb
> would impact these higher-level primitives, but maybe you have some
> idea?
> -Greg



-- 
Best Regards,

Wheat

Re: xattr spillout appears broken :(

[Haomai Wang]

/familiar/not familiar/

On Tue, Jun 3, 2014 at 10:33 PM, Haomai Wang <haomaiwang@gmail.com> wrote:
> Hi Gregory,
>
> I checked again and again each line change about spill out codes,
> still failed to find anything wrong.
>
> I ran "ceph_test_rados" then activate scrub process several times
> locally, nothing unusual. I'm familiar with teuthoghy jobs, maybe we
> can find the common thing among fail jobs.
>
>
> On Sat, May 31, 2014 at 2:06 AM, Gregory Farnum <greg@inktank.com> wrote:
>> On Fri, May 30, 2014 at 2:18 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
>>> Hi Gregory,
>>>
>>> I try to reproduce the bug in my local machine but failed.
>>>
>>> My test cmdline:
>>> ./ceph_test_rados --op read 100 --op write 100 --op delete 50
>>> --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000
>>> --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600
>>> --op copy_from 50 --op snap_create 50 --op snap_remove 50 --op
>>> rollback 50 --op setattr 25 --op rmattr 25 --pool unique_pool_0
>>>
>>> Is there any tip to reproduce it?
>>
>> Hmm. I've been directly running the teuthology tests that failed; that
>> is the command line it's running though so I think the only difference
>> would be that I've got OSD thrashing (ie, recovery) happening while
>> the test is running.
>> Most of the other failures were scrub turning up inconsistencies in
>> the xattrs at each replica/shard of an object. I didn't see any
>> obvious mechanism by which storing values in xattrs versus leveldb
>> would impact these higher-level primitives, but maybe you have some
>> idea?
>> -Greg
>
>
>
> --
> Best Regards,
>
> Wheat



-- 
Best Regards,

Wheat

Re: xattr spillout appears broken :(

[Haomai Wang]

Hi Greg,

I have found the reason. "user.cephos.spill_out" can't be apply to new
object when calling clone method. So if the origin object is spill
out, the new object still has no spill out marker.

I pushed a commit (https://github.com/ceph/ceph/pull/1932) to help
cover this situation.


On Tue, Jun 3, 2014 at 10:33 PM, Haomai Wang <haomaiwang@gmail.com> wrote:
> /familiar/not familiar/
>
> On Tue, Jun 3, 2014 at 10:33 PM, Haomai Wang <haomaiwang@gmail.com> wrote:
>> Hi Gregory,
>>
>> I checked again and again each line change about spill out codes,
>> still failed to find anything wrong.
>>
>> I ran "ceph_test_rados" then activate scrub process several times
>> locally, nothing unusual. I'm familiar with teuthoghy jobs, maybe we
>> can find the common thing among fail jobs.
>>
>>
>> On Sat, May 31, 2014 at 2:06 AM, Gregory Farnum <greg@inktank.com> wrote:
>>> On Fri, May 30, 2014 at 2:18 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
>>>> Hi Gregory,
>>>>
>>>> I try to reproduce the bug in my local machine but failed.
>>>>
>>>> My test cmdline:
>>>> ./ceph_test_rados --op read 100 --op write 100 --op delete 50
>>>> --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000
>>>> --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600
>>>> --op copy_from 50 --op snap_create 50 --op snap_remove 50 --op
>>>> rollback 50 --op setattr 25 --op rmattr 25 --pool unique_pool_0
>>>>
>>>> Is there any tip to reproduce it?
>>>
>>> Hmm. I've been directly running the teuthology tests that failed; that
>>> is the command line it's running though so I think the only difference
>>> would be that I've got OSD thrashing (ie, recovery) happening while
>>> the test is running.
>>> Most of the other failures were scrub turning up inconsistencies in
>>> the xattrs at each replica/shard of an object. I didn't see any
>>> obvious mechanism by which storing values in xattrs versus leveldb
>>> would impact these higher-level primitives, but maybe you have some
>>> idea?
>>> -Greg
>>
>>
>>
>> --
>> Best Regards,
>>
>> Wheat
>
>
>
> --
> Best Regards,
>
> Wheat



-- 
Best Regards,

Wheat

Re: xattr spillout appears broken :(

[Haomai Wang]

The fix should make clone method copy "cephos" prefix xattrs

On Sat, Jun 7, 2014 at 2:54 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
> Hi Greg,
>
> I have found the reason. "user.cephos.spill_out" can't be apply to new
> object when calling clone method. So if the origin object is spill
> out, the new object still has no spill out marker.
>
> I pushed a commit (https://github.com/ceph/ceph/pull/1932) to help
> cover this situation.
>
>
> On Tue, Jun 3, 2014 at 10:33 PM, Haomai Wang <haomaiwang@gmail.com> wrote:
>> /familiar/not familiar/
>>
>> On Tue, Jun 3, 2014 at 10:33 PM, Haomai Wang <haomaiwang@gmail.com> wrote:
>>> Hi Gregory,
>>>
>>> I checked again and again each line change about spill out codes,
>>> still failed to find anything wrong.
>>>
>>> I ran "ceph_test_rados" then activate scrub process several times
>>> locally, nothing unusual. I'm familiar with teuthoghy jobs, maybe we
>>> can find the common thing among fail jobs.
>>>
>>>
>>> On Sat, May 31, 2014 at 2:06 AM, Gregory Farnum <greg@inktank.com> wrote:
>>>> On Fri, May 30, 2014 at 2:18 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
>>>>> Hi Gregory,
>>>>>
>>>>> I try to reproduce the bug in my local machine but failed.
>>>>>
>>>>> My test cmdline:
>>>>> ./ceph_test_rados --op read 100 --op write 100 --op delete 50
>>>>> --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000
>>>>> --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600
>>>>> --op copy_from 50 --op snap_create 50 --op snap_remove 50 --op
>>>>> rollback 50 --op setattr 25 --op rmattr 25 --pool unique_pool_0
>>>>>
>>>>> Is there any tip to reproduce it?
>>>>
>>>> Hmm. I've been directly running the teuthology tests that failed; that
>>>> is the command line it's running though so I think the only difference
>>>> would be that I've got OSD thrashing (ie, recovery) happening while
>>>> the test is running.
>>>> Most of the other failures were scrub turning up inconsistencies in
>>>> the xattrs at each replica/shard of an object. I didn't see any
>>>> obvious mechanism by which storing values in xattrs versus leveldb
>>>> would impact these higher-level primitives, but maybe you have some
>>>> idea?
>>>> -Greg
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>>
>>> Wheat
>>
>>
>>
>> --
>> Best Regards,
>>
>> Wheat
>
>
>
> --
> Best Regards,
>
> Wheat



-- 
Best Regards,

Wheat

Re: xattr spillout appears broken :(

[Gregory Farnum]

On Fri, Jun 6, 2014 at 11:55 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
> The fix should make clone method copy "cephos" prefix xattrs
>
> On Sat, Jun 7, 2014 at 2:54 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
>> Hi Greg,
>>
>> I have found the reason. "user.cephos.spill_out" can't be apply to new
>> object when calling clone method. So if the origin object is spill
>> out, the new object still has no spill out marker.
>>
>> I pushed a commit (https://github.com/ceph/ceph/pull/1932) to help
>> cover this situation.

Awesome, thanks!

So that commit is a test that reveals the issue, but we still need to
come up with a fix?
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com

Re: xattr spillout appears broken :(

[Haomai Wang]

Yes, maybe you can add it in your branch. Because it will happen when
creating object and set spill out

On Sat, Jun 7, 2014 at 2:59 AM, Gregory Farnum <greg@inktank.com> wrote:
> On Fri, Jun 6, 2014 at 11:55 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
>> The fix should make clone method copy "cephos" prefix xattrs
>>
>> On Sat, Jun 7, 2014 at 2:54 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
>>> Hi Greg,
>>>
>>> I have found the reason. "user.cephos.spill_out" can't be apply to new
>>> object when calling clone method. So if the origin object is spill
>>> out, the new object still has no spill out marker.
>>>
>>> I pushed a commit (https://github.com/ceph/ceph/pull/1932) to help
>>> cover this situation.
>
> Awesome, thanks!
>
> So that commit is a test that reveals the issue, but we still need to
> come up with a fix?
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com



-- 
Best Regards,

Wheat

Re: xattr spillout appears broken :(

[Haomai Wang]

Greg, I have submit a patch to fix it. https://github.com/ceph/ceph/pull/1932

On Sat, Jun 7, 2014 at 3:00 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
> Yes, maybe you can add it in your branch. Because it will happen when
> creating object and set spill out
>
> On Sat, Jun 7, 2014 at 2:59 AM, Gregory Farnum <greg@inktank.com> wrote:
>> On Fri, Jun 6, 2014 at 11:55 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
>>> The fix should make clone method copy "cephos" prefix xattrs
>>>
>>> On Sat, Jun 7, 2014 at 2:54 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
>>>> Hi Greg,
>>>>
>>>> I have found the reason. "user.cephos.spill_out" can't be apply to new
>>>> object when calling clone method. So if the origin object is spill
>>>> out, the new object still has no spill out marker.
>>>>
>>>> I pushed a commit (https://github.com/ceph/ceph/pull/1932) to help
>>>> cover this situation.
>>
>> Awesome, thanks!
>>
>> So that commit is a test that reveals the issue, but we still need to
>> come up with a fix?
>> -Greg
>> Software Engineer #42 @ http://inktank.com | http://ceph.com
>
>
>
> --
> Best Regards,
>
> Wheat



-- 
Best Regards,

Wheat

Re: xattr spillout appears broken :(

[Gregory Farnum]

This is all merged in to master now, thanks very much Haomai!
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Fri, Jun 6, 2014 at 11:52 PM, Haomai Wang <haomaiwang@gmail.com> wrote:
> Greg, I have submit a patch to fix it. https://github.com/ceph/ceph/pull/1932
>
> On Sat, Jun 7, 2014 at 3:00 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
>> Yes, maybe you can add it in your branch. Because it will happen when
>> creating object and set spill out
>>
>> On Sat, Jun 7, 2014 at 2:59 AM, Gregory Farnum <greg@inktank.com> wrote:
>>> On Fri, Jun 6, 2014 at 11:55 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
>>>> The fix should make clone method copy "cephos" prefix xattrs
>>>>
>>>> On Sat, Jun 7, 2014 at 2:54 AM, Haomai Wang <haomaiwang@gmail.com> wrote:
>>>>> Hi Greg,
>>>>>
>>>>> I have found the reason. "user.cephos.spill_out" can't be apply to new
>>>>> object when calling clone method. So if the origin object is spill
>>>>> out, the new object still has no spill out marker.
>>>>>
>>>>> I pushed a commit (https://github.com/ceph/ceph/pull/1932) to help
>>>>> cover this situation.
>>>
>>> Awesome, thanks!
>>>
>>> So that commit is a test that reveals the issue, but we still need to
>>> come up with a fix?
>>> -Greg
>>> Software Engineer #42 @ http://inktank.com | http://ceph.com
>>
>>
>>
>> --
>> Best Regards,
>>
>> Wheat
>
>
>
> --
> Best Regards,
>
> Wheat