Help debugging RGW bug in jewel 10.2.8 integration branch

Help debugging RGW bug in jewel 10.2.8 integration branch

[Nathan Cutler]

Hello RGW developers:

Since 10.2.7 is almost out, I'm now running integration tests for 
10.2.8. As usual, there is quite a crop of RGW backports. The 
integration branch is "wip-jewel-backports" and the RGW suite [1] is not 
looking good: over 50 failures, and most of them are this:

======================================================================
FAIL: s3tests.functional.test_s3.test_versioning_obj_suspend_versions
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/nose/case.py", 
line 197, in runTest
     self.test(*self.arg)
   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py", 
line 6385, in test_versioning_obj_suspend_versions
     overwrite_suspended_versioning_obj(bucket, objname, k, c, 'null 
content 2')
   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py", 
line 6243, in overwrite_suspended_versioning_obj
     check_obj_versions(bucket, objname, k, c)
   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py", 
line 6080, in check_obj_versions
     eq(keys[i].version_id or 'null', key.version_id)
AssertionError: u'yGSvpxjEbJRBE.JL76y4OzeJISqDtmJ' != u'null'

Does anyone have an idea which of these PRs [2] might be the culprit?

[1] 
http://pulpito.ceph.com/smithfarm-2017-04-07_09:30:57-rgw-wip-jewel-backports-distro-basic-smithi/
[2] 
https://github.com/ceph/ceph/pulls?q=is%3Aopen+is%3Apr+milestone%3Ajewel+label%3Argw

Thanks in advance,
Nathan

Re: Help debugging RGW bug in jewel 10.2.8 integration branch

[Nathan Cutler]

All the failures appear to be in rgw/verify and rgw/multifs

On 04/09/2017 03:33 PM, Nathan Cutler wrote:
> Hello RGW developers:
>
> Since 10.2.7 is almost out, I'm now running integration tests for
> 10.2.8. As usual, there is quite a crop of RGW backports. The
> integration branch is "wip-jewel-backports" and the RGW suite [1] is not
> looking good: over 50 failures, and most of them are this:
>
> ======================================================================
> FAIL: s3tests.functional.test_s3.test_versioning_obj_suspend_versions
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/nose/case.py",
> line 197, in runTest
>     self.test(*self.arg)
>   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py",
> line 6385, in test_versioning_obj_suspend_versions
>     overwrite_suspended_versioning_obj(bucket, objname, k, c, 'null
> content 2')
>   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py",
> line 6243, in overwrite_suspended_versioning_obj
>     check_obj_versions(bucket, objname, k, c)
>   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py",
> line 6080, in check_obj_versions
>     eq(keys[i].version_id or 'null', key.version_id)
> AssertionError: u'yGSvpxjEbJRBE.JL76y4OzeJISqDtmJ' != u'null'
>
> Does anyone have an idea which of these PRs [2] might be the culprit?
>
> [1]
> http://pulpito.ceph.com/smithfarm-2017-04-07_09:30:57-rgw-wip-jewel-backports-distro-basic-smithi/
>
> [2]
> https://github.com/ceph/ceph/pulls?q=is%3Aopen+is%3Apr+milestone%3Ajewel+label%3Argw
>
>
> Thanks in advance,
> Nathan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Nathan Cutler
Software Engineer Distributed Storage
SUSE LINUX, s.r.o.
Tel.: +420 284 084 037

Re: Help debugging RGW bug in jewel 10.2.8 integration branch

[Yuri Weinstein]

Just a note that rgw passed for v10.2.7
So we may need to split PRs in smaller batches to find offending one.

On Sun, Apr 9, 2017 at 8:28 AM, Nathan Cutler <ncutler@suse.cz> wrote:
> All the failures appear to be in rgw/verify and rgw/multifs
>
>
> On 04/09/2017 03:33 PM, Nathan Cutler wrote:
>>
>> Hello RGW developers:
>>
>> Since 10.2.7 is almost out, I'm now running integration tests for
>> 10.2.8. As usual, there is quite a crop of RGW backports. The
>> integration branch is "wip-jewel-backports" and the RGW suite [1] is not
>> looking good: over 50 failures, and most of them are this:
>>
>> ======================================================================
>> FAIL: s3tests.functional.test_s3.test_versioning_obj_suspend_versions
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>   File
>>
>> "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/nose/case.py",
>> line 197, in runTest
>>     self.test(*self.arg)
>>   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py",
>> line 6385, in test_versioning_obj_suspend_versions
>>     overwrite_suspended_versioning_obj(bucket, objname, k, c, 'null
>> content 2')
>>   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py",
>> line 6243, in overwrite_suspended_versioning_obj
>>     check_obj_versions(bucket, objname, k, c)
>>   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py",
>> line 6080, in check_obj_versions
>>     eq(keys[i].version_id or 'null', key.version_id)
>> AssertionError: u'yGSvpxjEbJRBE.JL76y4OzeJISqDtmJ' != u'null'
>>
>> Does anyone have an idea which of these PRs [2] might be the culprit?
>>
>> [1]
>>
>> http://pulpito.ceph.com/smithfarm-2017-04-07_09:30:57-rgw-wip-jewel-backports-distro-basic-smithi/
>>
>> [2]
>>
>> https://github.com/ceph/ceph/pulls?q=is%3Aopen+is%3Apr+milestone%3Ajewel+label%3Argw
>>
>>
>> Thanks in advance,
>> Nathan
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
> --
> Nathan Cutler
> Software Engineer Distributed Storage
> SUSE LINUX, s.r.o.
> Tel.: +420 284 084 037
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help debugging RGW bug in jewel 10.2.8 integration branch

[Orit Wasserman]

Hi,
This pr is related to versioning:
https://github.com/ceph/ceph/pull/14368/
Try to see if removing it helps.

Regards,
Orit

On Sun, Apr 9, 2017 at 7:02 PM, Yuri Weinstein <yweinste@redhat.com> wrote:
> Just a note that rgw passed for v10.2.7
> So we may need to split PRs in smaller batches to find offending one.
>
> On Sun, Apr 9, 2017 at 8:28 AM, Nathan Cutler <ncutler@suse.cz> wrote:
>> All the failures appear to be in rgw/verify and rgw/multifs
>>
>>
>> On 04/09/2017 03:33 PM, Nathan Cutler wrote:
>>>
>>> Hello RGW developers:
>>>
>>> Since 10.2.7 is almost out, I'm now running integration tests for
>>> 10.2.8. As usual, there is quite a crop of RGW backports. The
>>> integration branch is "wip-jewel-backports" and the RGW suite [1] is not
>>> looking good: over 50 failures, and most of them are this:
>>>
>>> ======================================================================
>>> FAIL: s3tests.functional.test_s3.test_versioning_obj_suspend_versions
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>>   File
>>>
>>> "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/nose/case.py",
>>> line 197, in runTest
>>>     self.test(*self.arg)
>>>   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py",
>>> line 6385, in test_versioning_obj_suspend_versions
>>>     overwrite_suspended_versioning_obj(bucket, objname, k, c, 'null
>>> content 2')
>>>   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py",
>>> line 6243, in overwrite_suspended_versioning_obj
>>>     check_obj_versions(bucket, objname, k, c)
>>>   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py",
>>> line 6080, in check_obj_versions
>>>     eq(keys[i].version_id or 'null', key.version_id)
>>> AssertionError: u'yGSvpxjEbJRBE.JL76y4OzeJISqDtmJ' != u'null'
>>>
>>> Does anyone have an idea which of these PRs [2] might be the culprit?
>>>
>>> [1]
>>>
>>> http://pulpito.ceph.com/smithfarm-2017-04-07_09:30:57-rgw-wip-jewel-backports-distro-basic-smithi/
>>>
>>> [2]
>>>
>>> https://github.com/ceph/ceph/pulls?q=is%3Aopen+is%3Apr+milestone%3Ajewel+label%3Argw
>>>
>>>
>>> Thanks in advance,
>>> Nathan
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>> --
>> Nathan Cutler
>> Software Engineer Distributed Storage
>> SUSE LINUX, s.r.o.
>> Tel.: +420 284 084 037
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help debugging RGW bug in jewel 10.2.8 integration branch

[Nathan Cutler]

Hi Orit, Yehuda:

Regarding the 
s3tests.functional.test_s3.test_versioning_obj_suspend_versions failure 
that we were seeing in the 10.2.8 integration branch (see quoted email 
below), Abhishek and I bisected it with the conclusion that 
https://github.com/ceph/ceph/pull/13552 is causing the failure. I marked 
that PR "DNM" and will schedule a new round of jewel integration tests.

Nathan

On 04/09/2017 03:33 PM, Nathan Cutler wrote:
> Hello RGW developers:
>
> Since 10.2.7 is almost out, I'm now running integration tests for
> 10.2.8. As usual, there is quite a crop of RGW backports. The
> integration branch is "wip-jewel-backports" and the RGW suite [1] is not
> looking good: over 50 failures, and most of them are this:
>
> ======================================================================
> FAIL: s3tests.functional.test_s3.test_versioning_obj_suspend_versions
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/nose/case.py",
> line 197, in runTest
>     self.test(*self.arg)
>   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py",
> line 6385, in test_versioning_obj_suspend_versions
>     overwrite_suspended_versioning_obj(bucket, objname, k, c, 'null
> content 2')
>   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py",
> line 6243, in overwrite_suspended_versioning_obj
>     check_obj_versions(bucket, objname, k, c)
>   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py",
> line 6080, in check_obj_versions
>     eq(keys[i].version_id or 'null', key.version_id)
> AssertionError: u'yGSvpxjEbJRBE.JL76y4OzeJISqDtmJ' != u'null'
>
> Does anyone have an idea which of these PRs [2] might be the culprit?
>
> [1]
> http://pulpito.ceph.com/smithfarm-2017-04-07_09:30:57-rgw-wip-jewel-backports-distro-basic-smithi/
>
> [2]
> https://github.com/ceph/ceph/pulls?q=is%3Aopen+is%3Apr+milestone%3Ajewel+label%3Argw
>
>
> Thanks in advance,
> Nathan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Nathan Cutler
Software Engineer Distributed Storage
SUSE LINUX, s.r.o.
Tel.: +420 284 084 037