* RadosGW crashing on copy for one specific object
@ 2014-08-19 12:44 Sylvain Munaut
2014-08-19 17:44 ` Yehuda Sadeh
0 siblings, 1 reply; 9+ messages in thread
From: Sylvain Munaut @ 2014-08-19 12:44 UTC (permalink / raw)
To: ceph-devel
Hi,
Today I have an issue when trying to issue a COPY for one object I
have in RGW. It only happens for this object (at least that I noticed
and I did 1000's of COPYs in this batch) and I can do a GET of this
object just fine.
The stack trace :
ceph version 0.80.5-173-g7429f00 (7429f00caaae3c6197e13786c13bc55055d16951)
1: /usr/bin/radosgw() [0x5cd19a]
2: (()+0xfcb0) [0x7f3806340cb0]
3: (()+0x142740) [0x7f3805598740]
4: (ceph::buffer::ptr::append(char const*, unsigned int)+0x43) [0x7f380724f5b3]
5: (ceph::buffer::list::append(char const*, unsigned int)+0x91)
[0x7f3807251cf1]
6: (RGWRados::copy_obj_data(void*, std::string const&, void**, long,
rgw_obj&, rgw_obj&, unsigned long, long*, std::map<std::string,
ceph::buffer::list, std::less<std::string>,
std::allocator<std::pair<std::string const, ceph::buffer::list> > >&,
RGWObjCategory, std::string*, rgw_err*)+0x5cc) [0x52524c]
7: (RGWRados::copy_obj(void*, std::string const&, std::string const&,
std::string const&, req_info*, std::string const&, rgw_obj&, rgw_obj&,
RGWBucketInfo&, RGWBucketInfo&, long*, long const*, long const*, char
const*, char const*, bool, std::map<std::string, ceph::buffer::list,
std::less<std::string>, std::allocator<std::pair<std::string const,
ceph::buffer::list> > >&, RGWObjCategory, std::string*, rgw_err*, void
(*)(long, void*), void*)+0x19b5) [0x5336e5]
8: (RGWCopyObj::execute()+0x2bc) [0x557c0c]
9: /usr/bin/radosgw() [0x4c871c]
10: (RGWFCGXProcess::handle_request(RGWRequest*)+0x9c) [0x4c93fc]
11: (RGWProcess::RGWWQ::_process(RGWRequest*)+0x37) [0x4ca4e7]
12: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x7f38072373f6]
13: (ThreadPool::WorkThread::entry()+0x10) [0x7f3807239200]
14: (()+0x7e9a) [0x7f3806338e9a]
15: (clone()+0x6d) [0x7f380554a73d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
This is the content of the OMAP key in the directory index :
0000000: 0503 3201 0000 4d00 0000 3561 6531 6238 ..2...M...5ae1b8
0000010: 6362 3861 3262 6463 3363 3264 3765 3138 cb8a2bdc3c2d7e18
0000020: 3638 6236 3064 3736 6162 6561 3235 3336 68b60d76abea2536
0000030: 6634 3630 3464 3664 3331 3264 6639 3562 f4604d6d312df95b
0000040: 3731 3934 3730 6662 3362 2f72 656e 6465 719470fb3b/rende
0000050: 722d 696d 6167 651b 0100 0000 0000 0001 r-image.........
0000060: 0303 5800 0000 0107 a903 0000 0000 0032 ..X............2
0000070: 7c3c 5300 0000 0022 0000 0062 6665 3264 |<S...."...bfe2d
0000080: 6638 3632 6439 3266 3730 6365 6362 6564 f862d92f70cecbed
0000090: 6261 6233 3234 6533 6637 382d 3102 0000 bab324e3f78-1...
00000a0: 006b 700a 0000 004b 5020 5333 2055 7365 .kp....KP S3 Use
00000b0: 7209 0000 0069 6d61 6765 2f70 6e67 0000 r....image/png..
00000c0: 0000 4d00 0000 3561 6531 6238 6362 3861 ..M...5ae1b8cb8a
00000d0: 3262 6463 3363 3264 3765 3138 3638 6236 2bdc3c2d7e1868b6
00000e0: 3064 3736 6162 6561 3235 3336 6634 3630 0d76abea2536f460
00000f0: 3464 3664 3331 3264 6639 3562 3731 3934 4d6d312df95b7194
0000100: 3730 6662 3362 2f72 656e 6465 722d 696d 70fb3b/render-im
0000110: 6167 6501 0104 0000 0023 821b 0136 1600 age......#...6..
0000120: 0000 6465 6661 756c 742e 3434 3034 3839 ..default.440489
0000130: 2e31 3539 3035 3436 .1590546
The head object (
default.440489.58258_5ae1b8cb8a2bdc3c2d7e1868b60d76abea2536f4604d6d312df95b719470fb3b/render-image
) is empty (size=0) and the stored manifest (in the user.rgw.manifest
xattr) :
0000000: 0202 5d01 0000 07a9 0300 0000 0000 0100 ..].............
0000010: 0000 0000 0000 0000 0000 0202 4301 0000 ............C...
0000020: 0303 2d01 0000 1800 0000 6b70 2d72 656e ..-.......kp-ren
0000030: 6465 722d 696d 6167 652e 626f 7579 6775 der-image.bouygu
0000040: 6573 0000 0000 0900 0000 6d75 6c74 6970 es........multip
0000050: 6172 747a 0000 005f 6d75 6c74 6970 6172 artz..._multipar
0000060: 745f 3561 6531 6238 6362 3861 3262 6463 t_5ae1b8cb8a2bdc
0000070: 3363 3264 3765 3138 3638 6236 3064 3736 3c2d7e1868b60d76
0000080: 6162 6561 3235 3336 6634 3630 3464 3664 abea2536f4604d6d
0000090: 3331 3264 6639 3562 3731 3934 3730 6662 312df95b719470fb
00000a0: 3362 2f72 656e 6465 722d 696d 6167 652e 3b/render-image.
00000b0: 3872 5344 4438 4d56 4431 4b49 5844 5451 8rSDD8MVD1KIXDTQ
00000c0: 4473 6633 6736 6237 3164 7141 4450 5f2e Dsf3g6b71dqADP_.
00000d0: 3106 037c 0000 0018 0000 006b 702d 7265 1..|.......kp-re
00000e0: 6e64 6572 2d69 6d61 6765 2e62 6f75 7967 nder-image.bouyg
00000f0: 7565 7314 0000 002e 7267 772e 6b70 2d72 ues.....rgw.kp-r
0000100: 656e 6465 722d 696d 6167 6514 0000 0064 ender-image....d
0000110: 6566 6175 6c74 2e34 3430 3438 392e 3538 efault.440489.58
0000120: 3235 3814 0000 0064 6566 6175 6c74 2e34 258....default.4
0000130: 3430 3438 392e 3538 3235 3814 0000 002e 40489.58258.....
0000140: 7267 772e 6b70 2d72 656e 6465 722d 696d rgw.kp-render-im
0000150: 6167 6500 0000 0000 0000 0007 a903 0000 age.............
0000160: 0000 000a ....
The actual file content is in :
default.440489.58258__multipart_5ae1b8cb8a2bdc3c2d7e1868b60d76abea2536f4604d6d312df95b719470fb3b/render-image.8rSDD8MVD1KIXDTQDsf3g6b71dqADP_.1
and is correct (if I fetch the object with rados, it's all there).
Cheers,
Sylvain
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RadosGW crashing on copy for one specific object
2014-08-19 12:44 RadosGW crashing on copy for one specific object Sylvain Munaut
@ 2014-08-19 17:44 ` Yehuda Sadeh
2014-08-20 13:47 ` Sylvain Munaut
0 siblings, 1 reply; 9+ messages in thread
From: Yehuda Sadeh @ 2014-08-19 17:44 UTC (permalink / raw)
To: Sylvain Munaut; +Cc: ceph-devel
What does 'radosgw-admin object stat --bucket=<bucket> --object=<object>' show?
Also, can you bump logging up? debug rgw = 20, debug ms = 1
Thanks,
Yehuda
On Tue, Aug 19, 2014 at 5:44 AM, Sylvain Munaut
<s.munaut@whatever-company.com> wrote:
> Hi,
>
>
> Today I have an issue when trying to issue a COPY for one object I
> have in RGW. It only happens for this object (at least that I noticed
> and I did 1000's of COPYs in this batch) and I can do a GET of this
> object just fine.
>
> The stack trace :
>
> ceph version 0.80.5-173-g7429f00 (7429f00caaae3c6197e13786c13bc55055d16951)
> 1: /usr/bin/radosgw() [0x5cd19a]
> 2: (()+0xfcb0) [0x7f3806340cb0]
> 3: (()+0x142740) [0x7f3805598740]
> 4: (ceph::buffer::ptr::append(char const*, unsigned int)+0x43) [0x7f380724f5b3]
> 5: (ceph::buffer::list::append(char const*, unsigned int)+0x91)
> [0x7f3807251cf1]
> 6: (RGWRados::copy_obj_data(void*, std::string const&, void**, long,
> rgw_obj&, rgw_obj&, unsigned long, long*, std::map<std::string,
> ceph::buffer::list, std::less<std::string>,
> std::allocator<std::pair<std::string const, ceph::buffer::list> > >&,
> RGWObjCategory, std::string*, rgw_err*)+0x5cc) [0x52524c]
> 7: (RGWRados::copy_obj(void*, std::string const&, std::string const&,
> std::string const&, req_info*, std::string const&, rgw_obj&, rgw_obj&,
> RGWBucketInfo&, RGWBucketInfo&, long*, long const*, long const*, char
> const*, char const*, bool, std::map<std::string, ceph::buffer::list,
> std::less<std::string>, std::allocator<std::pair<std::string const,
> ceph::buffer::list> > >&, RGWObjCategory, std::string*, rgw_err*, void
> (*)(long, void*), void*)+0x19b5) [0x5336e5]
> 8: (RGWCopyObj::execute()+0x2bc) [0x557c0c]
> 9: /usr/bin/radosgw() [0x4c871c]
> 10: (RGWFCGXProcess::handle_request(RGWRequest*)+0x9c) [0x4c93fc]
> 11: (RGWProcess::RGWWQ::_process(RGWRequest*)+0x37) [0x4ca4e7]
> 12: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x7f38072373f6]
> 13: (ThreadPool::WorkThread::entry()+0x10) [0x7f3807239200]
> 14: (()+0x7e9a) [0x7f3806338e9a]
> 15: (clone()+0x6d) [0x7f380554a73d]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
>
> This is the content of the OMAP key in the directory index :
>
> 0000000: 0503 3201 0000 4d00 0000 3561 6531 6238 ..2...M...5ae1b8
> 0000010: 6362 3861 3262 6463 3363 3264 3765 3138 cb8a2bdc3c2d7e18
> 0000020: 3638 6236 3064 3736 6162 6561 3235 3336 68b60d76abea2536
> 0000030: 6634 3630 3464 3664 3331 3264 6639 3562 f4604d6d312df95b
> 0000040: 3731 3934 3730 6662 3362 2f72 656e 6465 719470fb3b/rende
> 0000050: 722d 696d 6167 651b 0100 0000 0000 0001 r-image.........
> 0000060: 0303 5800 0000 0107 a903 0000 0000 0032 ..X............2
> 0000070: 7c3c 5300 0000 0022 0000 0062 6665 3264 |<S...."...bfe2d
> 0000080: 6638 3632 6439 3266 3730 6365 6362 6564 f862d92f70cecbed
> 0000090: 6261 6233 3234 6533 6637 382d 3102 0000 bab324e3f78-1...
> 00000a0: 006b 700a 0000 004b 5020 5333 2055 7365 .kp....KP S3 Use
> 00000b0: 7209 0000 0069 6d61 6765 2f70 6e67 0000 r....image/png..
> 00000c0: 0000 4d00 0000 3561 6531 6238 6362 3861 ..M...5ae1b8cb8a
> 00000d0: 3262 6463 3363 3264 3765 3138 3638 6236 2bdc3c2d7e1868b6
> 00000e0: 3064 3736 6162 6561 3235 3336 6634 3630 0d76abea2536f460
> 00000f0: 3464 3664 3331 3264 6639 3562 3731 3934 4d6d312df95b7194
> 0000100: 3730 6662 3362 2f72 656e 6465 722d 696d 70fb3b/render-im
> 0000110: 6167 6501 0104 0000 0023 821b 0136 1600 age......#...6..
> 0000120: 0000 6465 6661 756c 742e 3434 3034 3839 ..default.440489
> 0000130: 2e31 3539 3035 3436 .1590546
>
>
> The head object (
> default.440489.58258_5ae1b8cb8a2bdc3c2d7e1868b60d76abea2536f4604d6d312df95b719470fb3b/render-image
> ) is empty (size=0) and the stored manifest (in the user.rgw.manifest
> xattr) :
>
> 0000000: 0202 5d01 0000 07a9 0300 0000 0000 0100 ..].............
> 0000010: 0000 0000 0000 0000 0000 0202 4301 0000 ............C...
> 0000020: 0303 2d01 0000 1800 0000 6b70 2d72 656e ..-.......kp-ren
> 0000030: 6465 722d 696d 6167 652e 626f 7579 6775 der-image.bouygu
> 0000040: 6573 0000 0000 0900 0000 6d75 6c74 6970 es........multip
> 0000050: 6172 747a 0000 005f 6d75 6c74 6970 6172 artz..._multipar
> 0000060: 745f 3561 6531 6238 6362 3861 3262 6463 t_5ae1b8cb8a2bdc
> 0000070: 3363 3264 3765 3138 3638 6236 3064 3736 3c2d7e1868b60d76
> 0000080: 6162 6561 3235 3336 6634 3630 3464 3664 abea2536f4604d6d
> 0000090: 3331 3264 6639 3562 3731 3934 3730 6662 312df95b719470fb
> 00000a0: 3362 2f72 656e 6465 722d 696d 6167 652e 3b/render-image.
> 00000b0: 3872 5344 4438 4d56 4431 4b49 5844 5451 8rSDD8MVD1KIXDTQ
> 00000c0: 4473 6633 6736 6237 3164 7141 4450 5f2e Dsf3g6b71dqADP_.
> 00000d0: 3106 037c 0000 0018 0000 006b 702d 7265 1..|.......kp-re
> 00000e0: 6e64 6572 2d69 6d61 6765 2e62 6f75 7967 nder-image.bouyg
> 00000f0: 7565 7314 0000 002e 7267 772e 6b70 2d72 ues.....rgw.kp-r
> 0000100: 656e 6465 722d 696d 6167 6514 0000 0064 ender-image....d
> 0000110: 6566 6175 6c74 2e34 3430 3438 392e 3538 efault.440489.58
> 0000120: 3235 3814 0000 0064 6566 6175 6c74 2e34 258....default.4
> 0000130: 3430 3438 392e 3538 3235 3814 0000 002e 40489.58258.....
> 0000140: 7267 772e 6b70 2d72 656e 6465 722d 696d rgw.kp-render-im
> 0000150: 6167 6500 0000 0000 0000 0007 a903 0000 age.............
> 0000160: 0000 000a ....
>
> The actual file content is in :
>
> default.440489.58258__multipart_5ae1b8cb8a2bdc3c2d7e1868b60d76abea2536f4604d6d312df95b719470fb3b/render-image.8rSDD8MVD1KIXDTQDsf3g6b71dqADP_.1
>
> and is correct (if I fetch the object with rados, it's all there).
>
>
> Cheers,
>
> Sylvain
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RadosGW crashing on copy for one specific object
2014-08-19 17:44 ` Yehuda Sadeh
@ 2014-08-20 13:47 ` Sylvain Munaut
2014-08-20 17:25 ` Yehuda Sadeh
0 siblings, 1 reply; 9+ messages in thread
From: Sylvain Munaut @ 2014-08-20 13:47 UTC (permalink / raw)
To: Yehuda Sadeh; +Cc: ceph-devel
Hi,
> What does 'radosgw-admin object stat --bucket=<bucket> --object=<object>' show?
{ "name": "5ae1b8cb8a2bdc3c2d7e1868b60d76abea2536f4604d6d312df95b719470fb3b\/render-image",
"size": 239879,
"policy": { "acl": { "acl_user_map": [
{ "user": "kp",
"acl": 15}],
"acl_group_map": [],
"grant_map": [
{ "id": "kp",
"grant": { "type": { "type": 0},
"id": "kp",
"email": "",
"permission": { "flags": 15},
"name": "KP S3 User",
"group": 0}}]},
"owner": { "id": "kp",
"display_name": "KP S3 User"}},
"etag": "bfe2df862d92f70cecbedbab324e3f78-1",
"tag": "default.440489.1590546",
"manifest": { "objs": [
0,
{ "loc": { "bucket": { "name": "kp-render-image.bouygues",
"pool": ".rgw.kp-render-image",
"data_extra_pool": "",
"index_pool": ".rgw.kp-render-image",
"marker": "default.440489.58258",
"bucket_id": "default.440489.58258"},
"key": "",
"ns": "multipart",
"object":
"_multipart_5ae1b8cb8a2bdc3c2d7e1868b60d76abea2536f4604d6d312df95b719470fb3b\/render-image.8rSDD8MVD1KIXDTQDsf3g6b71dqADP_.1"},
"loc_ofs": 0,
"size": 239879}],
"obj_size": 239879,
"explicit_objs": "true",
"head_obj": { "bucket": { "name": "",
"pool": "",
"data_extra_pool": "",
"index_pool": "",
"marker": "",
"bucket_id": ""},
"key": "",
"ns": "",
"object": ""},
"head_size": 0,
"max_head_size": 0,
"prefix": "",
"tail_bucket": { "name": "",
"pool": "",
"data_extra_pool": "",
"index_pool": "",
"marker": "",
"bucket_id": ""},
"rules": []},
"attrs": { "user.rgw.content_type": "image\/png"}}
> Also, can you bump logging up? debug rgw = 20, debug ms = 1
I'll try, but being on the production cluster, it's constantly
hammered with requests :p
Cheers,
Sylvain
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RadosGW crashing on copy for one specific object
2014-08-20 13:47 ` Sylvain Munaut
@ 2014-08-20 17:25 ` Yehuda Sadeh
2014-08-20 17:33 ` Yehuda Sadeh
0 siblings, 1 reply; 9+ messages in thread
From: Yehuda Sadeh @ 2014-08-20 17:25 UTC (permalink / raw)
To: Sylvain Munaut; +Cc: ceph-devel
On Wed, Aug 20, 2014 at 6:47 AM, Sylvain Munaut
<s.munaut@whatever-company.com> wrote:
> Hi,
>
>
>> What does 'radosgw-admin object stat --bucket=<bucket> --object=<object>' show?
>
> { "name": "5ae1b8cb8a2bdc3c2d7e1868b60d76abea2536f4604d6d312df95b719470fb3b\/render-image",
> "size": 239879,
> "policy": { "acl": { "acl_user_map": [
> { "user": "kp",
> "acl": 15}],
> "acl_group_map": [],
> "grant_map": [
> { "id": "kp",
> "grant": { "type": { "type": 0},
> "id": "kp",
> "email": "",
> "permission": { "flags": 15},
> "name": "KP S3 User",
> "group": 0}}]},
> "owner": { "id": "kp",
> "display_name": "KP S3 User"}},
> "etag": "bfe2df862d92f70cecbedbab324e3f78-1",
> "tag": "default.440489.1590546",
> "manifest": { "objs": [
> 0,
> { "loc": { "bucket": { "name": "kp-render-image.bouygues",
> "pool": ".rgw.kp-render-image",
> "data_extra_pool": "",
> "index_pool": ".rgw.kp-render-image",
> "marker": "default.440489.58258",
> "bucket_id": "default.440489.58258"},
> "key": "",
> "ns": "multipart",
> "object":
> "_multipart_5ae1b8cb8a2bdc3c2d7e1868b60d76abea2536f4604d6d312df95b719470fb3b\/render-image.8rSDD8MVD1KIXDTQDsf3g6b71dqADP_.1"},
> "loc_ofs": 0,
> "size": 239879}],
> "obj_size": 239879,
> "explicit_objs": "true",
> "head_obj": { "bucket": { "name": "",
> "pool": "",
> "data_extra_pool": "",
> "index_pool": "",
> "marker": "",
> "bucket_id": ""},
> "key": "",
> "ns": "",
> "object": ""},
> "head_size": 0,
> "max_head_size": 0,
> "prefix": "",
> "tail_bucket": { "name": "",
> "pool": "",
> "data_extra_pool": "",
> "index_pool": "",
> "marker": "",
> "bucket_id": ""},
> "rules": []},
> "attrs": { "user.rgw.content_type": "image\/png"}}
>
>
>> Also, can you bump logging up? debug rgw = 20, debug ms = 1
>
> I'll try, but being on the production cluster, it's constantly
> hammered with requests :p
>
>
The object has a zero sized head, which is ok, but I think it triggers
an issue. Can you try this:
diff --git a/src/rgw/rgw_dencoder.cc b/src/rgw/rgw_dencoder.cc
index e6dc2ba..fd12c4f 100644
--- a/src/rgw/rgw_dencoder.cc
+++ b/src/rgw/rgw_dencoder.cc
@@ -104,7 +104,7 @@ void RGWObjManifest::obj_iterator::update_location()
const rgw_obj& head = manifest->get_head();
- if (ofs < manifest->get_head_size()) {
+ if (ofs <= manifest->get_head_size()) {
location = head;
return;
}
diff --git a/src/rgw/rgw_rados.cc b/src/rgw/rgw_rados.cc
index 7ca4a9d..877c130 100644
--- a/src/rgw/rgw_rados.cc
+++ b/src/rgw/rgw_rados.cc
@@ -558,7 +558,7 @@ void RGWObjManifest::obj_iterator::operator++()
}
/* are we still pointing at the head? */
- if (ofs < head_size) {
+ if (ofs <= head_size) {
rule_iter = manifest->rules.begin();
RGWObjManifestRule *rule = &rule_iter->second;
ofs = MIN(head_size, obj_size);
Thanks,
Yehuda
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: RadosGW crashing on copy for one specific object
2014-08-20 17:25 ` Yehuda Sadeh
@ 2014-08-20 17:33 ` Yehuda Sadeh
2014-08-20 17:44 ` Yehuda Sadeh
0 siblings, 1 reply; 9+ messages in thread
From: Yehuda Sadeh @ 2014-08-20 17:33 UTC (permalink / raw)
To: Sylvain Munaut; +Cc: ceph-devel
Looking at the code some more, that change isn't going to fix it for
you. I'll send an updated fix soon.
Yehuda
On Wed, Aug 20, 2014 at 10:25 AM, Yehuda Sadeh <yehuda@redhat.com> wrote:
> On Wed, Aug 20, 2014 at 6:47 AM, Sylvain Munaut
> <s.munaut@whatever-company.com> wrote:
>> Hi,
>>
>>
>>> What does 'radosgw-admin object stat --bucket=<bucket> --object=<object>' show?
>>
>> { "name": "5ae1b8cb8a2bdc3c2d7e1868b60d76abea2536f4604d6d312df95b719470fb3b\/render-image",
>> "size": 239879,
>> "policy": { "acl": { "acl_user_map": [
>> { "user": "kp",
>> "acl": 15}],
>> "acl_group_map": [],
>> "grant_map": [
>> { "id": "kp",
>> "grant": { "type": { "type": 0},
>> "id": "kp",
>> "email": "",
>> "permission": { "flags": 15},
>> "name": "KP S3 User",
>> "group": 0}}]},
>> "owner": { "id": "kp",
>> "display_name": "KP S3 User"}},
>> "etag": "bfe2df862d92f70cecbedbab324e3f78-1",
>> "tag": "default.440489.1590546",
>> "manifest": { "objs": [
>> 0,
>> { "loc": { "bucket": { "name": "kp-render-image.bouygues",
>> "pool": ".rgw.kp-render-image",
>> "data_extra_pool": "",
>> "index_pool": ".rgw.kp-render-image",
>> "marker": "default.440489.58258",
>> "bucket_id": "default.440489.58258"},
>> "key": "",
>> "ns": "multipart",
>> "object":
>> "_multipart_5ae1b8cb8a2bdc3c2d7e1868b60d76abea2536f4604d6d312df95b719470fb3b\/render-image.8rSDD8MVD1KIXDTQDsf3g6b71dqADP_.1"},
>> "loc_ofs": 0,
>> "size": 239879}],
>> "obj_size": 239879,
>> "explicit_objs": "true",
>> "head_obj": { "bucket": { "name": "",
>> "pool": "",
>> "data_extra_pool": "",
>> "index_pool": "",
>> "marker": "",
>> "bucket_id": ""},
>> "key": "",
>> "ns": "",
>> "object": ""},
>> "head_size": 0,
>> "max_head_size": 0,
>> "prefix": "",
>> "tail_bucket": { "name": "",
>> "pool": "",
>> "data_extra_pool": "",
>> "index_pool": "",
>> "marker": "",
>> "bucket_id": ""},
>> "rules": []},
>> "attrs": { "user.rgw.content_type": "image\/png"}}
>>
>>
>>> Also, can you bump logging up? debug rgw = 20, debug ms = 1
>>
>> I'll try, but being on the production cluster, it's constantly
>> hammered with requests :p
>>
>>
>
> The object has a zero sized head, which is ok, but I think it triggers
> an issue. Can you try this:
>
> diff --git a/src/rgw/rgw_dencoder.cc b/src/rgw/rgw_dencoder.cc
> index e6dc2ba..fd12c4f 100644
> --- a/src/rgw/rgw_dencoder.cc
> +++ b/src/rgw/rgw_dencoder.cc
> @@ -104,7 +104,7 @@ void RGWObjManifest::obj_iterator::update_location()
>
> const rgw_obj& head = manifest->get_head();
>
> - if (ofs < manifest->get_head_size()) {
> + if (ofs <= manifest->get_head_size()) {
> location = head;
> return;
> }
> diff --git a/src/rgw/rgw_rados.cc b/src/rgw/rgw_rados.cc
> index 7ca4a9d..877c130 100644
> --- a/src/rgw/rgw_rados.cc
> +++ b/src/rgw/rgw_rados.cc
> @@ -558,7 +558,7 @@ void RGWObjManifest::obj_iterator::operator++()
> }
>
> /* are we still pointing at the head? */
> - if (ofs < head_size) {
> + if (ofs <= head_size) {
> rule_iter = manifest->rules.begin();
> RGWObjManifestRule *rule = &rule_iter->second;
> ofs = MIN(head_size, obj_size);
>
>
> Thanks,
> Yehuda
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RadosGW crashing on copy for one specific object
2014-08-20 17:33 ` Yehuda Sadeh
@ 2014-08-20 17:44 ` Yehuda Sadeh
2014-08-21 9:38 ` Sylvain Munaut
0 siblings, 1 reply; 9+ messages in thread
From: Yehuda Sadeh @ 2014-08-20 17:44 UTC (permalink / raw)
To: Sylvain Munaut; +Cc: ceph-devel
If by any chance you applied the previous patch, revert it, as it's
wrong. This might fix the issue:
diff --git a/src/rgw/rgw_rados.h b/src/rgw/rgw_rados.h
index d50fb59..0f13590 100644
--- a/src/rgw/rgw_rados.h
+++ b/src/rgw/rgw_rados.h
@@ -298,6 +298,9 @@ public:
bool has_tail() {
if (explicit_objs) {
+ if (objs.size() == 1) {
+ return head_obj != objs[0];
+ }
return (objs.size() >= 2);
}
return (obj_size > head_size);
Let me know if it works for you.
Thanks,
Yehuda
On Wed, Aug 20, 2014 at 10:33 AM, Yehuda Sadeh <yehuda@redhat.com> wrote:
> Looking at the code some more, that change isn't going to fix it for
> you. I'll send an updated fix soon.
>
> Yehuda
>
> On Wed, Aug 20, 2014 at 10:25 AM, Yehuda Sadeh <yehuda@redhat.com> wrote:
>> On Wed, Aug 20, 2014 at 6:47 AM, Sylvain Munaut
>> <s.munaut@whatever-company.com> wrote:
>>> Hi,
>>>
>>>
>>>> What does 'radosgw-admin object stat --bucket=<bucket> --object=<object>' show?
>>>
>>> { "name": "5ae1b8cb8a2bdc3c2d7e1868b60d76abea2536f4604d6d312df95b719470fb3b\/render-image",
>>> "size": 239879,
>>> "policy": { "acl": { "acl_user_map": [
>>> { "user": "kp",
>>> "acl": 15}],
>>> "acl_group_map": [],
>>> "grant_map": [
>>> { "id": "kp",
>>> "grant": { "type": { "type": 0},
>>> "id": "kp",
>>> "email": "",
>>> "permission": { "flags": 15},
>>> "name": "KP S3 User",
>>> "group": 0}}]},
>>> "owner": { "id": "kp",
>>> "display_name": "KP S3 User"}},
>>> "etag": "bfe2df862d92f70cecbedbab324e3f78-1",
>>> "tag": "default.440489.1590546",
>>> "manifest": { "objs": [
>>> 0,
>>> { "loc": { "bucket": { "name": "kp-render-image.bouygues",
>>> "pool": ".rgw.kp-render-image",
>>> "data_extra_pool": "",
>>> "index_pool": ".rgw.kp-render-image",
>>> "marker": "default.440489.58258",
>>> "bucket_id": "default.440489.58258"},
>>> "key": "",
>>> "ns": "multipart",
>>> "object":
>>> "_multipart_5ae1b8cb8a2bdc3c2d7e1868b60d76abea2536f4604d6d312df95b719470fb3b\/render-image.8rSDD8MVD1KIXDTQDsf3g6b71dqADP_.1"},
>>> "loc_ofs": 0,
>>> "size": 239879}],
>>> "obj_size": 239879,
>>> "explicit_objs": "true",
>>> "head_obj": { "bucket": { "name": "",
>>> "pool": "",
>>> "data_extra_pool": "",
>>> "index_pool": "",
>>> "marker": "",
>>> "bucket_id": ""},
>>> "key": "",
>>> "ns": "",
>>> "object": ""},
>>> "head_size": 0,
>>> "max_head_size": 0,
>>> "prefix": "",
>>> "tail_bucket": { "name": "",
>>> "pool": "",
>>> "data_extra_pool": "",
>>> "index_pool": "",
>>> "marker": "",
>>> "bucket_id": ""},
>>> "rules": []},
>>> "attrs": { "user.rgw.content_type": "image\/png"}}
>>>
>>>
>>>> Also, can you bump logging up? debug rgw = 20, debug ms = 1
>>>
>>> I'll try, but being on the production cluster, it's constantly
>>> hammered with requests :p
>>>
>>>
>>
>> The object has a zero sized head, which is ok, but I think it triggers
>> an issue. Can you try this:
>>
>> diff --git a/src/rgw/rgw_dencoder.cc b/src/rgw/rgw_dencoder.cc
>> index e6dc2ba..fd12c4f 100644
>> --- a/src/rgw/rgw_dencoder.cc
>> +++ b/src/rgw/rgw_dencoder.cc
>> @@ -104,7 +104,7 @@ void RGWObjManifest::obj_iterator::update_location()
>>
>> const rgw_obj& head = manifest->get_head();
>>
>> - if (ofs < manifest->get_head_size()) {
>> + if (ofs <= manifest->get_head_size()) {
>> location = head;
>> return;
>> }
>> diff --git a/src/rgw/rgw_rados.cc b/src/rgw/rgw_rados.cc
>> index 7ca4a9d..877c130 100644
>> --- a/src/rgw/rgw_rados.cc
>> +++ b/src/rgw/rgw_rados.cc
>> @@ -558,7 +558,7 @@ void RGWObjManifest::obj_iterator::operator++()
>> }
>>
>> /* are we still pointing at the head? */
>> - if (ofs < head_size) {
>> + if (ofs <= head_size) {
>> rule_iter = manifest->rules.begin();
>> RGWObjManifestRule *rule = &rule_iter->second;
>> ofs = MIN(head_size, obj_size);
>>
>>
>> Thanks,
>> Yehuda
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: RadosGW crashing on copy for one specific object
2014-08-20 17:44 ` Yehuda Sadeh
@ 2014-08-21 9:38 ` Sylvain Munaut
2014-08-21 20:39 ` Yehuda Sadeh
0 siblings, 1 reply; 9+ messages in thread
From: Sylvain Munaut @ 2014-08-21 9:38 UTC (permalink / raw)
To: Yehuda Sadeh; +Cc: ceph-devel
Hi,
> If by any chance you applied the previous patch, revert it, as it's
> wrong. This might fix the issue:
>
> diff --git a/src/rgw/rgw_rados.h b/src/rgw/rgw_rados.h
> index d50fb59..0f13590 100644
> --- a/src/rgw/rgw_rados.h
> +++ b/src/rgw/rgw_rados.h
> @@ -298,6 +298,9 @@ public:
>
> bool has_tail() {
> if (explicit_objs) {
> + if (objs.size() == 1) {
> + return head_obj != objs[0];
> + }
> return (objs.size() >= 2);
> }
> return (obj_size > head_size);
>
> Let me know if it works for you.
It doesn't build at all.
rgw/rgw_rados.h: In member function ‘bool RGWObjManifest::has_tail()’:
rgw/rgw_rados.h:302:34: error: no match for ‘operator!=’ in
‘((RGWObjManifest*)this)->RGWObjManifest::head_obj !=
((RGWObjManifest*)this)->RGWObjManifest::objs.std::map<_Key, _Tp,
_Compare, _Alloc>::operator[] [with _Key = long unsigned int, _Tp =
RGWObjManifestPart, _Compare = std::less<long unsigned int>, _Alloc =
std::allocator<std::pair<const long unsigned int, RGWObjManifestPart>
>, std::map<_Key, _Tp, _Compare, _Alloc>::mapped_type =
RGWObjManifestPart, std::map<_Key, _Tp, _Compare, _Alloc>::key_type =
long unsigned int]((* &0u))’
...
Cheers,
Sylvain
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: RadosGW crashing on copy for one specific object
2014-08-21 9:38 ` Sylvain Munaut
@ 2014-08-21 20:39 ` Yehuda Sadeh
2014-08-25 9:00 ` Sylvain Munaut
0 siblings, 1 reply; 9+ messages in thread
From: Yehuda Sadeh @ 2014-08-21 20:39 UTC (permalink / raw)
To: Sylvain Munaut; +Cc: ceph-devel
Try this:
diff --git a/src/rgw/rgw_rados.h b/src/rgw/rgw_rados.h
index ed8f02d..0042df2 100644
--- a/src/rgw/rgw_rados.h
+++ b/src/rgw/rgw_rados.h
@@ -306,6 +306,11 @@ public:
bool has_tail() {
if (explicit_objs) {
+ if (objs.size() == 1) {
+ map<uint64_t, RGWObjManifestPart>::iterator iter = objs.begin();
+ rgw_obj& obj = iter->second.loc;
+ return head_obj.object != obj.object;
+ }
return (objs.size() >= 2);
}
return (obj_size > head_size);
Thanks,
Yehuda
On Thu, Aug 21, 2014 at 2:38 AM, Sylvain Munaut
<s.munaut@whatever-company.com> wrote:
> Hi,
>
>
>> If by any chance you applied the previous patch, revert it, as it's
>> wrong. This might fix the issue:
>>
>> diff --git a/src/rgw/rgw_rados.h b/src/rgw/rgw_rados.h
>> index d50fb59..0f13590 100644
>> --- a/src/rgw/rgw_rados.h
>> +++ b/src/rgw/rgw_rados.h
>> @@ -298,6 +298,9 @@ public:
>>
>> bool has_tail() {
>> if (explicit_objs) {
>> + if (objs.size() == 1) {
>> + return head_obj != objs[0];
>> + }
>> return (objs.size() >= 2);
>> }
>> return (obj_size > head_size);
>>
>> Let me know if it works for you.
>
> It doesn't build at all.
>
> rgw/rgw_rados.h: In member function ‘bool RGWObjManifest::has_tail()’:
> rgw/rgw_rados.h:302:34: error: no match for ‘operator!=’ in
> ‘((RGWObjManifest*)this)->RGWObjManifest::head_obj !=
> ((RGWObjManifest*)this)->RGWObjManifest::objs.std::map<_Key, _Tp,
> _Compare, _Alloc>::operator[] [with _Key = long unsigned int, _Tp =
> RGWObjManifestPart, _Compare = std::less<long unsigned int>, _Alloc =
> std::allocator<std::pair<const long unsigned int, RGWObjManifestPart>
>>, std::map<_Key, _Tp, _Compare, _Alloc>::mapped_type =
> RGWObjManifestPart, std::map<_Key, _Tp, _Compare, _Alloc>::key_type =
> long unsigned int]((* &0u))’
> ...
>
>
>
> Cheers,
>
> Sylvain
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: RadosGW crashing on copy for one specific object
2014-08-21 20:39 ` Yehuda Sadeh
@ 2014-08-25 9:00 ` Sylvain Munaut
0 siblings, 0 replies; 9+ messages in thread
From: Sylvain Munaut @ 2014-08-25 9:00 UTC (permalink / raw)
To: Yehuda Sadeh; +Cc: ceph-devel
Hi Yehuda,
> diff --git a/src/rgw/rgw_rados.h b/src/rgw/rgw_rados.h
> index ed8f02d..0042df2 100644
> --- a/src/rgw/rgw_rados.h
> +++ b/src/rgw/rgw_rados.h
> @@ -306,6 +306,11 @@ public:
>
> bool has_tail() {
> if (explicit_objs) {
> + if (objs.size() == 1) {
> + map<uint64_t, RGWObjManifestPart>::iterator iter = objs.begin();
> + rgw_obj& obj = iter->second.loc;
> + return head_obj.object != obj.object;
> + }
> return (objs.size() >= 2);
> }
> return (obj_size > head_size);
>
I confirm that this fixes the issue.
I was able to copy the key and read the resulting copied key.
Cheers,
Sylvain
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2014-08-25 9:00 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-08-19 12:44 RadosGW crashing on copy for one specific object Sylvain Munaut
2014-08-19 17:44 ` Yehuda Sadeh
2014-08-20 13:47 ` Sylvain Munaut
2014-08-20 17:25 ` Yehuda Sadeh
2014-08-20 17:33 ` Yehuda Sadeh
2014-08-20 17:44 ` Yehuda Sadeh
2014-08-21 9:38 ` Sylvain Munaut
2014-08-21 20:39 ` Yehuda Sadeh
2014-08-25 9:00 ` Sylvain Munaut
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.