radosgw-agent failed to parse

radosgw-agent failed to parse

[ghislain.chevalier]

Hi all,
 
Context : Ubuntu 14.04 TLS firefly 0.80.8
 
I sent this post in ceph-users (identical subject) because I recently encountered the same issue.
Maybe I missed something between July and January...
 
I found that the http request wasn't correctly formed by /usr/lib/python2.7/dist-packages/radosgw_agent/client.py
 
I did the changes below
#    url = '{protocol}://{host}{path}'.format(protocol=request.protocol,
#                                             host=request.host,
#                                             path=request.path)
     url = '{path}'.format(protocol="", host="", path=request.path)
 
The request is then correctly formed and sent.
 
I still have problems setting the federation between 2 zones of 2 ceph clusters in 2 regions.

I go on investigating.

Best regards


_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

Re: radosgw-agent failed to parse

[Alfredo Deza]

What version of the agent are you using? And when it fails, how does
the log output look?

On Mon, Feb 2, 2015 at 8:56 AM,  <ghislain.chevalier@orange.com> wrote:
> Hi all,
>
> Context : Ubuntu 14.04 TLS firefly 0.80.8
>
> I sent this post in ceph-users (identical subject) because I recently encountered the same issue.
> Maybe I missed something between July and January...
>
> I found that the http request wasn't correctly formed by /usr/lib/python2.7/dist-packages/radosgw_agent/client.py
>
> I did the changes below
> #    url = '{protocol}://{host}{path}'.format(protocol=request.protocol,
> #                                             host=request.host,
> #                                             path=request.path)
>      url = '{path}'.format(protocol="", host="", path=request.path)
>
> The request is then correctly formed and sent.
>
> I still have problems setting the federation between 2 zones of 2 ceph clusters in 2 regions.
>
> I go on investigating.
>
> Best regards
>
>
> _________________________________________________________________________________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: radosgw-agent failed to parse

[ghislain.chevalier]

HI,

Thx for replying

According to dpkg -l, it's 1.2.1.

I noticed that the URL is malformed when launching using directly radosgw-agent -c <syncfile> 
but well formed when launching using service radosgw-agent start <syncfile>

best regards

-----Message d'origine-----
De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la part de Alfredo Deza
Envoyé : lundi 2 février 2015 15:18
À : CHEVALIER Ghislain IMT/OLPS
Cc : ceph-devel@vger.kernel.org
Objet : Re: [Ceph-devel] radosgw-agent failed to parse

What version of the agent are you using? And when it fails, how does the log output look?

On Mon, Feb 2, 2015 at 8:56 AM,  <ghislain.chevalier@orange.com> wrote:
> Hi all,
>
> Context : Ubuntu 14.04 TLS firefly 0.80.8
>
> I sent this post in ceph-users (identical subject) because I recently encountered the same issue.
> Maybe I missed something between July and January...
>
> I found that the http request wasn't correctly formed by 
> /usr/lib/python2.7/dist-packages/radosgw_agent/client.py
>
> I did the changes below
> #    url = '{protocol}://{host}{path}'.format(protocol=request.protocol,
> #                                             host=request.host,
> #                                             path=request.path)
>      url = '{path}'.format(protocol="", host="", path=request.path)
>
> The request is then correctly formed and sent.
>
> I still have problems setting the federation between 2 zones of 2 ceph clusters in 2 regions.
>
> I go on investigating.
>
> Best regards
>
>
> ______________________________________________________________________
> ___________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
> exploites ou copies sans autorisation. Si vous avez recu ce message 
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or 
> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

Re: radosgw-agent failed to parse

[Alfredo Deza]

That sounds suspicious because we haven't packaged a 1.2.1 release.

The latest on PyPI is 1.2, which is the same in the master branch.

Would you mind sending logs that show *how* the url is malformed (both
when it is OK and when it is not) ?

On Mon, Feb 2, 2015 at 9:53 AM,  <ghislain.chevalier@orange.com> wrote:
> HI,
>
> Thx for replying
>
> According to dpkg -l, it's 1.2.1.
>
> I noticed that the URL is malformed when launching using directly radosgw-agent -c <syncfile>
> but well formed when launching using service radosgw-agent start <syncfile>
>
> best regards
>
> -----Message d'origine-----
> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la part de Alfredo Deza
> Envoyé : lundi 2 février 2015 15:18
> À : CHEVALIER Ghislain IMT/OLPS
> Cc : ceph-devel@vger.kernel.org
> Objet : Re: [Ceph-devel] radosgw-agent failed to parse
>
> What version of the agent are you using? And when it fails, how does the log output look?
>
> On Mon, Feb 2, 2015 at 8:56 AM,  <ghislain.chevalier@orange.com> wrote:
>> Hi all,
>>
>> Context : Ubuntu 14.04 TLS firefly 0.80.8
>>
>> I sent this post in ceph-users (identical subject) because I recently encountered the same issue.
>> Maybe I missed something between July and January...
>>
>> I found that the http request wasn't correctly formed by
>> /usr/lib/python2.7/dist-packages/radosgw_agent/client.py
>>
>> I did the changes below
>> #    url = '{protocol}://{host}{path}'.format(protocol=request.protocol,
>> #                                             host=request.host,
>> #                                             path=request.path)
>>      url = '{path}'.format(protocol="", host="", path=request.path)
>>
>> The request is then correctly formed and sent.
>>
>> I still have problems setting the federation between 2 zones of 2 ceph clusters in 2 regions.
>>
>> I go on investigating.
>>
>> Best regards
>>
>>
>> ______________________________________________________________________
>> ___________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations
>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
>> exploites ou copies sans autorisation. Si vous avez recu ce message
>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or
>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>> Thank you.
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at  http://vger.kernel.org/majordomo-info.html
>
> _________________________________________________________________________________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: radosgw-agent failed to parse

[ghislain.chevalier]

OK I will send them asap

Logs are not very verbose.. Can I set a debug mode?

-----Message d'origine-----
De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la part de Alfredo Deza
Envoyé : lundi 2 février 2015 16:59
À : CHEVALIER Ghislain IMT/OLPS
Cc : 
Objet : Re: [Ceph-devel] radosgw-agent failed to parse

That sounds suspicious because we haven't packaged a 1.2.1 release.

The latest on PyPI is 1.2, which is the same in the master branch.

Would you mind sending logs that show *how* the url is malformed (both when it is OK and when it is not) ?

On Mon, Feb 2, 2015 at 9:53 AM,  <ghislain.chevalier@orange.com> wrote:
> HI,
>
> Thx for replying
>
> According to dpkg -l, it's 1.2.1.
>
> I noticed that the URL is malformed when launching using directly 
> radosgw-agent -c <syncfile> but well formed when launching using 
> service radosgw-agent start <syncfile>
>
> best regards
>
> -----Message d'origine-----
> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la 
> part de Alfredo Deza Envoyé : lundi 2 février 2015 15:18 À : CHEVALIER 
> Ghislain IMT/OLPS Cc : ceph-devel@vger.kernel.org Objet : Re: 
> [Ceph-devel] radosgw-agent failed to parse
>
> What version of the agent are you using? And when it fails, how does the log output look?
>
> On Mon, Feb 2, 2015 at 8:56 AM,  <ghislain.chevalier@orange.com> wrote:
>> Hi all,
>>
>> Context : Ubuntu 14.04 TLS firefly 0.80.8
>>
>> I sent this post in ceph-users (identical subject) because I recently encountered the same issue.
>> Maybe I missed something between July and January...
>>
>> I found that the http request wasn't correctly formed by 
>> /usr/lib/python2.7/dist-packages/radosgw_agent/client.py
>>
>> I did the changes below
>> #    url = '{protocol}://{host}{path}'.format(protocol=request.protocol,
>> #                                             host=request.host,
>> #                                             path=request.path)
>>      url = '{path}'.format(protocol="", host="", path=request.path)
>>
>> The request is then correctly formed and sent.
>>
>> I still have problems setting the federation between 2 zones of 2 ceph clusters in 2 regions.
>>
>> I go on investigating.
>>
>> Best regards
>>
>>
>> _____________________________________________________________________
>> _ ___________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations 
>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
>> exploites ou copies sans autorisation. Si vous avez recu ce message 
>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or 
>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>> Thank you.
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo 
>> info at  http://vger.kernel.org/majordomo-info.html
>
> ______________________________________________________________________
> ___________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
> exploites ou copies sans autorisation. Si vous avez recu ce message 
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or 
> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

Re: radosgw-agent failed to parse

[Alfredo Deza]

If you use the `-v` flag you will get very verbose output.

On Mon, Feb 2, 2015 at 12:12 PM,  <ghislain.chevalier@orange.com> wrote:
> OK I will send them asap
>
> Logs are not very verbose.. Can I set a debug mode?
>
> -----Message d'origine-----
> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la part de Alfredo Deza
> Envoyé : lundi 2 février 2015 16:59
> À : CHEVALIER Ghislain IMT/OLPS
> Cc :
> Objet : Re: [Ceph-devel] radosgw-agent failed to parse
>
> That sounds suspicious because we haven't packaged a 1.2.1 release.
>
> The latest on PyPI is 1.2, which is the same in the master branch.
>
> Would you mind sending logs that show *how* the url is malformed (both when it is OK and when it is not) ?
>
> On Mon, Feb 2, 2015 at 9:53 AM,  <ghislain.chevalier@orange.com> wrote:
>> HI,
>>
>> Thx for replying
>>
>> According to dpkg -l, it's 1.2.1.
>>
>> I noticed that the URL is malformed when launching using directly
>> radosgw-agent -c <syncfile> but well formed when launching using
>> service radosgw-agent start <syncfile>
>>
>> best regards
>>
>> -----Message d'origine-----
>> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la
>> part de Alfredo Deza Envoyé : lundi 2 février 2015 15:18 À : CHEVALIER
>> Ghislain IMT/OLPS Cc : ceph-devel@vger.kernel.org Objet : Re:
>> [Ceph-devel] radosgw-agent failed to parse
>>
>> What version of the agent are you using? And when it fails, how does the log output look?
>>
>> On Mon, Feb 2, 2015 at 8:56 AM,  <ghislain.chevalier@orange.com> wrote:
>>> Hi all,
>>>
>>> Context : Ubuntu 14.04 TLS firefly 0.80.8
>>>
>>> I sent this post in ceph-users (identical subject) because I recently encountered the same issue.
>>> Maybe I missed something between July and January...
>>>
>>> I found that the http request wasn't correctly formed by
>>> /usr/lib/python2.7/dist-packages/radosgw_agent/client.py
>>>
>>> I did the changes below
>>> #    url = '{protocol}://{host}{path}'.format(protocol=request.protocol,
>>> #                                             host=request.host,
>>> #                                             path=request.path)
>>>      url = '{path}'.format(protocol="", host="", path=request.path)
>>>
>>> The request is then correctly formed and sent.
>>>
>>> I still have problems setting the federation between 2 zones of 2 ceph clusters in 2 regions.
>>>
>>> I go on investigating.
>>>
>>> Best regards
>>>
>>>
>>> _____________________________________________________________________
>>> _ ___________________________________________________
>>>
>>> Ce message et ses pieces jointes peuvent contenir des informations
>>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
>>> exploites ou copies sans autorisation. Si vous avez recu ce message
>>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>>
>>> This message and its attachments may contain confidential or
>>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>>> Thank you.
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>> in the body of a message to majordomo@vger.kernel.org More majordomo
>>> info at  http://vger.kernel.org/majordomo-info.html
>>
>> ______________________________________________________________________
>> ___________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations
>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
>> exploites ou copies sans autorisation. Si vous avez recu ce message
>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or
>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>> Thank you.
>>
>
> _________________________________________________________________________________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: radosgw-agent failed to parse

[ghislain.chevalier]

Hi Alfredo,

Here are the logs you requested using the original client.py python script.

----------------------------------------------------------------------
DIRECT LAUNCH USING CLI:KO
radosgw-agent -v -c /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
the standard output is also written in the log file configured in the synchro file

DEBUG:boto:Using access key provided by client.
DEBUG:boto:Using secret key provided by client.
DEBUG:boto:StringToSign:
GET


Tue, 03 Feb 2015 09:37:54 GMT
/admin/config
DEBUG:boto:Signature:
AWS SBF7DQZ1ESR6Y34225H8:9i6/4do7tDpViolN1XdxgefMs3U=
DEBUG:boto:url = 'http://openstorage.ushttp://openstorage.us/admin/config'  
params={}
headers={'Date': 'Tue, 03 Feb 2015 09:37:54 GMT', 'Content-Length': '0', 'Authorization': 'AWS SBF7DQZ1ESR6Y34225H8:9i6/4do7tDpViolN1XdxgefMs3U=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.11.0-26-generic'}
data=None
ERROR:root:Could not retrieve region map from destination
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/cli.py", line 269, in main
    region_map = client.get_region_map(dest_conn)
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 391, in get_region_map
    region_map = request(connection, 'get', 'admin/config')
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 153, in request
    result = handler(url, params=params, headers=request.headers, data=data)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
    prep = self.prepare_request(req)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
    self.prepare_url(url, params)
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 334, in prepare_url
    scheme, auth, host, port, path, query, fragment = parse_url(url)
  File "/usr/lib/python2.7/dist-packages/urllib3/util.py", line 390, in parse_url
    raise LocationParseError("Failed to parse: %s" % url)
LocationParseError: Failed to parse: Failed to parse: openstorage.ushttp:

--------------------------------------------------------------------------------
LAUNCH USING /etc/init.d/radosgw-agent shell script:KO
/etc/init.d/radosgw-agent start /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
Standard output:
Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct

Log File :
2015-02-03T10:51:00.794 4735:ERROR:root:Could not retrieve region map from destination
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/cli.py", line 269, in main
    region_map = client.get_region_map(dest_conn)
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 391, in get_region_map
    region_map = request(connection, 'get', 'admin/config')
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 153, in request
    result = handler(url, params=params, headers=request.headers, data=data)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
    prep = self.prepare_request(req)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
    self.prepare_url(url, params)
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 334, in prepare_url
    scheme, auth, host, port, path, query, fragment = parse_url(url)
  File "/usr/lib/python2.7/dist-packages/urllib3/util.py", line 390, in parse_url
    raise LocationParseError("Failed to parse: %s" % url)
LocationParseError: Failed to parse: Failed to parse: openstorage.ushttp:

-------------------------------------------------------------------------------------------
LAUNCH AS A SERVICE: OK
service radosgw-agent start /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
standard output:
Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct

Log file:
2015-02-03T10:42:02.878 3162:INFO:urllib3.connectionpool:Starting new HTTP connection (1): openstorage.us

----------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------------------------------------
Here are the logs you requested using the altered client.py python script.

DIRECT LAUNCH USING CLI:OK
radosgw-agent -v -c /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
the standard output is also witten in the log file configured in the synchro file

2015-02-03T11:00:40.447 6158:DEBUG:boto:Using access key provided by client.
2015-02-03T11:00:40.447 6158:DEBUG:boto:Using secret key provided by client.
2015-02-03T11:00:40.448 6158:DEBUG:boto:StringToSign:
GET


Tue, 03 Feb 2015 10:00:40 GMT
/admin/config
2015-02-03T11:00:40.448 6158:DEBUG:boto:Signature:
AWS SBF7DQZ1ESR6Y34225H8:LUEBUrR9z1Bgbrj6F/XGrGF+8BM=
2015-02-03T11:00:40.448 6158:DEBUG:boto:url = 'http://openstorage.us/admin/config'
params={}
headers={'Date': 'Tue, 03 Feb 2015 10:00:40 GMT', 'Content-Length': '0', 'Authorization': 'AWS SBF7DQZ1ESR6Y34225H8:LUEBUrR9z1Bgbrj6F/XGrGF+8BM=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.11.0-26-generic'}
data=None
2015-02-03T11:00:40.457 6158:INFO:urllib3.connectionpool:Starting new HTTP connection (1): proxy
2015-02-03T11:00:40.467 6158:DEBUG:urllib3.connectionpool:Setting read timeout to None
2015-02-03T11:00:40.848 6158:DEBUG:urllib3.connectionpool:"GET http://openstorage.us/admin/config HTTP/1.1" 200 None

--------------------------------------------------------------------------------
LAUNCH USING /etc/init.d/radosgw-agent shell script:KO
/etc/init.d/radosgw-agent start /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
Standard output:
Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct

Log File :
2015-02-03T11:07:49.271 7513:INFO:urllib3.connectionpool:Starting new HTTP connection (1): proxy
2015-02-03T11:07:49.654 7513:INFO:urllib3.connectionpool:Starting new HTTP connection (1): proxy
2015-02-03T11:07:50.469 7513:ERROR:radosgw_agent.sync:finding number of shards failed
2015-02-03T11:07:50.470 7513:WARNING:radosgw_agent.sync:error preparing for sync, will retry. Traceback:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 26, in prepare_sync
    syncer.prepare()
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 202, in prepare
    self.init_num_shards()
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 73, in init_num_shards
    self.num_shards = client.num_log_shards(self.src_conn, self.type)
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 285, in num_log_shards
    out = request(connection, 'get', 'admin/log', dict(type=shard_type))
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 160, in request
    check_result_status(result)
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 116, in check_result_status
    HttpError)(result.status_code, result.content)
NotFound: Http error code 404 content <HTML><HEAD>


-------------------------------------------------------------------------------------------
LAUNCH AS A SERVICE:KO
service radosgw-agent start /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
standard output:
Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct

Log file:
2015-02-03T11:04:25.435 6872:ERROR:root:Could not retrieve region map from destination
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/cli.py", line 269, in main
    region_map = client.get_region_map(dest_conn)
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 396, in get_region_map
    region_map = request(connection, 'get', 'admin/config')
  File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 158, in request
    result = handler(url, params=params, headers=request.headers, data=data)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
    prep = self.prepare_request(req)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
    self.prepare_url(url, params)
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 338, in prepare_url
    "Perhaps you meant http://{0}?".format(url))
MissingSchema: Invalid URL u'/admin/config': No schema supplied. Perhaps you meant http:///admin/config?



Best regards

-----Message d'origine-----
De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la part de Alfredo Deza
Envoyé : lundi 2 février 2015 19:42
À : CHEVALIER Ghislain IMT/OLPS
Cc : ceph-devel@vger.kernel.org
Objet : Re: [Ceph-devel] radosgw-agent failed to parse

If you use the `-v` flag you will get very verbose output.

On Mon, Feb 2, 2015 at 12:12 PM,  <ghislain.chevalier@orange.com> wrote:
> OK I will send them asap
>
> Logs are not very verbose.. Can I set a debug mode?
>
> -----Message d'origine-----
> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la 
> part de Alfredo Deza Envoyé : lundi 2 février 2015 16:59 À : CHEVALIER 
> Ghislain IMT/OLPS Cc :
> Objet : Re: [Ceph-devel] radosgw-agent failed to parse
>
> That sounds suspicious because we haven't packaged a 1.2.1 release.
>
> The latest on PyPI is 1.2, which is the same in the master branch.
>
> Would you mind sending logs that show *how* the url is malformed (both when it is OK and when it is not) ?
>
> On Mon, Feb 2, 2015 at 9:53 AM,  <ghislain.chevalier@orange.com> wrote:
>> HI,
>>
>> Thx for replying
>>
>> According to dpkg -l, it's 1.2.1.
>>
>> I noticed that the URL is malformed when launching using directly 
>> radosgw-agent -c <syncfile> but well formed when launching using 
>> service radosgw-agent start <syncfile>
>>
>> best regards
>>
>> -----Message d'origine-----
>> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la 
>> part de Alfredo Deza Envoyé : lundi 2 février 2015 15:18 À : 
>> CHEVALIER Ghislain IMT/OLPS Cc : ceph-devel@vger.kernel.org Objet : Re:
>> [Ceph-devel] radosgw-agent failed to parse
>>
>> What version of the agent are you using? And when it fails, how does the log output look?
>>
>> On Mon, Feb 2, 2015 at 8:56 AM,  <ghislain.chevalier@orange.com> wrote:
>>> Hi all,
>>>
>>> Context : Ubuntu 14.04 TLS firefly 0.80.8
>>>
>>> I sent this post in ceph-users (identical subject) because I recently encountered the same issue.
>>> Maybe I missed something between July and January...
>>>
>>> I found that the http request wasn't correctly formed by 
>>> /usr/lib/python2.7/dist-packages/radosgw_agent/client.py
>>>
>>> I did the changes below
>>> #    url = '{protocol}://{host}{path}'.format(protocol=request.protocol,
>>> #                                             host=request.host,
>>> #                                             path=request.path)
>>>      url = '{path}'.format(protocol="", host="", path=request.path)
>>>
>>> The request is then correctly formed and sent.
>>>
>>> I still have problems setting the federation between 2 zones of 2 ceph clusters in 2 regions.
>>>
>>> I go on investigating.
>>>
>>> Best regards
>>>
>>>
>>> ____________________________________________________________________
>>> _ _ ___________________________________________________
>>>
>>> Ce message et ses pieces jointes peuvent contenir des informations 
>>> confidentielles ou privilegiees et ne doivent donc pas etre 
>>> diffuses, exploites ou copies sans autorisation. Si vous avez recu 
>>> ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>>
>>> This message and its attachments may contain confidential or 
>>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>>> Thank you.
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>> in the body of a message to majordomo@vger.kernel.org More majordomo 
>>> info at  http://vger.kernel.org/majordomo-info.html
>>
>> _____________________________________________________________________
>> _ ___________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations 
>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
>> exploites ou copies sans autorisation. Si vous avez recu ce message 
>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or 
>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>> Thank you.
>>
>
> ______________________________________________________________________
> ___________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
> exploites ou copies sans autorisation. Si vous avez recu ce message 
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or 
> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

Re: radosgw-agent failed to parse

[Alfredo Deza]

This tells me that you are not using the latest released version of
the agent because we dropped the requirement for the `requests`
library
that is showing up in your tracebacks.

You will need to make sure that the agent is completely removed,
including the `requests` library and then install the latest agent
available
from the Python package Index (PyPI) with a Python installer (e.g.
`pip install radosgw-agent`) or if you feel comfortable use the latest
tip
from the master branch (I advice to use the former).


On Tue, Feb 3, 2015 at 5:12 AM,  <ghislain.chevalier@orange.com> wrote:
> Hi Alfredo,
>
> Here are the logs you requested using the original client.py python script.
>
> ----------------------------------------------------------------------
> DIRECT LAUNCH USING CLI:KO
> radosgw-agent -v -c /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> the standard output is also written in the log file configured in the synchro file
>
> DEBUG:boto:Using access key provided by client.
> DEBUG:boto:Using secret key provided by client.
> DEBUG:boto:StringToSign:
> GET
>
>
> Tue, 03 Feb 2015 09:37:54 GMT
> /admin/config
> DEBUG:boto:Signature:
> AWS SBF7DQZ1ESR6Y34225H8:9i6/4do7tDpViolN1XdxgefMs3U=
> DEBUG:boto:url = 'http://openstorage.ushttp://openstorage.us/admin/config'
> params={}
> headers={'Date': 'Tue, 03 Feb 2015 09:37:54 GMT', 'Content-Length': '0', 'Authorization': 'AWS SBF7DQZ1ESR6Y34225H8:9i6/4do7tDpViolN1XdxgefMs3U=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.11.0-26-generic'}
> data=None
> ERROR:root:Could not retrieve region map from destination
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/cli.py", line 269, in main
>     region_map = client.get_region_map(dest_conn)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 391, in get_region_map
>     region_map = request(connection, 'get', 'admin/config')
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 153, in request
>     result = handler(url, params=params, headers=request.headers, data=data)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
>     return request('get', url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
>     return session.request(method=method, url=url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
>     prep = self.prepare_request(req)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
>     hooks=merge_hooks(request.hooks, self.hooks),
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
>     self.prepare_url(url, params)
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 334, in prepare_url
>     scheme, auth, host, port, path, query, fragment = parse_url(url)
>   File "/usr/lib/python2.7/dist-packages/urllib3/util.py", line 390, in parse_url
>     raise LocationParseError("Failed to parse: %s" % url)
> LocationParseError: Failed to parse: Failed to parse: openstorage.ushttp:
>
> --------------------------------------------------------------------------------
> LAUNCH USING /etc/init.d/radosgw-agent shell script:KO
> /etc/init.d/radosgw-agent start /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> Standard output:
> Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct
>
> Log File :
> 2015-02-03T10:51:00.794 4735:ERROR:root:Could not retrieve region map from destination
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/cli.py", line 269, in main
>     region_map = client.get_region_map(dest_conn)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 391, in get_region_map
>     region_map = request(connection, 'get', 'admin/config')
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 153, in request
>     result = handler(url, params=params, headers=request.headers, data=data)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
>     return request('get', url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
>     return session.request(method=method, url=url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
>     prep = self.prepare_request(req)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
>     hooks=merge_hooks(request.hooks, self.hooks),
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
>     self.prepare_url(url, params)
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 334, in prepare_url
>     scheme, auth, host, port, path, query, fragment = parse_url(url)
>   File "/usr/lib/python2.7/dist-packages/urllib3/util.py", line 390, in parse_url
>     raise LocationParseError("Failed to parse: %s" % url)
> LocationParseError: Failed to parse: Failed to parse: openstorage.ushttp:
>
> -------------------------------------------------------------------------------------------
> LAUNCH AS A SERVICE: OK
> service radosgw-agent start /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> standard output:
> Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct
>
> Log file:
> 2015-02-03T10:42:02.878 3162:INFO:urllib3.connectionpool:Starting new HTTP connection (1): openstorage.us
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
> ----------------------------------------------------------------------------------------------------------------------------------------------------
> Here are the logs you requested using the altered client.py python script.
>
> DIRECT LAUNCH USING CLI:OK
> radosgw-agent -v -c /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> the standard output is also witten in the log file configured in the synchro file
>
> 2015-02-03T11:00:40.447 6158:DEBUG:boto:Using access key provided by client.
> 2015-02-03T11:00:40.447 6158:DEBUG:boto:Using secret key provided by client.
> 2015-02-03T11:00:40.448 6158:DEBUG:boto:StringToSign:
> GET
>
>
> Tue, 03 Feb 2015 10:00:40 GMT
> /admin/config
> 2015-02-03T11:00:40.448 6158:DEBUG:boto:Signature:
> AWS SBF7DQZ1ESR6Y34225H8:LUEBUrR9z1Bgbrj6F/XGrGF+8BM=
> 2015-02-03T11:00:40.448 6158:DEBUG:boto:url = 'http://openstorage.us/admin/config'
> params={}
> headers={'Date': 'Tue, 03 Feb 2015 10:00:40 GMT', 'Content-Length': '0', 'Authorization': 'AWS SBF7DQZ1ESR6Y34225H8:LUEBUrR9z1Bgbrj6F/XGrGF+8BM=', 'User-Agent': 'Boto/2.20.1 Python/2.7.6 Linux/3.11.0-26-generic'}
> data=None
> 2015-02-03T11:00:40.457 6158:INFO:urllib3.connectionpool:Starting new HTTP connection (1): proxy
> 2015-02-03T11:00:40.467 6158:DEBUG:urllib3.connectionpool:Setting read timeout to None
> 2015-02-03T11:00:40.848 6158:DEBUG:urllib3.connectionpool:"GET http://openstorage.us/admin/config HTTP/1.1" 200 None
>
> --------------------------------------------------------------------------------
> LAUNCH USING /etc/init.d/radosgw-agent shell script:KO
> /etc/init.d/radosgw-agent start /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> Standard output:
> Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct
>
> Log File :
> 2015-02-03T11:07:49.271 7513:INFO:urllib3.connectionpool:Starting new HTTP connection (1): proxy
> 2015-02-03T11:07:49.654 7513:INFO:urllib3.connectionpool:Starting new HTTP connection (1): proxy
> 2015-02-03T11:07:50.469 7513:ERROR:radosgw_agent.sync:finding number of shards failed
> 2015-02-03T11:07:50.470 7513:WARNING:radosgw_agent.sync:error preparing for sync, will retry. Traceback:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 26, in prepare_sync
>     syncer.prepare()
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 202, in prepare
>     self.init_num_shards()
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 73, in init_num_shards
>     self.num_shards = client.num_log_shards(self.src_conn, self.type)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 285, in num_log_shards
>     out = request(connection, 'get', 'admin/log', dict(type=shard_type))
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 160, in request
>     check_result_status(result)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 116, in check_result_status
>     HttpError)(result.status_code, result.content)
> NotFound: Http error code 404 content <HTML><HEAD>
>
>
> -------------------------------------------------------------------------------------------
> LAUNCH AS A SERVICE:KO
> service radosgw-agent start /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> standard output:
> Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct
>
> Log file:
> 2015-02-03T11:04:25.435 6872:ERROR:root:Could not retrieve region map from destination
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/cli.py", line 269, in main
>     region_map = client.get_region_map(dest_conn)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 396, in get_region_map
>     region_map = request(connection, 'get', 'admin/config')
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 158, in request
>     result = handler(url, params=params, headers=request.headers, data=data)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
>     return request('get', url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
>     return session.request(method=method, url=url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
>     prep = self.prepare_request(req)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
>     hooks=merge_hooks(request.hooks, self.hooks),
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
>     self.prepare_url(url, params)
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 338, in prepare_url
>     "Perhaps you meant http://{0}?".format(url))
> MissingSchema: Invalid URL u'/admin/config': No schema supplied. Perhaps you meant http:///admin/config?
>
>
>
> Best regards
>
> -----Message d'origine-----
> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la part de Alfredo Deza
> Envoyé : lundi 2 février 2015 19:42
> À : CHEVALIER Ghislain IMT/OLPS
> Cc : ceph-devel@vger.kernel.org
> Objet : Re: [Ceph-devel] radosgw-agent failed to parse
>
> If you use the `-v` flag you will get very verbose output.
>
> On Mon, Feb 2, 2015 at 12:12 PM,  <ghislain.chevalier@orange.com> wrote:
>> OK I will send them asap
>>
>> Logs are not very verbose.. Can I set a debug mode?
>>
>> -----Message d'origine-----
>> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la
>> part de Alfredo Deza Envoyé : lundi 2 février 2015 16:59 À : CHEVALIER
>> Ghislain IMT/OLPS Cc :
>> Objet : Re: [Ceph-devel] radosgw-agent failed to parse
>>
>> That sounds suspicious because we haven't packaged a 1.2.1 release.
>>
>> The latest on PyPI is 1.2, which is the same in the master branch.
>>
>> Would you mind sending logs that show *how* the url is malformed (both when it is OK and when it is not) ?
>>
>> On Mon, Feb 2, 2015 at 9:53 AM,  <ghislain.chevalier@orange.com> wrote:
>>> HI,
>>>
>>> Thx for replying
>>>
>>> According to dpkg -l, it's 1.2.1.
>>>
>>> I noticed that the URL is malformed when launching using directly
>>> radosgw-agent -c <syncfile> but well formed when launching using
>>> service radosgw-agent start <syncfile>
>>>
>>> best regards
>>>
>>> -----Message d'origine-----
>>> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la
>>> part de Alfredo Deza Envoyé : lundi 2 février 2015 15:18 À :
>>> CHEVALIER Ghislain IMT/OLPS Cc : ceph-devel@vger.kernel.org Objet : Re:
>>> [Ceph-devel] radosgw-agent failed to parse
>>>
>>> What version of the agent are you using? And when it fails, how does the log output look?
>>>
>>> On Mon, Feb 2, 2015 at 8:56 AM,  <ghislain.chevalier@orange.com> wrote:
>>>> Hi all,
>>>>
>>>> Context : Ubuntu 14.04 TLS firefly 0.80.8
>>>>
>>>> I sent this post in ceph-users (identical subject) because I recently encountered the same issue.
>>>> Maybe I missed something between July and January...
>>>>
>>>> I found that the http request wasn't correctly formed by
>>>> /usr/lib/python2.7/dist-packages/radosgw_agent/client.py
>>>>
>>>> I did the changes below
>>>> #    url = '{protocol}://{host}{path}'.format(protocol=request.protocol,
>>>> #                                             host=request.host,
>>>> #                                             path=request.path)
>>>>      url = '{path}'.format(protocol="", host="", path=request.path)
>>>>
>>>> The request is then correctly formed and sent.
>>>>
>>>> I still have problems setting the federation between 2 zones of 2 ceph clusters in 2 regions.
>>>>
>>>> I go on investigating.
>>>>
>>>> Best regards
>>>>
>>>>
>>>> ____________________________________________________________________
>>>> _ _ ___________________________________________________
>>>>
>>>> Ce message et ses pieces jointes peuvent contenir des informations
>>>> confidentielles ou privilegiees et ne doivent donc pas etre
>>>> diffuses, exploites ou copies sans autorisation. Si vous avez recu
>>>> ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>>>
>>>> This message and its attachments may contain confidential or
>>>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>>>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>>>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>>>> Thank you.
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>>> in the body of a message to majordomo@vger.kernel.org More majordomo
>>>> info at  http://vger.kernel.org/majordomo-info.html
>>>
>>> _____________________________________________________________________
>>> _ ___________________________________________________
>>>
>>> Ce message et ses pieces jointes peuvent contenir des informations
>>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
>>> exploites ou copies sans autorisation. Si vous avez recu ce message
>>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>>
>>> This message and its attachments may contain confidential or
>>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>>> Thank you.
>>>
>>
>> ______________________________________________________________________
>> ___________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations
>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
>> exploites ou copies sans autorisation. Si vous avez recu ce message
>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or
>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>> Thank you.
>>
>
> _________________________________________________________________________________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: radosgw-agent failed to parse

[ghislain.chevalier]

HI

Thanks a lot

I installed the last version using (pip install radosgw-agent) as recommended after removing the current version (apt-get purge radosgw-agent)
The python scripts are now installed in /usr/local/bin/...../radosgw_agent instead of /usr/bin/....../radosgw_agent

Can you tell me what is required about the 'requests' library?

So, I will test it asap

Did you plan to update the package in order to be compliant with apt-get install radosgw-agent?

Best regards 

-----Message d'origine-----
De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la part de Alfredo Deza
Envoyé : mardi 3 février 2015 14:30
À : CHEVALIER Ghislain IMT/OLPS
Cc : ceph-devel@vger.kernel.org
Objet : Re: [Ceph-devel] radosgw-agent failed to parse

This tells me that you are not using the latest released version of the agent because we dropped the requirement for the `requests` library that is showing up in your tracebacks.

You will need to make sure that the agent is completely removed, including the `requests` library and then install the latest agent available from the Python package Index (PyPI) with a Python installer (e.g.
`pip install radosgw-agent`) or if you feel comfortable use the latest tip from the master branch (I advice to use the former).


On Tue, Feb 3, 2015 at 5:12 AM,  <ghislain.chevalier@orange.com> wrote:
> Hi Alfredo,
>
> Here are the logs you requested using the original client.py python script.
>
> ----------------------------------------------------------------------
> DIRECT LAUNCH USING CLI:KO
> radosgw-agent -v -c 
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> the standard output is also written in the log file configured in the 
> synchro file
>
> DEBUG:boto:Using access key provided by client.
> DEBUG:boto:Using secret key provided by client.
> DEBUG:boto:StringToSign:
> GET
>
>
> Tue, 03 Feb 2015 09:37:54 GMT
> /admin/config
> DEBUG:boto:Signature:
> AWS SBF7DQZ1ESR6Y34225H8:9i6/4do7tDpViolN1XdxgefMs3U=
> DEBUG:boto:url = 'http://openstorage.ushttp://openstorage.us/admin/config'
> params={}
> headers={'Date': 'Tue, 03 Feb 2015 09:37:54 GMT', 'Content-Length': 
> '0', 'Authorization': 'AWS 
> SBF7DQZ1ESR6Y34225H8:9i6/4do7tDpViolN1XdxgefMs3U=', 'User-Agent': 
> 'Boto/2.20.1 Python/2.7.6 Linux/3.11.0-26-generic'} data=None ERROR:root:Could not retrieve region map from destination Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/cli.py", line 269, in main
>     region_map = client.get_region_map(dest_conn)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 391, in get_region_map
>     region_map = request(connection, 'get', 'admin/config')
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 153, in request
>     result = handler(url, params=params, headers=request.headers, data=data)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
>     return request('get', url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
>     return session.request(method=method, url=url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
>     prep = self.prepare_request(req)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
>     hooks=merge_hooks(request.hooks, self.hooks),
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
>     self.prepare_url(url, params)
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 334, in prepare_url
>     scheme, auth, host, port, path, query, fragment = parse_url(url)
>   File "/usr/lib/python2.7/dist-packages/urllib3/util.py", line 390, in parse_url
>     raise LocationParseError("Failed to parse: %s" % url)
> LocationParseError: Failed to parse: Failed to parse: openstorage.ushttp:
>
> ----------------------------------------------------------------------
> ---------- LAUNCH USING /etc/init.d/radosgw-agent shell script:KO 
> /etc/init.d/radosgw-agent start 
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> Standard output:
> Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct
>
> Log File :
> 2015-02-03T10:51:00.794 4735:ERROR:root:Could not retrieve region map 
> from destination Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/cli.py", line 269, in main
>     region_map = client.get_region_map(dest_conn)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 391, in get_region_map
>     region_map = request(connection, 'get', 'admin/config')
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 153, in request
>     result = handler(url, params=params, headers=request.headers, data=data)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
>     return request('get', url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
>     return session.request(method=method, url=url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
>     prep = self.prepare_request(req)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
>     hooks=merge_hooks(request.hooks, self.hooks),
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
>     self.prepare_url(url, params)
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 334, in prepare_url
>     scheme, auth, host, port, path, query, fragment = parse_url(url)
>   File "/usr/lib/python2.7/dist-packages/urllib3/util.py", line 390, in parse_url
>     raise LocationParseError("Failed to parse: %s" % url)
> LocationParseError: Failed to parse: Failed to parse: openstorage.ushttp:
>
> ----------------------------------------------------------------------
> ---------------------
> LAUNCH AS A SERVICE: OK
> service radosgw-agent start 
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> standard output:
> Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct
>
> Log file:
> 2015-02-03T10:42:02.878 3162:INFO:urllib3.connectionpool:Starting new 
> HTTP connection (1): openstorage.us
>
> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> --------
> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> -------- Here are the logs you requested using the altered client.py 
> python script.
>
> DIRECT LAUNCH USING CLI:OK
> radosgw-agent -v -c 
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> the standard output is also witten in the log file configured in the 
> synchro file
>
> 2015-02-03T11:00:40.447 6158:DEBUG:boto:Using access key provided by client.
> 2015-02-03T11:00:40.447 6158:DEBUG:boto:Using secret key provided by client.
> 2015-02-03T11:00:40.448 6158:DEBUG:boto:StringToSign:
> GET
>
>
> Tue, 03 Feb 2015 10:00:40 GMT
> /admin/config
> 2015-02-03T11:00:40.448 6158:DEBUG:boto:Signature:
> AWS SBF7DQZ1ESR6Y34225H8:LUEBUrR9z1Bgbrj6F/XGrGF+8BM=
> 2015-02-03T11:00:40.448 6158:DEBUG:boto:url = 'http://openstorage.us/admin/config'
> params={}
> headers={'Date': 'Tue, 03 Feb 2015 10:00:40 GMT', 'Content-Length': 
> '0', 'Authorization': 'AWS 
> SBF7DQZ1ESR6Y34225H8:LUEBUrR9z1Bgbrj6F/XGrGF+8BM=', 'User-Agent': 
> 'Boto/2.20.1 Python/2.7.6 Linux/3.11.0-26-generic'} data=None
> 2015-02-03T11:00:40.457 6158:INFO:urllib3.connectionpool:Starting new 
> HTTP connection (1): proxy
> 2015-02-03T11:00:40.467 6158:DEBUG:urllib3.connectionpool:Setting read 
> timeout to None
> 2015-02-03T11:00:40.848 6158:DEBUG:urllib3.connectionpool:"GET 
> http://openstorage.us/admin/config HTTP/1.1" 200 None
>
> ----------------------------------------------------------------------
> ---------- LAUNCH USING /etc/init.d/radosgw-agent shell script:KO 
> /etc/init.d/radosgw-agent start 
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> Standard output:
> Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct
>
> Log File :
> 2015-02-03T11:07:49.271 7513:INFO:urllib3.connectionpool:Starting new 
> HTTP connection (1): proxy
> 2015-02-03T11:07:49.654 7513:INFO:urllib3.connectionpool:Starting new 
> HTTP connection (1): proxy
> 2015-02-03T11:07:50.469 7513:ERROR:radosgw_agent.sync:finding number 
> of shards failed
> 2015-02-03T11:07:50.470 7513:WARNING:radosgw_agent.sync:error preparing for sync, will retry. Traceback:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 26, in prepare_sync
>     syncer.prepare()
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 202, in prepare
>     self.init_num_shards()
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 73, in init_num_shards
>     self.num_shards = client.num_log_shards(self.src_conn, self.type)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 285, in num_log_shards
>     out = request(connection, 'get', 'admin/log', dict(type=shard_type))
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 160, in request
>     check_result_status(result)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 116, in check_result_status
>     HttpError)(result.status_code, result.content)
> NotFound: Http error code 404 content <HTML><HEAD>
>
>
> ----------------------------------------------------------------------
> ---------------------
> LAUNCH AS A SERVICE:KO
> service radosgw-agent start 
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> standard output:
> Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct
>
> Log file:
> 2015-02-03T11:04:25.435 6872:ERROR:root:Could not retrieve region map 
> from destination Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/cli.py", line 269, in main
>     region_map = client.get_region_map(dest_conn)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 396, in get_region_map
>     region_map = request(connection, 'get', 'admin/config')
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 158, in request
>     result = handler(url, params=params, headers=request.headers, data=data)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
>     return request('get', url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
>     return session.request(method=method, url=url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
>     prep = self.prepare_request(req)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
>     hooks=merge_hooks(request.hooks, self.hooks),
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
>     self.prepare_url(url, params)
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 338, in prepare_url
>     "Perhaps you meant http://{0}?".format(url))
> MissingSchema: Invalid URL u'/admin/config': No schema supplied. Perhaps you meant http:///admin/config?
>
>
>
> Best regards
>
> -----Message d'origine-----
> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la 
> part de Alfredo Deza Envoyé : lundi 2 février 2015 19:42 À : CHEVALIER 
> Ghislain IMT/OLPS Cc : ceph-devel@vger.kernel.org Objet : Re: 
> [Ceph-devel] radosgw-agent failed to parse
>
> If you use the `-v` flag you will get very verbose output.
>
> On Mon, Feb 2, 2015 at 12:12 PM,  <ghislain.chevalier@orange.com> wrote:
>> OK I will send them asap
>>
>> Logs are not very verbose.. Can I set a debug mode?
>>
>> -----Message d'origine-----
>> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la 
>> part de Alfredo Deza Envoyé : lundi 2 février 2015 16:59 À : 
>> CHEVALIER Ghislain IMT/OLPS Cc :
>> Objet : Re: [Ceph-devel] radosgw-agent failed to parse
>>
>> That sounds suspicious because we haven't packaged a 1.2.1 release.
>>
>> The latest on PyPI is 1.2, which is the same in the master branch.
>>
>> Would you mind sending logs that show *how* the url is malformed (both when it is OK and when it is not) ?
>>
>> On Mon, Feb 2, 2015 at 9:53 AM,  <ghislain.chevalier@orange.com> wrote:
>>> HI,
>>>
>>> Thx for replying
>>>
>>> According to dpkg -l, it's 1.2.1.
>>>
>>> I noticed that the URL is malformed when launching using directly 
>>> radosgw-agent -c <syncfile> but well formed when launching using 
>>> service radosgw-agent start <syncfile>
>>>
>>> best regards
>>>
>>> -----Message d'origine-----
>>> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De 
>>> la part de Alfredo Deza Envoyé : lundi 2 février 2015 15:18 À :
>>> CHEVALIER Ghislain IMT/OLPS Cc : ceph-devel@vger.kernel.org Objet : Re:
>>> [Ceph-devel] radosgw-agent failed to parse
>>>
>>> What version of the agent are you using? And when it fails, how does the log output look?
>>>
>>> On Mon, Feb 2, 2015 at 8:56 AM,  <ghislain.chevalier@orange.com> wrote:
>>>> Hi all,
>>>>
>>>> Context : Ubuntu 14.04 TLS firefly 0.80.8
>>>>
>>>> I sent this post in ceph-users (identical subject) because I recently encountered the same issue.
>>>> Maybe I missed something between July and January...
>>>>
>>>> I found that the http request wasn't correctly formed by 
>>>> /usr/lib/python2.7/dist-packages/radosgw_agent/client.py
>>>>
>>>> I did the changes below
>>>> #    url = '{protocol}://{host}{path}'.format(protocol=request.protocol,
>>>> #                                             host=request.host,
>>>> #                                             path=request.path)
>>>>      url = '{path}'.format(protocol="", host="", path=request.path)
>>>>
>>>> The request is then correctly formed and sent.
>>>>
>>>> I still have problems setting the federation between 2 zones of 2 ceph clusters in 2 regions.
>>>>
>>>> I go on investigating.
>>>>
>>>> Best regards
>>>>
>>>>
>>>> ___________________________________________________________________
>>>> _ _ _ ___________________________________________________
>>>>
>>>> Ce message et ses pieces jointes peuvent contenir des informations 
>>>> confidentielles ou privilegiees et ne doivent donc pas etre 
>>>> diffuses, exploites ou copies sans autorisation. Si vous avez recu 
>>>> ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>>>
>>>> This message and its attachments may contain confidential or 
>>>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>>>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>>>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>>>> Thank you.
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>>> in the body of a message to majordomo@vger.kernel.org More 
>>>> majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>> ____________________________________________________________________
>>> _ _ ___________________________________________________
>>>
>>> Ce message et ses pieces jointes peuvent contenir des informations 
>>> confidentielles ou privilegiees et ne doivent donc pas etre 
>>> diffuses, exploites ou copies sans autorisation. Si vous avez recu 
>>> ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>>
>>> This message and its attachments may contain confidential or 
>>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>>> Thank you.
>>>
>>
>> _____________________________________________________________________
>> _ ___________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations 
>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
>> exploites ou copies sans autorisation. Si vous avez recu ce message 
>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or 
>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>> Thank you.
>>
>
> ______________________________________________________________________
> ___________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
> exploites ou copies sans autorisation. Si vous avez recu ce message 
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or 
> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

Re: radosgw-agent failed to parse

[Alfredo Deza]

1.2.1 was just released bringing parity for both RPM/DEB and the PyPI versions.

You should now be able to get it either way. Let me know if you are still seeing the same issues.



----- Original Message -----
From: "ghislain chevalier" <ghislain.chevalier@orange.com>
To: "Alfredo Deza" <adeza@redhat.com>
Cc: ceph-devel@vger.kernel.org
Sent: Thursday, February 5, 2015 9:42:09 AM
Subject: RE: [Ceph-devel] radosgw-agent failed to parse

HI

Thanks a lot

I installed the last version using (pip install radosgw-agent) as recommended after removing the current version (apt-get purge radosgw-agent)
The python scripts are now installed in /usr/local/bin/...../radosgw_agent instead of /usr/bin/....../radosgw_agent

Can you tell me what is required about the 'requests' library?

So, I will test it asap

Did you plan to update the package in order to be compliant with apt-get install radosgw-agent?

Best regards 

-----Message d'origine-----
De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la part de Alfredo Deza
Envoyé : mardi 3 février 2015 14:30
À : CHEVALIER Ghislain IMT/OLPS
Cc : ceph-devel@vger.kernel.org
Objet : Re: [Ceph-devel] radosgw-agent failed to parse

This tells me that you are not using the latest released version of the agent because we dropped the requirement for the `requests` library that is showing up in your tracebacks.

You will need to make sure that the agent is completely removed, including the `requests` library and then install the latest agent available from the Python package Index (PyPI) with a Python installer (e.g.
`pip install radosgw-agent`) or if you feel comfortable use the latest tip from the master branch (I advice to use the former).


On Tue, Feb 3, 2015 at 5:12 AM,  <ghislain.chevalier@orange.com> wrote:
> Hi Alfredo,
>
> Here are the logs you requested using the original client.py python script.
>
> ----------------------------------------------------------------------
> DIRECT LAUNCH USING CLI:KO
> radosgw-agent -v -c 
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> the standard output is also written in the log file configured in the 
> synchro file
>
> DEBUG:boto:Using access key provided by client.
> DEBUG:boto:Using secret key provided by client.
> DEBUG:boto:StringToSign:
> GET
>
>
> Tue, 03 Feb 2015 09:37:54 GMT
> /admin/config
> DEBUG:boto:Signature:
> AWS SBF7DQZ1ESR6Y34225H8:9i6/4do7tDpViolN1XdxgefMs3U=
> DEBUG:boto:url = 'http://openstorage.ushttp://openstorage.us/admin/config'
> params={}
> headers={'Date': 'Tue, 03 Feb 2015 09:37:54 GMT', 'Content-Length': 
> '0', 'Authorization': 'AWS 
> SBF7DQZ1ESR6Y34225H8:9i6/4do7tDpViolN1XdxgefMs3U=', 'User-Agent': 
> 'Boto/2.20.1 Python/2.7.6 Linux/3.11.0-26-generic'} data=None ERROR:root:Could not retrieve region map from destination Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/cli.py", line 269, in main
>     region_map = client.get_region_map(dest_conn)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 391, in get_region_map
>     region_map = request(connection, 'get', 'admin/config')
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 153, in request
>     result = handler(url, params=params, headers=request.headers, data=data)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
>     return request('get', url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
>     return session.request(method=method, url=url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
>     prep = self.prepare_request(req)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
>     hooks=merge_hooks(request.hooks, self.hooks),
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
>     self.prepare_url(url, params)
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 334, in prepare_url
>     scheme, auth, host, port, path, query, fragment = parse_url(url)
>   File "/usr/lib/python2.7/dist-packages/urllib3/util.py", line 390, in parse_url
>     raise LocationParseError("Failed to parse: %s" % url)
> LocationParseError: Failed to parse: Failed to parse: openstorage.ushttp:
>
> ----------------------------------------------------------------------
> ---------- LAUNCH USING /etc/init.d/radosgw-agent shell script:KO 
> /etc/init.d/radosgw-agent start 
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> Standard output:
> Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct
>
> Log File :
> 2015-02-03T10:51:00.794 4735:ERROR:root:Could not retrieve region map 
> from destination Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/cli.py", line 269, in main
>     region_map = client.get_region_map(dest_conn)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 391, in get_region_map
>     region_map = request(connection, 'get', 'admin/config')
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 153, in request
>     result = handler(url, params=params, headers=request.headers, data=data)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
>     return request('get', url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
>     return session.request(method=method, url=url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
>     prep = self.prepare_request(req)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
>     hooks=merge_hooks(request.hooks, self.hooks),
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
>     self.prepare_url(url, params)
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 334, in prepare_url
>     scheme, auth, host, port, path, query, fragment = parse_url(url)
>   File "/usr/lib/python2.7/dist-packages/urllib3/util.py", line 390, in parse_url
>     raise LocationParseError("Failed to parse: %s" % url)
> LocationParseError: Failed to parse: Failed to parse: openstorage.ushttp:
>
> ----------------------------------------------------------------------
> ---------------------
> LAUNCH AS A SERVICE: OK
> service radosgw-agent start 
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> standard output:
> Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct
>
> Log file:
> 2015-02-03T10:42:02.878 3162:INFO:urllib3.connectionpool:Starting new 
> HTTP connection (1): openstorage.us
>
> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> --------
> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> -------- Here are the logs you requested using the altered client.py 
> python script.
>
> DIRECT LAUNCH USING CLI:OK
> radosgw-agent -v -c 
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> the standard output is also witten in the log file configured in the 
> synchro file
>
> 2015-02-03T11:00:40.447 6158:DEBUG:boto:Using access key provided by client.
> 2015-02-03T11:00:40.447 6158:DEBUG:boto:Using secret key provided by client.
> 2015-02-03T11:00:40.448 6158:DEBUG:boto:StringToSign:
> GET
>
>
> Tue, 03 Feb 2015 10:00:40 GMT
> /admin/config
> 2015-02-03T11:00:40.448 6158:DEBUG:boto:Signature:
> AWS SBF7DQZ1ESR6Y34225H8:LUEBUrR9z1Bgbrj6F/XGrGF+8BM=
> 2015-02-03T11:00:40.448 6158:DEBUG:boto:url = 'http://openstorage.us/admin/config'
> params={}
> headers={'Date': 'Tue, 03 Feb 2015 10:00:40 GMT', 'Content-Length': 
> '0', 'Authorization': 'AWS 
> SBF7DQZ1ESR6Y34225H8:LUEBUrR9z1Bgbrj6F/XGrGF+8BM=', 'User-Agent': 
> 'Boto/2.20.1 Python/2.7.6 Linux/3.11.0-26-generic'} data=None
> 2015-02-03T11:00:40.457 6158:INFO:urllib3.connectionpool:Starting new 
> HTTP connection (1): proxy
> 2015-02-03T11:00:40.467 6158:DEBUG:urllib3.connectionpool:Setting read 
> timeout to None
> 2015-02-03T11:00:40.848 6158:DEBUG:urllib3.connectionpool:"GET 
> http://openstorage.us/admin/config HTTP/1.1" 200 None
>
> ----------------------------------------------------------------------
> ---------- LAUNCH USING /etc/init.d/radosgw-agent shell script:KO 
> /etc/init.d/radosgw-agent start 
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> Standard output:
> Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct
>
> Log File :
> 2015-02-03T11:07:49.271 7513:INFO:urllib3.connectionpool:Starting new 
> HTTP connection (1): proxy
> 2015-02-03T11:07:49.654 7513:INFO:urllib3.connectionpool:Starting new 
> HTTP connection (1): proxy
> 2015-02-03T11:07:50.469 7513:ERROR:radosgw_agent.sync:finding number 
> of shards failed
> 2015-02-03T11:07:50.470 7513:WARNING:radosgw_agent.sync:error preparing for sync, will retry. Traceback:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 26, in prepare_sync
>     syncer.prepare()
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 202, in prepare
>     self.init_num_shards()
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 73, in init_num_shards
>     self.num_shards = client.num_log_shards(self.src_conn, self.type)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 285, in num_log_shards
>     out = request(connection, 'get', 'admin/log', dict(type=shard_type))
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 160, in request
>     check_result_status(result)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 116, in check_result_status
>     HttpError)(result.status_code, result.content)
> NotFound: Http error code 404 content <HTML><HEAD>
>
>
> ----------------------------------------------------------------------
> ---------------------
> LAUNCH AS A SERVICE:KO
> service radosgw-agent start 
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> standard output:
> Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct
>
> Log file:
> 2015-02-03T11:04:25.435 6872:ERROR:root:Could not retrieve region map 
> from destination Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/cli.py", line 269, in main
>     region_map = client.get_region_map(dest_conn)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 396, in get_region_map
>     region_map = request(connection, 'get', 'admin/config')
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 158, in request
>     result = handler(url, params=params, headers=request.headers, data=data)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
>     return request('get', url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
>     return session.request(method=method, url=url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
>     prep = self.prepare_request(req)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
>     hooks=merge_hooks(request.hooks, self.hooks),
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
>     self.prepare_url(url, params)
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 338, in prepare_url
>     "Perhaps you meant http://{0}?".format(url))
> MissingSchema: Invalid URL u'/admin/config': No schema supplied. Perhaps you meant http:///admin/config?
>
>
>
> Best regards
>
> -----Message d'origine-----
> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la 
> part de Alfredo Deza Envoyé : lundi 2 février 2015 19:42 À : CHEVALIER 
> Ghislain IMT/OLPS Cc : ceph-devel@vger.kernel.org Objet : Re: 
> [Ceph-devel] radosgw-agent failed to parse
>
> If you use the `-v` flag you will get very verbose output.
>
> On Mon, Feb 2, 2015 at 12:12 PM,  <ghislain.chevalier@orange.com> wrote:
>> OK I will send them asap
>>
>> Logs are not very verbose.. Can I set a debug mode?
>>
>> -----Message d'origine-----
>> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la 
>> part de Alfredo Deza Envoyé : lundi 2 février 2015 16:59 À : 
>> CHEVALIER Ghislain IMT/OLPS Cc :
>> Objet : Re: [Ceph-devel] radosgw-agent failed to parse
>>
>> That sounds suspicious because we haven't packaged a 1.2.1 release.
>>
>> The latest on PyPI is 1.2, which is the same in the master branch.
>>
>> Would you mind sending logs that show *how* the url is malformed (both when it is OK and when it is not) ?
>>
>> On Mon, Feb 2, 2015 at 9:53 AM,  <ghislain.chevalier@orange.com> wrote:
>>> HI,
>>>
>>> Thx for replying
>>>
>>> According to dpkg -l, it's 1.2.1.
>>>
>>> I noticed that the URL is malformed when launching using directly 
>>> radosgw-agent -c <syncfile> but well formed when launching using 
>>> service radosgw-agent start <syncfile>
>>>
>>> best regards
>>>
>>> -----Message d'origine-----
>>> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De 
>>> la part de Alfredo Deza Envoyé : lundi 2 février 2015 15:18 À :
>>> CHEVALIER Ghislain IMT/OLPS Cc : ceph-devel@vger.kernel.org Objet : Re:
>>> [Ceph-devel] radosgw-agent failed to parse
>>>
>>> What version of the agent are you using? And when it fails, how does the log output look?
>>>
>>> On Mon, Feb 2, 2015 at 8:56 AM,  <ghislain.chevalier@orange.com> wrote:
>>>> Hi all,
>>>>
>>>> Context : Ubuntu 14.04 TLS firefly 0.80.8
>>>>
>>>> I sent this post in ceph-users (identical subject) because I recently encountered the same issue.
>>>> Maybe I missed something between July and January...
>>>>
>>>> I found that the http request wasn't correctly formed by 
>>>> /usr/lib/python2.7/dist-packages/radosgw_agent/client.py
>>>>
>>>> I did the changes below
>>>> #    url = '{protocol}://{host}{path}'.format(protocol=request.protocol,
>>>> #                                             host=request.host,
>>>> #                                             path=request.path)
>>>>      url = '{path}'.format(protocol="", host="", path=request.path)
>>>>
>>>> The request is then correctly formed and sent.
>>>>
>>>> I still have problems setting the federation between 2 zones of 2 ceph clusters in 2 regions.
>>>>
>>>> I go on investigating.
>>>>
>>>> Best regards
>>>>
>>>>
>>>> ___________________________________________________________________
>>>> _ _ _ ___________________________________________________
>>>>
>>>> Ce message et ses pieces jointes peuvent contenir des informations 
>>>> confidentielles ou privilegiees et ne doivent donc pas etre 
>>>> diffuses, exploites ou copies sans autorisation. Si vous avez recu 
>>>> ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>>>
>>>> This message and its attachments may contain confidential or 
>>>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>>>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>>>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>>>> Thank you.
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>>> in the body of a message to majordomo@vger.kernel.org More 
>>>> majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>> ____________________________________________________________________
>>> _ _ ___________________________________________________
>>>
>>> Ce message et ses pieces jointes peuvent contenir des informations 
>>> confidentielles ou privilegiees et ne doivent donc pas etre 
>>> diffuses, exploites ou copies sans autorisation. Si vous avez recu 
>>> ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>>
>>> This message and its attachments may contain confidential or 
>>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>>> Thank you.
>>>
>>
>> _____________________________________________________________________
>> _ ___________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations 
>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
>> exploites ou copies sans autorisation. Si vous avez recu ce message 
>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or 
>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>> Thank you.
>>
>
> ______________________________________________________________________
> ___________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
> exploites ou copies sans autorisation. Si vous avez recu ce message 
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or 
> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: radosgw-agent failed to parse

[ghislain.chevalier]

Hi Alfredo,

Sorry for replying very very late. I had to temporarly descope my actions on rados gateway federation

The radosgw-agent is installed properly; that's fine.

We still have problems setting up federation between two sites.
One is at Orange Sillicon Valley (OSV@San Francisco), the other in France (Orange Labs@Rennes); 2 lab clusters.

We understood that data synchronization can only be set up between zones in a common region.
So we set up the 2 rgw with a common region with specific zones in our own clusters.

We understood that as the French zone is not master, I cannot store any objects in my buckets. 
We have 1 common s3 account and we set up 2 system users in order to drive the data synchronization.

As the master zone is at OSV, a radosgw-agent will be launched at this side and we would like the data to be synchronized between the 2 clusters.

Can you just tell us if we understood well the behavior of a rados gateway federation and if we will soon be able to federate our two rgw?

NB : we can provide our configuration if needed.

Best regards

-----Message d'origine-----
De : Alfredo Deza [mailto:adeza@redhat.com] 
Envoyé : mardi 10 février 2015 14:45
À : CHEVALIER Ghislain IMT/OLPS
Cc : ceph-devel@vger.kernel.org
Objet : Re: [Ceph-devel] radosgw-agent failed to parse

1.2.1 was just released bringing parity for both RPM/DEB and the PyPI versions.

You should now be able to get it either way. Let me know if you are still seeing the same issues.



----- Original Message -----
From: "ghislain chevalier" <ghislain.chevalier@orange.com>
To: "Alfredo Deza" <adeza@redhat.com>
Cc: ceph-devel@vger.kernel.org
Sent: Thursday, February 5, 2015 9:42:09 AM
Subject: RE: [Ceph-devel] radosgw-agent failed to parse

HI

Thanks a lot

I installed the last version using (pip install radosgw-agent) as recommended after removing the current version (apt-get purge radosgw-agent) The python scripts are now installed in /usr/local/bin/...../radosgw_agent instead of /usr/bin/....../radosgw_agent

Can you tell me what is required about the 'requests' library?

So, I will test it asap

Did you plan to update the package in order to be compliant with apt-get install radosgw-agent?

Best regards 

-----Message d'origine-----
De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la part de Alfredo Deza Envoyé : mardi 3 février 2015 14:30 À : CHEVALIER Ghislain IMT/OLPS Cc : ceph-devel@vger.kernel.org Objet : Re: [Ceph-devel] radosgw-agent failed to parse

This tells me that you are not using the latest released version of the agent because we dropped the requirement for the `requests` library that is showing up in your tracebacks.

You will need to make sure that the agent is completely removed, including the `requests` library and then install the latest agent available from the Python package Index (PyPI) with a Python installer (e.g.
`pip install radosgw-agent`) or if you feel comfortable use the latest tip from the master branch (I advice to use the former).


On Tue, Feb 3, 2015 at 5:12 AM,  <ghislain.chevalier@orange.com> wrote:
> Hi Alfredo,
>
> Here are the logs you requested using the original client.py python script.
>
> ----------------------------------------------------------------------
> DIRECT LAUNCH USING CLI:KO
> radosgw-agent -v -c
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> the standard output is also written in the log file configured in the 
> synchro file
>
> DEBUG:boto:Using access key provided by client.
> DEBUG:boto:Using secret key provided by client.
> DEBUG:boto:StringToSign:
> GET
>
>
> Tue, 03 Feb 2015 09:37:54 GMT
> /admin/config
> DEBUG:boto:Signature:
> AWS SBF7DQZ1ESR6Y34225H8:9i6/4do7tDpViolN1XdxgefMs3U=
> DEBUG:boto:url = 'http://openstorage.ushttp://openstorage.us/admin/config'
> params={}
> headers={'Date': 'Tue, 03 Feb 2015 09:37:54 GMT', 'Content-Length': 
> '0', 'Authorization': 'AWS
> SBF7DQZ1ESR6Y34225H8:9i6/4do7tDpViolN1XdxgefMs3U=', 'User-Agent': 
> 'Boto/2.20.1 Python/2.7.6 Linux/3.11.0-26-generic'} data=None ERROR:root:Could not retrieve region map from destination Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/cli.py", line 269, in main
>     region_map = client.get_region_map(dest_conn)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 391, in get_region_map
>     region_map = request(connection, 'get', 'admin/config')
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 153, in request
>     result = handler(url, params=params, headers=request.headers, data=data)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
>     return request('get', url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
>     return session.request(method=method, url=url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
>     prep = self.prepare_request(req)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
>     hooks=merge_hooks(request.hooks, self.hooks),
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
>     self.prepare_url(url, params)
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 334, in prepare_url
>     scheme, auth, host, port, path, query, fragment = parse_url(url)
>   File "/usr/lib/python2.7/dist-packages/urllib3/util.py", line 390, in parse_url
>     raise LocationParseError("Failed to parse: %s" % url)
> LocationParseError: Failed to parse: Failed to parse: openstorage.ushttp:
>
> ----------------------------------------------------------------------
> ---------- LAUNCH USING /etc/init.d/radosgw-agent shell script:KO 
> /etc/init.d/radosgw-agent start 
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> Standard output:
> Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct
>
> Log File :
> 2015-02-03T10:51:00.794 4735:ERROR:root:Could not retrieve region map 
> from destination Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/cli.py", line 269, in main
>     region_map = client.get_region_map(dest_conn)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 391, in get_region_map
>     region_map = request(connection, 'get', 'admin/config')
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 153, in request
>     result = handler(url, params=params, headers=request.headers, data=data)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
>     return request('get', url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
>     return session.request(method=method, url=url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
>     prep = self.prepare_request(req)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
>     hooks=merge_hooks(request.hooks, self.hooks),
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
>     self.prepare_url(url, params)
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 334, in prepare_url
>     scheme, auth, host, port, path, query, fragment = parse_url(url)
>   File "/usr/lib/python2.7/dist-packages/urllib3/util.py", line 390, in parse_url
>     raise LocationParseError("Failed to parse: %s" % url)
> LocationParseError: Failed to parse: Failed to parse: openstorage.ushttp:
>
> ----------------------------------------------------------------------
> ---------------------
> LAUNCH AS A SERVICE: OK
> service radosgw-agent start
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> standard output:
> Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct
>
> Log file:
> 2015-02-03T10:42:02.878 3162:INFO:urllib3.connectionpool:Starting new 
> HTTP connection (1): openstorage.us
>
> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> --------
> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> -------- Here are the logs you requested using the altered client.py 
> python script.
>
> DIRECT LAUNCH USING CLI:OK
> radosgw-agent -v -c
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> the standard output is also witten in the log file configured in the 
> synchro file
>
> 2015-02-03T11:00:40.447 6158:DEBUG:boto:Using access key provided by client.
> 2015-02-03T11:00:40.447 6158:DEBUG:boto:Using secret key provided by client.
> 2015-02-03T11:00:40.448 6158:DEBUG:boto:StringToSign:
> GET
>
>
> Tue, 03 Feb 2015 10:00:40 GMT
> /admin/config
> 2015-02-03T11:00:40.448 6158:DEBUG:boto:Signature:
> AWS SBF7DQZ1ESR6Y34225H8:LUEBUrR9z1Bgbrj6F/XGrGF+8BM=
> 2015-02-03T11:00:40.448 6158:DEBUG:boto:url = 'http://openstorage.us/admin/config'
> params={}
> headers={'Date': 'Tue, 03 Feb 2015 10:00:40 GMT', 'Content-Length': 
> '0', 'Authorization': 'AWS
> SBF7DQZ1ESR6Y34225H8:LUEBUrR9z1Bgbrj6F/XGrGF+8BM=', 'User-Agent': 
> 'Boto/2.20.1 Python/2.7.6 Linux/3.11.0-26-generic'} data=None
> 2015-02-03T11:00:40.457 6158:INFO:urllib3.connectionpool:Starting new 
> HTTP connection (1): proxy
> 2015-02-03T11:00:40.467 6158:DEBUG:urllib3.connectionpool:Setting read 
> timeout to None
> 2015-02-03T11:00:40.848 6158:DEBUG:urllib3.connectionpool:"GET
> http://openstorage.us/admin/config HTTP/1.1" 200 None
>
> ----------------------------------------------------------------------
> ---------- LAUNCH USING /etc/init.d/radosgw-agent shell script:KO 
> /etc/init.d/radosgw-agent start 
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> Standard output:
> Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct
>
> Log File :
> 2015-02-03T11:07:49.271 7513:INFO:urllib3.connectionpool:Starting new 
> HTTP connection (1): proxy
> 2015-02-03T11:07:49.654 7513:INFO:urllib3.connectionpool:Starting new 
> HTTP connection (1): proxy
> 2015-02-03T11:07:50.469 7513:ERROR:radosgw_agent.sync:finding number 
> of shards failed
> 2015-02-03T11:07:50.470 7513:WARNING:radosgw_agent.sync:error preparing for sync, will retry. Traceback:
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 26, in prepare_sync
>     syncer.prepare()
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 202, in prepare
>     self.init_num_shards()
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/sync.py", line 73, in init_num_shards
>     self.num_shards = client.num_log_shards(self.src_conn, self.type)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 285, in num_log_shards
>     out = request(connection, 'get', 'admin/log', dict(type=shard_type))
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 160, in request
>     check_result_status(result)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 116, in check_result_status
>     HttpError)(result.status_code, result.content)
> NotFound: Http error code 404 content <HTML><HEAD>
>
>
> ----------------------------------------------------------------------
> ---------------------
> LAUNCH AS A SERVICE:KO
> service radosgw-agent start
> /etc/ceph/radosgw-agent/fr-rennes-radosgw1-sync.conf_direct
> standard output:
> Starting radosgw-agent fr-rennes-radosgw1-sync.conf_direct
>
> Log file:
> 2015-02-03T11:04:25.435 6872:ERROR:root:Could not retrieve region map 
> from destination Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/cli.py", line 269, in main
>     region_map = client.get_region_map(dest_conn)
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 396, in get_region_map
>     region_map = request(connection, 'get', 'admin/config')
>   File "/usr/lib/python2.7/dist-packages/radosgw_agent/client.py", line 158, in request
>     result = handler(url, params=params, headers=request.headers, data=data)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
>     return request('get', url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
>     return session.request(method=method, url=url, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 421, in request
>     prep = self.prepare_request(req)
>   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 359, in prepare_request
>     hooks=merge_hooks(request.hooks, self.hooks),
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 287, in prepare
>     self.prepare_url(url, params)
>   File "/usr/lib/python2.7/dist-packages/requests/models.py", line 338, in prepare_url
>     "Perhaps you meant http://{0}?".format(url))
> MissingSchema: Invalid URL u'/admin/config': No schema supplied. Perhaps you meant http:///admin/config?
>
>
>
> Best regards
>
> -----Message d'origine-----
> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la 
> part de Alfredo Deza Envoyé : lundi 2 février 2015 19:42 À : CHEVALIER 
> Ghislain IMT/OLPS Cc : ceph-devel@vger.kernel.org Objet : Re:
> [Ceph-devel] radosgw-agent failed to parse
>
> If you use the `-v` flag you will get very verbose output.
>
> On Mon, Feb 2, 2015 at 12:12 PM,  <ghislain.chevalier@orange.com> wrote:
>> OK I will send them asap
>>
>> Logs are not very verbose.. Can I set a debug mode?
>>
>> -----Message d'origine-----
>> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De la 
>> part de Alfredo Deza Envoyé : lundi 2 février 2015 16:59 À :
>> CHEVALIER Ghislain IMT/OLPS Cc :
>> Objet : Re: [Ceph-devel] radosgw-agent failed to parse
>>
>> That sounds suspicious because we haven't packaged a 1.2.1 release.
>>
>> The latest on PyPI is 1.2, which is the same in the master branch.
>>
>> Would you mind sending logs that show *how* the url is malformed (both when it is OK and when it is not) ?
>>
>> On Mon, Feb 2, 2015 at 9:53 AM,  <ghislain.chevalier@orange.com> wrote:
>>> HI,
>>>
>>> Thx for replying
>>>
>>> According to dpkg -l, it's 1.2.1.
>>>
>>> I noticed that the URL is malformed when launching using directly 
>>> radosgw-agent -c <syncfile> but well formed when launching using 
>>> service radosgw-agent start <syncfile>
>>>
>>> best regards
>>>
>>> -----Message d'origine-----
>>> De : alfredo.deza@inktank.com [mailto:alfredo.deza@inktank.com] De 
>>> la part de Alfredo Deza Envoyé : lundi 2 février 2015 15:18 À :
>>> CHEVALIER Ghislain IMT/OLPS Cc : ceph-devel@vger.kernel.org Objet : Re:
>>> [Ceph-devel] radosgw-agent failed to parse
>>>
>>> What version of the agent are you using? And when it fails, how does the log output look?
>>>
>>> On Mon, Feb 2, 2015 at 8:56 AM,  <ghislain.chevalier@orange.com> wrote:
>>>> Hi all,
>>>>
>>>> Context : Ubuntu 14.04 TLS firefly 0.80.8
>>>>
>>>> I sent this post in ceph-users (identical subject) because I recently encountered the same issue.
>>>> Maybe I missed something between July and January...
>>>>
>>>> I found that the http request wasn't correctly formed by 
>>>> /usr/lib/python2.7/dist-packages/radosgw_agent/client.py
>>>>
>>>> I did the changes below
>>>> #    url = '{protocol}://{host}{path}'.format(protocol=request.protocol,
>>>> #                                             host=request.host,
>>>> #                                             path=request.path)
>>>>      url = '{path}'.format(protocol="", host="", path=request.path)
>>>>
>>>> The request is then correctly formed and sent.
>>>>
>>>> I still have problems setting the federation between 2 zones of 2 ceph clusters in 2 regions.
>>>>
>>>> I go on investigating.
>>>>
>>>> Best regards
>>>>
>>>>
>>>> ___________________________________________________________________
>>>> _ _ _ ___________________________________________________
>>>>
>>>> Ce message et ses pieces jointes peuvent contenir des informations 
>>>> confidentielles ou privilegiees et ne doivent donc pas etre 
>>>> diffuses, exploites ou copies sans autorisation. Si vous avez recu 
>>>> ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>>>
>>>> This message and its attachments may contain confidential or 
>>>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>>>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>>>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>>>> Thank you.
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>>> in the body of a message to majordomo@vger.kernel.org More 
>>>> majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>> ____________________________________________________________________
>>> _ _ ___________________________________________________
>>>
>>> Ce message et ses pieces jointes peuvent contenir des informations 
>>> confidentielles ou privilegiees et ne doivent donc pas etre 
>>> diffuses, exploites ou copies sans autorisation. Si vous avez recu 
>>> ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>>
>>> This message and its attachments may contain confidential or 
>>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>>> Thank you.
>>>
>>
>> _____________________________________________________________________
>> _ ___________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations 
>> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
>> exploites ou copies sans autorisation. Si vous avez recu ce message 
>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or 
>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>> Thank you.
>>
>
> ______________________________________________________________________
> ___________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
> exploites ou copies sans autorisation. Si vous avez recu ce message 
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
> This message and its attachments may contain confidential or 
> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
>

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.


_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.