getting started with RADOSGW

getting started with RADOSGW

[Jonathan Proulx]

Hi All,

I've created a testuser with testuser:swift subuser, setup apache with
fcgi but client access with either S3 or Swift clients result in HTTP
500 errors on the server:

[Sun Jul 22 10:14:33 2012] [error] [client 128.52.x.x] (2)No such file
or directory: FastCGI: failed to connect to server
"/var/www/s3gw.fcgi": connect() failed
[Sun Jul 22 10:14:33 2012] [error] [client 128.52.x.x] FastCGI:
incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"


 ls -lh /var/www/s3gw.fcgi
-rwxrwxr-x 1 root root 79 Jul 10 10:28 /var/www/s3gw.fcgi

cat /var/www/s3gw.fcgi
#!/bin/sh
exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.rados.gateway

/usr/bin/radosgw is in the right place and executable by all
/etc/ceph/ceph.conf is also in the correct location and readable by
all, so I'm a bit confused by the "No such file or directory" error.

I *think* I've followed all the steps but have obviously missed
something, any idea what?

Thanks,
-Jon

Re: getting started with RADOSGW

[Yehuda Sadeh]

On Sun, Jul 22, 2012 at 7:43 AM, Jonathan Proulx <jon@jonproulx.com> wrote:
> Hi All,
>
> I've created a testuser with testuser:swift subuser, setup apache with
> fcgi but client access with either S3 or Swift clients result in HTTP
> 500 errors on the server:
>
> [Sun Jul 22 10:14:33 2012] [error] [client 128.52.x.x] (2)No such file
> or directory: FastCGI: failed to connect to server
> "/var/www/s3gw.fcgi": connect() failed
> [Sun Jul 22 10:14:33 2012] [error] [client 128.52.x.x] FastCGI:
> incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"
>
>
>  ls -lh /var/www/s3gw.fcgi
> -rwxrwxr-x 1 root root 79 Jul 10 10:28 /var/www/s3gw.fcgi
>
> cat /var/www/s3gw.fcgi
> #!/bin/sh
> exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.rados.gateway
>
> /usr/bin/radosgw is in the right place and executable by all
> /etc/ceph/ceph.conf is also in the correct location and readable by
> all, so I'm a bit confused by the "No such file or directory" error.
>
> I *think* I've followed all the steps but have obviously missed
> something, any idea what?
>
I think you've set up your apache to use external fastcgi, but you
have to run radosgw manually using this method.

Yehuda

Re: getting started with RADOSGW

[Jonathan Proulx]

On Sun, Jul 22, 2012 at 10:59 AM, Yehuda Sadeh <yehuda@inktank.com> wrote:

> I think you've set up your apache to use external fastcgi, but you
> have to run radosgw manually using this method.

Spot on, thanks for the quick response.  By radosgw init script was
quietly exiting because I was using FQDN in ceph.conf but it was
trying to match on the short name.  Now it's failing to start, but at
least logging why.

2012-07-22 11:32:39.636760 7fd2b9be9780  0 librados:
client.radosgw.gateway authentication error (1) Operation not
permitted
2012-07-22 11:32:39.637018 7fd2b9be9780 -1 Couldn't init storage
provider (RADOS)

are these the right capabilities for that user
(http://ceph.com/docs/master/radosgw/config suggests they are)?

client.rados.gateway
	key: <redacted>
	caps: [mon] allow r
	caps: [osd] allow rwx

ceph.conf points to /etc/ceph/keyring.rados.gateway which has is
readable and has matching key

Thanks,
-Jon

Re: getting started with RADOSGW

[Yehuda Sadeh]

On Sun, Jul 22, 2012 at 8:46 AM, Jonathan Proulx <jon@jonproulx.com> wrote:
> On Sun, Jul 22, 2012 at 10:59 AM, Yehuda Sadeh <yehuda@inktank.com> wrote:
>
>> I think you've set up your apache to use external fastcgi, but you
>> have to run radosgw manually using this method.
>
> Spot on, thanks for the quick response.  By radosgw init script was
> quietly exiting because I was using FQDN in ceph.conf but it was
> trying to match on the short name.  Now it's failing to start, but at
> least logging why.
>
> 2012-07-22 11:32:39.636760 7fd2b9be9780  0 librados:
> client.radosgw.gateway authentication error (1) Operation not
> permitted
> 2012-07-22 11:32:39.637018 7fd2b9be9780 -1 Couldn't init storage
> provider (RADOS)
>
> are these the right capabilities for that user
> (http://ceph.com/docs/master/radosgw/config suggests they are)?
>
> client.rados.gateway
>         key: <redacted>
>         caps: [mon] allow r
>         caps: [osd] allow rwx

I think the radosgw needs the 'w' cap for the monitor for
automatically creating the rados pools. Though it may be that you'd be
better off creating the pools yourself with the required amount of pgs
than letting it do that by itself, as the default number of pgs that
will be created is very low.

>
> ceph.conf points to /etc/ceph/keyring.rados.gateway which has is
> readable and has matching key
>
Try running 'ceph auth list' and see if you see the auth info for that
user. If not then you'll need to 'ceph auth add' that keyring.

Yehuda

Re: getting started with RADOSGW

[Jonathan Proulx]

On Sun, Jul 22, 2012 at 12:31 PM, Yehuda Sadeh <yehuda@inktank.com> wrote:
> On Sun, Jul 22, 2012 at 8:46 AM, Jonathan Proulx <jon@jonproulx.com> wrote:

>> are these the right capabilities for that user
>> (http://ceph.com/docs/master/radosgw/config suggests they are)?
>>
>> client.rados.gateway
>>         key: <redacted>
>>         caps: [mon] allow r
>>         caps: [osd] allow rwx
>
> I think the radosgw needs the 'w' cap for the monitor for
> automatically creating the rados pools. Though it may be that you'd be
> better off creating the pools yourself with the required amount of pgs
> than letting it do that by itself, as the default number of pgs that
> will be created is very low.
>> ceph.conf points to /etc/ceph/keyring.rados.gateway which has is
>> readable and has matching key
>>
> Try running 'ceph auth list' and see if you see the auth info for that
> user. If not then you'll need to 'ceph auth add' that keyring.

'ceph auth list' is where I got the capabilites list, though the
keyring file above list the same caps

Hmmm, how do I change the capabilities of a key, that doc section is
blank http://ceph.com/docs/master/ops/manage/key/#capabilities.  I
tried "ceph-authtool -n client.rados.gateway --cap osd 'allow rwx'
--cap mon 'allow rw' /etc/ceph/keyring.rados.gateway" which changed
the keyfile but not the output of "ceph auth list"

And radosgw is still exiting with an auth error...

root@ceph-mon:/tmp/rbd# /etc/init.d/radosgw restart
No /usr/bin/radosgw found running; none killed.
Starting client.radosgw.gateway...
radosgw daemon started with pid 27614
root@ceph-mon:/tmp/rbd# ps 27614
  PID TTY      STAT   TIME COMMAND
root@ceph-all-0:/tmp/rbd# cat /var/log/ceph/radosgw.log
2012-07-22 17:01:38.709151 7f79c3689780  0 librados:
client.radosgw.gateway authentication error (1) Operation not
permitted
2012-07-22 17:01:38.709391 7f79c3689780 -1 Couldn't init storage
provider (RADOS)

Thanks,
-Jon

Re: getting started with RADOSGW

[Yehuda Sadeh]

On Sun, Jul 22, 2012 at 2:04 PM, Jonathan Proulx <jon@jonproulx.com> wrote:
> On Sun, Jul 22, 2012 at 12:31 PM, Yehuda Sadeh <yehuda@inktank.com> wrote:
>> On Sun, Jul 22, 2012 at 8:46 AM, Jonathan Proulx <jon@jonproulx.com> wrote:
>
>>> are these the right capabilities for that user
>>> (http://ceph.com/docs/master/radosgw/config suggests they are)?
>>>
>>> client.rados.gateway
>>>         key: <redacted>
>>>         caps: [mon] allow r
>>>         caps: [osd] allow rwx
>>
>> I think the radosgw needs the 'w' cap for the monitor for
>> automatically creating the rados pools. Though it may be that you'd be
>> better off creating the pools yourself with the required amount of pgs
>> than letting it do that by itself, as the default number of pgs that
>> will be created is very low.
>>> ceph.conf points to /etc/ceph/keyring.rados.gateway which has is
>>> readable and has matching key
>>>
>> Try running 'ceph auth list' and see if you see the auth info for that
>> user. If not then you'll need to 'ceph auth add' that keyring.
>
> 'ceph auth list' is where I got the capabilites list, though the
> keyring file above list the same caps
>
> Hmmm, how do I change the capabilities of a key, that doc section is
> blank http://ceph.com/docs/master/ops/manage/key/#capabilities.  I
> tried "ceph-authtool -n client.rados.gateway --cap osd 'allow rwx'
> --cap mon 'allow rw' /etc/ceph/keyring.rados.gateway" which changed
> the keyfile but not the output of "ceph auth list"
>
> And radosgw is still exiting with an auth error...
>
> root@ceph-mon:/tmp/rbd# /etc/init.d/radosgw restart
> No /usr/bin/radosgw found running; none killed.
> Starting client.radosgw.gateway...

You're starting radosgw with user 'client.radosgw.gateway', whereas
the key is for user 'client.rados.gateway'.

Yehuda