All of lore.kernel.org
 help / color / mirror / Atom feed
* RBD rights acting up.
@ 2017-07-03 17:51 Willem Jan Withagen
  2017-07-03 18:00 ` Gregory Farnum
  0 siblings, 1 reply; 7+ messages in thread
From: Willem Jan Withagen @ 2017-07-03 17:51 UTC (permalink / raw)
  To: Ceph Development


Hi,

I do not seem to have luck with working with the rights structure in
Ceph. But please help me out with my ignorance....

I have setup the FreeBSD cluster running in 5 ceph jails/vms. That works
as a charm, and I can use ceph-fuse to mount a FS and write/read/... on
it when I'm outside the ceph jails. For that I copies ceph.conf and
ceph.client.admin.keyring to the current system I would like to connect
with.

Also things like ceph -s and rados commands just work as a charm.

So the next check is to test rbd-ggate, the freebsd variant to map
rbd-images to a device.... But then I start to get things like:
----
# rbd info rbd/testrbdggate45846
2017-07-03 19:41:00.562567 80ee53b00 -1 librbd::image::OpenRequest:
failed to retrieve create_timestamp: (1) Operation not permitted
2017-07-03 19:41:00.562685 80f8b2000 -1 librbd::ImageState: 0x80ef23640
failed to open image: (1) Operation not permitted
rbd: error opening image testrbdggate45846: (1) Operation not permitted
----

Which is definitly a rights issue, because I can do that without much
trouble on one of the Ceph servers:
----
# jexec ceph_0 rbd info rbd/testrbdggate45846
rbd image 'testrbdggate45846':
        size 65536 kB in 16 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.10e3c0b1daf
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff,
deep-flatten
        flags:
----

So why does this work from a server that is actually running
mon/osd/mgr, but not from a server that has the same
ceph.client.admin.keyring:
[client.admin]
        key = AQBUXllZMNZSARAAsSexLBkWJQS6vHi9u3rrSA==
        auid = 0
        caps mds = "allow *"
        caps mgr = "allow *"
        caps mon = "allow *"
        caps osd = "allow *"

and ceph auth list:
installed auth entries:

mds.a
        key: AQA+YFlZEHtzKRAAm7Ihhsp2bq2z1YrfmMKnXg==
        caps: [mds] allow
        caps: [mgr] allow profile
 mds
        caps: [mon] allow profile mds
        caps: [osd] allow *
osd.0
        key: AQCLXllZ6NL+BBAAzZH3lnPOR7jOdpsHZz9+FA==
        caps: [mon] allow profile osd
        caps: [osd] allow *
osd.1
        key: AQDpXllZeNPKDBAAMzXKut/xcPQP43l2UzTyQw==
        caps: [mon] allow profile osd
        caps: [osd] allow *
osd.2
        key: AQAkX1lZmL4FOhAAcgVJ5lno+abOyGq7TOdpKg==
        caps: [mon] allow profile osd
        caps: [osd] allow *
osd.3
        key: AQBlX1lZyBxRARAAccH0W16MQ4EMPT+y3Bh5ag==
        caps: [mon] allow profile osd
        caps: [osd] allow *
osd.4
        key: AQCtX1lZaOnOGxAAFSK3sCq/pbqXARi9oELAxA==
        caps: [mon] allow profile osd
        caps: [osd] allow *
osd.5
        key: AQD5X1lZ2KhCGRAAFda2ttOumRXxLGU6CU26Fw==
        caps: [mon] allow profile osd
        caps: [osd] allow *
osd.6
        key: AQA8YFlZkLPDBxAAe6YlnCCUTQlgQh0UvWw6Fg==
        caps: [mon] allow profile osd
        caps: [osd] allow *
client.admin
        key: AQBUXllZMNZSARAAsSexLBkWJQS6vHi9u3rrSA==
        auid: 0
        caps: [mds] allow *
        caps: [mgr] allow *
        caps: [mon] allow *
        caps: [osd] allow *
client.bootstrap-mds
        key: AQBWXllZ6DHwMRAA53Zp9126rH/b5IZr0y2j/A==
        caps: [mon] allow profile bootstrap-mds
client.bootstrap-osd
        key: AQBWXllZWD3EDxAAhaJ41UcTwlQVJCHzw+tBkA==
        caps: [mon] allow profile bootstrap-osd
client.bootstrap-rgw
        key: AQBWXllZ0EUAIRAAfUgvfH7mIzF2cTKXbd8yPg==
        caps: [mon] allow profile bootstrap-rgw
client.ggate
        key: AQANP1pZGNDmNRAAVb8NRXUMmWYh9i1nju6rYA==
        caps: [mon] allow r
        caps: [osd] allow class-read object_prefix rbd_children, allow
rwx pool=rbd
client.kernelrbd
        key: AQBzP1pZEO4nLxAABljwihF2xHFzZfUcc+CJtg==
        caps: [mon] allow r
        caps: [osd] allow class-read object_prefix rbd_children, allow
rwx pool=rbd
mgr.a
        key: AQA/YFlZkA7tDxAAAvC49DNN2VuTTSfCJ1nH+w==
        caps: [mds] allow *
        caps: [mon] allow profile mgr
        caps: [osd] allow *
mgr.x
        key: AQA+YFlZcE52ERAASdHoQn7DZ4G6ialDJg922g==
        caps: [mds] allow *
        caps: [mon] allow profile mgr
        caps: [osd] allow *


Thanx,
--WjW



^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: RBD rights acting up.
  2017-07-03 17:51 RBD rights acting up Willem Jan Withagen
@ 2017-07-03 18:00 ` Gregory Farnum
  2017-07-03 19:24   ` Willem Jan Withagen
  0 siblings, 1 reply; 7+ messages in thread
From: Gregory Farnum @ 2017-07-03 18:00 UTC (permalink / raw)
  To: Willem Jan Withagen; +Cc: Ceph Development

On Mon, Jul 3, 2017 at 10:51 AM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>
> Hi,
>
> I do not seem to have luck with working with the rights structure in
> Ceph. But please help me out with my ignorance....
>
> I have setup the FreeBSD cluster running in 5 ceph jails/vms. That works
> as a charm, and I can use ceph-fuse to mount a FS and write/read/... on
> it when I'm outside the ceph jails. For that I copies ceph.conf and
> ceph.client.admin.keyring to the current system I would like to connect
> with.
>
> Also things like ceph -s and rados commands just work as a charm.
>
> So the next check is to test rbd-ggate, the freebsd variant to map
> rbd-images to a device.... But then I start to get things like:
> ----
> # rbd info rbd/testrbdggate45846
> 2017-07-03 19:41:00.562567 80ee53b00 -1 librbd::image::OpenRequest:
> failed to retrieve create_timestamp: (1) Operation not permitted
> 2017-07-03 19:41:00.562685 80f8b2000 -1 librbd::ImageState: 0x80ef23640
> failed to open image: (1) Operation not permitted
> rbd: error opening image testrbdggate45846: (1) Operation not permitted
> ----
>
> Which is definitly a rights issue, because I can do that without much
> trouble on one of the Ceph servers:
> ----
> # jexec ceph_0 rbd info rbd/testrbdggate45846
> rbd image 'testrbdggate45846':
>         size 65536 kB in 16 objects
>         order 22 (4096 kB objects)
>         block_name_prefix: rbd_data.10e3c0b1daf
>         format: 2
>         features: layering, exclusive-lock, object-map, fast-diff,
> deep-flatten
>         flags:
> ----
>
> So why does this work from a server that is actually running
> mon/osd/mgr, but not from a server that has the same
> ceph.client.admin.keyring:
> [client.admin]
>         key = AQBUXllZMNZSARAAsSexLBkWJQS6vHi9u3rrSA==
>         auid = 0
>         caps mds = "allow *"
>         caps mgr = "allow *"
>         caps mon = "allow *"
>         caps osd = "allow *"
>
> and ceph auth list:
> installed auth entries:
>
> mds.a
>         key: AQA+YFlZEHtzKRAAm7Ihhsp2bq2z1YrfmMKnXg==
>         caps: [mds] allow
>         caps: [mgr] allow profile
>  mds
>         caps: [mon] allow profile mds
>         caps: [osd] allow *
> osd.0
>         key: AQCLXllZ6NL+BBAAzZH3lnPOR7jOdpsHZz9+FA==
>         caps: [mon] allow profile osd
>         caps: [osd] allow *
> osd.1
>         key: AQDpXllZeNPKDBAAMzXKut/xcPQP43l2UzTyQw==
>         caps: [mon] allow profile osd
>         caps: [osd] allow *
> osd.2
>         key: AQAkX1lZmL4FOhAAcgVJ5lno+abOyGq7TOdpKg==
>         caps: [mon] allow profile osd
>         caps: [osd] allow *
> osd.3
>         key: AQBlX1lZyBxRARAAccH0W16MQ4EMPT+y3Bh5ag==
>         caps: [mon] allow profile osd
>         caps: [osd] allow *
> osd.4
>         key: AQCtX1lZaOnOGxAAFSK3sCq/pbqXARi9oELAxA==
>         caps: [mon] allow profile osd
>         caps: [osd] allow *
> osd.5
>         key: AQD5X1lZ2KhCGRAAFda2ttOumRXxLGU6CU26Fw==
>         caps: [mon] allow profile osd
>         caps: [osd] allow *
> osd.6
>         key: AQA8YFlZkLPDBxAAe6YlnCCUTQlgQh0UvWw6Fg==
>         caps: [mon] allow profile osd
>         caps: [osd] allow *
> client.admin
>         key: AQBUXllZMNZSARAAsSexLBkWJQS6vHi9u3rrSA==
>         auid: 0
>         caps: [mds] allow *
>         caps: [mgr] allow *
>         caps: [mon] allow *
>         caps: [osd] allow *
> client.bootstrap-mds
>         key: AQBWXllZ6DHwMRAA53Zp9126rH/b5IZr0y2j/A==
>         caps: [mon] allow profile bootstrap-mds
> client.bootstrap-osd
>         key: AQBWXllZWD3EDxAAhaJ41UcTwlQVJCHzw+tBkA==
>         caps: [mon] allow profile bootstrap-osd
> client.bootstrap-rgw
>         key: AQBWXllZ0EUAIRAAfUgvfH7mIzF2cTKXbd8yPg==
>         caps: [mon] allow profile bootstrap-rgw
> client.ggate
>         key: AQANP1pZGNDmNRAAVb8NRXUMmWYh9i1nju6rYA==
>         caps: [mon] allow r
>         caps: [osd] allow class-read object_prefix rbd_children, allow
> rwx pool=rbd

Presumably ggate is using this client. I'm not sure off-hand what
permissions RBD needs, but I presume it needs class-write in order to
create new images (and they probably won't be prefixed with
rbd_children?).
-Greg

> client.kernelrbd
>         key: AQBzP1pZEO4nLxAABljwihF2xHFzZfUcc+CJtg==
>         caps: [mon] allow r
>         caps: [osd] allow class-read object_prefix rbd_children, allow
> rwx pool=rbd
> mgr.a
>         key: AQA/YFlZkA7tDxAAAvC49DNN2VuTTSfCJ1nH+w==
>         caps: [mds] allow *
>         caps: [mon] allow profile mgr
>         caps: [osd] allow *
> mgr.x
>         key: AQA+YFlZcE52ERAASdHoQn7DZ4G6ialDJg922g==
>         caps: [mds] allow *
>         caps: [mon] allow profile mgr
>         caps: [osd] allow *
>
>
> Thanx,
> --WjW
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: RBD rights acting up.
  2017-07-03 18:00 ` Gregory Farnum
@ 2017-07-03 19:24   ` Willem Jan Withagen
  2017-07-06  1:20     ` Jason Dillaman
  0 siblings, 1 reply; 7+ messages in thread
From: Willem Jan Withagen @ 2017-07-03 19:24 UTC (permalink / raw)
  To: Gregory Farnum; +Cc: Ceph Development



Op 3-7-2017 om 20:00 schreef Gregory Farnum:
> On Mon, Jul 3, 2017 at 10:51 AM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>> Hi,
>>
>> I do not seem to have luck with working with the rights structure in
>> Ceph. But please help me out with my ignorance....
>>
>> I have setup the FreeBSD cluster running in 5 ceph jails/vms. That works
>> as a charm, and I can use ceph-fuse to mount a FS and write/read/... on
>> it when I'm outside the ceph jails. For that I copies ceph.conf and
>> ceph.client.admin.keyring to the current system I would like to connect
>> with.
>>
>> Also things like ceph -s and rados commands just work as a charm.
>>
>> So the next check is to test rbd-ggate, the freebsd variant to map
>> rbd-images to a device.... But then I start to get things like:
>> ----
>> # rbd info rbd/testrbdggate45846
>> 2017-07-03 19:41:00.562567 80ee53b00 -1 librbd::image::OpenRequest:
>> failed to retrieve create_timestamp: (1) Operation not permitted
>> 2017-07-03 19:41:00.562685 80f8b2000 -1 librbd::ImageState: 0x80ef23640
>> failed to open image: (1) Operation not permitted
>> rbd: error opening image testrbdggate45846: (1) Operation not permitted
>> ----
>>
>> Which is definitly a rights issue, because I can do that without much
>> trouble on one of the Ceph servers:
>> ----
>> # jexec ceph_0 rbd info rbd/testrbdggate45846
>> rbd image 'testrbdggate45846':
>>          size 65536 kB in 16 objects
>>          order 22 (4096 kB objects)
>>          block_name_prefix: rbd_data.10e3c0b1daf
>>          format: 2
>>          features: layering, exclusive-lock, object-map, fast-diff,
>> deep-flatten
>>          flags:
>> ----
>>
>> So why does this work from a server that is actually running
>> mon/osd/mgr, but not from a server that has the same
>> ceph.client.admin.keyring:
>> [client.admin]
>>          key = AQBUXllZMNZSARAAsSexLBkWJQS6vHi9u3rrSA==
>>          auid = 0
>>          caps mds = "allow *"
>>          caps mgr = "allow *"
>>          caps mon = "allow *"
>>          caps osd = "allow *"
>>
>> and ceph auth list:
>> installed auth entries:
>> client.admin
>>          key: AQBUXllZMNZSARAAsSexLBkWJQS6vHi9u3rrSA==
>>          auid: 0
>>          caps: [mds] allow *
>>          caps: [mgr] allow *
>>          caps: [mon] allow *
>>          caps: [osd] allow *

>> client.ggate
>>          key: AQANP1pZGNDmNRAAVb8NRXUMmWYh9i1nju6rYA==
>>          caps: [mon] allow r
>>          caps: [osd] allow class-read object_prefix rbd_children, allow
>> rwx pool=rbd
> Presumably ggate is using this client. I'm not sure off-hand what
> permissions RBD needs, but I presume it needs class-write in order to
> create new images (and they probably won't be prefixed with
> rbd_children?).

Well,  client.ggate was an attempt to mirror what we have in our working 
OpenStack ceph.

But my major concern is that it all works on a host that is actually 
part of the cluster (as admin),
but it does not work for all rbd commands once I start working on a 
separate host.
Ceph and rados are oke, but certain rbs commands fail.

So why doesn't it work as admin on an 'external'  host.

--WjW


^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: RBD rights acting up.
  2017-07-03 19:24   ` Willem Jan Withagen
@ 2017-07-06  1:20     ` Jason Dillaman
  2017-07-06  8:00       ` Willem Jan Withagen
  0 siblings, 1 reply; 7+ messages in thread
From: Jason Dillaman @ 2017-07-06  1:20 UTC (permalink / raw)
  To: Willem Jan Withagen; +Cc: Gregory Farnum, Ceph Development

Any chance that you have the OSD log for the PGs mapped to that
image's header? Since that method is brand new, I wonder if the
-EOPNOTSUPP error is being somehow mapped to -EPERM under FreeBSD(?).

On Mon, Jul 3, 2017 at 3:24 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>
>
> Op 3-7-2017 om 20:00 schreef Gregory Farnum:
>>
>> On Mon, Jul 3, 2017 at 10:51 AM, Willem Jan Withagen <wjw@digiware.nl>
>> wrote:
>>>
>>> Hi,
>>>
>>> I do not seem to have luck with working with the rights structure in
>>> Ceph. But please help me out with my ignorance....
>>>
>>> I have setup the FreeBSD cluster running in 5 ceph jails/vms. That works
>>> as a charm, and I can use ceph-fuse to mount a FS and write/read/... on
>>> it when I'm outside the ceph jails. For that I copies ceph.conf and
>>> ceph.client.admin.keyring to the current system I would like to connect
>>> with.
>>>
>>> Also things like ceph -s and rados commands just work as a charm.
>>>
>>> So the next check is to test rbd-ggate, the freebsd variant to map
>>> rbd-images to a device.... But then I start to get things like:
>>> ----
>>> # rbd info rbd/testrbdggate45846
>>> 2017-07-03 19:41:00.562567 80ee53b00 -1 librbd::image::OpenRequest:
>>> failed to retrieve create_timestamp: (1) Operation not permitted
>>> 2017-07-03 19:41:00.562685 80f8b2000 -1 librbd::ImageState: 0x80ef23640
>>> failed to open image: (1) Operation not permitted
>>> rbd: error opening image testrbdggate45846: (1) Operation not permitted
>>> ----
>>>
>>> Which is definitly a rights issue, because I can do that without much
>>> trouble on one of the Ceph servers:
>>> ----
>>> # jexec ceph_0 rbd info rbd/testrbdggate45846
>>> rbd image 'testrbdggate45846':
>>>          size 65536 kB in 16 objects
>>>          order 22 (4096 kB objects)
>>>          block_name_prefix: rbd_data.10e3c0b1daf
>>>          format: 2
>>>          features: layering, exclusive-lock, object-map, fast-diff,
>>> deep-flatten
>>>          flags:
>>> ----
>>>
>>> So why does this work from a server that is actually running
>>> mon/osd/mgr, but not from a server that has the same
>>> ceph.client.admin.keyring:
>>> [client.admin]
>>>          key = AQBUXllZMNZSARAAsSexLBkWJQS6vHi9u3rrSA==
>>>          auid = 0
>>>          caps mds = "allow *"
>>>          caps mgr = "allow *"
>>>          caps mon = "allow *"
>>>          caps osd = "allow *"
>>>
>>> and ceph auth list:
>>> installed auth entries:
>>> client.admin
>>>          key: AQBUXllZMNZSARAAsSexLBkWJQS6vHi9u3rrSA==
>>>          auid: 0
>>>          caps: [mds] allow *
>>>          caps: [mgr] allow *
>>>          caps: [mon] allow *
>>>          caps: [osd] allow *
>
>
>>> client.ggate
>>>          key: AQANP1pZGNDmNRAAVb8NRXUMmWYh9i1nju6rYA==
>>>          caps: [mon] allow r
>>>          caps: [osd] allow class-read object_prefix rbd_children, allow
>>> rwx pool=rbd
>>
>> Presumably ggate is using this client. I'm not sure off-hand what
>> permissions RBD needs, but I presume it needs class-write in order to
>> create new images (and they probably won't be prefixed with
>> rbd_children?).
>
>
> Well,  client.ggate was an attempt to mirror what we have in our working
> OpenStack ceph.
>
> But my major concern is that it all works on a host that is actually part of
> the cluster (as admin),
> but it does not work for all rbd commands once I start working on a separate
> host.
> Ceph and rados are oke, but certain rbs commands fail.
>
> So why doesn't it work as admin on an 'external'  host.
>
>
> --WjW
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Jason

^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: RBD rights acting up.
  2017-07-06  1:20     ` Jason Dillaman
@ 2017-07-06  8:00       ` Willem Jan Withagen
  2017-07-06 11:24         ` Jason Dillaman
  0 siblings, 1 reply; 7+ messages in thread
From: Willem Jan Withagen @ 2017-07-06  8:00 UTC (permalink / raw)
  To: dillaman; +Cc: Gregory Farnum, Ceph Development

On 6-7-2017 03:20, Jason Dillaman wrote:
> Any chance that you have the OSD log for the PGs mapped to that
> image's header? Since that method is brand new, I wonder if the
> -EOPNOTSUPP error is being somehow mapped to -EPERM under FreeBSD(?).

Hi Jason,

Getting annoyed witht the fact that I could not get it right, I thrashed
the cluster and redid it. So no more logs...
I think we must write this up to "pilot error".

And then it worked after I build a new cluster..
Now the difference is that the second time I created the mgr rights in
the client keyring right from the bat. It was missing that in earlier
version of my build-cluster script.

So my conclusion is that all tries I did to get a working keyring with
mgr rights were unsuccessful.

But your answer triggers something for me to check....
There is now conversion of the errorcodes in FreeBSD servers and
clients. All wire messages should contain the "Linux" version of the
errorcode, and are translated upon exit or arrival.
Problem is there what to do with errors that do not have a matching
counterpart. ATM they are all translated to EPERM, so once converted the
meaning is lost, and the errorcode stays at EPERM.

This gives me an idea though. Perhaps I should convert errorcodes
without Linux counterpart to 1000+errocode. Once they arrive at the
other end, it is time to decide how to translate back:
	if it is a FreeBSD system, return wirecode-1000
	if it is a linux system return EPERM.
(And start looking into the Errorcode class)

Fortunatelly that error type is known on both sides of the fence:
Linux:#define        EOPNOTSUPP      95
FreeBSD:#define      EOPNOTSUPP      45
And should be translatable without loss of information.

Thing is, the full extend of the PR set got merged like only yesterday
and I'm testing a version of v12.1.0 with without that translation part.
The package was last build on Tuesday 4th.

So I guess it must have been me, being clumsy with rights.

--WjW

> On Mon, Jul 3, 2017 at 3:24 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>>
>>
>> Op 3-7-2017 om 20:00 schreef Gregory Farnum:
>>>
>>> On Mon, Jul 3, 2017 at 10:51 AM, Willem Jan Withagen <wjw@digiware.nl>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I do not seem to have luck with working with the rights structure in
>>>> Ceph. But please help me out with my ignorance....
>>>>
>>>> I have setup the FreeBSD cluster running in 5 ceph jails/vms. That works
>>>> as a charm, and I can use ceph-fuse to mount a FS and write/read/... on
>>>> it when I'm outside the ceph jails. For that I copies ceph.conf and
>>>> ceph.client.admin.keyring to the current system I would like to connect
>>>> with.
>>>>
>>>> Also things like ceph -s and rados commands just work as a charm.
>>>>
>>>> So the next check is to test rbd-ggate, the freebsd variant to map
>>>> rbd-images to a device.... But then I start to get things like:
>>>> ----
>>>> # rbd info rbd/testrbdggate45846
>>>> 2017-07-03 19:41:00.562567 80ee53b00 -1 librbd::image::OpenRequest:
>>>> failed to retrieve create_timestamp: (1) Operation not permitted
>>>> 2017-07-03 19:41:00.562685 80f8b2000 -1 librbd::ImageState: 0x80ef23640
>>>> failed to open image: (1) Operation not permitted
>>>> rbd: error opening image testrbdggate45846: (1) Operation not permitted
>>>> ----
>>>>
>>>> Which is definitly a rights issue, because I can do that without much
>>>> trouble on one of the Ceph servers:
>>>> ----
>>>> # jexec ceph_0 rbd info rbd/testrbdggate45846
>>>> rbd image 'testrbdggate45846':
>>>>          size 65536 kB in 16 objects
>>>>          order 22 (4096 kB objects)
>>>>          block_name_prefix: rbd_data.10e3c0b1daf
>>>>          format: 2
>>>>          features: layering, exclusive-lock, object-map, fast-diff,
>>>> deep-flatten
>>>>          flags:
>>>> ----
>>>>
>>>> So why does this work from a server that is actually running
>>>> mon/osd/mgr, but not from a server that has the same
>>>> ceph.client.admin.keyring:
>>>> [client.admin]
>>>>          key = AQBUXllZMNZSARAAsSexLBkWJQS6vHi9u3rrSA==
>>>>          auid = 0
>>>>          caps mds = "allow *"
>>>>          caps mgr = "allow *"
>>>>          caps mon = "allow *"
>>>>          caps osd = "allow *"
>>>>
>>>> and ceph auth list:
>>>> installed auth entries:
>>>> client.admin
>>>>          key: AQBUXllZMNZSARAAsSexLBkWJQS6vHi9u3rrSA==
>>>>          auid: 0
>>>>          caps: [mds] allow *
>>>>          caps: [mgr] allow *
>>>>          caps: [mon] allow *
>>>>          caps: [osd] allow *
>>
>>
>>>> client.ggate
>>>>          key: AQANP1pZGNDmNRAAVb8NRXUMmWYh9i1nju6rYA==
>>>>          caps: [mon] allow r
>>>>          caps: [osd] allow class-read object_prefix rbd_children, allow
>>>> rwx pool=rbd
>>>
>>> Presumably ggate is using this client. I'm not sure off-hand what
>>> permissions RBD needs, but I presume it needs class-write in order to
>>> create new images (and they probably won't be prefixed with
>>> rbd_children?).
>>
>>
>> Well,  client.ggate was an attempt to mirror what we have in our working
>> OpenStack ceph.
>>
>> But my major concern is that it all works on a host that is actually part of
>> the cluster (as admin),
>> but it does not work for all rbd commands once I start working on a separate
>> host.
>> Ceph and rados are oke, but certain rbs commands fail.
>>
>> So why doesn't it work as admin on an 'external'  host.
>>
>>
>> --WjW
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 


^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: RBD rights acting up.
  2017-07-06  8:00       ` Willem Jan Withagen
@ 2017-07-06 11:24         ` Jason Dillaman
  2017-07-06 12:27           ` Willem Jan Withagen
  0 siblings, 1 reply; 7+ messages in thread
From: Jason Dillaman @ 2017-07-06 11:24 UTC (permalink / raw)
  To: Willem Jan Withagen; +Cc: Gregory Farnum, Ceph Development

I leaning towards not a real permission issue since by the time the
"open image" state machine reached that point of failure, it would
have already successfully executed several getter class methods on the
same object -- just like the one that failed.

On Thu, Jul 6, 2017 at 4:00 AM, Willem Jan Withagen <wjw@digiware.nl> wrote:
> On 6-7-2017 03:20, Jason Dillaman wrote:
>> Any chance that you have the OSD log for the PGs mapped to that
>> image's header? Since that method is brand new, I wonder if the
>> -EOPNOTSUPP error is being somehow mapped to -EPERM under FreeBSD(?).
>
> Hi Jason,
>
> Getting annoyed witht the fact that I could not get it right, I thrashed
> the cluster and redid it. So no more logs...
> I think we must write this up to "pilot error".
>
> And then it worked after I build a new cluster..
> Now the difference is that the second time I created the mgr rights in
> the client keyring right from the bat. It was missing that in earlier
> version of my build-cluster script.
>
> So my conclusion is that all tries I did to get a working keyring with
> mgr rights were unsuccessful.
>
> But your answer triggers something for me to check....
> There is now conversion of the errorcodes in FreeBSD servers and
> clients. All wire messages should contain the "Linux" version of the
> errorcode, and are translated upon exit or arrival.
> Problem is there what to do with errors that do not have a matching
> counterpart. ATM they are all translated to EPERM, so once converted the
> meaning is lost, and the errorcode stays at EPERM.
>
> This gives me an idea though. Perhaps I should convert errorcodes
> without Linux counterpart to 1000+errocode. Once they arrive at the
> other end, it is time to decide how to translate back:
>         if it is a FreeBSD system, return wirecode-1000
>         if it is a linux system return EPERM.
> (And start looking into the Errorcode class)
>
> Fortunatelly that error type is known on both sides of the fence:
> Linux:#define        EOPNOTSUPP      95
> FreeBSD:#define      EOPNOTSUPP      45
> And should be translatable without loss of information.
>
> Thing is, the full extend of the PR set got merged like only yesterday
> and I'm testing a version of v12.1.0 with without that translation part.
> The package was last build on Tuesday 4th.
>
> So I guess it must have been me, being clumsy with rights.
>
> --WjW
>
>> On Mon, Jul 3, 2017 at 3:24 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>>>
>>>
>>> Op 3-7-2017 om 20:00 schreef Gregory Farnum:
>>>>
>>>> On Mon, Jul 3, 2017 at 10:51 AM, Willem Jan Withagen <wjw@digiware.nl>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I do not seem to have luck with working with the rights structure in
>>>>> Ceph. But please help me out with my ignorance....
>>>>>
>>>>> I have setup the FreeBSD cluster running in 5 ceph jails/vms. That works
>>>>> as a charm, and I can use ceph-fuse to mount a FS and write/read/... on
>>>>> it when I'm outside the ceph jails. For that I copies ceph.conf and
>>>>> ceph.client.admin.keyring to the current system I would like to connect
>>>>> with.
>>>>>
>>>>> Also things like ceph -s and rados commands just work as a charm.
>>>>>
>>>>> So the next check is to test rbd-ggate, the freebsd variant to map
>>>>> rbd-images to a device.... But then I start to get things like:
>>>>> ----
>>>>> # rbd info rbd/testrbdggate45846
>>>>> 2017-07-03 19:41:00.562567 80ee53b00 -1 librbd::image::OpenRequest:
>>>>> failed to retrieve create_timestamp: (1) Operation not permitted
>>>>> 2017-07-03 19:41:00.562685 80f8b2000 -1 librbd::ImageState: 0x80ef23640
>>>>> failed to open image: (1) Operation not permitted
>>>>> rbd: error opening image testrbdggate45846: (1) Operation not permitted
>>>>> ----
>>>>>
>>>>> Which is definitly a rights issue, because I can do that without much
>>>>> trouble on one of the Ceph servers:
>>>>> ----
>>>>> # jexec ceph_0 rbd info rbd/testrbdggate45846
>>>>> rbd image 'testrbdggate45846':
>>>>>          size 65536 kB in 16 objects
>>>>>          order 22 (4096 kB objects)
>>>>>          block_name_prefix: rbd_data.10e3c0b1daf
>>>>>          format: 2
>>>>>          features: layering, exclusive-lock, object-map, fast-diff,
>>>>> deep-flatten
>>>>>          flags:
>>>>> ----
>>>>>
>>>>> So why does this work from a server that is actually running
>>>>> mon/osd/mgr, but not from a server that has the same
>>>>> ceph.client.admin.keyring:
>>>>> [client.admin]
>>>>>          key = AQBUXllZMNZSARAAsSexLBkWJQS6vHi9u3rrSA==
>>>>>          auid = 0
>>>>>          caps mds = "allow *"
>>>>>          caps mgr = "allow *"
>>>>>          caps mon = "allow *"
>>>>>          caps osd = "allow *"
>>>>>
>>>>> and ceph auth list:
>>>>> installed auth entries:
>>>>> client.admin
>>>>>          key: AQBUXllZMNZSARAAsSexLBkWJQS6vHi9u3rrSA==
>>>>>          auid: 0
>>>>>          caps: [mds] allow *
>>>>>          caps: [mgr] allow *
>>>>>          caps: [mon] allow *
>>>>>          caps: [osd] allow *
>>>
>>>
>>>>> client.ggate
>>>>>          key: AQANP1pZGNDmNRAAVb8NRXUMmWYh9i1nju6rYA==
>>>>>          caps: [mon] allow r
>>>>>          caps: [osd] allow class-read object_prefix rbd_children, allow
>>>>> rwx pool=rbd
>>>>
>>>> Presumably ggate is using this client. I'm not sure off-hand what
>>>> permissions RBD needs, but I presume it needs class-write in order to
>>>> create new images (and they probably won't be prefixed with
>>>> rbd_children?).
>>>
>>>
>>> Well,  client.ggate was an attempt to mirror what we have in our working
>>> OpenStack ceph.
>>>
>>> But my major concern is that it all works on a host that is actually part of
>>> the cluster (as admin),
>>> but it does not work for all rbd commands once I start working on a separate
>>> host.
>>> Ceph and rados are oke, but certain rbs commands fail.
>>>
>>> So why doesn't it work as admin on an 'external'  host.
>>>
>>>
>>> --WjW
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Jason

^ permalink raw reply	[flat|nested] 7+ messages in thread
* Re: RBD rights acting up.
  2017-07-06 11:24         ` Jason Dillaman
@ 2017-07-06 12:27           ` Willem Jan Withagen
  0 siblings, 0 replies; 7+ messages in thread
From: Willem Jan Withagen @ 2017-07-06 12:27 UTC (permalink / raw)
  To: dillaman; +Cc: Gregory Farnum, Ceph Development

On 6-7-2017 13:24, Jason Dillaman wrote:
> I leaning towards not a real permission issue since by the time the
> "open image" state machine reached that point of failure, it would
> have already successfully executed several getter class methods on the
> same object -- just like the one that failed.

Correct, that was indeed a bit surprising. I sort of assumed that it had
more to do with the type of operation and that opening an image the call
was that had access-right problems.
But if the are equal for all, then we might attribute it to errorcode
diffs which is very well possible since the changes went in with several
PRs at different days..

I'll post again, when I run into it.

Thanx for clearing this,

--WjW

> On Thu, Jul 6, 2017 at 4:00 AM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>> On 6-7-2017 03:20, Jason Dillaman wrote:
>>> Any chance that you have the OSD log for the PGs mapped to that
>>> image's header? Since that method is brand new, I wonder if the
>>> -EOPNOTSUPP error is being somehow mapped to -EPERM under FreeBSD(?).
>>
>> Hi Jason,
>>
>> Getting annoyed witht the fact that I could not get it right, I thrashed
>> the cluster and redid it. So no more logs...
>> I think we must write this up to "pilot error".
>>
>> And then it worked after I build a new cluster..
>> Now the difference is that the second time I created the mgr rights in
>> the client keyring right from the bat. It was missing that in earlier
>> version of my build-cluster script.
>>
>> So my conclusion is that all tries I did to get a working keyring with
>> mgr rights were unsuccessful.
>>
>> But your answer triggers something for me to check....
>> There is now conversion of the errorcodes in FreeBSD servers and
>> clients. All wire messages should contain the "Linux" version of the
>> errorcode, and are translated upon exit or arrival.
>> Problem is there what to do with errors that do not have a matching
>> counterpart. ATM they are all translated to EPERM, so once converted the
>> meaning is lost, and the errorcode stays at EPERM.
>>
>> This gives me an idea though. Perhaps I should convert errorcodes
>> without Linux counterpart to 1000+errocode. Once they arrive at the
>> other end, it is time to decide how to translate back:
>>         if it is a FreeBSD system, return wirecode-1000
>>         if it is a linux system return EPERM.
>> (And start looking into the Errorcode class)
>>
>> Fortunatelly that error type is known on both sides of the fence:
>> Linux:#define        EOPNOTSUPP      95
>> FreeBSD:#define      EOPNOTSUPP      45
>> And should be translatable without loss of information.
>>
>> Thing is, the full extend of the PR set got merged like only yesterday
>> and I'm testing a version of v12.1.0 with without that translation part.
>> The package was last build on Tuesday 4th.
>>
>> So I guess it must have been me, being clumsy with rights.
>>
>> --WjW
>>
>>> On Mon, Jul 3, 2017 at 3:24 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>>>>
>>>>
>>>> Op 3-7-2017 om 20:00 schreef Gregory Farnum:
>>>>>
>>>>> On Mon, Jul 3, 2017 at 10:51 AM, Willem Jan Withagen <wjw@digiware.nl>
>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I do not seem to have luck with working with the rights structure in
>>>>>> Ceph. But please help me out with my ignorance....
>>>>>>
>>>>>> I have setup the FreeBSD cluster running in 5 ceph jails/vms. That works
>>>>>> as a charm, and I can use ceph-fuse to mount a FS and write/read/... on
>>>>>> it when I'm outside the ceph jails. For that I copies ceph.conf and
>>>>>> ceph.client.admin.keyring to the current system I would like to connect
>>>>>> with.
>>>>>>
>>>>>> Also things like ceph -s and rados commands just work as a charm.
>>>>>>
>>>>>> So the next check is to test rbd-ggate, the freebsd variant to map
>>>>>> rbd-images to a device.... But then I start to get things like:
>>>>>> ----
>>>>>> # rbd info rbd/testrbdggate45846
>>>>>> 2017-07-03 19:41:00.562567 80ee53b00 -1 librbd::image::OpenRequest:
>>>>>> failed to retrieve create_timestamp: (1) Operation not permitted
>>>>>> 2017-07-03 19:41:00.562685 80f8b2000 -1 librbd::ImageState: 0x80ef23640
>>>>>> failed to open image: (1) Operation not permitted
>>>>>> rbd: error opening image testrbdggate45846: (1) Operation not permitted
>>>>>> ----
>>>>>>
>>>>>> Which is definitly a rights issue, because I can do that without much
>>>>>> trouble on one of the Ceph servers:
>>>>>> ----
>>>>>> # jexec ceph_0 rbd info rbd/testrbdggate45846
>>>>>> rbd image 'testrbdggate45846':
>>>>>>          size 65536 kB in 16 objects
>>>>>>          order 22 (4096 kB objects)
>>>>>>          block_name_prefix: rbd_data.10e3c0b1daf
>>>>>>          format: 2
>>>>>>          features: layering, exclusive-lock, object-map, fast-diff,
>>>>>> deep-flatten
>>>>>>          flags:
>>>>>> ----
>>>>>>
>>>>>> So why does this work from a server that is actually running
>>>>>> mon/osd/mgr, but not from a server that has the same
>>>>>> ceph.client.admin.keyring:
>>>>>> [client.admin]
>>>>>>          key = AQBUXllZMNZSARAAsSexLBkWJQS6vHi9u3rrSA==
>>>>>>          auid = 0
>>>>>>          caps mds = "allow *"
>>>>>>          caps mgr = "allow *"
>>>>>>          caps mon = "allow *"
>>>>>>          caps osd = "allow *"
>>>>>>
>>>>>> and ceph auth list:
>>>>>> installed auth entries:
>>>>>> client.admin
>>>>>>          key: AQBUXllZMNZSARAAsSexLBkWJQS6vHi9u3rrSA==
>>>>>>          auid: 0
>>>>>>          caps: [mds] allow *
>>>>>>          caps: [mgr] allow *
>>>>>>          caps: [mon] allow *
>>>>>>          caps: [osd] allow *
>>>>
>>>>
>>>>>> client.ggate
>>>>>>          key: AQANP1pZGNDmNRAAVb8NRXUMmWYh9i1nju6rYA==
>>>>>>          caps: [mon] allow r
>>>>>>          caps: [osd] allow class-read object_prefix rbd_children, allow
>>>>>> rwx pool=rbd
>>>>>
>>>>> Presumably ggate is using this client. I'm not sure off-hand what
>>>>> permissions RBD needs, but I presume it needs class-write in order to
>>>>> create new images (and they probably won't be prefixed with
>>>>> rbd_children?).
>>>>
>>>>
>>>> Well,  client.ggate was an attempt to mirror what we have in our working
>>>> OpenStack ceph.
>>>>
>>>> But my major concern is that it all works on a host that is actually part of
>>>> the cluster (as admin),
>>>> but it does not work for all rbd commands once I start working on a separate
>>>> host.
>>>> Ceph and rados are oke, but certain rbs commands fail.
>>>>
>>>> So why doesn't it work as admin on an 'external'  host.
>>>>
>>>>
>>>> --WjW
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 


^ permalink raw reply	[flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-07-06 12:27 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-07-03 17:51 RBD rights acting up Willem Jan Withagen
2017-07-03 18:00 ` Gregory Farnum
2017-07-03 19:24   ` Willem Jan Withagen
2017-07-06  1:20     ` Jason Dillaman
2017-07-06  8:00       ` Willem Jan Withagen
2017-07-06 11:24         ` Jason Dillaman
2017-07-06 12:27           ` Willem Jan Withagen

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.