OSD creation and device class

OSD creation and device class

[Loic Dachary]

Hi Sage,

When the OSD is prepared, we add a --device-class option to ceph-disk that creates a device_class file and activate can use it use when creating the the OSD[1]. The ceph osd create command is given an additional optional argument (device class). If the device class argument is present, ceph osd create[2] creates a device entry in the crushmap (it currently does nothing with the crushmap) so that the corresponding device class can be preserved. This happens before update_crush_location[3] is called and since the device is not included in any crush tree it won't be used.

I'm not happy about adding an argument to osd create. We could instead have a new dedicated command create-with-device-class but I like that even less. We could also add a "ceph osd crush create-device" and have ceph-disk call it right after "ceph osd create", followed by "ceph osd crush set-device-class". That would be my second favorite: the only drawback is having a multi-step device creation during activation but all steps are idempotents that should not be a problem.

What do you think ?

Cheers

[1] https://github.com/ceph/ceph/blob/master/src/ceph-disk/ceph_disk/main.py#L1055
[2] https://github.com/ceph/ceph/blob/master/src/mon/OSDMonitor.cc#L7604 
[3] https://github.com/ceph/ceph/blob/master/src/osd/OSD.cc#L3024
-- 
Loïc Dachary, Artisan Logiciel Libre

Re: OSD creation and device class

[Sage Weil]

On Mon, 10 Apr 2017, Loic Dachary wrote:
> Hi Sage,
> 
> When the OSD is prepared, we add a --device-class option to ceph-disk 
> that creates a device_class file and activate can use it use when 
> creating the the OSD[1]. The ceph osd create command is given an 
> additional optional argument (device class). If the device class 
> argument is present, ceph osd create[2] creates a device entry in the 
> crushmap (it currently does nothing with the crushmap) so that the 
> corresponding device class can be preserved. This happens before 
> update_crush_location[3] is called and since the device is not included 
> in any crush tree it won't be used.
> 
> I'm not happy about adding an argument to osd create. We could instead 
> have a new dedicated command create-with-device-class but I like that 
> even less. We could also add a "ceph osd crush create-device" and have 
> ceph-disk call it right after "ceph osd create", followed by "ceph osd 
> crush set-device-class". That would be my second favorite: the only 
> drawback is having a multi-step device creation during activation but 
> all steps are idempotents that should not be a problem.
> 
> What do you think ?

We're aiming toward a new osd create command anyway that does all of this 
stuff at once (create osd id and set uuid, set cephx key, set lockbox key 
for dmcrypt, add to crush map).  Having it also set the device class seems 
like an easy addition.  I believe Joao has started or is about to start 
work on this?

In the meantime, I would do the simplest thing, which is probably just 
calling 'ceph osd crush set-device-class'.  It could be done before the 
'osd crush create-or-move' in OSD::update_crush_location(), but then we'd 
probably want to make it only set it if a device class isn't already set 
(i.e., don't change it if the admin set it to something else).  Or at some 
convenient point during the initial activation that similarly avoids 
clobbering the value if the admin set it...

sage

Re: OSD creation and device class

[Loic Dachary]

On 04/10/2017 03:32 PM, Sage Weil wrote:
> On Mon, 10 Apr 2017, Loic Dachary wrote:
>> Hi Sage,
>>
>> When the OSD is prepared, we add a --device-class option to ceph-disk 
>> that creates a device_class file and activate can use it use when 
>> creating the the OSD[1]. The ceph osd create command is given an 
>> additional optional argument (device class). If the device class 
>> argument is present, ceph osd create[2] creates a device entry in the 
>> crushmap (it currently does nothing with the crushmap) so that the 
>> corresponding device class can be preserved. This happens before 
>> update_crush_location[3] is called and since the device is not included 
>> in any crush tree it won't be used.
>>
>> I'm not happy about adding an argument to osd create. We could instead 
>> have a new dedicated command create-with-device-class but I like that 
>> even less. We could also add a "ceph osd crush create-device" and have 
>> ceph-disk call it right after "ceph osd create", followed by "ceph osd 
>> crush set-device-class". That would be my second favorite: the only 
>> drawback is having a multi-step device creation during activation but 
>> all steps are idempotents that should not be a problem.
>>
>> What do you think ?
> 
> We're aiming toward a new osd create command anyway that does all of this 
> stuff at once (create osd id and set uuid, set cephx key, set lockbox key 
> for dmcrypt, add to crush map).  Having it also set the device class seems 
> like an easy addition.  I believe Joao has started or is about to start 
> work on this?
> 
> In the meantime, I would do the simplest thing, which is probably just 
> calling 'ceph osd crush set-device-class'.  It could be done before the 
> 'osd crush create-or-move' in OSD::update_crush_location(), 

Does that mean we should have a --device-class argument to ceph-osd --mkfs ? Or is there another way I'm not seeing ?

> but then we'd 
> probably want to make it only set it if a device class isn't already set 
> (i.e., don't change it if the admin set it to something else).  Or at some 
> convenient point during the initial activation that similarly avoids 
> clobbering the value if the admin set it...
> 
> sage
> 

-- 
Loïc Dachary, Artisan Logiciel Libre

Re: OSD creation and device class

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2716 bytes --]

On Mon, 10 Apr 2017, Loic Dachary wrote:
> On 04/10/2017 03:32 PM, Sage Weil wrote:
> > On Mon, 10 Apr 2017, Loic Dachary wrote:
> >> Hi Sage,
> >>
> >> When the OSD is prepared, we add a --device-class option to ceph-disk 
> >> that creates a device_class file and activate can use it use when 
> >> creating the the OSD[1]. The ceph osd create command is given an 
> >> additional optional argument (device class). If the device class 
> >> argument is present, ceph osd create[2] creates a device entry in the 
> >> crushmap (it currently does nothing with the crushmap) so that the 
> >> corresponding device class can be preserved. This happens before 
> >> update_crush_location[3] is called and since the device is not included 
> >> in any crush tree it won't be used.
> >>
> >> I'm not happy about adding an argument to osd create. We could instead 
> >> have a new dedicated command create-with-device-class but I like that 
> >> even less. We could also add a "ceph osd crush create-device" and have 
> >> ceph-disk call it right after "ceph osd create", followed by "ceph osd 
> >> crush set-device-class". That would be my second favorite: the only 
> >> drawback is having a multi-step device creation during activation but 
> >> all steps are idempotents that should not be a problem.
> >>
> >> What do you think ?
> > 
> > We're aiming toward a new osd create command anyway that does all of this 
> > stuff at once (create osd id and set uuid, set cephx key, set lockbox key 
> > for dmcrypt, add to crush map).  Having it also set the device class seems 
> > like an easy addition.  I believe Joao has started or is about to start 
> > work on this?
> > 
> > In the meantime, I would do the simplest thing, which is probably just 
> > calling 'ceph osd crush set-device-class'.  It could be done before the 
> > 'osd crush create-or-move' in OSD::update_crush_location(), 
> 
> Does that mean we should have a --device-class argument to ceph-osd --mkfs ? Or is there another way I'm not seeing ?

If you're putting a file in the osd_data directory, you can use 
ObjectStore::read_meta() (and/or write_meta()) to easily access it.

Thinking about this a bit more, if the admin has explicitly specified the 
type like this, I don't think we need to worry about set-but-do-not-change 
behavior like I suggested below...

sage

> 
> > but then we'd 
> > probably want to make it only set it if a device class isn't already set 
> > (i.e., don't change it if the admin set it to something else).  Or at some 
> > convenient point during the initial activation that similarly avoids 
> > clobbering the value if the admin set it...
> > 
> > sage
> > 
> 
> -- 
> Loïc Dachary, Artisan Logiciel Libre
> 
>