ceph osd create

ceph osd create

[Mandell Degerness]

As of this time, we are allocating OSD numbers based on servers.  The
"ceph osd create" command seems like it may be useful, but is not
documented nearly well enough yet. Can someone please provide answers
to the following:

1. Is this guaranteed to always produce an OSD number that is unique,
with no worries about race conditions?
2. Does this eliminate the need to worry about the value of max_osd?
3. Are there any other side effects of running the command?

Regards,
Mandell Degerness

Re: ceph osd create

[Tommi Virtanen]

On Wed, Aug 1, 2012 at 4:27 PM, Mandell Degerness
<mandell@pistoncloud.com> wrote:
> As of this time, we are allocating OSD numbers based on servers.  The
> "ceph osd create" command seems like it may be useful, but is not
> documented nearly well enough yet. Can someone please provide answers
> to the following:
>
> 1. Is this guaranteed to always produce an OSD number that is unique,
> with no worries about race conditions?

Yes.

You can also use "ceph osd create UUID", so it's safe to run multiple
times, get the same result for the same UUID, and not waste numbers.

> 2. Does this eliminate the need to worry about the value of max_osd?

It increases max_osd as necessary.
https://github.com/ceph/ceph/blob/master/src/mon/OSDMonitor.cc#L2167

> 3. Are there any other side effects of running the command?

Nope.


You may be interested in digging into the mechanics of the new chef
deployment stuff, and "osd hotplugging" as I call it. It completely
automates osd id allocation.

http://ceph.com/docs/master/install/chef/
http://ceph.com/docs/master/config-cluster/chef/

More architectural docs don't really exist yet, sorry.

Re: ceph osd create

[Mandell Degerness]

Follow up question:

Is there now, or will there be a migration path that works to add
existing OSDs to the function such that future calls to "ceph osd
create <uuid>" return the current OSD number used in an existing
configuration?  I am content if it is an undocumented hack that I can
run on each OSD to register the information with the monitors (I am
assuming that is where the correlation is stored).

Regards,
Mandell Degerness

On Wed, Aug 1, 2012 at 4:43 PM, Tommi Virtanen <tv@inktank.com> wrote:
> On Wed, Aug 1, 2012 at 4:27 PM, Mandell Degerness
> <mandell@pistoncloud.com> wrote:
>> As of this time, we are allocating OSD numbers based on servers.  The
>> "ceph osd create" command seems like it may be useful, but is not
>> documented nearly well enough yet. Can someone please provide answers
>> to the following:
>>
>> 1. Is this guaranteed to always produce an OSD number that is unique,
>> with no worries about race conditions?
>
> Yes.
>
> You can also use "ceph osd create UUID", so it's safe to run multiple
> times, get the same result for the same UUID, and not waste numbers.
>
>> 2. Does this eliminate the need to worry about the value of max_osd?
>
> It increases max_osd as necessary.
> https://github.com/ceph/ceph/blob/master/src/mon/OSDMonitor.cc#L2167
>
>> 3. Are there any other side effects of running the command?
>
> Nope.
>
>
> You may be interested in digging into the mechanics of the new chef
> deployment stuff, and "osd hotplugging" as I call it. It completely
> automates osd id allocation.
>
> http://ceph.com/docs/master/install/chef/
> http://ceph.com/docs/master/config-cluster/chef/
>
> More architectural docs don't really exist yet, sorry.

Re: ceph osd create

[Tommi Virtanen]

On Mon, Aug 20, 2012 at 1:30 PM, Mandell Degerness
<mandell@pistoncloud.com> wrote:
> Is there now, or will there be a migration path that works to add
> existing OSDs to the function such that future calls to "ceph osd
> create <uuid>" return the current OSD number used in an existing
> configuration?  I am content if it is an undocumented hack that I can
> run on each OSD to register the information with the monitors (I am
> assuming that is where the correlation is stored).

Hmm. My understanding is that OSDs have had the uuid for a while, and
are submitting it to be part of the osdmap, so that *should* just
start working right if you're running recent code.

It's stored in the "whoami" file in the osd_data dir. Pick an OSD, cat
its whoami file, run "ceph osd create <uuid>", and if you get back an
id that doesn't match with what that osd was, say oops and "ceph osd
rm <new_bad_id>".

Re: ceph osd create

[Mandell Degerness]

We're running Argonaut and it only has the OSD id in the whoami file
and nothing else.

-Mandell

On Mon, Aug 20, 2012 at 1:37 PM, Tommi Virtanen <tv@inktank.com> wrote:
> On Mon, Aug 20, 2012 at 1:30 PM, Mandell Degerness
> <mandell@pistoncloud.com> wrote:
>> Is there now, or will there be a migration path that works to add
>> existing OSDs to the function such that future calls to "ceph osd
>> create <uuid>" return the current OSD number used in an existing
>> configuration?  I am content if it is an undocumented hack that I can
>> run on each OSD to register the information with the monitors (I am
>> assuming that is where the correlation is stored).
>
> Hmm. My understanding is that OSDs have had the uuid for a while, and
> are submitting it to be part of the osdmap, so that *should* just
> start working right if you're running recent code.
>
> It's stored in the "whoami" file in the osd_data dir. Pick an OSD, cat
> its whoami file, run "ceph osd create <uuid>", and if you get back an
> id that doesn't match with what that osd was, say oops and "ceph osd
> rm <new_bad_id>".

Re: ceph osd create

[Tommi Virtanen]

On Mon, Aug 20, 2012 at 3:53 PM, Mandell Degerness
<mandell@pistoncloud.com> wrote:
> We're running Argonaut and it only has the OSD id in the whoami file
> and nothing else.

My bad, I meant the file "fsid" (note, not "ceph_fsid").

Re: ceph osd create

[Mandell Degerness]

OK.  I think I'm getting there.

I want to be able to generate the fsid to be used in the OSD (from the
file system fsid, if that matters).  Is there a way to inject the fsid
when initializing the OSD directory?  It doesn't seem to be
documented.  The alternative would require that we mount the OSD in a
temp dir to read the fsid file, determine the OSD number, and then
re-mount it where it belongs, which seems the wrong way to go.

Regards,
Mandell Degerness

On Mon, Aug 20, 2012 at 4:26 PM, Tommi Virtanen <tv@inktank.com> wrote:
> On Mon, Aug 20, 2012 at 3:53 PM, Mandell Degerness
> <mandell@pistoncloud.com> wrote:
>> We're running Argonaut and it only has the OSD id in the whoami file
>> and nothing else.
>
> My bad, I meant the file "fsid" (note, not "ceph_fsid").

Re: ceph osd create

[Tommi Virtanen]

On Tue, Aug 21, 2012 at 11:01 AM, Mandell Degerness
<mandell@pistoncloud.com> wrote:
> I want to be able to generate the fsid to be used in the OSD (from the
> file system fsid, if that matters).  Is there a way to inject the fsid
> when initializing the OSD directory?  It doesn't seem to be
> documented.  The alternative would require that we mount the OSD in a
> temp dir to read the fsid file, determine the OSD number, and then
> re-mount it where it belongs, which seems the wrong way to go.

It's just a uuid.

Oh and you don't have to read the files if you don't want to get your
hands dirty; these days ceph-osd has --get-cluster-fsid,
--get-osd-fsid, --get-journal-fsid options:
http://ceph.com/docs/master/man/8/ceph-osd/

You can set the osd uuid at ceph-osd --mkfs time, just pass in --osd-uuid.

Sounds like you want to explore the world of OSD hotplugging. It comes
down to these two guys:

1. ceph-disk-prepare will take a block device, create a partition
table from scratch, and put the right stuff in place for the disk to
later become an active osd.

2. ceph-disk-activate, triggered by udev->upstart, or manually (later
udev->other init systems), will ceph-osd --mkfs the disk and start an
osd for it (currently upstart only).

https://github.com/ceph/ceph/blob/master/src/ceph-disk-activate
https://github.com/ceph/ceph/blob/master/src/ceph-disk-prepare

I'll happily explain this more, but I'd recommend you take a couple of
Ubuntu 12.04 virtual machines and run through the Chef installation
scenario in
http://ceph.com/docs/master/install/chef/
http://ceph.com/docs/master/config-cluster/
-- that'll probably help you understand the components better. None of
the underlying functionality is tied to Chef, that's just the first
thing getting ready.

Re: ceph osd create

[Sage Weil]

On Tue, 21 Aug 2012, Mandell Degerness wrote:
> OK.  I think I'm getting there.
> 
> I want to be able to generate the fsid to be used in the OSD (from the
> file system fsid, if that matters).  Is there a way to inject the fsid
> when initializing the OSD directory?  It doesn't seem to be
> documented.  The alternative would require that we mount the OSD in a
> temp dir to read the fsid file, determine the OSD number, and then
> re-mount it where it belongs, which seems the wrong way to go.

You can feed in the fsid to ceph-osd --mkfs with --osd-fsid <uuid>.

sage

> 
> Regards,
> Mandell Degerness
> 
> On Mon, Aug 20, 2012 at 4:26 PM, Tommi Virtanen <tv@inktank.com> wrote:
> > On Mon, Aug 20, 2012 at 3:53 PM, Mandell Degerness
> > <mandell@pistoncloud.com> wrote:
> >> We're running Argonaut and it only has the OSD id in the whoami file
> >> and nothing else.
> >
> > My bad, I meant the file "fsid" (note, not "ceph_fsid").
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: ceph osd create

[Mandell Degerness]

Thanks, Sage.  This is what I was looking for, but what version of
ceph do I need for this to work (it isn't there in Argonaut)?  See
below:

# ceph-osd -c /etc/ceph/ceph.conf --fsid
8296cc23-9c11-44d7-84c1-16866ef9c4f7 -i 50 --mkfs --osd-fsid
e1097bd8-c931-4e2e-8ccb-332a954adace
  --conf/-c        Read configuration from the given configuration file
  -d               Run in foreground, log to stderr.
  -f               Run in foreground, log to usual location.
  --id/-i          set ID portion of my name
  --name/-n        set name (TYPE.ID)
  --version        show version and quit

  --debug_ms N
        set message debug level (e.g. 1)
2012-08-21 23:26:50.774858 7f1be9ac1780 -1 unrecognized arg --osd-fsid
2012-08-21 23:26:50.774864 7f1be9ac1780 -1 usage: ceph-osd -i osdid
[--osd-data=path] [--osd-journal=path] [--mkfs] [--mkjournal]
[--convert-filestore]
2012-08-21 23:26:50.774915 7f1be9ac1780 -1    --debug_osd N   set
debug level (e.g. 10)

# ceph --version
ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)

Tommi - Thank you for the suggestion of ceph-disk-prepare and
ceph-disk-activate, but they work at too high of a level for our
usage.  We need finer control of the block devices.

Regards,
Mandell Degerness

On Tue, Aug 21, 2012 at 11:15 AM, Sage Weil <sage@inktank.com> wrote:
> On Tue, 21 Aug 2012, Mandell Degerness wrote:
>> OK.  I think I'm getting there.
>>
>> I want to be able to generate the fsid to be used in the OSD (from the
>> file system fsid, if that matters).  Is there a way to inject the fsid
>> when initializing the OSD directory?  It doesn't seem to be
>> documented.  The alternative would require that we mount the OSD in a
>> temp dir to read the fsid file, determine the OSD number, and then
>> re-mount it where it belongs, which seems the wrong way to go.
>
> You can feed in the fsid to ceph-osd --mkfs with --osd-fsid <uuid>.
>
> sage
>
>>
>> Regards,
>> Mandell Degerness
>>
>> On Mon, Aug 20, 2012 at 4:26 PM, Tommi Virtanen <tv@inktank.com> wrote:
>> > On Mon, Aug 20, 2012 at 3:53 PM, Mandell Degerness
>> > <mandell@pistoncloud.com> wrote:
>> >> We're running Argonaut and it only has the OSD id in the whoami file
>> >> and nothing else.
>> >
>> > My bad, I meant the file "fsid" (note, not "ceph_fsid").
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>

Re: ceph osd create

[Mandell Degerness]

Found it (digging through the source code to find a guess, since it is
in no way obvious):  --osd-uuid <uuid>

On Tue, Aug 21, 2012 at 4:38 PM, Mandell Degerness
<mandell@pistoncloud.com> wrote:
> Thanks, Sage.  This is what I was looking for, but what version of
> ceph do I need for this to work (it isn't there in Argonaut)?  See
> below:
>
> # ceph-osd -c /etc/ceph/ceph.conf --fsid
> 8296cc23-9c11-44d7-84c1-16866ef9c4f7 -i 50 --mkfs --osd-fsid
> e1097bd8-c931-4e2e-8ccb-332a954adace
>   --conf/-c        Read configuration from the given configuration file
>   -d               Run in foreground, log to stderr.
>   -f               Run in foreground, log to usual location.
>   --id/-i          set ID portion of my name
>   --name/-n        set name (TYPE.ID)
>   --version        show version and quit
>
>   --debug_ms N
>         set message debug level (e.g. 1)
> 2012-08-21 23:26:50.774858 7f1be9ac1780 -1 unrecognized arg --osd-fsid
> 2012-08-21 23:26:50.774864 7f1be9ac1780 -1 usage: ceph-osd -i osdid
> [--osd-data=path] [--osd-journal=path] [--mkfs] [--mkjournal]
> [--convert-filestore]
> 2012-08-21 23:26:50.774915 7f1be9ac1780 -1    --debug_osd N   set
> debug level (e.g. 10)
>
> # ceph --version
> ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
>
> Tommi - Thank you for the suggestion of ceph-disk-prepare and
> ceph-disk-activate, but they work at too high of a level for our
> usage.  We need finer control of the block devices.
>
> Regards,
> Mandell Degerness
>
> On Tue, Aug 21, 2012 at 11:15 AM, Sage Weil <sage@inktank.com> wrote:
>> On Tue, 21 Aug 2012, Mandell Degerness wrote:
>>> OK.  I think I'm getting there.
>>>
>>> I want to be able to generate the fsid to be used in the OSD (from the
>>> file system fsid, if that matters).  Is there a way to inject the fsid
>>> when initializing the OSD directory?  It doesn't seem to be
>>> documented.  The alternative would require that we mount the OSD in a
>>> temp dir to read the fsid file, determine the OSD number, and then
>>> re-mount it where it belongs, which seems the wrong way to go.
>>
>> You can feed in the fsid to ceph-osd --mkfs with --osd-fsid <uuid>.
>>
>> sage
>>
>>>
>>> Regards,
>>> Mandell Degerness
>>>
>>> On Mon, Aug 20, 2012 at 4:26 PM, Tommi Virtanen <tv@inktank.com> wrote:
>>> > On Mon, Aug 20, 2012 at 3:53 PM, Mandell Degerness
>>> > <mandell@pistoncloud.com> wrote:
>>> >> We're running Argonaut and it only has the OSD id in the whoami file
>>> >> and nothing else.
>>> >
>>> > My bad, I meant the file "fsid" (note, not "ceph_fsid").
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>

Re: ceph osd create

[Sage Weil]

On Tue, 21 Aug 2012, Mandell Degerness wrote:
> Found it (digging through the source code to find a guess, since it is
> in no way obvious):  --osd-uuid <uuid>

Whoops, sorry, yeah.  It appeared in 0.47.

sage

> 
> On Tue, Aug 21, 2012 at 4:38 PM, Mandell Degerness
> <mandell@pistoncloud.com> wrote:
> > Thanks, Sage.  This is what I was looking for, but what version of
> > ceph do I need for this to work (it isn't there in Argonaut)?  See
> > below:
> >
> > # ceph-osd -c /etc/ceph/ceph.conf --fsid
> > 8296cc23-9c11-44d7-84c1-16866ef9c4f7 -i 50 --mkfs --osd-fsid
> > e1097bd8-c931-4e2e-8ccb-332a954adace
> >   --conf/-c        Read configuration from the given configuration file
> >   -d               Run in foreground, log to stderr.
> >   -f               Run in foreground, log to usual location.
> >   --id/-i          set ID portion of my name
> >   --name/-n        set name (TYPE.ID)
> >   --version        show version and quit
> >
> >   --debug_ms N
> >         set message debug level (e.g. 1)
> > 2012-08-21 23:26:50.774858 7f1be9ac1780 -1 unrecognized arg --osd-fsid
> > 2012-08-21 23:26:50.774864 7f1be9ac1780 -1 usage: ceph-osd -i osdid
> > [--osd-data=path] [--osd-journal=path] [--mkfs] [--mkjournal]
> > [--convert-filestore]
> > 2012-08-21 23:26:50.774915 7f1be9ac1780 -1    --debug_osd N   set
> > debug level (e.g. 10)
> >
> > # ceph --version
> > ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
> >
> > Tommi - Thank you for the suggestion of ceph-disk-prepare and
> > ceph-disk-activate, but they work at too high of a level for our
> > usage.  We need finer control of the block devices.
> >
> > Regards,
> > Mandell Degerness
> >
> > On Tue, Aug 21, 2012 at 11:15 AM, Sage Weil <sage@inktank.com> wrote:
> >> On Tue, 21 Aug 2012, Mandell Degerness wrote:
> >>> OK.  I think I'm getting there.
> >>>
> >>> I want to be able to generate the fsid to be used in the OSD (from the
> >>> file system fsid, if that matters).  Is there a way to inject the fsid
> >>> when initializing the OSD directory?  It doesn't seem to be
> >>> documented.  The alternative would require that we mount the OSD in a
> >>> temp dir to read the fsid file, determine the OSD number, and then
> >>> re-mount it where it belongs, which seems the wrong way to go.
> >>
> >> You can feed in the fsid to ceph-osd --mkfs with --osd-fsid <uuid>.
> >>
> >> sage
> >>
> >>>
> >>> Regards,
> >>> Mandell Degerness
> >>>
> >>> On Mon, Aug 20, 2012 at 4:26 PM, Tommi Virtanen <tv@inktank.com> wrote:
> >>> > On Mon, Aug 20, 2012 at 3:53 PM, Mandell Degerness
> >>> > <mandell@pistoncloud.com> wrote:
> >>> >> We're running Argonaut and it only has the OSD id in the whoami file
> >>> >> and nothing else.
> >>> >
> >>> > My bad, I meant the file "fsid" (note, not "ceph_fsid").
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >>> the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>
> >>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
>