Ceph-deploy for FreeBSD

Ceph-deploy for FreeBSD

[Willem Jan Withagen]

Hi,

I'm looking for somebody to help me get ceph-deploy running on FreeBSD...

I can get it to install with pip install, and the local part runs.
But then rouble starts already rather fast:

----
 # ceph-deploy -v -v --username wjw install cephtest
.....
[ceph_deploy.install][DEBUG ] Installing stable version jewel on cluster
ceph hosts cephtest
[ceph_deploy.install][DEBUG ] Detecting platform for host cephtest ...
Password for wjw@cephtest:
Password for wjw@cephtest:
[cephtest][DEBUG ] connection detected need for sudo
Password for wjw@cephtest:
[cephtest][DEBUG ] connected to host: wjw@cephtest
[cephtest][DEBUG ] detect platform information from remote host
[ceph_deploy][ERROR ] UnsupportedPlatform: Platform is not supported:
----

Which is due to the fact that python on FreeBSD does not fill the
Linux_distribution info:
>>> print platform.linux_distribution()
('', '', '')

Anybody out there that can help me get my head wrapped around this
remote exec stuff and get  platform_information() to return some
informative info back, like in ceph-detect-init.

After that I'll have to add the FreeBSD pkg package manager, but that
code seems more clear to me. Which would result in largely a copy/modify
exercise.

Thanx,
--WjW

Re: Ceph-deploy for FreeBSD

[Willem Jan Withagen]

Having solved the first few step in ceph-deploy, I can now get to the
point where it actually wants to start the first monitor.
Like:
    /usr/local/bin/ceph-deploy mon create-initial

And if I look at the other platform booting the inital monitor is
started thru the init|systemd|upstart....

However under FreeBSD bsdrc I require atleast in ceph.conf
[mon]
    host = cephtest

This is not the case on the other platforms?

And how would I otherwise get this into the ceph.conf?

--WjW

Re: Ceph-deploy for FreeBSD

[Gregory Farnum]

On Fri, Apr 14, 2017 at 1:44 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
> Having solved the first few step in ceph-deploy, I can now get to the
> point where it actually wants to start the first monitor.
> Like:
>     /usr/local/bin/ceph-deploy mon create-initial
>
> And if I look at the other platform booting the inital monitor is
> started thru the init|systemd|upstart....
>
> However under FreeBSD bsdrc I require atleast in ceph.conf
> [mon]
>     host = cephtest
>
> This is not the case on the other platforms?
>
> And how would I otherwise get this into the ceph.conf?

I think you're running into some annoying deployment sharp edges here.
The monitor "host" config value has gone in and out of being necessary
at various times — the whole monitor boot sequence is pretty finicky
so I've added Joao to the thread. It's possible that the way things
are default-configured in FreeBSD that's necessary to work around some
other default configs?
-Greg

Re: Ceph-deploy for FreeBSD

[Willem Jan Withagen]

On 19-4-2017 22:47, Gregory Farnum wrote:
> On Fri, Apr 14, 2017 at 1:44 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>> Having solved the first few step in ceph-deploy, I can now get to the
>> point where it actually wants to start the first monitor.
>> Like:
>>     /usr/local/bin/ceph-deploy mon create-initial
>>
>> And if I look at the other platform booting the inital monitor is
>> started thru the init|systemd|upstart....
>>
>> However under FreeBSD bsdrc I require atleast in ceph.conf
>> [mon]
>>     host = cephtest
>>
>> This is not the case on the other platforms?
>>
>> And how would I otherwise get this into the ceph.conf?
> 
> I think you're running into some annoying deployment sharp edges here.
> The monitor "host" config value has gone in and out of being necessary
> at various times — the whole monitor boot sequence is pretty finicky
> so I've added Joao to the thread. It's possible that the way things
> are default-configured in FreeBSD that's necessary to work around some
> other default configs?

Hi Greg, Joao,

Could very well be like you describe...

The call sequence in of course rather convoluted:

	ceph-deply
	-> remote call ceph-disk prepare
	-> remote call ceph-disk activat

And expect after the activate to actually have a running OSD.
But to do so, ceph-disk calls the FreeBSD init script:
	service ceph start osd.x
that activates actual initscript:
	/usr/local/etc/rc.d/ceph
which calls:
	init-ceph

And init-ceph goes thru /etc/ceph/ceph.conf for OSD that it can start on
this host. Now if the OSD description/config is not there, it doesn't
get started.

I'm doing the service -> rc.d/ceph -> init-ceph way, because that seems
to me the natural way of doing things, and that will take into account
all the settings/bits/other stuff that is set on that host.
AND by keeping init-ceph as main executor, I can try to keep in pace
with the code in the ceph-repo.

So I guess the best fix is to add routines to add the OSD to the config
and then copy it to the remote host.

Why would it be not wanted on the other systems?
Otherwise I'd call those adds only from the FreeBSD init part.

--WjW

Re: Ceph-deploy for FreeBSD

[Nathan Cutler]

Hi Willem:

It sounds like you are trying to use the sysvinit scripts? These have 
been unmaintained (presumably with lots of weeds growing up) since 
infernalis. Until now I have been assuming that all init systems other 
than systemd (sysvinit, upstart, etc.) are deprecated in Ceph.

Cheers,
Nathan

On 04/20/2017 10:55 AM, Willem Jan Withagen wrote:
> On 19-4-2017 22:47, Gregory Farnum wrote:
>> On Fri, Apr 14, 2017 at 1:44 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>>> Having solved the first few step in ceph-deploy, I can now get to the
>>> point where it actually wants to start the first monitor.
>>> Like:
>>>     /usr/local/bin/ceph-deploy mon create-initial
>>>
>>> And if I look at the other platform booting the inital monitor is
>>> started thru the init|systemd|upstart....
>>>
>>> However under FreeBSD bsdrc I require atleast in ceph.conf
>>> [mon]
>>>     host = cephtest
>>>
>>> This is not the case on the other platforms?
>>>
>>> And how would I otherwise get this into the ceph.conf?
>>
>> I think you're running into some annoying deployment sharp edges here.
>> The monitor "host" config value has gone in and out of being necessary
>> at various times — the whole monitor boot sequence is pretty finicky
>> so I've added Joao to the thread. It's possible that the way things
>> are default-configured in FreeBSD that's necessary to work around some
>> other default configs?
>
> Hi Greg, Joao,
>
> Could very well be like you describe...
>
> The call sequence in of course rather convoluted:
>
> 	ceph-deply
> 	-> remote call ceph-disk prepare
> 	-> remote call ceph-disk activat
>
> And expect after the activate to actually have a running OSD.
> But to do so, ceph-disk calls the FreeBSD init script:
> 	service ceph start osd.x
> that activates actual initscript:
> 	/usr/local/etc/rc.d/ceph
> which calls:
> 	init-ceph
>
> And init-ceph goes thru /etc/ceph/ceph.conf for OSD that it can start on
> this host. Now if the OSD description/config is not there, it doesn't
> get started.
>
> I'm doing the service -> rc.d/ceph -> init-ceph way, because that seems
> to me the natural way of doing things, and that will take into account
> all the settings/bits/other stuff that is set on that host.
> AND by keeping init-ceph as main executor, I can try to keep in pace
> with the code in the ceph-repo.
>
> So I guess the best fix is to add routines to add the OSD to the config
> and then copy it to the remote host.
>
> Why would it be not wanted on the other systems?
> Otherwise I'd call those adds only from the FreeBSD init part.
>
> --WjW
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Ceph-deploy for FreeBSD

[Gregory Farnum]

On Thu, Apr 20, 2017 at 11:11 AM, Nathan Cutler <ncutler@suse.cz> wrote:
> Hi Willem:
>
> It sounds like you are trying to use the sysvinit scripts? These have been
> unmaintained (presumably with lots of weeds growing up) since infernalis.
> Until now I have been assuming that all init systems other than systemd
> (sysvinit, upstart, etc.) are deprecated in Ceph.
>

Yeah. In particular, the scripts generally set up some magic to turn
on daemons instead of enumerating them in the ceph.conf. I forget how
the OSDs work but it's some hotplug system, and I think maybe the
monitor scripts just look for any monitor directories and start them?
Don't really remember.
-Greg

> Cheers,
> Nathan
>
>
> On 04/20/2017 10:55 AM, Willem Jan Withagen wrote:
>>
>> On 19-4-2017 22:47, Gregory Farnum wrote:
>>>
>>> On Fri, Apr 14, 2017 at 1:44 PM, Willem Jan Withagen <wjw@digiware.nl>
>>> wrote:
>>>>
>>>> Having solved the first few step in ceph-deploy, I can now get to the
>>>> point where it actually wants to start the first monitor.
>>>> Like:
>>>>     /usr/local/bin/ceph-deploy mon create-initial
>>>>
>>>> And if I look at the other platform booting the inital monitor is
>>>> started thru the init|systemd|upstart....
>>>>
>>>> However under FreeBSD bsdrc I require atleast in ceph.conf
>>>> [mon]
>>>>     host = cephtest
>>>>
>>>> This is not the case on the other platforms?
>>>>
>>>> And how would I otherwise get this into the ceph.conf?
>>>
>>>
>>> I think you're running into some annoying deployment sharp edges here.
>>> The monitor "host" config value has gone in and out of being necessary
>>> at various times — the whole monitor boot sequence is pretty finicky
>>> so I've added Joao to the thread. It's possible that the way things
>>> are default-configured in FreeBSD that's necessary to work around some
>>> other default configs?
>>
>>
>> Hi Greg, Joao,
>>
>> Could very well be like you describe...
>>
>> The call sequence in of course rather convoluted:
>>
>>         ceph-deply
>>         -> remote call ceph-disk prepare
>>         -> remote call ceph-disk activat
>>
>> And expect after the activate to actually have a running OSD.
>> But to do so, ceph-disk calls the FreeBSD init script:
>>         service ceph start osd.x
>> that activates actual initscript:
>>         /usr/local/etc/rc.d/ceph
>> which calls:
>>         init-ceph
>>
>> And init-ceph goes thru /etc/ceph/ceph.conf for OSD that it can start on
>> this host. Now if the OSD description/config is not there, it doesn't
>> get started.
>>
>> I'm doing the service -> rc.d/ceph -> init-ceph way, because that seems
>> to me the natural way of doing things, and that will take into account
>> all the settings/bits/other stuff that is set on that host.
>> AND by keeping init-ceph as main executor, I can try to keep in pace
>> with the code in the ceph-repo.
>>
>> So I guess the best fix is to add routines to add the OSD to the config
>> and then copy it to the remote host.
>>
>> Why would it be not wanted on the other systems?
>> Otherwise I'd call those adds only from the FreeBSD init part.
>>
>> --WjW
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

Re: Ceph-deploy for FreeBSD

[Willem Jan Withagen]

On 20-04-17 20:18, Gregory Farnum wrote:
> On Thu, Apr 20, 2017 at 11:11 AM, Nathan Cutler <ncutler@suse.cz> wrote:
>> Hi Willem:
>>
>> It sounds like you are trying to use the sysvinit scripts? These have been
>> unmaintained (presumably with lots of weeds growing up) since infernalis.
>> Until now I have been assuming that all init systems other than systemd
>> (sysvinit, upstart, etc.) are deprecated in Ceph.

sysvinit => init-ceph?

If that is the case and is "abandoned", I could start making it more to
my liking to use it under FreeBSD. FreeBSD does not (yet) have this wild
choice of init systems.

Attempts to port Linux one to FreeBSD are in progress, but if they make
it to a large crowd is questionable. Unless it is incorporated in the
the FreeBSD release itself.

> Yeah. In particular, the scripts generally set up some magic to turn
> on daemons instead of enumerating them in the ceph.conf. I forget how
> the OSDs work but it's some hotplug system, and I think maybe the
> monitor scripts just look for any monitor directories and start them?
> Don't really remember.

So systemd is digging around in /dev and /sys proding for devices with
Ceph properties, and then starts the required ceph-(mon|osd|mds|mgr).
Or Udev-events telling systemd to act on a device that presents itself
in the system?

So on an average system there is no OSD info in /etc/ceph/ceph.conf

> -Greg
> 
>> Cheers,
>> Nathan
>>
>>
>> On 04/20/2017 10:55 AM, Willem Jan Withagen wrote:
>>>
>>> On 19-4-2017 22:47, Gregory Farnum wrote:
>>>>
>>>> On Fri, Apr 14, 2017 at 1:44 PM, Willem Jan Withagen <wjw@digiware.nl>
>>>> wrote:
>>>>>
>>>>> Having solved the first few step in ceph-deploy, I can now get to the
>>>>> point where it actually wants to start the first monitor.
>>>>> Like:
>>>>>     /usr/local/bin/ceph-deploy mon create-initial
>>>>>
>>>>> And if I look at the other platform booting the inital monitor is
>>>>> started thru the init|systemd|upstart....
>>>>>
>>>>> However under FreeBSD bsdrc I require atleast in ceph.conf
>>>>> [mon]
>>>>>     host = cephtest
>>>>>
>>>>> This is not the case on the other platforms?
>>>>>
>>>>> And how would I otherwise get this into the ceph.conf?
>>>>
>>>>
>>>> I think you're running into some annoying deployment sharp edges here.
>>>> The monitor "host" config value has gone in and out of being necessary
>>>> at various times — the whole monitor boot sequence is pretty finicky
>>>> so I've added Joao to the thread. It's possible that the way things
>>>> are default-configured in FreeBSD that's necessary to work around some
>>>> other default configs?
>>>
>>>
>>> Hi Greg, Joao,
>>>
>>> Could very well be like you describe...
>>>
>>> The call sequence in of course rather convoluted:
>>>
>>>         ceph-deply
>>>         -> remote call ceph-disk prepare
>>>         -> remote call ceph-disk activat
>>>
>>> And expect after the activate to actually have a running OSD.
>>> But to do so, ceph-disk calls the FreeBSD init script:
>>>         service ceph start osd.x
>>> that activates actual initscript:
>>>         /usr/local/etc/rc.d/ceph
>>> which calls:
>>>         init-ceph
>>>
>>> And init-ceph goes thru /etc/ceph/ceph.conf for OSD that it can start on
>>> this host. Now if the OSD description/config is not there, it doesn't
>>> get started.
>>>
>>> I'm doing the service -> rc.d/ceph -> init-ceph way, because that seems
>>> to me the natural way of doing things, and that will take into account
>>> all the settings/bits/other stuff that is set on that host.
>>> AND by keeping init-ceph as main executor, I can try to keep in pace
>>> with the code in the ceph-repo.
>>>
>>> So I guess the best fix is to add routines to add the OSD to the config
>>> and then copy it to the remote host.
>>>
>>> Why would it be not wanted on the other systems?
>>> Otherwise I'd call those adds only from the FreeBSD init part.
>>>
>>> --WjW
>>>
>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>

Re: Ceph-deploy for FreeBSD

[Willem Jan Withagen]

On 21-4-2017 09:33, Willem Jan Withagen wrote:
> On 20-04-17 20:18, Gregory Farnum wrote:
>> On Thu, Apr 20, 2017 at 11:11 AM, Nathan Cutler <ncutler@suse.cz> wrote:
>>> Hi Willem:
>>>
>>> It sounds like you are trying to use the sysvinit scripts? These have been
>>> unmaintained (presumably with lots of weeds growing up) since infernalis.
>>> Until now I have been assuming that all init systems other than systemd
>>> (sysvinit, upstart, etc.) are deprecated in Ceph.
> 
> sysvinit => init-ceph?
> 
> If that is the case and is "abandoned", I could start making it more to
> my liking to use it under FreeBSD. FreeBSD does not (yet) have this wild
> choice of init systems.

Right,

Did some digging into this.
1) init-ceph.in sort of looks abondoned. There are very few changes to
this file during 2016 and 2017, mostly by me for FreeBSD, and some
others have to do with the move to cmake.

2) I checked in our cluster, and yes the ceph.conf file only holds
global settings. There is only a global definition of the monitor hosts
and that is all there is about hosts.

So I'll have to figure out what to do in ceph-deploy, ceph-disk and or
init-ceph.
Least invasive in this would be to modify init-ceph to analyze
/var/lib/ceph for what should be started.

--WjW

> Attempts to port Linux one to FreeBSD are in progress, but if they make
> it to a large crowd is questionable. Unless it is incorporated in the
> the FreeBSD release itself.
> 
>> Yeah. In particular, the scripts generally set up some magic to turn
>> on daemons instead of enumerating them in the ceph.conf. I forget how
>> the OSDs work but it's some hotplug system, and I think maybe the
>> monitor scripts just look for any monitor directories and start them?
>> Don't really remember.
> 
> So systemd is digging around in /dev and /sys proding for devices with
> Ceph properties, and then starts the required ceph-(mon|osd|mds|mgr).
> Or Udev-events telling systemd to act on a device that presents itself
> in the system?
> 
> So on an average system there is no OSD info in /etc/ceph/ceph.conf
> 
>> -Greg
>>
>>> Cheers,
>>> Nathan
>>>
>>>
>>> On 04/20/2017 10:55 AM, Willem Jan Withagen wrote:
>>>>
>>>> On 19-4-2017 22:47, Gregory Farnum wrote:
>>>>>
>>>>> On Fri, Apr 14, 2017 at 1:44 PM, Willem Jan Withagen <wjw@digiware.nl>
>>>>> wrote:
>>>>>>
>>>>>> Having solved the first few step in ceph-deploy, I can now get to the
>>>>>> point where it actually wants to start the first monitor.
>>>>>> Like:
>>>>>>     /usr/local/bin/ceph-deploy mon create-initial
>>>>>>
>>>>>> And if I look at the other platform booting the inital monitor is
>>>>>> started thru the init|systemd|upstart....
>>>>>>
>>>>>> However under FreeBSD bsdrc I require atleast in ceph.conf
>>>>>> [mon]
>>>>>>     host = cephtest
>>>>>>
>>>>>> This is not the case on the other platforms?
>>>>>>
>>>>>> And how would I otherwise get this into the ceph.conf?
>>>>>
>>>>>
>>>>> I think you're running into some annoying deployment sharp edges here.
>>>>> The monitor "host" config value has gone in and out of being necessary
>>>>> at various times — the whole monitor boot sequence is pretty finicky
>>>>> so I've added Joao to the thread. It's possible that the way things
>>>>> are default-configured in FreeBSD that's necessary to work around some
>>>>> other default configs?
>>>>
>>>>
>>>> Hi Greg, Joao,
>>>>
>>>> Could very well be like you describe...
>>>>
>>>> The call sequence in of course rather convoluted:
>>>>
>>>>         ceph-deply
>>>>         -> remote call ceph-disk prepare
>>>>         -> remote call ceph-disk activat
>>>>
>>>> And expect after the activate to actually have a running OSD.
>>>> But to do so, ceph-disk calls the FreeBSD init script:
>>>>         service ceph start osd.x
>>>> that activates actual initscript:
>>>>         /usr/local/etc/rc.d/ceph
>>>> which calls:
>>>>         init-ceph
>>>>
>>>> And init-ceph goes thru /etc/ceph/ceph.conf for OSD that it can start on
>>>> this host. Now if the OSD description/config is not there, it doesn't
>>>> get started.
>>>>
>>>> I'm doing the service -> rc.d/ceph -> init-ceph way, because that seems
>>>> to me the natural way of doing things, and that will take into account
>>>> all the settings/bits/other stuff that is set on that host.
>>>> AND by keeping init-ceph as main executor, I can try to keep in pace
>>>> with the code in the ceph-repo.
>>>>
>>>> So I guess the best fix is to add routines to add the OSD to the config
>>>> and then copy it to the remote host.
>>>>
>>>> Why would it be not wanted on the other systems?
>>>> Otherwise I'd call those adds only from the FreeBSD init part.

Re: Ceph-deploy for FreeBSD

[Fabian Grünbichler]

On Thu, Apr 20, 2017 at 08:11:38PM +0200, Nathan Cutler wrote:
> Hi Willem:
> 
> It sounds like you are trying to use the sysvinit scripts? These have been
> unmaintained (presumably with lots of weeds growing up) since infernalis.
> Until now I have been assuming that all init systems other than systemd
> (sysvinit, upstart, etc.) are deprecated in Ceph.

Slightly OT, but AFAICT http://tracker.ceph.com/issues/18305 still
applies to the official ceph.com Kraken Debian packages, i.e., ceph-base
installs and activates the init.d script, which then races against the
(udev-activated) ceph-osd systemd units. If anything except systemd is
indeed deprecated, I wonder why the Debian packages (still) ship AND
activate both systemd units and Sys V init scripts?

(Note that the proposed fix probably does not apply as is anymore,
because ceph-disk and the systemd units have been changed in the
meantime).

Re: Ceph-deploy for FreeBSD

[Nathan Cutler]

Bringing Loic into CC in hopes he'll correct my mistakes and 
misconceptions below.

> So systemd is digging around in /dev and /sys proding for devices with
> Ceph properties, and then starts the required ceph-(mon|osd|mds|mgr).
> Or Udev-events telling systemd to act on a device that presents itself
> in the system?

I don't know about ceph-(mon|mds|mgr), but for osd it works 
approximately like the following:

See udev/95-ceph-osd.rules

systemd triggers udev, and the udev rules look for partitions with 
certain "partition GUID codes" ("ceph data" and "ceph journal" 
respectively) and run "ceph-disk trigger" on them.

Cluster deployment tooling is expected to run "ceph-disk prepare" to set 
up the partition GUID codes. AFAIK "ceph-disk activate" is run for each 
OSD at startup as a consequence of "ceph-disk trigger".

HTH,
Nathan

Re: Ceph-deploy for FreeBSD

[Willem Jan Withagen]

On 21-04-17 11:13, Fabian Grünbichler wrote:
> On Thu, Apr 20, 2017 at 08:11:38PM +0200, Nathan Cutler wrote:
>> Hi Willem:
>>
>> It sounds like you are trying to use the sysvinit scripts? These have been
>> unmaintained (presumably with lots of weeds growing up) since infernalis.
>> Until now I have been assuming that all init systems other than systemd
>> (sysvinit, upstart, etc.) are deprecated in Ceph.
> 
> Slightly OT, but AFAICT http://tracker.ceph.com/issues/18305 still
> applies to the official ceph.com Kraken Debian packages, i.e., ceph-base
> installs and activates the init.d script, which then races against the
> (udev-activated) ceph-osd systemd units. If anything except systemd is
> indeed deprecated, I wonder why the Debian packages (still) ship AND
> activate both systemd units and Sys V init scripts?
> 
> (Note that the proposed fix probably does not apply as is anymore,
> because ceph-disk and the systemd units have been changed in the
> meantime).
> 
Thanx for the headsup.

It means that I cannot get my axe out and start whacking at init-ceph
just at will.

--WjW

Re: Ceph-deploy for FreeBSD

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1295 bytes --]

On Fri, 21 Apr 2017, Fabian Grünbichler wrote:
> On Thu, Apr 20, 2017 at 08:11:38PM +0200, Nathan Cutler wrote:
> > Hi Willem:
> > 
> > It sounds like you are trying to use the sysvinit scripts? These have been
> > unmaintained (presumably with lots of weeds growing up) since infernalis.
> > Until now I have been assuming that all init systems other than systemd
> > (sysvinit, upstart, etc.) are deprecated in Ceph.
> 
> Slightly OT, but AFAICT http://tracker.ceph.com/issues/18305 still
> applies to the official ceph.com Kraken Debian packages, i.e., ceph-base
> installs and activates the init.d script, which then races against the
> (udev-activated) ceph-osd systemd units. If anything except systemd is
> indeed deprecated, I wonder why the Debian packages (still) ship AND
> activate both systemd units and Sys V init scripts?
> 
> (Note that the proposed fix probably does not apply as is anymore,
> because ceph-disk and the systemd units have been changed in the
> meantime).

I think the issue with Debian (generally) is that it "supports" multiple 
init systems (sysvinit and systemd both), even though systemd is the one 
installed default.  Which means we ship the sysvinit script and systemd 
unit files.

(There may very well be a bug in how we "activate" them, though!)

sage

Re: Ceph-deploy for FreeBSD

[Sage Weil]

On Fri, 21 Apr 2017, Willem Jan Withagen wrote:
> On 21-4-2017 09:33, Willem Jan Withagen wrote:
> > On 20-04-17 20:18, Gregory Farnum wrote:
> >> On Thu, Apr 20, 2017 at 11:11 AM, Nathan Cutler <ncutler@suse.cz> wrote:
> >>> Hi Willem:
> >>>
> >>> It sounds like you are trying to use the sysvinit scripts? These have been
> >>> unmaintained (presumably with lots of weeds growing up) since infernalis.
> >>> Until now I have been assuming that all init systems other than systemd
> >>> (sysvinit, upstart, etc.) are deprecated in Ceph.
> > 
> > sysvinit => init-ceph?
> > 
> > If that is the case and is "abandoned", I could start making it more to
> > my liking to use it under FreeBSD. FreeBSD does not (yet) have this wild
> > choice of init systems.
> 
> Right,
> 
> Did some digging into this.
> 1) init-ceph.in sort of looks abondoned. There are very few changes to
> this file during 2016 and 2017, mostly by me for FreeBSD, and some
> others have to do with the move to cmake.
> 
> 2) I checked in our cluster, and yes the ceph.conf file only holds
> global settings. There is only a global definition of the monitor hosts
> and that is all there is about hosts.
> 
> So I'll have to figure out what to do in ceph-deploy, ceph-disk and or
> init-ceph.
> Least invasive in this would be to modify init-ceph to analyze
> /var/lib/ceph for what should be started.

I think the primary user of init-ceph at this point is vstart.sh; 
developers use it to start and stop daemons from their source directory.

init-ceph suffers from the fact that it maintains compatibility back 
through its original version, circa 2007, when we defined all daemons in 
ceph.conf, and every variation of weirdness since then.  Since I 
don't think many/any distros are using it by default, now, I think we can 
probably afford to drop some of that old cruft and modernize it a bit.  
Unfortunatly vstart is making use of the daemons-defined-in-conf thing, 
though, but it wouldn't bother me to modernize vstart, either.

sage

Re: Ceph-deploy for FreeBSD

[Fabian Grünbichler]

On Fri, Apr 21, 2017 at 01:16:23PM +0000, Sage Weil wrote:
> On Fri, 21 Apr 2017, Fabian Grünbichler wrote:
> > On Thu, Apr 20, 2017 at 08:11:38PM +0200, Nathan Cutler wrote:
> > > Hi Willem:
> > > 
> > > It sounds like you are trying to use the sysvinit scripts? These have been
> > > unmaintained (presumably with lots of weeds growing up) since infernalis.
> > > Until now I have been assuming that all init systems other than systemd
> > > (sysvinit, upstart, etc.) are deprecated in Ceph.
> > 
> > Slightly OT, but AFAICT http://tracker.ceph.com/issues/18305 still
> > applies to the official ceph.com Kraken Debian packages, i.e., ceph-base
> > installs and activates the init.d script, which then races against the
> > (udev-activated) ceph-osd systemd units. If anything except systemd is
> > indeed deprecated, I wonder why the Debian packages (still) ship AND
> > activate both systemd units and Sys V init scripts?
> > 
> > (Note that the proposed fix probably does not apply as is anymore,
> > because ceph-disk and the systemd units have been changed in the
> > meantime).
> 
> I think the issue with Debian (generally) is that it "supports" multiple 
> init systems (sysvinit and systemd both), even though systemd is the one 
> installed default.  Which means we ship the sysvinit script and systemd 
> unit files.
> 
> (There may very well be a bug in how we "activate" them, though!)
> 

that's why I reported the original issue - in general you never want to
have both a multi-daemon init script (like the "old" ceph one) and the
replacing split up systemd units active at the same time.

since systemd will generate a unit for every init script for which a
unit of the same name does not already exist, you either need to mask
the auto-generated unit (i.e., symlink it to /dev/null) or write a
replacement unit that has the identical name (so the "ceph" init script
becomes the "ceph.service" unit). if you don't do that and your units
are named differently than your init script, both will be active (this
is not a Debianism, it is how the LSB generator in systemd is supposed
to work to ease the transition from Sys V init to systemd..).

what exacerbates the issue in this case is that the systemd units + udev
actually don't completely replace the old init scripts, because some of
the udev events might have been processed before the system was fully
booted, and osds might not be properly activated on boot as a result.

hence my proposal to add a ceph.service that simply calls "ceph-disk
activate-all", which is AFAICT the only part of the init script that is
not covered by the current systemd units / udev rules.

the whole ceph service startup is pretty messy in general IMHO,
especially for OSDs where (IIRC?) udev rules are calling python scripts
which are starting systemd units which are in turn calling python
scripts with different parameters that end up starting systemd units
which actually start daemons (somewhere in there the mounting happens as
well..).

Re: Ceph-deploy for FreeBSD

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3755 bytes --]

On Fri, 21 Apr 2017, Fabian Grünbichler wrote:
> On Fri, Apr 21, 2017 at 01:16:23PM +0000, Sage Weil wrote:
> > On Fri, 21 Apr 2017, Fabian Grünbichler wrote:
> > > On Thu, Apr 20, 2017 at 08:11:38PM +0200, Nathan Cutler wrote:
> > > > Hi Willem:
> > > > 
> > > > It sounds like you are trying to use the sysvinit scripts? These have been
> > > > unmaintained (presumably with lots of weeds growing up) since infernalis.
> > > > Until now I have been assuming that all init systems other than systemd
> > > > (sysvinit, upstart, etc.) are deprecated in Ceph.
> > > 
> > > Slightly OT, but AFAICT http://tracker.ceph.com/issues/18305 still
> > > applies to the official ceph.com Kraken Debian packages, i.e., ceph-base
> > > installs and activates the init.d script, which then races against the
> > > (udev-activated) ceph-osd systemd units. If anything except systemd is
> > > indeed deprecated, I wonder why the Debian packages (still) ship AND
> > > activate both systemd units and Sys V init scripts?
> > > 
> > > (Note that the proposed fix probably does not apply as is anymore,
> > > because ceph-disk and the systemd units have been changed in the
> > > meantime).
> > 
> > I think the issue with Debian (generally) is that it "supports" multiple 
> > init systems (sysvinit and systemd both), even though systemd is the one 
> > installed default.  Which means we ship the sysvinit script and systemd 
> > unit files.
> > 
> > (There may very well be a bug in how we "activate" them, though!)
> > 
> 
> that's why I reported the original issue - in general you never want to
> have both a multi-daemon init script (like the "old" ceph one) and the
> replacing split up systemd units active at the same time.
> 
> since systemd will generate a unit for every init script for which a
> unit of the same name does not already exist, you either need to mask
> the auto-generated unit (i.e., symlink it to /dev/null) or write a
> replacement unit that has the identical name (so the "ceph" init script
> becomes the "ceph.service" unit). if you don't do that and your units
> are named differently than your init script, both will be active (this
> is not a Debianism, it is how the LSB generator in systemd is supposed
> to work to ease the transition from Sys V init to systemd..).
> 
> what exacerbates the issue in this case is that the systemd units + udev
> actually don't completely replace the old init scripts, because some of
> the udev events might have been processed before the system was fully
> booted, and osds might not be properly activated on boot as a result.
> 
> hence my proposal to add a ceph.service that simply calls "ceph-disk
> activate-all", which is AFAICT the only part of the init script that is
> not covered by the current systemd units / udev rules.

This sounds reasonable to me.  It could also do nothing... IIRC the 
ceph-disk activate-all was a workaround for racy/buggy udev interactions 
that preventing all osds from starting on large boxes with lots of disks 
(see below).  Given that we don't have such a workaround on systemd 
anymore I'm not sure if it's still necessary or not.  (I guess it can't 
hurt, though!)

> the whole ceph service startup is pretty messy in general IMHO,
> especially for OSDs where (IIRC?) udev rules are calling python scripts
> which are starting systemd units which are in turn calling python
> scripts with different parameters that end up starting systemd units
> which actually start daemons (somewhere in there the mounting happens as
> well..).

Agreed.  The goal with using udev like this was to make it all hotplug, 
but I'm not sure if any operators actually take advantage of this.  If 
they don't, we could consider going back to something a bit less weird...

sage

Re: Ceph-deploy for FreeBSD

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3584 bytes --]

On Fri, 21 Apr 2017, Sage Weil wrote:
> On Fri, 21 Apr 2017, Fabian Grünbichler wrote:
> > On Fri, Apr 21, 2017 at 01:16:23PM +0000, Sage Weil wrote:
> > > On Fri, 21 Apr 2017, Fabian Grünbichler wrote:
> > > > On Thu, Apr 20, 2017 at 08:11:38PM +0200, Nathan Cutler wrote:
> > > > > Hi Willem:
> > > > > 
> > > > > It sounds like you are trying to use the sysvinit scripts? These have been
> > > > > unmaintained (presumably with lots of weeds growing up) since infernalis.
> > > > > Until now I have been assuming that all init systems other than systemd
> > > > > (sysvinit, upstart, etc.) are deprecated in Ceph.
> > > > 
> > > > Slightly OT, but AFAICT http://tracker.ceph.com/issues/18305 still
> > > > applies to the official ceph.com Kraken Debian packages, i.e., ceph-base
> > > > installs and activates the init.d script, which then races against the
> > > > (udev-activated) ceph-osd systemd units. If anything except systemd is
> > > > indeed deprecated, I wonder why the Debian packages (still) ship AND
> > > > activate both systemd units and Sys V init scripts?
> > > > 
> > > > (Note that the proposed fix probably does not apply as is anymore,
> > > > because ceph-disk and the systemd units have been changed in the
> > > > meantime).
> > > 
> > > I think the issue with Debian (generally) is that it "supports" multiple 
> > > init systems (sysvinit and systemd both), even though systemd is the one 
> > > installed default.  Which means we ship the sysvinit script and systemd 
> > > unit files.
> > > 
> > > (There may very well be a bug in how we "activate" them, though!)
> > > 
> > 
> > that's why I reported the original issue - in general you never want to
> > have both a multi-daemon init script (like the "old" ceph one) and the
> > replacing split up systemd units active at the same time.
> > 
> > since systemd will generate a unit for every init script for which a
> > unit of the same name does not already exist, you either need to mask
> > the auto-generated unit (i.e., symlink it to /dev/null) or write a
> > replacement unit that has the identical name (so the "ceph" init script
> > becomes the "ceph.service" unit). if you don't do that and your units
> > are named differently than your init script, both will be active (this
> > is not a Debianism, it is how the LSB generator in systemd is supposed
> > to work to ease the transition from Sys V init to systemd..).
> > 
> > what exacerbates the issue in this case is that the systemd units + udev
> > actually don't completely replace the old init scripts, because some of
> > the udev events might have been processed before the system was fully
> > booted, and osds might not be properly activated on boot as a result.
> > 
> > hence my proposal to add a ceph.service that simply calls "ceph-disk
> > activate-all", which is AFAICT the only part of the init script that is
> > not covered by the current systemd units / udev rules.
> 
> This sounds reasonable to me.  It could also do nothing... IIRC the 
> ceph-disk activate-all was a workaround for racy/buggy udev interactions 
> that preventing all osds from starting on large boxes with lots of disks 
> (see below).  Given that we don't have such a workaround on systemd 
> anymore I'm not sure if it's still necessary or not.  (I guess it can't 
> hurt, though!)

Hmm, it's a bit tricky because of the package separation between 
ceph-base, ceph-common, and ceph-osd.  The ceph sysvinit script is in the 
base or common package but ceph-disk is in ceph-osd... I'm not sure where 
the ceph.service masking service should go.

sage

Re: Ceph-deploy for FreeBSD

[Fabian Grünbichler]

On Fri, Apr 21, 2017 at 01:45:05PM +0000, Sage Weil wrote:
> On Fri, 21 Apr 2017, Fabian Grünbichler wrote:
> > On Fri, Apr 21, 2017 at 01:16:23PM +0000, Sage Weil wrote:
> > > On Fri, 21 Apr 2017, Fabian Grünbichler wrote:
> > > > On Thu, Apr 20, 2017 at 08:11:38PM +0200, Nathan Cutler wrote:
> > > > > Hi Willem:
> > > > > 
> > > > > It sounds like you are trying to use the sysvinit scripts? These have been
> > > > > unmaintained (presumably with lots of weeds growing up) since infernalis.
> > > > > Until now I have been assuming that all init systems other than systemd
> > > > > (sysvinit, upstart, etc.) are deprecated in Ceph.
> > > > 
> > > > Slightly OT, but AFAICT http://tracker.ceph.com/issues/18305 still
> > > > applies to the official ceph.com Kraken Debian packages, i.e., ceph-base
> > > > installs and activates the init.d script, which then races against the
> > > > (udev-activated) ceph-osd systemd units. If anything except systemd is
> > > > indeed deprecated, I wonder why the Debian packages (still) ship AND
> > > > activate both systemd units and Sys V init scripts?
> > > > 
> > > > (Note that the proposed fix probably does not apply as is anymore,
> > > > because ceph-disk and the systemd units have been changed in the
> > > > meantime).
> > > 
> > > I think the issue with Debian (generally) is that it "supports" multiple 
> > > init systems (sysvinit and systemd both), even though systemd is the one 
> > > installed default.  Which means we ship the sysvinit script and systemd 
> > > unit files.
> > > 
> > > (There may very well be a bug in how we "activate" them, though!)
> > > 
> > 
> > that's why I reported the original issue - in general you never want to
> > have both a multi-daemon init script (like the "old" ceph one) and the
> > replacing split up systemd units active at the same time.
> > 
> > since systemd will generate a unit for every init script for which a
> > unit of the same name does not already exist, you either need to mask
> > the auto-generated unit (i.e., symlink it to /dev/null) or write a
> > replacement unit that has the identical name (so the "ceph" init script
> > becomes the "ceph.service" unit). if you don't do that and your units
> > are named differently than your init script, both will be active (this
> > is not a Debianism, it is how the LSB generator in systemd is supposed
> > to work to ease the transition from Sys V init to systemd..).
> > 
> > what exacerbates the issue in this case is that the systemd units + udev
> > actually don't completely replace the old init scripts, because some of
> > the udev events might have been processed before the system was fully
> > booted, and osds might not be properly activated on boot as a result.
> > 
> > hence my proposal to add a ceph.service that simply calls "ceph-disk
> > activate-all", which is AFAICT the only part of the init script that is
> > not covered by the current systemd units / udev rules.
> 
> This sounds reasonable to me.  It could also do nothing... IIRC the 
> ceph-disk activate-all was a workaround for racy/buggy udev interactions 
> that preventing all osds from starting on large boxes with lots of disks 
> (see below).  Given that we don't have such a workaround on systemd 
> anymore I'm not sure if it's still necessary or not.  (I guess it can't 
> hurt, though!)

I am sorry if that was not clear enough - the incompleteness I mentioned
in my previous mail is not theoretical, if I "systemctl mask
ceph.service" (which disables the init script) on a Debian Jessie /
Ceph Jewel based system, not all OSDs will be activated on boot (in
fact, most of the time none will). The ceph-osd services are only
started if a udev add event for a OSD partition happens late enough in
the boot process (e.g., if I hot unplug and replug the OSD disk, they
are correctly started). I last tested this around 10.2.5 (and have had
the "ceph-disk activate-all" ceph.service in place since), but it was
very reproducible.

> 
> > the whole ceph service startup is pretty messy in general IMHO,
> > especially for OSDs where (IIRC?) udev rules are calling python scripts
> > which are starting systemd units which are in turn calling python
> > scripts with different parameters that end up starting systemd units
> > which actually start daemons (somewhere in there the mounting happens as
> > well..).
> 
> Agreed.  The goal with using udev like this was to make it all hotplug, 
> but I'm not sure if any operators actually take advantage of this.  If 
> they don't, we could consider going back to something a bit less weird...
> 
> sage

I am undecided on this - the current state is far from elegant, but
OTOH, simply moving OSD disks between hosts and having them work OOTB is
a nice feature...

Re: Ceph-deploy for FreeBSD

[Willem Jan Withagen]

On 21-04-17 15:52, Sage Weil wrote:
> On Fri, 21 Apr 2017, Sage Weil wrote:
>> On Fri, 21 Apr 2017, Fabian Grünbichler wrote:
>>> On Fri, Apr 21, 2017 at 01:16:23PM +0000, Sage Weil wrote:
>>>> On Fri, 21 Apr 2017, Fabian Grünbichler wrote:
>>>>> On Thu, Apr 20, 2017 at 08:11:38PM +0200, Nathan Cutler wrote:
>>>>>> Hi Willem:
>>>>>>
>>>>>> It sounds like you are trying to use the sysvinit scripts? These have been
>>>>>> unmaintained (presumably with lots of weeds growing up) since infernalis.
>>>>>> Until now I have been assuming that all init systems other than systemd
>>>>>> (sysvinit, upstart, etc.) are deprecated in Ceph.
>>>>>
>>>>> Slightly OT, but AFAICT http://tracker.ceph.com/issues/18305 still
>>>>> applies to the official ceph.com Kraken Debian packages, i.e., ceph-base
>>>>> installs and activates the init.d script, which then races against the
>>>>> (udev-activated) ceph-osd systemd units. If anything except systemd is
>>>>> indeed deprecated, I wonder why the Debian packages (still) ship AND
>>>>> activate both systemd units and Sys V init scripts?
>>>>>
>>>>> (Note that the proposed fix probably does not apply as is anymore,
>>>>> because ceph-disk and the systemd units have been changed in the
>>>>> meantime).
>>>>
>>>> I think the issue with Debian (generally) is that it "supports" multiple 
>>>> init systems (sysvinit and systemd both), even though systemd is the one 
>>>> installed default.  Which means we ship the sysvinit script and systemd 
>>>> unit files.
>>>>
>>>> (There may very well be a bug in how we "activate" them, though!)
>>>>
>>>
>>> that's why I reported the original issue - in general you never want to
>>> have both a multi-daemon init script (like the "old" ceph one) and the
>>> replacing split up systemd units active at the same time.
>>>
>>> since systemd will generate a unit for every init script for which a
>>> unit of the same name does not already exist, you either need to mask
>>> the auto-generated unit (i.e., symlink it to /dev/null) or write a
>>> replacement unit that has the identical name (so the "ceph" init script
>>> becomes the "ceph.service" unit). if you don't do that and your units
>>> are named differently than your init script, both will be active (this
>>> is not a Debianism, it is how the LSB generator in systemd is supposed
>>> to work to ease the transition from Sys V init to systemd..).
>>>
>>> what exacerbates the issue in this case is that the systemd units + udev
>>> actually don't completely replace the old init scripts, because some of
>>> the udev events might have been processed before the system was fully
>>> booted, and osds might not be properly activated on boot as a result.
>>>
>>> hence my proposal to add a ceph.service that simply calls "ceph-disk
>>> activate-all", which is AFAICT the only part of the init script that is
>>> not covered by the current systemd units / udev rules.
>>
>> This sounds reasonable to me.  It could also do nothing... IIRC the 
>> ceph-disk activate-all was a workaround for racy/buggy udev interactions 
>> that preventing all osds from starting on large boxes with lots of disks 
>> (see below).  Given that we don't have such a workaround on systemd 
>> anymore I'm not sure if it's still necessary or not.  (I guess it can't 
>> hurt, though!)
> 
> Hmm, it's a bit tricky because of the package separation between 
> ceph-base, ceph-common, and ceph-osd.  The ceph sysvinit script is in the 
> base or common package but ceph-disk is in ceph-osd... I'm not sure where 
> the ceph.service masking service should go.

Well that is certainly something I cannot test, nor repair.
So that has to go on somebody else's plate.

--WjW

Re: Ceph-deploy for FreeBSD

[Fabian Grünbichler]

On Fri, Apr 21, 2017 at 01:52:45PM +0000, Sage Weil wrote:
> On Fri, 21 Apr 2017, Sage Weil wrote:
> > On Fri, 21 Apr 2017, Fabian Grünbichler wrote:
> > > On Fri, Apr 21, 2017 at 01:16:23PM +0000, Sage Weil wrote:
> > > > On Fri, 21 Apr 2017, Fabian Grünbichler wrote:
> > > > > On Thu, Apr 20, 2017 at 08:11:38PM +0200, Nathan Cutler wrote:
> > > > > > Hi Willem:
> > > > > > 
> > > > > > It sounds like you are trying to use the sysvinit scripts? These have been
> > > > > > unmaintained (presumably with lots of weeds growing up) since infernalis.
> > > > > > Until now I have been assuming that all init systems other than systemd
> > > > > > (sysvinit, upstart, etc.) are deprecated in Ceph.
> > > > > 
> > > > > Slightly OT, but AFAICT http://tracker.ceph.com/issues/18305 still
> > > > > applies to the official ceph.com Kraken Debian packages, i.e., ceph-base
> > > > > installs and activates the init.d script, which then races against the
> > > > > (udev-activated) ceph-osd systemd units. If anything except systemd is
> > > > > indeed deprecated, I wonder why the Debian packages (still) ship AND
> > > > > activate both systemd units and Sys V init scripts?
> > > > > 
> > > > > (Note that the proposed fix probably does not apply as is anymore,
> > > > > because ceph-disk and the systemd units have been changed in the
> > > > > meantime).
> > > > 
> > > > I think the issue with Debian (generally) is that it "supports" multiple 
> > > > init systems (sysvinit and systemd both), even though systemd is the one 
> > > > installed default.  Which means we ship the sysvinit script and systemd 
> > > > unit files.
> > > > 
> > > > (There may very well be a bug in how we "activate" them, though!)
> > > > 
> > > 
> > > that's why I reported the original issue - in general you never want to
> > > have both a multi-daemon init script (like the "old" ceph one) and the
> > > replacing split up systemd units active at the same time.
> > > 
> > > since systemd will generate a unit for every init script for which a
> > > unit of the same name does not already exist, you either need to mask
> > > the auto-generated unit (i.e., symlink it to /dev/null) or write a
> > > replacement unit that has the identical name (so the "ceph" init script
> > > becomes the "ceph.service" unit). if you don't do that and your units
> > > are named differently than your init script, both will be active (this
> > > is not a Debianism, it is how the LSB generator in systemd is supposed
> > > to work to ease the transition from Sys V init to systemd..).
> > > 
> > > what exacerbates the issue in this case is that the systemd units + udev
> > > actually don't completely replace the old init scripts, because some of
> > > the udev events might have been processed before the system was fully
> > > booted, and osds might not be properly activated on boot as a result.
> > > 
> > > hence my proposal to add a ceph.service that simply calls "ceph-disk
> > > activate-all", which is AFAICT the only part of the init script that is
> > > not covered by the current systemd units / udev rules.
> > 
> > This sounds reasonable to me.  It could also do nothing... IIRC the 
> > ceph-disk activate-all was a workaround for racy/buggy udev interactions 
> > that preventing all osds from starting on large boxes with lots of disks 
> > (see below).  Given that we don't have such a workaround on systemd 
> > anymore I'm not sure if it's still necessary or not.  (I guess it can't 
> > hurt, though!)
> 
> Hmm, it's a bit tricky because of the package separation between 
> ceph-base, ceph-common, and ceph-osd.  The ceph sysvinit script is in the 
> base or common package but ceph-disk is in ceph-osd... I'm not sure where 
> the ceph.service masking service should go.
> 

well, both variants (ceph-base and ceph-osd) make sense, and I don't
think that the bottom-line changes much?

ceph-base:
- init script is already there, and does only work when ceph-mon,
  ceph-osd etc are installed as well
- future packaging modifications won't miss the connection between init
  script and override service

ceph-osd:
- arguably where it belongs when looking at the functionality it
  provides