Puppet modules for Ceph

Puppet modules for Ceph

[François Charlier]

Hi,

I'm currently working on writing a Puppet module for Ceph.

As after some research I found no existing module, I'll start from
scratch but I would be glad to hear from people who would already have
started working or this or having any idea or pointers regarding this
subject.

Thanks,

[ By the way, I'm fc on #ceph ! ]
-- 
François Charlier                     Software Engineer
// eNovance labs               http://labs.enovance.com
// ✉ francois.charlier@enovance.com ☎ +33 1 49 70 99 81
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Puppet modules for Ceph

[Mark Nelson]

On 7/18/12 8:58 AM, François Charlier wrote:
> Hi,
>
> I'm currently working on writing a Puppet module for Ceph.
>
> As after some research I found no existing module, I'll start from
> scratch but I would be glad to hear from people who would already have
> started working or this or having any idea or pointers regarding this
> subject.
>
> Thanks,
>
> [ By the way, I'm fc on #ceph ! ]
>

Hi Francois,

That's great!  You might want to look at the chef work that has been 
done as a base to start from.  I'm not very familiar with what is in 
place, but Tommi or Dan may chime in later with more details.  Some of 
the folks from Mediawiki were actually just talking about puppet modules 
yesterday on the IRC channel so they may be interested in collaborating too.

Thanks,
Mark
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Puppet modules for Ceph

[François Charlier]

On Jul 18 09:41, Mark Nelson wrote:
> On 7/18/12 8:58 AM, François Charlier wrote:
> >Hi,
> >
> >I'm currently working on writing a Puppet module for Ceph.
> >
> >As after some research I found no existing module, I'll start from
> >scratch but I would be glad to hear from people who would already have
> >started working or this or having any idea or pointers regarding this
> >subject.
> >
> >Thanks,
> >
> >[ By the way, I'm fc on #ceph ! ]
> >
> 
> Hi Francois,
> 
> That's great!  You might want to look at the chef work that has been
> done as a base to start from.  I'm not very familiar with what is in
> place, but Tommi or Dan may chime in later with more details.  Some
> of the folks from Mediawiki were actually just talking about puppet
> modules yesterday on the IRC channel so they may be interested in
> collaborating too.
> 
> Thanks,
> Mark

Hi,

I'd like to keep you updated on the progress I made so far :

- the module is published on github: https://github.com/fcharlier/puppet-ceph

- it's currently (only) able to install some MONs

- I hope to be able to add support for OSDs before next tuesday (because
  I'm on vacation after that …)

- I listed the necessary “manual” actions needed to install a Ceph
  cluster (gathered from the Ceph Chef Cookbook and the mkcephfs script)
  on our wiki : https://labs.enovance.com/projects/puppet/wiki/Puppet-ceph
  but it might still contain some errors.

- I've decided to stick with exported resources to transfer information
  between the nodes.

Any comment or contribution would be much appreciated !

Sincerely,
-- 
François Charlier                     Software Engineer
// eNovance labs               http://labs.enovance.com
// ✉ francois.charlier@enovance.com ☎ +33 1 49 70 99 81
// fc on OFTC / fc___ on FreeNode
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Puppet modules for Ceph

[Tommi Virtanen]

On Wed, Jul 18, 2012 at 6:58 AM, François Charlier
<francois.charlier@enovance.com> wrote:
> I'm currently working on writing a Puppet module for Ceph.
>
> As after some research I found no existing module, I'll start from
> scratch but I would be glad to hear from people who would already have
> started working or this or having any idea or pointers regarding this
> subject.

Hi. I don't remember anyone actively working on puppet modules for
Ceph. A quick search gives me just this:

http://git.sans.ethz.ch/?p=puppet-modules/ceph;a=summary

The Chef cookbook at https://github.com/ceph/ceph-cookbooks is
starting to get into a pretty good stage. It radically changes how we
do deployment and management, so I'd recommend you look at it in
detail, and don't imitate mkcephfs. We've been actively changing core
Ceph to make deployment and management simpler; I think the best proof
of that is that the cookbook is already shorter than the mkcephfs
shell script, and will probably just become a thinner layer in the
future. The Juju charms for Ceph are also adopting a model quite close
to what the Chef cookbook does.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Puppet modules for Ceph

[Tommi Virtanen]

On Wed, Jul 18, 2012 at 2:59 PM, Teyo Tyree <teyo@puppetlabs.com> wrote:
> As you probably know, Puppet Labs is based in Portland. Are you attending
> OScon? It might be a good opportunity for us to have some face to face
> hacking time on a Puppet module. Let me know if you would like for us to
> arrange sometime to get together if you are in town.

Sorry, I'm not at OScon, I'm intentionally limiting my travel right
now. A large chunk of Inktank is based in Los Angeles, so we're not
far away even outside of conferences.

Frankly, we still have a bit of cleanup work to do on the chef
cookbook side, and you'd probably be most productive writing puppet
modules once that stuff is all flushed out. Soon, we'll start to
de-emphasize mkcephfs in favor of other, more flexible, deployment
mechanisms; I think bringing together some Puppet, Juju and Chef
experts at that point would be most beneficial.

Re: Puppet modules for Ceph

[Tommi Virtanen]

On Wed, Jul 18, 2012 at 3:26 PM, Teyo Tyree <teyo@puppetlabs.com> wrote:
> Ha, that would be an interesting experiment indeed. I think Francois would
> like to have the Puppet module done sooner rather than later. Are the
> current Chef cookbooks functional enough for us to get started with them as
> a reference?

I think so. They Work For Me(tm). The ugly stuff is mostly things like
needing to wait a few rounds due to Chef's asynchronous data store,
and it's missing just about all internal documentation; the
user-visible aspects have a decent write-up, but nothing explains e.g.
what exactly /var/lib/ceph/bootstrap-osd/ is about.

I'd love to help you work through that though, so please keep talking
to me and make me explain everything in enough detail. It's just that
I don't have anything except source to give you right now.

The end user Chef deployment docs are at
http://ceph.com/docs/master/install/chef/
http://ceph.com/docs/master/config-cluster/chef/

The cookbook is at https://github.com/ceph/ceph-cookbooks and
currently it assumes Ubuntu 12.04.

Re: Puppet modules for Ceph

[Tommi Virtanen]

On Tue, Jul 24, 2012 at 6:15 AM,  <loic.dachary@enovance.com> wrote:
> Note that if puppet client was run on nodeB before it was run on nodeA, all
> three steps would have been run in sequence instead of being spread over two
> puppet client invocations.

Unfortunately, even that is not enough.

The relevant keys (client.admin, client.bootstrap-osd, later
bootstrap-mds radosgw etc also) can only be created once the mons have
reached quorum. This is some time after they have started, even in the
best case. Making the puppet/chef run wait for that sounds like a bad
idea; especially since I use further chef-client runs to feed ceph-mon
information about its peers, which may be necessary for it to ever
reach quorum.

While I can find ways of making the key generation happen as soon as
quorum is reached, communicating the keys to other nodes only happens
at the mercy of the configuration management system; both puppet and
chef seem to be in the mindset of "run every N minutes" option. So
even if we generate the keys best case 2 seconds after ceph-mon
startup, it needs a full configuration manager run on the source node,
and then a run on the destination node, before OSD bring-up etc can
succeed.

I have found no satisfying solution to this, so far.

Re: Puppet modules for Ceph

[Sage Weil]

On Tue, 31 Jul 2012, Tommi Virtanen wrote:
> On Tue, Jul 24, 2012 at 6:15 AM,  <loic.dachary@enovance.com> wrote:
> > Note that if puppet client was run on nodeB before it was run on nodeA, all
> > three steps would have been run in sequence instead of being spread over two
> > puppet client invocations.
> 
> Unfortunately, even that is not enough.
> 
> The relevant keys (client.admin, client.bootstrap-osd, later
> bootstrap-mds radosgw etc also) can only be created once the mons have
> reached quorum. This is some time after they have started, even in the
> best case. Making the puppet/chef run wait for that sounds like a bad
> idea; especially since I use further chef-client runs to feed ceph-mon
> information about its peers, which may be necessary for it to ever
> reach quorum.
> 
> While I can find ways of making the key generation happen as soon as
> quorum is reached, communicating the keys to other nodes only happens
> at the mercy of the configuration management system; both puppet and
> chef seem to be in the mindset of "run every N minutes" option. So
> even if we generate the keys best case 2 seconds after ceph-mon
> startup, it needs a full configuration manager run on the source node,
> and then a run on the destination node, before OSD bring-up etc can
> succeed.
> 
> I have found no satisfying solution to this, so far.

It is also possible to feed initial keys to the monitors during the 'mkfs' 
stage.  If the keys can be agreed on somehow beforehand, then they will 
already be in place when the initial quorum is reached.  Not sure if that 
helps in this situation or not...

sage

Re: Puppet modules for Ceph

[Tommi Virtanen]

On Tue, Jul 31, 2012 at 11:51 AM, Sage Weil <sage@inktank.com> wrote:
> It is also possible to feed initial keys to the monitors during the 'mkfs'
> stage.  If the keys can be agreed on somehow beforehand, then they will
> already be in place when the initial quorum is reached.  Not sure if that
> helps in this situation or not...

Yeah, we're going that way for the "mon." key in the chef cookbooks
(to get the mons talking to each other at all, that *has* to be done
that way), but putting more and more stuff there is not very nice.

Your typical CM framework does not let the recipe run arbitrary code
at that sort of an instantiation time, and pushing this work on the
admin makes it laborous and brittle; what happens when we need a new
type of a bootstrap-foo key? Get all admins to cram an extra entry
into their environment json?
http://ceph.com/docs/master/config-cluster/chef/#configure-your-ceph-environment

That just does not seem like a good way.

Juju seems to provide a real-time notification mechanism between
peers, using it's <name>-relation-changed hook. Other CM frameworks
may need to step up their game, or be subject to the "keep re-running
chef-client until it works" limitation.

If the CM makes it safe to trigger a run manually (e.g. "sudo
chef-client" whenever you feel like it), we can trigger that locally
when we finally create the keys. This still doesn't help the receiving
side to notice any faster.

If the CM makes it safe for us to change node attributes outside of
the full CM run, we can do trigger that when we finally create the
keys. Chef seems to have a "full overwrites only" semantic, so this is
probably not safe with it. And as above, this does not help the
receiving side to notice that it has information to fetch.

What I want to do longer term, is make the Chef cookbook for Ceph very
thin, push everything except the cross-node communication into Ceph
proper, and then write a "mkcephfs v2.0" that uses SSH connections as
appropriate, from a central "workstation" host that can SSH anywhere,
to trigger these actions ASAP. Then that becomes the goal for CM
frameworks: "provide me a communication mechanism between these nodes
that can do *this*".

Re: Puppet modules for Ceph

[Jonathan D. Proulx]

On Tue, Jul 31, 2012 at 12:09:13PM -0700, Tommi Virtanen wrote:

:Juju seems to provide a real-time notification mechanism between
:peers, using it's <name>-relation-changed hook. Other CM frameworks
:may need to step up their game, or be subject to the "keep re-running
:chef-client until it works" limitation.

Have you lloked at "puppet kick"

puppet-kick(8) -- Remotely control puppet agent
========

SYNOPSIS
--------
Trigger a puppet agent run on a set of hosts.

I'm not using it in my environment and it does require some specific
puppet config on the agent side to make it work so less than ideal,
but might atleast provide a path.

-Jon

Re: Puppet modules for Ceph

[Tommi Virtanen]

On Tue, Aug 7, 2012 at 6:51 AM, Jonathan D. Proulx <jon@csail.mit.edu> wrote:
> :Juju seems to provide a real-time notification mechanism between
> :peers, using it's <name>-relation-changed hook. Other CM frameworks
> :may need to step up their game, or be subject to the "keep re-running
> :chef-client until it works" limitation.
>
> Have you lloked at "puppet kick"
>
> puppet-kick(8) -- Remotely control puppet agent

I haven't really used puppet, but that doesn't look like it's peer to
peer, the manpage talks about reading the "master's" config file. I
may be wrong in that.

The other nice thing Juju is let you transfer data between the peers
in a relationship -- e.g. we could transport the crypto secret to a
new OSD that way.

Re: Puppet modules for Ceph

[Jimmy Tang]

Hi Francois

On 18 Jul 2012, at 14:58, François Charlier wrote:

> Hi,
> 
> I'm currently working on writing a Puppet module for Ceph.
> 
> As after some research I found no existing module, I'll start from
> scratch but I would be glad to hear from people who would already have
> started working or this or having any idea or pointers regarding this
> subject.
> 
> Thanks,
> 



+1 for puppet modules.

I'm also interested in the possibility of puppet modules for ceph, we've been looking at evaluating ceph here at our site and one of the things we'd like to do is automate deployments in vagrant vm's for testing/development.

Thanks,
Jimmy.
--
Senior Software Engineer, Digital Repository of Ireland (DRI)
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
http://www.tchpc.tcd.ie/

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html