I’m not sure how I missed this earlier in the lists but having done a lot of work on Ceph helm charts, this is of definite interest to us. We’ve been running various states of Ceph in Docker and Kubernetes (in production environments) for over a year now.

There is a lot of overlap between the Rook and the Ceph related projects (ceph-docker/ceph-container and now ceph-helm) and I agree with Bassam about finding ways of bringing things closer. Having felt (and contributed to) the pain of the complex ceph-container entrypoint scripts, the importance of the simpler initial configuration and user experience with Rook and it’s Operator approach can’t be understated. There is a definite need for a vanilla project to run Ceph on Kubernetes but the most useful part lies in encapsulating the day-to-day operation of a Ceph cluster that builds on the strengths of Kubernetes (CRDs, Operators, dynamic scaling, etc).

Looking forward to seeing where this goes (and joining the discussion)

Hunter

On Sat, Nov 4, 2017 at 5:13 AM Bassam Tabbara <bassam-7zw3Wi+BkiFBDgjK7y7TUQ@public.gmane.org> wrote:

(sorry for the late response, just catching up on ceph-users)

> Probably the main difference is that ceph-helm aims to run Ceph as part of
> the container infrastructure. The containers are privileged so they can
> interact with hardware where needed (e.g., lvm for dm-crypt) and the
> cluster runs on the host network. We use kubernetes some orchestration:
> kube is a bit of a headache for mons and osds but will be very helpful for
> scheduling everything else: mgrs, rgw, rgw-nfs, iscsi, mds, ganesha,
> samba, rbd-mirror, etc.
>
> Rook, as I understand it at least (the rook folks on the list can speak up
> here), aims to run Ceph more as a tenant of kubernetes. The cluster runs
> in the container network space, and the aim is to be able to deploy ceph
> more like an unprivileged application on e.g., a public cloud providing
> kubernetes as the cloud api.

Yes Rook’s goal is to run wherever Kubernetes runs without making changes at the host level. Eventually we plan to remove the need to run some of the containers privileged, and automatically work with different kernel versions and heterogeneous environments. It's fair to think of Rook as an application of Kubernetes. As a result you could run it on AWS, Google, bare-metal or wherever.

> The other difference is around rook-operator, which is the thing that lets
> you declare what you want (ceph clusters, pools, etc) via kubectl and goes
> off and creates the cluster(s) and tells it/them what to do. It makes the
> storage look like it is tightly integrated with and part of kubernetes but
> means that kubectl becomes the interface for ceph cluster management.

Rook extends Kubernetes to understand storage concepts like Pool, Object Store, FileSystems. Our goal is for storage to be integrated deeply into Kuberentes. That said, you can easily launch the Rook toolbox and use the ceph tools at any point. I don’t think the goal is for Rook to replace the ceph tools, but instead to offer a Kuberentes-native alternative to them.

> Some of that seems useful to me (still developing opinions here!) and
> perhaps isn't so different than the declarations in your chart's
> values.yaml but I'm unsure about the wisdom of going too far down the road
> of administering ceph via yaml.
>
> Anyway, I'm still pretty new to kubernetes-land and very interested in
> hearing what people are interested in or looking for here!

Would be great to find ways to get these two projects closer.

Bassam

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html