Re: Rook orchestrator module

[Jason Dillaman <jdillama@redhat.com>]

On Tue, Sep 29, 2020 at 3:33 PM Travis Nielsen <tnielsen@redhat.com> wrote:
>
> Sebastian and fellow orchestrators,
>
> Some questions have come up recently about issues in the Rook
> orchestrator module and its state of disrepair. Patrick, Varsha, and I
> have been discussing these recently as Varsha has been working on the
> module. Before we fix all the issues that are being found, I want to
> start a higher level conversation. I’ll join the leads meeting
> tomorrow to discuss, and would be good to include in the Monday
> orchestrator agenda as well, which unfortunately I haven’t been able
> to attend recently...
>
> First, Rook is driven by the K8s APIs, including CRDs, an operator,
> the CSI driver, etc. When the admin needs to configure the Ceph
> cluster, they create the CRDs and other resources directly with the
> K8s tools such as kubectl. Rook does everything with K8s patterns so
> that the admin doesn’t need to leave their standard administration
> sandbox in order to configure Rook or Ceph. If any Ceph-specific
> command needs to be run, the rook toolbox can be used. However, we
> prefer to avoid the toolbox for common scenarios that should have CRDs
> for declaring desired state.
>
> The fundamental question then is, **what scenarios require the Rook
> orchestrator mgr module**? The module is not enabled by default in
> Rook clusters and I am not aware of upstream users consuming it.
>
> The purpose of the orchestrator module was originally to provide a
> common entry point either for Ceph CLI tools or the dashboard. This
> would provide the constant interface to work with both Rook or cephadm
> clusters. Patrick pointed out that the dashboard isn’t really a
> scenario anymore for the orchestrator module.

Is that true? [1]

> If so, the only
> remaining usage is for CLI tools. And if we only have the CLI
> scenario, this means that the CLI commands would be run from the
> toolbox. But we are trying to avoid the toolbox. We should be putting
> our effort into the CRDs, CSI driver, etc.
>
> If the orchestrator module is creating CRs, we are likely doing
> something wrong. We expect the cluster admin to create CRs.
>
> Thus, I’d like to understand the scenarios where the rook orchestrator
> module is needed. If there isn’t a need anymore since dashboard
> requirements have changed, I’d propose the module can be removed.

I don't have a current stake in the outcome, but I could foresee the
future need/desire for letting the Ceph cluster itself spin up
resources on-demand in k8s via Rook. Let's say that I want to convert
an XFS on RBD image to CephFS, the MGR could instruct the orchestrator
to kick off a job to translate between the two formats. I'd imagine
the same could be argued for on-demand NFS/SMB gateways or anywhere
else there is a delta between a storage administrator setting up the
basic Ceph system and Ceph attempting to self-regulate/optimize.

> Thanks,
> Travis
> Rook
>

[1] https://tracker.ceph.com/issues/46756

-- 
Jason